Noblogs archive uploader

These scripts upload WACZ archives to a dufs server. Uploads that stop partway through are resumed automatically, and files that already exist on the server with the correct size are skipped.

Requirements

• Bash
• curl
• GNU coreutils (dd, realpath, sha512sum, and stat)
• awk

Configuration

Create the local environment file from the provided example:

cp dufs.env.example dufs.env

Edit dufs.env and set the server URL, credentials, and archive directory:

DUFS_SERVER_URL="https://files.server.tld"
DUFS_USERNAME="username"
DUFS_PASSWORD="password"
NOBLOGS_DIR="noblogs"

The real dufs.env is ignored by Git. Do not commit credentials. Keep this configuration file in the uploader project directory next to the scripts.

NOBLOGS_DIR is resolved relative to the directory from which a script is called, not relative to the script itself. The scripts can therefore be called from outside the project directory. An absolute path can also be used when the archive directory should not depend on the current working directory.

For end users, keep the archive directory outside the uploader project so the archive data remains separate from the repository. With the default NOBLOGS_DIR="noblogs", change to the directory containing noblogs before running a script. A recommended layout is:

/path/to/
├── noblogs-uploader/       # This project
└── archive-data/           # Current working directory
    └── noblogs/            # NOBLOGS_DIR
        └── seg_du/
            ├── noblogs-example-one.wacz
            ├── noblogs-example-two.wacz
            └── SHA512SUMS

Upload a segment

From /path/to/archive-data, pass the segment directory name to upload-segment.sh:

/path/to/noblogs-uploader/upload-segment.sh seg_du

Before uploading, the script creates the segment's SHA512SUMS file if it is missing. It regenerates the file when any .wacz file has a newer modification time. It then uploads every .wacz file in $NOBLOGS_DIR/seg_du, followed by SHA512SUMS.

Checksums can also be refreshed separately:

/path/to/noblogs-uploader/compute-checksums.sh seg_du

It is safe to run the segment upload again after adding files. Archives already present on the server with the correct size are skipped.

Upload one domain

From /path/to/archive-data, pass the segment directory name and WACZ filename to upload-domain.sh:

/path/to/noblogs-uploader/upload-domain.sh seg_du noblogs-example-one.wacz

The script first compares the local and remote sizes. It skips a complete remote file, otherwise starts an upload and repeatedly resumes from the remote size until the uploaded file is complete.

License

This project is dedicated to the public domain under the CC0 1.0 Universal public domain dedication.