Noblogs archive uploader

These scripts upload WACZ archives to a dufs server. Uploads that stop partway through are resumed automatically, and files that already exist on the server with the correct size are skipped.

Requirements

• Bash
• curl
• GNU coreutils (dd, realpath, sha512sum, and stat)
• awk

Configuration

Create the local environment file from the provided example:

cp dufs.env.example dufs.env

Edit dufs.env and set the server URL, credentials, and archive directory:

DUFS_SERVER_URL="https://files.server.tld"
DUFS_USERNAME="username"
DUFS_PASSWORD="password"
NOBLOGS_DIR="noblogs"

The real dufs.env is ignored by Git. Do not commit credentials. Keep this configuration file in the uploader project directory next to the scripts.

NOBLOGS_DIR is resolved relative to the directory from which a script is called, not relative to the script itself. The scripts can therefore be called from outside the project directory. An absolute path can also be used when the archive directory should not depend on the current working directory.

For end users, keep the archive directory outside the uploader project so the archive data remains separate from the repository. With the default NOBLOGS_DIR="noblogs", change to the directory containing noblogs before running a script. A recommended layout is:

/path/to/
├── noblogs-uploader/       # This project
└── archive-data/           # Current working directory
    └── noblogs/            # NOBLOGS_DIR
        └── seg_du/
            ├── noblogs-example-one.wacz
            ├── noblogs-example-two.wacz
            └── SHA512SUMS

Upload a segment

From /path/to/archive-data, pass the segment directory name to upload-segment.sh:

/path/to/noblogs-uploader/upload-segment.sh seg_du

Before uploading, the script creates the segment's SHA512SUMS file if it is missing. It regenerates the file when any .wacz file has a newer modification time. It then uploads every .wacz file in $NOBLOGS_DIR/seg_du, followed by SHA512SUMS.

Checksums can also be refreshed separately:

/path/to/noblogs-uploader/compute-checksums.sh seg_du

It is safe to run the segment upload again after adding files. Archives already present on the server with the correct size are skipped.

Upload one domain

From /path/to/archive-data, pass the segment directory name and WACZ filename to upload-domain.sh:

/path/to/noblogs-uploader/upload-domain.sh seg_du noblogs-example-one.wacz

The script first compares the local and remote sizes. It skips a complete remote file, otherwise starts an upload and repeatedly resumes from the remote size until the uploaded file is complete.

Upload files reported as missing

From /path/to/archive-data, run upload-missing.sh to fetch the current noblogs/missing.txt from the DUFS server and upload locally available files reported by that list:

/path/to/noblogs-uploader/upload-missing.sh

The missing list contains entries in this format:

Domain example-one is missing in seg_du.

For each entry, the script looks for these files in $NOBLOGS_DIR:

noblogs-example-one.wacz
noblogs-example-one.wacz.sha512

Each file that exists is uploaded to the corresponding remote segment directory, such as noblogs/seg_du/. Files that are not available locally are skipped. During a normal run, the downloaded list replaces $NOBLOGS_DIR/missing.txt.

Before uploading a WACZ file, the script compares its local size with the remote size. Complete files are skipped, and partial files resume from the number of bytes already stored on the server. If a remote WACZ file is larger than its local counterpart, it is restarted from the beginning. Checksum files are also skipped when their sizes match, but a mismatched checksum is small enough to be uploaded again in full.

To use the existing local $NOBLOGS_DIR/missing.txt without downloading it again, pass --reuse-missing:

/path/to/noblogs-uploader/upload-missing.sh --reuse-missing

Use --dry-run to print each local source and remote destination without uploading anything:

/path/to/noblogs-uploader/upload-missing.sh --dry-run

Dry-run mode checks remote sizes and only prints files that need to be uploaded or resumed. It still fetches the current list from the server, but does not replace the local missing.txt file. The --dry-run and --reuse-missing options can be used together.

License

This project is dedicated to the public domain under the CC0 1.0 Universal public domain dedication.