diff --git a/Dockerfile b/Dockerfile index 3d5874d..837e430 100644 --- a/Dockerfile +++ b/Dockerfile @@ -7,8 +7,8 @@ RUN apt-get -y update && \ apt-get -y install --no-install-recommends \ git vim parted \ quilt coreutils qemu-user-static debootstrap zerofree zip dosfstools \ - bsdtar libcap2-bin rsync grep udev xz-utils curl xxd file kmod bc\ - binfmt-support ca-certificates \ + libarchive-tools libcap2-bin rsync grep udev xz-utils curl xxd file kmod bc\ + binfmt-support ca-certificates qemu-utils kpartx \ && rm -rf /var/lib/apt/lists/* COPY . /pi-gen/ diff --git a/README.md b/README.md index 1563bd8..da4c7a1 100644 --- a/README.md +++ b/README.md @@ -14,7 +14,8 @@ To install the required dependencies for `pi-gen` you should run: ```bash apt-get install coreutils quilt parted qemu-user-static debootstrap zerofree zip \ -dosfstools bsdtar libcap2-bin grep rsync xz-utils file git curl bc +dosfstools libarchive-tools libcap2-bin grep rsync xz-utils file git curl bc \ +qemu-utils kpartx ``` The file `depends` contains a list of tools needed. The format of this @@ -36,7 +37,30 @@ The following environment variables are supported: but you should use something else for a customized version. Export files in stages may add suffixes to `IMG_NAME`. - * `RELEASE` (Default: buster) +* `USE_QCOW2`(Default: `1` ) + + Instead of using traditional way of building the rootfs of every stage in + single subdirectories and copying over the previous one to the next one, + qcow2 based virtual disks with backing images are used in every stage. + This speeds up the build process and reduces overall space consumption + significantly. + + Additional optional parameters regarding qcow2 build: + + * `BASE_QCOW2_SIZE` (Default: 12G) + + Size of the virtual qcow2 disk. + Note: it will not actually use that much of space at once but defines the + maximum size of the virtual disk. If you change the build process by adding + a lot of bigger packages or additional build stages, it can be necessary to + increase the value because the virtual disk can run out of space like a normal + hard drive would. + + **CAUTION:** Although the qcow2 build mechanism will run fine inside Docker, it can happen + that the network block device is not disconnected correctly after the Docker process has + ended abnormally. In that case see [Disconnect an image if something went wrong](#Disconnect-an-image-if-something-went-wrong) + +* `RELEASE` (Default: buster) The release version to build images against. Valid values are jessie, stretch buster, bullseye, and testing. @@ -240,6 +264,10 @@ fix is to ensure `binfmt-support` is installed on the host machine before starting the `./build-docker.sh` script (or using your own docker build solution). +### Passing arguments to Docker + +When the docker image is run various required command line arguments are provided. For example the system mounts the `/dev` directory to the `/dev` directory within the docker container. If other arguments are required they may be specified in the PIGEN_DOCKER_OPTS environment variable. For example setting `PIGEN_DOCKER_OPTS="--add-host foo:192.168.0.23"` will add '192.168.0.23 foo' to the `/etc/hosts` file in the container. The `--name` +and `--privileged` options are already set by the script and should not be redefined. ## Stage Anatomy @@ -332,6 +360,71 @@ follows: * Once you're happy with the image you can remove the SKIP_IMAGES files and export your image to test +# Regarding Qcow2 image building + +### Get infos about the image in use + +If you issue the two commands shown in the example below in a second command shell while a build +is running you can find out, which network block device is currently being used and which qcow2 image +is bound to it. + +Example: + +```bash +root@build-machine:~/$ lsblk | grep nbd +nbd1 43:32 0 10G 0 disk +├─nbd1p1 43:33 0 10G 0 part +└─nbd1p1 253:0 0 10G 0 part + +root@build-machine:~/$ ps xa | grep qemu-nbd + 2392 pts/6 S+ 0:00 grep --color=auto qemu-nbd +31294 ? Ssl 0:12 qemu-nbd --discard=unmap -c /dev/nbd1 image-stage4.qcow2 +``` + +Here you can see, that the qcow2 image `image-stage4.qcow2` is currently connected to `/dev/nbd1` with +the associated partition map `/dev/mapper/nbd1p1`. Don't worry that `lsblk` shows two entries. It is totally fine, because the device map is accessible via `/dev/mapper/nbd1p1` and also via `/dev/dm-0`. This is all part of the device mapper functionality of the kernel. See `dmsetup` for further information. + +### Mount a qcow2 image + +If you want to examine the content of a a single stage, you can simply mount the qcow2 image found in the `WORK_DIR` directory with the tool `./imagetool.sh`. + +See `./imagetool.sh -h` for further details on how to use it. + +### Disconnect an image if something went wrong + +It can happen, that your build stops in case of an error. Normally `./build.sh` should handle image disconnection appropriately, but in rare cases, especially during a Docker build, this may not work as expected. If that happens, starting a new build will fail and you may have to disconnect the image and/or device yourself. + +A typical message indicating that there are some orphaned device mapper entries is this: + +``` +Failed to set NBD socket +Disconnect client, due to: Unexpected end-of-file before all bytes were read +``` + +If that happens go through the following steps: + +1. First, check if the image is somehow mounted to a directory entry and umount it as you would any other block device, like i.e. a hard disk or USB stick. + +2. Second, to disconnect an image from `qemu-nbd`, the QEMU Disk Network Block Device Server, issue the following command (be sure to change the device name to the one actually used): + + ```bash + sudo qemu-nbd -d /dev/nbd1 + ``` + + Note: if you use Docker build, normally no active `qemu-nbd` process exists anymore as it will be terminated when the Docker container stops. + +3. To disconnect a device partition map from the network block device, execute: + + ```bash + sudo kpartx -d /dev/nbd1 + or + sudo ./imagetool.sh --cleanup + ``` + + Note: The `imagetool.sh` command will cleanup any /dev/nbdX that is not connected to a running `qemu-nbd` daemon. Be careful if you use network block devices for other tasks utilizing NBDs on your build machine as well. + +Now you should be able to start a new build without running into troubles again. Most of the time, especially when using Docker build, you will only need no. 3 to get everything up and running again. + # Troubleshooting ## `64 Bit Systems` diff --git a/build-docker.sh b/build-docker.sh index 350f722..01fd517 100755 --- a/build-docker.sh +++ b/build-docker.sh @@ -1,4 +1,5 @@ #!/bin/bash -eu + DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )" BUILD_OPTS="$*" @@ -47,6 +48,7 @@ fi CONTAINER_NAME=${CONTAINER_NAME:-pigen_work} CONTINUE=${CONTINUE:-0} PRESERVE_CONTAINER=${PRESERVE_CONTAINER:-0} +PIGEN_DOCKER_OPTS=${PIGEN_DOCKER_OPTS:-""} if [ -z "${IMG_NAME}" ]; then echo "IMG_NAME not set in 'config'" 1>&2 @@ -87,6 +89,10 @@ ${DOCKER} build --build-arg BASE_IMAGE=${BASE_IMAGE} -t pi-gen "${DIR}" if [ "${CONTAINER_EXISTS}" != "" ]; then trap 'echo "got CTRL+C... please wait 5s" && ${DOCKER} stop -t 5 ${CONTAINER_NAME}_cont' SIGINT SIGTERM time ${DOCKER} run --rm --privileged \ + --cap-add=ALL \ + -v /dev:/dev \ + -v /lib/modules:/lib/modules \ + ${PIGEN_DOCKER_OPTS} \ --volume "${CONFIG_FILE}":/config:ro \ -e "GIT_HASH=${GIT_HASH}" \ --volumes-from="${CONTAINER_NAME}" --name "${CONTAINER_NAME}_cont" \ @@ -98,6 +104,10 @@ if [ "${CONTAINER_EXISTS}" != "" ]; then else trap 'echo "got CTRL+C... please wait 5s" && ${DOCKER} stop -t 5 ${CONTAINER_NAME}' SIGINT SIGTERM time ${DOCKER} run --name "${CONTAINER_NAME}" --privileged \ + --cap-add=ALL \ + -v /dev:/dev \ + -v /lib/modules:/lib/modules \ + ${PIGEN_DOCKER_OPTS} \ --volume "${CONFIG_FILE}":/config:ro \ -e "GIT_HASH=${GIT_HASH}" \ pi-gen \ @@ -106,6 +116,7 @@ else rsync -av work/*/build.log deploy/" & wait "$!" fi + echo "copying results from deploy/" ${DOCKER} cp "${CONTAINER_NAME}":/pi-gen/deploy . ls -lah deploy diff --git a/build.sh b/build.sh index 725ab61..5b91d3b 100755 --- a/build.sh +++ b/build.sh @@ -1,4 +1,5 @@ #!/bin/bash -e + # shellcheck disable=SC2119 run_sub_stage() { @@ -13,7 +14,7 @@ $(cat "${i}-debconf") SELEOF EOF - log "End ${SUB_STAGE_DIR}/${i}-debconf" + log "End ${SUB_STAGE_DIR}/${i}-debconf" fi if [ -f "${i}-packages-nr" ]; then log "Begin ${SUB_STAGE_DIR}/${i}-packages-nr" @@ -22,6 +23,11 @@ EOF on_chroot << EOF apt-get -o APT::Acquire::Retries=3 install --no-install-recommends -y $PACKAGES EOF + if [ "${USE_QCOW2}" = "1" ]; then + on_chroot << EOF +apt-get clean +EOF + fi fi log "End ${SUB_STAGE_DIR}/${i}-packages-nr" fi @@ -32,6 +38,11 @@ EOF on_chroot << EOF apt-get -o APT::Acquire::Retries=3 install -y $PACKAGES EOF + if [ "${USE_QCOW2}" = "1" ]; then + on_chroot << EOF +apt-get clean +EOF + fi fi log "End ${SUB_STAGE_DIR}/${i}-packages" fi @@ -82,17 +93,30 @@ EOF run_stage(){ log "Begin ${STAGE_DIR}" STAGE="$(basename "${STAGE_DIR}")" + pushd "${STAGE_DIR}" > /dev/null - unmount "${WORK_DIR}/${STAGE}" + STAGE_WORK_DIR="${WORK_DIR}/${STAGE}" ROOTFS_DIR="${STAGE_WORK_DIR}"/rootfs + + if [ "${USE_QCOW2}" = "1" ]; then + if [ ! -f SKIP ]; then + load_qimage + fi + else + # make sure we are not umounting during export-image stage + if [ "${USE_QCOW2}" = "0" ] && [ "${NO_PRERUN_QCOW2}" = "0" ]; then + unmount "${WORK_DIR}/${STAGE}" + fi + fi + if [ ! -f SKIP_IMAGES ]; then if [ -f "${STAGE_DIR}/EXPORT_IMAGE" ]; then EXPORT_DIRS="${EXPORT_DIRS} ${STAGE_DIR}" fi fi if [ ! -f SKIP ]; then - if [ "${CLEAN}" = "1" ]; then + if [ "${CLEAN}" = "1" ] && [ "${USE_QCOW2}" = "0" ] ; then if [ -d "${ROOTFS_DIR}" ]; then rm -rf "${ROOTFS_DIR}" fi @@ -103,13 +127,21 @@ run_stage(){ log "End ${STAGE_DIR}/prerun.sh" fi for SUB_STAGE_DIR in "${STAGE_DIR}"/*; do - if [ -d "${SUB_STAGE_DIR}" ] && - [ ! -f "${SUB_STAGE_DIR}/SKIP" ]; then + if [ -d "${SUB_STAGE_DIR}" ] && [ ! -f "${SUB_STAGE_DIR}/SKIP" ]; then run_sub_stage fi done fi - unmount "${WORK_DIR}/${STAGE}" + + if [ "${USE_QCOW2}" = "1" ]; then + unload_qimage + else + # make sure we are not umounting during export-image stage + if [ "${USE_QCOW2}" = "0" ] && [ "${NO_PRERUN_QCOW2}" = "0" ]; then + unmount "${WORK_DIR}/${STAGE}" + fi + fi + PREV_STAGE="${STAGE}" PREV_STAGE_DIR="${STAGE_DIR}" PREV_ROOTFS_DIR="${ROOTFS_DIR}" @@ -143,6 +175,15 @@ do esac done +term() { + if [ "${USE_QCOW2}" = "1" ]; then + log "Unloading image" + unload_qimage + fi +} + +trap term EXIT INT TERM + export PI_GEN=${PI_GEN:-pi-gen} export PI_GEN_REPO=${PI_GEN_REPO:-https://github.com/RPi-Distro/pi-gen} @@ -157,7 +198,7 @@ export IMG_FILENAME="${IMG_FILENAME:-"${IMG_DATE}-${IMG_NAME}"}" export ZIP_FILENAME="${ZIP_FILENAME:-"image_${IMG_DATE}-${IMG_NAME}"}" export SCRIPT_DIR="${BASE_DIR}/scripts" -export WORK_DIR="${WORK_DIR:-"${BASE_DIR}/work/${IMG_DATE}-${IMG_NAME}"}" +export WORK_DIR="${WORK_DIR:-"${BASE_DIR}/work/${IMG_NAME}"}" export DEPLOY_DIR=${DEPLOY_DIR:-"${BASE_DIR}/deploy"} export DEPLOY_ZIP="${DEPLOY_ZIP:-1}" export LOG_FILE="${WORK_DIR}/build.log" @@ -209,6 +250,18 @@ source "${SCRIPT_DIR}/common" # shellcheck source=scripts/dependencies_check source "${SCRIPT_DIR}/dependencies_check" +export NO_PRERUN_QCOW2="${NO_PRERUN_QCOW2:-1}" +export USE_QCOW2="${USE_QCOW2:-1}" +export BASE_QCOW2_SIZE=${BASE_QCOW2_SIZE:-12G} +source "${SCRIPT_DIR}/qcow2_handling" +if [ "${USE_QCOW2}" = "1" ]; then + NO_PRERUN_QCOW2=1 +else + NO_PRERUN_QCOW2=0 +fi + +export NO_PRERUN_QCOW2="${NO_PRERUN_QCOW2:-1}" + dependencies_check "${BASE_DIR}/depends" if [[ -n "${APT_PROXY}" ]] && ! curl --silent "${APT_PROXY}" >/dev/null ; then @@ -242,22 +295,98 @@ for EXPORT_DIR in ${EXPORT_DIRS}; do # shellcheck source=/dev/null source "${EXPORT_DIR}/EXPORT_IMAGE" EXPORT_ROOTFS_DIR=${WORK_DIR}/$(basename "${EXPORT_DIR}")/rootfs - run_stage + if [ "${USE_QCOW2}" = "1" ]; then + USE_QCOW2=0 + EXPORT_NAME="${IMG_FILENAME}${IMG_SUFFIX}" + echo "------------------------------------------------------------------------" + echo "Running export stage for ${EXPORT_NAME}" + rm -f "${WORK_DIR}/export-image/${EXPORT_NAME}.img" || true + rm -f "${WORK_DIR}/export-image/${EXPORT_NAME}.qcow2" || true + rm -f "${WORK_DIR}/${EXPORT_NAME}.img" || true + rm -f "${WORK_DIR}/${EXPORT_NAME}.qcow2" || true + EXPORT_STAGE=$(basename "${EXPORT_DIR}") + for s in $STAGE_LIST; do + TMP_LIST=${TMP_LIST:+$TMP_LIST }$(basename "${s}") + done + FIRST_STAGE=${TMP_LIST%% *} + FIRST_IMAGE="image-${FIRST_STAGE}.qcow2" + + pushd "${WORK_DIR}" > /dev/null + echo "Creating new base "${EXPORT_NAME}.qcow2" from ${FIRST_IMAGE}" + cp "./${FIRST_IMAGE}" "${EXPORT_NAME}.qcow2" + + ARR=($TMP_LIST) + # rebase stage images to new export base + for CURR_STAGE in "${ARR[@]}"; do + if [ "${CURR_STAGE}" = "${FIRST_STAGE}" ]; then + PREV_IMG="${EXPORT_NAME}" + continue + fi + echo "Rebasing image-${CURR_STAGE}.qcow2 onto ${PREV_IMG}.qcow2" + qemu-img rebase -f qcow2 -u -b ${PREV_IMG}.qcow2 image-${CURR_STAGE}.qcow2 + if [ "${CURR_STAGE}" = "${EXPORT_STAGE}" ]; then + break + fi + PREV_IMG="image-${CURR_STAGE}" + done + + # commit current export stage into base export image + echo "Committing image-${EXPORT_STAGE}.qcow2 to ${EXPORT_NAME}.qcow2" + qemu-img commit -f qcow2 -p -b "${EXPORT_NAME}.qcow2" image-${EXPORT_STAGE}.qcow2 + + # rebase stage images back to original first stage for easy re-run + for CURR_STAGE in "${ARR[@]}"; do + if [ "${CURR_STAGE}" = "${FIRST_STAGE}" ]; then + PREV_IMG="image-${CURR_STAGE}" + continue + fi + echo "Rebasing back image-${CURR_STAGE}.qcow2 onto ${PREV_IMG}.qcow2" + qemu-img rebase -f qcow2 -u -b ${PREV_IMG}.qcow2 image-${CURR_STAGE}.qcow2 + if [ "${CURR_STAGE}" = "${EXPORT_STAGE}" ]; then + break + fi + PREV_IMG="image-${CURR_STAGE}" + done + popd > /dev/null + + mkdir -p "${WORK_DIR}/export-image/rootfs" + mv "${WORK_DIR}/${EXPORT_NAME}.qcow2" "${WORK_DIR}/export-image/" + echo "Mounting image ${WORK_DIR}/export-image/${EXPORT_NAME}.qcow2 to rootfs ${WORK_DIR}/export-image/rootfs" + mount_qimage "${WORK_DIR}/export-image/${EXPORT_NAME}.qcow2" "${WORK_DIR}/export-image/rootfs" + + CLEAN=0 + run_stage + CLEAN=1 + USE_QCOW2=1 + + else + run_stage + fi if [ "${USE_QEMU}" != "1" ]; then if [ -e "${EXPORT_DIR}/EXPORT_NOOBS" ]; then # shellcheck source=/dev/null source "${EXPORT_DIR}/EXPORT_NOOBS" STAGE_DIR="${BASE_DIR}/export-noobs" - run_stage + if [ "${USE_QCOW2}" = "1" ]; then + USE_QCOW2=0 + run_stage + USE_QCOW2=1 + else + run_stage + fi fi fi done -if [ -x ${BASE_DIR}/postrun.sh ]; then +if [ -x postrun.sh ]; then log "Begin postrun.sh" cd "${BASE_DIR}" ./postrun.sh log "End postrun.sh" fi +if [ "${USE_QCOW2}" = "1" ]; then + unload_qimage +fi + log "End ${BASE_DIR}" diff --git a/depends b/depends index a86bc82..eeb2490 100644 --- a/depends +++ b/depends @@ -7,7 +7,7 @@ zerofree zip mkdosfs:dosfstools capsh:libcap2-bin -bsdtar +bsdtar:libarchive-tools grep rsync xz:xz-utils @@ -17,3 +17,5 @@ file git lsmod:kmod bc +qemu-nbd:qemu-utils +kpartx diff --git a/export-image/03-set-partuuid/00-run.sh b/export-image/03-set-partuuid/00-run.sh index 1538c07..16e1b15 100755 --- a/export-image/03-set-partuuid/00-run.sh +++ b/export-image/03-set-partuuid/00-run.sh @@ -1,13 +1,18 @@ #!/bin/bash -e -IMG_FILE="${STAGE_WORK_DIR}/${IMG_FILENAME}${IMG_SUFFIX}.img" +if [ "${NO_PRERUN_QCOW2}" = "0" ]; then -IMGID="$(dd if="${IMG_FILE}" skip=440 bs=1 count=4 2>/dev/null | xxd -e | cut -f 2 -d' ')" + IMG_FILE="${STAGE_WORK_DIR}/${IMG_FILENAME}${IMG_SUFFIX}.img" -BOOT_PARTUUID="${IMGID}-01" -ROOT_PARTUUID="${IMGID}-02" + IMGID="$(dd if="${IMG_FILE}" skip=440 bs=1 count=4 2>/dev/null | xxd -e | cut -f 2 -d' ')" -sed -i "s/BOOTDEV/PARTUUID=${BOOT_PARTUUID}/" "${ROOTFS_DIR}/etc/fstab" -sed -i "s/ROOTDEV/PARTUUID=${ROOT_PARTUUID}/" "${ROOTFS_DIR}/etc/fstab" + BOOT_PARTUUID="${IMGID}-01" + ROOT_PARTUUID="${IMGID}-02" + + sed -i "s/BOOTDEV/PARTUUID=${BOOT_PARTUUID}/" "${ROOTFS_DIR}/etc/fstab" + sed -i "s/ROOTDEV/PARTUUID=${ROOT_PARTUUID}/" "${ROOTFS_DIR}/etc/fstab" + + sed -i "s/ROOTDEV/PARTUUID=${ROOT_PARTUUID}/" "${ROOTFS_DIR}/boot/cmdline.txt" + +fi -sed -i "s/ROOTDEV/PARTUUID=${ROOT_PARTUUID}/" "${ROOTFS_DIR}/boot/cmdline.txt" diff --git a/export-image/04-finalise/01-run.sh b/export-image/04-finalise/01-run.sh index 91264e5..d7ace58 100755 --- a/export-image/04-finalise/01-run.sh +++ b/export-image/04-finalise/01-run.sh @@ -76,25 +76,30 @@ cp "$ROOTFS_DIR/etc/rpi-issue" "$INFO_FILE" dpkg -l --root "$ROOTFS_DIR" } >> "$INFO_FILE" -ROOT_DEV="$(mount | grep "${ROOTFS_DIR} " | cut -f1 -d' ')" - -unmount "${ROOTFS_DIR}" -zerofree "${ROOT_DEV}" - -unmount_image "${IMG_FILE}" - mkdir -p "${DEPLOY_DIR}" rm -f "${DEPLOY_DIR}/${ZIP_FILENAME}${IMG_SUFFIX}.zip" rm -f "${DEPLOY_DIR}/${IMG_FILENAME}${IMG_SUFFIX}.img" +mv "$INFO_FILE" "$DEPLOY_DIR/" + +if [ "${USE_QCOW2}" = "0" ] && [ "${NO_PRERUN_QCOW2}" = "0" ]; then + ROOT_DEV="$(mount | grep "${ROOTFS_DIR} " | cut -f1 -d' ')" + + unmount "${ROOTFS_DIR}" + zerofree "${ROOT_DEV}" + + unmount_image "${IMG_FILE}" +else + unload_qimage + make_bootable_image "${STAGE_WORK_DIR}/${IMG_FILENAME}${IMG_SUFFIX}.qcow2" "$IMG_FILE" +fi + if [ "${DEPLOY_ZIP}" == "1" ]; then pushd "${STAGE_WORK_DIR}" > /dev/null zip "${DEPLOY_DIR}/${ZIP_FILENAME}${IMG_SUFFIX}.zip" \ "$(basename "${IMG_FILE}")" popd > /dev/null else - cp "$IMG_FILE" "$DEPLOY_DIR" + mv "$IMG_FILE" "$DEPLOY_DIR/" fi - -cp "$INFO_FILE" "$DEPLOY_DIR" diff --git a/export-image/prerun.sh b/export-image/prerun.sh index 4b5cf8c..fad7f80 100755 --- a/export-image/prerun.sh +++ b/export-image/prerun.sh @@ -1,85 +1,87 @@ #!/bin/bash -e -IMG_FILE="${STAGE_WORK_DIR}/${IMG_FILENAME}${IMG_SUFFIX}.img" +if [ "${NO_PRERUN_QCOW2}" = "0" ]; then + IMG_FILE="${STAGE_WORK_DIR}/${IMG_FILENAME}${IMG_SUFFIX}.img" -unmount_image "${IMG_FILE}" + unmount_image "${IMG_FILE}" -rm -f "${IMG_FILE}" + rm -f "${IMG_FILE}" -rm -rf "${ROOTFS_DIR}" -mkdir -p "${ROOTFS_DIR}" + rm -rf "${ROOTFS_DIR}" + mkdir -p "${ROOTFS_DIR}" -BOOT_SIZE="$((256 * 1024 * 1024))" -ROOT_SIZE=$(du --apparent-size -s "${EXPORT_ROOTFS_DIR}" --exclude var/cache/apt/archives --exclude boot --block-size=1 | cut -f 1) + BOOT_SIZE="$((256 * 1024 * 1024))" + ROOT_SIZE=$(du --apparent-size -s "${EXPORT_ROOTFS_DIR}" --exclude var/cache/apt/archives --exclude boot --block-size=1 | cut -f 1) -# All partition sizes and starts will be aligned to this size -ALIGN="$((4 * 1024 * 1024))" -# Add this much space to the calculated file size. This allows for -# some overhead (since actual space usage is usually rounded up to the -# filesystem block size) and gives some free space on the resulting -# image. -ROOT_MARGIN="$(echo "($ROOT_SIZE * 0.2 + 200 * 1024 * 1024) / 1" | bc)" + # All partition sizes and starts will be aligned to this size + ALIGN="$((4 * 1024 * 1024))" + # Add this much space to the calculated file size. This allows for + # some overhead (since actual space usage is usually rounded up to the + # filesystem block size) and gives some free space on the resulting + # image. + ROOT_MARGIN="$(echo "($ROOT_SIZE * 0.2 + 200 * 1024 * 1024) / 1" | bc)" -BOOT_PART_START=$((ALIGN)) -BOOT_PART_SIZE=$(((BOOT_SIZE + ALIGN - 1) / ALIGN * ALIGN)) -ROOT_PART_START=$((BOOT_PART_START + BOOT_PART_SIZE)) -ROOT_PART_SIZE=$(((ROOT_SIZE + ROOT_MARGIN + ALIGN - 1) / ALIGN * ALIGN)) -IMG_SIZE=$((BOOT_PART_START + BOOT_PART_SIZE + ROOT_PART_SIZE)) + BOOT_PART_START=$((ALIGN)) + BOOT_PART_SIZE=$(((BOOT_SIZE + ALIGN - 1) / ALIGN * ALIGN)) + ROOT_PART_START=$((BOOT_PART_START + BOOT_PART_SIZE)) + ROOT_PART_SIZE=$(((ROOT_SIZE + ROOT_MARGIN + ALIGN - 1) / ALIGN * ALIGN)) + IMG_SIZE=$((BOOT_PART_START + BOOT_PART_SIZE + ROOT_PART_SIZE)) -truncate -s "${IMG_SIZE}" "${IMG_FILE}" + truncate -s "${IMG_SIZE}" "${IMG_FILE}" -parted --script "${IMG_FILE}" mklabel msdos -parted --script "${IMG_FILE}" unit B mkpart primary fat32 "${BOOT_PART_START}" "$((BOOT_PART_START + BOOT_PART_SIZE - 1))" -parted --script "${IMG_FILE}" unit B mkpart primary ext4 "${ROOT_PART_START}" "$((ROOT_PART_START + ROOT_PART_SIZE - 1))" + parted --script "${IMG_FILE}" mklabel msdos + parted --script "${IMG_FILE}" unit B mkpart primary fat32 "${BOOT_PART_START}" "$((BOOT_PART_START + BOOT_PART_SIZE - 1))" + parted --script "${IMG_FILE}" unit B mkpart primary ext4 "${ROOT_PART_START}" "$((ROOT_PART_START + ROOT_PART_SIZE - 1))" -PARTED_OUT=$(parted -sm "${IMG_FILE}" unit b print) -BOOT_OFFSET=$(echo "$PARTED_OUT" | grep -e '^1:' | cut -d':' -f 2 | tr -d B) -BOOT_LENGTH=$(echo "$PARTED_OUT" | grep -e '^1:' | cut -d':' -f 4 | tr -d B) + PARTED_OUT=$(parted -sm "${IMG_FILE}" unit b print) + BOOT_OFFSET=$(echo "$PARTED_OUT" | grep -e '^1:' | cut -d':' -f 2 | tr -d B) + BOOT_LENGTH=$(echo "$PARTED_OUT" | grep -e '^1:' | cut -d':' -f 4 | tr -d B) -ROOT_OFFSET=$(echo "$PARTED_OUT" | grep -e '^2:' | cut -d':' -f 2 | tr -d B) -ROOT_LENGTH=$(echo "$PARTED_OUT" | grep -e '^2:' | cut -d':' -f 4 | tr -d B) + ROOT_OFFSET=$(echo "$PARTED_OUT" | grep -e '^2:' | cut -d':' -f 2 | tr -d B) + ROOT_LENGTH=$(echo "$PARTED_OUT" | grep -e '^2:' | cut -d':' -f 4 | tr -d B) -echo "Mounting BOOT_DEV..." -cnt=0 -until BOOT_DEV=$(losetup --show -f -o "${BOOT_OFFSET}" --sizelimit "${BOOT_LENGTH}" "${IMG_FILE}"); do - if [ $cnt -lt 5 ]; then - cnt=$((cnt + 1)) - echo "Error in losetup for BOOT_DEV. Retrying..." - sleep 5 - else - echo "ERROR: losetup for BOOT_DEV failed; exiting" - exit 1 - fi -done + echo "Mounting BOOT_DEV..." + cnt=0 + until BOOT_DEV=$(losetup --show -f -o "${BOOT_OFFSET}" --sizelimit "${BOOT_LENGTH}" "${IMG_FILE}"); do + if [ $cnt -lt 5 ]; then + cnt=$((cnt + 1)) + echo "Error in losetup for BOOT_DEV. Retrying..." + sleep 5 + else + echo "ERROR: losetup for BOOT_DEV failed; exiting" + exit 1 + fi + done -echo "Mounting ROOT_DEV..." -cnt=0 -until ROOT_DEV=$(losetup --show -f -o "${ROOT_OFFSET}" --sizelimit "${ROOT_LENGTH}" "${IMG_FILE}"); do - if [ $cnt -lt 5 ]; then - cnt=$((cnt + 1)) - echo "Error in losetup for ROOT_DEV. Retrying..." - sleep 5 - else - echo "ERROR: losetup for ROOT_DEV failed; exiting" - exit 1 - fi -done + echo "Mounting ROOT_DEV..." + cnt=0 + until ROOT_DEV=$(losetup --show -f -o "${ROOT_OFFSET}" --sizelimit "${ROOT_LENGTH}" "${IMG_FILE}"); do + if [ $cnt -lt 5 ]; then + cnt=$((cnt + 1)) + echo "Error in losetup for ROOT_DEV. Retrying..." + sleep 5 + else + echo "ERROR: losetup for ROOT_DEV failed; exiting" + exit 1 + fi + done -echo "/boot: offset $BOOT_OFFSET, length $BOOT_LENGTH" -echo "/: offset $ROOT_OFFSET, length $ROOT_LENGTH" + echo "/boot: offset $BOOT_OFFSET, length $BOOT_LENGTH" + echo "/: offset $ROOT_OFFSET, length $ROOT_LENGTH" -ROOT_FEATURES="^huge_file" -for FEATURE in metadata_csum 64bit; do + ROOT_FEATURES="^huge_file" + for FEATURE in metadata_csum 64bit; do if grep -q "$FEATURE" /etc/mke2fs.conf; then - ROOT_FEATURES="^$FEATURE,$ROOT_FEATURES" + ROOT_FEATURES="^$FEATURE,$ROOT_FEATURES" fi -done -mkdosfs -n boot -F 32 -v "$BOOT_DEV" > /dev/null -mkfs.ext4 -L rootfs -O "$ROOT_FEATURES" "$ROOT_DEV" > /dev/null + done + mkdosfs -n boot -F 32 -v "$BOOT_DEV" > /dev/null + mkfs.ext4 -L rootfs -O "$ROOT_FEATURES" "$ROOT_DEV" > /dev/null -mount -v "$ROOT_DEV" "${ROOTFS_DIR}" -t ext4 -mkdir -p "${ROOTFS_DIR}/boot" -mount -v "$BOOT_DEV" "${ROOTFS_DIR}/boot" -t vfat + mount -v "$ROOT_DEV" "${ROOTFS_DIR}" -t ext4 + mkdir -p "${ROOTFS_DIR}/boot" + mount -v "$BOOT_DEV" "${ROOTFS_DIR}/boot" -t vfat -rsync -aHAXx --exclude /var/cache/apt/archives --exclude /boot "${EXPORT_ROOTFS_DIR}/" "${ROOTFS_DIR}/" -rsync -rtx "${EXPORT_ROOTFS_DIR}/boot/" "${ROOTFS_DIR}/boot/" + rsync -aHAXx --exclude /var/cache/apt/archives --exclude /boot "${EXPORT_ROOTFS_DIR}/" "${ROOTFS_DIR}/" + rsync -rtx "${EXPORT_ROOTFS_DIR}/boot/" "${ROOTFS_DIR}/boot/" +fi diff --git a/export-noobs/00-release/00-run.sh b/export-noobs/00-release/00-run.sh index 18a3a5d..bfaea9f 100755 --- a/export-noobs/00-release/00-run.sh +++ b/export-noobs/00-release/00-run.sh @@ -1,6 +1,6 @@ #!/bin/bash -e -NOOBS_DIR="${STAGE_WORK_DIR}/${IMG_DATE}-${IMG_NAME}${IMG_SUFFIX}" +NOOBS_DIR="${STAGE_WORK_DIR}/${IMG_NAME}${IMG_SUFFIX}" install -v -m 744 files/partition_setup.sh "${NOOBS_DIR}/" install -v files/partitions.json "${NOOBS_DIR}/" @@ -20,7 +20,7 @@ BOOT_SIZE="$(( BOOT_SIZE / 1024 / 1024 + 1))" ROOT_SIZE="$(( ROOT_SIZE / 1024 / 1024 + 1))" BOOT_NOM="256" -ROOT_NOM="$(( ROOT_SIZE + 400 ))" +ROOT_NOM="$(echo "$ROOT_SIZE" | awk '{printf "%.0f", (($1 + 400) * 1.2) + 0.5 }')" mv "${NOOBS_DIR}/OS.png" "${NOOBS_DIR}/${NOOBS_NAME// /_}.png" @@ -37,7 +37,12 @@ sed "${NOOBS_DIR}/os.json" -i -e "s|UNRELEASED|${IMG_DATE}|" sed "${NOOBS_DIR}/os.json" -i -e "s|NOOBS_NAME|${NOOBS_NAME}|" sed "${NOOBS_DIR}/os.json" -i -e "s|NOOBS_DESCRIPTION|${NOOBS_DESCRIPTION}|" sed "${NOOBS_DIR}/os.json" -i -e "s|RELEASE|${RELEASE}|" +sed "${NOOBS_DIR}/os.json" -i -e "s|KERNEL|$(cat "${STAGE_WORK_DIR}/kernel_version")|" sed "${NOOBS_DIR}/release_notes.txt" -i -e "s|UNRELEASED|${IMG_DATE}|" -cp -a "${NOOBS_DIR}" "${DEPLOY_DIR}/" +if [ "${USE_QCOW2}" = "1" ]; then + mv "${NOOBS_DIR}" "${DEPLOY_DIR}/" +else + cp -a "${NOOBS_DIR}" "${DEPLOY_DIR}/" +fi diff --git a/export-noobs/00-release/files/os.json b/export-noobs/00-release/files/os.json index e231238..6cc0e36 100644 --- a/export-noobs/00-release/files/os.json +++ b/export-noobs/00-release/files/os.json @@ -1,7 +1,7 @@ { "description": "NOOBS_DESCRIPTION", "feature_level": 35120124, - "kernel": "4.19", + "kernel": "KERNEL", "name": "NOOBS_NAME", "password": "raspberry", "release_date": "UNRELEASED", diff --git a/export-noobs/00-release/files/release_notes.txt b/export-noobs/00-release/files/release_notes.txt index be998d9..6e9db37 100644 --- a/export-noobs/00-release/files/release_notes.txt +++ b/export-noobs/00-release/files/release_notes.txt @@ -1,4 +1,57 @@ UNRELEASED: + * Chromium upgraded to version 88.0.4324.187 + * NuScratch upgraded to version 20210507 + * Node-RED upgraded to version 1.3.4 + * pigpio upgraded to version 1.79 + * Thonny upgraded to version 3.3.6 + * Icelandic and Italian translations updated for several packages + * piclone: Remove hiding of application in other desktops + * agnostics: Remove hiding of app in other desktops + * rp-bookshelf: + - Remove hiding of app in other desktops + - GTK+3 version + * lxplug-bluetooth: + - Fix some memory leaks + - Add authorisation dialog required by some BT-LE pairings + * alsa-utils: Add custom init files for bcm2835 on Raspberry Pi to set volume correctly + * rp-prefapps: Remove hiding of app in other desktops + * OpenSSH and OpenSSL speed improvements + * Install gpiozero in lite images + * Raspberry Pi firmware 518ee7c871aaa9aaa88116953d57e73787ee6e43 + * Linux kernel 5.10.17 +2021-03-04: + * Thonny upgraded to version 3.3.5 + * SD Card Copier made compatible with NVMe devices; now built against GTK+3 toolkit + * Composite video options removed from Raspberry Pi 4 in Raspberry Pi Configuration + * Boot order options in raspi-config adjusted for more flexibility + * Recommended Software now built against GTK+3 toolkit + * Fix for crash in volume plugin when using keyboard could push value out of range + * Fix for focus changing between windows in file manager when using keyboard to navigate directory view + * Fix for Raspberry Pi 400 keyboard country not being read correctly in startup wizard + * Armenian and Japanese translations added to several packages + * Automatically load aes-neon-bs on ARM64 to speed up OpenSSL + * Raspberry Pi firmware fcf8d2f7639ad8d0330db9c8db9b71bd33eaaa28 + * Linux kernel 5.10.17 +2021-01-11: + * Chromium version 86.0.4240.197 included + * Screen reader support enabled in Chromium + * Adobe have end-of-lifed Flash Player, so it has been removed + * Scratch 2 required Flash, so it has been removed + * Added Epson printer drivers + * Added timeout to hide messages from USB device monitor after 5 seconds + * Bug fix - PulseAudio output was in mono + * Bug fix - brief audio interruptions at start of playback in VLC + * Bug fix - old ALSA output settings being used instead of PulseAudio settings by some applications + * Bug fix - crash in PulseAudio volume controller when used on multichannel devices + * Bug fix - battery monitor failing to load on x86 platforms + * Bug fix - setting of password in startup wizard failed if language was changed + * Bug fix - Chromium video playback lockup on small number of devices + * Bug fix - Chromium Google Maps 3D view artefacts + * Slovak, Italian and Norwegian translations updated + * Added Epson printer drivers + * Raspberry Pi firmware 70f1581eec2c036b7e9309f1af41c651fb125447 + * Linux kernel 5.4.83 +2020-12-02: * PulseAudio now included and running by default * Bluealsa Bluetooth interface removed - Bluetooth audio is now handled by PulseAudio * LXPanel volume control plugin replaced with PulseAudio version diff --git a/export-noobs/prerun.sh b/export-noobs/prerun.sh index c161ee6..d97508a 100755 --- a/export-noobs/prerun.sh +++ b/export-noobs/prerun.sh @@ -1,11 +1,15 @@ #!/bin/bash -e -IMG_FILE="${STAGE_WORK_DIR}/${IMG_FILENAME}${IMG_SUFFIX}.img" -NOOBS_DIR="${STAGE_WORK_DIR}/${IMG_DATE}-${IMG_NAME}${IMG_SUFFIX}" -unmount_image "${IMG_FILE}" - +NOOBS_DIR="${STAGE_WORK_DIR}/${IMG_NAME}${IMG_SUFFIX}" mkdir -p "${STAGE_WORK_DIR}" -cp "${WORK_DIR}/export-image/${IMG_FILENAME}${IMG_SUFFIX}.img" "${STAGE_WORK_DIR}/" + +if [ "${DEPLOY_ZIP}" == "1" ]; then + IMG_FILE="${WORK_DIR}/export-image/${IMG_FILENAME}${IMG_SUFFIX}.img" +else + IMG_FILE="${DEPLOY_DIR}/${IMG_FILENAME}${IMG_SUFFIX}.img" +fi + +unmount_image "${IMG_FILE}" rm -rf "${NOOBS_DIR}" @@ -53,8 +57,15 @@ mount "$BOOT_DEV" "${STAGE_WORK_DIR}/rootfs/boot" ln -sv "/lib/systemd/system/apply_noobs_os_config.service" "$ROOTFS_DIR/etc/systemd/system/multi-user.target.wants/apply_noobs_os_config.service" +KERNEL_VER="$(zgrep -oPm 1 "Linux version \K(.*)$" "${STAGE_WORK_DIR}/rootfs/usr/share/doc/raspberrypi-kernel/changelog.Debian.gz" | cut -f-2 -d.)" +echo "$KERNEL_VER" > "${STAGE_WORK_DIR}/kernel_version" + bsdtar --numeric-owner --format gnutar -C "${STAGE_WORK_DIR}/rootfs/boot" -cpf - . | xz -T0 > "${NOOBS_DIR}/boot.tar.xz" umount "${STAGE_WORK_DIR}/rootfs/boot" bsdtar --numeric-owner --format gnutar -C "${STAGE_WORK_DIR}/rootfs" --one-file-system -cpf - . | xz -T0 > "${NOOBS_DIR}/root.tar.xz" +if [ "${USE_QCOW2}" = "1" ]; then + rm "$ROOTFS_DIR/etc/systemd/system/multi-user.target.wants/apply_noobs_os_config.service" +fi + unmount_image "${IMG_FILE}" diff --git a/imagetool.sh b/imagetool.sh new file mode 100755 index 0000000..e8467e1 --- /dev/null +++ b/imagetool.sh @@ -0,0 +1,114 @@ +#!/bin/bash + +if [ "$(id -u)" != "0" ]; then + echo "Please run as root" 1>&2 + exit 1 +fi + +progname=$(basename $0) + +function usage() +{ + cat << HEREDOC + +Usage: + Mount Image : $progname [--mount] [--image-name ] [--mount-point ] + Umount Image: $progname [--umount] [--mount-point ] + Cleanup NBD : $progname [--cleanup] + + arguments: + -h, --help show this help message and exit + -c, --cleanup cleanup orphaned device mappings + -m, --mount mount image + -u, --umount umount image + -i, --image-name path to qcow2 image + -p, --mount-point mount point for image + + This tool will use /dev/nbd1 as default for mounting an image. If you want to use another device, execute like this: + NBD_DEV=/dev/nbd2 ./$progname --mount --image --mount-point + +HEREDOC +} + +MOUNT=0 +UMOUNT=0 +IMAGE="" +MOUNTPOINT="" + +nbd_cleanup() { + DEVS="$(lsblk | grep nbd | grep disk | cut -d" " -f1)" + if [ ! -z "${DEVS}" ]; then + for d in $DEVS; do + if [ ! -z "${d}" ]; then + QDEV="$(ps xa | grep $d | grep -v grep)" + if [ -z "${QDEV}" ]; then + kpartx -d /dev/$d && echo "Unconnected device map removed: /dev/$d" + fi + fi + done + fi +} + +# As long as there is at least one more argument, keep looping +while [[ $# -gt 0 ]]; do + key="$1" + case "$key" in + -h|--help) + usage + exit + ;; + -c|--cleanup) + nbd_cleanup + ;; + -m|--mount) + MOUNT=1 + ;; + -u|--umount) + UMOUNT=1 + ;; + -i|--image-name) + shift + IMAGE="$1" + ;; + -p|--mount-point) + shift + MOUNTPOINT="$1" + ;; + *) + echo "Unknown option '$key'" + usage + exit + ;; + esac + # Shift after checking all the cases to get the next option + shift +done + +if [ "${MOUNT}" = "1" ] && [ "${UMOUNT}" = "1" ]; then + usage + echo "Concurrent mount options not possible." + exit +fi + +if [ "${MOUNT}" = "1" ] && ([ -z "${IMAGE}" ] || [ -z "${MOUNTPOINT}" ]); then + usage + echo "Can not mount image. Image path and/or mount point missing." + exit +fi + +if [ "${UMOUNT}" = "1" ] && [ -z "${MOUNTPOINT}" ]; then + usage + echo "Can not umount. Mount point parameter missing." + exit +fi + +export NBD_DEV="${NBD_DEV:-/dev/nbd1}" +export MAP_BOOT_DEV=/dev/mapper/nbd1p1 +export MAP_ROOT_DEV=/dev/mapper/nbd1p2 +source scripts/qcow2_handling + +if [ "${MOUNT}" = "1" ]; then + mount_qimage "${IMAGE}" "${MOUNTPOINT}" +elif [ "${UMOUNT}" = "1" ]; then + umount_qimage "${MOUNTPOINT}" +fi diff --git a/scripts/common b/scripts/common index e2048d9..300d30b 100644 --- a/scripts/common +++ b/scripts/common @@ -21,8 +21,10 @@ bootstrap(){ setarch linux32 capsh --drop=cap_setfcap -- -c "'${BOOTSTRAP_CMD}' $BOOTSTRAP_STR" || true - if [ -d "$2/debootstrap" ]; then - rmdir "$2/debootstrap" + if [ -d "$2/debootstrap" ] && ! rmdir "$2/debootstrap"; then + cp "$2/debootstrap/debootstrap.log" "${STAGE_WORK_DIR}" + log "bootstrap failed: please check ${STAGE_WORK_DIR}/debootstrap.log" + return 1 fi } export -f bootstrap diff --git a/scripts/dependencies_check b/scripts/dependencies_check index 375512b..32c42ef 100644 --- a/scripts/dependencies_check +++ b/scripts/dependencies_check @@ -28,11 +28,26 @@ dependencies_check() false fi + # If we're building on a native arm platform, we don't need to check for + # binfmt_misc or require it to be loaded. - if ! grep -q "/proc/sys/fs/binfmt_misc" /proc/mounts; then - echo "Module binfmt_misc not loaded in host" - echo "Please run:" - echo " sudo modprobe binfmt_misc" - exit 1 + binfmt_misc_required=1 + + case $(uname -m) in + aarch64) + binfmt_misc_required=0 + ;; + arm*) + binfmt_misc_required=0 + ;; + esac + + if [[ "${binfmt_misc_required}" == "1" ]]; then + if ! grep -q "/proc/sys/fs/binfmt_misc" /proc/mounts; then + echo "Module binfmt_misc not loaded in host" + echo "Please run:" + echo " sudo modprobe binfmt_misc" + exit 1 + fi fi } diff --git a/scripts/qcow2_handling b/scripts/qcow2_handling new file mode 100644 index 0000000..6b2a56b --- /dev/null +++ b/scripts/qcow2_handling @@ -0,0 +1,256 @@ +#!/bin/bash + +# QCOW2 Routines + +export CURRENT_IMAGE +export CURRENT_MOUNTPOINT + +export NBD_DEV +export MAP_BOOT_DEV +export MAP_ROOT_DEV + +# set in build.sh +# should be fairly enough for the beginning +# overwrite here by uncommenting following lines +# BASE_QCOW2_SIZE=12G + +# find and initialize free block device nodes +init_nbd() { + modprobe nbd max_part=16 + if [ -z "${NBD_DEV}" ]; then + for x in /sys/class/block/nbd* ; do + S=`cat $x/size` + if [ "$S" == "0" ] ; then + NBD_DEV=/dev/$(basename $x) + MAP_BOOT_DEV=/dev/mapper/$(basename $x)p1 + MAP_ROOT_DEV=/dev/mapper/$(basename $x)p2 + break + fi + done + fi +} +export -f init_nbd + +# connect image to block device +connect_blkdev() { + init_nbd + qemu-nbd --discard=unmap -c $NBD_DEV "$1" + sync + kpartx -as $NBD_DEV + sync + CURRENT_IMAGE="$1" +} +export -f connect_blkdev + +# disconnect image from block device +disconnect_blkdev() { + kpartx -d $NBD_DEV + qemu-nbd -d $NBD_DEV + NBD_DEV= + MAP_BOOT_DEV= + MAP_ROOT_DEV= + CURRENT_IMAGE= +} +export -f disconnect_blkdev + +# mount qcow2 image: mount_image +mount_qimage() { + connect_blkdev "$1" + mount -v -t ext4 $MAP_ROOT_DEV "$2" + mkdir -p "${ROOTFS_DIR}/boot" + mount -v -t vfat $MAP_BOOT_DEV "$2/boot" + CURRENT_MOUNTPOINT="$2" +} +export -f mount_qimage + +# umount qcow2 image: umount_image +umount_qimage() { + sync + #umount "$1/boot" + while mount | grep -q "$1"; do + local LOCS + LOCS=$(mount | grep "$1" | cut -f 3 -d ' ' | sort -r) + for loc in $LOCS; do + echo "$loc" + while mountpoint -q "$loc" && ! umount "$loc"; do + sleep 0.1 + done + done + done + CURRENT_MOUNTPOINT= + disconnect_blkdev +} +export -f umount_qimage + +# create base image / backing image / mount image +load_qimage() { + if [ -z "${CURRENT_MOUNTPOINT}" ]; then + if [ ! -d "${ROOTFS_DIR}" ]; then + mkdir -p "${ROOTFS_DIR}"; + fi + + if [ "${CLEAN}" = "1" ] && [ -f "${WORK_DIR}/image-${STAGE}.qcow2" ]; then + rm -f "${WORK_DIR}/image-${STAGE}.qcow2"; + fi + + if [ ! -f "${WORK_DIR}/image-${STAGE}.qcow2" ]; then + pushd ${WORK_DIR} > /dev/null + init_nbd + if [ -z "${PREV_STAGE}" ]; then + echo "Creating base image: image-${STAGE}.qcow2" + # -o preallocation=falloc + qemu-img create -f qcow2 image-${STAGE}.qcow2 $BASE_QCOW2_SIZE + sync + qemu-nbd --discard=unmap -c $NBD_DEV image-${STAGE}.qcow2 + sync + sfdisk $NBD_DEV << EOF +4MiB,250MiB,c,* +254MiB,,83; +EOF + sync + kpartx -as $NBD_DEV + mkdosfs -n boot -F 32 -v $MAP_BOOT_DEV + mkfs.ext4 -L rootfs -O "^huge_file,^metadata_csum,^64bit" $MAP_ROOT_DEV + sync + else + if [ ! -f "${WORK_DIR}/image-${PREV_STAGE}.qcow2" ]; then + exit 1; + fi + echo "Creating backing image: image-${STAGE}.qcow2 <- ${WORK_DIR}/image-${PREV_STAGE}.qcow2" + qemu-img create -f qcow2 \ + -o backing_file=${WORK_DIR}/image-${PREV_STAGE}.qcow2 \ + ${WORK_DIR}/image-${STAGE}.qcow2 + sync + qemu-nbd --discard=unmap -c $NBD_DEV image-${STAGE}.qcow2 + sync + kpartx -as $NBD_DEV + fi + + mount -v -t ext4 $MAP_ROOT_DEV "${ROOTFS_DIR}" + mkdir -p "${ROOTFS_DIR}/boot" + mount -v -t vfat $MAP_BOOT_DEV "${ROOTFS_DIR}/boot" + CURRENT_IMAGE=${WORK_DIR}/image-${STAGE}.qcow2 + CURRENT_MOUNTPOINT=${ROOTFS_DIR} + popd > /dev/null + else + mount_qimage "${WORK_DIR}/image-${STAGE}.qcow2" "${ROOTFS_DIR}" + fi + echo "Current image in use: ${CURRENT_IMAGE} (MP: ${CURRENT_MOUNTPOINT})" + fi +} +export -f load_qimage + +# umount current image and refresh mount point env var +unload_qimage() { + if [ ! -z "${CURRENT_MOUNTPOINT}" ]; then + fstrim -v "${CURRENT_MOUNTPOINT}" || true + umount_qimage "${CURRENT_MOUNTPOINT}" + fi +} +export -f unload_qimage + +# based on: https://github.com/SirLagz/RaspberryPi-ImgAutoSizer +# helper function for make_bootable_image, do not call directly +function resize_qcow2() { + if [ -z "$CALL_FROM_MBI" ]; then + echo "resize_qcow2: cannot be called directly, use make_bootable_image instead" + return 1 + fi + + # ROOT_MARGIN=$((800*1024*1024)) + ROOT_MARGIN=$((1*1024*1024)) + PARTED_OUT=`parted -s -m "$NBD_DEV" unit B print` + PART_NO=`echo "$PARTED_OUT" | grep ext4 | awk -F: ' { print $1 } '` + PART_START=`echo "$PARTED_OUT" | grep ext4 | awk -F: ' { print substr($2,1,length($2)-1) } '` + + e2fsck -y -f $MAP_ROOT_DEV || true + + DATA_SIZE=`resize2fs -P $MAP_ROOT_DEV | awk -F': ' ' { print $2 } '` + BLOCK_SIZE=$(dumpe2fs -h $MAP_ROOT_DEV | grep 'Block size' | awk -F': ' ' { print $2 }') + BLOCK_SIZE=${BLOCK_SIZE// /} + + let DATA_SIZE=$DATA_SIZE+$ROOT_MARGIN/$BLOCK_SIZE + resize2fs -p $MAP_ROOT_DEV $DATA_SIZE + sleep 1 + + let PART_NEW_SIZE=$DATA_SIZE*$BLOCK_SIZE + let PART_NEW_END=$PART_START+$PART_NEW_SIZE + ACT1=`parted -s "$NBD_DEV" rm 2` + ACT2=`parted -s "$NBD_DEV" unit B mkpart primary $PART_START $PART_NEW_END` + NEW_IMG_SIZE=`parted -s -m "$NBD_DEV" unit B print free | tail -1 | awk -F: ' { print substr($2,1,length($2)-1) } '` +} +export -f resize_qcow2 + +# create raw img from qcow2: make_bootable_image +function make_bootable_image() { + + EXPORT_QCOW2="$1" + EXPORT_IMAGE="$2" + + echo "Connect block device to source qcow2" + connect_blkdev "${EXPORT_QCOW2}" + + echo "Resize fs and partition" + CALL_FROM_MBI=1 + resize_qcow2 + sync + CALL_FROM_MBI= + + echo "Disconnect block device" + disconnect_blkdev + + if [ -z "$NEW_IMG_SIZE" ]; then + echo "NEW_IMG_SIZE could not be calculated, cannot process image. Exit." + exit 1 + fi + + echo "Shrinking qcow2 image" + qemu-img resize --shrink "${EXPORT_QCOW2}" $NEW_IMG_SIZE + sync + + echo "Convert qcow2 to raw image" + qemu-img convert -f qcow2 -O raw "${EXPORT_QCOW2}" "${EXPORT_IMAGE}" + sync + + echo "Get PARTUUIDs from image" + IMGID="$(blkid -o value -s PTUUID "${EXPORT_IMAGE}")" + + BOOT_PARTUUID="${IMGID}-01" + echo "Boot: $BOOT_PARTUUID" + ROOT_PARTUUID="${IMGID}-02" + echo "Root: $ROOT_PARTUUID" + + echo "Mount image" + MOUNTROOT=${WORK_DIR}/tmpimage + mkdir -p $MOUNTROOT + + MOUNTPT=$MOUNTROOT + PARTITION=2 + mount "${EXPORT_IMAGE}" "$MOUNTPT" -o loop,offset=$[ `/sbin/sfdisk -d "${EXPORT_IMAGE}" | grep "start=" | head -n $PARTITION | tail -n1 | sed 's/.*start=[ ]*//' | sed 's/,.*//'` * 512 ],sizelimit=$[ `/sbin/sfdisk -d "${EXPORT_IMAGE}" | grep "start=" | head -n $PARTITION | tail -n1 | sed 's/.*size=[ ]*//' | sed 's/,.*//'` * 512 ] || exit 1 + + MOUNTPT=$MOUNTROOT/boot + PARTITION=1 + mount "${EXPORT_IMAGE}" "$MOUNTPT" -o loop,offset=$[ `/sbin/sfdisk -d "${EXPORT_IMAGE}" | grep "start=" | head -n $PARTITION | tail -n1 | sed 's/.*start=[ ]*//' | sed 's/,.*//'` * 512 ],sizelimit=$[ `/sbin/sfdisk -d "${EXPORT_IMAGE}" | grep "start=" | head -n $PARTITION | tail -n1 | sed 's/.*size=[ ]*//' | sed 's/,.*//'` * 512 ] || exit 1 + + if [ ! -d "${MOUNTROOT}/root" ]; then + echo "Image damaged or not mounted. Exit." + exit 1 + fi + + echo "Setup PARTUUIDs" + if [ ! -z "$BOOT_PARTUUID" ] && [ ! -z "$ROOT_PARTUUID" ]; then + echo "Set UUIDs to make it bootable" + sed -i "s/BOOTDEV/PARTUUID=${BOOT_PARTUUID}/" "${MOUNTROOT}/etc/fstab" + sed -i "s/ROOTDEV/PARTUUID=${ROOT_PARTUUID}/" "${MOUNTROOT}/etc/fstab" + sed -i "s/ROOTDEV/PARTUUID=${ROOT_PARTUUID}/" "${MOUNTROOT}/boot/cmdline.txt" + fi + + echo "Umount image" + sync + umount "${MOUNTROOT}/boot" || exit 1 + umount "${MOUNTROOT}" || exit 1 + + echo "Remove qcow2 export image" + rm -f "${EXPORT_QCOW2}" +} +export -f make_bootable_image diff --git a/stage0/prerun.sh b/stage0/prerun.sh index 3b25783..d4dd0a1 100755 --- a/stage0/prerun.sh +++ b/stage0/prerun.sh @@ -1,5 +1,5 @@ #!/bin/bash -e -if [ ! -d "${ROOTFS_DIR}" ]; then +if [ ! -d "${ROOTFS_DIR}" ] || [ "${USE_QCOW2}" = "1" ]; then bootstrap ${RELEASE} "${ROOTFS_DIR}" http://raspbian.raspberrypi.org/raspbian/ fi diff --git a/stage1/00-boot-files/files/cmdline.txt b/stage1/00-boot-files/files/cmdline.txt index b815bd8..03cde68 100644 --- a/stage1/00-boot-files/files/cmdline.txt +++ b/stage1/00-boot-files/files/cmdline.txt @@ -1 +1 @@ -console=serial0,115200 console=tty1 root=ROOTDEV rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait +console=serial0,115200 console=tty1 root=ROOTDEV rootfstype=ext4 fsck.repair=yes rootwait diff --git a/stage2/01-sys-tweaks/00-packages b/stage2/01-sys-tweaks/00-packages index 83ec74c..a1f4b59 100644 --- a/stage2/01-sys-tweaks/00-packages +++ b/stage2/01-sys-tweaks/00-packages @@ -2,6 +2,8 @@ ssh less fbset sudo psmisc strace ed ncdu crda console-setup keyboard-configuration debconf-utils parted unzip build-essential manpages-dev python bash-completion gdb pkg-config python-rpi.gpio v4l-utils +python-gpiozero +python3-gpiozero avahi-daemon lua5.1 luajit diff --git a/stage2/01-sys-tweaks/00-patches/07-resize-init.diff b/stage2/01-sys-tweaks/00-patches/07-resize-init.diff index 8c41357..cb160ae 100644 --- a/stage2/01-sys-tweaks/00-patches/07-resize-init.diff +++ b/stage2/01-sys-tweaks/00-patches/07-resize-init.diff @@ -1,5 +1,5 @@ --- stage2.orig/rootfs/boot/cmdline.txt +++ stage2/rootfs/boot/cmdline.txt @@ -1 +1 @@ --console=serial0,115200 console=tty1 root=ROOTDEV rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait -+console=serial0,115200 console=tty1 root=ROOTDEV rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait quiet init=/usr/lib/raspi-config/init_resize.sh +-console=serial0,115200 console=tty1 root=ROOTDEV rootfstype=ext4 fsck.repair=yes rootwait ++console=serial0,115200 console=tty1 root=ROOTDEV rootfstype=ext4 fsck.repair=yes rootwait quiet init=/usr/lib/raspi-config/init_resize.sh