Merge branch 'master' of https://github.com/RPi-Distro/pi-gen into RPi-Distro-master

This commit is contained in:
Alexandre Aubin 2021-11-05 17:50:02 +01:00
commit 59babe0987
20 changed files with 822 additions and 117 deletions

View file

@ -7,8 +7,8 @@ RUN apt-get -y update && \
apt-get -y install --no-install-recommends \ apt-get -y install --no-install-recommends \
git vim parted \ git vim parted \
quilt coreutils qemu-user-static debootstrap zerofree zip dosfstools \ quilt coreutils qemu-user-static debootstrap zerofree zip dosfstools \
bsdtar libcap2-bin rsync grep udev xz-utils curl xxd file kmod bc\ libarchive-tools libcap2-bin rsync grep udev xz-utils curl xxd file kmod bc\
binfmt-support ca-certificates \ binfmt-support ca-certificates qemu-utils kpartx \
&& rm -rf /var/lib/apt/lists/* && rm -rf /var/lib/apt/lists/*
COPY . /pi-gen/ COPY . /pi-gen/

View file

@ -14,7 +14,8 @@ To install the required dependencies for `pi-gen` you should run:
```bash ```bash
apt-get install coreutils quilt parted qemu-user-static debootstrap zerofree zip \ apt-get install coreutils quilt parted qemu-user-static debootstrap zerofree zip \
dosfstools bsdtar libcap2-bin grep rsync xz-utils file git curl bc dosfstools libarchive-tools libcap2-bin grep rsync xz-utils file git curl bc \
qemu-utils kpartx
``` ```
The file `depends` contains a list of tools needed. The format of this The file `depends` contains a list of tools needed. The format of this
@ -36,7 +37,30 @@ The following environment variables are supported:
but you should use something else for a customized version. Export files but you should use something else for a customized version. Export files
in stages may add suffixes to `IMG_NAME`. in stages may add suffixes to `IMG_NAME`.
* `RELEASE` (Default: buster) * `USE_QCOW2`(Default: `1` )
Instead of using traditional way of building the rootfs of every stage in
single subdirectories and copying over the previous one to the next one,
qcow2 based virtual disks with backing images are used in every stage.
This speeds up the build process and reduces overall space consumption
significantly.
<u>Additional optional parameters regarding qcow2 build:</u>
* `BASE_QCOW2_SIZE` (Default: 12G)
Size of the virtual qcow2 disk.
Note: it will not actually use that much of space at once but defines the
maximum size of the virtual disk. If you change the build process by adding
a lot of bigger packages or additional build stages, it can be necessary to
increase the value because the virtual disk can run out of space like a normal
hard drive would.
**CAUTION:** Although the qcow2 build mechanism will run fine inside Docker, it can happen
that the network block device is not disconnected correctly after the Docker process has
ended abnormally. In that case see [Disconnect an image if something went wrong](#Disconnect-an-image-if-something-went-wrong)
* `RELEASE` (Default: buster)
The release version to build images against. Valid values are jessie, stretch The release version to build images against. Valid values are jessie, stretch
buster, bullseye, and testing. buster, bullseye, and testing.
@ -240,6 +264,10 @@ fix is to ensure `binfmt-support` is installed on the host machine before
starting the `./build-docker.sh` script (or using your own docker build starting the `./build-docker.sh` script (or using your own docker build
solution). solution).
### Passing arguments to Docker
When the docker image is run various required command line arguments are provided. For example the system mounts the `/dev` directory to the `/dev` directory within the docker container. If other arguments are required they may be specified in the PIGEN_DOCKER_OPTS environment variable. For example setting `PIGEN_DOCKER_OPTS="--add-host foo:192.168.0.23"` will add '192.168.0.23 foo' to the `/etc/hosts` file in the container. The `--name`
and `--privileged` options are already set by the script and should not be redefined.
## Stage Anatomy ## Stage Anatomy
@ -332,6 +360,71 @@ follows:
* Once you're happy with the image you can remove the SKIP_IMAGES files and * Once you're happy with the image you can remove the SKIP_IMAGES files and
export your image to test export your image to test
# Regarding Qcow2 image building
### Get infos about the image in use
If you issue the two commands shown in the example below in a second command shell while a build
is running you can find out, which network block device is currently being used and which qcow2 image
is bound to it.
Example:
```bash
root@build-machine:~/$ lsblk | grep nbd
nbd1 43:32 0 10G 0 disk
├─nbd1p1 43:33 0 10G 0 part
└─nbd1p1 253:0 0 10G 0 part
root@build-machine:~/$ ps xa | grep qemu-nbd
2392 pts/6 S+ 0:00 grep --color=auto qemu-nbd
31294 ? Ssl 0:12 qemu-nbd --discard=unmap -c /dev/nbd1 image-stage4.qcow2
```
Here you can see, that the qcow2 image `image-stage4.qcow2` is currently connected to `/dev/nbd1` with
the associated partition map `/dev/mapper/nbd1p1`. Don't worry that `lsblk` shows two entries. It is totally fine, because the device map is accessible via `/dev/mapper/nbd1p1` and also via `/dev/dm-0`. This is all part of the device mapper functionality of the kernel. See `dmsetup` for further information.
### Mount a qcow2 image
If you want to examine the content of a a single stage, you can simply mount the qcow2 image found in the `WORK_DIR` directory with the tool `./imagetool.sh`.
See `./imagetool.sh -h` for further details on how to use it.
### Disconnect an image if something went wrong
It can happen, that your build stops in case of an error. Normally `./build.sh` should handle image disconnection appropriately, but in rare cases, especially during a Docker build, this may not work as expected. If that happens, starting a new build will fail and you may have to disconnect the image and/or device yourself.
A typical message indicating that there are some orphaned device mapper entries is this:
```
Failed to set NBD socket
Disconnect client, due to: Unexpected end-of-file before all bytes were read
```
If that happens go through the following steps:
1. First, check if the image is somehow mounted to a directory entry and umount it as you would any other block device, like i.e. a hard disk or USB stick.
2. Second, to disconnect an image from `qemu-nbd`, the QEMU Disk Network Block Device Server, issue the following command (be sure to change the device name to the one actually used):
```bash
sudo qemu-nbd -d /dev/nbd1
```
Note: if you use Docker build, normally no active `qemu-nbd` process exists anymore as it will be terminated when the Docker container stops.
3. To disconnect a device partition map from the network block device, execute:
```bash
sudo kpartx -d /dev/nbd1
or
sudo ./imagetool.sh --cleanup
```
Note: The `imagetool.sh` command will cleanup any /dev/nbdX that is not connected to a running `qemu-nbd` daemon. Be careful if you use network block devices for other tasks utilizing NBDs on your build machine as well.
Now you should be able to start a new build without running into troubles again. Most of the time, especially when using Docker build, you will only need no. 3 to get everything up and running again.
# Troubleshooting # Troubleshooting
## `64 Bit Systems` ## `64 Bit Systems`

View file

@ -1,4 +1,5 @@
#!/bin/bash -eu #!/bin/bash -eu
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )" DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
BUILD_OPTS="$*" BUILD_OPTS="$*"
@ -47,6 +48,7 @@ fi
CONTAINER_NAME=${CONTAINER_NAME:-pigen_work} CONTAINER_NAME=${CONTAINER_NAME:-pigen_work}
CONTINUE=${CONTINUE:-0} CONTINUE=${CONTINUE:-0}
PRESERVE_CONTAINER=${PRESERVE_CONTAINER:-0} PRESERVE_CONTAINER=${PRESERVE_CONTAINER:-0}
PIGEN_DOCKER_OPTS=${PIGEN_DOCKER_OPTS:-""}
if [ -z "${IMG_NAME}" ]; then if [ -z "${IMG_NAME}" ]; then
echo "IMG_NAME not set in 'config'" 1>&2 echo "IMG_NAME not set in 'config'" 1>&2
@ -87,6 +89,10 @@ ${DOCKER} build --build-arg BASE_IMAGE=${BASE_IMAGE} -t pi-gen "${DIR}"
if [ "${CONTAINER_EXISTS}" != "" ]; then if [ "${CONTAINER_EXISTS}" != "" ]; then
trap 'echo "got CTRL+C... please wait 5s" && ${DOCKER} stop -t 5 ${CONTAINER_NAME}_cont' SIGINT SIGTERM trap 'echo "got CTRL+C... please wait 5s" && ${DOCKER} stop -t 5 ${CONTAINER_NAME}_cont' SIGINT SIGTERM
time ${DOCKER} run --rm --privileged \ time ${DOCKER} run --rm --privileged \
--cap-add=ALL \
-v /dev:/dev \
-v /lib/modules:/lib/modules \
${PIGEN_DOCKER_OPTS} \
--volume "${CONFIG_FILE}":/config:ro \ --volume "${CONFIG_FILE}":/config:ro \
-e "GIT_HASH=${GIT_HASH}" \ -e "GIT_HASH=${GIT_HASH}" \
--volumes-from="${CONTAINER_NAME}" --name "${CONTAINER_NAME}_cont" \ --volumes-from="${CONTAINER_NAME}" --name "${CONTAINER_NAME}_cont" \
@ -98,6 +104,10 @@ if [ "${CONTAINER_EXISTS}" != "" ]; then
else else
trap 'echo "got CTRL+C... please wait 5s" && ${DOCKER} stop -t 5 ${CONTAINER_NAME}' SIGINT SIGTERM trap 'echo "got CTRL+C... please wait 5s" && ${DOCKER} stop -t 5 ${CONTAINER_NAME}' SIGINT SIGTERM
time ${DOCKER} run --name "${CONTAINER_NAME}" --privileged \ time ${DOCKER} run --name "${CONTAINER_NAME}" --privileged \
--cap-add=ALL \
-v /dev:/dev \
-v /lib/modules:/lib/modules \
${PIGEN_DOCKER_OPTS} \
--volume "${CONFIG_FILE}":/config:ro \ --volume "${CONFIG_FILE}":/config:ro \
-e "GIT_HASH=${GIT_HASH}" \ -e "GIT_HASH=${GIT_HASH}" \
pi-gen \ pi-gen \
@ -106,6 +116,7 @@ else
rsync -av work/*/build.log deploy/" & rsync -av work/*/build.log deploy/" &
wait "$!" wait "$!"
fi fi
echo "copying results from deploy/" echo "copying results from deploy/"
${DOCKER} cp "${CONTAINER_NAME}":/pi-gen/deploy . ${DOCKER} cp "${CONTAINER_NAME}":/pi-gen/deploy .
ls -lah deploy ls -lah deploy

141
build.sh
View file

@ -1,4 +1,5 @@
#!/bin/bash -e #!/bin/bash -e
# shellcheck disable=SC2119 # shellcheck disable=SC2119
run_sub_stage() run_sub_stage()
{ {
@ -22,6 +23,11 @@ EOF
on_chroot << EOF on_chroot << EOF
apt-get -o APT::Acquire::Retries=3 install --no-install-recommends -y $PACKAGES apt-get -o APT::Acquire::Retries=3 install --no-install-recommends -y $PACKAGES
EOF EOF
if [ "${USE_QCOW2}" = "1" ]; then
on_chroot << EOF
apt-get clean
EOF
fi
fi fi
log "End ${SUB_STAGE_DIR}/${i}-packages-nr" log "End ${SUB_STAGE_DIR}/${i}-packages-nr"
fi fi
@ -32,6 +38,11 @@ EOF
on_chroot << EOF on_chroot << EOF
apt-get -o APT::Acquire::Retries=3 install -y $PACKAGES apt-get -o APT::Acquire::Retries=3 install -y $PACKAGES
EOF EOF
if [ "${USE_QCOW2}" = "1" ]; then
on_chroot << EOF
apt-get clean
EOF
fi
fi fi
log "End ${SUB_STAGE_DIR}/${i}-packages" log "End ${SUB_STAGE_DIR}/${i}-packages"
fi fi
@ -82,17 +93,30 @@ EOF
run_stage(){ run_stage(){
log "Begin ${STAGE_DIR}" log "Begin ${STAGE_DIR}"
STAGE="$(basename "${STAGE_DIR}")" STAGE="$(basename "${STAGE_DIR}")"
pushd "${STAGE_DIR}" > /dev/null pushd "${STAGE_DIR}" > /dev/null
unmount "${WORK_DIR}/${STAGE}"
STAGE_WORK_DIR="${WORK_DIR}/${STAGE}" STAGE_WORK_DIR="${WORK_DIR}/${STAGE}"
ROOTFS_DIR="${STAGE_WORK_DIR}"/rootfs ROOTFS_DIR="${STAGE_WORK_DIR}"/rootfs
if [ "${USE_QCOW2}" = "1" ]; then
if [ ! -f SKIP ]; then
load_qimage
fi
else
# make sure we are not umounting during export-image stage
if [ "${USE_QCOW2}" = "0" ] && [ "${NO_PRERUN_QCOW2}" = "0" ]; then
unmount "${WORK_DIR}/${STAGE}"
fi
fi
if [ ! -f SKIP_IMAGES ]; then if [ ! -f SKIP_IMAGES ]; then
if [ -f "${STAGE_DIR}/EXPORT_IMAGE" ]; then if [ -f "${STAGE_DIR}/EXPORT_IMAGE" ]; then
EXPORT_DIRS="${EXPORT_DIRS} ${STAGE_DIR}" EXPORT_DIRS="${EXPORT_DIRS} ${STAGE_DIR}"
fi fi
fi fi
if [ ! -f SKIP ]; then if [ ! -f SKIP ]; then
if [ "${CLEAN}" = "1" ]; then if [ "${CLEAN}" = "1" ] && [ "${USE_QCOW2}" = "0" ] ; then
if [ -d "${ROOTFS_DIR}" ]; then if [ -d "${ROOTFS_DIR}" ]; then
rm -rf "${ROOTFS_DIR}" rm -rf "${ROOTFS_DIR}"
fi fi
@ -103,13 +127,21 @@ run_stage(){
log "End ${STAGE_DIR}/prerun.sh" log "End ${STAGE_DIR}/prerun.sh"
fi fi
for SUB_STAGE_DIR in "${STAGE_DIR}"/*; do for SUB_STAGE_DIR in "${STAGE_DIR}"/*; do
if [ -d "${SUB_STAGE_DIR}" ] && if [ -d "${SUB_STAGE_DIR}" ] && [ ! -f "${SUB_STAGE_DIR}/SKIP" ]; then
[ ! -f "${SUB_STAGE_DIR}/SKIP" ]; then
run_sub_stage run_sub_stage
fi fi
done done
fi fi
if [ "${USE_QCOW2}" = "1" ]; then
unload_qimage
else
# make sure we are not umounting during export-image stage
if [ "${USE_QCOW2}" = "0" ] && [ "${NO_PRERUN_QCOW2}" = "0" ]; then
unmount "${WORK_DIR}/${STAGE}" unmount "${WORK_DIR}/${STAGE}"
fi
fi
PREV_STAGE="${STAGE}" PREV_STAGE="${STAGE}"
PREV_STAGE_DIR="${STAGE_DIR}" PREV_STAGE_DIR="${STAGE_DIR}"
PREV_ROOTFS_DIR="${ROOTFS_DIR}" PREV_ROOTFS_DIR="${ROOTFS_DIR}"
@ -143,6 +175,15 @@ do
esac esac
done done
term() {
if [ "${USE_QCOW2}" = "1" ]; then
log "Unloading image"
unload_qimage
fi
}
trap term EXIT INT TERM
export PI_GEN=${PI_GEN:-pi-gen} export PI_GEN=${PI_GEN:-pi-gen}
export PI_GEN_REPO=${PI_GEN_REPO:-https://github.com/RPi-Distro/pi-gen} export PI_GEN_REPO=${PI_GEN_REPO:-https://github.com/RPi-Distro/pi-gen}
@ -157,7 +198,7 @@ export IMG_FILENAME="${IMG_FILENAME:-"${IMG_DATE}-${IMG_NAME}"}"
export ZIP_FILENAME="${ZIP_FILENAME:-"image_${IMG_DATE}-${IMG_NAME}"}" export ZIP_FILENAME="${ZIP_FILENAME:-"image_${IMG_DATE}-${IMG_NAME}"}"
export SCRIPT_DIR="${BASE_DIR}/scripts" export SCRIPT_DIR="${BASE_DIR}/scripts"
export WORK_DIR="${WORK_DIR:-"${BASE_DIR}/work/${IMG_DATE}-${IMG_NAME}"}" export WORK_DIR="${WORK_DIR:-"${BASE_DIR}/work/${IMG_NAME}"}"
export DEPLOY_DIR=${DEPLOY_DIR:-"${BASE_DIR}/deploy"} export DEPLOY_DIR=${DEPLOY_DIR:-"${BASE_DIR}/deploy"}
export DEPLOY_ZIP="${DEPLOY_ZIP:-1}" export DEPLOY_ZIP="${DEPLOY_ZIP:-1}"
export LOG_FILE="${WORK_DIR}/build.log" export LOG_FILE="${WORK_DIR}/build.log"
@ -209,6 +250,18 @@ source "${SCRIPT_DIR}/common"
# shellcheck source=scripts/dependencies_check # shellcheck source=scripts/dependencies_check
source "${SCRIPT_DIR}/dependencies_check" source "${SCRIPT_DIR}/dependencies_check"
export NO_PRERUN_QCOW2="${NO_PRERUN_QCOW2:-1}"
export USE_QCOW2="${USE_QCOW2:-1}"
export BASE_QCOW2_SIZE=${BASE_QCOW2_SIZE:-12G}
source "${SCRIPT_DIR}/qcow2_handling"
if [ "${USE_QCOW2}" = "1" ]; then
NO_PRERUN_QCOW2=1
else
NO_PRERUN_QCOW2=0
fi
export NO_PRERUN_QCOW2="${NO_PRERUN_QCOW2:-1}"
dependencies_check "${BASE_DIR}/depends" dependencies_check "${BASE_DIR}/depends"
if [[ -n "${APT_PROXY}" ]] && ! curl --silent "${APT_PROXY}" >/dev/null ; then if [[ -n "${APT_PROXY}" ]] && ! curl --silent "${APT_PROXY}" >/dev/null ; then
@ -242,22 +295,98 @@ for EXPORT_DIR in ${EXPORT_DIRS}; do
# shellcheck source=/dev/null # shellcheck source=/dev/null
source "${EXPORT_DIR}/EXPORT_IMAGE" source "${EXPORT_DIR}/EXPORT_IMAGE"
EXPORT_ROOTFS_DIR=${WORK_DIR}/$(basename "${EXPORT_DIR}")/rootfs EXPORT_ROOTFS_DIR=${WORK_DIR}/$(basename "${EXPORT_DIR}")/rootfs
if [ "${USE_QCOW2}" = "1" ]; then
USE_QCOW2=0
EXPORT_NAME="${IMG_FILENAME}${IMG_SUFFIX}"
echo "------------------------------------------------------------------------"
echo "Running export stage for ${EXPORT_NAME}"
rm -f "${WORK_DIR}/export-image/${EXPORT_NAME}.img" || true
rm -f "${WORK_DIR}/export-image/${EXPORT_NAME}.qcow2" || true
rm -f "${WORK_DIR}/${EXPORT_NAME}.img" || true
rm -f "${WORK_DIR}/${EXPORT_NAME}.qcow2" || true
EXPORT_STAGE=$(basename "${EXPORT_DIR}")
for s in $STAGE_LIST; do
TMP_LIST=${TMP_LIST:+$TMP_LIST }$(basename "${s}")
done
FIRST_STAGE=${TMP_LIST%% *}
FIRST_IMAGE="image-${FIRST_STAGE}.qcow2"
pushd "${WORK_DIR}" > /dev/null
echo "Creating new base "${EXPORT_NAME}.qcow2" from ${FIRST_IMAGE}"
cp "./${FIRST_IMAGE}" "${EXPORT_NAME}.qcow2"
ARR=($TMP_LIST)
# rebase stage images to new export base
for CURR_STAGE in "${ARR[@]}"; do
if [ "${CURR_STAGE}" = "${FIRST_STAGE}" ]; then
PREV_IMG="${EXPORT_NAME}"
continue
fi
echo "Rebasing image-${CURR_STAGE}.qcow2 onto ${PREV_IMG}.qcow2"
qemu-img rebase -f qcow2 -u -b ${PREV_IMG}.qcow2 image-${CURR_STAGE}.qcow2
if [ "${CURR_STAGE}" = "${EXPORT_STAGE}" ]; then
break
fi
PREV_IMG="image-${CURR_STAGE}"
done
# commit current export stage into base export image
echo "Committing image-${EXPORT_STAGE}.qcow2 to ${EXPORT_NAME}.qcow2"
qemu-img commit -f qcow2 -p -b "${EXPORT_NAME}.qcow2" image-${EXPORT_STAGE}.qcow2
# rebase stage images back to original first stage for easy re-run
for CURR_STAGE in "${ARR[@]}"; do
if [ "${CURR_STAGE}" = "${FIRST_STAGE}" ]; then
PREV_IMG="image-${CURR_STAGE}"
continue
fi
echo "Rebasing back image-${CURR_STAGE}.qcow2 onto ${PREV_IMG}.qcow2"
qemu-img rebase -f qcow2 -u -b ${PREV_IMG}.qcow2 image-${CURR_STAGE}.qcow2
if [ "${CURR_STAGE}" = "${EXPORT_STAGE}" ]; then
break
fi
PREV_IMG="image-${CURR_STAGE}"
done
popd > /dev/null
mkdir -p "${WORK_DIR}/export-image/rootfs"
mv "${WORK_DIR}/${EXPORT_NAME}.qcow2" "${WORK_DIR}/export-image/"
echo "Mounting image ${WORK_DIR}/export-image/${EXPORT_NAME}.qcow2 to rootfs ${WORK_DIR}/export-image/rootfs"
mount_qimage "${WORK_DIR}/export-image/${EXPORT_NAME}.qcow2" "${WORK_DIR}/export-image/rootfs"
CLEAN=0
run_stage run_stage
CLEAN=1
USE_QCOW2=1
else
run_stage
fi
if [ "${USE_QEMU}" != "1" ]; then if [ "${USE_QEMU}" != "1" ]; then
if [ -e "${EXPORT_DIR}/EXPORT_NOOBS" ]; then if [ -e "${EXPORT_DIR}/EXPORT_NOOBS" ]; then
# shellcheck source=/dev/null # shellcheck source=/dev/null
source "${EXPORT_DIR}/EXPORT_NOOBS" source "${EXPORT_DIR}/EXPORT_NOOBS"
STAGE_DIR="${BASE_DIR}/export-noobs" STAGE_DIR="${BASE_DIR}/export-noobs"
if [ "${USE_QCOW2}" = "1" ]; then
USE_QCOW2=0
run_stage run_stage
USE_QCOW2=1
else
run_stage
fi
fi fi
fi fi
done done
if [ -x ${BASE_DIR}/postrun.sh ]; then if [ -x postrun.sh ]; then
log "Begin postrun.sh" log "Begin postrun.sh"
cd "${BASE_DIR}" cd "${BASE_DIR}"
./postrun.sh ./postrun.sh
log "End postrun.sh" log "End postrun.sh"
fi fi
if [ "${USE_QCOW2}" = "1" ]; then
unload_qimage
fi
log "End ${BASE_DIR}" log "End ${BASE_DIR}"

View file

@ -7,7 +7,7 @@ zerofree
zip zip
mkdosfs:dosfstools mkdosfs:dosfstools
capsh:libcap2-bin capsh:libcap2-bin
bsdtar bsdtar:libarchive-tools
grep grep
rsync rsync
xz:xz-utils xz:xz-utils
@ -17,3 +17,5 @@ file
git git
lsmod:kmod lsmod:kmod
bc bc
qemu-nbd:qemu-utils
kpartx

View file

@ -1,13 +1,18 @@
#!/bin/bash -e #!/bin/bash -e
IMG_FILE="${STAGE_WORK_DIR}/${IMG_FILENAME}${IMG_SUFFIX}.img" if [ "${NO_PRERUN_QCOW2}" = "0" ]; then
IMGID="$(dd if="${IMG_FILE}" skip=440 bs=1 count=4 2>/dev/null | xxd -e | cut -f 2 -d' ')" IMG_FILE="${STAGE_WORK_DIR}/${IMG_FILENAME}${IMG_SUFFIX}.img"
BOOT_PARTUUID="${IMGID}-01" IMGID="$(dd if="${IMG_FILE}" skip=440 bs=1 count=4 2>/dev/null | xxd -e | cut -f 2 -d' ')"
ROOT_PARTUUID="${IMGID}-02"
sed -i "s/BOOTDEV/PARTUUID=${BOOT_PARTUUID}/" "${ROOTFS_DIR}/etc/fstab" BOOT_PARTUUID="${IMGID}-01"
sed -i "s/ROOTDEV/PARTUUID=${ROOT_PARTUUID}/" "${ROOTFS_DIR}/etc/fstab" ROOT_PARTUUID="${IMGID}-02"
sed -i "s/BOOTDEV/PARTUUID=${BOOT_PARTUUID}/" "${ROOTFS_DIR}/etc/fstab"
sed -i "s/ROOTDEV/PARTUUID=${ROOT_PARTUUID}/" "${ROOTFS_DIR}/etc/fstab"
sed -i "s/ROOTDEV/PARTUUID=${ROOT_PARTUUID}/" "${ROOTFS_DIR}/boot/cmdline.txt"
fi
sed -i "s/ROOTDEV/PARTUUID=${ROOT_PARTUUID}/" "${ROOTFS_DIR}/boot/cmdline.txt"

View file

@ -76,25 +76,30 @@ cp "$ROOTFS_DIR/etc/rpi-issue" "$INFO_FILE"
dpkg -l --root "$ROOTFS_DIR" dpkg -l --root "$ROOTFS_DIR"
} >> "$INFO_FILE" } >> "$INFO_FILE"
ROOT_DEV="$(mount | grep "${ROOTFS_DIR} " | cut -f1 -d' ')"
unmount "${ROOTFS_DIR}"
zerofree "${ROOT_DEV}"
unmount_image "${IMG_FILE}"
mkdir -p "${DEPLOY_DIR}" mkdir -p "${DEPLOY_DIR}"
rm -f "${DEPLOY_DIR}/${ZIP_FILENAME}${IMG_SUFFIX}.zip" rm -f "${DEPLOY_DIR}/${ZIP_FILENAME}${IMG_SUFFIX}.zip"
rm -f "${DEPLOY_DIR}/${IMG_FILENAME}${IMG_SUFFIX}.img" rm -f "${DEPLOY_DIR}/${IMG_FILENAME}${IMG_SUFFIX}.img"
mv "$INFO_FILE" "$DEPLOY_DIR/"
if [ "${USE_QCOW2}" = "0" ] && [ "${NO_PRERUN_QCOW2}" = "0" ]; then
ROOT_DEV="$(mount | grep "${ROOTFS_DIR} " | cut -f1 -d' ')"
unmount "${ROOTFS_DIR}"
zerofree "${ROOT_DEV}"
unmount_image "${IMG_FILE}"
else
unload_qimage
make_bootable_image "${STAGE_WORK_DIR}/${IMG_FILENAME}${IMG_SUFFIX}.qcow2" "$IMG_FILE"
fi
if [ "${DEPLOY_ZIP}" == "1" ]; then if [ "${DEPLOY_ZIP}" == "1" ]; then
pushd "${STAGE_WORK_DIR}" > /dev/null pushd "${STAGE_WORK_DIR}" > /dev/null
zip "${DEPLOY_DIR}/${ZIP_FILENAME}${IMG_SUFFIX}.zip" \ zip "${DEPLOY_DIR}/${ZIP_FILENAME}${IMG_SUFFIX}.zip" \
"$(basename "${IMG_FILE}")" "$(basename "${IMG_FILE}")"
popd > /dev/null popd > /dev/null
else else
cp "$IMG_FILE" "$DEPLOY_DIR" mv "$IMG_FILE" "$DEPLOY_DIR/"
fi fi
cp "$INFO_FILE" "$DEPLOY_DIR"

View file

@ -1,47 +1,48 @@
#!/bin/bash -e #!/bin/bash -e
IMG_FILE="${STAGE_WORK_DIR}/${IMG_FILENAME}${IMG_SUFFIX}.img" if [ "${NO_PRERUN_QCOW2}" = "0" ]; then
IMG_FILE="${STAGE_WORK_DIR}/${IMG_FILENAME}${IMG_SUFFIX}.img"
unmount_image "${IMG_FILE}" unmount_image "${IMG_FILE}"
rm -f "${IMG_FILE}" rm -f "${IMG_FILE}"
rm -rf "${ROOTFS_DIR}" rm -rf "${ROOTFS_DIR}"
mkdir -p "${ROOTFS_DIR}" mkdir -p "${ROOTFS_DIR}"
BOOT_SIZE="$((256 * 1024 * 1024))" BOOT_SIZE="$((256 * 1024 * 1024))"
ROOT_SIZE=$(du --apparent-size -s "${EXPORT_ROOTFS_DIR}" --exclude var/cache/apt/archives --exclude boot --block-size=1 | cut -f 1) ROOT_SIZE=$(du --apparent-size -s "${EXPORT_ROOTFS_DIR}" --exclude var/cache/apt/archives --exclude boot --block-size=1 | cut -f 1)
# All partition sizes and starts will be aligned to this size # All partition sizes and starts will be aligned to this size
ALIGN="$((4 * 1024 * 1024))" ALIGN="$((4 * 1024 * 1024))"
# Add this much space to the calculated file size. This allows for # Add this much space to the calculated file size. This allows for
# some overhead (since actual space usage is usually rounded up to the # some overhead (since actual space usage is usually rounded up to the
# filesystem block size) and gives some free space on the resulting # filesystem block size) and gives some free space on the resulting
# image. # image.
ROOT_MARGIN="$(echo "($ROOT_SIZE * 0.2 + 200 * 1024 * 1024) / 1" | bc)" ROOT_MARGIN="$(echo "($ROOT_SIZE * 0.2 + 200 * 1024 * 1024) / 1" | bc)"
BOOT_PART_START=$((ALIGN)) BOOT_PART_START=$((ALIGN))
BOOT_PART_SIZE=$(((BOOT_SIZE + ALIGN - 1) / ALIGN * ALIGN)) BOOT_PART_SIZE=$(((BOOT_SIZE + ALIGN - 1) / ALIGN * ALIGN))
ROOT_PART_START=$((BOOT_PART_START + BOOT_PART_SIZE)) ROOT_PART_START=$((BOOT_PART_START + BOOT_PART_SIZE))
ROOT_PART_SIZE=$(((ROOT_SIZE + ROOT_MARGIN + ALIGN - 1) / ALIGN * ALIGN)) ROOT_PART_SIZE=$(((ROOT_SIZE + ROOT_MARGIN + ALIGN - 1) / ALIGN * ALIGN))
IMG_SIZE=$((BOOT_PART_START + BOOT_PART_SIZE + ROOT_PART_SIZE)) IMG_SIZE=$((BOOT_PART_START + BOOT_PART_SIZE + ROOT_PART_SIZE))
truncate -s "${IMG_SIZE}" "${IMG_FILE}" truncate -s "${IMG_SIZE}" "${IMG_FILE}"
parted --script "${IMG_FILE}" mklabel msdos parted --script "${IMG_FILE}" mklabel msdos
parted --script "${IMG_FILE}" unit B mkpart primary fat32 "${BOOT_PART_START}" "$((BOOT_PART_START + BOOT_PART_SIZE - 1))" parted --script "${IMG_FILE}" unit B mkpart primary fat32 "${BOOT_PART_START}" "$((BOOT_PART_START + BOOT_PART_SIZE - 1))"
parted --script "${IMG_FILE}" unit B mkpart primary ext4 "${ROOT_PART_START}" "$((ROOT_PART_START + ROOT_PART_SIZE - 1))" parted --script "${IMG_FILE}" unit B mkpart primary ext4 "${ROOT_PART_START}" "$((ROOT_PART_START + ROOT_PART_SIZE - 1))"
PARTED_OUT=$(parted -sm "${IMG_FILE}" unit b print) PARTED_OUT=$(parted -sm "${IMG_FILE}" unit b print)
BOOT_OFFSET=$(echo "$PARTED_OUT" | grep -e '^1:' | cut -d':' -f 2 | tr -d B) BOOT_OFFSET=$(echo "$PARTED_OUT" | grep -e '^1:' | cut -d':' -f 2 | tr -d B)
BOOT_LENGTH=$(echo "$PARTED_OUT" | grep -e '^1:' | cut -d':' -f 4 | tr -d B) BOOT_LENGTH=$(echo "$PARTED_OUT" | grep -e '^1:' | cut -d':' -f 4 | tr -d B)
ROOT_OFFSET=$(echo "$PARTED_OUT" | grep -e '^2:' | cut -d':' -f 2 | tr -d B) ROOT_OFFSET=$(echo "$PARTED_OUT" | grep -e '^2:' | cut -d':' -f 2 | tr -d B)
ROOT_LENGTH=$(echo "$PARTED_OUT" | grep -e '^2:' | cut -d':' -f 4 | tr -d B) ROOT_LENGTH=$(echo "$PARTED_OUT" | grep -e '^2:' | cut -d':' -f 4 | tr -d B)
echo "Mounting BOOT_DEV..." echo "Mounting BOOT_DEV..."
cnt=0 cnt=0
until BOOT_DEV=$(losetup --show -f -o "${BOOT_OFFSET}" --sizelimit "${BOOT_LENGTH}" "${IMG_FILE}"); do until BOOT_DEV=$(losetup --show -f -o "${BOOT_OFFSET}" --sizelimit "${BOOT_LENGTH}" "${IMG_FILE}"); do
if [ $cnt -lt 5 ]; then if [ $cnt -lt 5 ]; then
cnt=$((cnt + 1)) cnt=$((cnt + 1))
echo "Error in losetup for BOOT_DEV. Retrying..." echo "Error in losetup for BOOT_DEV. Retrying..."
@ -50,11 +51,11 @@ until BOOT_DEV=$(losetup --show -f -o "${BOOT_OFFSET}" --sizelimit "${BOOT_LENGT
echo "ERROR: losetup for BOOT_DEV failed; exiting" echo "ERROR: losetup for BOOT_DEV failed; exiting"
exit 1 exit 1
fi fi
done done
echo "Mounting ROOT_DEV..." echo "Mounting ROOT_DEV..."
cnt=0 cnt=0
until ROOT_DEV=$(losetup --show -f -o "${ROOT_OFFSET}" --sizelimit "${ROOT_LENGTH}" "${IMG_FILE}"); do until ROOT_DEV=$(losetup --show -f -o "${ROOT_OFFSET}" --sizelimit "${ROOT_LENGTH}" "${IMG_FILE}"); do
if [ $cnt -lt 5 ]; then if [ $cnt -lt 5 ]; then
cnt=$((cnt + 1)) cnt=$((cnt + 1))
echo "Error in losetup for ROOT_DEV. Retrying..." echo "Error in losetup for ROOT_DEV. Retrying..."
@ -63,23 +64,24 @@ until ROOT_DEV=$(losetup --show -f -o "${ROOT_OFFSET}" --sizelimit "${ROOT_LENGT
echo "ERROR: losetup for ROOT_DEV failed; exiting" echo "ERROR: losetup for ROOT_DEV failed; exiting"
exit 1 exit 1
fi fi
done done
echo "/boot: offset $BOOT_OFFSET, length $BOOT_LENGTH" echo "/boot: offset $BOOT_OFFSET, length $BOOT_LENGTH"
echo "/: offset $ROOT_OFFSET, length $ROOT_LENGTH" echo "/: offset $ROOT_OFFSET, length $ROOT_LENGTH"
ROOT_FEATURES="^huge_file" ROOT_FEATURES="^huge_file"
for FEATURE in metadata_csum 64bit; do for FEATURE in metadata_csum 64bit; do
if grep -q "$FEATURE" /etc/mke2fs.conf; then if grep -q "$FEATURE" /etc/mke2fs.conf; then
ROOT_FEATURES="^$FEATURE,$ROOT_FEATURES" ROOT_FEATURES="^$FEATURE,$ROOT_FEATURES"
fi fi
done done
mkdosfs -n boot -F 32 -v "$BOOT_DEV" > /dev/null mkdosfs -n boot -F 32 -v "$BOOT_DEV" > /dev/null
mkfs.ext4 -L rootfs -O "$ROOT_FEATURES" "$ROOT_DEV" > /dev/null mkfs.ext4 -L rootfs -O "$ROOT_FEATURES" "$ROOT_DEV" > /dev/null
mount -v "$ROOT_DEV" "${ROOTFS_DIR}" -t ext4 mount -v "$ROOT_DEV" "${ROOTFS_DIR}" -t ext4
mkdir -p "${ROOTFS_DIR}/boot" mkdir -p "${ROOTFS_DIR}/boot"
mount -v "$BOOT_DEV" "${ROOTFS_DIR}/boot" -t vfat mount -v "$BOOT_DEV" "${ROOTFS_DIR}/boot" -t vfat
rsync -aHAXx --exclude /var/cache/apt/archives --exclude /boot "${EXPORT_ROOTFS_DIR}/" "${ROOTFS_DIR}/" rsync -aHAXx --exclude /var/cache/apt/archives --exclude /boot "${EXPORT_ROOTFS_DIR}/" "${ROOTFS_DIR}/"
rsync -rtx "${EXPORT_ROOTFS_DIR}/boot/" "${ROOTFS_DIR}/boot/" rsync -rtx "${EXPORT_ROOTFS_DIR}/boot/" "${ROOTFS_DIR}/boot/"
fi

View file

@ -1,6 +1,6 @@
#!/bin/bash -e #!/bin/bash -e
NOOBS_DIR="${STAGE_WORK_DIR}/${IMG_DATE}-${IMG_NAME}${IMG_SUFFIX}" NOOBS_DIR="${STAGE_WORK_DIR}/${IMG_NAME}${IMG_SUFFIX}"
install -v -m 744 files/partition_setup.sh "${NOOBS_DIR}/" install -v -m 744 files/partition_setup.sh "${NOOBS_DIR}/"
install -v files/partitions.json "${NOOBS_DIR}/" install -v files/partitions.json "${NOOBS_DIR}/"
@ -20,7 +20,7 @@ BOOT_SIZE="$(( BOOT_SIZE / 1024 / 1024 + 1))"
ROOT_SIZE="$(( ROOT_SIZE / 1024 / 1024 + 1))" ROOT_SIZE="$(( ROOT_SIZE / 1024 / 1024 + 1))"
BOOT_NOM="256" BOOT_NOM="256"
ROOT_NOM="$(( ROOT_SIZE + 400 ))" ROOT_NOM="$(echo "$ROOT_SIZE" | awk '{printf "%.0f", (($1 + 400) * 1.2) + 0.5 }')"
mv "${NOOBS_DIR}/OS.png" "${NOOBS_DIR}/${NOOBS_NAME// /_}.png" mv "${NOOBS_DIR}/OS.png" "${NOOBS_DIR}/${NOOBS_NAME// /_}.png"
@ -37,7 +37,12 @@ sed "${NOOBS_DIR}/os.json" -i -e "s|UNRELEASED|${IMG_DATE}|"
sed "${NOOBS_DIR}/os.json" -i -e "s|NOOBS_NAME|${NOOBS_NAME}|" sed "${NOOBS_DIR}/os.json" -i -e "s|NOOBS_NAME|${NOOBS_NAME}|"
sed "${NOOBS_DIR}/os.json" -i -e "s|NOOBS_DESCRIPTION|${NOOBS_DESCRIPTION}|" sed "${NOOBS_DIR}/os.json" -i -e "s|NOOBS_DESCRIPTION|${NOOBS_DESCRIPTION}|"
sed "${NOOBS_DIR}/os.json" -i -e "s|RELEASE|${RELEASE}|" sed "${NOOBS_DIR}/os.json" -i -e "s|RELEASE|${RELEASE}|"
sed "${NOOBS_DIR}/os.json" -i -e "s|KERNEL|$(cat "${STAGE_WORK_DIR}/kernel_version")|"
sed "${NOOBS_DIR}/release_notes.txt" -i -e "s|UNRELEASED|${IMG_DATE}|" sed "${NOOBS_DIR}/release_notes.txt" -i -e "s|UNRELEASED|${IMG_DATE}|"
cp -a "${NOOBS_DIR}" "${DEPLOY_DIR}/" if [ "${USE_QCOW2}" = "1" ]; then
mv "${NOOBS_DIR}" "${DEPLOY_DIR}/"
else
cp -a "${NOOBS_DIR}" "${DEPLOY_DIR}/"
fi

View file

@ -1,7 +1,7 @@
{ {
"description": "NOOBS_DESCRIPTION", "description": "NOOBS_DESCRIPTION",
"feature_level": 35120124, "feature_level": 35120124,
"kernel": "4.19", "kernel": "KERNEL",
"name": "NOOBS_NAME", "name": "NOOBS_NAME",
"password": "raspberry", "password": "raspberry",
"release_date": "UNRELEASED", "release_date": "UNRELEASED",

View file

@ -1,4 +1,57 @@
UNRELEASED: UNRELEASED:
* Chromium upgraded to version 88.0.4324.187
* NuScratch upgraded to version 20210507
* Node-RED upgraded to version 1.3.4
* pigpio upgraded to version 1.79
* Thonny upgraded to version 3.3.6
* Icelandic and Italian translations updated for several packages
* piclone: Remove hiding of application in other desktops
* agnostics: Remove hiding of app in other desktops
* rp-bookshelf:
- Remove hiding of app in other desktops
- GTK+3 version
* lxplug-bluetooth:
- Fix some memory leaks
- Add authorisation dialog required by some BT-LE pairings
* alsa-utils: Add custom init files for bcm2835 on Raspberry Pi to set volume correctly
* rp-prefapps: Remove hiding of app in other desktops
* OpenSSH and OpenSSL speed improvements
* Install gpiozero in lite images
* Raspberry Pi firmware 518ee7c871aaa9aaa88116953d57e73787ee6e43
* Linux kernel 5.10.17
2021-03-04:
* Thonny upgraded to version 3.3.5
* SD Card Copier made compatible with NVMe devices; now built against GTK+3 toolkit
* Composite video options removed from Raspberry Pi 4 in Raspberry Pi Configuration
* Boot order options in raspi-config adjusted for more flexibility
* Recommended Software now built against GTK+3 toolkit
* Fix for crash in volume plugin when using keyboard could push value out of range
* Fix for focus changing between windows in file manager when using keyboard to navigate directory view
* Fix for Raspberry Pi 400 keyboard country not being read correctly in startup wizard
* Armenian and Japanese translations added to several packages
* Automatically load aes-neon-bs on ARM64 to speed up OpenSSL
* Raspberry Pi firmware fcf8d2f7639ad8d0330db9c8db9b71bd33eaaa28
* Linux kernel 5.10.17
2021-01-11:
* Chromium version 86.0.4240.197 included
* Screen reader support enabled in Chromium
* Adobe have end-of-lifed Flash Player, so it has been removed
* Scratch 2 required Flash, so it has been removed
* Added Epson printer drivers
* Added timeout to hide messages from USB device monitor after 5 seconds
* Bug fix - PulseAudio output was in mono
* Bug fix - brief audio interruptions at start of playback in VLC
* Bug fix - old ALSA output settings being used instead of PulseAudio settings by some applications
* Bug fix - crash in PulseAudio volume controller when used on multichannel devices
* Bug fix - battery monitor failing to load on x86 platforms
* Bug fix - setting of password in startup wizard failed if language was changed
* Bug fix - Chromium video playback lockup on small number of devices
* Bug fix - Chromium Google Maps 3D view artefacts
* Slovak, Italian and Norwegian translations updated
* Added Epson printer drivers
* Raspberry Pi firmware 70f1581eec2c036b7e9309f1af41c651fb125447
* Linux kernel 5.4.83
2020-12-02:
* PulseAudio now included and running by default * PulseAudio now included and running by default
* Bluealsa Bluetooth interface removed - Bluetooth audio is now handled by PulseAudio * Bluealsa Bluetooth interface removed - Bluetooth audio is now handled by PulseAudio
* LXPanel volume control plugin replaced with PulseAudio version * LXPanel volume control plugin replaced with PulseAudio version

View file

@ -1,11 +1,15 @@
#!/bin/bash -e #!/bin/bash -e
IMG_FILE="${STAGE_WORK_DIR}/${IMG_FILENAME}${IMG_SUFFIX}.img" NOOBS_DIR="${STAGE_WORK_DIR}/${IMG_NAME}${IMG_SUFFIX}"
NOOBS_DIR="${STAGE_WORK_DIR}/${IMG_DATE}-${IMG_NAME}${IMG_SUFFIX}"
unmount_image "${IMG_FILE}"
mkdir -p "${STAGE_WORK_DIR}" mkdir -p "${STAGE_WORK_DIR}"
cp "${WORK_DIR}/export-image/${IMG_FILENAME}${IMG_SUFFIX}.img" "${STAGE_WORK_DIR}/"
if [ "${DEPLOY_ZIP}" == "1" ]; then
IMG_FILE="${WORK_DIR}/export-image/${IMG_FILENAME}${IMG_SUFFIX}.img"
else
IMG_FILE="${DEPLOY_DIR}/${IMG_FILENAME}${IMG_SUFFIX}.img"
fi
unmount_image "${IMG_FILE}"
rm -rf "${NOOBS_DIR}" rm -rf "${NOOBS_DIR}"
@ -53,8 +57,15 @@ mount "$BOOT_DEV" "${STAGE_WORK_DIR}/rootfs/boot"
ln -sv "/lib/systemd/system/apply_noobs_os_config.service" "$ROOTFS_DIR/etc/systemd/system/multi-user.target.wants/apply_noobs_os_config.service" ln -sv "/lib/systemd/system/apply_noobs_os_config.service" "$ROOTFS_DIR/etc/systemd/system/multi-user.target.wants/apply_noobs_os_config.service"
KERNEL_VER="$(zgrep -oPm 1 "Linux version \K(.*)$" "${STAGE_WORK_DIR}/rootfs/usr/share/doc/raspberrypi-kernel/changelog.Debian.gz" | cut -f-2 -d.)"
echo "$KERNEL_VER" > "${STAGE_WORK_DIR}/kernel_version"
bsdtar --numeric-owner --format gnutar -C "${STAGE_WORK_DIR}/rootfs/boot" -cpf - . | xz -T0 > "${NOOBS_DIR}/boot.tar.xz" bsdtar --numeric-owner --format gnutar -C "${STAGE_WORK_DIR}/rootfs/boot" -cpf - . | xz -T0 > "${NOOBS_DIR}/boot.tar.xz"
umount "${STAGE_WORK_DIR}/rootfs/boot" umount "${STAGE_WORK_DIR}/rootfs/boot"
bsdtar --numeric-owner --format gnutar -C "${STAGE_WORK_DIR}/rootfs" --one-file-system -cpf - . | xz -T0 > "${NOOBS_DIR}/root.tar.xz" bsdtar --numeric-owner --format gnutar -C "${STAGE_WORK_DIR}/rootfs" --one-file-system -cpf - . | xz -T0 > "${NOOBS_DIR}/root.tar.xz"
if [ "${USE_QCOW2}" = "1" ]; then
rm "$ROOTFS_DIR/etc/systemd/system/multi-user.target.wants/apply_noobs_os_config.service"
fi
unmount_image "${IMG_FILE}" unmount_image "${IMG_FILE}"

114
imagetool.sh Executable file
View file

@ -0,0 +1,114 @@
#!/bin/bash
if [ "$(id -u)" != "0" ]; then
echo "Please run as root" 1>&2
exit 1
fi
progname=$(basename $0)
function usage()
{
cat << HEREDOC
Usage:
Mount Image : $progname [--mount] [--image-name <path to qcow2 image>] [--mount-point <mount point>]
Umount Image: $progname [--umount] [--mount-point <mount point>]
Cleanup NBD : $progname [--cleanup]
arguments:
-h, --help show this help message and exit
-c, --cleanup cleanup orphaned device mappings
-m, --mount mount image
-u, --umount umount image
-i, --image-name path to qcow2 image
-p, --mount-point mount point for image
This tool will use /dev/nbd1 as default for mounting an image. If you want to use another device, execute like this:
NBD_DEV=/dev/nbd2 ./$progname --mount --image <your image> --mount-point <your path>
HEREDOC
}
MOUNT=0
UMOUNT=0
IMAGE=""
MOUNTPOINT=""
nbd_cleanup() {
DEVS="$(lsblk | grep nbd | grep disk | cut -d" " -f1)"
if [ ! -z "${DEVS}" ]; then
for d in $DEVS; do
if [ ! -z "${d}" ]; then
QDEV="$(ps xa | grep $d | grep -v grep)"
if [ -z "${QDEV}" ]; then
kpartx -d /dev/$d && echo "Unconnected device map removed: /dev/$d"
fi
fi
done
fi
}
# As long as there is at least one more argument, keep looping
while [[ $# -gt 0 ]]; do
key="$1"
case "$key" in
-h|--help)
usage
exit
;;
-c|--cleanup)
nbd_cleanup
;;
-m|--mount)
MOUNT=1
;;
-u|--umount)
UMOUNT=1
;;
-i|--image-name)
shift
IMAGE="$1"
;;
-p|--mount-point)
shift
MOUNTPOINT="$1"
;;
*)
echo "Unknown option '$key'"
usage
exit
;;
esac
# Shift after checking all the cases to get the next option
shift
done
if [ "${MOUNT}" = "1" ] && [ "${UMOUNT}" = "1" ]; then
usage
echo "Concurrent mount options not possible."
exit
fi
if [ "${MOUNT}" = "1" ] && ([ -z "${IMAGE}" ] || [ -z "${MOUNTPOINT}" ]); then
usage
echo "Can not mount image. Image path and/or mount point missing."
exit
fi
if [ "${UMOUNT}" = "1" ] && [ -z "${MOUNTPOINT}" ]; then
usage
echo "Can not umount. Mount point parameter missing."
exit
fi
export NBD_DEV="${NBD_DEV:-/dev/nbd1}"
export MAP_BOOT_DEV=/dev/mapper/nbd1p1
export MAP_ROOT_DEV=/dev/mapper/nbd1p2
source scripts/qcow2_handling
if [ "${MOUNT}" = "1" ]; then
mount_qimage "${IMAGE}" "${MOUNTPOINT}"
elif [ "${UMOUNT}" = "1" ]; then
umount_qimage "${MOUNTPOINT}"
fi

View file

@ -21,8 +21,10 @@ bootstrap(){
setarch linux32 capsh --drop=cap_setfcap -- -c "'${BOOTSTRAP_CMD}' $BOOTSTRAP_STR" || true setarch linux32 capsh --drop=cap_setfcap -- -c "'${BOOTSTRAP_CMD}' $BOOTSTRAP_STR" || true
if [ -d "$2/debootstrap" ]; then if [ -d "$2/debootstrap" ] && ! rmdir "$2/debootstrap"; then
rmdir "$2/debootstrap" cp "$2/debootstrap/debootstrap.log" "${STAGE_WORK_DIR}"
log "bootstrap failed: please check ${STAGE_WORK_DIR}/debootstrap.log"
return 1
fi fi
} }
export -f bootstrap export -f bootstrap

View file

@ -28,11 +28,26 @@ dependencies_check()
false false
fi fi
# If we're building on a native arm platform, we don't need to check for
# binfmt_misc or require it to be loaded.
binfmt_misc_required=1
case $(uname -m) in
aarch64)
binfmt_misc_required=0
;;
arm*)
binfmt_misc_required=0
;;
esac
if [[ "${binfmt_misc_required}" == "1" ]]; then
if ! grep -q "/proc/sys/fs/binfmt_misc" /proc/mounts; then if ! grep -q "/proc/sys/fs/binfmt_misc" /proc/mounts; then
echo "Module binfmt_misc not loaded in host" echo "Module binfmt_misc not loaded in host"
echo "Please run:" echo "Please run:"
echo " sudo modprobe binfmt_misc" echo " sudo modprobe binfmt_misc"
exit 1 exit 1
fi fi
fi
} }

256
scripts/qcow2_handling Normal file
View file

@ -0,0 +1,256 @@
#!/bin/bash
# QCOW2 Routines
export CURRENT_IMAGE
export CURRENT_MOUNTPOINT
export NBD_DEV
export MAP_BOOT_DEV
export MAP_ROOT_DEV
# set in build.sh
# should be fairly enough for the beginning
# overwrite here by uncommenting following lines
# BASE_QCOW2_SIZE=12G
# find and initialize free block device nodes
init_nbd() {
modprobe nbd max_part=16
if [ -z "${NBD_DEV}" ]; then
for x in /sys/class/block/nbd* ; do
S=`cat $x/size`
if [ "$S" == "0" ] ; then
NBD_DEV=/dev/$(basename $x)
MAP_BOOT_DEV=/dev/mapper/$(basename $x)p1
MAP_ROOT_DEV=/dev/mapper/$(basename $x)p2
break
fi
done
fi
}
export -f init_nbd
# connect image to block device
connect_blkdev() {
init_nbd
qemu-nbd --discard=unmap -c $NBD_DEV "$1"
sync
kpartx -as $NBD_DEV
sync
CURRENT_IMAGE="$1"
}
export -f connect_blkdev
# disconnect image from block device
disconnect_blkdev() {
kpartx -d $NBD_DEV
qemu-nbd -d $NBD_DEV
NBD_DEV=
MAP_BOOT_DEV=
MAP_ROOT_DEV=
CURRENT_IMAGE=
}
export -f disconnect_blkdev
# mount qcow2 image: mount_image <image file> <mountpoint>
mount_qimage() {
connect_blkdev "$1"
mount -v -t ext4 $MAP_ROOT_DEV "$2"
mkdir -p "${ROOTFS_DIR}/boot"
mount -v -t vfat $MAP_BOOT_DEV "$2/boot"
CURRENT_MOUNTPOINT="$2"
}
export -f mount_qimage
# umount qcow2 image: umount_image <current mountpoint>
umount_qimage() {
sync
#umount "$1/boot"
while mount | grep -q "$1"; do
local LOCS
LOCS=$(mount | grep "$1" | cut -f 3 -d ' ' | sort -r)
for loc in $LOCS; do
echo "$loc"
while mountpoint -q "$loc" && ! umount "$loc"; do
sleep 0.1
done
done
done
CURRENT_MOUNTPOINT=
disconnect_blkdev
}
export -f umount_qimage
# create base image / backing image / mount image
load_qimage() {
if [ -z "${CURRENT_MOUNTPOINT}" ]; then
if [ ! -d "${ROOTFS_DIR}" ]; then
mkdir -p "${ROOTFS_DIR}";
fi
if [ "${CLEAN}" = "1" ] && [ -f "${WORK_DIR}/image-${STAGE}.qcow2" ]; then
rm -f "${WORK_DIR}/image-${STAGE}.qcow2";
fi
if [ ! -f "${WORK_DIR}/image-${STAGE}.qcow2" ]; then
pushd ${WORK_DIR} > /dev/null
init_nbd
if [ -z "${PREV_STAGE}" ]; then
echo "Creating base image: image-${STAGE}.qcow2"
# -o preallocation=falloc
qemu-img create -f qcow2 image-${STAGE}.qcow2 $BASE_QCOW2_SIZE
sync
qemu-nbd --discard=unmap -c $NBD_DEV image-${STAGE}.qcow2
sync
sfdisk $NBD_DEV << EOF
4MiB,250MiB,c,*
254MiB,,83;
EOF
sync
kpartx -as $NBD_DEV
mkdosfs -n boot -F 32 -v $MAP_BOOT_DEV
mkfs.ext4 -L rootfs -O "^huge_file,^metadata_csum,^64bit" $MAP_ROOT_DEV
sync
else
if [ ! -f "${WORK_DIR}/image-${PREV_STAGE}.qcow2" ]; then
exit 1;
fi
echo "Creating backing image: image-${STAGE}.qcow2 <- ${WORK_DIR}/image-${PREV_STAGE}.qcow2"
qemu-img create -f qcow2 \
-o backing_file=${WORK_DIR}/image-${PREV_STAGE}.qcow2 \
${WORK_DIR}/image-${STAGE}.qcow2
sync
qemu-nbd --discard=unmap -c $NBD_DEV image-${STAGE}.qcow2
sync
kpartx -as $NBD_DEV
fi
mount -v -t ext4 $MAP_ROOT_DEV "${ROOTFS_DIR}"
mkdir -p "${ROOTFS_DIR}/boot"
mount -v -t vfat $MAP_BOOT_DEV "${ROOTFS_DIR}/boot"
CURRENT_IMAGE=${WORK_DIR}/image-${STAGE}.qcow2
CURRENT_MOUNTPOINT=${ROOTFS_DIR}
popd > /dev/null
else
mount_qimage "${WORK_DIR}/image-${STAGE}.qcow2" "${ROOTFS_DIR}"
fi
echo "Current image in use: ${CURRENT_IMAGE} (MP: ${CURRENT_MOUNTPOINT})"
fi
}
export -f load_qimage
# umount current image and refresh mount point env var
unload_qimage() {
if [ ! -z "${CURRENT_MOUNTPOINT}" ]; then
fstrim -v "${CURRENT_MOUNTPOINT}" || true
umount_qimage "${CURRENT_MOUNTPOINT}"
fi
}
export -f unload_qimage
# based on: https://github.com/SirLagz/RaspberryPi-ImgAutoSizer
# helper function for make_bootable_image, do not call directly
function resize_qcow2() {
if [ -z "$CALL_FROM_MBI" ]; then
echo "resize_qcow2: cannot be called directly, use make_bootable_image instead"
return 1
fi
# ROOT_MARGIN=$((800*1024*1024))
ROOT_MARGIN=$((1*1024*1024))
PARTED_OUT=`parted -s -m "$NBD_DEV" unit B print`
PART_NO=`echo "$PARTED_OUT" | grep ext4 | awk -F: ' { print $1 } '`
PART_START=`echo "$PARTED_OUT" | grep ext4 | awk -F: ' { print substr($2,1,length($2)-1) } '`
e2fsck -y -f $MAP_ROOT_DEV || true
DATA_SIZE=`resize2fs -P $MAP_ROOT_DEV | awk -F': ' ' { print $2 } '`
BLOCK_SIZE=$(dumpe2fs -h $MAP_ROOT_DEV | grep 'Block size' | awk -F': ' ' { print $2 }')
BLOCK_SIZE=${BLOCK_SIZE// /}
let DATA_SIZE=$DATA_SIZE+$ROOT_MARGIN/$BLOCK_SIZE
resize2fs -p $MAP_ROOT_DEV $DATA_SIZE
sleep 1
let PART_NEW_SIZE=$DATA_SIZE*$BLOCK_SIZE
let PART_NEW_END=$PART_START+$PART_NEW_SIZE
ACT1=`parted -s "$NBD_DEV" rm 2`
ACT2=`parted -s "$NBD_DEV" unit B mkpart primary $PART_START $PART_NEW_END`
NEW_IMG_SIZE=`parted -s -m "$NBD_DEV" unit B print free | tail -1 | awk -F: ' { print substr($2,1,length($2)-1) } '`
}
export -f resize_qcow2
# create raw img from qcow2: make_bootable_image <in.qcow2> <out.img>
function make_bootable_image() {
EXPORT_QCOW2="$1"
EXPORT_IMAGE="$2"
echo "Connect block device to source qcow2"
connect_blkdev "${EXPORT_QCOW2}"
echo "Resize fs and partition"
CALL_FROM_MBI=1
resize_qcow2
sync
CALL_FROM_MBI=
echo "Disconnect block device"
disconnect_blkdev
if [ -z "$NEW_IMG_SIZE" ]; then
echo "NEW_IMG_SIZE could not be calculated, cannot process image. Exit."
exit 1
fi
echo "Shrinking qcow2 image"
qemu-img resize --shrink "${EXPORT_QCOW2}" $NEW_IMG_SIZE
sync
echo "Convert qcow2 to raw image"
qemu-img convert -f qcow2 -O raw "${EXPORT_QCOW2}" "${EXPORT_IMAGE}"
sync
echo "Get PARTUUIDs from image"
IMGID="$(blkid -o value -s PTUUID "${EXPORT_IMAGE}")"
BOOT_PARTUUID="${IMGID}-01"
echo "Boot: $BOOT_PARTUUID"
ROOT_PARTUUID="${IMGID}-02"
echo "Root: $ROOT_PARTUUID"
echo "Mount image"
MOUNTROOT=${WORK_DIR}/tmpimage
mkdir -p $MOUNTROOT
MOUNTPT=$MOUNTROOT
PARTITION=2
mount "${EXPORT_IMAGE}" "$MOUNTPT" -o loop,offset=$[ `/sbin/sfdisk -d "${EXPORT_IMAGE}" | grep "start=" | head -n $PARTITION | tail -n1 | sed 's/.*start=[ ]*//' | sed 's/,.*//'` * 512 ],sizelimit=$[ `/sbin/sfdisk -d "${EXPORT_IMAGE}" | grep "start=" | head -n $PARTITION | tail -n1 | sed 's/.*size=[ ]*//' | sed 's/,.*//'` * 512 ] || exit 1
MOUNTPT=$MOUNTROOT/boot
PARTITION=1
mount "${EXPORT_IMAGE}" "$MOUNTPT" -o loop,offset=$[ `/sbin/sfdisk -d "${EXPORT_IMAGE}" | grep "start=" | head -n $PARTITION | tail -n1 | sed 's/.*start=[ ]*//' | sed 's/,.*//'` * 512 ],sizelimit=$[ `/sbin/sfdisk -d "${EXPORT_IMAGE}" | grep "start=" | head -n $PARTITION | tail -n1 | sed 's/.*size=[ ]*//' | sed 's/,.*//'` * 512 ] || exit 1
if [ ! -d "${MOUNTROOT}/root" ]; then
echo "Image damaged or not mounted. Exit."
exit 1
fi
echo "Setup PARTUUIDs"
if [ ! -z "$BOOT_PARTUUID" ] && [ ! -z "$ROOT_PARTUUID" ]; then
echo "Set UUIDs to make it bootable"
sed -i "s/BOOTDEV/PARTUUID=${BOOT_PARTUUID}/" "${MOUNTROOT}/etc/fstab"
sed -i "s/ROOTDEV/PARTUUID=${ROOT_PARTUUID}/" "${MOUNTROOT}/etc/fstab"
sed -i "s/ROOTDEV/PARTUUID=${ROOT_PARTUUID}/" "${MOUNTROOT}/boot/cmdline.txt"
fi
echo "Umount image"
sync
umount "${MOUNTROOT}/boot" || exit 1
umount "${MOUNTROOT}" || exit 1
echo "Remove qcow2 export image"
rm -f "${EXPORT_QCOW2}"
}
export -f make_bootable_image

View file

@ -1,5 +1,5 @@
#!/bin/bash -e #!/bin/bash -e
if [ ! -d "${ROOTFS_DIR}" ]; then if [ ! -d "${ROOTFS_DIR}" ] || [ "${USE_QCOW2}" = "1" ]; then
bootstrap ${RELEASE} "${ROOTFS_DIR}" http://raspbian.raspberrypi.org/raspbian/ bootstrap ${RELEASE} "${ROOTFS_DIR}" http://raspbian.raspberrypi.org/raspbian/
fi fi

View file

@ -1 +1 @@
console=serial0,115200 console=tty1 root=ROOTDEV rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait console=serial0,115200 console=tty1 root=ROOTDEV rootfstype=ext4 fsck.repair=yes rootwait

View file

@ -2,6 +2,8 @@ ssh less fbset sudo psmisc strace ed ncdu crda
console-setup keyboard-configuration debconf-utils parted unzip console-setup keyboard-configuration debconf-utils parted unzip
build-essential manpages-dev python bash-completion gdb pkg-config build-essential manpages-dev python bash-completion gdb pkg-config
python-rpi.gpio v4l-utils python-rpi.gpio v4l-utils
python-gpiozero
python3-gpiozero
avahi-daemon avahi-daemon
lua5.1 lua5.1
luajit luajit

View file

@ -1,5 +1,5 @@
--- stage2.orig/rootfs/boot/cmdline.txt --- stage2.orig/rootfs/boot/cmdline.txt
+++ stage2/rootfs/boot/cmdline.txt +++ stage2/rootfs/boot/cmdline.txt
@@ -1 +1 @@ @@ -1 +1 @@
-console=serial0,115200 console=tty1 root=ROOTDEV rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait -console=serial0,115200 console=tty1 root=ROOTDEV rootfstype=ext4 fsck.repair=yes rootwait
+console=serial0,115200 console=tty1 root=ROOTDEV rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait quiet init=/usr/lib/raspi-config/init_resize.sh +console=serial0,115200 console=tty1 root=ROOTDEV rootfstype=ext4 fsck.repair=yes rootwait quiet init=/usr/lib/raspi-config/init_resize.sh