Search by Tags

OSTree

 

Article updated at 16 Jul 2020
Compare with Revision

Subscribe for this article updates

Warning: the Torizon OTA is a project currently under development at Toradex Labs. It is still an experimental project in it's early stages which is subject to changes without notice. This might impact new releases and/or iterations.

Introduction

OSTree is a library - actually libostree - that handles updates for filesystem trees, that is, the entire Linux root filesystem. It applies updates atomically, therefore guaranteeing the system integrity in case of a power-cut or dropped internet connectivity for instance. Different from the A/B partition mechanism of updates, OSTree works in a git-like model, providing file updates and thus saving storage space as well as connection bandwidth and allowing more than 2 software revisions on the device at the same time. In addition, some directories such as /var and /etc and handled as special cases, providing persistent configuration and user data to be preserved between updates.

OSTree is a key technology used by the Torizon Update System based on aktualizr. If aktualizr is used, OSTree is managed by it and should not be manipulated directly. This article shows how to use OSTree directly from command-line, useful for development or when the Torizon Update System/aktualizr is not being used.

This article complies to the Typographic Conventions for Torizon Documentation.

Prerequisites

Direct OSTree Update

This article shows how-to perform a TorizonCore update using the ostree command directly in a shell on the device.

Whenever TorizonCore is built using Yocto/OpenEmbedded's bitbake, a OSTree repository is automatically initialized and a new OSTree commit is created. For every following build, a new commit is created where only changed files are added to the repository. Inside the OSTree repository, the builds are organized in branches (or REFs). REFs follow a naming convention consisting of the TorizonCore major version number, machine name, distribution name, image name and release type, e.g.:

4/colibri-imx6/torizon/torizon-core-docker/nightly

REFs with a major version 0 are builds from sources taken from the master branch, typically meant for TorizonCore development only.

Toradex uploads the OSTree repository of TorizonCore builds produced by the Toradex CI infrastructure to make them available publicly https://feeds.toradex.com/ostree/.

This article shows how to update from both sources:

  • Toradex TorizonCore OSTree repository
  • Local OSTree repository from a local OpenEmbedded/bitbake build

Note: OSTree allows to switch between different sources transparently. E.g. it is possible to deploy a OSTree from a local build on a TorizonCore installation made from the official builds.

Warning: All device commands must be executed as root, so you have to prefix the following command lines with "sudo" to execute them as root.

Update From Toradex TorizonCore OSTree repository

This chapter shows how to update from a TorizonCore installation to a newer build obtained from the Toradex provided TorizonCore OSTree.

On the board, add the Toradex OSTree repository as a new remote using ostree remote, similar to how one would add a remote in a git repo:

# ostree remote add --no-gpg-verify torizon https://feeds.toradex.com/ostree/

Note: Currently Toradex does not sign the vanilla TorizonCore OSTree commits (hence the --no-gpg-verify is required). In the Torizon Update System the TorizonCore commits are cryptographic signed by Uptane/aktualizr.

The ostree admin commands allow to manage the OSTree on the host system. The command ostree admin status shows details about the currently active deployments (root filesystems).

# ostree admin status
* torizon 0b8e2f573de3d9c913abbb064674b04e2ce86da25ba7ea226b08a23db5cbd31c.0
    Version: 4.0.0-devel-202007+build.17
    origin refspec: torizon:4/colibri-imx6/torizon/torizon-core-docker/monthly

This shows an installation of a monthly build on a Colibri iMX6. At this point, there is only a single deployment active, the one from the original installation.

The ostree admin upgrade command can be used to upgrade to the latest monthly release. This command downloads the latest commit of the origin branch, and deploys that in one step. At the time of writing (July 16th) the monthly branch is not available on the OSTree repository. Future monthly releases are expected to be available.

Switch To a Different OSTree Branch

To switch to a different branch, e.g. to nightly build or a new major release, the branch name needs to be explicitly stated. The ostree remote refs command allows to get a list of branches available on a remote OSTree repository:

# ostree remote refs torizon
...
torizon:4/colibri-imx6/torizon/torizon-core-docker/nightly
...

Use OSTree pull to download the lastest version of a particular branch:

# ostree pull torizon:4/colibri-imx6/torizon/torizon-core-docker/nightly
32 metadata, 53 content objects fetched; 51978 KiB transferred in 21 seconds 

To see more details about the fetched OSTree commits, ostree log can be used:

# ostree log torizon:4/colibri-imx6/torizon/torizon-core-docker/nightly
commit 8ad8f2f63be22b271abfeec7a9fbb4d105b882ac08b33aeacfbdce412f79383e
ContentChecksum:  fc8b18d816f877671d52193221f888d9b651732f1644b08827bdc9bdec61d996
Date:  2020-07-15 19:43:31 +0000
Version: 4.0.0-devel-20200715+build.166

    4.0.0-devel-20200715+build.166

    meta-toradex-torizon    HEAD:75acc4a3acfd2104e47769b2a7e038f2503d57dc
    meta-toradex-distro     HEAD:276c966a04bbe4d19866becb3b8fc147d1079049
    meta-toradex-bsp-common HEAD:419eba81a51fb8f8e5a4eef9f23f1734c7044c78
    meta-oe                 HEAD:2b5dd1eb81cd08bc065bc76125f2856e9383e98b
    meta-networking         HEAD:2b5dd1eb81cd08bc065bc76125f2856e9383e98b
    meta-filesystems        HEAD:2b5dd1eb81cd08bc065bc76125f2856e9383e98b
    meta-python             HEAD:2b5dd1eb81cd08bc065bc76125f2856e9383e98b
    meta-perl               HEAD:2b5dd1eb81cd08bc065bc76125f2856e9383e98b
    meta-virtualization     HEAD:514ce5f6b240600f06b9956737be1e33900bacac
    meta-updater            HEAD:12e0cd9e18afadfdd90d81fb947950b4620901fc
    meta-toradex-nxp        HEAD:acbfe83716e6a045d9290d932eec29a9afb468f1
    meta-freescale          HEAD:f535ddcd5cd3da846a34b1721ceda3c8b18b37a0
    meta-freescale-3rdparty HEAD:dbcc686f52c3c84db8cb86aa8973a4e373651b98
    meta-yocto-bsp          HEAD:2d2dc20359ca75203f2194415b53731ad8ff0066
    meta-poky               HEAD:2d2dc20359ca75203f2194415b53731ad8ff0066
    meta-security           HEAD:440c37f0b623ccc0aa0328613908608d6362adda
    meta-lmp-base           HEAD:067487d1d4a823237eb6e7bbdebf9b023b58affa
    meta                    HEAD:ee95a399285abbde84e0148ca957b59d65bcad0a

<< History beyond this commit not fetched >>

To see which files have been changed, ostree diff can be used. To show the complete diff of the current running commit and the target REF, you have to specify the commit from the ostree admin status command explicitly (leave away the part after the dot).

# ostree diff 0b8e2f573de3d9c913abbb064674b04e2ce86da25ba7ea226b08a23db5cbd31c torizon:4/colibri-imx6/torizon/torizon-core-docker/nightly
M    /usr/package.manifest
M    /usr/bin/chardetect3
...

The command ostree admin deploy allows to create a new deployment using the given branch which will be used upon next boot:

# ostree admin deploy torizon:4/colibri-imx6/torizon/torizon-core-docker/nightly
Copying /etc changes: 4 modified, 1 removed, 7 added
Transaction complete; bootconfig swap: yes; deployment count change: 1

OSTree shows that there is a switch to a new root filesystem tree pending:

# ostree admin status
  torizon 8ad8f2f63be22b271abfeec7a9fbb4d105b882ac08b33aeacfbdce412f79383e.0 (pending)
    Version: 4.0.0-devel-20200715+build.166
    origin refspec: torizon:4/colibri-imx6/torizon/torizon-core-docker/nightly
* torizon 0b8e2f573de3d9c913abbb064674b04e2ce86da25ba7ea226b08a23db5cbd31c.0
    Version: 4.0.0-devel-202007+build.17
    origin refspec: torizon:4/colibri-imx6/torizon/torizon-core-docker/monthly

On the next boot, the system will boot using the new filesystem tree.

# reboot

After reboot you can view your current and previous deployment (still available for rollback):

# ostree admin status
* torizon 8ad8f2f63be22b271abfeec7a9fbb4d105b882ac08b33aeacfbdce412f79383e.0
    Version: 4.0.0-devel-20200715+build.166
    origin refspec: torizon:4/colibri-imx6/torizon/torizon-core-docker/nightly
  torizon 0b8e2f573de3d9c913abbb064674b04e2ce86da25ba7ea226b08a23db5cbd31c.0 (rollback)
    Version: 4.0.0-devel-202007+build.17
    origin refspec: torizon:4/colibri-imx6/torizon/torizon-core-docker/monthly

OSTree Update From a Local TorizonCore Build

If you are building TorizonCore image locally you may provide an ostree repo directly from your PC. If you have Python installed, just move to the ostree_repo folder of your build directory (located in the OpenEmbedded build directory under deploy/images/$MACHINE/ostree_repo) and bring-up the Python built-in simple HTTP server:

$ python -m SimpleHTTPServer 8081

In this way your PC will be sharing the image you just built on port 8081.

On the board, add the local OSTree repository using ostree remote, similar to how one would add a remote in a git repo:

# ostree remote add --no-gpg-verify local-build \
       http://<your PC host name/ip address>:8081/

You then can list the available branches and deploy a new tree using the same commands documented above (using the remote name local-build).

OSTree Update from a USB flash drive

If you don't have network connectivity on target board an update can also be deployed using a USB flash drive or SD card.

If you are building TorizonCore image locally the OSTree repository will be in deploy/images/$MACHINE/ostree_repo. The repository is in archive mode and hence can be copied with a regular copy command:

$ cp -r deploy/images/$MACHINE/ostree_repo path/to/flash-drive

You can also create a new OSTree repository and fetch an update from the Toradex TorizonCore OSTree repository:

$ cd path/to/flash-drive
$ ostree init --repo=ostree_repo --mode=archive
$ ostree remote add --repo=ostree_repo --no-gpg-verify torizon https://feeds.toradex.com/ostree/
$ ostree pull --repo=ostree_repo torizon:4/colibri-imx6/torizon/torizon-core-docker/nightly
$ ostree refs --repo=ostree_repo --create=4/colibri-imx6/torizon/torizon-core-docker/nightly torizon:4/colibri-imx6/torizon/torizon-core-docker/nightly

Note: It seems that the ostree pull-local command used on the target cannot handle REFs with a colon, hence a new remote independent reference needs to be created.

On the target, list the refs available on the given repository:

# ostree refs --repo=path/to/flash-drive
torizon:4/colibri-imx6/torizon/torizon-core-docker/nightly
4/colibri-imx6/torizon/torizon-core-docker/nightly

Pull the commit from the given branch name from the USB flash drive to the target board:

# ostree pull-local /home/root/pendrive/ostree_repo/ 4/colibri-imx6/torizon/torizon-core-docker/nightly

Create a new deployment from the given branch name:

# ostree admin deploy 4/colibri-imx6/torizon/torizon-core-docker/nightly

OSTree shows that there is a switch to a new root filesystem tree pending:

# ostree admin status
  torizon 1d98e25015a390556e7ab93c9b94e01e8632d30d800b36bff1ddd54fa9fc7cf4.0 (pending)
    origin refspec: apalis-imx6/torizon/torizon-core-docker
* torizon d0d53487d305358ad50c37b1b2a94fc78eaea5e770749a33f5607573e2447660.0
    origin refspec: d0d53487d305358ad50c37b1b2a94fc78eaea5e770749a33f5607573e2447660

On the next boot, the system will boot using the new filesystem tree.

# reboot

OSTree Rollback

OSTree allows to rollback to the last root filesystem in case of a bad update. A rollback can be manually triggered by setting the U-Boot environment variable rollback:

> env set rollback 1

The boot script will load the kernel/device tree and initramfs from the rollback deployment. This is shown in the boot message as well as visible when looking at ostree admin status

...
Rollback enabled. Booting previously deployed version.
...
# ostree admin status
  torizon 8ad8f2f63be22b271abfeec7a9fbb4d105b882ac08b33aeacfbdce412f79383e.0 (pending)
    Version: 4.0.0-devel-20200715+build.166
    origin refspec: torizon:4/colibri-imx6/torizon/torizon-core-docker/nightly
* torizon 0b8e2f573de3d9c913abbb064674b04e2ce86da25ba7ea226b08a23db5cbd31c.0
    Version: 4.0.0-devel-202007+build.17
    origin refspec: torizon:4/colibri-imx6/torizon/torizon-core-docker/monthly

In this state a new (hopefully working) OSTree can be deployed.

Automatic Rollback

TorizonCore relies on U-Boot boot count support to provide automatic rollback in case a bad update has been deployed. Any update that causes issues is expected to reboot. For that TorizonCore configures the kernel to automatically reboot in case of a kernel panic, and instructs systemd to reboot in case a user defined critical application (by default docker.service) does not start. If the system detects more reboots than what is defined in U-Boot's bootlimit environment variable, the rollback mechanism will be triggered. With aktualizr the U-Boot boot count is automatically enabled after an upgrade. When manually deploying an update using OSTree (ostree admin deploy), the boot count support needs to be enabled manually to enable automatic rollback support:

  • For emmc based modules
# boot_partition=$(readlink /dev/emmc-boot0)
# echo 0 > /sys/block/${boot_partition}/force_ro
# fw_setenv upgrade_available 1
# fw_setenv bootcount 0
# echo 1 > /sys/block/${boot_partition}/force_ro
  • For MTD based modules
# fw_setenv upgrade_available 1
# fw_setenv bootcount 0

After the reboot, the output of the ostree admin status can be used to see if the upgrade succeeded.

After an unsuccessful update, the system stays in "rollback" mode. A new update need to be deployed and the rollback variable needs to be cleared:

# fw_setenv rollback 0

After a successful update, the boot count should be disabled again (since regular flash is used as boot counter, this prevents unnecessary wear on the flash used for the U-Boot environment)

# fw_setenv upgrade_available 0