Search by Tags

TorizonCore Builder Tool

 

Article updated at 17 Sep 2020
Compare with Revision

Subscribe for this article updates

Select the version of your OS from the tabs below. If you don't know the version you are using, run the command cat /etc/os-release or cat /etc/issue on the board.

Torizon 5.0.0

Introduction

Toradex provides Easy Installer to flash images on modules. Developer may make changes in configuration (/etc) of the system, need to compile device tree, apply device tree overlay or work with containers for the end user solution. Now how to replicate the same on other modules? Toradex has developed a new tool, TorizonCore Builder, to let the developer pick those modifications/containers, compile device trees, apply device tree overlays and create another image out of it, keeping the original Toradex Easy Installer image of Torizon as base. TorizonCore builder is containerized and run on the host system.

This article complies to the Typographic Conventions for Torizon Documentation

Workflow


  • TorizonCore Builder Workflow

    TorizonCore Builder Workflow

Commands

TorizonCore Builder is provided in the form of Docker image and has following commands to create Toradex Easy Installer image of Torizon with required changes

  • Unpack

  • Bundle

  • Dt (Device Tree)

  • Isolate

  • Union

  • Deploy

  • Combine

Operation

Firstly, user needs a base Toradex Easy Installer image of Torizon which will be used by TorizonCore Builder container to add user changes in it. User has two options, either to perform all tasks in a “working directory”, which must be a file system capable of carrying Linux file system metadata (such as Unix style file system permissions and extended attributes), by setting appropriate arguments OR let the commands use Docker volumes for intermediate storage to perform the operations on image.

  • The very first command to use is unpack, which unpacks the OSTree file system from user provided Toradex Easy Installer image of Torizon

  • If pre-installation of Docker container(s) onto a Torizon image with Docker engine is required, bundle command is to be used by providing a Docker compose file stating all image(s).

  • If device tree related functionality like building a complete device tree or applying device tree overlays are needed, dt with subcommands is to be used.

  • If user has made any changes in configuration (/etc), isolate command is to be used to get all those changes (modifications, addition and deletion). It requires ssh connection to the target board.

  • If user needs to change the splash screen, splash command is to be used.

  • If user has used isolate, splash and/or dt command, union command is to be used to capture all changes and create a branch out of it for OSTree repo.

  • If user has used union command, deploy command is to be used to deploy the OSTree branch either on the target machine connected via ssh OR onto the OSTree file system (from unpacked Toradex Easy Installer image of Torizon) and create a new Toradex Easy Installer image of Torizon.

  • Combine is used to create a Toradex Easy Installer image of Torizon by combining output of bundle command with a Toradex Easy Installer Image of Torizon. That Toradex Easy Installer Image of Torizon can be either output image from deploy command or unpacked base image.

Usage

Warning: Make sure to use an empty working directory and navigate to it. This will be bind mounted to the container this tool is running in and will be used to store the final image, and may also use to store temporary files.

Note: Toradex Easy Installer image of Torizon can be downloaded from Toradex Easy Installer

Extract the downloaded image tar file

$ mkdir <directory to extract image into>
$ tar -xf <downloaded image.tar> -C <directory to extract image into>

Basic

For convenience, create alias first

$ alias torizoncore-builder='docker run --rm -it -v $(pwd):/workdir -v storage:/storage -v /deploy --net=host -v /var/run/docker.sock:/var/run/docker.sock torizon/torizoncore-builder:1'

TorizonCore Builder tool uses Docker volumes as persistent storage for intermediate outputs of sub-commands.

TorizonCore Builder tool provides different logging levels, which can be set by using --log-level option. There are five logging levels i.e. debug, info, warning, error and critical and default is set to info. Formatted logging can be enabled by using --verbose option.

Sub-commands of TorizonCore Builder tool are used as follows

Unpack

unpack gets the base Toradex Easy Installer image of Torizon and unpacks the rootfs (OSTree filesystem) to Docker volume

$ torizoncore-builder unpack --image-directory <base Toradex Easy Installer image of Torizon>

Dt (Device Tree)

dt has subcommands to perform device tree related operations as shown below.

dt checkout

The dt checkout subcommand allows to checkout Toradex maintained overlays, base device trees (dts files), device tree include files (.dtsi files) and required header files (.h files). The command clones the repository device-trees and makes sure to checkout the kernel branch used by the currently unpacked TorizonCore image.

Note: Refer to Device Tree Overlays on Torizon for more advanced options

$ torizoncore-builder dt checkout
dt custom

dt custom subcommand is to be used if device tree source file is to be compiled. --include-directory can be used multiple times to provide multiple directories to search for .dtsi or .h file(s).

$ torizoncore-builder dt custom --devicetree-source <path to device tree source file> --include-dir <path to directory containing dtsi and/or .h files> --include-dir <optionally can be used multiple times to provided multiple directories>
dt list-overlays

dt list-overlays subcommand is used to search and list compatible device tree overlay source files from user provided directory, for either user provided device tree source file or device tree binary file.

for device tree source file

$ torizoncore-builder dt list-overlays --devicetree-source <path to device tree source file> --overlays-dir <path to directory containing overlay files to be filtered out based on compatibility>

for device tree binary file

$ torizoncore-builder dt list-overlays --devicetree-bin <path to device tree binary file> --overlays-dir <path to directory containing overlay files to be filtered out based on compatibility>
dt list-devicetrees

dt list-devicetrees subcommand is used to list available device tree binaries in unpacked Toradex Easy Installer image of Torizon.

$ torizoncore-builder dt list-devicetrees
dt overlay

dt overlay subcommand is used for compiling and applying the device tree overlay to user provided device tree binary file. --include-dir can be used multiple times to provide multiple directories. Overlay(s) are to be provided as positional argument i.e. provide them without specifying any argument.

If user does not provide --devicetree option, devicetree named device tree binary is searched in unpacked Toradex Easy Installer image of Torizon and then, if available, overlay(s) are applied to it.

If user provides a device tree binary without specifying any path, it is searched in unpacked Toradex Easy Installer image of Torizon and then, if available, overlay(s) are applied to it.

If user provides a device tree binary with a path, it is searched in that path and overlays are applied to it. If it is not found in user provided path, it is searched inside unpacked Toradex Easy Installer image of Torizon and then, if available, overlay(s) are applied to it.

$ torizoncore-builder dt overlay --include-dir <path to directory containing dtsi and/or .h files> --include-dir <optionally can be used multiple times to provided multiple directories> <overlay file(s)>

Or user can provide device tree binary that needs to be overlaid

$ torizoncore-builder dt overlay --devicetree <devcie tree name or path to it> --include-dir <path to directory containing dtsi and/or .h files> --include-dir <optionally can be used multiple times to provided multiple directories>  <overlay file(s)>

Isolate

isolate is used to get configuration changes(modifications/addition/deletion) from the target board. It creates ssh session with target board to get changes and store them within Docker volume on host.

$ torizoncore-builder isolate --remote-ip <IP of target board> --remote-username <username on target board> --remote-password <password of user on target board>

Note: --mdns-source option can be specified optionally to provide IP address as mDNS source, when multiple interfaces are used, and mDNS multicast requests are sent out the wrong network interface.

Union

union is used to make an OSTree branch(containing the commit for changes) for all changes provided by user, to be made in OSTree rootfs of unpacked base Torizon image. It picks all changes made as a result of isolate, splash dt and any other user provided change to be made.

$ torizoncore-builder union --union-branch <name to be given to the branch containing the changes> --extra-changes-directory <additional path with user changes to be committed> --extra-changes-directory <optionally can be provided multiple times to provide multiple directories>

Deploy

deploy is used to take branch(containing the commit for changes) from union command, make it default deployment. Deployment can be either set on unpacked Toradex Easy Installer image of Torizon or directly on the target board after creating a ssh session.

If deployment is to be set on the unpacked Toradex Easy Installer image of Torizon, it makes directory (inside bind mounted working directory) with user provided name and store Torizon image in it as output.

Note: Output directory is not needed to be created.

$ torizoncore-builder deploy --output-directory <name of directory to store Torizon image> <name or checksum of branch from Union command>

If deployment is to be set on the target board, user needs to provide host name or IP, username and password.

$ torizoncore-builder deploy --remote-host <IP or hostname(.local) of target board> --remote-username <username of target board> --remote-password <password of user on target board>  <name or checksum of branch from Union command> 

Bundle

bundle is used to bundle the Docker container(s), which is used by combine command to pre-provision container(s) bundle with Torizon image.

User provides the docker-compose.yml, platform of target board (linux/arm/v7 or linux/arm64) and working directory of host to store data used for DIND (docker in docker) and bundle command stores the Docker container(s) tar archive in directory named “bundle” inside the working directory.

$ torizoncore-builder bundle --file <path to docker-compose.yml> --platform <target board like linux/arm/v7 or linux/arm64> --host-workdir $(pwd)

Combine

combine command is used to create Toradex Easy Installer image of Torizon by combining user provided Toradex Easy Installer image of Torizon (either base Torizon Image OR output (output image) from deploy command) with bundled Docker container(s) from bundle command.

$ torizoncore-builder combine --image-directory <path to Torizon image> --output-directory <Torizon image with pre-provisioned Docker container(s)>

Example

Note: This section does not show how to use dt command. For dt command usage, refer to Device Tree Overlays on Torizon

Following is the usage of tool with Docker volume

Creating alias

$ alias torizoncore-builder='docker run -it --rm -v $(pwd):/workdir -v storage:/storage -v /deploy --net=host -v /var/run/docker.sock:/var/run/docker.sock torizon/torizoncore-builder:1'

Unpack base Toradex Easy Installer image of Torizon

$ torizoncore-builder unpack --image-directory colibri-imx7
Copying Toradex Easy Installer image.
Unpacking TorizonCore Toradex Easy Installer image.
Unpacked OSTree from oradex Easy Installer image:
Commit ref: 3c5795a0dcd0ceca552c916f6515a84011425443fd14d118caa09b869dbe4804
TorizonCore Version: 4.0.0-devel-20200607+build.117

Get configuration changes from target board

$ torizoncore-builder isolate --remote-ip 192.168.1.146 --remote-username torizon --remote-password 1234 
isolation command completed

Merge changes to base Toradex Easy Installer image of Torizon

$ torizoncore-builder union --union-branch=changes_committed
Commit d77ed9d5ddec41b7b61f9a571e810de2447133391ee19fe5f34d1a81009967d1 has been generated for changes and ready to be deployed. 

Set branch to deploy at boot. Output image can be deployed on target board.

$ torizoncore-builder deploy --output-directory deployed_image d77ed9d5ddec41b7b61f9a571e810de2447133391ee19fe5f34d1a81009967d1
Running tar command: tar --xattrs --xattrs-include='*' -cf /workdir/deployed_image/torizon-core-docker-colibri-imx7.ota.tar.xz --xz -S -C /deploy -p .
Using unpacked Toradex Easy Installer image as base:
  Commit ref: 3c5795a0dcd0ceca552c916f6515a84011425443fd14d118caa09b869dbe4804
  TorizonCore Version: 4.0.0-devel-20200607+build.117
  Kernel arguments: quiet logo.nologo vt.global_cursor_default=0 plymouth.ignore-serial-consoles splash ostree=/ostree/boot.1/torizon/d103ac800d2c9a67cce7ec68ad1accb836d766ff959895820deea195f16eaeed/0

Deploying commit ref: d77ed9d5ddec41b7b61f9a571e810de2447133391ee19fe5f34d1a81009967d1
Pulling OSTree with ref d77ed9d5ddec41b7b61f9a571e810de2447133391ee19fe5f34d1a81009967d1 from local repository...
Pulling done.
Deploying OSTree with ref d77ed9d5ddec41b7b61f9a571e810de2447133391ee19fe5f34d1a81009967d1
Deploying done.
Copy rootdirs such as /home from original deployment.
Packing rootfs...
Packing rootfs done.

If branch is to be deployed on the target board

$ torizoncore-builder deploy d36d7f8b393e9226af780d16c0813eaa087fd35ff1251c86b5dc0d61a9ca29b3 --remote-host 192.168.1.111 --remote-username torizon --remote-password 1234
Pulling OSTree with ref d36d7f8b393e9226af780d16c0813eaa087fd35ff1251c86b5dc0d61a9ca29b3 (checksum d36d7f8b393e9226af780d16c0813eaa087fd35ff1251c86b5dc0d61a9ca29b3) from local archive repository...
Starting http server to serve OSTree.
Starting OSTree pull on the device...
Deploying new OSTree on the device...
Deploying successfully finished.
Please reboot the device to boot into the new deployment.

If required, create tar archive of Docker container(s)

$ torizoncore-builder --bundle-directory bundled_containers  bundle -f docker-compose.yml --platform linux/arm/v7 --host-workdir $(pwd)

Create image with pre-provisioned docker containers. --image-directory can be either base image or output from deploy command.

$ mkdir preprov_cont_image
$ torizoncore-builder --bundle-directory bundled_containers combine --image-directory deployed_image --output-directory preprov_cont_image
Creating copy of TorizonCore source image.
Combining TorizonCore image with Docker Container bundle.
Successfully created a TorizonCore image with Docker Containers preprovisioned in preprov_cont_image

Advanced

User can use TorizonCore Builder tool by either providing the working directory (capable of carrying Linux filesystem metadata) or Docker volumes for intermediate data storage. So all commands are explained with both approaches. Commands can be used with mix of Docker volumes and working directory but that will complicate the usage. So it is recommended to use either working directory or Docker volumes for all the commands.

As it is provided in the form of Docker image, any of the following commands (based on either working directory or Docker volumes are to be used) can be executed to create an alias to be easy to use.

Note: Usage of all commands explained later are based on the following aliases.

Warning: Move to the working directory before executing alias commands, so the required one gets bind mounted when alias is executed

With Docker volumes:

$ alias torizoncore-builder='docker run --rm -it -v $(pwd):/workdir -v storage:/storage -v /deploy --net=host -v /var/run/docker.sock:/var/run/docker.sock torizon/torizoncore-builder:1'

Here, two volumes are being created and mounted at /storage and /deploy of container for internal usage and host current directory is bind mounted at /workdir within container.

With user provided working directory (capable of handling Linux filesystem metadata):

$ alias torizoncore-builder='docker run --rm -it -v $(pwd):/workdir --net=host -v /var/run/docker.sock:/var/run/docker.sock torizon/torizoncore-builder:1'

Base Command

Base command is now alias torizoncore-builder and has below optional options

--verbose: enable formatted logs

--log-level: set log level from options debug, info, warning, error and critical
default: info

--log-file: write logs to specified file instead of console

--bundle-directory: path to directory to store output of bundle command
default: creates and use directory with name bundle in working directory

--storage-directory: path to directory to use as intermediate storage for sub-commands.
default: Docker volume mounted at/storage Note: Do not set it if Docker volumes are to be used.

Unpack

Unpack command is used to unpack the OSTree filesystem from user provided Toradex Easy Installer image of Torizon base image.

Arguments

--image-directory : path to base Toradex Easy Installer image of Torizon in the working directory

Example

Note: If there is already an unpacked OSTree filesystem, user is prompted to delete it because currently a single OSTree filesystem can be handled.

With docker volume
$ torizoncore-builder unpack --image-directory <path to base Toradex Easy Installer image of Torizon in the working directory>  
Without Docker Volume

Additional argument --storage-directory specified with alias torizoncore-builder, specifies path(capable of handling Linux filesystem metadata) to be used for storage of intermediate data. It will be used to store unpacked OSTree rootfs as a result of this command.

$ torizoncore-builder unpack --image-directory <path to base Toradex Easy Installer image of Torizon in working directory>

Dt (Device Tree)

Refer to Device Tree Overlays on Torizon for details on advanced usage.

Isolate

Isolate command is used to get the configuration (/etc) changes (modifications, additions and deletions) from user target module. It ignores following files/directories for any change

'gshadow', 'machine-id', 'group', 'shadow', 'systemd/system/sysinit.target.wants/run-postinsts.service',
'ostree/remotes.d/toradex-nightly.conf', 'docker/key.json', '.updated', '.pwd.lock', 'group-',
'gshadow-', 'hostname', 'ssh/ssh\_host\_rsa\_key', 'ssh/ssh\_host\_rsa\_key.pub', 'ssh/ssh\_host\_ecdsa\_key',
'ssh/ssh\_host\_ecdsa\_key.pub','ssh/ssh\_host\_ed25519\_key','ssh/ssh\_host\_ed25519\_key.pub',
'ipk-postinsts', 'fw\_env.conf'

Isolate command establishes a ssh session with the target board, gets all the changes and places them in either user provided directory or within docker volume.

Arguments

--remote-ip: IP of target board

--remote-username: username of target board, to be used for creating ssh session

--remote-password: user password of target board, to be used for creating ssh session

--changes-directory (optional): If docker volume is not to be used, directory for storing configuration changes
default: changes directory inside either user provided storage_directory using --storage-directory option OR docker volume (mounted at /storage)

`--mdns-source:` Use the given IP address as mDNS source. This is useful when multiple interfaces are used, and mDNS multicast requests are sent out the wrong                         network interface.
Example
With docker volume
$torizoncore-builder  isolate --remote-ip <target board IP> --remote-username <username of target board > --remote-password <user password on target board> --mdns-source <Use the given IP address as mDNS source>
without Docker volume

To use without Docker volume, directory path is needed (--changes-directory) to store changes inside user provided working directory

$ torizoncore-builder --storage-directory <Path to store intermediate data. Must be a file system capable of carrying Linux file system metadata (Unix file permissions and xattr)>  isolate --remote-ip <target board IP> --remote-username <username of target board> --remote-password <user password on target board > --changes-directory <directory to store changes> --mdns-source <Use the given IP address as mDNS source>

Union

It is to be run to commit changes made by isolate, splash and dt commands to the OSTree repo i.e. unpacked Toradex Easy Installer image of Torizon. If changes are stored in Docker volume, it is to be run with Docker volume and if changes are stored in working directory, same directory is needed to be provided to the command.

Arguments

----union-branch: branch name containing the changes committed to OSTree repo of unpacked OSTree filesystem

--changes-directory(optional): path (within working directory) to changes made by isolate, splash and dt commands. Each change can be provided with single option, so use it multiple times to provide multiple directories. default: changes directory inside either user provided storage_directory using --storage-directory option OR docker volume (mounted at /storage)

--extra-changes-directory(optional): additional path with user changes to be committed. Each change can be provided with single option, so use it multiple times to provide multiple directories. default: None

--subject: OSTree commit subject default: TorizonCore Builder [timestamp]

--body: OSTree commit body message

Example
with Docker volume
$ torizoncore-builder union --union-branch <name to be given to the branch> --extra-changes-directory <path to directory containing changes to be committed> --extra-changes-directory <optionally can be provided multiple times to provide multiple directories containing changes> --subject <TorizonCore Builder [timestamp]> --body <OSTree commit body message>
without Docker volume

If any changes are made by isolate, splash and dt commands within working directory, same directories are needed to be provided with --changes-directory option. Each change can be provided with single option, so use it multiple times to provide multiple directories. If user has made any other changes, it can be provided with --extra-changes-directory option. --extra-changes-directory is also needed to be used multiple times to provide multiple directories.

$ torizoncore-builder --storage-directory <Path to store intermediate data. Must be a file system capable of carrying Linux file system metadata (Unix file permissions and xattr)> union --union-branch <name to be given to the branch> --changes-directory <path to directory containing configuration changes>  --changes-directory <optionally can be provided multiple times to provide multiple directories containing changes>  --extra-changes-directory <additional directories containing changes> --extra-changes-directory <optionally can be provided multiple times to provide multiple directories containing changes> --subject <TorizonCore Builder [timestamp]> --body <OSTree commit body message>

Deploy

It takes a branch from union command, sets it up for the boot either directly on target machine connected via ssh or packs the rootfs and creates the Toradex Easy Installer image of Torizon ready to be deployed (if no pre-install container(s) are needed).

If pre-installed container(s) are needed, combine command is to be used to create a new Toradex Easy Installer image of Torizon with bundled container(s) from bundle command.

Arguments

--output-directory: directory to store Toradex Easy Installer image of Torizon with deployment set to branch from union command. Output directory should not be created by the user. --remote-host: remote target machine IP or hostname (.local) --remote-username: username of remote target machine --remote-password: user password of remote target machine --reboot: reboot remote target machine after deploying default: do not reboot after deployment --deploy-sysroot-directory (optional): directory to store the intermittent deployment sysroot (used internally by tool). NOTE: OSTree needs to be able to write extended attributes in this directory. [better to be used for without Docker volume option] default: docker volume mounted at /deploy

Positional Argument

branch name/checksum to be deployed from union command
default: active deployment in OSTree repo

Example
with Docker volume

To create output image

$ torizoncore-builder deploy --output-directory <directory to store Toradex Easy Installer image of Torizon with deployment set to provided reference> <name or checksum of branch within unpacked OSTree repo (output from union command or any other branch)> 

To set branch for deployment on remote target machine

$ torizoncore-builder deploy <name or checksum of branch within unpacked OSTree repo (output from union command or any other branch)> --remote-host <IP or hostname(.local) or remote target machine> --remote-username <user name of remote target machine> --remote-password <user password of remote target machine> --reboot <indicates reboot machine after deploying>
without Docker volumes

To create output image

$ torizoncore-builder --storage-directory <Path to store intermediate data. Must be a file system capable of carrying Linux file system metadata (Unix file permissions and xattr)> deploy <name or checksum of branch within unpacked OSTree repo (output from union command or any other branch)> --output-directory <directory to store Toradex Easy Installer image of Torizon image with deployment set to provided reference> --deploy-sysroot-directory <directory to store the intermittent deployment sysroot>

To set branch for deployment on remote target machine

$ torizoncore-builder deploy <name or checksum of branch within unpacked OSTree repo (output from union command or any other branch)> --remote-host <IP or hostname(.local) or remote target machine> --remote-username <user name of remote target machine> --remote-password <password of remote target machine> --reboot <indicates reboot machine after deploying>

Bundle

It creates a tar archive of bundle container(s) from provided docker-compose file and stores it in ether "bundle" named directory inside working directory or with user provided name and path using

--bundle-directory

option of

torizonbuilder

. Docker-compose file and tar archive is later combined with Toradex Easy Installer image of Torizon using combine command to pre-install containers in Toradex Easy Installer image of Torizon. Toradex EasyInstaller tool copies containers data to appropriate places in rootfs during flashing the image on Target board.

Arguments

--compose-file: path to docker-compose file
default: docker-compose.yml in working directory

--platform: specify platform to make sure fetching the correct container image when multi-platform container images are specified (e.g. linux/arm/v7 or linux/arm64).
default: linux/arm/v7

--host-workdir: directory to save some internal information for DIND(Docker in Docker) and output container tar archive.

Examples
$ torizoncore-builder --bundle-directory <path to store docker bundle tar archive> --compose-file <pat to docker compose file> --platform <platform of container> --host-workdir <path to directory to store continaer bundle>

Combine

It is used to create a Toradex Easy Installer image of Torizon by combining tar archive with user provided Toradex Easy Installer image of Torizon. User provided Toradex Easy Installer image of Torizon can be either image created by deploy command or base Toradex Easy Installer image of Torizon.

Arguments

--image-directory: path to Toradex Easy Installer image of Torizon to be combined with docker bundle

--output-directory: path to combined Toradex Easy Installer image of Torizon with bundled containers

--image-name (optional) : image name to be used in Easy Installer image json

--image-description (optional): image description to be used in Easy Installer image json

--image-licence (optional): path to license file which will be shown on image installation

--image-release-notes (optional): path to release notes file which will be shown on image installation

Example
$ torizoncore-builder --bundle-directory <path to docker bundle tar archive> --image-directory <path to Tezi image to be combined with docker bundle> --output-directory <directory to store combined Tezi image> --image-name <name of image to be set in image json file> --image-description <image desription to be set in image json file> --image-licence <path to license file to be shown at time of installation of image> --image-release-notes <path to release notes file which will be shown on image installation>

Torizon 4.0.0

Introduction

Toradex provides Easy Installer to flash images on modules. Developer may make changes in configuration (/etc) of the system and/or work with containers for the end user solution. Now how to replicate the same on other modules? Toradex has developed a new tool, TorizonCore Builder, to let the developer pick those modifications/containers and create another image out of it, keeping the original Toradex Easy Installer image of Torizon as base. TorizonCore Builder is containerized and run on the host system.

This article complies to the Typographic Conventions for Torizon Documentation

Workflow


  • TorizonCore Builder Workflow

    TorizonCore Builder Workflow

Commands

TorizonCore Builder is provided in the form of Docker image and has following commands to create Toradex Easy Installer image of Torizon with required changes

  • Unpack

  • Bundle

  • Isolate

  • Union

  • Deploy

  • Combine

Operation

Firstly, user needs a base Toradex Easy Installer image of Torizon which will be used by TorizonCore Builder container to add user changes in it. User has two options, either to perform all tasks in a “working directory”, which must be a file system capable of carrying Linux file system metadata (such as Unix style file system permissions and extended attributes), by setting appropriate arguments OR let the commands use Docker volumes for intermediate storage to perform the operations on image.

  • The very first command to use is unpack, which unpacks the OSTree file system from user provided Toradex Easy Installer image of Torizon

  • If pre-installation of Docker container(s) onto a Torizon image with Docker engine is required, bundle command is to be used by providing a Docker compose file stating all image(s).

  • If user has made any changes in configuration (/etc), isolate command is to be used to get all those changes (modifications, addition and deletion). It requires ssh connection to the target board.

  • If user has used isolate command, union command is to be used to capture all changes in the form of a commit.

  • If user has used union command, deploy command is to be used to deploy the OSTree commit onto the OSTree file system and create a new Toradex Easy Installer image of Torizon.

  • Combine is used to create a Toradex Easy Installer image of Torizon by combining outputs from deploy command and bundle command i.e. combine user configuration changes and pre-provisioned Docker container(s).

Usage

Warning: Make sure to use an empty working directory and navigate to it. This will be bind mounted to the container this tool is running in and will be used to store the final image and may also use to store temporary files.

Note: Toradex Easy Installer image of Torizon can be downloaded from Toradex Easy Installer

Extract the downloaded image tar file

$ mkdir <directory to extract image into>
$ tar -xf <downloaded image.tar> -C <directory to extract image into>

Basic

For convenience, create alias first

$ alias torizoncore-builder='docker run --rm -it -v $(pwd):/workdir -v storage:/storage -v /deploy --net=host -v /var/run/docker.sock:/var/run/docker.sock torizon/torizoncore-builder:1'

TorizonCore Builder tool uses Docker volumes as persistent storage for intermediate outputs of sub-commands.

TorizonCore Builder tool provides different logging levels, which can be set by using --log-level option. There are five logging levels i.e. debug, info, warning, error and critical and default is set to info. Formatted logging can be enabled by using --verbose option.

Sub-commands of TorizonCore Builder tool can be used as follows

unpack

unpack gets the base Toradex Easy Installer image of Torizon and unpacks the rootfs (OSTree filesystem) to Docker volume

$ torizoncore-builder unpack --image-directory <base Toradex Easy Installer image of Torizon>

Isolate

isolate is used to get configuration changes(modifications/addition/deletion) from the Target board. It creates ssh session with Target board to get changes and store them within Docker volume on host.

$ torizoncore-builder isolate --remote-ip <ip of target board> --remote-username <username on target board> --remote-password <password of user on target board> 

Union

union is used to make an OStree commit of all changes in OSTree repo of unpacked base Torizon image.

$ torizoncore-builder union --union-branch <name to be given to the branch containing the changes commit>

Deploy

deploy is used to take branch from union command, make it default deployment, create the new Torizon image, make directory (inside bind mounted working directory ) with user provided name and store Torizon Image in it.

Note: Output directory is not needed to be created.

$ torizoncore-builder deploy --output-directory <name of directory to store Torizon image>

Bundle

bundle is used to bundle the Docker container(s), which is used by combine command to pre-provision container(s) bundle with Torizon Image.

User provides the docker-compose.yml, platform of target board (linux/arm/v7 or linux/arm64) and working directory of host to store data used for DIND (docker in docker) and bundle command stores the Docker container(s) tar archive in directory named “bundle” inside the working directory.

$ torizoncore-builder bundle --file <path to docker-compose.yml> --platform <target board like linux/arm/v7 or linux/arm64> --host-workdir $(pwd)

Combine

combine command is used to create Torizon image by combining user provided Torizon image (either base Torizon Image OR output of deploy;depending on what is needed) with bundled Docker container(s) from bundle command.

$ torizoncore-builder combine --image-directory <Path to Torizon image> --output-directory <Torizon image with pre-provisioned Docker container(s)>

Example

Creating alias

$ alias torizoncore-builder='docker run -it --rm -v $(pwd):/workdir -v storage:/storage -v /deploy --net=host -v /var/run/docker.sock:/var/run/docker.sock torizon/torizoncore-builder:1'

Unpack base Toradex Easy Installer image of Torizon

$ torizoncore-builder unpack --image-directory colibri-imx7
Copying Toradex Easy Installer image.
Unpacking TorizonCore Toradex Easy Installer image.
Unpacked OSTree from oradex Easy Installer image:
Commit ref: 3c5795a0dcd0ceca552c916f6515a84011425443fd14d118caa09b869dbe4804
TorizonCore Version: 4.0.0-devel-20200607+build.117

Get configuration changes from target board

$ torizoncore-builder isolate --remote-ip 192.168.1.146 --remote-username torizon --remote-password 1234 
isolation command completed

Merge changes to base Toradex Easy Installer image of Torizon

$ torizoncore-builder union --union-branch=changes_committed
Commit d77ed9d5ddec41b7b61f9a571e810de2447133391ee19fe5f34d1a81009967d1 has been generated for changes and ready to be deployed. 

Set branch to deploy at boot. Output image can be deployed on target board. Note: deployed_image is not needed to be created

$ torizoncore-builder deploy --output-directory deployed_image --ref=d77ed9d5ddec41b7b61f9a571e810de2447133391ee19fe5f34d1a81009967d1
Running tar command: tar --xattrs --xattrs-include='*' -cf /workdir/deployed_image/torizon-core-docker-colibri-imx7.ota.tar.xz --xz -S -C /deploy -p .
Using unpacked Toradex Easy Installer image as base:
  Commit ref: 3c5795a0dcd0ceca552c916f6515a84011425443fd14d118caa09b869dbe4804
  TorizonCore Version: 4.0.0-devel-20200607+build.117
  Kernel arguments: quiet logo.nologo vt.global_cursor_default=0 plymouth.ignore-serial-consoles splash ostree=/ostree/boot.1/torizon/d103ac800d2c9a67cce7ec68ad1accb836d766ff959895820deea195f16eaeed/0

Deploying commit ref: d77ed9d5ddec41b7b61f9a571e810de2447133391ee19fe5f34d1a81009967d1
Pulling OSTree with ref d77ed9d5ddec41b7b61f9a571e810de2447133391ee19fe5f34d1a81009967d1 from local repository...
Pulling done.
Deploying OSTree with ref d77ed9d5ddec41b7b61f9a571e810de2447133391ee19fe5f34d1a81009967d1
Deploying done.
Copy rootdirs such as /home from original deployment.
Packing rootfs...
Packing rootfs done.

If required, create tar archive of Docker container(s)

$ torizoncore-builder --bundle-directory bundled_containers  bundle -f docker-compose.yml --platform linux/arm/v7 --host-workdir $(pwd)

Create image with pre-provisioned docker containers. --image-directory can be either base image or output from deploy command.

$ mkdir preprov_cont_image
$ torizoncore-builder --bundle-directory bundled_containers combine --image-directory deployed_image --output-directory preprov_cont_image
Creating copy of TorizonCore source image.
Combining TorizonCore image with Docker Container bundle.
Successfully created a TorizonCore image with Docker Containers preprovisioned in preprov_cont_image

Advanced

User can use TorizonCore Builder tool by either providing the working directory (capable of carrying Linux filesystem metadata) or Docker volumes for intermediate data storage. So all commands are explained with both approaches. Commands can be used with mix of Docker volumes and working directory but that will complicate the usage. So it is recommended to use either working directory or Docker volumes for all the commands.

As it is provided in the form of Docker image, any of the following commands (based on working directory or Docker volumes) can be executed to create an alias to be easy to use.

Usage of all commands explained later are based on the following aliases.

With Docker volumes:

$ alias torizoncore-builder='docker run --rm -it -v $(pwd):/workdir -v storage:/storage -v /deploy --net=host -v /var/run/docker.sock:/var/run/docker.sock torizon/torizoncore-builder:1'

Here, two volumes are being created and mounted at /storage and /deploy of container for internal usage and host current directory is bind mounted at /workdir within container.

With user provided working directory (capable of handling Linux filesystem metadata)

$ alias torizoncore-builder='docker run --rm -it -v $(pwd):/workdir --net=host -v /var/run/docker.sock:/var/run/docker.sock torizon/torizoncore-builder:1'

move to the working directory so the required one gets bind mounted when alias is executed

Base Command

Base command is now alias torizoncore-builder and has below optional options

--verbose: enable formatted logs

--log-level: set log level from options debug, info, warning, error and critical
default: info

--log-file: write logs to specified file instead of console

--bundle-directory: path to directory to store output of bundle command
default: creates and use directory with name bundle in working directory

--storage-directory: path to directory to use as intermediate storage for sub-commands.
default: Docker volume mounted at/storage
Note: do not set it if Docker volumes are to be used.

Unpack

Unpack command is used to unpack the ostree filesystem from user provided Toradex Easy Installer image of Torizon base image.

Arguments

--image-directory : path to base Toradex Easy Installer image of Torizon in the working directory

--sysroot-directory (optional): path to where ostree filesystem from Toradex Easy Installer image of Torizon will be unpacked
default: sysroot directory inside either user provided storage_directory using --storage-directory option OR docker volume (mounted at /storage)

Example
With docker volume
$ torizoncore-builder unpack --image-directory <path to base Toradex Easy Installer image of Torizon in the working directory>  
Without Docker Volume

Additional argument --sysroot-directory, specifying path to store unpacked ostree filesystem, is needed in case of user provided working directory (capable of handling Linux filesystem metadata) is to be used for storage of intermediate data.

$ torizoncore-builder unpack --image-directory <path to base Toradex Easy Installer image of Torizon in working directory> --sysroot-directory <path to where ostree filesystem from Toradex Easy Installer image of Torizon will be unpacked>

If there is already an unpacked ostree filesystem, user is prompted to delete it because currently a single ostree filesystem can be handled.

Isolate

Isolate command is used to get the configuration (/etc) changes (modifications, additions and deletions) from user target module. It ignores following files/directories for any change

'gshadow', 'machine-id', 'group', 'shadow', 'systemd/system/sysinit.target.wants/run-postinsts.service', 'ostree/remotes.d/toradex-nightly.conf', 'docker/key.json', '.updated', '.pwd.lock', 'group-', 'gshadow-', 'hostname', 'ssh/ssh_host_rsa_key', 'ssh/ssh_host_rsa_key.pub', 'ssh/ssh_host_ecdsa_key', 'ssh/ssh_host_ecdsa_key.pub','ssh/ssh_host_ed25519_key','ssh/ssh_host_ed25519_key.pub', 'ipk-postinsts', 'fw_env.conf'

Isolate command establishes a ssh session with the target board, gets all the changes and places them in either user provided directory or within docker volumes.

Arguments

--remote-ip: IP of target board

--remote-username: Username of target board to be used for creating ssh session

--remote-password: Password of target board to be used for creating ssh session

--diff-dir (optional): If docker volume is not to be used, directory for storing configuration changes
default: changes directory inside either user provided storage_directory using --storage-directory option OR docker volume (mounted at /storage)

Example
with docker volume
$torizoncore-builder  isolate --remote-ip <target board ip> --remote-username <target board username> --remote-password <taregtboard password>
without docker volume

Directory path is needed (--diff-dir) to store changes within user provided working directory

$ torizoncore-builder  isolate --remote-ip <target board ip> --remote-username <target board username> --remote-password <taregtboard password> --diff-dir <directory to store changes>

Union

It is to be run after isolate command to commit changes to the ostree filesystem. If changes are stored in docker volume, it is to be run with docker volume and if changes are stored in working directory, same directory is needed to be provided.

Arguments

----union-branch: branch name containing the changes committed to ostree repo of unpacked ostree filesystem

--diff-directory(optional): path (within working directory) to configuration changes picked by isolate command.
default: changes directory inside either user provided storage_directory using --storage-directory option OR docker volume (mounted at /storage)

--sysroot-directory(optional): path to unpacked ostree filesystem from Toradex Easy Installer image of Torizon (same as used in unpack command)
default: sysroot directory inside either user provided storage_directory using --storage-directory option OR docker volume (mounted at /storage)

Example
with docker volume
$ torizoncore-builder union --union-branch <name to be given to the branch>
without docker volume

isolate command is used to store changes in a directory within working directory. Same directory is needed here with --diff-directory option.

$ torizoncore-builder union --union-branch <name to be given to the branch> --diff-directory <path to directory containing configuration changes> --sysroot-directory <path to unpacked ostree filesystem from Toradex Easy Installer image of Torizon>  

Deploy

It takes a branch from union command, sets it up for the boot, packs the rootfs and creates the Toradex Easy Installer image of Torizon ready to be deployed (if no pre-install container(s) are needed).

If pre-installed container(s) are needed, “combine command“ is used to create a new Toradex Easy Installer image of Torizon with bundled container(s) from bundle command.

Arguments

--output-directory: directory to store Toradex Easy Installer image of Torizon with deployment set to branch from union command. It is not needed to be created by the user.

--diff-ref: branch name/checksum to be deployed from union command
default: active deployment in ostree repo

--sysroot-directory (optional): path to unpacked ostree filesystem from Toradex Easy Installer image of Torizon
default: sysroot directory inside either user provided storage_directory using --storage-directory option OR docker volume (mounted at /storage)

--deploy-sysroot-directory (optional): directory to store the intermittent deployment sysroot (used internally by tool).
default: docker volume mounted at /deploy

Example
with docker volume
$ torizoncore-builder deploy --output-directory <directory to store Toradex Easy Installer image of Torizon with deployment set to provided reference> --diff-ref <name or checksum of branch within unpacked ostree repo> 
without docker volumes
$ torizoncore-builder deploy --output-directory <directory to store Toradex Easy Installer image of Torizonm image with deployment set to provided reference> --diff-ref <name or checksum of branch within unpacked ostree repo> --sysroot-directory <path to unpacked ostree filesystem from Toradex Easy Installer image of Torizon> --deploy-sysroot-directory <directory to store the intermittent deployment sysroot>

Bundle

It creates a tar archive of bundle container(s) from provided docker-compose file and stores it in ether "bundle" named directory inside working directory or with user provided name and path using

--bundle-directory

option of

torizonbuilder

. Docker-compose file and tar archive is later combined with Toradex Easy Installer image of Torizon using combine command to pre-install containers in Toradex Easy Installer image of Torizon. Toradex EasyInstaller tool copies containers data to appropriate places in rootfs during flashing the image on Target board.

Arguments

--compose-file: path to docker-compose file
default: docker-compose.yml in working directory

--platform: specify platform to make sure fetching the correct container image when multi-platform container images are specified (e.g. linux/arm/v7 or linux/arm64).
default: linux/arm/v7

--host-workdir: directory to save some internal information for DIND(Docker in Docker) and output container tar archive.

Examples
$ torizoncore-builder --bundle-directory <path to store docker bundle tar archive> --compose-file <pat to docker compose file> --platform <platform of container> --host-workdir <path to directory to store continaer bundle>

Combine

It is used to create a Toradex Easy Installer image of Torizon by combining tar archive with user provided Toradex Easy Installer image of Torizon. User provided Toradex Easy Installer image of Torizon can be either image created by deploy command or base Toradex Easy Installer image of Torizon as per need.

Arguments

--image-directory: path to Toradex Easy Installer image of Torizon to be combined with docker bundle

--output-directory: path to combined Toradex Easy Installer image of Torizon with bundled containers

--image-name (optional) : image name to be used in Easy Installer image json

--image-description (optional): image description to be used in Easy Installer image json

--image-licence (optional): path to license file which will be shown on image installation

--image-release-notes (optional): path to release notes file which will be shown on image installation

Example
$ torizoncore-builder --bundle-directory <path to docker bundle tar archive> --image-directory <path to Tezi image to be combined with docker bundle> --output-directory <directory to store combined Tezi image> --image-name <name of image to be set in image json file> --image-description <image desription to be set in image json file> --image-licence <path to license file to be shown at time of installation of image> --image-release-notes <path to release notes file which will be shown on image installation>