Toradex provides Easy Installer to flash images on modules. Developer may make changes in configuration (/etc) of the system and/or work with containers for the end user solution. Now how to replicate the same on other modules? Toradex has developed a new tool, TorizonCore Builder, to let the developer pick those modifications/containers and create another image out of it, keeping the original Toradex Easy Installer image of Torizon as base. TorizonCore Builder is containerized and run on the host system.
This article complies to the Typographic Conventions for Torizon Documentation
TorizonCore Builder Workflow
TorizonCore Builder is provided in the form of Docker image and has following commands to create Toradex Easy Installer image of Torizon with required changes
Unpack
Bundle
Isolate
Union
Deploy
Combine
Firstly, user needs a base Toradex Easy Installer image of Torizon which will be used by TorizonCore Builder container to add user changes in it. User has two options, either to perform all tasks in a “working directory”, which must be a file system capable of carrying Linux file system metadata (such as Unix style file system permissions and extended attributes), by setting appropriate arguments OR let the commands use Docker volumes for intermediate storage to perform the operations on image.
The very first command to use is unpack, which unpacks the OSTree file system from user provided Toradex Easy Installer image of Torizon
If pre-installation of Docker container(s) onto a Torizon image with Docker engine is required, bundle command is to be used by providing a Docker compose file stating all image(s).
If user has made any changes in configuration (/etc), isolate command is to be used to get all those changes (modifications, addition and deletion). It requires ssh connection to the target board.
If user has used isolate command, union command is to be used to capture all changes in the form of a commit.
If user has used union command, deploy command is to be used to deploy the OSTree commit onto the OSTree file system and create a new Toradex Easy Installer image of Torizon.
Combine is used to create a Toradex Easy Installer image of Torizon by combining outputs from deploy command and bundle command i.e. combine user configuration changes and pre-provisioned Docker container(s).
Warning: Make sure to use an empty working directory and navigate to it. This will be bind mounted to the container this tool is running in and will be used to store the final image and may also use to store temporary files.
Note: Toradex Easy Installer image of Torizon can be downloaded from Toradex Easy Installer
Extract the downloaded image tar file
$ mkdir <directory to extract image into>
$ tar -xf <downloaded image.tar> -C <directory to extract image into>
For convenience, create alias first
$ alias torizoncore-builder='docker run --rm -it -v $(pwd):/workdir -v storage:/storage -v /deploy --net=host -v /var/run/docker.sock:/var/run/docker.sock torizon/torizoncore-builder'
TorizonCore Builder tool uses Docker volumes as persistent storage for intermediate outputs of sub-commands.
TorizonCore Builder tool provides different logging levels, which can be set by using --log-level option. There are five logging levels i.e. debug, info, warning, error and critical and default is set to info. Formatted logging can be enabled by using --verbose option.
Sub-commands of TorizonCore Builder tool can be used as follows
unpack gets the base Toradex Easy Installer image of Torizon and unpacks the rootfs (OSTree filesystem) to Docker volume
$ torizoncore-builder unpack --image-directory <base Toradex Easy Installer image of Torizon>
isolate is used to get configuration changes(modifications/addition/deletion) from the Target board. It creates ssh session with Target board to get changes and store them within Docker volume on host.
$ torizoncore-builder isolate --remote-ip <ip of target board> --remote-username <username on target board> --remote-password <password of user on target board>
union is used to make an OStree commit of all changes in OSTree repo of unpacked base Torizon image.
$ torizoncore-builder union --union-branch <name to be given to the branch containing the changes commit>
deploy is used to take branch from union command, make it default deployment, create the new Torizon image, make directory (inside bind mounted working directory ) with user provided name and store Torizon Image in it.
Note: Output directory is not needed to be created.
$ torizoncore-builder deploy --output-directory <name of directory to store Torizon image>
bundle is used to bundle the Docker container(s), which is used by combine command to pre-provision container(s) bundle with Torizon Image.
User provides the docker-compose.yml, platform of target board (linux/arm/v7 or linux/arm64) and working directory of host to store data used for DIND (docker in docker) and bundle command stores the Docker container(s) tar archive in directory named “bundle” inside the working directory.
$ torizoncore-builder bundle --file <path to docker-compose.yml> --platform <target board like linux/arm/v7 or linux/arm64> --host-workdir $(pwd)
combine command is used to create Torizon image by combining user provided Torizon image (either base Torizon Image OR output of deploy;depending on what is needed) with bundled Docker container(s) from bundle command.
$ torizoncore-builder --image-directory <Path to Torizon image> --output-directory <Torizon image with pre-provisioned Docker container(s)>
Creating alias
$ alias torizoncore-builder='docker run -it --rm -v $(pwd):/workdir -v storage:/storage -v /deploy --net=host -v /var/run/docker.sock:/var/run/docker.sock torizon/torizoncore-builder'
Unpack base Toradex Easy Installer image of Torizon
$ torizoncore-builder unpack --image-directory colibri-imx7
Copying Toradex Easy Installer image.
Unpacking TorizonCore Toradex Easy Installer image.
Unpacked OSTree from oradex Easy Installer image:
Commit ref: 3c5795a0dcd0ceca552c916f6515a84011425443fd14d118caa09b869dbe4804
TorizonCore Version: 4.0.0-devel-20200607+build.117
Get configuration changes from target board
$ torizoncore-builder isolate --remote-ip 192.168.1.146 --remote-username torizon --remote-password 1234
isolation command completed
Merge changes to base Toradex Easy Installer image of Torizon
$ torizoncore-builder union --union-branch=changes_committed
Commit d77ed9d5ddec41b7b61f9a571e810de2447133391ee19fe5f34d1a81009967d1 has been generated for changes and ready to be deployed.
Set branch to deploy at boot. Output image can be deployed on target board.
Note: deployed_image is not needed to be created
$ torizoncore-builder deploy --output-directory deployed_image --ref=d77ed9d5ddec41b7b61f9a571e810de2447133391ee19fe5f34d1a81009967d1
Running tar command: tar --xattrs --xattrs-include='*' -cf /workdir/deployed_image/torizon-core-docker-colibri-imx7.ota.tar.xz --xz -S -C /deploy -p .
Using unpacked Toradex Easy Installer image as base:
Commit ref: 3c5795a0dcd0ceca552c916f6515a84011425443fd14d118caa09b869dbe4804
TorizonCore Version: 4.0.0-devel-20200607+build.117
Kernel arguments: quiet logo.nologo vt.global_cursor_default=0 plymouth.ignore-serial-consoles splash ostree=/ostree/boot.1/torizon/d103ac800d2c9a67cce7ec68ad1accb836d766ff959895820deea195f16eaeed/0
Deploying commit ref: d77ed9d5ddec41b7b61f9a571e810de2447133391ee19fe5f34d1a81009967d1
Pulling OSTree with ref d77ed9d5ddec41b7b61f9a571e810de2447133391ee19fe5f34d1a81009967d1 from local repository...
Pulling done.
Deploying OSTree with ref d77ed9d5ddec41b7b61f9a571e810de2447133391ee19fe5f34d1a81009967d1
Deploying done.
Copy rootdirs such as /home from original deployment.
Packing rootfs...
Packing rootfs done.
If required, create tar archive of Docker container(s)
$ torizoncore-builder --bundle-directory bundled_containers bundle -f docker-compose.yml --platform linux/arm/v7 --host-workdir $(pwd)
Create image with pre-provisioned docker containers. --image-directory can be either base image or output from deploy command.
$ mkdir preprov_cont_image
$ torizoncore-builder --bundle-directory bundled_containers combine --image-directory deployed_image --output-directory preprov_cont_image
Creating copy of TorizonCore source image.
Combining TorizonCore image with Docker Container bundle.
Successfully created a TorizonCore image with Docker Containers preprovisioned in preprov_cont_image
User can use TorizonCore Builder tool by either providing the working directory (capable of carrying Linux filesystem metadata) or Docker volumes for intermediate data storage. So all commands are explained with both approaches. Commands can be used with mix of Docker volumes and working directory but that will complicate the usage. So it is recommended to use either working directory or Docker volumes for all the commands.
As it is provided in the form of Docker image, any of the following commands (based on working directory or Docker volumes) can be executed to create an alias to be easy to use.
Usage of all commands explained later are based on the following aliases.
With Docker volumes:
$ alias torizoncore-builder='docker run --rm -it -v $(pwd):/workdir -v storage:/storage -v /deploy --net=host -v /var/run/docker.sock:/var/run/docker.sock torizon/torizoncore-builder'
Here, two volumes are being created and mounted at /storage and /deploy of container for internal usage and host current directory is bind mounted at /workdir within container.
With user provided working directory (capable of handling Linux filesystem metadata)
$ alias torizoncore-builder='docker run --rm -it -v $(pwd):/workdir --net=host -v /var/run/docker.sock:/var/run/docker.sock torizon/torizoncore-builder'
move to the working directory so the required one gets bind mounted when alias is executed
Base command is now alias torizoncore-builder and has below optional options
--verbose: enable formatted logs
--log-level: set log level from options debug, info, warning, error and critical
default: info
--log-file: write logs to specified file instead of console
--bundle-directory: path to directory to store output of bundle command
default: creates and use directory with name bundle in working directory
--storage-directory: path to directory to use as intermediate storage for sub-commands.
default: Docker volume mounted at/storage
Note: do not set it if Docker volumes are to be used.
Unpack command is used to unpack the ostree filesystem from user provided Toradex Easy Installer image of Torizon base image.
--image-directory : path to base Toradex Easy Installer image of Torizon in the working directory
--sysroot-directory (optional): path to where ostree filesystem from Toradex Easy Installer image of Torizon will be unpacked
default: sysroot directory inside either user provided storage_directory using --storage-directory option OR docker volume (mounted at /storage)
$ torizoncore-builder unpack --image-directory <path to base Toradex Easy Installer image of Torizon in the working directory>
Additional argument --sysroot-directory, specifying path to store unpacked ostree filesystem, is needed in case of user provided working directory (capable of handling Linux filesystem metadata) is to be used for storage of intermediate data.
$ torizoncore-builder unpack --image-directory <path to base Toradex Easy Installer image of Torizon in working directory> --sysroot-directory <path to where ostree filesystem from Toradex Easy Installer image of Torizon will be unpacked>
If there is already an unpacked ostree filesystem, user is prompted to delete it because currently a single ostree filesystem can be handled.
Isolate command is used to get the configuration (/etc) changes (modifications, additions and deletions) from user target module. It ignores following files/directories for any change
'gshadow', 'machine-id', 'group', 'shadow', 'systemd/system/sysinit.target.wants/run-postinsts.service', 'ostree/remotes.d/toradex-nightly.conf', 'docker/key.json', '.updated', '.pwd.lock', 'group-', 'gshadow-', 'hostname', 'ssh/ssh_host_rsa_key', 'ssh/ssh_host_rsa_key.pub', 'ssh/ssh_host_ecdsa_key', 'ssh/ssh_host_ecdsa_key.pub','ssh/ssh_host_ed25519_key','ssh/ssh_host_ed25519_key.pub', 'ipk-postinsts', 'fw_env.conf'
Isolate command establishes a ssh session with the target board, gets all the changes and places them in either user provided directory or within docker volumes.
--remote-ip: IP of target board
--remote-username: Username of target board to be used for creating ssh session
--remote-password: Password of target board to be used for creating ssh session
--diff-dir (optional): If docker volume is not to be used, directory for storing configuration changes
default: changes directory inside either user provided storage_directory using --storage-directory option OR docker volume (mounted at /storage)
$torizoncore-builder isolate --remote-ip <target board ip> --remote-username <target board username> --remote-password <taregtboard password>
Directory path is needed (--diff-dir) to store changes within user provided working directory
$ torizoncore-builder isolate --remote-ip <target board ip> --remote-username <target board username> --remote-password <taregtboard password> --diff-dir <directory to store changes>
It is to be run after isolate command to commit changes to the ostree filesystem. If changes are stored in docker volume, it is to be run with docker volume and if changes are stored in working directory, same directory is needed to be provided.
----union-branch: branch name containing the changes committed to ostree repo of unpacked ostree filesystem
--diff-directory(optional): path (within working directory) to configuration changes picked by isolate command.
default: changes directory inside either user provided storage_directory using --storage-directory option OR docker volume (mounted at /storage)
--sysroot-directory(optional): path to unpacked ostree filesystem from Toradex Easy Installer image of Torizon (same as used in unpack command)
default: sysroot directory inside either user provided storage_directory using --storage-directory option OR docker volume (mounted at /storage)
$ torizoncore-builder union --union-branch <name to be given to the branch>
isolate command is used to store changes in a directory within working directory. Same directory is needed here with --diff-directory option.
$ torizoncore-builder union --union-branch <name to be given to the branch> --diff-directory <path to directory containing configuration changes> --sysroot-directory <path to unpacked ostree filesystem from Toradex Easy Installer image of Torizon>
It takes a branch from union command, sets it up for the boot, packs the rootfs and creates the Toradex Easy Installer image of Torizon ready to be deployed (if no pre-install container(s) are needed).
If pre-installed container(s) are needed, “combine command“ is used to create a new Toradex Easy Installer image of Torizon with bundled container(s) from bundle command.
--output-directory: directory to store Toradex Easy Installer image of Torizon with deployment set to branch from union command. It is not needed to be created by the user.
--diff-ref: branch name/checksum to be deployed from union command
default: active deployment in ostree repo
--sysroot-directory (optional): path to unpacked ostree filesystem from Toradex Easy Installer image of Torizon
default: sysroot directory inside either user provided storage_directory using --storage-directory option OR docker volume (mounted at /storage)
--deploy-sysroot-directory (optional): directory to store the intermittent deployment sysroot (used internally by tool).
default: docker volume mounted at /deploy
$ torizoncore-builder deploy --output-directory <directory to store Toradex Easy Installer image of Torizon with deployment set to provided reference> --diff-ref <name or checksum of branch within unpacked ostree repo>
$ torizoncore-builder deploy --output-directory <directory to store Toradex Easy Installer image of Torizonm image with deployment set to provided reference> --diff-ref <name or checksum of branch within unpacked ostree repo> --sysroot-directory <path to unpacked ostree filesystem from Toradex Easy Installer image of Torizon> --deploy-sysroot-directory <directory to store the intermittent deployment sysroot>
It creates a tar archive of bundle container(s) from provided docker-compose file and stores it in ether "bundle" named directory inside working directory or with user provided name and path using
--bundle-directory
option of
torizonbuilder
. Docker-compose file and tar archive is later combined with Toradex Easy Installer image of Torizon using combine command to pre-install containers in Toradex Easy Installer image of Torizon. Toradex EasyInstaller tool copies containers data to appropriate places in rootfs during flashing the image on Target board.
--compose-file: path to docker-compose file
default: docker-compose.yml in working directory
--platform: specify platform to make sure fetching the correct container image when multi-platform container images are specified (e.g. linux/arm/v7 or linux/arm64).
default: linux/arm/v7
--host-workdir: directory to save some internal information for DIND(Docker in Docker) and output container tar archive.
$ torizoncore-builder --bundle-directory <path to store docker bundle tar archive> --compose-file <pat to docker compose file> --platform <platform of container> --host-workdir <path to directory to store continaer bundle>
It is used to create a Toradex Easy Installer image of Torizon by combining tar archive with user provided Toradex Easy Installer image of Torizon. User provided Toradex Easy Installer image of Torizon can be either image created by deploy command or base Toradex Easy Installer image of Torizon as per need.
--image-directory: path to Toradex Easy Installer image of Torizon to be combined with docker bundle
--output-directory: path to combined Toradex Easy Installer image of Torizon with bundled containers
--image-name (optional) : image name to be used in Easy Installer image json
--image-description (optional): image description to be used in Easy Installer image json
--image-licence (optional): path to license file which will be shown on image installation
--image-release-notes (optional): path to release notes file which will be shown on image installation
$ torizoncore-builder --bundle-directory <path to docker bundle tar archive> --image-directory <path to Tezi image to be combined with docker bundle> --output-directory <directory to store combined Tezi image> --image-name <name of image to be set in image json file> --image-description <image desription to be set in image json file> --image-licence <path to license file to be shown at time of installation of image> --image-release-notes <path to release notes file which will be shown on image installation>
