Search by Tags

TorizonCore Builder Tool - Customizing TorizonCore Images

 

Article updated at 27 Sep 2021
Compare with Revision




Subscribe for this article updates

Select the version of your OS from the tabs below. If you don't know the version you are using, run the command cat /etc/os-release or cat /etc/issue on the board.



Remember that you can always refer to the Torizon Documentation, there you can find a lot of relevant articles that might help you in the application development.

Torizon 5.0.0

Introduction

TorizonCore Builder is a tool that allows you to easily generate a custom image of TorizonCore with device configuration changes, a custom device tree and overlays, pre-provisioned containers, external kernel modules, and a splash screen. In the end, you will have a custom TorizonCore image that you can flash onto the SoM using the Toradex Easy Installer, deploy to the SoM through SSH, or push to Torizon Platform for deploying it remotely. The tool is designed to be used on production programming and on development as well.


  • TorizonCore Builder Workflow

    TorizonCore Builder Workflow

*OS Customization Files are device tree and overlays source codes, external kernel modules source files, a splash screen image, and a directory with the configuration changes captured from the device. They can be also called version-controlled artifacts.

Resources

Browse for TorizonCore Builder resources from the tabs below.

This article complies to the Typographic Conventions for Torizon Documentation.

Requirements

  • A Toradex SoM with Torizon installed

Typical Use Cases

TorizonCore builder tool provides multiple commands to be used in different situations. We will describe below typical TorizonCore Builder use cases and on which section the customization related to these use cases is:

Customizing a TorizonCore Image for Production

TorizonCore Builder provides standalone commands that allow you to easily customize a TorizonCore image. Such changes could be:

Deploying TorizonCore OS Images

Users can deploy the image in three different ways:

To learn how to do it, refer to the Deploying the image section of this page.

Installing TorizonCore Builder

OS Compatibility

TorizonCore Builder executes in a Docker Runtime and it was tested on the following Operating Systems:

  • Linux
  • Windows 10 with Windows Subsystem for Linux (WSL2)

Warning: WSL2 requires Windows 10, updated to version 2004 (also known as the Windows 10 May 2020 Update) or higher.

Note for Users Installing TorizonCore Builder on Windows Machine

TorizonCore Builder support on a Windows desktop requires the "Windows Subsystem for Linux 2" (WSL 2). This is due to TorizonCore Builder relying on a Linux-based filesystem.

Before proceeding with any TorizonCore Builder instructions please set up a WSL 2 distro and Docker on your development host as shown on Configure Build Environment for Torizon Containers. As a reminder, any and all TorizonCore Builder commands should be run in your WSL 2 distro.

Finally, as a quick tip, you'll occasionally need to transfer files to and from your WSL 2 distro and your Windows host. You can actually do this by using the built-in Windows file explorer utility. This and other Windows utilities can actually be invoked from within your WSL 2 Distro. For example by doing the following in your WSL 2 Distro:

$ explorer.exe .

The above command will open the Windows file explorer in your current directory. From here you'll be able to copy/paste and move files between the WSL 2 world and the normal Windows world.

Setup Script

Create and enter a working directory where your customization will be stored:

$ mkdir ~/tcbworkdir
$ cd ~/tcbworkdir

To use TorizonCore Builder, you need to install it on your host machine by creating an alias. We provide a setup script named tcb-env-setup.sh on Toradex Github to facilitate the installation and setup TorizonCore Builder.

For basic usage it can be run as follows with no arguments:

Warning: Make sure to source the script. Otherwise, it will not work as intended and most importantly the torizoncore-builder alias will not be created properly.

$ wget https://raw.githubusercontent.com/toradex/tcb-env-setup/master/tcb-env-setup.sh
$ source tcb-env-setup.sh

For more advanced usage of the setup script please run source tcb-env-setup.sh -h, or see the project README.

Example expected output for the first installation:

TorizonCore Builder is not installed. Pulling the latest version from Docker Hub...
Setting up TorizonCore Builder with version 2.

Pulling TorizonCore Builder...
2: Pulling from torizon/torizoncore-builder
7783ecb52c24: Pull complete
d2c061578333: Pull complete
96046ec03d42: Pull complete
feb49f8f9a32: Pull complete
6f5a136384f4: Pull complete
1b448ab72455: Pull complete
9738893ca531: Pull complete
eb0c196b0855: Pull complete
0ef22c696c8d: Pull complete
58da04a4b089: Pull complete
bc364f234f04: Pull complete
3c10f77095c8: Pull complete
2c4d076b4725: Pull complete
65d31292f35e: Pull complete
f553605a42f8: Pull complete
6824da993f58: Pull complete
86f495c51638: Pull complete
Digest: sha256:b5b94cbc962885752ebe2bef498e9a8bcf2bee4fe89c0c162dc857655284ed84
Status: Downloaded newer image for torizon/torizoncore-builder:2
docker.io/torizon/torizoncore-builder:2
Done!

Setup complete! TorizonCore Builder is now ready to use.
For more information, run 'torizoncore-builder -h' or go to https://developer.toradex.com/knowledge-base/torizoncore-builder-tool

As shown the script will guide you to ensure that the latest version of TorizonCore Builder is set up on your machine if no previous versions were found. If there are previous versions you will be asked if you'd like to use these or look for the latest updates online instead.

Tip: When configuring the latest version in your machine, the setup script will also install a bash completion script for TorizonCore Builder, making it possible to autocomplete commands and parameters by just pressing the TAB key.

Using TorizonCore Builder

This section shows how to use TorizonCore Builder in a simple way, yet covering almost all use cases. It is based on the build command, which allows the full image customization in a single step, using a configuration YAML file. The other commands that are necessary in conjunction with the build command in order to cover most use cases are also explained in this section.

If your use case is not covered here, please refer to the TorizonCore Builder - Commands Manual section for a detailed explanation of all TorizonCore Builder commands.

Workflow

  • Create a configuration file with the desired settings (the build command can even help you with that);
  • Customize the configuration file in order to select the desired input image, include the desired customization, and get the desired outputs. Also, run the auxiliary commands where needed so all local artifacts possibly referenced by the configuration file are available;
  • Run the build command;
  • Deploy the output to the board or to Torizon Platform Services server;

To learn about TorizonCore Builder workflow and the different approaches to use the tool, with explanatory diagrams, please refer to the TorizonCore Builder - Workflow article.

Creating a Configuration File

The easiest and thus recommended way for creating an initial configuration file is by running:

$ torizoncore-builder build --create-template

The previous command will create a file named tcbuild.yaml containing all possible configuration parameters accepted by the command.

The configuration file is in a standard YAML format which should be recognized by most modern programming editors. Any other name can be chosen for the file, but tcbuild.yaml is the name that the build command will expect by default when running in build mode. Most parameters in the generated file will be commented out which can be seen by the # character at the beginning of each line. Lines having the characters >> are just explaining the contents of the file and should be left as-is or removed if desired.

The contents of the file are organized in a tree structure where the leaves are configurable properties. In order to set a certain property, you need to make sure all its parent properties are also uncommented (by removing the first # character).

The configuration file follows the YAML format and syntax. The tool expects it to be made up of three sections, represented by the top-level properties (keys): input, customization, and output as outlined below:

tcbuild.yaml
# Configuration file outline:
input:
  # Input section items (required).
customization:
  # Customization section items.
output:
  # Output section items (required).

Customizing The Configuration File

Minimal Configuration File

A minimal configuration file has to specify at least one input and one output. Below we assume that the input is a previously downloaded Toradex Easy Installer image that is stored in a directory called images relative to the working directory (set by the alias of the tool).

When downloading a Toradex Easy Installer image, be sure to download an image without evaluation containers, because the input image to be customized MUST NOT contain bundled container images. To download the image, go to the Toradex Download Links (Torizon, Linux BSP, WinCE and Partner Demos) article.

tcbuild.yaml
# Sample configuration file:
input:
  easy-installer:
    local: images/torizon-core-docker-colibri-imx6-Tezi_5.3.0+build.7.tar
output:
  easy-installer:
    local: torizon-core-docker-colibri-imx6-Tezi_5.3.0.CUSTOM

The directory structure of such a "project" would look like this:

.
├── images
│   └── torizon-core-docker-colibri-imx6-Tezi_5.3.0+build.7.tar
└── tcbuild.yaml

Fetching Remote Images

The input to the tool can also be specified as a URL to a remote server instead of a directory with a downloaded image, as in the Minimal Configuration section. If the input is passed as a remote image, the image will be downloaded every time the image is built.

In the sample configuration below, we directly download an image from Toradex's artifacts server:

tcbuild.yaml
# Sample configuration input, specifying a remote:
input:
  easy-installer:
    remote: "https://artifacts.toradex.com:443/artifactory/torizoncore-oe-prod-frankfurt/dunfell-5.x.y/release/7/colibri-imx6/torizon-upstream/torizon-core-docker/oedeploy/torizon-core-docker-colibri-imx6-Tezi_5.3.0%2Bbuild.7.tar;sha256sum=49fc6a32ab3fc21a184fb8333c19ad53f22875975219883c1087d929bedf470f"

Notice the presence of the sha256sum parameter which can be used to ensure that the downloaded file has a certain SHA-256 checksum – the tool will stop if the actual file checksum does not match that value. For more information about how a remote location can be specified please refer to the build command Detailed Manual.

Fetching remote images: URL autogenerated

Another way to specify an input is the sample configuration below:

tcbuild.yaml
# Sample configuration input, specifying parameters for URL autogeneration:
input:
  easy-installer:
    toradex-feed:
      version: "5.3.0"
      release: quarterly
      machine: colibri-imx6
      distro: torizon-upstream
      variant: torizon-core-docker
      build-number: "7"
      #build-date: "202107"

The build command then autogenerates the URL shown in the remote parameter of the Fetching Remote Images section. For monthly and nightly releases the parameter build-date is needed.

Splash Screen

To customize the splash screen, insert the configuration below:

tcbuild.yaml
#Sample customization: insert a Splash Screen
customization:
  # >> Splash screen:
  splash-screen: splash.png

The file splash.png should be available before running the build command. A demo image can be downloaded by clicking on the button below:

Example Splash Screen Image

Check the Splash Screen on TorizonCore article for more details.

Device Tree

This customization can be used both with Toradex's device trees and overlays as well as any other compatible device tree and overlay. The files containing the source code of the device trees and overlays should be available before running the build command.

i.MX 8/8X/8MM/8MP Downstream Device Tree Overlays Versions

BSP Version Device Tree Overlays Git Branch Device Tree Overlays Binaries
4 toradex_4.14-2.3.x-imx *.dtbo
5.0.0 / 5.1.0 toradex_5.4-2.1.x-imx *.dtbo
5.2.0+ toradex_5.4-2.3.x-imx *.dtbo

i.MX 7 Downstream Device Tree Overlays Versions

BSP Version Device Tree Overlays Git Branch Device Tree Overlays Binaries
4 toradex_4.14-2.3.x-imx *.dtbo
5.0.0 / 5.1.0 toradex_5.4-2.1.x-imx *.dtbo
5.2.0+ toradex_5.4-2.3.x-imx *.dtbo

iMX7 Upstream Device Tree Overlays Versions

BSP Version Device Tree Overlays Git Branch Device Tree Overlays Binaries
4 / 5 toradex_5.4.y *.dtbo

i.MX 6 Downstream Device Tree Overlays Versions

BSP Version Device Tree Overlays Git Branch Device Tree Overlays Binaries
4 toradex_4.14-2.3.x-imx *.dtbo
5.0.0 / 5.1.0 toradex_5.4-2.1.x-imx *.dtbo
5.2.0+ toradex_5.4-2.3.x-imx *.dtbo

i.MX 6 Upstream Device Tree Overlays Versions

BSP Version Device Tree Overlays Git Branch Device Tree Overlays Binaries
4 / 5 toradex_5.4.y *.dtbo

i.MX 6ULL Downstream Device Tree Overlays Versions

BSP Version Device Tree Overlays Git Branch Device Tree Overlays Binaries
4 toradex_4.14-2.3.x-imx *.dtbo
5.0.0 / 5.1.0 toradex_5.4-2.1.x-imx *.dtbo
5.2.0+ toradex_5.4-2.3.x-imx *.dtbo

i.MX 6ULL Upstream Device Tree Overlays Versions

BSP Version Device Tree Overlays Git Branch Device Tree Overlays Binaries
4 / 5 toradex_5.4.y *.dtbo

Tegra K1 Upstream Device Tree Overlays Versions

BSP Version Device Tree Overlays Git Branch Device Tree Overlays Binaries
5 toradex_5.4.y *.dtbo

You just need to execute this once. To get the source code of the Toradex supplied device tree files (including overlays), one could use the command below, accordingly to the table above:

$ git clone -b <git branch> https://github.com/toradex/device-trees.git

To see the available device trees and select the appropriate one for your device, run the command below, passing the parameter -name accordingly to your device.

$ find device-trees/dts-arm32 -name "*imx6q-apalis*.dts"

device-trees/dts-arm32/imx6q-apalis-ixora-v1.2.dts
device-trees/dts-arm32/imx6q-apalis-ixora.dts
device-trees/dts-arm32/imx6q-apalis-ixora-v1.1.dts
device-trees/dts-arm32/imx6q-apalis-eval.dts
$ find device-trees/dts-arm64 -name "*imx8qp-apalis*.dts" # for Apalis iMX8 QuadPlus versions
$ find device-trees/dts-arm64 -name "*imx8qm-apalis*.dts" # for Apalis iMX8 QuadMax versions

device-trees/dts-arm64/imx8qp-apalis-v1.1-ixora-v1.1.dts
device-trees/dts-arm64/imx8qp-apalis-v1.1-eval.dts
device-trees/dts-arm64/imx8qm-apalis-eval.dts
device-trees/dts-arm64/imx8qm-apalis-ixora-v1.1.dts
device-trees/dts-arm64/imx8qm-apalis-v1.1-eval.dts
device-trees/dts-arm64/imx8qm-apalis-v1.1-ixora-v1.1.dts
$ find device-trees/dts-arm64 -name "*imx8qxp-apalis*.dts"

device-trees/dts-arm64/imx8qxp-apalis-v1.1-eval.dts
$ find device-trees/dts-arm32 -name "*imx6dl-colibri*.dts"

device-trees/dts-arm32/imx6dl-colibri-eval-v3.dts
$ find device-trees/dts-arm32 -name "*imx7d-colibri*.dts"

device-trees/dts-arm32/imx7d-colibri-aster.dts
device-trees/dts-arm32/imx7d-colibri-emmc-eval-v3.dts
device-trees/dts-arm32/imx7d-colibri-emmc-aster.dts
device-trees/dts-arm32/imx7d-colibri-eval-v3.dts
$ find device-trees/dts-arm64 -name "*imx8qxp-colibri*.dts"

device-trees/dts-arm64/imx8qxp-colibri-aster.dts
device-trees/dts-arm64/imx8qxp-colibri-dsihdmi-eval-v3.dts
device-trees/dts-arm64/imx8qxp-colibri-eval-v3.dt
$ find device-trees/dts-arm64 -name "*imx8mm-verdin*.dts"

device-trees/dts-arm64/imx8mm-verdin-nonwifi-dev.dts
device-trees/dts-arm64/imx8mm-verdin-wifi-dev.dts
device-trees/dts-arm64/imx8mm-verdin-wifi-dahlia.dts
device-trees/dts-arm64/imx8mm-verdin-nonwifi-dahlia.dts
$ find device-trees/dts-arm64 -name "*imx8mp-verdin*.dts"

device-trees/dts-arm64/imx8mp-verdin-wifi-dahlia.dts
device-trees/dts-arm64/imx8mp-verdin-nonwifi-dahlia.dts
device-trees/dts-arm64/imx8mp-verdin-nonwifi-dev.dts
device-trees/dts-arm64/imx8mp-verdin-wifi-dev.dts

Then to check the available overlays names for a particular device tree (imx6dl-colibri-eval-v3.dtb in this case), run the command below:

$ torizoncore-builder dto list --device-tree ./device-trees/dts-arm32/imx6dl-colibri-eval-v3.dts

Overlays compatible with device tree imx6dl-colibri-eval-v3.dts:
- device-trees/overlays/colibri-imx6_atmel-mxt-adapter_overlay.dts
- device-trees/overlays/colibri-imx6_atmel-mxt-connector_overlay.dts
- device-trees/overlays/colibri-imx6_hdmi_overlay.dts
- device-trees/overlays/colibri-imx6_parallel-rgb-lvds_overlay.dts
- device-trees/overlays/colibri-imx6_parallel-rgb_overlay.dts
- device-trees/overlays/colibri-imx6_stmpe-ts_overlay.dts
- device-trees/overlays/display-dpi-lt170410_overlay.dts
- device-trees/overlays/display-edt5.7_overlay.dts
- device-trees/overlays/display-edt7_overlay.dts
- device-trees/overlays/display-fullhd_overlay.dts
- device-trees/overlays/display-lt161010_overlay.dts
- device-trees/overlays/display-vga_overlay.dts

Then to include the device tree and the overlays in the image, use the customization below:

tcbuild.yaml
#Sample customization: insert the resistive 7" display overlay in the IMX6 device tree with Aster Carrier Board
customization:
  device-tree:
    include-dirs:
      - device-trees/include/
    custom: device-trees/dts-arm32/imx6dl-colibri-eval-v3.dts
    overlays:
      add:
        - device-trees/overlays/display-edt7_overlay.dts

See the following articles for more information:

Also, for more details on how the dt commands work, please check the dt command in the commands manual.

Kernel Modules

The build command allows you to build and install kernel modules into the deployed OS image. This is demonstrated in this configuration:

tcbuild.yaml
# Sample customization: build hello-mod module into the image
customization:
  kernel:
    modules:
      - source-dir: hello-mod/
        autoload: no

The directory hello-mod containing the source code files of the kernel modules should be available before running the command. To get the hello-mod module from our Github repository, run the command below:

$ git clone https://github.com/toradex/hello-mod

For more general information about the subject, please refer to Building External Kernel Modules With Torizon article.

Kernel Arguments (Parameters)

Kernel arguments (parameters) can be included in the OS image or modified, if they already exist, with the build command. This is done in the configuration file below:

tcbuild.yaml
# Sample customization: include or change (if already exists) key1 and key2 kernel arguments
customization:
  kernel:
    arguments:
      - key1=val1
      - key2=val2

For more general information about the subject, please refer to Customizing Kernel Arguments in Torizon article.

Capturing Configuration Changes in the Target Device

If the user made configuration changes on TorizonCore in the SoM, this command is necessary to capture them. For more details about what it does and for which cases it is applicable, please check the Typical Use Cases section.

To capture the configuration changes made in the target device, run the command below:

$ torizoncore-builder isolate --remote-host 192.168.1.117 --remote-username torizon --remote-password torizon --changes-directory changes1

Then to include these changes in the image, use the customization below:

tcbuild.yaml
#Sample customization: insert the configuration changes made in the device
customization:
  filesystem:
     - changes1/

See the Capturing Changes in the Configuration of a Board on TorizonCore article for detailed instructions.

Also, for more details on how the isolate command works, please check the isolate command in the commands manual.

Pre-provisioning Docker Container Images

The build command also simplifies the process of bundling container images into an installer image. A sample configuration leveraging that feature is this:

tcbuild.yaml
#Sample customization: insert pre-provisioned Docker container images
output:
  easy-installer:
    local: torizon-core-docker-colibri-imx6-Tezi_5.3.0.CUSTOM
  bundle:
      compose-file: custom/docker-compose.yml

With such an input, the tool will fetch the container images referenced by the specified docker-compose.yml file and combine them with the installer image.

A sample Docker Compose YAML file can be downloaded, based on your device, by clicking in one of the buttons below:

Example 32-bit Docker Compose file

Example 64-bit Docker Compose file

Pre-provisioning Docker Container Images: Pre-bundled Container Images

If you don't want the tool to download the images every time the build command is run, you can pass a directory with the pre-bundled container images instead of the Docker Compose file as a parameter.

To create a tarball with the container images from a Docker Compose file, in the directory bundle, use the command below:

$ torizoncore-builder bundle docker-compose.yml --bundle-directory bundle --platform linux/arm/v7

Then to include these pre-bundled container images in the image, through the directory, use the configuration below:

tcbuild.yaml
#Sample customization: insert pre-provisioned Docker container images with pre-built container images
output:
  easy-installer:
    local: torizon-core-docker-colibri-imx6-Tezi_5.3.0.CUSTOM
  bundle:
    dir: bundle/

Find more details in the Pre-provisioning Docker Containers onto a TorizonCore image article.

For more details on how the bundle command works, please check the bundle command in the commands manual.

Building The Custom Image of TorizonCore

To run the tool one would simply execute:

$ torizoncore-builder build

...
1091 metadata, 12741 content objects imported; 412.2 MB content written                                                                                                                                   
Pulling done.
Deploying OSTree with checksum 58629613a342197c31c5911d0874aac1b0fcb46b68a63f59760c03bacc4df08a
Deploying done.
Copy files not under OSTree control from original deployment.
Packing rootfs...
Packing rootfs done.

=>> Build command successfully executed!

Once the build process finishes, the custom Easy Installer image would be present in the directory defined in the local of the output section of the configuration file, as it can be seen in the Minimal Configuration File section.

Different configuration file name

The default configuration file used is the tcbuild.yaml file, created in the Creating a Configuration file section. In case of using a configuration file with a different name, run the command specifying the configuration file name:

$ torizoncore-builder build -f <configuration_file_name>

Variable substitution

A feature that can be especially useful in CI scripts is called variable substitution which allows the replacement of property values (or parts of them) based on parameters passed on the command line of the tool. To use this feature, click on the link below and follow the steps.

Variable Substitution

Deploying The Custom Image Of TorizonCore

The output image can be deployed in three ways to the board:

  • Using Toradex Easy Installer, the recommended method for production programming.
  • Directly on the board through SSH, the recommended method during development.
  • Through Torizon Platform Services server, the recommended method to perform updates on a device already deployed to the field.

Note: Despite the recommendations, it is also possible to use a different method, using the Torizon Platform Services method during development for example.

To learn more about when to use each method, please refer to the Deployment Methods Comparison section of the TorizonCore Builder - Workflow article.

Toradex Easy Installer

Toradex Easy Installer Tool allows users to install an image into the internal flash memory of Toradex modules in an extremely simple way.

Copy the output image into a USB stick and follow the steps in the Toradex Easy Installer article linked above. The copy of the output image can be done with the command below:

$ cp -a torizon-core-docker-colibri-imx6-Tezi_5.3.0.CUSTOM /media/user/myUSBstick

Where, in my case, /media/user/myUSBstick is the path to USB stick mount point and torizon-core-docker-colibri-imx6-Tezi_5.3.0.CUSTOM is the directory containing the custom Toradex Easy Installer image.

Directly, through SSH

In this case, before deployment the output image needs to be unpacked, using the command below:

$ torizoncore-builder images unpack torizon-core-docker-colibri-imx6-Tezi_5.3.0.CUSTOM

Change the argument torizon-core-docker-colibri-imx6-Tezi_5.3.0.CUSTOM to the directory containing your custom Toradex Easy Installer image.

For more details on how this command works, please check the images unpack command in the commands manual.

To deploy it directly to the board, through SSH, use the command below:

$ torizoncore-builder deploy --remote-host 192.168.1.117 --remote-username torizon --remote-password torizon --reboot

For more details on how this command works, please check the deploy command in the commands manual.

Change the arguments --remote-host,--remote-username and --remote-password to your board IP Address, username and password, respectively.

Tip: From TorizonCore Builder 3.1.0 after, the default value of --remote-username and --remote-password is torizon, so if the username or the password is torizon the argument can be omitted.

Warning: This way does not support the deployment of the pre-provisioned containers.

Torizon Platform Remote Update Service

In this case, before deployment the output image needs to be unpacked, using the command below:

$ torizoncore-builder images unpack torizon-core-docker-colibri-imx6-Tezi_5.3.0.CUSTOM

Change the argument torizon-core-docker-colibri-imx6-Tezi_5.3.0.CUSTOM to the directory containing your custom Toradex Easy Installer image.

For more details on how this command works, please check the images unpack command in the commands manual.

To deploy it to the Torizon Platform Remote Update Service, use the command below:

$ torizoncore-builder push --credentials credentials.zip base

For more details on how this command works, please check the push command in the commands manual.

Warning: This way does not support the deployment of the pre-provisioned containers, so their Docker Compose file will need to be deployed separately in the server.

To deploy the image from the Torizon Platform Remote Update Service to the board, please follow the steps in the OTA Web Interface article.

To obtain credentials (credentials.zip file) and to obtain more information on how to sign and securely deploy a custom image using Over-the-Air (OTA) updates, check the Signing and pushing the image to Torizon OTA article.

Verifying The New Custom Image On The Device

After rebooting, in your target device's terminal, verify that your new custom image of TorizonCore is active on the device with the command below:

# sudo ostree admin status

Password: 
* torizon 58629613a342197c31c5911d0874aac1b0fcb46b68a63f59760c03bacc4df08a.0
    Version: 5.3.0+build.7-tcbuilder.20211008140217
    origin refspec: tcbuilder:58629613a342197c31c5911d0874aac1b0fcb46b68a63f59760c03bacc4df08a
  torizon 36ad904617b170339b6ded7b9dce87ed8cf0f76473b897fdd832d91e82eb1ddc.0 (rollback)
    Version: 5.3.0+build.7
    origin refspec: tcbuilder:36ad904617b170339b6ded7b9dce87ed8cf0f76473b897fdd832d91e82eb1ddc

Where 58629613a342197c31c5911d0874aac1b0fcb46b68a63f59760c03bacc4df08a is the OSTree commit hash and should be the same as the one in the "Deploying OSTree with checksum ..." part of the output of the build command.

Additional Resources

Proceed to the Documentation tab on this page.

Extra Resources for TorizonCore Customization

See below more documents related to TorizonCore Builder and TorizonCore Customization

Article Brief Description
TorizonCore Builder Tool - Customizing TorizonCore Images Create a custom TorizonCore image that you can use in production programming with Toradex Easy Installer, or push your changes to Torizon Platform Services
TorizonCore Builder - Workflow Workflow of TorizonCore Builder, with a comparison between the approaches to use the tool and diagrams to graphically present the workflow and each approach
TorizonCore Builder Tool “build” command Learn how to fully customize an image as a single step using the build command from TorizonCore Builder
Device Tree Overlays on Torizon How to modify the device tree without having to re-compile it
Device Tree Customization Examples Some examples of Device Tree Customization
Pin Multiplexing - Changing Pin Functionalities in the Linux Device Tree How to create a dts file to change the pin muxing configuration
Building External Kernel Modules With Torizon How to deploy an external kernel module using TorizonCore Builder with an example
Customizing Kernel Arguments in Torizon How to use TorizonCore Builder to customize kernel arguments in Torizon
Splash Screen Customization on TorizonCore Learn how to change the splash screen using initramfs
Touch Screen Calibration (Torizon) How to use weston-touch-calibrator to calibrate the touch
Capturing Changes in the Configuration of a Board on TorizonCore Use TorizonCore Builder to save and reproduce board customization
Pre-provisioning Docker Containers onto a TorizonCore image How to preinstall a Docker Container onto a TorizonCore image using your PC
TorizonCore Builder - Commands Manual Commands manual of TorizonCore Builder, showing all commands of the tool and explaining them
TorizonCore Boot Time Optimization How to measure and optimize TorizonCore boot time
Reliability on Torizon Learn how to improve reliability on TorizonCore
How to Store Docker Data on an External Storage Device (USB/SD Card) How to storage Docker data on an external device like an SD Card or a USB stick
Persistent Journald Logging How switch from in-RAM to persistent Journald logging
Using Private Registries With Torizon Platform How to deploy private registry credentials to your devices

Torizon 4.0.0

Introduction

Toradex provides Easy Installer to flash images on modules. Developer may make changes in configuration (/etc) of the system and/or work with containers for the end user solution. Now how to replicate the same on other modules? Toradex has developed a new tool, TorizonCore Builder, to let the developer pick those modifications/containers and create another image out of it, keeping the original Toradex Easy Installer image of Torizon as base. TorizonCore Builder is containerized and run on the host system.

This article complies to the Typographic Conventions for Torizon Documentation

Workflow


  • TorizonCore Builder Workflow

    TorizonCore Builder Workflow

Commands

TorizonCore Builder is provided in the form of Docker image and has following commands to create Toradex Easy Installer image of Torizon with required changes

  • Unpack

  • Bundle

  • Isolate

  • Union

  • Deploy

  • Combine

Operation

Firstly, user needs a base Toradex Easy Installer image of Torizon which will be used by TorizonCore Builder container to add user changes in it. User has two options, either to perform all tasks in a “working directory”, which must be a file system capable of carrying Linux file system metadata (such as Unix style file system permissions and extended attributes), by setting appropriate arguments OR let the commands use Docker volumes for intermediate storage to perform the operations on image.

  • The very first command to use is unpack, which unpacks the OSTree file system from user provided Toradex Easy Installer image of Torizon

  • If pre-installation of Docker container(s) onto a Torizon image with Docker engine is required, bundle command is to be used by providing a Docker compose file stating all image(s).

  • If user has made any changes in configuration (/etc), isolate command is to be used to get all those changes (modifications, addition and deletion). It requires ssh connection to the target board.

  • If user has used isolate command, union command is to be used to capture all changes in the form of a commit.

  • If user has used union command, deploy command is to be used to deploy the OSTree commit onto the OSTree file system and create a new Toradex Easy Installer image of Torizon.

  • Combine is used to create a Toradex Easy Installer image of Torizon by combining outputs from deploy command and bundle command i.e. combine user configuration changes and pre-provisioned Docker container(s).

Usage

Warning: Make sure to use an empty working directory and navigate to it. This will be bind mounted to the container this tool is running in and will be used to store the final image and may also use to store temporary files.

Note: Toradex Easy Installer image of Torizon can be downloaded from Toradex Easy Installer

Extract the downloaded image tar file

$ mkdir <directory to extract image into>
$ tar -xf <downloaded image.tar> -C <directory to extract image into>

Basic

For convenience, create alias first

$ alias torizoncore-builder='docker run --rm -it -v $(pwd):/workdir -v storage:/storage -v /deploy --net=host -v /var/run/docker.sock:/var/run/docker.sock torizon/torizoncore-builder:1'

TorizonCore Builder tool uses Docker volumes as persistent storage for intermediate outputs of sub-commands.

TorizonCore Builder tool provides different logging levels, which can be set by using --log-level option. There are five logging levels i.e. debug, info, warning, error and critical and default is set to info. Formatted logging can be enabled by using --verbose option.

Sub-commands of TorizonCore Builder tool can be used as follows

unpack

unpack gets the base Toradex Easy Installer image of Torizon and unpacks the rootfs (OSTree filesystem) to Docker volume

$ torizoncore-builder unpack --image-directory <base Toradex Easy Installer image of Torizon>

Isolate

isolate is used to get configuration changes(modifications/addition/deletion) from the Target board. It creates ssh session with Target board to get changes and store them within Docker volume on host.

$ torizoncore-builder isolate --remote-ip <ip of target board> --remote-username <username on target board> --remote-password <password of user on target board> 

Union

union is used to make an OStree commit of all changes in OSTree repo of unpacked base Torizon image.

$ torizoncore-builder union --union-branch <name to be given to the branch containing the changes commit>

Deploy

deploy is used to take branch from union command, make it default deployment, create the new Torizon image, make directory (inside bind mounted working directory ) with user provided name and store Torizon Image in it.

Note: Output directory is not needed to be created.

$ torizoncore-builder deploy --output-directory <name of directory to store Torizon image>

Bundle

bundle is used to bundle the Docker container(s), which is used by combine command to pre-provision container(s) bundle with Torizon Image.

User provides the docker-compose.yml, platform of target board (linux/arm/v7 or linux/arm64) and working directory of host to store data used for DIND (docker in docker) and bundle command stores the Docker container(s) tar archive in directory named “bundle” inside the working directory.

$ torizoncore-builder bundle --file <path to docker-compose.yml> --platform <target board like linux/arm/v7 or linux/arm64> --host-workdir $(pwd)

Combine

combine command is used to create Torizon image by combining user provided Torizon image (either base Torizon Image OR output of deploy;depending on what is needed) with bundled Docker container(s) from bundle command.

$ torizoncore-builder combine --image-directory <Path to Torizon image> --output-directory <Torizon image with pre-provisioned Docker container(s)>

Example

Creating alias

$ alias torizoncore-builder='docker run -it --rm -v $(pwd):/workdir -v storage:/storage -v /deploy --net=host -v /var/run/docker.sock:/var/run/docker.sock torizon/torizoncore-builder:1'

Unpack base Toradex Easy Installer image of Torizon

$ torizoncore-builder unpack --image-directory colibri-imx7
Copying Toradex Easy Installer image.
Unpacking TorizonCore Toradex Easy Installer image.
Unpacked OSTree from oradex Easy Installer image:
Commit ref: 3c5795a0dcd0ceca552c916f6515a84011425443fd14d118caa09b869dbe4804
TorizonCore Version: 4.0.0-devel-20200607+build.117

Get configuration changes from target board

$ torizoncore-builder isolate --remote-ip 192.168.1.146 --remote-username torizon --remote-password 1234 
isolation command completed

Merge changes to base Toradex Easy Installer image of Torizon

$ torizoncore-builder union --union-branch=changes_committed
Commit d77ed9d5ddec41b7b61f9a571e810de2447133391ee19fe5f34d1a81009967d1 has been generated for changes and ready to be deployed. 

Set branch to deploy at boot. Output image can be deployed on target board. Note: deployed_image is not needed to be created

$ torizoncore-builder deploy --output-directory deployed_image --ref=d77ed9d5ddec41b7b61f9a571e810de2447133391ee19fe5f34d1a81009967d1
Running tar command: tar --xattrs --xattrs-include='*' -cf /workdir/deployed_image/torizon-core-docker-colibri-imx7.ota.tar.xz --xz -S -C /deploy -p .
Using unpacked Toradex Easy Installer image as base:
  Commit ref: 3c5795a0dcd0ceca552c916f6515a84011425443fd14d118caa09b869dbe4804
  TorizonCore Version: 4.0.0-devel-20200607+build.117
  Kernel arguments: quiet logo.nologo vt.global_cursor_default=0 plymouth.ignore-serial-consoles splash ostree=/ostree/boot.1/torizon/d103ac800d2c9a67cce7ec68ad1accb836d766ff959895820deea195f16eaeed/0

Deploying commit ref: d77ed9d5ddec41b7b61f9a571e810de2447133391ee19fe5f34d1a81009967d1
Pulling OSTree with ref d77ed9d5ddec41b7b61f9a571e810de2447133391ee19fe5f34d1a81009967d1 from local repository...
Pulling done.
Deploying OSTree with ref d77ed9d5ddec41b7b61f9a571e810de2447133391ee19fe5f34d1a81009967d1
Deploying done.
Copy rootdirs such as /home from original deployment.
Packing rootfs...
Packing rootfs done.

If required, create tar archive of Docker container(s)

$ torizoncore-builder --bundle-directory bundled_containers  bundle -f docker-compose.yml --platform linux/arm/v7 --host-workdir $(pwd)

Create image with pre-provisioned docker containers. --image-directory can be either base image or output from deploy command.

$ mkdir preprov_cont_image
$ torizoncore-builder --bundle-directory bundled_containers combine --image-directory deployed_image --output-directory preprov_cont_image
Creating copy of TorizonCore source image.
Combining TorizonCore image with Docker Container bundle.
Successfully created a TorizonCore image with Docker Containers preprovisioned in preprov_cont_image

Advanced

User can use TorizonCore Builder tool by either providing the working directory (capable of carrying Linux filesystem metadata) or Docker volumes for intermediate data storage. So all commands are explained with both approaches. Commands can be used with mix of Docker volumes and working directory but that will complicate the usage. So it is recommended to use either working directory or Docker volumes for all the commands.

As it is provided in the form of Docker image, any of the following commands (based on working directory or Docker volumes) can be executed to create an alias to be easy to use.

Usage of all commands explained later are based on the following aliases.

With Docker volumes:

$ alias torizoncore-builder='docker run --rm -it -v $(pwd):/workdir -v storage:/storage -v /deploy --net=host -v /var/run/docker.sock:/var/run/docker.sock torizon/torizoncore-builder:1'

Here, two volumes are being created and mounted at /storage and /deploy of container for internal usage and host current directory is bind mounted at /workdir within container.

With user provided working directory (capable of handling Linux filesystem metadata)

$ alias torizoncore-builder='docker run --rm -it -v $(pwd):/workdir --net=host -v /var/run/docker.sock:/var/run/docker.sock torizon/torizoncore-builder:1'

move to the working directory so the required one gets bind mounted when alias is executed

Base Command

Base command is now alias torizoncore-builder and has below optional options

--verbose: enable formatted logs

--log-level: set log level from options debug, info, warning, error and critical
default: info

--log-file: write logs to specified file instead of console

--bundle-directory: path to directory to store output of bundle command
default: creates and use directory with name bundle in working directory

--storage-directory: path to directory to use as intermediate storage for sub-commands.
default: Docker volume mounted at/storage
Note: do not set it if Docker volumes are to be used.

Unpack

Unpack command is used to unpack the ostree filesystem from user provided Toradex Easy Installer image of Torizon base image.

Arguments

--image-directory : path to base Toradex Easy Installer image of Torizon in the working directory

--sysroot-directory (optional): path to where ostree filesystem from Toradex Easy Installer image of Torizon will be unpacked
default: sysroot directory inside either user provided storage_directory using --storage-directory option OR docker volume (mounted at /storage)

Example
With docker volume
$ torizoncore-builder unpack --image-directory <path to base Toradex Easy Installer image of Torizon in the working directory>  
Without Docker Volume

Additional argument --sysroot-directory, specifying path to store unpacked ostree filesystem, is needed in case of user provided working directory (capable of handling Linux filesystem metadata) is to be used for storage of intermediate data.

$ torizoncore-builder unpack --image-directory <path to base Toradex Easy Installer image of Torizon in working directory> --sysroot-directory <path to where ostree filesystem from Toradex Easy Installer image of Torizon will be unpacked>

If there is already an unpacked ostree filesystem, user is prompted to delete it because currently a single ostree filesystem can be handled.

Isolate

Isolate command is used to get the configuration (/etc) changes (modifications, additions and deletions) from user target module. It ignores following files/directories for any change

'gshadow', 'machine-id', 'group', 'shadow', 'systemd/system/sysinit.target.wants/run-postinsts.service', 'ostree/remotes.d/toradex-nightly.conf', 'docker/key.json', '.updated', '.pwd.lock', 'group-', 'gshadow-', 'hostname', 'ssh/ssh_host_rsa_key', 'ssh/ssh_host_rsa_key.pub', 'ssh/ssh_host_ecdsa_key', 'ssh/ssh_host_ecdsa_key.pub','ssh/ssh_host_ed25519_key','ssh/ssh_host_ed25519_key.pub', 'ipk-postinsts', 'fw_env.conf'

Isolate command establishes a ssh session with the target board, gets all the changes and places them in either user provided directory or within docker volumes.

Arguments

--remote-ip: IP of target board

--remote-username: Username of target board to be used for creating ssh session

--remote-password: Password of target board to be used for creating ssh session

--diff-dir (optional): If docker volume is not to be used, directory for storing configuration changes
default: changes directory inside either user provided storage_directory using --storage-directory option OR docker volume (mounted at /storage)

Example
with docker volume
$torizoncore-builder  isolate --remote-ip <target board ip> --remote-username <target board username> --remote-password <taregtboard password>
without docker volume

Directory path is needed (--diff-dir) to store changes within user provided working directory

$ torizoncore-builder  isolate --remote-ip <target board ip> --remote-username <target board username> --remote-password <taregtboard password> --diff-dir <directory to store changes>

Union

It is to be run after isolate command to commit changes to the ostree filesystem. If changes are stored in docker volume, it is to be run with docker volume and if changes are stored in working directory, same directory is needed to be provided.

Arguments

----union-branch: branch name containing the changes committed to ostree repo of unpacked ostree filesystem

--diff-directory(optional): path (within working directory) to configuration changes picked by isolate command.
default: changes directory inside either user provided storage_directory using --storage-directory option OR docker volume (mounted at /storage)

--sysroot-directory(optional): path to unpacked ostree filesystem from Toradex Easy Installer image of Torizon (same as used in unpack command)
default: sysroot directory inside either user provided storage_directory using --storage-directory option OR docker volume (mounted at /storage)

Example
with docker volume
$ torizoncore-builder union --union-branch <name to be given to the branch>
without docker volume

isolate command is used to store changes in a directory within working directory. Same directory is needed here with --diff-directory option.

$ torizoncore-builder union --union-branch <name to be given to the branch> --diff-directory <path to directory containing configuration changes> --sysroot-directory <path to unpacked ostree filesystem from Toradex Easy Installer image of Torizon>  

Deploy

It takes a branch from union command, sets it up for the boot, packs the rootfs and creates the Toradex Easy Installer image of Torizon ready to be deployed (if no pre-install container(s) are needed).

If pre-installed container(s) are needed, “combine command“ is used to create a new Toradex Easy Installer image of Torizon with bundled container(s) from bundle command.

Arguments

--output-directory: directory to store Toradex Easy Installer image of Torizon with deployment set to branch from union command. It is not needed to be created by the user.

--diff-ref: branch name/checksum to be deployed from union command
default: active deployment in ostree repo

--sysroot-directory (optional): path to unpacked ostree filesystem from Toradex Easy Installer image of Torizon
default: sysroot directory inside either user provided storage_directory using --storage-directory option OR docker volume (mounted at /storage)

--deploy-sysroot-directory (optional): directory to store the intermittent deployment sysroot (used internally by tool).
default: docker volume mounted at /deploy

Example
with docker volume
$ torizoncore-builder deploy --output-directory <directory to store Toradex Easy Installer image of Torizon with deployment set to provided reference> --diff-ref <name or checksum of branch within unpacked ostree repo> 
without docker volumes
$ torizoncore-builder deploy --output-directory <directory to store Toradex Easy Installer image of Torizonm image with deployment set to provided reference> --diff-ref <name or checksum of branch within unpacked ostree repo> --sysroot-directory <path to unpacked ostree filesystem from Toradex Easy Installer image of Torizon> --deploy-sysroot-directory <directory to store the intermittent deployment sysroot>

Bundle

It creates a tar archive of bundle container(s) from provided docker-compose file and stores it in ether "bundle" named directory inside working directory or with user provided name and path using --bundle-directory option of torizonbuilder. Docker-compose file and tar archive is later combined with Toradex Easy Installer image of Torizon using combine command to pre-install containers in Toradex Easy Installer image of Torizon. Toradex EasyInstaller tool copies containers data to appropriate places in rootfs during flashing the image on Target board.

Arguments

--compose-file: path to docker-compose file
default: docker-compose.yml in working directory

--platform: specify platform to make sure fetching the correct container image when multi-platform container images are specified (e.g. linux/arm/v7 or linux/arm64).
default: linux/arm/v7

--host-workdir: directory to save some internal information for DIND(Docker in Docker) and output container tar archive.

Examples
$ torizoncore-builder --bundle-directory <path to store docker bundle tar archive> --compose-file <pat to docker compose file> --platform <platform of container> --host-workdir <path to directory to store continaer bundle>

Combine

It is used to create a Toradex Easy Installer image of Torizon by combining tar archive with user provided Toradex Easy Installer image of Torizon. User provided Toradex Easy Installer image of Torizon can be either image created by deploy command or base Toradex Easy Installer image of Torizon as per need.

Arguments

--image-directory: path to Toradex Easy Installer image of Torizon to be combined with docker bundle

--output-directory: path to combined Toradex Easy Installer image of Torizon with bundled containers

--image-name (optional) : image name to be used in Easy Installer image json

--image-description (optional): image description to be used in Easy Installer image json

--image-licence (optional): path to license file which will be shown on image installation

--image-release-notes (optional): path to release notes file which will be shown on image installation

Example
$ torizoncore-builder --bundle-directory <path to docker bundle tar archive> --image-directory <path to Tezi image to be combined with docker bundle> --output-directory <directory to store combined Tezi image> --image-name <name of image to be set in image json file> --image-description <image desription to be set in image json file> --image-licence <path to license file to be shown at time of installation of image> --image-release-notes <path to release notes file which will be shown on image installation>