In-Toto Attestations in Torizon OS

Introduction

Software supply chain security has become critical in embedded systems. Attacks on the software supply chain are one of the most dangerous class of attacks, because they often lead to deep, long-term compromise that can go undetected for years. Transparency in the supply chain is critical for protecting against this type of attack.

At Toradex, we leverage in-toto attestations to ensure the transparency, integrity, and traceability of our Torizon OS builds. This documentation explains what in-toto is, how it is integrated into the build and release process for Torizon OS, and how customers can benefit from it.

Why In-Toto?

The Challenge

Cybersecurity is a broad discipline, and different actors might choose different tools or controls to verify system security. Tools like automated scanners and system analyzers are often employed to fingerprint running services. Torizon OS strikes a balance between stability of the base OS and security updates. When new vulnerabilities arise, we evaluate what the best course of action is; usually, the choice is between backporting a patch or updating the package.

For example, Torizon OS 7 ships with OpenSSH 9.6p1. Since OpenSSH 9.6p1 was released, some important vulnerabilities have been discovered in that version, including CVE-2024-6387. Every Torizon release includes a large number of patches for various security issues, usually inherited from the upstream Yocto project patches, so more recent Torizon OS versions are patched against this vulnerability. However, automated scanners may fingerprint the OpenSSH version as 9.6p1, see that there are vulnerabilities in the original, unpatched version of that release, and flag it as a security issue.

As Torizon OS is 100% open source, it's always been possible to check the build recipes and verify exactly what goes into each build. But that can be cumbersome. Additionally, some security-conscious users may wish to verify that the versions tagged in git actually correspond to the binary releases of Torizon OS, and simply taking the tagged release and checking the recipes can't fulfill that requirement.

So, we have two issues to solve:

• Scanner mismatch: Tools often flag vulnerabilities simply based on version strings, without validating that the vulnerabilities are actually present. This can lead to unnecessary alarm over false-positive results.
• Proof gap: Rebuilding Torizon OS from source is always possible, and looking at the sources for any release can show what packages should have gone into the release build. But there should be a way for a user to verify that the build actually matches the sources.

The Solution

Toradex uses in-toto attestations to record and cryptographically sign the inputs, and outputs of each step of our build process. This provides verifiable evidence of:

• Which sources and patches were included
• How artifacts were produced
• That no tampering occurred between build and release

What is In-Toto?

In-toto is a CNCF-graduated framework for securing software supply chains. It consists of:

• Specification: Defines how to describe supply chain steps
• Metadata & Attestations: Signed records of what happened at each build stage
• Verification tooling: To validate that expected steps were followed

Key Concepts

• Run / Record: Commands that track inputs, outputs, and commands executed
• Layout: A recipe defining the allowed build steps and their expected outputs
• Verify: Ensures attestations match the layout and are signed by the right keys

Integration in Torizon OS

Since 2023, Toradex has been generating signed in-toto attestations for all Torizon OS release builds. Since the release of Torizon 6.6.1 in April 2024, we have made these attestations public (including for all releases of Torizon 7). These are published alongside the other release artifacts on our public server, the same place that Toradex Easy Installer downloads the releases from.

Build Steps Attested

For each build step, the following process is followed:

1. in-toto records the filenames and hashes of all of the inputs to the step
2. Our CI system performs the build step
3. in-toto records the filenames and hashes of all the outputs of the build step
4. The build server cryptographically signs the in-toto attestation, which thus includes the names and hashes of all input and output files

The build steps we define are:

1. Repo Init – Captures all Yocto layers downloaded, including the hashes of each recipe and patch file
2. BitBake Conf – Records build configuration (local.conf, etc.)
3. BitBake Fetch – Records the recipe files and configuration from steps 1 and 2 as inputs, then fetches and records the bitbake download step as the output--i.e., the actual complete source code that will be used to build the final image. Depending on the package recipe, this may be a source tarball or a git commit.
4. BitBake Build – Records the results of the first 3 steps (recipes, configuration, and downloaded source code) as its input, and then performs the actual build. Its recorded output is the final deployed artifacts that are publicly released.

Each step links inputs → outputs with cryptographic integrity.

Example: OpenSSH Patches

Attestations are very important as a security measure, a public record of what went into a build. However, they are also extremely useful as a way to dig into exactly what has been patched or not for a particular package. By convention in Yocto, backported patches for security fixes are named according to the CVE patched. So a patch for CVE-2024-6387, the OpenSSH vulnerability mentioned above, would have the filename CVE-2024-6387.patch. We can take the published in-toto attestation for the build and, using in-toto, confirm exactly which patches were applied in a given release.

One-Liner Check

This bash one-liner will show all of the security patches applied to the OpenSSH binary built into Torizon OS 7.3.0 for the Verdin i.MX8M Plus:

$ curl https://artifacts.toradex.com/artifactory/torizoncore-oe-prod-frankfurt/scarthgap-7.x.y/release/18/verdin-imx8mp/torizon/torizon-docker/attestations/bitbake-build.f261b475.link \
 | jq -r .payload | base64 -d \
 | jq '.materials | with_entries(select((.key|contains("openssh")) and (.key|contains("CVE"))))'

Output Example

{
  "oe/layers/openembedded-core/meta/recipes-connectivity/openssh/openssh/CVE-2024-39894.patch": {
    "sha256": "15c663cb89fccf1af0918652ce372f741746815bb6bc1aaa62906f738e12f97e"
  },
  "oe/layers/openembedded-core/meta/recipes-connectivity/openssh/openssh/CVE-2024-6387.patch": {
    "sha256": "b0ff779d1a87fd2f4aa84f4361de4c4fcf21d996148293d3c82c4c700277745b"
  },
  "oe/layers/openembedded-core/meta/recipes-connectivity/openssh/openssh/CVE-2025-26465.patch": {
    "sha256": "47423885dc3f8e59c19904bc4131b25d4e612a14ff549a45fa52abe1c2e73c2e"
  },
  "oe/layers/openembedded-core/meta/recipes-connectivity/openssh/openssh/CVE-2025-26466.patch": {
    "sha256": "b3854a7d427a69c4b25cf1b6d6d5d9fda676dd03683ed4103b8fd386d1c9fb57"
  },
  "oe/layers/openembedded-core/meta/recipes-connectivity/openssh/openssh/CVE-2025-32728.patch": {
    "sha256": "98b7d80307230a96ccafff2741332eb6581f0aa0c790c5b7cc6f6904439538b1"
  }
}

Step-by-step breakdown

This one-liner combines several steps:

• Locate the build on the Toradex release server. In this case, our release build was scarthgap-7.x.y/release/18/verdin-imx8mp/torizon/torizon-docker.
• Look in the attestations directory for the bitbake-build attestation
• Download the attestation with curl and extract the base64-encoded payload containing the recorded inputs (jq -r .payload)
• Decode the base64 payload (base64 -d)
• Query the decoded payload using jq:
  • Take the materials object where build inputs are stored (.materials)
  • Filter the results to only show the filenames that include the strings openssh and CVE (with_entries(select((.key|contains("openssh")) and (.key|contains("CVE")))))

Further verification steps

This approach is quick and convenient, and tells us that a patch was applied, but it's also possible to go further. Here are some more verifications you can do using the in-toto attestations.

Check that the security patches weren't modified between download and build

For this control, we could check that the hashes listed in the step 1 (repo-init) output match the hashes in the step 4 (bitbake-build) input:

$ curl https://artifacts.toradex.com/artifactory/torizoncore-oe-prod-frankfurt/scarthgap-7.x.y/release/18/verdin-imx8mp/torizon/torizon-docker/attestations/repo-init.f261b475.link \
 | jq -r .payload | base64 -d \
 | jq '.products | with_entries(select((.key|contains("openssh")) and (.key|contains("CVE"))))'

Note that here, we are checking the products (outputs) of step 1, not the materials (inputs) like we did for step 4. This verifies that the initial recipes we downloaded are the same ones the bitbake build command used.

Check for tampering in the release image

To verify that the Toradex Easy Installer image we published actually matches the image listed in the attestation of the bitbake-build output:

$ curl https://artifacts.toradex.com/artifactory/torizoncore-oe-prod-frankfurt/scarthgap-7.x.y/release/18/verdin-imx8mp/torizon/torizon-docker/attestations/bitbake-build.f261b475.link \
 | jq -r .payload | base64 -d \
 | jq '.products | with_entries(select(.key|contains("torizon-docker-verdin-imx8mp.ota.tar.zst")))'
$ curl https://artifacts.toradex.com/artifactory/torizoncore-oe-prod-frankfurt/scarthgap-7.x.y/release/18/verdin-imx8mp/torizon/torizon-docker/teziimage/torizon-docker-verdin-imx8mp.ota.tar.zst | sha256sum

The hashes from these two commands should match.

Audit the actual contents of the security patch

You also might want to audit the contents of the patch: what if the patch we downloaded didn't actually address the issue? This is where the strength of open source shines. None of the inputs to our build are secret, so you can find the actual patch from an independent source, audit it, download it, and check that the hash of the patch you audited matches the hash of the patch our build server applied.

We can see that the applied patch came from openembedded-core. We can find the patch on the scarthgap branch of the openembedded-core git repository. Then, we can simply check that the hash of this patch matches the attested hash of the patch that our build server applied:

$ curl https://raw.githubusercontent.com/openembedded/openembedded-core/refs/heads/scarthgap/meta/recipes-connectivity/openssh/openssh/CVE-2024-6387.patch | sha256sum

The hash from this command should match the hash of the patch file in the attestation.

Independently reproduce our Torizon OS builds

If you wanted to go even further, you could reproduce the build yourself locally, and verify that the hashes for each step match the hashes that our build server produced.

Build binary reproducibility

Unfortunately, Yocto builds aren't yet 100% bit-for-bit reproducible, so you won't necessarily get precisely the same results that we did for the outputs of the bitbake-build step. The other steps are fully reproducible, however.

Conclusion

Going back to our example, where a concerned security auditor found that a potentially-vulnerable version of OpenSSH was contained in the Torizon OS image, we solved the issue, and got some meaningful benefits by looking at the in-toto build attestations:

• Patched Since 7.0.0: The OpenSSH CVE we were worried about had already been patched. In fact, if we go back and look at previous releases, it was patched in every Torizon OS 7 release.
• Strong links between source code, build step, and artifact: The in-toto attestations give us a place to quickly query information about what sources went into a given build, but they also provide a strong public audit trail that can be independently verified.
• Assurance against spurious security scanner reports: Security tools often only flag by version, and don't know about applied patches. In-toto provides proof.

As a final note, we should point out that Torizon OS also offers a much better alternative to actually running an SSH server on production devices. It's usually not recommended to leave an SSH server listening; even if it's hardened and patched, it represents an attack vector that needs to be secured. Torizon OS (with Torizon Cloud) has the Secure Remote Access feature, allowing you to have remote-access sessions using vanilla SSH without having to have an SSH server running at all times. Secure Remote Access leverages the same root of trust as our software update system, and instructs the device to start an SSH session on-demand, for a limited amount of time, and with a locked-down configuration that only allows access via SSH keys that are authorized on a per-session basis.