Support cocoon-tpm's OpenSSL backend by osteffenrh · Pull Request #1210 · coconut-svsm/svsm

osteffenrh · 2026-09-04T10:42:39Z

Add support for cocoon-tpm's OpenSSL backend (cocoon-tpm PR: coconut-svsm/cocoon-tpm#30).

cocoon-tpm-crypto is the designated crypto library for SVSM. It supports
multiple backends: pure Rust, BoringSSL, and — with the above PR — OpenSSL.
This PR integrates the new OpenSSL backend on the SVSM side.

Previously, the vTPM code (libtcgtpm) maintained its own OpenSSL build —
including a git submodule, build config, and a bundled copy of libcrt (a
minimal C runtime). With cocoon-tpm now providing an OpenSSL backend, this
duplication is eliminated: a single libcrypto.a is built by cocoon-tpm's
ossl-bare-sys crate and shared by both cocoon-tpm-crypto and libtcgtpm.
The OpenSSL submodule and build infrastructure are removed from libtcgtpm.

The OpenSSL backend is selected via the cargo feature openssl. The vtpm
feature now implies openssl, since the TCG TPM reference code requires
OpenSSL at the C level. Without vtpm, the pure-Rust backend remains the
default.

To build OpenSSL for the freestanding SVSM environment, cocoon-tpm's
ossl-bare-sys uses a target-integration pattern: the upstream crate depends
on a stub crate (cocoon-tpm-ossl-bare-sys-target-integration) that is
overridden via [patch.crates-io] in each target environment. The SVSM
implementation of this crate provides the OpenSSL build configuration
(Configure flags and target config file) and depends on our libcrt crate for
the C runtime headers needed during the OpenSSL build.

libcrt itself is extracted from libtcgtpm into a standalone crate
(libcrt). It has no Rust code — it just builds the C runtime and provides
it at link time. Both the target-integration crate and libtcgtpm depend on
it. Include paths and library locations are communicated between crates via
cargo metadata (DEP_CRT_*, DEP_OSSL_* environment variables).

This PR does not consolidate all of SVSM's crypto usage under
cocoon-tpm-crypto; that is left as follow-up work.

The vtpm and openssl features do not work for host-side targets like tests
and doc tests, since they pull in the target-integration crate for OpenSSL
which is specific to the bare-metal environment of SVSM.

Commit Outline

The first two commits refactor the random number generator pool in the crypto crate,
making it more generic and easier to add other RNG implementations later.

kernel: utils: add static array backed object pool
kernel: crypto: refactor RNG pool

The following commit is only temporary and shall be dropped before this PR is merged,
which requires that the cocoon-tpm PR is merged and a new release is published on crates.io first.

workspace: override cocoon-tpm source to use osteffen's fork (drop)

Debug improvement, possible with the cocoon-tpm PR.

kernel: print cocoon-tpm crypto backend name

libcrt refactoring: Pull the C runtime out from libtcgtpm into its own
top-level crate, allowing multiple consumers. It does not contain any Rust code,
just builds the C code and provides a static library to the linker. Include path
and library location are communicated between build.rs scripts via environment variables.

workspace: add libcrt from libtcgtpm as its own crate
libcrt: add missing visibility pop to match push in libcrt.h
libtcgtpm: use libcrt from dedicated crate

The center piece of this PR: Add the target integration library for cocoon-tpm OpenSSL backend.
Provides OpenSSL build config and libcrt (by depending on the libcrt crate).
Bisect-ability between these three commits is broken: The RNG implementation is needed to build with
the openssl feature enabled (fixed by the second one) and the openssl feature conflicts with the vtpm
feature due to duplicate linkage against libcrypto.a instances. This is resolved by the last commit that
drops the openssl copy from libtcgtpm and uses the cocoon-tpm provided one instead.

workspace: add cocoon-tpm OpenSSL backend integration lib
kernel: crypto: add RNG implementation using OpenSSL
libtcgtpm: use libcrypto from cocoon-tpm-ossl-bare-sys

osteffenrh · 2026-09-04T10:43:14Z

cc: @nicstange

osteffenrh · 2026-09-21T09:12:17Z

cocoon-tpm-ossl-bare-sys-target-integration: Change edition from 2021 to 2024 as done on the cocoon-tpm side (see coconut-svsm/cocoon-tpm#30 (comment))

osteffenrh · 2026-09-21T15:40:58Z

Applied suggestion to use ManuallyDrop for the object pool (@00xc)
Amended commits with Cargo.lock changes
Rebased to latest main

luigix25

make test is failing on my machine

several errors all similar to each other:

error[E0080]: evaluation of constant value failed
     --> /home/lleonard/repos/svsm/target/debug/build/cocoon-tpm-ossl-bare-sys-a17970f9705b7cdb/out/wrapper.rs:12761:30
      |
12761 |     ["Size of err_state_st"][::core::mem::size_of::<err_state_st>() - 904usize];
      |                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ attempt to compute `1_usize - 904_usize`, which would overflow

osteffenrh · 2026-09-29T13:11:34Z

Updated the cocoon-tpm revs to match the cocoon-tpm PR.

osteffenrh · 2026-09-29T13:47:38Z

make test is failing on my machine

several errors all similar to each other:

error[E0080]: evaluation of constant value failed
     --> /home/lleonard/repos/svsm/target/debug/build/cocoon-tpm-ossl-bare-sys-a17970f9705b7cdb/out/wrapper.rs:12761:30
      |
12761 |     ["Size of err_state_st"][::core::mem::size_of::<err_state_st>() - 904usize];
      |                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ attempt to compute `1_usize - 904_usize`, which would overflow

Hm. Fedora 44 issue. Investigating.

osteffenrh · 2026-10-09T09:23:56Z

v3:

Match the v4 of cocoon-tpm PR
cross-compile fixes
Move OpenSSL algorithm selection config t ococoon-tpm, keep target environment specific ones

luigix25 · 2026-10-09T09:33:31Z

@osteffenrh what about moving the bindgen bump to the first commit? so bisectability is fine.

osteffenrh · 2026-10-09T10:22:46Z

NOTE: There is an issue with conflicting dependencies right now: bindgen seems to pull in 2 versions of syn (2 and 3). I am working around that in the last commit by pinning.
This commit needs to get dropped before merging.
We need to change/drop an earlier commit too (cocoon-tpm source override), so commit history will change anyway.

Add a simple object pool implementation usable for static pools. Signed-off-by: Oliver Steffen <osteffen@redhat.com>

Refactor the random number generator pool using the new object pool implementation from "utils". Move the DRBG generator implementation to its own submodule to further abstract it. We will gain another RNG implementation in a following commit. Signed-off-by: Oliver Steffen <osteffen@redhat.com>

osteffenrh · 2026-10-15T11:21:58Z

Cocoon-tpm PR adding the OpenSSL backend was merged.
Here we consume cocoon-tpm now directly from the current tip of the main branch, instead of using my fork. Cocoon-tpm changes are not on crates.io yet.

v4:

Rebased to main
Use cocoon-tpm main repo branch insted of fork

luigix25 · 2026-10-15T11:44:55Z

@osteffenrh I think last commit should we dropped

osteffenrh · 2026-10-15T11:57:19Z

@osteffenrh I think last commit should we dropped

It is still needed. I need to figure out how to fix properly.

Use the same version of bindgen as we do in cocoon-tpm (where 0.73.2 is needed). Pin prettyplease to 0.2.29 in Cargo.lock. bindgen 0.73.2 accepts prettyplease >=0.2.7, <0.4 but uses syn 2.x internally. If Cargo resolves prettyplease to 0.3.0 (which depends on syn 3.x), the build fails due to conflicting syn versions. Signed-off-by: Oliver Steffen <osteffen@redhat.com>

Use the latest releases of the cocoon-tpm crates from crates.io. This includes the new OpenSSL backend for cocoon-tpm-crypto, which we will use in a following commit. Signed-off-by: Oliver Steffen <osteffen@redhat.com>

Print the name of the compile-time selected backend of cocoon-tpm-crypto to the log. Signed-off-by: Oliver Steffen <osteffen@redhat.com>

Copy the libcrt C library used to integrate OpenSSL from the libtcgtpm crate to its own. It will gain an additional consumer in the future and thus should live in a common place. The new libcrt crate has no Rust code. It just builds libcrt and provides it at link time. libtcgtpm will depend on the new crate and drop its copy of the C code in a following commit. The C code is only compiled for bare-metal targets, to allow regular host-side tests to compile against system libc. Signed-off-by: Oliver Steffen <osteffen@redhat.com>

libcrt.h opens with `#pragma GCC visibility push(hidden)` when __pie__ is defined, but never pops it. This leaks hidden visibility into every translation unit that transitively includes any libcrt header (stddef.h, stdio.h, stdlib.h, etc. all include libcrt.h). For the SVSM kernel built with gcc -nostdinc this is harmless because gcc does not default to PIE and the SVSM link step uses explicit symbol visibility anyway. However, when a host-side tool such as bindgen uses libclang to parse OpenSSL headers that pull in libcrt's stdio.h, the missing pop causes all subsequent function declarations (BN_*, EC_*, EVP_*, ...) to inherit hidden visibility. On hosts where clang defaults to PIE (Ubuntu, Debian), __pie__ is defined and the pragma fires — bindgen's enable_function_attribute_detection() then silently drops the hidden functions, producing only 15 out of ~1873 expected bindings. Hosts without PIE defaults (Fedora, RHEL) are unaffected, hiding the bug. Add the matching `#pragma GCC visibility pop` before the include guard close. Signed-off-by: Oliver Steffen <osteffen@redhat.com>

The Makefile used gcc as the compiler (hard-coded). For cross-builds, this needs to be prefixed with the target architecture. Signed-off-by: Oliver Steffen <osteffen@redhat.com>

Use the libcrt crate instead of building libcrt locally. The libcrt crate provides include and lib paths via cargo metadata (DEP_CRT_*), and handles linking at build time. Signed-off-by: Oliver Steffen <osteffen@redhat.com>

Add an implementation of cocoon-tpm's OpenSSL backend integration package. It provides an OpenSSL build config (based on the libtcgtpm one), matching compiler flags to the cocoon-tpm OpenSSL backend. It depends on our libcrt crate providing a C runtime for OpenSSL. A cargo feature called "openssl" is added to select OpenSSL as the backend in cocoon-tpm. Note: After this commit the openssl and vtpm cargo feature conflict, due to duplicate libcrypto symbols at link time. This will be resolved in a later commit. Configuration options specific to the target environment are passed to the OpenSSL backend of cocoon-tpm; this includes flags for cross-compilation. The set of enabled/disabled algorithms is selected on the cocoon-tpm side and satisfies our needs. Note: The new openssl feature does not build yet, we need to implement a matching RNG in kernel/crypto. This will happen in the following commit. Signed-off-by: Oliver Steffen <osteffen@redhat.com>

The DRBG generator from cocoon-tpm crypto are not available when the OpenSSL backend is selected. Add a new RNG implementation which uses OsslRandBytes() next to the DRBG one and switch between them at build time depending on the "openssl" cargo feature. Signed-off-by: Oliver Steffen <osteffen@redhat.com>

Stop building OpenSSL locally in the libtcgtpm Makefile. Instead, depend on cocoon-tpm-ossl-bare-sys which provides libcrypto.a and exports include/lib paths via cargo metadata (DEP_OSSL_*). The Makefile now expects OPENSSL_INCLUDE_DIR and OPENSSL_SRC_INCLUDE_DIR from the caller and only builds the tcgtpm libraries. Remove the OpenSSL submodule and build config. After this commit libtcgtpm can be used again. It now depends on OpenSSL via cocoon-tpm-crypto. The vtpm cargo feature enables the openssl feature automatically. The vtpm can not be built without OpenSSL. Signed-off-by: Oliver Steffen <osteffen@redhat.com>

Move cross-compile detection to build.rs and use cargo's HOST settings instead of handling it in the makefile (calls uname). Signed-off-by: Oliver Steffen <osteffen@redhat.com>

osteffenrh · 2026-10-16T10:43:04Z

v5:

Use cocoon-tpm crates from crates.io
Pin prettyplease version in Cargo.toml as a workaround for upstream bindgen bug
(cleans up a "DROP" commit)

osteffenrh force-pushed the cocoon-tpm-openssl branch 13 times, most recently from a46ac80 to 7bc6315 Compare September 18, 2026 16:00

osteffenrh marked this pull request as ready for review September 18, 2026 16:34

osteffenrh requested review from 00xc, joergroedel, msft-jlange, peterfang and stefano-garzarella as code owners September 18, 2026 16:34

osteffenrh force-pushed the cocoon-tpm-openssl branch from 7bc6315 to f5bdfc4 Compare September 21, 2026 09:12

00xc reviewed Sep 21, 2026

View reviewed changes

Comment thread kernel/src/utils/pool.rs

osteffenrh force-pushed the cocoon-tpm-openssl branch from f5bdfc4 to 88edb71 Compare September 21, 2026 15:38

stefano-garzarella self-assigned this Sep 22, 2026

luigix25 reviewed Sep 28, 2026

View reviewed changes

Comment thread Makefile

Comment thread libtcgtpm/Makefile

Comment thread cocoon-tpm_ossl-bare-sys-target-integration/Cargo.toml

Comment thread cocoon-tpm_ossl-bare-sys-target-integration/Cargo.toml

Comment thread cocoon-tpm_ossl-bare-sys-target-integration/src/lib.rs

osteffenrh force-pushed the cocoon-tpm-openssl branch from 88edb71 to 3d81c40 Compare September 29, 2026 13:09

osteffenrh force-pushed the cocoon-tpm-openssl branch from 8f2d202 to 9b46477 Compare October 9, 2026 09:21

osteffenrh requested review from 00xc and luigix25 October 9, 2026 09:25

osteffenrh force-pushed the cocoon-tpm-openssl branch from 9b46477 to 8843058 Compare October 9, 2026 09:27

osteffenrh force-pushed the cocoon-tpm-openssl branch from 8843058 to e73a699 Compare October 9, 2026 10:20

osteffenrh force-pushed the cocoon-tpm-openssl branch 2 times, most recently from e44afc1 to 6cfc814 Compare October 9, 2026 10:27

stefano-garzarella added the needs-rebase The PR needs to be rebased to the latest upstream branch label Oct 14, 2026

osteffenrh added 2 commits October 15, 2026 12:51

kernel: utils: add static array backed object pool

c80ca6b

Add a simple object pool implementation usable for static pools. Signed-off-by: Oliver Steffen <osteffen@redhat.com>

osteffenrh force-pushed the cocoon-tpm-openssl branch from 6cfc814 to 2e84834 Compare October 15, 2026 11:18

osteffenrh added 11 commits October 16, 2026 12:24

workspace: Bump cocoon-tpm crates to latest version

0fe11bf

Use the latest releases of the cocoon-tpm crates from crates.io. This includes the new OpenSSL backend for cocoon-tpm-crypto, which we will use in a following commit. Signed-off-by: Oliver Steffen <osteffen@redhat.com>

kernel: print cocoon-tpm crypto backend name

d270492

Print the name of the compile-time selected backend of cocoon-tpm-crypto to the log. Signed-off-by: Oliver Steffen <osteffen@redhat.com>

libcrt: Auto-detect CC cross-build prefix

eb1315a

The Makefile used gcc as the compiler (hard-coded). For cross-builds, this needs to be prefixed with the target architecture. Signed-off-by: Oliver Steffen <osteffen@redhat.com>

libtcgtpm: use libcrt from dedicated crate

c2069d0

Use the libcrt crate instead of building libcrt locally. The libcrt crate provides include and lib paths via cargo metadata (DEP_CRT_*), and handles linking at build time. Signed-off-by: Oliver Steffen <osteffen@redhat.com>

libtcgtpm: Move cross-compile detection into build.rs

79c1cd0

Move cross-compile detection to build.rs and use cargo's HOST settings instead of handling it in the makefile (calls uname). Signed-off-by: Oliver Steffen <osteffen@redhat.com>

osteffenrh force-pushed the cocoon-tpm-openssl branch from 2e84834 to 79c1cd0 Compare October 16, 2026 10:40

Conversation

osteffenrh commented Sep 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Commit Outline

Uh oh!

osteffenrh commented Sep 4, 2026

Uh oh!

osteffenrh commented Sep 21, 2026

Uh oh!

Uh oh!

osteffenrh commented Sep 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

luigix25 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

osteffenrh commented Sep 29, 2026

Uh oh!

osteffenrh commented Sep 29, 2026

Uh oh!

osteffenrh commented Oct 9, 2026

Uh oh!

luigix25 commented Oct 9, 2026

Uh oh!

osteffenrh commented Oct 9, 2026

Uh oh!

osteffenrh commented Oct 15, 2026

Uh oh!

luigix25 commented Oct 15, 2026

Uh oh!

osteffenrh commented Oct 15, 2026

Uh oh!

osteffenrh commented Oct 16, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

osteffenrh commented Sep 4, 2026 •

edited

Loading

osteffenrh commented Sep 21, 2026 •

edited

Loading