Link to or incorporate C dependencies?

[MajorDallas]

I'm trying to deploy a .pex with code that uses the python-ldap library, which links against the OpenLDAP C library. This library is installed both on my machine (where the .pex is built) and the target machine, but the two machines are running different versions of Ubuntu (24.04 and 22.04 LTS respectively) and the binary that ldap wants to use has a different name in the two environments. When I try to run it, I get the error

ImportError: libldap.so.2: cannot open shared object file: No such file or directory

The binary on the target machine is libldap.so (without the .2), but on my machine that is the correct binary. My working hypothesis is that python-ldap's internal _ldap module is being built against that binary by uv, and then the built module is being packaged into the .pex. This seems to be supported by the fact that my prod environment, which is not yet using PEX, seems to be linking against libldap.so.

This is not the first time I've run into issues building and deploying python-ldap specifically and I'm about to rewrite with ldap3 over it, but I have other C-linked libraries without pure-Python alternatives like psycopg, as well, and I worry it's only a matter of time before one of them doesn't provide prebuilt binaries and causes a similar problem. The docs and -h option aren't making it obvious if/how I can roll C binaries into the .pex or at least override the location(s) that modules should look for them.

I thought --scie might be the answer, but it seems that just includes the CPython interpreter. I also see -D, but the description only suggests it includes files in the .pex but is silent on whether those files would be extracted somewhere as with setuptools' data-files option. (If it is completely analogous to data-files, then this is probably the answer I'm looking for!)

While I'm open to workarounds for OpenLDAP/python-ldap specifically, my actual question is:

In general, how can I ensure that C libraries linked to by Python modules can be either included in the .pex archive itself or made discoverable to such modules, or delay building of the Python C module to after the .pex is deployed so it can link to that environment?

[jsirois]

In general, how can I ensure that C libraries linked to by Python modules can be either included in the .pex archive itself or made discoverable to such modules, or delay building of the Python C module to after the .pex is deployed so it can link to that environment?

In short, you can't via Pex now and likely forevermore. Pex supports Python packaging standards such as they are and that's enough work to keep up with. Supporting native, non-wheeled deps is not planned.

I also see -D, but the description only suggests it includes files in the .pex but is silent on whether those files would be extracted somewhere as with setuptools' data-files option. (If it is completely analogous to data-files, then this is probably the answer I'm looking for!)

It is not analogous. That said, just use setuptools to package your code and point Pex at your project directory (as a requirement) instead of specifying sources manually.

Here Pex dogsfoods that idea packaging itself in a PEX by using . (the Pex project directory), as a requirement. That means Pex uses the build-system found for . to package Pex into a PEX. It does not use -D, etc.:

pex/scripts/create-packages.py

Lines 33 to 127 in 5263d73

	def build_pex_pex(
	output_file: PurePath, verbosity: int = 0, env: Optional[Dict[str, str]] = None
	) -> PurePath:
	# NB: We do not include the subprocess extra (which would be spelled: `.[subprocess]`) since we
	# would then produce a pex that would not be consumable by all python interpreters otherwise
	# meeting `python_requires`; ie: we'd need to then come up with a deploy environment / deploy
	# tooling, that built subprocess32 for linux cp27m, cp27mu, pypy, ... etc. Even with all the
	# work expended to do this, we'd still miss some platform someone wanted to run the Pex PEX on.
	# As such, we just ship unadorned Pex which is pure-python and universal. Any user wanting the
	# extra is encouraged to build a Pex PEX for their particular platform themselves.
	pex_requirement = "."
	args = [
	sys.executable,
	"-m",
	"pex",
	*["-v" for _ in range(verbosity)],
	"--disable-cache",
	"--no-build",
	"--no-compile",
	"--no-use-system-time",
	"--python-shebang",
	"/usr/bin/env python",
	"--no-strip-pex-env",
	"--include-tools",
	"--record-git-state",
	"--pip-version",
	"latest-compatible",
	"-o",
	str(output_file),
	"-c",
	"pex",
	pex_requirement,
	]
	subprocess.run(args=args, env=env, check=True)
	return output_file
	def build_pex_scie(
	platform_info: tuple[PlatformConfig, Path],
	*,
	lock: Path,
	pex_requirement: str,
	verbosity: int = 0,
	env: Optional[Dict[str, str]] = None,
	) -> SpawnedJob[tuple[str, PlatformConfig]]:
	platform_config, complete_platform = platform_info
	command = [
	sys.executable,
	"-m",
	"pex",
	*["-v" for _ in range(verbosity)],
	"--disable-cache",
	"--no-build",
	"--no-compile",
	"--no-use-system-time",
	"--record-git-state",
	"--venv",
	"--no-strip-pex-env",
	"--complete-platform",
	str(complete_platform),
	"--lock",
	str(lock),
	"--scie",
	"eager",
	"--scie-only",
	"--scie-name-style",
	"platform-file-suffix",
	"--scie-platform",
	platform_config.name,
	"--scie-pbs-release",
	platform_config.pbs_release,
	"--scie-python-version",
	platform_config.python_version,
	"--scie-pbs-stripped",
	"--scie-hash-alg",
	"sha256",
	"--scie-busybox",
	"@pex",
	"--scie-busybox-pex-entrypoint-env-passthrough",
	"--pip-version",
	PipVersion.LATEST_COMPATIBLE.value,
	"-o",
	os.path.join(safe_mkdtemp(), "pex"),
	"-c",
	"pex",
	"--project",
	pex_requirement,
	]
	return SpawnedJob.stdout(
	job=Job(
	command=command, process=subprocess.Popen(args=command, env=env, stdout=subprocess.PIPE)
	),
	result_func=lambda stdout: (stdout.decode("utf-8").strip(), platform_config),
	)

[MajorDallas]

At the moment I'm pointing pex at a wheel built by uv. Wouldn't pointing it at . run into the same problem? If it invokes uv, uv will build (or probably re-use) .venv/lib64/python3.13/site-packages/_ldap.cpython-313-x86_64-linux-gnu.so, which is linked against libldap.so.2.

[jsirois]

Yes, uv will run into the same problem. You mentioned data_files, which is a setuptools thing; so I was responding to that. Quite simply, there is no magic here at all in the Python ecosystem I'm aware of. If you cannot fully package your complete dependency set (including native libraries like libldap) in wheels, which do support identifying glibc version compatibility, you're out of luck using Python ecosystem tools.