How to integrate into a setuptools package?

[patrislav1]

I'm using the setuptools method to build/install my pybind11-based module. I'd like to integrate pybind11-stubgen into the setup.py, so it creates and installs the .pyi after the pybind11-based module is built. I tried to set up a post-install function using the setuptools.command.install override (cmdclass={'install': new_install}). But at this point, the wheel is built but the module is not installed yet - so when I run pybind11-stubgen, it will not find the module to annotate.

I don't know the Python packaging internals well enough, but this doesn't look like a very exotic problem to me, so maybe someone has already figured it out?

[sizmailov]

Hi, thanks for the good question!

I think the solution would be to customize (subclass) Pybind11Extension. After the module is built, we can set env variables to make just built .so/.dll discoverable by the interpreter and generate the stubs. Unfortunately, I don't have a working example at hand.

If you found a solution, please share.

[patrislav1]

Thank you for the suggestion. In the meantime I resorted to calling pybind11-stubgen manually, when the API of my module changes (which isn't too often right now), and checking in the stubs together with the other code. When that ever becomes too annoying I'll look into the subclassing solution.

[sizmailov]

In one of my projects, I use the following approach. First, I install the package as is, without stubs. Then I use pybind11-stubgen to place the stubs in the right location and assemble the distribution packages with stubs for PyPI.

Note that you have to explicitly list stubs in setup.py:

...

def find_stubs(path: Path):
    return [str(pyi.relative_to(path)) for pyi in path.rglob("*.pyi")]

package_name = ... # name of your package

setup(
    name=package_name,
    ...,
    # Add `py.typed` and `*.pyi` files
    package_data={package_name: ["py.typed", *find_stubs(path=Path(package_name))]},
)

Example of package-distributions.sh script:

#!/bin/bash

# Show commands
set -x
# Exit on the first error
set -e

MODULE_NAME="..." # e.g. `my_package._core`
# Could be either name of your package (same as in `setup.py`)
# or its binary part only (e.g. `my_package._core`)
# if you don't want to duplicate pure python parts of your library in `*.pyi` files

# Replace '.' with '/'
MODULE_PATH=$(echo "${MODULE_NAME}" | sed 's/\./\//g' -)

# Install the package from sources
pip install .

# Make sure required tools are installed
pip install black isort pybind11-stubgen

# Generate stubs and apply formatting
pybind11-stubgen "${MODULE_NAME}" -o ./ --numpy-array-wrap-with-annotated --exit-code
black "${MODULE_PATH}"
isort --profile=black "${MODULE_PATH}"

# Assemble final packages
python setup.py sdist bdist_wheel

[ColorsWind]

Currently, the limitation of pybind11-stubgen is that the package must be importable using import to generate the stubs. We can utilize PYTHONPATH Mechanism to achieve "executing import without following the package structure." Below is a part of my project configuration:

setup.py

class PostInstallCommand:

    def run(self, install_lib):
        self.py_dir = os.path.abspath(os.path.join(os.path.dirname(__file__), '..', 'mpool'))
        self.build_dir = os.path.abspath(os.path.join(os.path.dirname(__file__), '..', 'build'))
        self.target_dir = os.path.join(install_lib, 'mpool')
        self._install_lib = install_lib
        self.copy_python_file()
        self.copy_shared_library()
        self.generate_stub()
    
    def generate_stub(self):
        env = os.environ.copy()
        if 'PYTHONPATH' not in env:
            env['PYTHONPATH'] = os.path.abspath(self._install_lib)
        else:
            env['PYTHONPATH'] += ':' + os.path.abspath(self._install_lib)
        subprocess.check_call(['pybind11-stubgen', 'mpool', '-o', self._install_lib, '--ignore-all-errors'], env=env)