@@ -345,7 +345,7 @@ Generating small synthetic datasets to try it on:
345345
346346 # JoinByTimestamp is a local plugin transformer (see Plugins below) - needed both to
347347 # build the pipeline above and to load it back for damast process
348- export DAMAST_PLUGIN_PATH=docs/examples/plugins
348+ export DAMAST_PLUGIN_PATH=ais_osint= docs/examples/plugins
349349 damast process --pipeline pipelines/osint_ais_preparation.damast.ppl \
350350 --input-data df=docs/examples/data/ais.parquet \
351351 --input-data osint_events=docs/examples/data/osint.parquet \
@@ -365,7 +365,7 @@ resolvable, i.e., ``DAMAST_PLUGIN_PATH`` must be set for a local plugin, while i
365365
366366::
367367
368- export DAMAST_PLUGIN_PATH=docs/examples/plugins
368+ export DAMAST_PLUGIN_PATH=ais_osint= docs/examples/plugins
369369 damast process --pipeline pipelines/osint_ais_preparation.damast.ppl --describe
370370
371371.. highlight :: none
@@ -624,20 +624,35 @@ registering such plugin transformers are supported - see :class:`damast.core.tra
624624for the full API:
625625
626626- installable packages that declare their :class: `damast.core.transformations.PipelineElement ` subclasses via the
627- ``damast.transformers `` entry-point group in their own ``pyproject.toml ``::
627+ ``damast.transformers `` entry-point group in their own ``pyproject.toml `` - either one entry per class, or
628+ one entry per module, which registers every transformer defined in that module (or, for a package, in its
629+ top-level submodules)::
628630
629631 [project.entry-points."damast.transformers"]
630632 MyTransformer = "acme_pkg.transformers:MyTransformer"
633+ acme_pkg = "acme_pkg.transformers"
631634
632- - local, ad-hoc ``*.py `` files that are not part of any installed package, made discoverable by
633- pointing the ``DAMAST_PLUGIN_PATH `` environment variable at the directory (or directories,
634- separated with ``os.pathsep ``) that contains them
635+ - local directories that are not part of any installed package, made discoverable via the
636+ ``DAMAST_PLUGIN_PATH `` environment variable (several entries separated with ``os.pathsep ``). An entry
637+ ``name=path `` loads the directory as a package called ``name ``: its top-level ``*.py `` files become
638+ ``name.<file> ``, and may use relative imports (``from .helpers import x ``), also into subdirectories.
639+ The same can be done in code via ``plugin_manager.register_plugin_package(name, path) ``.
640+ Choose a distinctive name - one that is already importable is rejected. A bare ``path `` entry
641+ (deprecated) loads each file as a flat module named after the file, without relative imports.
642+
643+ The package name of a local directory is recorded in saved pipelines (e.g.
644+ ``module_name: my_plugins.my_transformers ``), so a pipeline can only be replayed with the directory
645+ registered under the same name - or with an installed package of that name, e.g. once the directory has
646+ been turned into an installable package with a module entry-point.
635647
636648Regardless of which of the two a transformer comes from, it is resolvable in code the same way,
637- via the ``damast.plugins `` namespace::
649+ via ``damast.plugins.<package> `` - where ``<package> `` is the top-level package of the module
650+ defining it, i.e. ``acme_pkg `` for ``acme_pkg.transformers:MyTransformer ``, or the name a local
651+ directory was registered under::
638652
639- from damast.plugins import MyTransformer
653+ from damast.plugins.acme_pkg import MyTransformer
640654
655+ Scoping by package means two plugins can provide a transformer of the same name without clashing.
641656``damast.plugins `` resolves names lazily on first access, so nothing beyond the requested class
642657is ever imported - see :mod: `damast.plugins ` for details.
643658
@@ -650,27 +665,29 @@ subclass in a loose ``*.py`` file, written like any other transformer:
650665.. literalinclude :: ./examples/plugins/my_transformers.py
651666 :language: Python
652667
653- With ``DAMAST_PLUGIN_PATH `` pointing at the directory containing that file, `` damast plugins ``
654- lists it without requiring any further Python code:
668+ With ``DAMAST_PLUGIN_PATH `` registering the directory containing that file (here as package
669+ `` my_plugins ``), `` damast plugins `` lists it without requiring any further Python code:
655670
656671.. highlight :: none
657672
658673::
659674
660- $ export DAMAST_PLUGIN_PATH=./examples/plugins
675+ $ export DAMAST_PLUGIN_PATH=my_plugins= ./examples/plugins
661676 $ damast plugins
662677
663- MyTripler: my_transformers:MyTripler
678+ my_plugins (local: examples/plugins)
679+ JoinByTimestamp .osint_ais_transformers
680+ MyTripler .my_transformers
664681
665- ``MyTripler `` is now resolvable via ``damast.plugins `` and can be used in a pipeline like any
666- other transformer:
682+ ``MyTripler `` is now resolvable via ``damast.plugins.my_plugins `` and can be used in a pipeline
683+ like any other transformer:
667684
668685.. literalinclude :: ./examples/damast-plugin-pipeline.py
669686 :language: Python
670687
671688The resulting pipeline can be applied like any other, e.g. via ``damast process `` (see `Process `_
672- above), as long as ``DAMAST_PLUGIN_PATH `` is still set to a directory containing
673- ``my_transformers.py ``:
689+ above), as long as ``DAMAST_PLUGIN_PATH `` still registers the directory containing
690+ ``my_transformers.py `` as `` my_plugins `` :
674691
675692.. highlight :: none
676693
@@ -679,7 +696,7 @@ above), as long as ``DAMAST_PLUGIN_PATH`` is still set to a directory containing
679696 damast process --input-data data.parquet --pipeline pipelines/my-plugin-pipeline.damast.ppl
680697
681698Pipelines saved with a plugin transformer record where it came from under ``requires `` (the
682- installed distribution and version, or the original local file path ), so that loading the pipeline
683- elsewhere fails with an actionable message - naming the missing package to ``pip install ``, or the
684- ``DAMAST_PLUGIN_PATH `` directory to add - instead of a bare import error.
699+ installed distribution and version, or the local package name and directory ), so that loading the
700+ pipeline elsewhere fails with an actionable message - naming the missing package to ``pip install ``, or
701+ the ``DAMAST_PLUGIN_PATH `` entry to add - instead of a bare import error.
685702
0 commit comments