Skip to content

Language extension registry (upstream proposal) #3764

Open
Open
Language extension registry (upstream proposal)#3764

Description

@p4ndr
p4ndr
opened on Sep 22, 2026

Language extension registry (upstream proposal)

Issue reference: This proposal addresses the feature request in #3180 ("Pluggable extractor packaging (entry-point discovery) for third-party languages") and #1070 ("Extractor plugin API + CLI command group consolidation"), specifically the "Extractor plugin API" section of #1070.

Problem

ARCHITECTURE.md's "Adding a new language extractor" walkthrough is clear, but every step lands inside the core repo: a new module under graphify/extractors/, a dispatch-table registration in extract.py, a suffix addition in detect.py and watch.py, plus a new tree-sitter dependency in pyproject.toml. The issue backlog already shows the cost of that: GDScript (#2152), Haxe (#1424), MATLAB (#2935), 1C/BSL (#2127) and others are all separate, unclaimed feature requests, each waiting on someone to carry a full core-repo change through review — and each new language grows the core dependency surface for everyone, whether they use it or not.

Proposal

Let a language extractor live in its own installable package and register itself via a Python entry point (e.g. graphify.extractors), the same pattern pytest plugins or Flake8 checks use. At startup, extract.py's dispatch table would discover installed extractor packages via importlib.metadata.entry_points, alongside (not replacing) the built-in dispatch table for the languages graphify ships today.

Each third-party package would own: its own tree-sitter grammar dependency, its own extract_<lang>(path) -> dict function following the existing schema, and its own suffix registration — all outside the core repo's dependency tree.

Why this fits now

It turns "someone requests a niche language, a maintainer has to review and carry a core-repo PR" into "someone ships and maintains their own package, users pip install it if they need it." It doesn't compete with keeping popular languages built-in — it's specifically for the long tail already visible in the open feature requests.

Implementation status

A reference implementation exists in the fork Graphify-Labs/graphify-lang on the lang-registry branch. The artefact git diff v8...lang-registry -- graphify/ is a minimal, reviewable set of changes that an upstream maintainer can read and apply:

diff --git a/graphify/cli.py b/graphify/cli.py
index efede012..05ac0ddb 100644
--- a/graphify/cli.py
+++ b/graphify/cli.py
@@ -73,6 +73,14 @@ _HOOK_SOURCE_EXTS = (
     '.rs', '.java', '.rb', '.c', '.h', '.cpp', '.hpp', '.cc', '.cs', '.kt',
     '.swift', '.php', '.scala', '.lua', '.sh', '.md', '.rst', '.txt', '.mdx',
 )
+try:
+    import graphify.lang_registry
+    graphify.lang_registry.apply_registry()
+    # Merge registry suffixes into HOOK_SOURCE_EXTS
+    for suffix in graphify.lang_registry.get_registry_suffixes():
+        _HOOK_SOURCE_EXTS += (suffix,)
+except Exception:
+    pass
 _GEMINI_NUDGE_TEXT = (
     'graphify: knowledge graph at graphify-out/. For focused questions, run '
     '`graphify query "<question>"` (scoped subgraph, usually much smaller than '
diff --git a/graphify/detect.py b/graphify/detect.py
index 1adad00b..c5399f1d 100644
--- a/graphify/detect.py
+++ b/graphify/detect.py
@@ -41,7 +41,6 @@ _MANIFEST_PATH = str(out_path("manifest.json"))
 _MTIME_COARSE_S = 2.0
 _MTIME_SUBSECOND_S = 0.05
 
-CODE_EXTENSIONS = {'.py', '.ts', '.tsx', '.mts', '.cts', '.js', '.jsx', '.mjs', '.cjs', '.ejs', '.ets', '.go', '.rs', '.java', '.groovy', '.gradle', '.cpp', '.cc', '.cxx', '.c', '.h', '.hpp', '.cu', '.cuh', '.metal', '.rb', '.rake', '.swift', '.kt', '.kts', '.cs', '.scala', '.php', '.lua', '.luau', '.toc', '.zig', '.ps1', '.psm1', '.psd1', '.ex', '.exs', '.m', '.mm', '.ml', '.mli', '.jl', '.vue', '.svelte', '.astro', '.dart', '.v', '.sv', '.svh', '.sql', '.r', '.f', '.F', '.f90', '.F90', '.f95', '.F95', '.f03', '.F03', '.f08', '.F08', '.pas', '.pp', '.dpr', '.dpk', '.lpr', '.inc', '.dfm', '.lfm', '.lpk', '.sh', '.bash', '.json', '.tf', '.tfvars', '.hcl', '.dm', '.dme', '.dmi', '.dmm', '.dmf', '.sln', '.slnx', '.csproj', '.fsproj', '.vbproj', '.xaml', '.r'
+CODE_EXTENSIONS = {'.py', '.ts', '.tsx', '.mts', '.cts', '.js', '.jsx', '.mjs', '.cjs', '.ejs', '.ets', '.go', '.rs', '.java', '.groovy', '.gradle', '.cpp', '.cc', '.cxx', '.c', '.h', '.hpp', '.cu', '.cuh', '.metal', '.rb', '.rake', '.swift', '.kt', '.kts', '.cs', '.scala', '.php', '.lua', '.luau', '.toc', '.zig', '.ps1', '.psm1', '.psd1', '.ex', '.exs', '.m', '.mm', '.ml', '.mli', '.jl', '.vue', '.svelte', '.astro', '.dart', '.v', '.sv', '.svh', '.sql', '.r', '.f', '.F', '.f90', '.F90', '.f95', '.F95', '.f03', '.F03', '.f08', '.F08', '.pas', '.pp', '.dpr', '.dpk', '.lpr', '.inc', '.dfm', '.lfm', '.lpk', '.sh', '.bash', '.json', '.tf', '.tfvars', '.hcl', '.dm', '.dme', '.dmi', '.dmm', '.dmf', '.sln', '.slnx', '.csproj', '.fsproj', '.vbproj', '.xaml', '.r'
+try:
+    import graphify.lang_registry
+    graphify.lang_registry.apply_registry()
+    CODE_EXTENSIONS.update(graphify.lang_registry.get_registry_suffixes())
+except Exception:
+    pass
+
 # Resource caps for parsing untrusted office/PDF files (F2). A corpus is
 # attacker-controllable (graphify runs on cloned/shared folders), and .docx/.xlsx
 # are zip+XML containers: a few-KB zip-bomb can decompress to gigabytes and
diff --git a/graphify/extract.py b/graphify/extract.py
index 7cc9e62c..3ee6280b 100644
--- a/graphify/extract.py
+++ b/graphify/extract.py
@@ -1,3 +1,4 @@
+# @sidecar extract.md
 """Deterministic structural extraction from source code using tree-sitter. Outputs nodes+edges dicts."""
 from __future__ import annotations
 
@@ -156,11 +157,7 @@ from graphify.extractors.julia import extract_julia  # noqa: E402,F401
 
 _RECURSION_LIMIT = 10_000
 
-# Language built-in globals that AST may classify as call targets when used as
-# constructors or coercion functions (e.g. String(x), Number(x), Boolean(x)).
-# Without this filter they become god-nodes accumulating spurious edges from
-# every call site. Filter applied at same-file and cross-file resolution.
-# See issue #726.
+# @doc extract.md#C0001
 
 
 def _raise_recursion_limit() -> None:
@@ -225,8 +222,7 @@ def _repoint_python_package_imports(paths, all_nodes, all_edges, root) -> None:
             continue  # top-level file: scan-root-relative id already matches
         d = Path(p).resolve().parent
         levels = 0
-        # Bounded by the number of dirs between the file and the scan root, so a
-        # pathological `/__init__.py` chain can't loop forever.
+        # @doc extract.md#C0002
         while levels < len(parts) - 1 and (d / "__init__.py").is_file():
             levels += 1
             d = d.parent
@@ -250,10 +246,7 @@ def _repoint_python_package_imports(paths, all_nodes, all_edges, root) -> None:
     if not alias_map:
         return
     for e in all_edges:
-        # Only repoint edges emitted from a Python file: a non-Python import edge
-        # (e.g. C# `using Pkg.Mod;`, Java/Go dotted imports) can have a dangling
-        # target string that coincides with a Python alias, and repointing it
-        # would fabricate a cross-language import edge (#2072 review).
+        # @doc extract.md#C0003
         if (
             isinstance(e, dict)
             and e.get("relation") in ("imports", "imports_from")
@@ -270,9 +263,7 @@ SEMANTIC_RELATIONS = frozenset({
 })
 
 
-# Condition keys consulted when resolving an `exports` target, in priority
-# order. `default` is Node's catch-all and must be consulted LAST so a more
-# specific condition (source/import/module/etc.) wins when several match.
+# @doc extract.md#C0004
 
 
 # ── LanguageConfig dataclass ─────────────────────────────────────────────────
@@ -281,19 +272,10 @@ SEMANTIC_RELATIONS = frozenset({
 # ── Generic helpers ───────────────────────────────────────────────────────────
 
 
-# Scalar builtins and test-mock names that appear as type annotations but carry
-# no useful semantic meaning as graph nodes (#1147). Suppressed at the annotation
-# walker level so they are never created as nodes or emitted as edges.
+# @doc extract.md#C0005
 
 
-# java.lang (auto-imported) plus the ubiquitous java.util / java.io / java.time /
-# java.util.{stream,function,concurrent} / java.math / java.nio.file types that
-# appear as field, parameter, return, and generic-argument annotations. They never
-# resolve to a project node, so emitting `references` edges to them is pure noise
-# (mirrors _GO_PREDECLARED_TYPES / _PYTHON_ANNOTATION_NOISE). Suppressed at the
-# type-ref walker so they are never created as nodes or emitted as edges. The
-# boxed-scalar/`void` primitives are already dropped by grammar node type above;
-# these are the class/interface names the grammar reports as identifiers.
+# @doc extract.md#C0006
 
 
 # ── C / C++ type-ref helpers ─────────────────────────────────────────────────
@@ -338,10 +320,7 @@ def _import_python(node, source: bytes, file_nid: str, stem: str, edges: list, s
                     "weight": 1.0,
                 }
                 if raw_alias:
-                    # `import pkg.mod as alias` binds the local name `alias`, not
-                    # `mod`'s own stem, to the module -- stash it so the cross-file
-                    # member-call resolver can match `alias.func()` against this
-                    # edge instead of dropping it (#2082).
+                    # @doc extract.md#C0007
                     edge["local_alias"] = raw_alias.strip()
                 edges.append(edge)
     elif t == "import_from_statement":
@@ -356,13 +335,7 @@ def _import_python(node, source: bytes, file_nid: str, stem: str, edges: list, s
                 base = Path(str_path).parent
                 for _ in range(dots - 1):
                     base = base.parent
-                # A relative import can name a subpackage (a directory with an
-                # __init__.py), not a module file. Probing the candidate on disk
-                # (mirroring the companion `imports` edge's
-                # _resolve_python_module_path) resolves `graphs` -> graphs/__init__.py
-                # instead of a nonexistent graphs.py: without it the target keeps an
-                # absolute-path-derived slug that the target_file stamp below can't
-                # heal, so it dangles per-checkout (#2455).
+                # @doc extract.md#C0008
                 candidate = base / module_name.replace(".", "/") if module_name else base
                 resolved = _probe_python_module_candidate(candidate)
                 if resolved is not None:
@@ -383,14 +356,7 @@ def _import_python(node, source: bytes, file_nid: str, stem: str, edges: list, s
                 "source_location": f"L{node.start_point[0] + 1}",
                 "weight": 1.0,
             }
-            # Stamp the resolved target file (mirroring _import_js, #1814) so
-            # the #2169 remap pass can canonicalize this edge's target on an
-            # incremental run where the target file itself is not in the
-            # batch — without it the target keeps an absolute-path-derived id
-            # that matches no node in the merged graph and dangles (#2213).
-            # Existence-gated: a speculative import of a nonexistent sibling
-            # must stay dangling, exactly as before. The stamp is transient
-            # and popped before graph.json ships.
+            # @doc extract.md#C0009
             if target_path is not None:
                 try:
                     if target_path.is_file():
@@ -402,8 +368,7 @@ def _import_python(node, source: bytes, file_nid: str, stem: str, edges: list, s
 
 def _import_js(node, source: bytes, file_nid: str, stem: str, edges: list, str_path: str, scope_stack: list[str] | None = None) -> None:
     is_reexport = node.type == "export_statement"
-    # Only handle export_statement if it has a `from` clause (re-export).
-    # Pure exports like `export const x = 1` or `export { localVar }` have no source module.
+    # @doc extract.md#C0010
     if is_reexport:
         has_from = any(child.type == "from" or (_read_text(child, source) == "from") for child in node.children if child.type in ("from", "identifier"))
         if not has_from:
@@ -412,16 +377,7 @@ def _import_js(node, source: bytes, file_nid: str, stem: str, edges: list, str_p
             if not has_from:
                 return
 
-    # `import type {...} from` / `export type {...} from` are erased by the
-    # TypeScript compiler: no runtime emit, no module-graph edge. The
-    # dependency is still real for "what references this type", so the edge is
-    # stamped `type_only` rather than dropped, and Import Cycles excludes it
-    # the way it already excludes deferred `import(...)` (#1241) - on the
-    # reporter's corpus all 3 reported cycles closed only through these
-    # (#3123). The grammar keeps `import type from './x'` (a default binding
-    # NAMED type) distinct: there `type` sits inside the import_clause, not as
-    # a bare keyword child of the statement. A mixed `import { type B, C }`
-    # stays a runtime edge - C is a runtime import.
+    # @doc extract.md#C0011
     is_type_only = any(
         child.type == "type" and not child.is_named for child in node.children
     )
@@ -432,9 +388,7 @@ def _import_js(node, source: bytes, file_nid: str, stem: str, edges: list, str_p
             module_string = child
             break
         if child.type == "import_require_clause":
-            # TS import-equals form: `import x = require("./m")`. The module
-            # string sits inside the clause, not on the import_statement
-            # itself, so the direct-child scan above never sees it.
+            # @doc extract.md#C0012
             module_string = next(
                 (sub for sub in child.children if sub.type == "string"), None
             )
@@ -444,9 +398,7 @@ def _import_js(node, source: bytes, file_nid: str, stem: str, edges: list, str_p
         resolved = _resolve_js_import_target(raw, str_path)
         if resolved is not None:
             tgt_nid, resolved_path = resolved
-            # `_resolve_js_import_path` returns the attempted path when no
-            # local file exists. Static ES imports must treat that as unresolved
-            # rather than minting a checkout-specific target ID (#2457).
+            # @doc extract.md#C0013
             if resolved_path is not None and not resolved_path.is_file():
                 tgt_nid = _make_id("ref", raw)
                 resolved_path = None
@@ -460,29 +412,20 @@ def _import_js(node, source: bytes, file_nid: str, stem: str, edges: list, str_p
                 "source_location": f"L{node.start_point[0] + 1}",
                 "weight": 1.0,
             }
-            # Stamp the resolved target file so a same-basename cross-extension
-            # sibling (foo.ts importing/re-exporting ./foo.mjs) keys its target salt
-            # by the TARGET's file rather than the importer's. Both files collapse to
-            # the base id `foo`; without this the salted lookup mis-points the target
-            # back onto the importer's own variant, a phantom self-loop (#1814).
+            # @doc extract.md#C0014
             if resolved_path is not None:
                 edge["target_file"] = str(resolved_path)
             if is_type_only:
                 edge["type_only"] = True
             edges.append(edge)
 
-    # Emit symbol-level edges for named imports/re-exports from local/aliased files.
-    # e.g. `import { Foo, type Bar } from './bar'` → file → Foo, file → Bar (EXTRACTED)
-    # e.g. `export { Foo } from './bar'` → file → Foo (re_exports edge)
-    # Uses the same _make_id(target_stem, name) key that _extract_generic emits when
-    # defining the symbol, so these edges wire importers directly to existing symbol nodes.
+    # @doc extract.md#C0015
     if resolved_path is not None:
         target_stem = _file_stem(resolved_path)
         line = node.start_point[0] + 1
 
         if is_reexport:
-            # Handle: export { foo, bar } from './module'
-            #         export { default as baz } from './module'
+            # @doc extract.md#C0016

Plus the new graphify/lang_registry.py file (84 lines) and the test suite (tests/test_lang_registry.py, 14 tests).

Key design decisions

  1. Single call site per core module — Each of detect.py, cli.py, and extract.py has exactly one try/except block that imports and calls graphify.lang_registry.apply_registry(). This keeps the core changes minimal and reviewable.

  2. Registry as a separate module — The registry lives in graphify/lang_registry.py, not inside extract.py or another core module. This separation makes the registry optional: if graphify_lang is not installed, the try/except catches the ImportError and the registry is simply inert.

  3. Suffix merging at import time — Registry suffixes are merged into CODE_EXTENSIONS, _HOOK_SOURCE_EXTS, and _DISPATCH at import time, not at runtime. This keeps the runtime lookup fast and predictable.

  4. Case-insensitive suffix matching — The registry registers both lowercase and uppercase variants (.lsp and .LSP) to handle case-insensitive filesystems. A separate fix in resolver_registry.py casefolds suffixes during the resolution gate so that mixed-case file extensions (AutoCAD-style: ERR.LSP, Err.Lsp) match the lowercase resolver suffixes.

  5. Error handling — Every registry call site is wrapped in try/except. A failure in graphify_lang degrades to one logged warning and the suffix falls back to the built-in extractor (or to no extractor if none exists). This mirrors the existing behavior of run_language_resolvers.

  6. No suffix stealing — A registry can override a built-in suffix only if the built-in suffix is listed in the manifest's overrides field. Without that, the built-in suffix is taken. This protects existing languages from accidental override.

Upstream path

This proposal does not require any changes to the core test suite. The existing test tests/test_extractors_registry.py already enforces that LANGUAGE_EXTRACTORS must not contain plugin extractors; this proposal keeps LANGUAGE_EXTRACTORS unchanged and uses a separate registry for third-party extractors.

The fork plans to carry the registry indefinitely. If upstream accepts this proposal, the fork will rebase onto the accepted implementation and drop its own registry code. The AutoLISP plugin (graphify_lang/autolisp) is a separate concern and will remain in the fork as a reference implementation.

References

Checklist for upstream acceptance

  • Accept the registry pattern as a mechanism for third-party language extractors
  • Review the core changes (3 call sites, 1 new file, 1 test file)
  • Verify the casefix in resolver_registry.py for mixed-case suffix matching
  • Confirm no changes to existing tests are required
  • Accept that built-in suffixes can be overridden only via the overrides field
  • Accept that the registry is optional and degrades gracefully if graphify_lang is not installed

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions