Skip to content

jvm: resolve protobuf codegen ambiguity per requesting language - #23647

Closed
robertpi wants to merge 4 commits into
pantsbuild:mainfrom
robertpi:jvm/protobuf-classpath-per-consumer-language
Closed

jvm: resolve protobuf codegen ambiguity per requesting language#23647
robertpi wants to merge 4 commits into
pantsbuild:mainfrom
robertpi:jvm/protobuf-classpath-per-consumer-language

Conversation

@robertpi

@robertpi robertpi commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Summary

Fixes #23619: a single, unparametrized protobuf_sources() target can now be depended on by java_sources, scala_sources, and python_sources at the same time, with no disambiguating field and no need to declare multiple protobuf_sources targets.

Previously, depending on the same protobuf_sources target from both a java_sources and a scala_sources target raised ambiguity errors, because two independent parts of the JVM/codegen machinery couldn't tell which generated language a given consumer wanted:

  1. Classpath-entry dispatch (ClasspathEntryRequestFactory.classify_impl in jvm/compile.py) couldn't decide between CompileJavaSourceRequest and CompileScalaSourceRequest for the shared codegen target.
  2. Source hydration/codegen dispatch (hydrate_sources in engine/internals/graph.py) couldn't decide between GenerateJavaFromProtobufRequest and GenerateScalaFromProtobufRequest when compile_scala_source hydrates its own component's sources (it requests both ScalaSourceField and JavaSourceField to support scalac's mixed Java/Scala compilation).

This PR fixes both:

  • classify_impl/for_targets gain a preferred_impl param; classpath_dependency_requests passes the concrete requester's own ClasspathEntryRequest type as the preference, so classpath-entry ambiguity resolves per (component, requesting-language) edge instead of per-component alone.
  • hydrate_sources narrows ambiguous codegen candidates using the order of the caller's for_sources_types tuple: if restricting to the first type in that tuple with any matching generator leaves exactly one candidate, it's used instead of raising. Genuine ambiguity (multiple generators producing the same output type) still raises, unchanged.

Neither fix requires new fields, new BUILD syntax, or any per-backend wiring — dispatch is resolved implicitly from context that already exists at each call site.

Added test_protobuf_consumed_by_java_and_scala (jvm/compile_test.py), a real BUILD-level integration test with a fresh JVM lockfile fixture, proving a single protobuf_sources target compiles correctly for both a Java and a Scala consumer in the same graph.

Design choice — why not a jvm_codegen_type field?

An earlier, narrower version of this fix lives on jvm/protobuf-codegen-type-parametrize (see 3435af0). That approach added a jvm_codegen_type field to protobuf_sources/protobuf_source and used parametrize to generate a distinct target per JVM language, letting users write e.g. protobuf_sources(jvm_codegen_type=parametrize("java", "scala")).

That branch is kept around, unmerged, as a smaller and more conservative alternative, but has real drawbacks surfaced during review:

  • The field is JVM-specific but lives on a target type also used by non-JVM consumers (Python) — python_sources has no reason to know about jvm_codegen_type.
  • To disambiguate cleanly, users still end up declaring multiple protobuf_sources targets (one per JVM language via parametrize), which leaks the JVM implementation detail into the user's BUILD files instead of hiding it.
  • It's also a strictly smaller diff by construction: it adds a field and wires it through target-level parametrization rather than touching the shared dispatch machinery, so its "smaller PR" is really "narrower fix, more BUILD-visible surface."

This PR trades a bit more engine-internal surface area for a fix that requires zero BUILD-visible changes and generalizes to any two conflicting codegen outputs sharing a dependency edge, not just protobuf's java/scala split.

Test plan

  • pants test src/python/pants/jvm/compile_test.py — new end-to-end test passes; unit-level test_request_classification coverage of preferred_impl passes
  • pants test on backend/codegen/protobuf/{java,scala}/rules_integration_test.py, backend/{java,scala,kotlin}/compile/*_test.py, engine/internals/graph_test.py, core/util_rules/source_files_test.py — no regressions
  • pants lint fmt check on all touched files

… per component

Thread the direct dependent's own ClasspathEntryRequest type through
classpath_dependency_requests as a `preferred_impl` hint. When a codegen
input (e.g. protobuf_sources) is consumed by more than one JVM language's
codegen, this lets classify_impl prefer whichever language is actually
asking for it on that dependency edge, instead of requiring the shared
target to declare which language it's for. A single, unparametrized
protobuf_sources() declaration can now be depended on by java_sources and
scala_sources (and non-JVM consumers) without any disambiguating field.

Root-level resolution (no requester) is unchanged and still raises
ClasspathSourceAmbiguity, but protobuf_sources/protobuf_source are never
valid roots for JVM compile/check, so this doesn't arise in practice.

Environment: Datadog workspace
…e_sources gap

Adds `test_protobuf_consumed_by_java_and_scala`, a real BUILD-level
integration test with a fresh JVM lockfile fixture, showing that a
single unparametrized `protobuf_sources` target can be depended on by
both `java_sources` and `scala_sources` without any disambiguating
field or additional target declaration.

The test is marked `xfail(strict=True)`: it currently fails with
`AmbiguousCodegenImplementationsException` because `hydrate_sources`
is a separate codegen-dispatch mechanism from the classpath-entry
dispatch that `preferred_impl` already fixes.
`compile_scala_source` hydrates its own component's sources with
`for_sources_types=(ScalaSourceField, JavaSourceField)` to support
scalac's mixed-source compilation, so `hydrate_sources` still can't
choose between `GenerateJavaFromProtobufRequest` and
`GenerateScalaFromProtobufRequest` for the shared `protos` target.

Environment: Datadog workspace
…types order

`hydrate_sources` raised `AmbiguousCodegenImplementationsException` whenever
more than one registered code generator could satisfy a `for_sources_types`
request, even when the caller's tuple already expresses a preference order
(e.g. `compile_scala_source` passes `(ScalaSourceField, JavaSourceField)` to
accept Java sources coarsened into the same compile, alongside its own
Scala sources). This meant a single `protobuf_sources` target depended on by
both `java_sources` and `scala_sources` could not be hydrated on the Scala
side at all, independent of the classpath-entry-level ambiguity already
fixed by `preferred_impl` in `jvm/compile.py`.

Narrow to the first `for_sources_types` entry that has any matching
generator before raising: if exactly one generator remains, use it. Genuine
ambiguity (multiple generators producing the same output type) still
raises, unchanged.

Removes the `xfail` marker from `test_protobuf_consumed_by_java_and_scala`
now that it passes, and fixes its classpath-entry-name assertions to match
the real jar naming.

Environment: Datadog workspace
@robertpi
robertpi force-pushed the jvm/protobuf-classpath-per-consumer-language branch from d789595 to 3740582 Compare August 25, 2026 09:42
@robertpi robertpi closed this Aug 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant