KAFKA-20032: KIP-1265 Automatic detection of internal API usage

[ashwinpankaj]

Summary

Wires up the KIP-1265 mechanism for explicit declaration and automated
enforcement of Apache Kafka's public API surface. Adds two annotations
(@InterfaceAudience.Public / .Private, modelled on Hadoop and
orthogonal to the existing @InterfaceStability), an auditable escape
hatch (@SuppressKafkaInternalApiUsage), a Gradle checker that
cross-validates the javadoc HTML against the bytecode annotations and
cascades through public method signatures, and a Gradle/Maven checker
plugin for external plugin developers. Every javadoc'd class across the
10 docsJar modules is annotated, and ./gradlew docsJar now finalizes
with the checker on every project — 0 violations, 20 suppressions, every
suppression carrying a reason the next reviewer can audit.

JIRA: KAFKA-20032 ·
KIP:
KIP-1265

What's in this PR

Annotations (in o.a.k.common.annotation)

• @InterfaceAudience.Public / .Private — marker annotations, RUNTIME
  retention, @Documented. The audience dimension is orthogonal to
  @InterfaceStability; a class may carry both.
• @SuppressKafkaInternalApiUsage("reason") —
  class/method/field/constructor target, RUNTIME retention. Required
  reason text is surfaced in every checker report so each escape hatch is
  reviewable inline.

Audience inheritance

A class's effective audience is its own direct @InterfaceAudience
annotation if present, otherwise the audience of its nearest annotated
enclosing class. Default everywhere is @Private per the KIP. A nested
class can override an inherited @Public with explicit
@InterfaceAudience.Private. This matches Hadoop's model and aligns
with how javadoc treats nested classes — they're documented under the
outer's page, so requiring a separate annotation on every public nested
class would be redundant noise. Inheritance also handles protected and
package-private nested classes correctly: they appear in javadoc only as
part of the outer's docs, so the checker only enforces direct @Public
for the MISSING_JAVADOC check, while cascade reference checks accept
inherited @Public for any nested type. Net effect: ~640 javadoc'd
classes are covered by ~550 top-level annotations, with the rest carried
by inheritance.

Producer-side checker (kafkaPublicApiChecker — runs as part of

docsJar)
Cross-validates each module's javadoc jar against its bytecode-level
annotations. Reports three categories of finding:

• MISSING_PUBLICAPI_ANNOTATION — a class is in the javadoc HTML but
  isn't effectively @Public.
• MISSING_JAVADOC — a class carries a direct @Public but no HTML
  page in the javadoc jar.
• Cascade types (INVALID_RETURN_TYPE / INVALID_PARAMETER_TYPE /
  INVALID_EXCEPTION_TYPE) — a public method signature on a @Public
  class leaks an internal type. Cascade findings whose containing method
  or class carries @SuppressKafkaInternalApiUsage divert to a separate
  suppressions section with the supplied reason.

Consumer-side checker (kafkaInternalApiChecker plugin)

Gradle and Maven plugins
(o.a.k:kafka-internal-api-checker-maven-plugin) for external plugin
developers — walk compiled bytecode (Java/Scala/Kotlin, uniformly) and
flag references to any org.apache.kafka.* type that isn't @Public.
The same @SuppressKafkaInternalApiUsage annotation applies to
known/intentional consumer-side exposures.

Cross-module support

Per-module checks include sibling Kafka jars as referenceJarFiles —
reference classes contribute to the surface so cross-module @Public
resolution works (e.g. :storage:storage-api referencing
o.a.k.common.TopicIdPartition from :clients), but they don't
participate in the module's own MISSING_JAVADOC or cascade iteration.
Gradle wires this via
configurations.compileClasspath.incoming.artifactView { componentFilter { it instanceof ProjectComponentIdentifier } … } filtered to the JAR
artifact variant.

Coverage

./gradlew docsJar triggers the checker on every subproject. Per-module
results across the 10 docsJar modules:

| Module | Violations | Suppressions | |---|---:|---:| | :clients |
0 | 10 | | :streams | 0 | 9 | | :streams:test-utils | 0 | 1 | |
:streams:streams-scala | 0 | 0 | | :connect:api | 0 | 0 | |
:connect:mirror-client | 0 | 0 | | :connect:test-plugins | 0 | 0 |
| :storage:storage-api | 0 | 0 | | :tools:tools-api | 0 | 0 | |
:group-coordinator:group-coordinator-api | 0 | 0 | | Total |
0 | 20 |

Suppressions in this PR

Every suppression carries a KIP-1265: … reason in the source and is
rendered in the checker report. They mark cascade leaks that can't be
cleanly resolved without API-shape changes the KIP scope doesn't cover:

• :clients (10) — Metrics(Time) × 6, KafkaMetric(Time),
  JwtBearerJwtRetriever(Time), ClientCredentialsJwtRetriever(Time),
  DefaultJwtValidator(CloseableVerificationKeyResolver). All cases of
  internal o.a.k.common.utils.Time (or other internal-package types)
  reaching the public ctor surface for test-injection purposes.
• :streams (9) — QueryableStoreType.create(StateStoreProvider)
  plus the five QueryableStoreTypes nested overrides;
  KafkaStreams.<init>(…, Time, …) × 3 (same Time-injection pattern as
  Metrics).
• :streams:test-utils (1) —
  MockProcessorContext.recordCollector() returns internal
  RecordCollector.

These are deliberate, reviewable, and easy to grep. Follow-up KIPs can
decide per-case whether to promote the internal type or refactor the
method signature.

Genuine public-API leaks surfaced and fixed

The checker found two cases where the de-facto public surface didn't
match the published javadoc:

1. o.a.k.common.config.types.Password — already public class,
   returned by AbstractConfig.getPassword() which is @Public, but the
   …common.config.types.* package was missing from the :clients javadoc
   include list. Brought it into the include list and annotated Password
   with @Public.
2. SampledStat.Sample — protected static class returned by two
   public methods (current(long), oldest(long)) of @Public abstract SampledStat. Subclasses extending SampledStat have always been able
   to touch this type; the annotation now makes the de-facto contract
   explicit. Visibility unchanged.

Implementation notes

• PublicApiChecker (~100 LOC) — facade. Constructor takes (List<File> projectJars, List<File> referenceJars) and pre-scans into an immutable
  ApiSurface. Subsequent calls (checkPublicApiConsistency,
  isPublicApi, checkBytecode) are thin delegations.
• ApiSurface — immutable scan result. Hides the dotted-vs-binary name
  distinction at the boundary (all lookups accept either form and
  normalize internally).
• ApiSurfaceScanner — two-pass static scan(projectJars, referenceJars) → ApiSurface. Pass 1 reads bytecode facts per class via
  ASM; pass 2 walks the enclosing-class chain to resolve effective
  audience.
• CascadeValidator — method-signature cascade +
  @SuppressKafkaInternalApiUsage detection.
• JavadocConsistencyValidator — javadoc HTML reader +
  MISSING_JAVADOC / MISSING_PUBLICAPI_ANNOTATION cross-checks.
• ClassFacts — per-class bytecode facts (immutable, builder pattern
  for the ASM visitor).
• PluginDeveloperApiUsageScanner — consumer-side ASM walker.

No classloading anywhere. All bytecode reading goes through ASM, so
a class with broken transitive deps on the checker's classpath (gRPC
stubs, telemetry shims, etc.) can't trip LinkageError /
NoClassDefFoundError.

Test plan

• ./gradlew docsJar — every module's kafkaPublicApiChecker runs
  and reports 0 violations.
• Per-module reports inspected: 0 violations, 20 auditable
  suppressions across :clients (10), :streams (9),
  :streams:test-utils (1).
• :buildSrc:compileJava + :buildSrc:compileTestJava clean.
• End-to-end Gradle and Maven consumer tests (sample plugin app
  referencing KafkaProducer, ProducerRecord, Callback,
  RecordMetadata, StringSerializer) produce the expected violations
  and suppressions.
• Existing Kafka tests continue to pass.

Reviewers: Manikumar Reddy manikumar.reddy@gmail.com

[github-actions]

A label of 'needs-attention' was automatically added to this PR in order to raise the
attention of the committers. Once this issue has been triaged, the triage label
should be removed to prevent this automation from happening again.

[github-actions]

This PR is being marked as stale since it has not had any activity in 90 days. If you
would like to keep this PR alive, please leave a comment asking for a review. If the PR has
merge conflicts, update it with the latest from the base branch.

If you are having difficulty finding a reviewer, please reach out on the [mailing list](https://kafka.apache.org/contact).

If this PR is no longer valid or desired, please feel free to close it. If no activity occurs in the next 30 days, it will be automatically closed.

[github-actions]

A label of 'needs-attention' was automatically added to this PR in order to raise the
attention of the committers. Once this issue has been triaged, the triage label
should be removed to prevent this automation from happening again.

[github-actions]

A label of 'needs-attention' was automatically added to this PR in order to raise the
attention of the committers. Once this issue has been triaged, the triage label
should be removed to prevent this automation from happening again.

[omkreddy]

Can we use a module name such as kafka-build-plugins or kafka-api-checker instead of buildSrc?

[omkreddy]

Can we use the id as kafka-internal-api-checker-gradle-plugin' which will be similar to kafka-internal-api-checker-maven-plugin

[omkreddy]

Can we confirm that this artifact uses the same version as the Kafka release? It also looks like the -PscalaVersion=2.13 flag is not required here.