(#2) Make indexer-elastic work with ES8.#3
Open
blairlearn wants to merge 1 commit into
Open
Conversation
blairlearn
changed the title
blairlearn
force-pushed
the
v1.22-with-es8
branch
4 times, most recently
from
July 21, 2026 21:03
0768cec to
0f3bf1f
Compare
- Rewrite ElasticIndexRewriter to use elasticsearch-java-8.16.7. - Add unit tests for ElasticIndexWriter. - Add nci-develop to list of pull-request build triggers. - Build/run on OpenJDK 17. - This mirrors the apache/nutch master branch. - Continues to target 11 for runtime. - Force curl to use HTTP 1.1 in scripts to avoid CVE-2026-27135 - Update packages: - all io.netty packages to 4.1.135.Final - org.apache.cxf to v3.6.11 - log4j from 2.52.2 to 2.25.4 - org.apache.tika to 3.3.1. -Added forced Hadoop-managed overrides: - com.nimbusds/nimbus-jose-jwt -> 9.37.4 - org.apache.zookeeper/zookeeper -> 3.8.6 - org.apache.zookeeper/zookeeper-jute -> 3.8.6 - Bumped root Jackson set to 2.18.6 - Added explicit jackson-core 2.18.6 - Override org.bouncycastle for org.apache.cxf, forcing it to v1.84. - Override org.apache.commons2 for org.apache.hadoop, forcing to v2.15.0
blairlearn
force-pushed
the
v1.22-with-es8
branch
from
July 21, 2026 21:17
0f3bf1f to
17ea20c
Compare
blairlearn
marked this pull request as ready for review
July 21, 2026 21:21
There was a problem hiding this comment.
Pull request overview
This PR updates Nutch’s Elasticsearch indexer plugin to work with Elasticsearch 8 by migrating from the deprecated high-level REST client/BulkProcessor to the Elasticsearch Java API Client and BulkIngester. It also bumps multiple dependency versions to address vulnerabilities and adds Testcontainers-based integration testing plus CI wiring to exercise indexer plugins.
Changes:
- Migrate
indexer-elasticto ES8 client APIs (Java API Client + low-level REST client) and implement bulk retry handling. - Add Testcontainers-powered integration test scaffolding for index writers and wire new Ant/CI targets to run them.
- Update plugin/core dependency pins (Netty, Tika, CXF, Log4j, Jackson, etc.) and refresh binary NOTICE/LICENSE entries accordingly.
Reviewed changes
Copilot reviewed 28 out of 28 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| src/test/org/apache/nutch/indexer/IndexWriterIntegrationTest.java | Introduces an integration-test contract for IndexWriter implementations. |
| src/test/org/apache/nutch/indexer/AbstractIndexWriterIT.java | Adds shared Testcontainers-oriented integration test logic for IndexWriters. |
| src/plugin/protocol-http/ivy.xml | Scopes LittleProxy dependency to tests. |
| src/plugin/parse-tika/plugin.xml | Updates bundled Tika dependency jar list to match newer Tika/parsers stack. |
| src/plugin/parse-tika/ivy.xml | Bumps Tika parser/boilerpipe artifacts to 3.3.1. |
| src/plugin/lib-htmlunit/plugin.xml | Updates Netty jar versions embedded in the plugin runtime manifest. |
| src/plugin/lib-htmlunit/ivy.xml | Forces Netty modules to 4.1.135.Final for this plugin. |
| src/plugin/language-identifier/plugin.xml | Updates bundled jar list for Tika language detector dependencies. |
| src/plugin/language-identifier/ivy.xml | Bumps tika-langdetect-optimaize to 3.3.1. |
| src/plugin/indexer-solr/plugin.xml | Updates Netty jar versions referenced by the Solr indexer plugin. |
| src/plugin/indexer-solr/ivy.xml | Forces Netty modules to 4.1.135.Final for Solr indexer dependencies. |
| src/plugin/indexer-kafka/plugin.xml | Updates Netty jar versions referenced by the Kafka indexer plugin. |
| src/plugin/indexer-kafka/ivy.xml | Forces Netty modules to 4.1.135.Final for Kafka indexer dependencies. |
| src/plugin/indexer-elastic/src/test/org/apache/nutch/indexwriter/elastic/TestElasticIndexWriter.java | Adds unit tests for ES8 writer mapping and retry/backoff helpers. |
| src/plugin/indexer-elastic/src/test/org/apache/nutch/indexwriter/elastic/ElasticIndexWriterIT.java | Adds ES8 Testcontainers integration tests for indexing and delete behavior. |
| src/plugin/indexer-elastic/src/java/org/apache/nutch/indexwriter/elastic/ElasticIndexWriter.java | Core ES8 migration: new client, BulkIngester ingestion, and retry handling. |
| src/plugin/indexer-elastic/README.md | Updates docs to reflect REST port 9200 and new BulkIngester-based retry descriptions. |
| src/plugin/indexer-elastic/plugin.xml | Replaces ES7-era runtime jar list with ES8 client/runtime dependencies. |
| src/plugin/indexer-elastic/ivy.xml | Updates dependencies to ES8 client libs and adds Testcontainers Elasticsearch test dependency. |
| src/plugin/indexer-elastic/howto_upgrade_es.md | Refreshes upgrade instructions for ES8 client artifacts and new plugin.xml section markers. |
| src/plugin/build.xml | Adds a top-level plugin target to run indexer integration tests. |
| src/plugin/build-plugin.xml | Adds integration-test target + ensures test dependency resolution and includes Testcontainers jars on test classpath. |
| NOTICE-binary | Updates binary notice entries for new/removed Elasticsearch-related dependencies. |
| LICENSE-binary | Updates binary license list to reflect new/removed Elasticsearch-related dependencies. |
| ivy/ivy.xml | Updates core dependency pins (Log4j, CXF, Jackson, Tika, Netty, security overrides) and adds Testcontainers test deps. |
| docker/Dockerfile | Switches Docker build to Java 17 and builds from a specific fork branch. |
| build.xml | Adds a top-level Ant target to run indexer integration tests. |
| .github/workflows/master-build.yml | Expands PR branch trigger and adds a CI step to run indexer integration tests on relevant changes. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
324
to
+331
| public void close() throws IOException { | ||
| // Close BulkProcessor (automatically flushes) | ||
| IOException closeException = null; | ||
| try { | ||
| bulkProcessor.awaitClose(bulkCloseTimeout, TimeUnit.SECONDS); | ||
| if (bulkIngester != null) { | ||
| bulkIngester.flush(); | ||
| awaitBulkCompletion(bulkCloseTimeout, TimeUnit.SECONDS); | ||
| bulkIngester.close(); | ||
| } |
Comment on lines
+416
to
+430
| long delayMillis = computeExponentialBackoffMillis(expBackoffMillis, | ||
| context.attempt); | ||
| RetryContext retryContext = context.nextAttempt(); | ||
| scheduledRetries.incrementAndGet(); | ||
| retryScheduler.schedule(() -> { | ||
| try { | ||
| bulkIngester.add(retryContext.operation, retryContext); | ||
| } catch (RuntimeException e) { | ||
| LOG.error("Elasticsearch retry failed for {}:", retryContext.description, e); | ||
| } finally { | ||
| scheduledRetries.decrementAndGet(); | ||
| } | ||
| }, delayMillis, TimeUnit.MILLISECONDS); | ||
| LOG.debug("Scheduled Elasticsearch retry for {} in {} ms after {}", | ||
| context.description, delayMillis, reason); |
Comment on lines
+137
to
+139
| <!-- Testcontainers for indexer plugin integration tests --> | ||
| <dependency org="org.testcontainers" name="testcontainers" rev="2.0.3" conf="test->default"/> | ||
| <dependency org="org.testcontainers" name="junit-jupiter" rev="1.21.4" conf="test->default"/> |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Dockerfile
Address vulnerabilities
Closes #2