Skip to content

Vulnerability fix changes #121

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
maheshkumargangula merged 2 commits into from
Apr 22, 2026
Merged

Vulnerability fix changes #121

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
35dc050
fix: vulnerability fix changes
vinodbhorge Apr 2, 2026
4cf006d
Merge branch 'main' into Vulnerability-fixes
maheshkumargangula Apr 22, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
72 changes: 25 additions & 47 deletions Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,59 +1,37 @@
FROM maven:3.9.4-eclipse-temurin-11-focal AS build-core
FROM public.ecr.aws/docker/library/maven:3.9.4-eclipse-temurin-11-focal AS build-core
COPY . /app
RUN mvn clean install -DskipTests -f /app/pom.xml
# RUN mvn clean install -DskipTests -f /app/dataset-registry/pom.xml
# RUN mvn clean install -DskipTests -f /app/transformation-sdk/pom.xml

FROM maven:3.9.4-eclipse-temurin-11-focal AS build-pipeline
FROM public.ecr.aws/docker/library/maven:3.9.4-eclipse-temurin-11-focal AS build-pipeline
COPY --from=build-core /root/.m2 /root/.m2
COPY . /app
RUN mvn clean package -DskipTests -f /app/pipeline/pom.xml

FROM sanketikahub/flink:1.20-scala_2.12-java11 AS extractor-image
FROM public.ecr.aws/docker/library/flink:1.20-scala_2.12-java11 AS unified-image

@coderabbitai coderabbitai Bot Apr 22, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

Base image violates coding guideline.

As per coding guidelines, "Docker images for deployment must use base image: sanketikahub/flink:1.20-scala_2.12-java11". This stage (and cache-indexer-image at line 27) uses public.ecr.aws/docker/library/flink:1.20-scala_2.12-java11, which is the upstream Apache Flink image rather than the project's hardened base. Revert to sanketikahub/flink:1.20-scala_2.12-java11 unless the guideline is being formally changed — in which case update CLAUDE.md in the same PR.

🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@Dockerfile` at line 10, The Dockerfile uses the upstream Flink base image;
change the FROM references for the unified-image stage (symbol: unified-image)
and the cache-indexer-image stage (symbol: cache-indexer-image) from
public.ecr.aws/docker/library/flink:1.20-scala_2.12-java11 to the approved
sanketikahub/flink:1.20-scala_2.12-java11; if you intend to change the project
guideline instead, update CLAUDE.md in the same PR to document the new approved
base image.

USER flink
RUN mkdir -p $FLINK_HOME/usrlib
COPY --from=build-pipeline /app/pipeline/extractor/target/extractor-1.0.0.jar $FLINK_HOME/usrlib/

FROM sanketikahub/flink:1.20-scala_2.12-java11 AS preprocessor-image
USER flink
RUN mkdir -p $FLINK_HOME/usrlib
COPY --from=build-pipeline /app/pipeline/preprocessor/target/preprocessor-1.0.0.jar $FLINK_HOME/usrlib/

FROM sanketikahub/flink:1.20-scala_2.12-java11 AS denormalizer-image
USER flink
RUN mkdir -p $FLINK_HOME/usrlib
COPY --from=build-pipeline /app/pipeline/denormalizer/target/denormalizer-1.0.0.jar $FLINK_HOME/usrlib/

FROM sanketikahub/flink:1.20-scala_2.12-java11 AS transformer-image
USER flink
RUN mkdir -p $FLINK_HOME/usrlib
COPY --from=build-pipeline /app/pipeline/transformer/target/transformer-1.0.0.jar $FLINK_HOME/usrlib/

FROM sanketikahub/flink:1.20-scala_2.12-java11 AS dataset-router-image
USER flink
RUN mkdir -p $FLINK_HOME/usrlib
COPY --from=build-pipeline /app/pipeline/dataset-router/target/dataset-router-1.0.0.jar $FLINK_HOME/usrlib/

# unified image build
FROM sanketikahub/flink:1.20-scala_2.12-java11 AS unified-image
USER flink
RUN mkdir -p $FLINK_HOME/usrlib
# Move the bundled flink-s3-fs-hadoop plugin from opt/ to the required plugins subfolder.
# This avoids a network download and guarantees the plugin version matches the runtime.
RUN mkdir -p $FLINK_HOME/usrlib && \
mkdir -p $FLINK_HOME/plugins/flink-s3-fs-hadoop && \
mv $FLINK_HOME/opt/flink-s3-fs-hadoop-*.jar $FLINK_HOME/plugins/flink-s3-fs-hadoop/
# Use IRSA/OIDC (Web Identity Token) for S3 auth instead of static access keys.
# EKS injects AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE into pods whose service
# account has an IAM role annotation; WebIdentityTokenCredentialsProvider reads them.
RUN if [ -f "$FLINK_HOME/conf/config.yaml" ]; then \
echo 's3.aws.credentials.provider: com.amazonaws.auth.WebIdentityTokenCredentialsProvider' >> $FLINK_HOME/conf/config.yaml; \
else \
echo 's3.aws.credentials.provider: com.amazonaws.auth.WebIdentityTokenCredentialsProvider' >> $FLINK_HOME/conf/flink-conf.yaml; \
fi
COPY --from=build-pipeline /app/pipeline/unified-pipeline/target/unified-pipeline-1.0.0.jar $FLINK_HOME/usrlib/

# # Lakehouse connector image build
# FROM sanketikahub/flink:1.17.2-scala_2.12-java11 AS lakehouse-connector-image
# USER flink
# RUN wget https://repo1.maven.org/maven2/org/apache/flink/flink-shaded-hadoop-2-uber/2.8.3-10.0/flink-shaded-hadoop-2-uber-2.8.3-10.0.jar
# RUN wget https://repo1.maven.org/maven2/org/apache/flink/flink-s3-fs-hadoop/1.17.2/flink-s3-fs-hadoop-1.17.2.jar
# RUN wget https://repo.maven.apache.org/maven2/org/apache/hudi/hudi-flink1.17-bundle/1.0.2/hudi-flink1.17-bundle-1.0.2.jar
# RUN mv flink-shaded-hadoop-2-uber-2.8.3-10.0.jar $FLINK_HOME/lib
# RUN mv flink-s3-fs-hadoop-1.17.2.jar $FLINK_HOME/lib
# RUN mv hudi-flink1.17-bundle-1.0.2.jar $FLINK_HOME/lib
# # RUN mkdir $FLINK_HOME/custom-lib
# COPY --from=build-pipeline /app/pipeline/hudi-connector/target/hudi-connector-1.0.0.jar $FLINK_HOME/lib

# cache indexer image build
FROM sanketikahub/flink:1.20-scala_2.12-java11 AS cache-indexer-image
FROM public.ecr.aws/docker/library/flink:1.20-scala_2.12-java11 AS cache-indexer-image
USER flink
RUN mkdir -p $FLINK_HOME/usrlib
RUN mkdir -p $FLINK_HOME/usrlib && \
mkdir -p $FLINK_HOME/plugins/flink-s3-fs-hadoop && \
mv $FLINK_HOME/opt/flink-s3-fs-hadoop-*.jar $FLINK_HOME/plugins/flink-s3-fs-hadoop/
RUN if [ -f "$FLINK_HOME/conf/config.yaml" ]; then \
echo 's3.aws.credentials.provider: com.amazonaws.auth.WebIdentityTokenCredentialsProvider' >> $FLINK_HOME/conf/config.yaml; \
else \
echo 's3.aws.credentials.provider: com.amazonaws.auth.WebIdentityTokenCredentialsProvider' >> $FLINK_HOME/conf/flink-conf.yaml; \
fi
COPY --from=build-pipeline /app/pipeline/cache-indexer/target/cache-indexer-1.0.0.jar $FLINK_HOME/usrlib/
Comment on lines +10 to 37

@coderabbitai coderabbitai Bot Apr 22, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

CI/CD build matrix references removed Docker targets — pipeline will fail.

The removed stages (extractor-image, preprocessor-image, denormalizer-image, transformer-image, dataset-router-image) are still present in .github/workflows/build_and_deploy.yaml (lines 19–31). After this change, the workflow will fail on every one of those matrix entries with failed to solve: target <name>-image: not found. Either:

  1. Add the stages back (aliased to unified-image if the consolidation is intentional), or
  2. Update build_and_deploy.yaml in this PR to drop those matrix entries.
🧰 Tools
🪛 Trivy (0.69.3)

[error] 22-22: 'RUN update' instruction alone

The instruction 'RUN update' should always be followed by ' install' in the same RUN statement.

Rule: DS-0017

Learn more

(IaC/Dockerfile)

[error] 23-23: 'apt-get' missing '--no-install-recommends'

'--no-install-recommends' flag is missed: 'apt-get install libcurl4 curl -y'

Rule: DS-0029

Learn more

(IaC/Dockerfile)

[error] 26-29: 'apt-get' missing '--no-install-recommends'

'--no-install-recommends' flag is missed: 'set -ex; apt-get update; apt-get -y install gpg libsnappy1v5 gettext-base libjemalloc-dev; rm -rf /var/lib/apt/lists/*'

Rule: DS-0029

Learn more

(IaC/Dockerfile)

🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@Dockerfile` around lines 10 - 37, The Dockerfile removed several build stages
(extractor-image, preprocessor-image, denormalizer-image, transformer-image,
dataset-router-image) while keeping unified-image and cache-indexer-image, which
breaks the CI matrix in build_and_deploy.yaml; fix by either re-adding the
missing stages as aliases that point to the consolidated artifact (e.g., create
additional FROM ... AS extractor-image / AS preprocessor-image entries that
mirror unified-image behavior or add simple stage aliases referencing the same
jar) or update the GitHub Actions matrix to remove those stage names, ensuring
the matrix entries match existing Dockerfile targets (refer to the Dockerfile
stage identifiers unified-image and cache-indexer-image and the COPY targets
like unified-pipeline-1.0.0.jar and cache-indexer-1.0.0.jar).

2 changes: 1 addition & 1 deletion FlinkDockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
FROM --platform=linux/x86_64 flink:1.20-scala_2.12-java11
FROM --platform=linux/x86_64 public.ecr.aws/docker/library/flink:1.20-scala_2.12-java11

@coderabbitai coderabbitai Bot Apr 22, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

Base image violates coding guideline.

As per coding guidelines, "Docker images for deployment must use base image: sanketikahub/flink:1.20-scala_2.12-java11". Revert to the project's hardened base, or update the guideline in CLAUDE.md in the same PR if the switch to the ECR public mirror is deliberate.

🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@FlinkDockerfile` at line 1, The Dockerfile currently uses the public ECR base
image string "public.ecr.aws/docker/library/flink:1.20-scala_2.12-java11" which
violates the project's guideline requiring the hardened base
"sanketikahub/flink:1.20-scala_2.12-java11"; update the FROM line to use the
sanctioned base image (replace the image string), or if the ECR mirror is
intentional, add/update rationale and change the guideline in CLAUDE.md within
this PR to permit the new base.

USER flink
# RUN mkdir $FLINK_HOME/custom-lib
RUN wget https://repo1.maven.org/maven2/org/apache/flink/flink-shaded-hadoop-2-uber/2.8.3-10.0/flink-shaded-hadoop-2-uber-2.8.3-10.0.jar
Expand Down
77 changes: 74 additions & 3 deletions dataset-registry/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,18 @@
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
<version>4.5.1</version>
<version>4.5.14</version>
<exclusions>
<exclusion>
<groupId>commons-codec</groupId>
<artifactId>commons-codec</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>commons-codec</groupId>
<artifactId>commons-codec</artifactId>
<version>1.15</version>
</dependency>
<dependency>
<groupId>com.google.code.gson</groupId>
Expand All @@ -55,14 +66,40 @@
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.12</version>
<version>4.13.2</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>io.zonky.test</groupId>
<artifactId>embedded-postgres</artifactId>
<version>2.0.3</version>
<scope>test</scope>
<exclusions>
<exclusion>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.commons</groupId>
<artifactId>commons-compress</artifactId>
</exclusion>
<exclusion>
<groupId>commons-io</groupId>
<artifactId>commons-io</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-compress</artifactId>
<version>1.26.0</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
<version>3.18.0</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.github.codemonstur</groupId>
Expand All @@ -89,11 +126,45 @@
<version>${flink.version}</version>
<scope>test</scope>
<classifier>tests</classifier>
<exclusions>
<exclusion>
<groupId>com.esotericsoftware.kryo</groupId>
<artifactId>kryo</artifactId>
</exclusion>
<exclusion>
<groupId>org.lz4</groupId>
<artifactId>lz4-java</artifactId>
</exclusion>
<exclusion>
<groupId>org.xerial.snappy</groupId>
<artifactId>snappy-java</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>com.esotericsoftware</groupId>
<artifactId>kryo</artifactId>
<version>4.0.3</version>
<scope>test</scope>
Comment on lines +129 to +148

@coderabbitai coderabbitai Bot Apr 22, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Exclusions add no replacements for lz4-java and snappy-java in test classpath.

flink-runtime tests require lz4-java and snappy-java for codec use in serializers and records. Only kryo is re-added. Tests that exercise compressed records/network stack will fail with ClassNotFoundException. The framework/pom.xml pins snappy-java:1.1.10.5 and at.yawk.lz4:lz4-java:1.10.3 in compile scope — verify the same pins are transitively visible to this module for tests.

🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@dataset-registry/pom.xml` around lines 129 - 148, The exclusions removed
lz4-java and snappy-java from the test classpath but only re-adds kryo, causing
runtime ClassNotFoundExceptions in flink-runtime tests; update this module's pom
to explicitly add test-scope dependencies for at.yawk.lz4:lz4-java:1.10.3 and
org.xerial.snappy:snappy-java:1.1.10.5 (the same versions pinned in
framework/pom.xml) or otherwise ensure those compile-scope pins are transitively
visible to tests, keeping the existing kryo test dependency and leaving the
flink-runtime exclusion block intact.

</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-connector-kafka</artifactId>
<version>3.3.0-1.20</version>
<exclusions>
<exclusion>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
</exclusion>
<exclusion>
<groupId>org.xerial.snappy</groupId>
<artifactId>snappy-java</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.zookeeper</groupId>
<artifactId>zookeeper</artifactId>
</exclusion>
</exclusions>
</dependency>
</dependencies>

Expand Down Expand Up @@ -164,7 +235,7 @@
<plugin>
<groupId>net.alchim31.maven</groupId>
<artifactId>scala-maven-plugin</artifactId>
<version>4.4.0</version>
<version>4.8.1</version>
<configuration>
<source>${java.target.runtime}</source>
<target>${java.target.runtime}</target>
Expand Down
Loading
Loading