Skip to content

dockerfile: install libzstd-dev for kafka zstd compression #12003

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
dbottini2 wants to merge 2 commits into
base: master
Choose a base branch
from
+28 −1
Open

dockerfile: install libzstd-dev for kafka zstd compression #12003

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
38240b1
dockerfile: install libzstd-dev for kafka zstd compression
dbottini2 Jun 25, 2026
99acbc8
tests: runtime: cover librdkafka zstd compression support
dbottini2 Jun 25, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion dockerfiles/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@ RUN apt-get update && \
pkg-config \
libsystemd-dev \
zlib1g-dev \
libzstd-dev \
libpq-dev \
postgresql-server-dev-all \
flex \
Expand Down Expand Up @@ -263,7 +264,7 @@ RUN apt-get update && \
openssl \
htop atop strace iotop sysstat ncdu logrotate hdparm pciutils psmisc tree pv \
make tar flex bison \
libssl-dev libsasl2-dev libsystemd-dev zlib1g-dev libpq-dev libyaml-dev postgresql-server-dev-all \
libssl-dev libsasl2-dev libsystemd-dev zlib1g-dev libzstd-dev libpq-dev libyaml-dev postgresql-server-dev-all \
&& apt-get satisfy -y cmake "cmake (<< 4.0)" \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
Expand Down
26 changes: 26 additions & 0 deletions tests/runtime/out_kafka.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,34 @@
#include <fluent-bit.h>
#include "flb_tests_runtime.h"

#include "rdkafka.h"

/* Test data */
#include "data/td/json_td.h"

/*
* Ensure librdkafka was compiled with zstd support so Kafka producers can
* negotiate zstd compression. Setting compression.codec to zstd only succeeds
* when WITH_ZSTD was enabled at build time, otherwise rd_kafka_conf_set
* reports the codec as not built in.
*/
void flb_test_zstd_compression_available()
{
rd_kafka_conf_t *conf;
rd_kafka_conf_res_t res;
char errstr[512] = {0};

conf = rd_kafka_conf_new();
TEST_CHECK(conf != NULL);

res = rd_kafka_conf_set(conf, "compression.codec", "zstd",
errstr, sizeof(errstr));
TEST_CHECK(res == RD_KAFKA_CONF_OK);

@chatgpt-codex-connector chatgpt-codex-connector Bot Jun 25, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Gate the zstd check to zstd-enabled builds

In regular source builds where FLB_OUT_KAFKA=On but the host lacks a system libzstd-dev, librdkafka still builds with WITH_ZSTD=OFF because the CMake path only lets librdkafka's own find_package(ZSTD) decide; the Dockerfile install does not affect those builds. This unconditional assertion then makes flb-rt-out_kafka fail for an otherwise supported non-container build, so either wire zstd into the CMake Kafka build or only run this check when zstd support was actually required/enabled.

Useful? React with 👍 / 👎.

TEST_MSG("compression.codec=zstd rejected: %s", errstr);

rd_kafka_conf_destroy(conf);
}


void flb_test_raw_format()
{
Expand Down Expand Up @@ -44,6 +69,7 @@ void flb_test_raw_format()
}

TEST_LIST = {
{ "zstd_compression_available", flb_test_zstd_compression_available },
{ "raw_format", flb_test_raw_format },
{ NULL, NULL },
};
Loading