Skip to content

dockerfile: install libzstd-dev for kafka zstd compression (4.2 backport) #12004

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
dbottini2 wants to merge 2 commits into
base: 4.2
Choose a base branch
from
+28 −1
Open

dockerfile: install libzstd-dev for kafka zstd compression (4.2 backport) #12004

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
bf213c1
dockerfile: install libzstd-dev for kafka zstd compression
dbottini2 Jun 25, 2026
cbca3a9
tests: runtime: cover librdkafka zstd compression support
dbottini2 Jun 25, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion dockerfiles/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@ RUN apt-get update && \
pkg-config \
libsystemd-dev \
zlib1g-dev \
libzstd-dev \
libpq-dev \
postgresql-server-dev-all \
flex \
Expand Down Expand Up @@ -262,7 +263,7 @@ RUN apt-get update && \
openssl \
htop atop strace iotop sysstat ncdu logrotate hdparm pciutils psmisc tree pv \
make tar flex bison \
libssl-dev libsasl2-dev libsystemd-dev zlib1g-dev libpq-dev libyaml-dev postgresql-server-dev-all \
libssl-dev libsasl2-dev libsystemd-dev zlib1g-dev libzstd-dev libpq-dev libyaml-dev postgresql-server-dev-all \
&& apt-get satisfy -y cmake "cmake (<< 4.0)" \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
Expand Down
26 changes: 26 additions & 0 deletions tests/runtime/out_kafka.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,34 @@
#include <fluent-bit.h>
#include "flb_tests_runtime.h"

#include "rdkafka.h"

/* Test data */
#include "data/td/json_td.h"

/*
* Ensure librdkafka was compiled with zstd support so Kafka producers can
* negotiate zstd compression. Setting compression.codec to zstd only succeeds
* when WITH_ZSTD was enabled at build time, otherwise rd_kafka_conf_set
* reports the codec as not built in.
*/
void flb_test_zstd_compression_available()
{
rd_kafka_conf_t *conf;
rd_kafka_conf_res_t res;
char errstr[512] = {0};

conf = rd_kafka_conf_new();
TEST_CHECK(conf != NULL);

res = rd_kafka_conf_set(conf, "compression.codec", "zstd",
errstr, sizeof(errstr));
TEST_CHECK(res == RD_KAFKA_CONF_OK);

@chatgpt-codex-connector chatgpt-codex-connector Bot Jun 25, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Guard zstd-only Kafka runtime test

In runtime-test builds where FLB_OUT_KAFKA is enabled but librdkafka was configured without WITH_ZSTD (for example, non-Docker source builds without a system libzstd-dev, or builds that explicitly disable zstd), this assertion now fails even though the Kafka output can still build and run with other codecs. The Dockerfile change only fixes the container builder stages, while cmake/kafka.cmake does not force librdkafka to use Fluent Bit's bundled zstd, so this generic runtime test makes supported Kafka test configurations fail unless zstd happens to be available.

Useful? React with 👍 / 👎.

TEST_MSG("compression.codec=zstd rejected: %s", errstr);

rd_kafka_conf_destroy(conf);
}


void flb_test_raw_format()
{
Expand Down Expand Up @@ -44,6 +69,7 @@ void flb_test_raw_format()
}

TEST_LIST = {
{ "zstd_compression_available", flb_test_zstd_compression_available },
{ "raw_format", flb_test_raw_format },
{ NULL, NULL },
};
Loading