[beatreceiver][packetbeat] Delegate shutdown_timeout logic to libbeat

[khushijain21]

Proposed commit message

This PR delegatesshutdown_timeout logic to libbeat. It stays consistent with how other beats also handle the same.

This is also required so that we do not wait for shutdown_timeout period every time the config is validated here

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works. Where relevant, I have used the stresstest.sh script to run them under stress conditions and race detector to verify their stability.
• I have added an entry in ./changelog/fragments using the changelog tool.

Disruptive User Impact

None

Will also close #52031

[github-actions]

🤖 GitHub comments

Just comment with:

• run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)
• /test : Run the Buildkite pipeline.

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @khushijain21? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit
• backport-active-all is the label that automatically backports to all active branches.
• backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
• backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

[infra-vault-gh-plugin-prod]

Pinging @elastic/sec-linux-platform (Team:Security-Linux Platform)

[github-actions]

TL;DR

The five failed Packetbeat unit-test jobs share a shutdown-drain regression introduced by this PR, not an OS-specific failure. processor.Stop now stops the transaction publisher immediately, so queued events are discarded before the framework can disconnect/flush the pipeline. Restore an explicit bounded drain or change the publisher stop path to drain queued events before returning.

Remediation

• Update packetbeat/beater/processor.go so processor.Stop does not call TransactionPublisher.Stop until the configured shutdown drain has completed; alternatively make TransactionPublisher.Stop drain its worker queue before stopping.
• Keep the libbeat watchdog registration, but do not rely on Beat.ShutdownTimeout alone: it only sizes the force-disconnect watchdog and is bypassed when beater.Run returns promptly.
• Re-run the scoped Packetbeat unit tests on Linux and Windows, especially test_0002_thrift_basics.py, test_0041_memcache_udp_*, test_0066_dhcp.py, and the SIP golden tests.

Investigation details

Root Cause

The PR removes the wait in packetbeat/beater/processor.go function processor.Stop that previously allowed the asynchronous publisher worker to process pending events, then calls TransactionPublisher.Stop immediately. In packetbeat/publish/publish.go:86-89, TransactionPublisher.Stop closes done and waits; its worker exits on done at packetbeat/publish/publish.go:143-155, even when events remain buffered in the channel.

The new packetbeat/beater/packetbeat.go:106-109 assignment of b.ShutdownTimeout does not replace that drain: libbeat uses it only to calculate the shutdown watchdog grace in libbeat/cmd/instance/beat.go:546-571, while packetbeat returns from Run after the deferred runner stop. The watchdog therefore does not wait for the publisher worker. The system-test template sets packetbeat.shutdown_timeout to 400ms by default at packetbeat/tests/system/config/packetbeat.yml.j2:200, which matches the prior bounded drain behavior.

Evidence

• Build: https://buildkite.com/elastic/beats/builds/49626
• Jobs: Packetbeat Ubuntu x86_64, RHEL9, Windows 2016, Windows 2022, and Windows 2025 unit tests all fail in the Python/system-test phase.
• Linux log: test_0002_thrift_basics.py reports assert 9 == 17, multiple Memcache tests report missing packetbeat-20260716.ndjson, DHCP reports assert 1 == 4, and SIP golden output is short.
• Windows logs show the same truncation: Thrift/Memcache output files are missing, DHCP reports assert 1 == 4, and SIP golden tests produce fewer events than expected.
• The failure counts vary by platform and timing (17/15/22/18 failures), but the event-loss pattern is consistent; the Go tests/build complete and the failures are concentrated in event-producing system tests.

Verification

Not run locally; the available evidence is from the five failed Buildkite jobs.

---

What is this? | From workflow: PR Buildkite Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

[khushijain21]

There is fundamental difference in what shutdown_timeout in packetbeat does compared to filebeat.

In filebeat, shutdown_timeout is configured to control the pipeline shutdown. See. But in packetbeat - this same field controls when to stop sending events to the pipeline. This is more of a producer side control . See and #5200 cc: @swiatekm

Since this PR immediately stops sending any buffered events to the pipeline on stop signal, we are seeing certain python tests fail. I think shutdown_timeout here was introduced for python tests to pass. This field has never been exposed to users via packetbeat.reference.yaml or via docs.

Can the @elastic/sec-linux-platform team confirm why we need this config? shutdown_timeout should be a reserved field and only be used to control the pipeline shutdown timeout instead. See

[swiatekm]

There is fundamental difference in what shutdown_timeout in packetbeat does compared to filebeat.

In filebeat, shutdown_timeout is configured to control the pipeline shutdown. See. But in packetbeat - this same field controls "how long to continue sending buffered events from packetbeat to the pipeline". This is more of a producer side control . See. cc: @swiatekm

Since this PR immediately stops sending any buffered events to the pipeline on stop signal, we are seeing certain python tests fail. I think shutdown_timeout here was introduced for python tests to pass. This field has never been exposed to users via packetbeat.reference.yaml or via docs.

Can the @elastic/sec-linux-platform team confirm why we need this config? shutdown_timeout should be a reserved field and only be used to control the pipeline shutdown timeout instead. See

So it sounds like we simply shouldn't set this for packetbeat receiver until we figure out what it should do. Ideally shutdown_timeout should work the same way for packetbeat as it does for other beats and the internal setting should have a different name.

[nfritts]

This is a really old parameter. I don't personally know of a reason it couldn't be renamed in packetbeat. But maybe we should see if @ruflin has any context. It looks like the parameter was renamed to unify it with FileBeat as part of #3588 but then when the delay was executed was adjusted to make tests pass here: c0de062

@khushijain21 I don't know that anyone on my team will have the context needed to answer this really well. I see that @fearful-symmetry and @efd6 have both contributed near this so it is probably worth having both of them weigh in on if they remember anything about it.

[fearful-symmetry]

Hmm, yeah, I don't think I've ever tinkered with this, so I don't have any context to add here. There's a lot of old cruft in Packetbeat. I suspect there wasn't a whole lot of reasoning going into those changes, however. We should probably bias towards "whatever makes it work"

[nfritts]

From talking to both Alex and Dan none of us have a lot of context with this but we think the change makes sense.

[mergify]

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

• Queue this pull request

[github-actions]

@Mergifyio backport 9.5 9.4 9.3 8.19

[mergify]

backport 9.5 9.4 9.3 8.19

✅ Backports have been created

Details

• #52265 [9.5](backport #52005) [beatreceiver][packetbeat] Delegate shutdown_timeout logic to libbeat has been created for branch 9.5
• #52266 [9.4](backport #52005) [beatreceiver][packetbeat] Delegate shutdown_timeout logic to libbeat has been created for branch 9.4 but encountered conflicts

Cherry-pick of 663f454 has failed:

On branch mergify/bp/9.4/pr-52005
Your branch is up to date with 'origin/9.4'.

You are currently cherry-picking commit 663f45459.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   packetbeat/config/config.go
	modified:   packetbeat/tests/system/config/packetbeat.yml.j2
	modified:   packetbeat/tests/system/test_0060_flows.py

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   packetbeat/beater/packetbeat.go
	both modified:   packetbeat/beater/processor.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

• #52267 [9.3](backport #52005) [beatreceiver][packetbeat] Delegate shutdown_timeout logic to libbeat has been created for branch 9.3 but encountered conflicts

Cherry-pick of 663f454 has failed:

On branch mergify/bp/9.3/pr-52005
Your branch is up to date with 'origin/9.3'.

You are currently cherry-picking commit 663f45459.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   packetbeat/config/config.go
	modified:   packetbeat/tests/system/config/packetbeat.yml.j2
	modified:   packetbeat/tests/system/test_0060_flows.py

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   packetbeat/beater/packetbeat.go
	both modified:   packetbeat/beater/processor.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

• #52268 [8.19](backport #52005) [beatreceiver][packetbeat] Delegate shutdown_timeout logic to libbeat has been created for branch 8.19 but encountered conflicts

Cherry-pick of 663f454 has failed:

On branch mergify/bp/8.19/pr-52005
Your branch is up to date with 'origin/8.19'.

You are currently cherry-picking commit 663f45459.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   packetbeat/config/config.go
	modified:   packetbeat/tests/system/config/packetbeat.yml.j2
	modified:   packetbeat/tests/system/test_0060_flows.py

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   packetbeat/beater/packetbeat.go
	both modified:   packetbeat/beater/processor.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally