Mark ML controller non-dumpable before accepting commands by edsavage · Pull Request #3081 · elastic/ml-cpp

edsavage · 2026-07-21T01:14:31Z

Summary

Follow-up to a privately reported security finding (details tracked internally): the ML controller shared UID 1000 with the JVM / pytorch_inference and remained dumpable, so a compromised peer could write /proc/<controller-pid>/mem, overwrite access@GOT, and turn access(path, X_OK) into dlopen(path, RTLD_LAZY) (X_OK == RTLD_LAZY == 1) when handling a spawn command — loading attacker code under the controller's laxer RuntimeDefault-only seccomp.

Fix

After logging is reconfigured and before opening command/output pipes (i.e. before accepting commands):

prctl(PR_SET_DUMPABLE, 0)
Confirm with prctl(PR_GET_DUMPABLE) == 0
Fail closed (LOG_FATAL + EXIT_FAILURE) if either step fails

No-op on non-Linux platforms (/proc/<pid>/mem same-UID writes are the Linux concern).

Test plan

Linux CI: controller still starts and processes commands normally.
On Linux: after start, /proc/<controller-pid>/status shows Dumpable: 0 (or equivalent).
Confirm fail-closed path if PR_SET_DUMPABLE cannot be applied (manual / mocked).

Related: #3078 (__setstate__ pre-load), #3080 (seccomp arch/ABI check).

Same-UID peers in the shared ES/ml-cpp pod could write /proc/<controller-pid>/mem and overwrite access@GOT so a spawn's access(path, X_OK) became dlopen(path, RTLD_LAZY) (elastic/security#12621). Call prctl(PR_SET_DUMPABLE, 0) after logging is up and before opening command pipes; verify with PR_GET_DUMPABLE and fail closed if the boundary cannot be established. No-op on non-Linux platforms where /proc/<pid>/mem is not the concern. Co-authored-by: Cursor <cursoragent@cursor.com>

Co-authored-by: Cursor <cursoragent@cursor.com>

elasticsearchmachine · 2026-07-23T00:49:17Z

Pinging @elastic/ml-core (Team:ML)

Copilot

Pull request overview

Hardens the ML controller on Linux by making the process non-dumpable before it opens command/output pipes, mitigating same-UID /proc/<pid>/mem write attacks described in elastic/security#12621.

Changes:

Add a Linux-only prctl(PR_SET_DUMPABLE, 0) + verification step, and fail-closed if it cannot be applied.
Call the hardening step after logging is reconfigured and before accepting commands (before opening command/output pipes).
Add a changelog entry for the security hardening fix.

Reviewed changes

Copilot reviewed 1 out of 2 changed files in this pull request and generated no comments.

File	Description
docs/changelog/3081.yaml	Documents the security hardening change in the changelog.
bin/controller/Main.cc	Applies Linux `PR_SET_DUMPABLE` hardening before opening controller command/output pipes.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

edsavage · 2026-07-23T01:16:59Z

Test plan verification

All three items checked off. Item 1 is confirmed by CI; items 2–3 were verified against the real controller binary from this PR's CI build (ml-cpp-9.6.0-SNAPSHOT-linux-aarch64, build 88ade68d3, PR CI build 2806), run in a native arm64 Linux container (no emulation).

1 & CI — controller starts and processes commands normally

ml_test_controller (9 cases incl. CCommandProcessorTest_testStartPermitted) passed on Linux x86_64 + aarch64.

End-to-end with the real binary: it logs the version, becomes non-dumpable, opens the command/output pipes, and answers a command:

INFO  controller (64 bit): Version 9.6.0-SNAPSHOT (Build 88ade68d3bd13c) ...
output pipe response: [{"id":1,"success":false,"reason":"Did not understand verb 'nope'"}]

2 — non-dumpable after start (Linux)

This kernel's /proc/<pid>/status has no Dumpable: line, so I verified the authoritative observable of PR_SET_DUMPABLE(0): /proc/<pid> flips to root:root and same-UID peers are denied mem/maps — i.e. exactly the /proc/<pid>/mem write vector from security#12621. Running as a non-root uid (1000), with a normal sleep as a dumpable control:

                          /proc/<pid>/mem owner
control  sleep (dumpable):  1000:1000        maps: readable
CONTROLLER    (this PR)  :  root:root         maps: Permission denied
same-uid peer open(/proc/<controller>/mem, O_RDWR): DENIED  [Errno 13] Permission denied

The controller is caught after makeProcessNonDumpable() and before it opens the command pipe, so the boundary is established before any command is accepted — as intended.

3 — fail-closed (mocked)

prctl(PR_SET_DUMPABLE,0) doesn't fail in practice, so I compiled makeProcessNonDumpable() verbatim (only LOG_ERROR→fprintf, ::prctl→mock) with the same caller logic and forced each failure branch:

A: SET ok, GET==0 (normal)                 -> exit 0 (continue accepting commands)
B: PR_SET_DUMPABLE fails (EPERM)           -> exit 1 (EXIT_FAILURE, LOG_FATAL)
C: SET 'ok' but still dumpable (GET!=0)    -> exit 1 (EXIT_FAILURE, LOG_FATAL)

Both failure paths refuse to accept commands (fail closed); the normal path continues.

edsavage · 2026-07-23T02:43:04Z

buildkite run_pytorch_tests

valeriy42

LGTM. Minor suggestion to improve diagnostic

Distinguish a PR_GET_DUMPABLE syscall failure (-1) from the process still being dumpable, and log the actual value, per review feedback on elastic#3081. Co-authored-by: Cursor <cursoragent@cursor.com>

Co-authored-by: Cursor <cursoragent@cursor.com>

github-actions · 2026-07-24T03:08:12Z

💚 All backports created successfully

Status	Branch	Result
✅	8.19
✅	9.4
✅	9.5

Questions ?

Please refer to the Backport tool documentation and see the Github Action logs for details

) * Mark ML controller non-dumpable before accepting commands Same-UID peers in the shared ES/ml-cpp pod could write /proc/<controller-pid>/mem and overwrite access@GOT so a spawn's access(path, X_OK) became dlopen(path, RTLD_LAZY) (elastic/security#12621). Call prctl(PR_SET_DUMPABLE, 0) after logging is up and before opening command pipes; verify with PR_GET_DUMPABLE and fail closed if the boundary cannot be established. No-op on non-Linux platforms where /proc/<pid>/mem is not the concern. * Add changelog for #3081 * [ML] Improve PR_GET_DUMPABLE diagnostic in controller non-dumpable guard Distinguish a PR_GET_DUMPABLE syscall failure (-1) from the process still being dumpable, and log the actual value, per review feedback on #3081. * [ML] Remove public-to-private security-tracker references * [ML] Fix clang-format in controller Main.cc --------- (cherry picked from commit 9c46fa0) Co-authored-by: Ed Savage <ed.savage@elastic.co> Co-authored-by: Cursor <cursoragent@cursor.com>

edsavage added >bug ci:skip-es-tests labels Jul 21, 2026

Add changelog for elastic#3081

88ade68

Co-authored-by: Cursor <cursoragent@cursor.com>

elasticsearchmachine added the v9.6.0 label Jul 21, 2026

edsavage added the :ml label Jul 21, 2026

edsavage mentioned this pull request Jul 23, 2026

Harden model validation: reject __setstate__ hooks and forbid inductor::_reinterpret_tensor #3078

Open

3 tasks

edsavage marked this pull request as ready for review July 23, 2026 00:48

edsavage requested a review from Copilot July 23, 2026 00:52

Copilot started reviewing on behalf of edsavage July 23, 2026 00:52 View session

Copilot AI reviewed Jul 23, 2026

View reviewed changes

edsavage mentioned this pull request Jul 23, 2026

Reject non-native ABIs in ML seccomp BPF filter #3080

Merged

2 tasks

edsavage requested a review from valeriy42 July 23, 2026 01:42

edsavage added the ci:run-pytorch-tests label Jul 23, 2026

valeriy42 approved these changes Jul 23, 2026

View reviewed changes

Comment thread bin/controller/Main.cc Outdated

edsavage and others added 2 commits July 24, 2026 10:41

[ML] Improve PR_GET_DUMPABLE diagnostic in controller non-dumpable guard

3de96c9

Distinguish a PR_GET_DUMPABLE syscall failure (-1) from the process still being dumpable, and log the actual value, per review feedback on elastic#3081. Co-authored-by: Cursor <cursoragent@cursor.com>

[ML] Remove public-to-private security-tracker references

9b47f5e

Co-authored-by: Cursor <cursoragent@cursor.com>

edsavage added auto-backport Automatically merge backport PRs when CI passes v8.19.20 v9.4.5 v9.5.1 labels Jul 23, 2026

[ML] Fix clang-format in controller Main.cc

388d6a9

Co-authored-by: Cursor <cursoragent@cursor.com>

edsavage merged commit 9c46fa0 into elastic:main Jul 24, 2026
18 checks passed

github-actions Bot mentioned this pull request Jul 24, 2026

[8.19] Mark ML controller non-dumpable before accepting commands (#3081) #3107

Merged

github-actions Bot added the backport-pending label Jul 24, 2026

This was referenced Jul 24, 2026

[9.4] Mark ML controller non-dumpable before accepting commands (#3081) #3109

Merged

[9.5] Mark ML controller non-dumpable before accepting commands (#3081) #3111

Merged

This was referenced Jul 24, 2026

[ML] Publish ml-cpp-ci rollup commit status for PR builds #3112

Merged

[ML] Disable auto-merge of backport PRs; merge only after CI is green #3114

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Mark ML controller non-dumpable before accepting commands#3081

Mark ML controller non-dumpable before accepting commands#3081
edsavage merged 5 commits into
elastic:mainfrom
edsavage:security/controller-non-dumpable

edsavage commented Jul 21, 2026 •

edited

Loading

Uh oh!

elasticsearchmachine commented Jul 23, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

edsavage commented Jul 23, 2026

Uh oh!

edsavage commented Jul 23, 2026

Uh oh!

valeriy42 left a comment

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jul 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

edsavage commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Fix

Test plan

Uh oh!

elasticsearchmachine commented Jul 23, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

edsavage commented Jul 23, 2026

Test plan verification

1 & CI — controller starts and processes commands normally

2 — non-dumpable after start (Linux)

3 — fail-closed (mocked)

Uh oh!

edsavage commented Jul 23, 2026

Uh oh!

valeriy42 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jul 24, 2026

💚 All backports created successfully

Questions ?

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

edsavage commented Jul 21, 2026 •

edited

Loading