Skip to content

topology-aware: publish machine DRA devices - #877

Open
bart0sh wants to merge 3 commits into
stacked/bart0sh/DRA/015-socket-devicesfrom
stacked/bart0sh/DRA/016-tree-devices
Open

bart0sh wants to merge 3 commits into
stacked/bart0sh/DRA/015-socket-devicesfrom
stacked/bart0sh/DRA/016-tree-devices

Conversation

@bart0sh

@bart0sh bart0sh commented Oct 27, 2026 •

Copy link
Copy Markdown
Collaborator

topology-aware publishes a DRA device for each NUMA node and each socket. Now it also publishes cpudevmachine, with level machine, which has all CPUs of the machine. A claim on it is for any number of CPUs, when it does not matter which socket or NUMA node they come from. The policy picks them from one NUMA node when one has enough free CPUs, otherwise from one socket, otherwise from several sockets. Every level is published, also when a group has the same CPUs as the one above it, as on a machine with one socket. So a claim written for one level works on every machine.

The ID attributes of a device depend only on its level. A NUMA device has numaNode and socketID, a socket device has socketID, and the machine device has no IDs. This is also true when groups of different levels have the same CPUs, so a selector works the same way on every machine. Since the machine device has no socketID, a selector on socketID checks the level as well, as a selector on numaNode already does.

A device lists the PCIe roots of each socket that has some of its CPUs. A root that belongs to several sockets is listed once.

The move from #875 now works for groups at any level. When a NUMA claim needs CPUs that a socket claim holds, and a machine claim holds the other CPUs of the socket, the machine claim moves out of the socket first, then the socket claim moves. If the NUMA claim still fails, every moved claim gets its old CPUs back.

Known issue, the same as in #875: a move updates running containers while the resource manager holds its lock. Until #871 merges, containerd can time out and drop the plugin.

Testing:

  • go test -race -count=1 ./cmd/plugins/topology-aware/... ./pkg/resmgr/... passes.
  • golangci-lint v2.11.4 reports no issues.
  • e2e on Kubernetes 1.37 with an image built from this branch: test42, test43, test45 and test48 pass. test48 now also checks a machine claim larger than one socket, and a NUMA claim which moves a socket claim, which first moves a machine claim out of its socket. e2e does not run in CI.

@bart0sh
bart0sh added this pull request to stack #807 October 27, 2026 13:03
@bart0sh bart0sh changed the title stacked/bart0sh/DRA/016 tree devices topology-aware: publish machine DRA devices Oct 27, 2026
@bart0sh
bart0sh marked this pull request as draft October 27, 2026 13:04
@bart0sh
bart0sh force-pushed the stacked/bart0sh/DRA/016-tree-devices branch from cb238b9 to f6f00b9 Compare October 27, 2026 13:44
@bart0sh
bart0sh force-pushed the stacked/bart0sh/DRA/016-tree-devices branch from f6f00b9 to 1493307 Compare October 28, 2026 11:02
@bart0sh
bart0sh requested a balanced review from Copilot October 29, 2026 09:09

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Incorrect migration-headroom accounting can reject NUMA claims despite advertised capacity, while unfiltered PCIe-root advertising can expose sockets with no grantable CPUs.

2 open issues
What changed in this PR

Extends topology-aware DRA support with a machine-wide CPU device, building on socket-level allocation and claim migration.

Changes:

  • Publishes cpudevmachine with level-specific attributes and deduplicated PCIe roots.
  • Extends capacity accounting and claim migration across NUMA, socket, and machine levels.
  • Updates documentation, unit tests, and end-to-end coverage for machine claims and chained moves.
File Description
test/​e2e/​policies.test-suite/​topology-aware/​n4c16/​test48-dra-socket-devices/​machine-claim.yaml.in Adds a machine-level claim template.
test/​e2e/​policies.test-suite/​topology-aware/​n4c16/​test48-dra-socket-devices/​code.var.sh Tests machine capacity, cross-socket allocation, and chained migration.
test/​e2e/​policies.test-suite/​topology-aware/​n4c16/​test42-dra-devices/​code.var.sh Updates the expected device count.
docs/​resource-policy/​policy/​topology-aware.md Documents machine devices, selectors, and migration behavior.
cmd/​plugins/​topology-aware/​policy/​topology-aware-policy.go Initializes topology CPU groups during setup.
cmd/​plugins/​topology-aware/​policy/​dra.go Implements machine devices and recursive claim migration.
cmd/​plugins/​topology-aware/​policy/​dra_test.go Adds coverage for machine devices, attributes, capacity, and migration rollback.

Review effort: Balanced

@bart0sh
bart0sh force-pushed the stacked/bart0sh/DRA/016-tree-devices branch from 1493307 to baad308 Compare October 29, 2026 10:30
@bart0sh
bart0sh force-pushed the stacked/bart0sh/DRA/016-tree-devices branch from baad308 to 71bae60 Compare October 29, 2026 11:51
@bart0sh
bart0sh force-pushed the stacked/bart0sh/DRA/016-tree-devices branch from 71bae60 to fc38e72 Compare October 29, 2026 12:43
@bart0sh
bart0sh force-pushed the stacked/bart0sh/DRA/016-tree-devices branch from fc38e72 to bf8d8eb Compare October 30, 2026 10:53
@bart0sh
bart0sh force-pushed the stacked/bart0sh/DRA/016-tree-devices branch from bf8d8eb to e1e327e Compare October 30, 2026 12:36
@bart0sh
bart0sh requested a balanced review from Copilot October 30, 2026 12:40

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

A critical mixed-result rollback cycle can leave both claims displaced despite restorable original allocations, requiring a rollback fix and regression test.

1 open issue
2 resolved since last review

Review effort: Balanced

@bart0sh
bart0sh force-pushed the stacked/bart0sh/DRA/016-tree-devices branch 2 times, most recently from bb6a314 to 9b55bd6 Compare October 30, 2026 14:20
@bart0sh
bart0sh requested a balanced review from Copilot October 30, 2026 14:29

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

CPU selection can unnecessarily span NUMA nodes despite a suitable memoryless node, and failed rollback attempts can leave claims displaced because restoration dependencies are not retried.

2 open issues
1 resolved since last review

Review effort: Balanced

@bart0sh
bart0sh force-pushed the stacked/bart0sh/DRA/016-tree-devices branch from 9b55bd6 to cf89a97 Compare November 2, 2026 10:44
@bart0sh
bart0sh force-pushed the stacked/bart0sh/DRA/016-tree-devices branch from cf89a97 to 70136ca Compare November 2, 2026 11:55
@bart0sh
bart0sh requested a balanced review from Copilot November 2, 2026 12:12

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues block approval of the machine-wide CPU device support.

No open issues 🎉

2 resolved since last review

Review effort: Balanced

@bart0sh
bart0sh force-pushed the stacked/bart0sh/DRA/016-tree-devices branch 2 times, most recently from 2f1840d to b6d9f44 Compare November 2, 2026 13:42
@bart0sh
bart0sh force-pushed the stacked/bart0sh/DRA/016-tree-devices branch from b6d9f44 to 6d66303 Compare November 2, 2026 16:27
Publish a device for the machine next to the NUMA node and socket
devices, also when it has the same CPUs as a socket. A NUMA node device
has the IDs of its node and socket, a socket device the ID of its
socket, and the machine device no IDs.

Assisted-by: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Ed Bartosh <eduard.bartosh@intel.com>
When a claim moves for a claim on a smaller device, and the CPUs it could
move to are held by claims on larger devices, move those claims out of its
device first.

Assisted-by: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Ed Bartosh <eduard.bartosh@intel.com>
A machine claim gets CPUs of both sockets, and moves out of a socket
when a socket claim has to move there.

Assisted-by: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Ed Bartosh <eduard.bartosh@intel.com>
@bart0sh
bart0sh force-pushed the stacked/bart0sh/DRA/016-tree-devices branch from 6d66303 to a4a33d6 Compare November 2, 2026 20:31
@bart0sh
bart0sh marked this pull request as ready for review November 3, 2026 07:05

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants