codegen_ssa: pack small const aggregates into immediate stores by ChuanqiXu9 · Pull Request #157690 · rust-lang/rust

ChuanqiXu9 · 2026-06-10T02:56:38Z

For small repr(C) aggregates with padding, direct constant
initialization can still lower into field-wise construction plus
memcpy. That leaves the backend to rediscover that the whole object
is a single constant byte pattern.

This is especially visible for non-zero constant aggregates. Instead
of materializing them as separate field stores, we want codegen_ssa to
emit the packed value directly. For example, a value like

InnerPadded { a: 0, b: 1, c: 0 }

can otherwise lower to something like

store i16 0, ptr %val, align 4
store i8 1, ptr %val_plus_2, align 2
store i32 0, ptr %val_plus_4, align 4
call void @llvm.memcpy(..., ptr %val, ...)

while this change produces

store i64 65536, ptr %val, align 4
call void @llvm.memcpy(..., ptr %val, ...)

Why not solve this in LLVM?

At the problematic lowering point, rustc still knows that the MIR
aggregate is small and fully constant. LLVM only sees a lowered stack
temporary built from per-field stores and then copied out. Recovering
that packed constant there would require rediscovering front-end
aggregate semantics after lowering, so emitting the packed store in
rustc is the simpler and more local fix.

Why not keep the previous typed-copy approach?

An earlier approach zeroed padding on typed copies whose source could
be traced back to a constant assignment. That helped some cases, but it
also widened the optimization to runtime copy paths and could introduce
extra runtime stores purely to maintain padding knowledge. That is not
an acceptable tradeoff here.

Keep the scope narrow instead: only handle direct MIR aggregates whose
fields are all constants, and pack them according to the target
endianness before emitting a single integer store.

ChuanqiXu9 · 2026-06-12T03:09:00Z

r? rust-lang/codegen

dianqk · 2026-06-12T03:52:35Z

@bors try @rust-timer queue

[Optimization] Zero padding in typed copies to help LLVM merge stores

dianqk

I think we should leave this to LLVM. Maybe we can emit store undef in SROA and let instcombine or memcpyopt consume it.

View changes since this review

rust-bors · 2026-06-12T06:01:05Z

☀️ Try build successful (CI)
Build commit: 68b77d3 (68b77d32f1786d41b84fa6dfba36f5c3809b1f9f, parent: b30f3df3ba3c4c9de2f58f1a75dd9500b79b3f8d)

rust-timer · 2026-06-12T06:41:59Z

Finished benchmarking commit (68b77d3): comparison URL.

Overall result: ❌ regressions - please read:

Benchmarking means the PR may be perf-sensitive. It's automatically marked not fit for rolling up. Overriding is possible but disadvised: it risks changing compiler perf.

Next, please: If you can, justify the regressions found in this try perf run in writing along with @rustbot label: +perf-regression-triaged. If not, fix the regressions and do another perf run. Neutral or positive results will clear the label automatically.

@bors rollup=never
@rustbot label: -S-waiting-on-perf +perf-regression

Instruction count

Our most reliable metric. Used to determine the overall result above. However, even this metric can be noisy.

	mean	range	count
Regressions ❌ (primary)	0.4%	[0.2%, 0.6%]	16
Regressions ❌ (secondary)	0.4%	[0.2%, 0.6%]	10
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	0.4%	[0.2%, 0.6%]	16

Max RSS (memory usage)

Results (secondary 2.3%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	2.3%	[2.3%, 2.3%]	1
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	-	-	0

Cycles

This perf run didn't have relevant results for this metric.

Binary size

Results (primary 0.1%, secondary 0.3%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	0.1%	[0.0%, 0.2%]	22
Regressions ❌ (secondary)	0.3%	[0.0%, 0.4%]	8
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	0.1%	[0.0%, 0.2%]	22

Bootstrap: 518.98s -> 524.616s (1.09%)
Artifact size: 400.92 MiB -> 400.88 MiB (-0.01%)

theemathas · 2026-06-17T13:00:14Z

What exactly is nesting structs doing that affects this optimization? See #157373 (comment)

ChuanqiXu9 · 2026-06-24T07:33:19Z

I think we should leave this to LLVM. Maybe we can emit store undef in SROA and let instcombine or memcpyopt consume it.

View changes since this review

Yeah, it makes sense to do this in LLVM too. I just thought it is still meaningful to do this in rust. As I tried to make the patch as small as possible.

I was trying to train my self to get familiar with the rust compiler setups. So if you think it is meaningless to do this in rust side, please let me know and I'll try to look at other stuffs. (I'll be happy if you can tell me what is worth next).

ChuanqiXu9 · 2026-06-24T07:35:48Z

As the last implementation shows some runtime regressions, I think it is not good. I rewrote the whole patch to avoid any runtime regressions. The high level view of the patch is, we try to recognize the pattern in the MIR level, and if we find it, we will try to transform it into a single store explicitly.

ChuanqiXu9 · 2026-06-24T08:56:10Z

What exactly is nesting structs doing that affects this optimization? See #157373 (comment)

I feel this is the same. With nested struct, it has some temp variables. But these temp variables helps the LLVM to understand that it is a whole object. But without the nested struct, LLVM's SROA inserted some store undef and finally makes the other pass failed to recognize it is a whole object.

My thought is, this shows some complexities and randomness of middle end optimizations. Yeah, we can blame that. But it is still complex and randomness. To avoid such cases, what the current patch did may be meaningful. We did the optimization we want in our part. We introduces the determinism.

dianqk · 2026-06-25T01:22:29Z

The new approach sounds good to me, although I haven’t read it thoroughly yet.
@bors try @rust-timer queue

[Optimization] Zero padding in typed copies to help LLVM merge stores

rust-bors · 2026-06-25T03:42:25Z

☀️ Try build successful (CI)
Build commit: b506142 (b50614215251a44128e69d72272cc8f1bfe4ae05)
Base parent: f28ac76 (f28ac764c36004fa6a6e098d15b4016a838c13c6)

ChuanqiXu9 · 2026-07-08T08:11:27Z

An earlier approach zeroed padding on typed copies whose source could
be traced back to a constant assignment. That helped some cases, but it
also widened the optimization to runtime copy paths and could introduce
extra runtime stores purely to maintain padding knowledge. That is not
an acceptable tradeoff here.

Is that not the current approach any more? It's still in the PR title.

@bors r- so we can avoid confusing commit messages permanently living in our history.

Thanks for the review. I've updated the title.

RalfJung · 2026-07-08T09:54:37Z

@bors r=dianqk
Thanks :)

rust-bors · 2026-07-08T09:54:40Z

📌 Commit 08d5ee3 has been approved by dianqk

It is now in the queue for this repository.

JonathanBrouwer · 2026-07-08T10:19:10Z

@bors rollup=iffy
Perf regression has been solved

codegen_ssa: pack small const aggregates into immediate stores Close rust-lang#157373 For small repr(C) aggregates with padding, direct constant initialization can still lower into field-wise construction plus memcpy. That leaves the backend to rediscover that the whole object is a single constant byte pattern. This is especially visible for non-zero constant aggregates. Instead of materializing them as separate field stores, we want codegen_ssa to emit the packed value directly. For example, a value like InnerPadded { a: 0, b: 1, c: 0 } can otherwise lower to something like store i16 0, ptr %val, align 4 store i8 1, ptr %val_plus_2, align 2 store i32 0, ptr %val_plus_4, align 4 call void @llvm.memcpy(..., ptr %val, ...) while this change produces store i64 65536, ptr %val, align 4 call void @llvm.memcpy(..., ptr %val, ...) Why not solve this in LLVM? At the problematic lowering point, rustc still knows that the MIR aggregate is small and fully constant. LLVM only sees a lowered stack temporary built from per-field stores and then copied out. Recovering that packed constant there would require rediscovering front-end aggregate semantics after lowering, so emitting the packed store in rustc is the simpler and more local fix. Why not keep the previous typed-copy approach? An earlier approach zeroed padding on typed copies whose source could be traced back to a constant assignment. That helped some cases, but it also widened the optimization to runtime copy paths and could introduce extra runtime stores purely to maintain padding knowledge. That is not an acceptable tradeoff here. Keep the scope narrow instead: only handle direct MIR aggregates whose fields are all constants, and pack them according to the target endianness before emitting a single integer store.

Rollup of 25 pull requests Successful merges: - #158871 (add relnotes for 1.97.0) - #158968 (stdarch subtree update) - #157690 (codegen_ssa: pack small const aggregates into immediate stores) - #158541 (Move `std::io::Write` to `core::io`) - #154445 (rustdoc: Represent `--output-format=json` coverage and ir differently) - #156370 (Reject linked dylib EII default overrides) - #157153 (allow `Allocator`s to be used as `#[global_allocator]`s) - #158495 (Rename HAS_CT_PROJECTION to HAS_CONST_ALIAS) - #158617 (allow mGCA const arguments to fall back to anon consts) - #158645 (Fix splat ICEs and ban it in closures) - #158655 (Fix coroutine MIR saved local remapping) - #158666 (Carry the `b_offset` inside `BackendRepr::ScalarPair`) - #158870 (std: merge the unix-like io::error modules into one file) - #158920 (Update wasm-component-ld to 0.5.26) - #158926 (wrapping_sh* methods: clarify underspecified reference) - #158927 (add core test run with `-Zforce-intrinsic-fallback`) - #158932 (Do not build the compiler when invoking `x perf compare`) - #158937 (Emit the emscripten entry point as `__main_argc_argv`) - #151379 (Stabilize `VecDeque::retain_back` from `truncate_front`) - #156144 (Better docs for PartialEq (includes macro rename)) - #156548 ( Library support for aarch64-unknown-linux-pauthtest target) - #158307 (CI job for parallel frontend ui tests) - #158347 (Improve generic parameters handling for #[diagnostic::on_const]) - #158722 (delegation: do not always inherit `ConstArgHasType` predicates) - #158741 (Simplify `Option::into_flat_iter` signature)

tmiasko · 2026-07-09T06:09:30Z

It looks like debug assertion added here failed in the rollup #158986 (comment):

2:48.1480834Z thread 'rustc' (94320) panicked at compiler/rustc_codegen_ssa/src/mir/rvalue.rs:36:9:
2026-07-09T00:32:48.1489684Z assertion `left == right` failed
2026-07-09T00:32:48.1497701Z   left: 1
2026-07-09T00:32:48.1505838Z  right: 0

@bors r-

rust-bors · 2026-07-09T06:09:35Z

This pull request was unapproved.

View changes since this unapproval

rustbot · 2026-07-09T08:17:55Z

⚠️ Warning ⚠️

This PR is based on an upstream commit that is older than 28 days.

It's recommended to update your branch according to the rustc-dev-guide.

ChuanqiXu9 · 2026-07-09T08:18:46Z

It looks like debug assertion added here failed in the rollup #158986 (comment):

2:48.1480834Z thread 'rustc' (94320) panicked at compiler/rustc_codegen_ssa/src/mir/rvalue.rs:36:9:
2026-07-09T00:32:48.1489684Z assertion `left == right` failed
2026-07-09T00:32:48.1497701Z   left: 1
2026-07-09T00:32:48.1505838Z  right: 0

@bors r-

Didn't get the case for variant right. Now I've updated it and the test now cover the new case.

For small repr(C) aggregates with padding, direct constant initialization can still lower into field-wise construction plus memcpy. That leaves the backend to rediscover that the whole object is a single constant byte pattern. This is especially visible for non-zero constant aggregates. Instead of materializing them as separate field stores, we want codegen_ssa to emit the packed value directly. For example, a value like InnerPadded { a: 0, b: 1, c: 0 } can otherwise lower to something like store i16 0, ptr %val, align 4 store i8 1, ptr %val_plus_2, align 2 store i32 0, ptr %val_plus_4, align 4 call void @llvm.memcpy(..., ptr %val, ...) while this change produces store i64 65536, ptr %val, align 4 call void @llvm.memcpy(..., ptr %val, ...) Why not solve this in LLVM? At the problematic lowering point, rustc still knows that the MIR aggregate is small and fully constant. LLVM only sees a lowered stack temporary built from per-field stores and then copied out. Recovering that packed constant there would require rediscovering front-end aggregate semantics after lowering, so emitting the packed store in rustc is the simpler and more local fix. Why not keep the previous typed-copy approach? An earlier approach zeroed padding on typed copies whose source could be traced back to a constant assignment. That helped some cases, but it also widened the optimization to runtime copy paths and could introduce extra runtime stores purely to maintain padding knowledge. That is not an acceptable tradeoff here. Keep the scope narrow instead: only handle direct MIR aggregates whose fields are all constants, and pack them according to the target endianness before emitting a single integer store. Add a focused codegen test covering the original PR 157690 entry points together with non-zero constant cases. The test uses -Cno-prepopulate-passes so it checks the immediate-store shape directly at rustc codegen time, instead of depending on later LLVM store merging.

dianqk · 2026-07-09T13:27:31Z

@bors try jobs=aarch64-gnu-debug,test-various

codegen_ssa: pack small const aggregates into immediate stores try-job: aarch64-gnu-debug try-job: test-various

rust-log-analyzer · 2026-07-09T14:40:16Z

The job test-various failed! Check out the build log: (web) (plain enhanced) (plain)

Click to see the possible cause of the failure (guessed by this bot)

test [debuginfo-gdb] tests/debuginfo/zst-interferes-with-prologue.rs ... ok

failures:

---- [debuginfo-gdb] tests/debuginfo/function-call.rs stdout ----
NOTE: compiletest thinks it is using GDB version 12001000
------gdb stdout------------------------------
GNU gdb (Ubuntu 12.1-0ubuntu1~22.04.2) 12.1
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word".
Breakpoint 1 at 0x9532: file /checkout/tests/debuginfo/function-call.rs, line 36.

Breakpoint 1, function_call::main () at /checkout/tests/debuginfo/function-call.rs:36
36     zzz(); // #break

------gdb stderr------------------------------
/checkout/obj/build/x86_64-unknown-linux-gnu/test/debuginfo/function-call.gdb/function-call.debugger.script:11: Error in sourced command file:
Couldn't write extended state status: Bad address.

------------------------------------------

error: gdb failed to execute
status: exit status: 1
command: PYTHONPATH="/checkout/src/etc" "/usr/bin/gdb" "-quiet" "-batch" "-nx" "-command=/checkout/obj/build/x86_64-unknown-linux-gnu/test/debuginfo/function-call.gdb/function-call.debugger.script"
--- stdout -------------------------------
GNU gdb (Ubuntu 12.1-0ubuntu1~22.04.2) 12.1
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word".
Breakpoint 1 at 0x9532: file /checkout/tests/debuginfo/function-call.rs, line 36.

Breakpoint 1, function_call::main () at /checkout/tests/debuginfo/function-call.rs:36
36     zzz(); // #break
------------------------------------------
--- stderr -------------------------------
/checkout/obj/build/x86_64-unknown-linux-gnu/test/debuginfo/function-call.gdb/function-call.debugger.script:11: Error in sourced command file:
Couldn't write extended state status: Bad address.
------------------------------------------

rust-bors · 2026-07-09T14:43:33Z

💔 Test for 46695be failed: CI. Failed job:

try - test-various (web logs, enhanced plaintext logs)

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels Jun 10, 2026

This comment has been minimized.

Sign in to view

ChuanqiXu9 force-pushed the PaddingZero branch from 0c29089 to de5e31b Compare June 10, 2026 05:42

This comment has been minimized.

Sign in to view

rustbot assigned dianqk Jun 12, 2026

This comment has been minimized.

Sign in to view

rustbot added the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Jun 12, 2026

This comment has been minimized.

Sign in to view

rust-bors Bot pushed a commit that referenced this pull request Jun 12, 2026

Auto merge of #157690 - ChuanqiXu9:PaddingZero, r=<try>

68b77d3

[Optimization] Zero padding in typed copies to help LLVM merge stores

dianqk reviewed Jun 12, 2026

View reviewed changes

Comment thread tests/codegen-llvm/aggregate-padding-zero.rs

This comment has been minimized.

Sign in to view

rustbot added perf-regression Performance regression. and removed S-waiting-on-perf Status: Waiting on a perf run to be completed. labels Jun 12, 2026

ChuanqiXu9 force-pushed the PaddingZero branch from de5e31b to 2434916 Compare June 24, 2026 07:26

ChuanqiXu9 force-pushed the PaddingZero branch from 2434916 to cbaf03b Compare June 24, 2026 08:32

This comment has been minimized.

Sign in to view

rustbot added the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Jun 25, 2026

rust-bors Bot pushed a commit that referenced this pull request Jun 25, 2026

Auto merge of #157690 - ChuanqiXu9:PaddingZero, r=<try>

b506142

[Optimization] Zero padding in typed copies to help LLVM merge stores

ChuanqiXu9 force-pushed the PaddingZero branch from 167c7ae to 08d5ee3 Compare July 8, 2026 08:09

ChuanqiXu9 changed the title ~~[Optimization] Zero padding in typed copies to help LLVM merge stores~~ codegen_ssa: pack small const aggregates into immediate stores Jul 8, 2026

rust-bors Bot added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Jul 8, 2026

JonathanBrouwer mentioned this pull request Jul 8, 2026

Rollup of 12 pull requests #158963

Closed

JonathanBrouwer mentioned this pull request Jul 8, 2026

Rollup of 14 pull requests #158971

Closed

jhpratt mentioned this pull request Jul 8, 2026

Rollup of 25 pull requests #158986

Closed

rust-bors Bot added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Jul 9, 2026

ChuanqiXu9 force-pushed the PaddingZero branch from 08d5ee3 to 1661071 Compare July 9, 2026 08:17

This comment has been minimized.

Sign in to view

ChuanqiXu9 force-pushed the PaddingZero branch from 1661071 to 87bb7c9 Compare July 9, 2026 08:53

This comment has been minimized.

Sign in to view

rust-bors Bot pushed a commit that referenced this pull request Jul 9, 2026

Auto merge of #157690 - ChuanqiXu9:PaddingZero, r=<try>

46695be

codegen_ssa: pack small const aggregates into immediate stores try-job: aarch64-gnu-debug try-job: test-various

Uh oh!

Uh oh!

Conversation

ChuanqiXu9 commented Jun 10, 2026 • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment has been minimized.

This comment has been minimized.

ChuanqiXu9 commented Jun 12, 2026

Uh oh!

dianqk commented Jun 12, 2026

Uh oh!

This comment has been minimized.

This comment has been minimized.

dianqk left a comment • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

rust-bors Bot commented Jun 12, 2026

Uh oh!

This comment has been minimized.

rust-timer commented Jun 12, 2026

Overall result: ❌ regressions - please read:

Instruction count

Max RSS (memory usage)

Cycles

Binary size

Uh oh!

theemathas commented Jun 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ChuanqiXu9 commented Jun 24, 2026

Uh oh!

ChuanqiXu9 commented Jun 24, 2026

Uh oh!

ChuanqiXu9 commented Jun 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dianqk commented Jun 25, 2026

Uh oh!

This comment has been minimized.

This comment has been minimized.

rust-bors Bot commented Jun 25, 2026

Uh oh!

ChuanqiXu9 commented Jul 8, 2026

Uh oh!

RalfJung commented Jul 8, 2026

Uh oh!

rust-bors Bot commented Jul 8, 2026

Uh oh!

JonathanBrouwer commented Jul 8, 2026

Uh oh!

tmiasko commented Jul 9, 2026

Uh oh!

rust-bors Bot commented Jul 9, 2026

Uh oh!

rustbot commented Jul 9, 2026

Uh oh!

ChuanqiXu9 commented Jul 9, 2026

Uh oh!

This comment has been minimized.

dianqk commented Jul 9, 2026

Uh oh!

This comment has been minimized.

rust-log-analyzer commented Jul 9, 2026

Uh oh!

rust-bors Bot commented Jul 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

ChuanqiXu9 commented Jun 10, 2026 •

edited by rustbot

Loading

dianqk left a comment •

edited by rustbot

Loading

theemathas commented Jun 17, 2026 •

edited

Loading

ChuanqiXu9 commented Jun 24, 2026 •

edited

Loading