Shared memory spilling and better inlining policies

[Qubitol]

This PR implements two things:

• shared memory spilling feature of CUDA 13: registers will prefer shared memory when spilling
• better inlining policies for the HELAS routines: it is possible to select in between 3 behaviours by using the following env variable before compilation:
  • HELINL=-1: prevent inlining
  • HELINL=0 (default): compiler default
  • HELINL=1: forced inlining

Performance comparison

Regarding performance comparison before/after, I tested the 3 processes:

• g g > t t~ g
• g g > t t~ g g
• g g > t t~ g g g

in standalone mode, measuring check_sa throughput, and register spilling, fixing 256 threads per block, and by varying number of blocks.
I analysed all the possible combinations across inlining policy and shared memory spilling dimensions, testing both CUDA and AVX2 backends (for CPU backend of course shared memory spilling is not a thing).

Benchmark throughput across work size

Throughput change due to inlining (heatmap)

Throughput change due to shared memory spilling enabled (heatmap)

Register pressure and spill destination

Most kernels use the architectural maximum of 255 registers/thread.
The exceptions are no-inline $+1g$ at 130 and no-inline $+2g$ at 242; no-inline $+3g$ remains at 255.

When shared spilling is active, the compiler reserves 19,456 bytes/block.
It allocates nothing for no-inline $+1g$ and $+2g$, where there are no compiler-reported spill loads/stores to redirect.

Conclusions

• There is no universally best inlining policy.
  Compiler-default is best for `g g > t t~ g g g$, no-inline is strong for $g g > t t~ g g$.
• The shared-memory-spilling pragma does what it is supposed to do: when it is active, compiler-reported local spill traffic falls and about 19 KiB of shared memory is allocated per block.
  The effect on end-to-end benchmark throughput is usually modest.

What's next?

• Merging the shared memory spilling pragma would require some work to make the madevent side work, since as of today it doesn't compile: to be honest, I don't know if it is worth merging given the little effect it seems to have.
• The inlining policies are now better defined, however, inlining seems to not affect the throughput except for some cases: either we merge in order to have the feature and then we can decide to test it to see if some processes are running faster, or we don't merge, and we force compiler defaults (incidentally removing also what was implemented before).

If we decide to merge only one single feature, I would create a separate branch and PR, while closing this.

[roiser]

LGTM, I just looked at the merge conflict, should be as easy as removing the ALWAYS_INLINE as the attribute can be put in one place no matter if its CPU or GPU compilation.