[Security] Hypervisor NonBlockingDCache does not invalidate DTLB for HFENCE.VVMA/GVMA #3817

Hello Rocket Chip maintainers,

I tried to report this privately through security@lists.chipsalliance.org, but multiple delivery attempts were rejected by the groups.io mail server as spam, including an attempt without attachments.

Summary

In Rocket Chip configurations that enable both Hypervisor support and NonBlockingDCache, NonBlockingDCache appears not to assert the DTLB sfence/invalidation valid signal for internal hypervisor fence commands M_HFENCEV and M_HFENCEG.

This can leave stale VS-stage or G-stage data-side translations usable after mappings are revoked or remapped and the corresponding hypervisor fence is executed.

Tested commit:

55bcad0f59436de98ea510334121de8546b9e9d7

Affected configuration:

HypervisorNonblockingRocketConfig
WithHypervisor + WithL1DCacheNonblocking

Root Cause

DCache.scala and NBDcache.scala handle the DTLB sfence path differently.

In the blocking DCache path, hypervisor fence commands are included in s1_sfence:

val s1_sfence =
  s1_req.cmd === M_SFENCE ||
  s1_req.cmd === M_HFENCEV ||
  s1_req.cmd === M_HFENCEG

But in NonBlockingDCache, only M_SFENCE is treated as sfence-valid:

val s1_sfence = s1_req.cmd === M_SFENCE

Later, NonBlockingDCache still wires the hypervisor fence bits:

dtlb.io.sfence.valid := s1_valid && !io.cpu.s1_kill && s1_sfence
dtlb.io.sfence.bits.hv := s1_req.cmd === M_HFENCEV
dtlb.io.sfence.bits.hg := s1_req.cmd === M_HFENCEG

Because dtlb.io.sfence.valid is not asserted for M_HFENCEV or M_HFENCEG, the hv / hg bits cannot take effect for those commands.

The generated RTL shows the same behavior in HypervisorNonblockingRocketConfig:

.io_sfence_valid   (s1_valid & ~io_cpu_s1_kill & s1_req_cmd == 5'h14)
.io_sfence_bits_hv (s1_req_cmd == 5'h15)
.io_sfence_bits_hg (s1_req_cmd == 5'h16)

So M_SFENCE (5'h14) asserts valid, but M_HFENCEV (5'h15) and M_HFENCEG (5'h16) do not.

Validation

I validated this with focused bare-metal tests comparing Spike, Rocket with blocking DCache, and Rocket with NonBlockingDCache.

Result matrix:

VS-stage stale translation test:
- Spike rv64gc_h: PASS
- Rocket HypervisorRocketConfig with blocking DCache: PASS
- Rocket HypervisorNonblockingRocketConfig: stale translation reproduced

G-stage stale translation test:
- Spike rv64gc_h: PASS
- Rocket HypervisorRocketConfig with blocking DCache: PASS
- Rocket HypervisorNonblockingRocketConfig: stale translation reproduced

The NonBlockingDCache failures are consistent with the missing DTLB invalidation path for M_HFENCEV / M_HFENCEG.

Impact

Confirmed impact in the tested configuration:

• stale VS-stage data translation reuse after virtualized SFENCE.VMA / internal M_HFENCEV
• stale G-stage data translation reuse after HFENCE.GVMA / internal M_HFENCEG
• VM isolation boundary impact is possible when stale G-stage translations survive after HS remaps or revokes guest physical mappings

Suggested Fix

At minimum, NonBlockingDCache should match the blocking DCache behavior:

val s1_sfence =
  s1_req.cmd === M_SFENCE ||
  s1_req.cmd === M_HFENCEV ||
  s1_req.cmd === M_HFENCEG

Suggested regression coverage:

• assert that dtlb.io.sfence.valid is asserted for M_HFENCEV
• assert that dtlb.io.sfence.valid is asserted for M_HFENCEG
• add a VS-stage remap/revoke test followed by SFENCE.VMA
• add a G-stage remap/revoke test followed by HFENCE.GVMA

I can provide the full validation logs and minimal reproducers if maintainers provide a working private channel or confirm that they are comfortable receiving them here.