docs(lvm-local-storage): document chunk size + expand tuning section by abonillabeeche · Pull Request #1083 · harvester/docs

abonillabeeche · 2026-07-14T12:42:11Z

Summary

Follow-up to csi-driver-lvm PR #57 — documents the new chunkSize, poolMetadataSize, and zeroBlocks StorageClass parameters and expands the existing "Tuning the dm-thin Pool" section (introduced in #1075) with:

NEW subsection: Chunk size — the primary content of this PR. Explains why the default is 1M (aligns with the RAID full-stripe width common on Harvester hardware, avoiding partial-stripe read-modify-write and, on parity RAID, write-hole exposure), when to override (match your RAID full stripe; 512K/128K only for sub-stripe or non-RAID VGs, never >2M), the hard "cannot change after pool creation" constraint, and how to verify the effective chunk size of a live pool.
Chunk zeroing — expanded to mention the zeroBlocks StorageClass parameter (in addition to the existing post-creation lvchange --zero n workflow).
Pool metadata size — expanded to mention the poolMetadataSize StorageClass parameter (default 16G covers 256 TB pools at 1M chunks) and adds the metadata sizing formula.
NEW: Full example StorageClass — shows all three parameters set explicitly.

Motivation

A field customer running everything from our runbook ended up with a 16 MiB chunk_size on their thin pool. LVM's default auto-select scales chunk_size with pool size to keep metadata bounded, which for multi-TB pools produces 8-16 MiB chunks — appropriate for metadata sizing but disastrous for random-IOPS workloads (up to 4096:1 allocation write amplification on 4K writes to virgin regions).

Customer's measured DI_RANDOM: 326 IOPS. Our lab measurement on hardware in the same performance class with an explicit, stripe-aligned chunk size + zero=n: 15,000+ QD1 4K IOPS. The gap is entirely explained by chunk size + zero-on-allocate.

Test plan

Local Docusaurus render (npm start) — page renders, no broken links
Markdown syntax validated
Docs CI (markdownlint + link check) — will run on this PR

Base branch note

This PR includes the "Tuning the dm-thin Pool" section from PR #1075 (APPROVED + MERGEABLE) because it builds on that section. Please merge #1075 first — this PR will show a clean diff (just the chunk-size additions) once it rebases against post-#1075 main.

Scope

Only edits docs/advanced/addons/lvm-local-storage.md (current-dev / v1.9 tree). Not backported to versioned_docs/ because the chunkSize StorageClass parameter ships in a CSI driver version targeting Harvester v1.9. The lvchange/lvextend post-creation workflows work on older versions but were already covered by #1075.

References

csi-driver-lvm PR #57 — driver + chart changes this documents
LVM upstream performance policy — the 512 KiB general-purpose seed (Harvester defaults to 1M for RAID full-stripe alignment)
Linux kernel dm-thin admin guide — "512 KiB general; 64 KiB snapshot-heavy"
Red Hat Gluster admin guide — the 2 MiB cap "to reduce COW problems"

🤖 Generated with Claude Code

Add a brief comparison of the two volume group types to help operators pick the right StorageClass for their workload, and append a small "Additional Notes" section covering thin-pool tuning, VM disk-bus recommendation, and Longhorn v2 coexistence. Signed-off-by: Alejandro Bonilla <abonilla@suse.com>

Co-authored-by: Jillian Maroket <67180770+jillian-maroket@users.noreply.github.com> Signed-off-by: Alejandro Bonilla <abonilla@suse.com>

Jillian moved the striped vs dm-thin explanation into the procedure step above (Volume Group Type), so the standalone section is now duplicate content. The "For more information, see [StorageClass]" link is kept. Co-authored-by: Jillian Maroket <67180770+jillian-maroket@users.noreply.github.com> Signed-off-by: Alejandro Bonilla <abonilla@suse.com>

Field investigation showed that Harvester's LVM CSI driver, when creating a new thin pool, was letting LVM auto-select the chunk_size, which for multi-TB pools produces 8-16 MiB chunks. On random 4K workloads this causes severe write amplification (up to 4096:1). A field customer measured DI_RANDOM = 326 IOPS on hardware capable of 10x that, traced to a 16 MiB chunk size. harvester/csi-driver-lvm PR harvester#57 (still in review) exposes three new StorageClass parameters (chunkSize, poolMetadataSize, zeroBlocks) and ships sensible defaults (512K/16G/false). This docs change explains the choice, when to override, the hard "cannot change after pool creation" constraint, and gives a full example StorageClass with all three parameters set. Also expands the existing chunk-zeroing and pool-metadata subsections to reference the new StorageClass parameters (in addition to the post-creation lvchange/lvextend workflows they already documented). Only edits the current-dev docs/ tree - not backported to versioned_docs/, since the StorageClass parameters ship in a CSI driver version that will target Harvester v1.9. The post-creation lvchange/lvextend commands work on older Harvester versions, but the chunkSize parameter itself does not. Base is docs-lvm-considerations (PR harvester#1075), which introduces the initial "Tuning the dm-thin Pool" section. Rebasing onto main once harvester#1075 merges. Signed-off-by: Alejandro Bonilla <abonilla@suse.com>

…stripe alignment Recommend 1M as the default dm-thin chunk size so it matches the full-stripe width of common hardware-RAID layouts. A sub-stripe chunk (e.g. 512K on a 1M stripe) forces partial-stripe read-modify-write and, on parity RAID (5/6), widens the write-hole window, risking parity inconsistency on unclean shutdown without a protected controller cache. Update the values table, metadata sizing note, and example StorageClass accordingly. Signed-off-by: Alejandro Bonilla <abonilla@suse.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

mergify · 2026-07-30T04:10:42Z

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

Queue this pull request

abonillabeeche · 2026-08-06T00:33:15Z

@jillian-maroket PING

jillian-maroket · 2026-08-06T11:14:49Z

@abonillabeeche I'll work on this next week. Too many release deliverables.

jillian-maroket

Review done

Accept all 13 pending suggested edits from the 2026-08-11 review: tighten wording throughout the chunk-size/pool-metadata/chunk-zeroing guidance, restructure the value-selection table, and align the striped-vs-dm-thin section (main doc and v1.8 versioned copy) with the house style. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: Alejandro Bonilla <abonilla@suse.com>

abonillabeeche · 2026-09-15T14:51:51Z

All 13 pending suggestions from the 2026-08-11 review are applied in 839f918 — accepted as written except where a suggestion's context had already shifted slightly from later commits, in which case I matched the surrounding text to the intent of the suggestion. Covers both the main doc's chunk-size/pool-metadata rewording and the v1.8 versioned copy's striped-vs-dm-thin section and Longhorn coexistence heading.

@jillian-maroket please take another look when you have a chance.

WebberHuang1118

Please address the inline documentation concerns before publishing this guidance, thanks.

WebberHuang1118 · 2026-09-16T10:59:32Z

-1. Click **Save**.
+    - **Volume Group Type**: Select a type based on how the workload uses snapshots and how pool capacity is allocated. Harvester supports the following options:
+
+        - **striped**: Best suited for workloads requiring direct, high-performance volume access distributed across the physical devices in the volume group. Each logical volume is fully allocated its requested capacity at provisioning time. Snapshots are created as independent logical volumes sized to match the source volume's maximum capacity. For example, a snapshot of a 100 GiB volume reserves an additional 100 GiB of volume group space upon creation, regardless of the actual quantity of data written to the source..


Could we avoid recommending striped for new workloads here? harvester/harvester#11334 tracks its deprecation before GA and identifies dm-thin as the supported type. Please label striped as deprecated and outside GA scope, recommend dm-thin for new configurations, and add compatibility or migration guidance for existing pre-GA striped volumes.

WebberHuang1118 · 2026-09-16T10:59:32Z

+
+- **Chunk zeroing**: By default, the thin pool writes zeros to each newly allocated block chunk before exposing it to a write operation. On single-tenant clusters, you can disable chunk zeroing to significantly reduce write amplification during initial data allocations.
+
+  Set `zeroBlocks: "false"` on the StorageClass at pool creation time, or apply it to an existing pool with:


Please remove the instructions to run lvchange and lvextend on a live CSI-managed thin pool. Direct pool changes can conflict with attached volumes and driver operations. Any manual offline procedure should require that all volumes are detached and no corresponding dm devices exist before changing the pool.

WebberHuang1118 · 2026-09-16T10:59:32Z

+  parameters:
+    type: dm-thin
+    vgName: vmvg
+    chunkSize: "1M"            # Applied on first-time pool creation only.


These settings belong to the thin pool shared by all LVs in the volume group, not to an individual StorageClass. The first pool creation determines them, so presenting them as per-StorageClass tuning knobs is misleading and can lead to conflicting configurations. Please remove this example and recommendation; #11335 tracks a dedicated pool-level configuration and status resource.

WebberHuang1118 · 2026-09-16T10:59:32Z

+
+### Choosing a Virtual Machine Disk Bus
+
+When attaching an LVM CSI PersistentVolumeClaim to a VirtualMachine, `virtio-scsi` (`bus: scsi`) generally outperforms the default `virtio-blk` (`bus: virtio`) for sustained-write workloads on thin-provisioned pools, particularly on RAID-backed storage. `virtio-scsi` performs better because it supports multiple queues and uses a more efficient DISCARD path.


Could we remove this general performance recommendation or support it with a reproducible Harvester LVM CSI benchmark? QEMU recommends virtio-blk for performance-critical use cases, and KubeVirt supports multiqueue on virtio-blk and enables discard passthrough by default. Our Windows VM guide also reports better virtio-blk results on LVM CSI. We can describe SCSI-specific feature needs without claiming it generally writes faster.

WebberHuang1118 · 2026-09-16T10:59:32Z

+
+### Coexistence with Longhorn v2 Block-Mode Disks
+
+If the same node hosts a Longhorn V2 disk in block mode, the underlying device is held exclusively by the SPDK Instance Manager. You can add this device to the LVM `global_filter` to exclude it from LVM device scans and prevent resource conflicts.


Could we leave this coexistence guidance with @tserong as part of #11098? The issue is still under investigation and discusses a possible devices-file fix, while this section presents the global_filter workaround as a general user procedure. Please have the issue owner validate the supported guidance before publishing it.

abonillabeeche and others added 10 commits July 9, 2026 20:45

Update docs/advanced/addons/lvm-local-storage.md

4d047a8

Co-authored-by: Jillian Maroket <67180770+jillian-maroket@users.noreply.github.com> Signed-off-by: Alejandro Bonilla <abonilla@suse.com>

Update docs/advanced/addons/lvm-local-storage.md

f2730fb

Co-authored-by: Jillian Maroket <67180770+jillian-maroket@users.noreply.github.com> Signed-off-by: Alejandro Bonilla <abonilla@suse.com>

Update docs/advanced/addons/lvm-local-storage.md

959d31b

Co-authored-by: Jillian Maroket <67180770+jillian-maroket@users.noreply.github.com> Signed-off-by: Alejandro Bonilla <abonilla@suse.com>

Update docs/advanced/addons/lvm-local-storage.md

c94296d

Co-authored-by: Jillian Maroket <67180770+jillian-maroket@users.noreply.github.com> Signed-off-by: Alejandro Bonilla <abonilla@suse.com>

Update docs/advanced/addons/lvm-local-storage.md

3ac3f63

Co-authored-by: Jillian Maroket <67180770+jillian-maroket@users.noreply.github.com> Signed-off-by: Alejandro Bonilla <abonilla@suse.com>

Update docs/advanced/addons/lvm-local-storage.md

2e8b93c

Co-authored-by: Jillian Maroket <67180770+jillian-maroket@users.noreply.github.com> Signed-off-by: Alejandro Bonilla <abonilla@suse.com>

Update docs/advanced/addons/lvm-local-storage.md

3ca804e

Co-authored-by: Jillian Maroket <67180770+jillian-maroket@users.noreply.github.com> Signed-off-by: Alejandro Bonilla <abonilla@suse.com>

abonillabeeche requested review from a team and jillian-maroket as code owners July 14, 2026 12:42

abonillabeeche requested a review from ibrokethecloud July 14, 2026 12:42

ibrokethecloud previously approved these changes Jul 30, 2026

View reviewed changes

jillian-maroket reviewed Aug 11, 2026

View reviewed changes

This was referenced Sep 2, 2026

[FEATURE] Expose dm-thin pool configuration and capacity status through a custom resource harvester/harvester#11335

Open

feat: expose thin-pool chunk_size, metadata size, and zeroing as StorageClass params harvester/csi-driver-lvm#57

Open

abonillabeeche dismissed ibrokethecloud’s stale review via 839f918 September 15, 2026 14:51

Merge branch 'main' into docs-lvm-chunk-size

07715fe

jillian-maroket approved these changes Sep 15, 2026

View reviewed changes

Vicente-Cheng requested a review from WebberHuang1118 September 16, 2026 02:28

WebberHuang1118 reviewed Sep 16, 2026

View reviewed changes

WebberHuang1118 requested a review from tserong September 16, 2026 11:01


		- Chunk zeroing: By default, the thin pool writes zeros to each newly allocated block chunk before exposing it to a write operation. On single-tenant clusters, you can disable chunk zeroing to significantly reduce write amplification during initial data allocations.

		Set `zeroBlocks: "false"` on the StorageClass at pool creation time, or apply it to an existing pool with:


		### Choosing a Virtual Machine Disk Bus

		When attaching an LVM CSI PersistentVolumeClaim to a VirtualMachine, `virtio-scsi` (`bus: scsi`) generally outperforms the default `virtio-blk` (`bus: virtio`) for sustained-write workloads on thin-provisioned pools, particularly on RAID-backed storage. `virtio-scsi` performs better because it supports multiple queues and uses a more efficient DISCARD path.


		### Coexistence with Longhorn v2 Block-Mode Disks

		If the same node hosts a Longhorn V2 disk in block mode, the underlying device is held exclusively by the SPDK Instance Manager. You can add this device to the LVM `global_filter` to exclude it from LVM device scans and prevent resource conflicts.

Conversation

abonillabeeche commented Jul 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Motivation

Test plan

Base branch note

Scope

References

Uh oh!

mergify Bot commented Jul 30, 2026

Uh oh!

abonillabeeche commented Aug 6, 2026

Uh oh!

jillian-maroket commented Aug 6, 2026

Uh oh!

jillian-maroket left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

abonillabeeche commented Sep 15, 2026

Uh oh!

WebberHuang1118 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

WebberHuang1118 Sep 16, 2026

Choose a reason for hiding this comment

Uh oh!

WebberHuang1118 Sep 16, 2026

Choose a reason for hiding this comment

Uh oh!

WebberHuang1118 Sep 16, 2026

Choose a reason for hiding this comment

Uh oh!

WebberHuang1118 Sep 16, 2026

Choose a reason for hiding this comment

Uh oh!

WebberHuang1118 Sep 16, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

abonillabeeche commented Jul 14, 2026 •

edited

Loading

WebberHuang1118 left a comment •

edited

Loading