-
Notifications
You must be signed in to change notification settings - Fork 119
docs(lvm-local-storage): document chunk size + expand tuning section #1083
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from 11 commits
1f4ccc8
4d047a8
f2730fb
959d31b
c94296d
3ac3f63
2e8b93c
3ca804e
c36d7d7
72d8748
dc947a2
839f918
07715fe
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -98,7 +98,19 @@ You can only use one type of local volume in each volume group. If necessary, cr | |
|
|
||
|  | ||
|
|
||
| 1. Click **Save**. | ||
| - **Volume Group Type**: Select a type based on how the workload uses snapshots and how pool capacity is allocated. Harvester supports the following options: | ||
|
|
||
| - **striped**: Best suited for workloads requiring direct, high-performance volume access distributed across the physical devices in the volume group. Each logical volume is fully allocated its requested capacity at provisioning time. Snapshots are created as independent logical volumes sized to match the source volume's maximum capacity. For example, a snapshot of a 100 GiB volume reserves an additional 100 GiB of volume group space upon creation, regardless of the actual quantity of data written to the source.. | ||
|
|
||
| - **dm-thin**: Best suited for virtual machine workloads that frequently use snapshots or clones, and for environments that require over-provisioned pool capacity. Thin-provisioned volumes consume physical space only as blocks are written. Snapshots leverage true copy-on-write functionality at the thin-pool chunk level. For example, a fresh snapshot of a 100 GiB volume consumes practically zero pool capacity at creation, only growing as modified blocks accumulate over time. | ||
|
|
||
| :::tip | ||
|
|
||
| Select **dm-thin** if you expect to create regular snapshots or scheduled backups. This option is generally optimal for standard virtual machine workloads because of its efficient space utilization. | ||
|
|
||
| ::: | ||
|
|
||
|  | ||
|
|
||
| 1. On the **Storage** screen, verify that the StorageClass was created and the correct provisioner was set. | ||
|
|
||
|
|
@@ -177,4 +189,93 @@ You can also create a new virtual machine with the volume of the LVM StorageClas | |
|
|
||
| Backup creation is currently not supported. This limitation will be addressed in a future release. | ||
|
|
||
| ::: | ||
| ::: | ||
|
|
||
| ## Additional Notes | ||
|
|
||
| ### Tuning the `dm-thin` Pool | ||
|
|
||
| When the first PersistentVolumeClaim is created against a `dm-thin` StorageClass, the driver creates an LVM thin pool named `<vgName>-thinpool` using `-l 90%FREE` (allocating 90% of the volume group's remaining free space). Consider tuning the following settings based on your workload demands: | ||
|
|
||
| - **Chunk size**: The chunk size determines the smallest unit of physical space that a thin pool allocates in response to a write. A write to a previously-unallocated region always allocates a full chunk, so small random writes to virgin space against a large chunk size cause severe write amplification (a 4 KiB write against a 16 MiB chunk allocates 16 MiB of pool space). When you install the harvester-csi-driver-lvm add-on version 0.4.0 or later, the default StorageClass sets `chunkSize: "1M"`. On the hardware-RAID-backed volume groups typical of Harvester nodes, `1M` matches the full-stripe width of common layouts (for example, four data disks at a 256 KiB strip yields a 1 MiB stripe), so each chunk allocation maps to whole stripes rather than partial ones. `1M` also keeps thin-pool metadata bounded and comfortably supports pools up to 256 TB. See the [Linux kernel dm-thin admin guide](https://docs.kernel.org/admin-guide/device-mapper/thin-provisioning.html) for background on chunk sizing. | ||
|
jillian-maroket marked this conversation as resolved.
Outdated
|
||
|
|
||
| :::warning | ||
|
|
||
| Chunk size cannot be changed after the thin pool is created. The value must be chosen before the first PersistentVolumeClaim is created against the StorageClass. Reducing chunk size on an existing pool requires evacuating all volumes, destroying the pool, recreating it with the new chunk size, and restoring the volumes. | ||
|
|
||
| ::: | ||
|
|
||
| Align the chunk size to your RAID full-stripe width — that is, (number of data disks) × (RAID strip size). A chunk smaller than the full stripe (for example, a `512K` chunk on a 1 MiB stripe) forces every first-touch allocation into a partial-stripe **read-modify-write**. On parity RAID (5/6), partial-stripe writes also widen the write-hole window, so an unclean shutdown without a protected controller cache (BBU/FBWC) can leave a stripe with inconsistent parity. Choosing a chunk size equal to — or an exact multiple of — the full stripe avoids both problems; `1M` is the safe default because it aligns with the most common power-of-two RAID geometries. | ||
|
jillian-maroket marked this conversation as resolved.
Outdated
|
||
|
|
||
| Override the default with the `chunkSize` parameter on the StorageClass. Common values: | ||
|
|
||
| | Value | When to use | | ||
| |---|---| | ||
| | `1M` | **Default** — matches the full-stripe width of common hardware-RAID layouts; general-purpose VM workloads | | ||
| | `512K` / `128K` | Only when the RAID full stripe is that size, or on non-RAID / single-disk volume groups where stripe alignment does not apply and minimizing snapshot copy-on-write cost is the priority | | ||
| | `2M` | Very large sequential-write pools whose RAID full stripe is 2 MiB | | ||
|
jillian-maroket marked this conversation as resolved.
Outdated
|
||
|
|
||
| Never exceed `2M`. Larger chunks trigger disproportionate copy-on-write costs for snapshots, as documented in the [Red Hat Gluster admin guide](https://docs.redhat.com/en/documentation/red_hat_gluster_storage/3.5/html/administration_guide/chap-configuring_red_hat_storage_for_enhancing_performance). If you rely on LVM's built-in auto-selection instead of setting `chunkSize` explicitly, LVM chooses the chunk size based on the pool size to keep metadata bounded, which for multi-TB pools produces 8–16 MiB chunks — appropriate for metadata sizing but often not for random-write performance. | ||
|
jillian-maroket marked this conversation as resolved.
Outdated
|
||
|
|
||
| Verify the effective chunk size of a live pool: | ||
|
|
||
| ``` | ||
| sudo lvs -o vg_name,lv_name,chunk_size,zero <vgName>/<vgName>-thinpool | ||
| sudo dmsetup table <vgName>-<vgName>--thinpool-tpool | ||
| ``` | ||
|
|
||
| - **Chunk zeroing**: By default, the thin pool writes zeros to each newly allocated block chunk before exposing it to a write operation. On single-tenant clusters, you can disable chunk zeroing to significantly reduce write amplification during initial data allocations. | ||
|
|
||
| Set `zeroBlocks: "false"` on the StorageClass at pool creation time, or apply it to an existing pool with: | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Please remove the instructions to run |
||
|
|
||
| ``` | ||
| sudo lvchange --zero n <vgName>/<vgName>-thinpool | ||
| ``` | ||
|
|
||
| You can fully reverse the change by running the command with `--zero y`. | ||
|
|
||
| - **Pool metadata size**: When the thin pool is created, the driver sizes its metadata logical volume based on the `poolMetadataSize` StorageClass parameter (default `16G`, sufficient for pools up to 256 TB at the default 1 MiB chunk size). A larger chunk size needs less metadata to address the same capacity, and a smaller chunk size needs more: metadata bytes ≈ 64 × (pool_size ÷ chunk_size). If you did not set the parameter, or if the pool grew significantly after creation, extend the metadata volume proactively to prevent the pool from running out of space and becoming unresponsive later. | ||
|
jillian-maroket marked this conversation as resolved.
Outdated
|
||
|
|
||
| ``` | ||
| sudo lvextend --poolmetadatasize +1G <vgName>/<vgName>-thinpool | ||
| ``` | ||
|
|
||
| - **Example `dm-thin` StorageClass with all tuning parameters set explicitly**: | ||
|
|
||
| ```yaml | ||
| apiVersion: storage.k8s.io/v1 | ||
| kind: StorageClass | ||
| metadata: | ||
| name: harvester-lvm-thin | ||
| provisioner: lvm.driver.harvesterhci.io | ||
| parameters: | ||
| type: dm-thin | ||
| vgName: vmvg | ||
| chunkSize: "1M" # Applied on first-time pool creation only. | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. These settings belong to the thin pool shared by all LVs in the volume group, not to an individual StorageClass. The first pool creation determines them, so presenting them as per-StorageClass tuning knobs is misleading and can lead to conflicting configurations. Please remove this example and recommendation; #11335 tracks a dedicated pool-level configuration and status resource. |
||
| poolMetadataSize: "16G" # Applied on first-time pool creation only. | ||
| zeroBlocks: "false" # Applied on first-time pool creation only. | ||
| reclaimPolicy: Delete | ||
| volumeBindingMode: WaitForFirstConsumer | ||
| allowVolumeExpansion: true | ||
| ``` | ||
|
|
||
| ### Choosing a Virtual Machine Disk Bus | ||
|
|
||
| When attaching an LVM CSI PersistentVolumeClaim to a VirtualMachine, `virtio-scsi` (`bus: scsi`) generally outperforms the default `virtio-blk` (`bus: virtio`) for sustained-write workloads on thin-provisioned pools, particularly on RAID-backed storage. `virtio-scsi` performs better because it supports multiple queues and uses a more efficient DISCARD path. | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Could we remove this general performance recommendation or support it with a reproducible Harvester LVM CSI benchmark? QEMU recommends |
||
|
|
||
| ### Coexistence with Longhorn v2 Block-Mode Disks | ||
|
|
||
| If the same node hosts a Longhorn V2 disk in block mode, the underlying device is held exclusively by the SPDK Instance Manager. You can add this device to the LVM `global_filter` to exclude it from LVM device scans and prevent resource conflicts. | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Could we leave this coexistence guidance with @tserong as part of #11098? The issue is still under investigation and discusses a possible devices-file fix, while this section presents the |
||
|
|
||
| ``` | ||
| devices { | ||
| global_filter = [ | ||
| "r|/dev/loop.*|", | ||
| "r|/dev/disk/by-path/.*longhorn.*|", | ||
| "r|/dev/mapper/pvc-.*|", | ||
| "r|/dev/disk/by-id/wwn-<lhv2-disk-wwn>|", | ||
| ] | ||
| } | ||
| ``` | ||
|
|
||
| Because Harvester's operating system is immutable, you must persist this change through an `/oem/*.yaml` cloud-config file to ensure it survives system reboots and upgrades.For more information, see issue [#11098](https://github.com/harvester/harvester/issues/11098). | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could we avoid recommending
stripedfor new workloads here? harvester/harvester#11334 tracks its deprecation before GA and identifiesdm-thinas the supported type. Please labelstripedas deprecated and outside GA scope, recommenddm-thinfor new configurations, and add compatibility or migration guidance for existing pre-GA striped volumes.