Feature #3060
trim/discard for Ceph RBD
Status: | Closed | Start date: | 07/17/2014 | |
---|---|---|---|---|
Priority: | Sponsored | Due date: | ||
Assignee: | Jaime Melis | % Done: | 0% | |
Category: | Core & System | |||
Target version: | Release 5.4 | |||
Resolution: | fixed | Pull request: |
Description
If hosts are running at least qemu 1.5 and libvirt 1.0.6, and Ceph RBD is used with either IDE od SCSI driver (virtio doesn't support trim/discard), you could add discard=unmap to the qemu -drive option. For example:
<disk type='network' device='disk'>
<driver name='qemu' type='raw' cache='writeback' discard='unmap'/>
<source protocol='rbd' name='one/one-123'/>
<target dev='sda' bus='scsi'/>
<address type='drive' controller='0' bus='0' target='0' unit='0'/>
</disk>
<controller type='scsi' index='0' model='virtio-scsi'></controller>
Trim/discard option would allow reducing storage allocation on Ceph RBD if files inside the VM gets deleted. Without this option storage allocation on Ceph RBD is not getting any smaller if you delete something inside the VM.
Related issues
Associated revisions
feature-3060: enable disk discard option for qemu-kvm
This is proposed solution for feature in Backlog #3060
By default this feature is not touching the xml configuration for backward compatibility.
There is global VM config in vmm/remotes/kvm/kvmrc named 'DEFAULT_ATTACH_DISCARD' that if set
will enforce the discard option to all VMs if there is no configuranito per template.
It is possible to enable the discard feature per VM in their template by adding
DISCARD=unmap
to enable and
DISCARD=ignore
to disable the discard
(cherry picked from commit 35b864b897816094222536c63f3ae1df0afd698b)
feature #3060: Minor change in if to conform previous ones.
(cherry picked from commit d9c1999719e4f1d06c4518a35feb598ed33c98bb)
feature #3060: Fix wrong format of discard. Thanks to Anton Todorov
History
#1 Updated by Ruben S. Montero almost 7 years ago
- Tracker changed from Feature to Backlog
- Category set to Drivers - VM
- Priority changed from Normal to High
Moving it to backlog to plan it after 4.8
#2 Updated by Data Hero over 6 years ago
Hi Drazen,
you can work around this issue with a hack-ish configuration in the file /etc/one/vmm_exec/vmm_exec_kvm.conf.
DISK = [ driver = "raw", cache = "writeback' discard='unmap" ]
OpenNebula does no sanitization of configuration input whatsoever. ( see: https://github.com/OpenNebula/one/blob/master/src/vmm/LibVirtDriverKVM.cc#L606 )
#3 Updated by Anton Todorov about 6 years ago
Hello,
The TRIM/DISCARD feature is supported by LVM, Ceph RBD and now with our StorPool block device driver too.
Please review this pull request [[https://github.com/OpenNebula/one/pull/65]]
We already have customers that are eager for this feature.
Regarding the Copyrights, this is small patch to enable an option to qemu-kvm, but if you are considering this as big one I or our company are ready to start a procedure of signing a (Company) Contributor License Agreement. In this case please point us to the right document (Company vs Personal one) to proceed.
Kind Regards,
Anton Todorov
StorPool
#4 Updated by Ruben S. Montero about 6 years ago
- Tracker changed from Backlog to Feature
- Status changed from Pending to New
- Target version set to Release 4.14
Needs to update documentation with new option, KVM driver guide
#5 Updated by Ruben S. Montero almost 6 years ago
- Category changed from Drivers - VM to Documentation
#6 Updated by Ruben S. Montero almost 6 years ago
- Target version changed from Release 4.14 to Release 4.14.2
#7 Updated by Laurent Grawet almost 6 years ago
Hi,
I just want to add that trim/discard IS supported by virtio-scsi but not by old virtio-blk driver.
As a consequence, to be able to use vistio-iscsi inside opennebula, feature #3053 has to be implemented.
Regards,
Laurent
#8 Updated by Anton Todorov almost 6 years ago
Hi,
I can confirm that it is working in 4.14.0 with virtio-scsi with following options in Advanced Options of VM template/storage tab set:
Device Prefix: sd
Discard: unmap
IO: native
Cache: none
Here is the result inside the VM
[root@localhost ~]# lsblk -D NAME DISC-ALN DISC-GRAN DISC-MAX DISC-ZERO sda 0 4K 1G 0 └─sda1 0 4K 1G 0 sr0 0 0B 0B 0
I am almost sure that only Device Prefix and Discard are mandatory tough.
IMO feature #3053 will improve virtio-scsi performance (is it equivalent of virtio-bkl-dataplane ?).
Here is example in action: after system update and followed fstrim here are the stats shown on storpool:
[root@s07 ]# storpool volume usedSpace ----------------------------------------------------------------------- | volume | size | repl. | stored | used | ----------------------------------------------------------------------- | one-img-0 | 10 GB | 3 | 0 B | 912 MB | | one-img-0-12-0 | 10 GB | 3 | 587 MB | 1.3 GB | -----------------------------------------------------------------------
And after `fstrim /` inside the VM:
[root@s07 ]# storpool volume usedSpace ----------------------------------------------------------------------- | volume | size | repl. | stored | used | ----------------------------------------------------------------------- | one-img-0 | 10 GB | 3 | 0 B | 912 MB | | one-img-0-12-0 | 10 GB | 3 | 402 MB | 1.1 GB | -----------------------------------------------------------------------
Where one-img-0 is the parent image, one-img-0-12-0 is the disk in test(non-persistent clone), 'stored' is tha space stored on physical disks -after fstrim it is less than before.
'used' is the space as shown with df -m from inside VM:
[root@localhost ~]# df -m Filesystem 1M-blocks Used Available Use% Mounted on /dev/sda1 10228 1133 9096 12% / devtmpfs 354 0 354 0% /dev tmpfs 371 0 371 0% /dev/shm tmpfs 371 10 362 3% /run tmpfs 371 0 371 0% /sys/fs/cgroup
Kind Regards,
Anton
#9 Updated by Tino Vázquez over 5 years ago
- Assignee set to Jaime Melis
#10 Updated by Jaime Melis over 5 years ago
Support for <controller type='scsi' index='0' model='virtio-scsi'></controller> must be implemented in the core.
#11 Updated by Jaime Melis over 5 years ago
- Category changed from Documentation to Core & System
- Target version changed from Release 4.14.2 to Release 5.0
#12 Updated by Anton Todorov over 5 years ago
Please consider adding option to configure multi-queues on virtio-scsi
<controller type='scsi' index='0' model='virtio-scsi'> <driver queues='N' /> </controller>
#13 Updated by Ruben S. Montero about 5 years ago
- Tracker changed from Feature to Backlog
#14 Updated by OpenNebula Systems Support Team about 5 years ago
- Priority changed from High to Sponsored
#15 Updated by Ruben S. Montero almost 5 years ago
- Target version changed from Release 5.0 to Release 5.4
#16 Updated by Javi Fontan about 4 years ago
- Tracker changed from Backlog to Feature
- Status changed from New to Closed
- Resolution set to fixed
virtio-scsi controller queue can be configured with changes from issue #3053
#17 Updated by Javi Fontan about 4 years ago
- Related to Feature #3053: Multi-Queue virtio-scsi added