Feature #3060

trim/discard for Ceph RBD

Added by Drazen Ferencic about 5 years ago. Updated about 2 years ago.

Status:ClosedStart date:07/17/2014
Priority:SponsoredDue date:
Assignee:Jaime Melis% Done:

0%

Category:Core & System
Target version:Release 5.4
Resolution:fixed Pull request:

Description

If hosts are running at least qemu 1.5 and libvirt 1.0.6, and Ceph RBD is used with either IDE od SCSI driver (virtio doesn't support trim/discard), you could add discard=unmap to the qemu -drive option. For example:

<disk type='network' device='disk'>
<driver name='qemu' type='raw' cache='writeback' discard='unmap'/>
<source protocol='rbd' name='one/one-123'/>
<target dev='sda' bus='scsi'/>
<address type='drive' controller='0' bus='0' target='0' unit='0'/>
</disk>
<controller type='scsi' index='0' model='virtio-scsi'></controller>

Trim/discard option would allow reducing storage allocation on Ceph RBD if files inside the VM gets deleted. Without this option storage allocation on Ceph RBD is not getting any smaller if you delete something inside the VM.


Related issues

Related to Feature #3053: Multi-Queue virtio-scsi Closed 07/16/2014

Associated revisions

Revision 21484817
Added by Anton Todorov about 4 years ago

feature-3060: enable disk discard option for qemu-kvm

This is proposed solution for feature in Backlog #3060

By default this feature is not touching the xml configuration for backward compatibility.

There is global VM config in vmm/remotes/kvm/kvmrc named 'DEFAULT_ATTACH_DISCARD' that if set
will enforce the discard option to all VMs if there is no configuranito per template.

It is possible to enable the discard feature per VM in their template by adding
DISCARD=unmap
to enable and
DISCARD=ignore
to disable the discard

(cherry picked from commit 35b864b897816094222536c63f3ae1df0afd698b)

Revision ed9526ac
Added by Ruben S. Montero about 4 years ago

feature #3060: Minor change in if to conform previous ones.

(cherry picked from commit d9c1999719e4f1d06c4518a35feb598ed33c98bb)

Revision 997eaa02
Added by Ruben S. Montero about 4 years ago

feature #3060: Fix wrong format of discard. Thanks to Anton Todorov

History

#1 Updated by Ruben S. Montero about 5 years ago

  • Tracker changed from Feature to Backlog
  • Category set to Drivers - VM
  • Priority changed from Normal to High

Moving it to backlog to plan it after 4.8

#2 Updated by Data Hero over 4 years ago

Hi Drazen,

you can work around this issue with a hack-ish configuration in the file /etc/one/vmm_exec/vmm_exec_kvm.conf.

DISK = [ driver = "raw", cache = "writeback' discard='unmap" ]

OpenNebula does no sanitization of configuration input whatsoever. ( see: https://github.com/OpenNebula/one/blob/master/src/vmm/LibVirtDriverKVM.cc#L606 )

#3 Updated by Anton Todorov about 4 years ago

Hello,

The TRIM/DISCARD feature is supported by LVM, Ceph RBD and now with our StorPool block device driver too.

Please review this pull request [[https://github.com/OpenNebula/one/pull/65]]

We already have customers that are eager for this feature.

Regarding the Copyrights, this is small patch to enable an option to qemu-kvm, but if you are considering this as big one I or our company are ready to start a procedure of signing a (Company) Contributor License Agreement. In this case please point us to the right document (Company vs Personal one) to proceed.

Kind Regards,
Anton Todorov
StorPool

#4 Updated by Ruben S. Montero about 4 years ago

  • Tracker changed from Backlog to Feature
  • Status changed from Pending to New
  • Target version set to Release 4.14

Needs to update documentation with new option, KVM driver guide

#5 Updated by Ruben S. Montero about 4 years ago

  • Category changed from Drivers - VM to Documentation

#6 Updated by Ruben S. Montero almost 4 years ago

  • Target version changed from Release 4.14 to Release 4.14.2

#7 Updated by Laurent Grawet almost 4 years ago

Hi,

I just want to add that trim/discard IS supported by virtio-scsi but not by old virtio-blk driver.
As a consequence, to be able to use vistio-iscsi inside opennebula, feature #3053 has to be implemented.

Regards,

Laurent

#8 Updated by Anton Todorov almost 4 years ago

Hi,

I can confirm that it is working in 4.14.0 with virtio-scsi with following options in Advanced Options of VM template/storage tab set:

Device Prefix: sd
Discard: unmap
IO: native
Cache: none

Here is the result inside the VM

[root@localhost ~]# lsblk -D
NAME   DISC-ALN DISC-GRAN DISC-MAX DISC-ZERO
sda           0        4K       1G         0
└─sda1        0        4K       1G         0
sr0           0        0B       0B         0

I am almost sure that only Device Prefix and Discard are mandatory tough.

IMO feature #3053 will improve virtio-scsi performance (is it equivalent of virtio-bkl-dataplane ?).

Here is example in action: after system update and followed fstrim here are the stats shown on storpool:

[root@s07 ]# storpool volume usedSpace
-----------------------------------------------------------------------
| volume               |       size | repl. |     stored |       used |
-----------------------------------------------------------------------
| one-img-0            |      10 GB |     3 |       0  B |     912 MB |
| one-img-0-12-0       |      10 GB |     3 |     587 MB |     1.3 GB |
-----------------------------------------------------------------------

And after `fstrim /` inside the VM:
[root@s07 ]# storpool volume usedSpace
-----------------------------------------------------------------------
| volume               |       size | repl. |     stored |       used |
-----------------------------------------------------------------------
| one-img-0            |      10 GB |     3 |       0  B |     912 MB |
| one-img-0-12-0       |      10 GB |     3 |     402 MB |     1.1 GB |
-----------------------------------------------------------------------

Where one-img-0 is the parent image, one-img-0-12-0 is the disk in test(non-persistent clone), 'stored' is tha space stored on physical disks -after fstrim it is less than before.
'used' is the space as shown with df -m from inside VM:
[root@localhost ~]# df -m
Filesystem     1M-blocks  Used Available Use% Mounted on
/dev/sda1          10228  1133      9096  12% /
devtmpfs             354     0       354   0% /dev
tmpfs                371     0       371   0% /dev/shm
tmpfs                371    10       362   3% /run
tmpfs                371     0       371   0% /sys/fs/cgroup

Kind Regards,
Anton

#9 Updated by Tino Vázquez over 3 years ago

  • Assignee set to Jaime Melis

#10 Updated by Jaime Melis over 3 years ago

Support for <controller type='scsi' index='0' model='virtio-scsi'></controller> must be implemented in the core.

#11 Updated by Jaime Melis over 3 years ago

  • Category changed from Documentation to Core & System
  • Target version changed from Release 4.14.2 to Release 5.0

#13 Updated by Ruben S. Montero about 3 years ago

  • Tracker changed from Feature to Backlog

#14 Updated by OpenNebula Systems Support Team about 3 years ago

  • Priority changed from High to Sponsored

#15 Updated by Ruben S. Montero almost 3 years ago

  • Target version changed from Release 5.0 to Release 5.4

#16 Updated by Javi Fontan about 2 years ago

  • Tracker changed from Backlog to Feature
  • Status changed from New to Closed
  • Resolution set to fixed

virtio-scsi controller queue can be configured with changes from issue #3053

#17 Updated by Javi Fontan about 2 years ago

Also available in: Atom PDF