OpenNebula

After some discussion with the people at CERN we have now a clearer idea on how this feature can be implemented. In fact, they are currently using this approach, and have contributed the Transfer Manager scripts.

• Volume name generation
• Specify the LVM size, volume group and the like in the DISK attribute

Thanks to Sebastien Goasguen for his kind contribution...

Another contribution to this issue. I personally support the approach by Sander, we can easily merge the patches with minor modifications. From the Mailing list:

I made some minimalistic changes to XenDriver.cc and TransferManager.cc to support LVM over iSCSI. Details of what I did can be found under "Implementation issues" on this page:

http://www.nikhef.nl/pub/projects/grid/gridwiki/index.php/Virtual_Machines_working_group

Basically the code checks for the first 5 characters of a path. If these characters match "/dev/" then it is assumed to be a block device and handled as "LVM over iSCSI". The changes should be backward compatible.

I attached the diffs for both src files. The tm scripts and configuration changes are available at the above mentioned web page.

The big advantage of using LVM over iSCSI is the possibility to work on locally stored copy-on-write clones from the LVM images. If this can be combined with a significant iSCSI read-cache (still need to investigate this), it would significantly reduce the overhead of image transfers between the image repository and the nodes.

Please feel free to contact me for comments or more information.
Thanks,
Sander

I've been working a bit more on this (still in opennebula-1.2).

With the current implementation I have an iSCSI repository containing base virtual machine images. Once a virtual machine is deployed, opennebula creates a copy-on-write clone for it. This makes it possible to boot multiple virtual machines from the same base image. Once the VM is shutdown there are two options. The first one is "save=no", just throw the copy-on-write clone away. The second is "save=yes", the copy-on-write clone is saved in the image repository.

The technical details are a bit tricky. Because snapshots and clones on (c)LVM are not (yet) cluster aware. Fortunately, for this purpose they do not need to be (because the base virtual machine image doesn't change during usage). The workaround is to create a cluster aware normal partition in LVM intended for Copy-On-Write purposes. This partition can be enabled exclusively on the worker node and mapped together with the base (cluster aware) virtual machine image to a local snapshot with 'dmsetup'. When modifications have to be stored after a shutdown of the virtual machine, it is sufficient to remove the mapping and synchronize (i.e. disable) the cluster aware Copy-On-Write partition. Work can continue when the Copy-On-Write image is enabled on any worker node and the local mapping is reestablished. This last bit works when I do it by hand, but I am still trying to find a solution on how to do that from opennebula (it has to know it is booting a cow-image somehow).

Ruben S. Montero wrote:

Another contribution to this issue. I personally support the approach by Sander, we can easily merge the patches with minor modifications.

Sander Klous wrote:

Work can continue when the Copy-On-Write image is enabled on any worker node and the local mapping is reestablished. This last bit works when I do it by hand, but I am still trying to find a solution on how to do that from opennebula (it has to know it is booting a cow-image somehow).

This is now working. In fact, COW images are recursive. So, you can modify the original image and save it. Using this image (which in fact is a COW clone with modifications of the original), will result in another COW clone, pointing to the original COW clone, pointing to the original image.

Why is this important? The use case I am trying to fulfill is the following: take a base image. Modify it to your liking and store it. Submit 100 VMs based on the modified image. All of this with minimal disk/network overhead and without access conflicts.

In the next version I will add caching with dm-cache to minimize the network overhead. Please find attached two tar-balls with minor modifications to the original patches I made in TransferManager.cc and the tm_commands.

The core has been modified to accept block devices, tm scripts have been written to handle lvm partitions.

Transfer Manager LVM scripts have been tested. The LVM scenario has been documented.