OpenNebula

A little note, we updated to OpenNebula 4.6 and Ubuntu 14.04 across the whole cluster yesterday. This driver seems to work in OpenNebula 4.6, although more testing is needed.

I've just started implementing a better-engineered patch. One thing I've observed is that the scheduler appears to be blissfully unaware of any kind of local storage on the hosts, and is even less aware about how much the VM uses.

I'm therefore starting on adding some support in the core of OpenNebula for two components:

• a Local Storage MAD (LS_MAD): which will be configured per host (in the host template, much like how virtual networking is configured) that provisions local storage for cache and volumes
• a Local Cache MAD (LC_MAD): which configures a local cache driver (e.g. FlashCache, Enhanced I/O, dm-cache) to take the block device presented by the transfer manager and combine it with the locally provisioned cache.

Then my existing cephcache driver can have the cache and LVM parts stripped out: the new driver would be called "kceph", the "k" denoting that this uses the kernel RBD driver, rather than the userspace librbd driver used by "ceph".

Presently, TM_MAD and its ln step, links directly to storage.

The datastore/transfer manager drivers might opt to override some caching settings for the sake of compatibility, or raise an error. e.g. the ssh transfer manager can't support any cache mode other than OFFLINE. Modes other than NONE require the image to be in/converted to a raw format.

Here, I plan to first call on LS_MAD to provision whatever storage is needed to store the local image, there'd be provision and unprovision scripts that do this. So LS_MAD would be asked to provision some cache area, this returns a device path to the storage (i.e. /dev/dm-123 for LVM, /dev/loop123 for file). This would be skipped when cache_mode_==_NONE. For a LVM-based LS_MAD driver, this would be a wrapper around lvcreate, lvdisplay and lvremove.

TM_MAD, instead would use a mount step, which would make calls on centralised storage to mount or retrieve the image, returning the path to the image on the host. The TM_MAD would be passed the path to that local storage.

• For the shared TM_MAD driver, it simply return the path to where that image resides on the host (i.e. /var/lib/one/datastores/123/...)
• For the ssh TM_MAD driver, it could download a copy of the image, writing it directly over the locally provisioned block device, then return that device as the local path.
• For kceph, it would do an rbd mount and return the /dev/rbdX device.
• An iSCSI driver would do similar: mount the iSCSI target and return the /dev/sdX device it generates.

• If (source_device local_device) or (cache_mode NONE):
  • let vm_device = source_device
• Else:
  • Call set-up functions to create cached block device (e.g. flashcache-create)
  • let vm_device = /path/to/provisioned/cache/device (e.g. /dev/dm-345)
  • If image_precache > 0:
    • Read in image_precache MB of vm_device to fill some of the cache with the initial image_precache MB of data (to speed up booting of some VMs)
• Link vm_device to the system datastore

As an addition, I'll be making some extensions to the scheduler to improve the placement decisions, allowing the scheduler to place a VM based on the amount of local storage required by the VM. In order to do this, I plan to make the following changes:

• Images/disks will carry 3 new attributes:
  • CACHE_MODE: The caching mode to use for a given image or disk, where:
    • NONE = No caching, the image is directly accessed from shared storage and is not transferred to the host
    • AROUND = Write-Around caching: only reads are cached, writes bypass cache completely.
    • THRU = Write-Thru caching: Writes are written simultaneously to back-end storage and cache.
    • BACK = Write-Back caching: Writes are stored locally in cache first, then later flushed to back-end storage.
    • OFFLINE = The entire image is copied to cache during the VM's PROLOG phase, and is copied back in the EPILOG phase. (i.e. like when TM_MAD==ssh)
  • CACHE_SIZE: The size of the cache in MB
  • PRELOAD_SIZE: The amount to pre-read during the PROLOG phase to speed up launching of VMs.
• It may be desirable for the datastore to have these parameters which can be inherited as defaults.

• Hosts will carry 4 new parameters:
  • LS_MAD: Local Storage Middleware Access Driver, specifies what driver is used for local storage:
    • dummy: No local storage provisioning
    • file: Provisioning using raw loopback files
    • lvm: Provisioning using a LVM volume
  • LS_PATH: Path on the host for local storage
    • For LS_MAD==file, the directory where volume files will be kept
    • For LS_MAD==lvm, the LVM volume group
  • LS_MAX: The maximum size of a cache partition in MB.
  • LC_MAD: The Local Cache Middleware Access Driver, e.g.
    • dummy: No cache provisioning
    • flashcache: Local caching using FlashCache

For the sake of simplicity, each host will have just one LS_MAD driver (and pool), and one LC_MAD driver, which will be used for all VMs deployed to that host.

• http://git.longlandclan.yi.org/?p=for-upstream/one.git;a=shortlog;h=feature-cache
• git://git.longlandclan.yi.org/for-upstream/one.git branch: feature-cache

Hi Stuart,

I've had the same issues with ssd-cached disk backends of other types.
I'd love to try out at least a part of what you built there, unfortunately your git is a tad broken.

If you happen to fix it at some point, could you post an update here?
No idea why it didn't go into the addon marketplace, I think it would also help with the newer ceph cache pools.

Ah and finally, in one case a fw update of the caching SSDs has improved things a lot for me. They were worn and tired and now they feel better :-)