Feature #4316

VM deployement does not honor datastore BASE_PATH

Added by EOLE Team about 3 years ago. Updated almost 3 years ago.

Status:ClosedStart date:02/03/2016
Priority:NormalDue date:
Assignee:Jaime Melis% Done:

0%

Category:Core & System
Target version:Release 5.0
Resolution:fixed Pull request:

Description

Hello,

We have a two VDC OpenNebula running and we have an issue regarding to datastores.

Here is the global setup:

  • we use qcow2 TM for VM images and SYSTEM datastores
  • we use shared TM for ISO images
  • we use SAN storage on the physical machines and expose it to the frontend through NFS
  • ONE frontend is a VM running on the nebula physical hosts (corosync/pacemaker)

The SAN storage is allocated like this:

  • a SAN storage is common to all clusters, mounted under /var/lib/one/datastores (GFS2), to store VM images and ISOs common to all VDCs
  • a VDC has a dedicated SAN storage for its stored VM images (DS ID 104) and running VMs (DS ID 103), mounted under /var/lib/one/datastores/<VDC-NAME> only for the hosts of the cluster used by this VDC, it's NFS exported to the frontend which mount it under the same path

The dedicated VDC datastores have their BASE_PATH set to /var/lib/one/datastores/<VDC-NAME>.

Unfortunately, this parameter seems not honored by the qcow2 TM, neither with the libvirt XML generator.

  • Log from qcow2 TM: the path on nebula5 should be /var/lib/one//datastores/<VDC>/103/30901/disk.0
    oneadmin@one-frontend:~$ cat vms/30901/transfer.0.prolog 
    CLONE qcow2 one-frontend:/var/lib/one/datastores/envole/104/f4735e653caf14377c016db8ecf16a29 nebula5:/var/lib/one//datastores/103/30901/disk.0 30901 104
    CONTEXT qcow2 /var/lib/one/vms/30901/context.sh nebula5:/var/lib/one//datastores/103/30901/disk.1 30901 103
    
  • Log for the VM: the deployement file path should be /var/lib/one//datastores/<VDC>/103/30901/deployment.0
    Wed Feb  3 10:50:11 2016 [Z0][VM][I]: New state is ACTIVE
    Wed Feb  3 10:50:11 2016 [Z0][VM][I]: New LCM state is PROLOG
    Wed Feb  3 10:50:13 2016 [Z0][VM][I]: New LCM state is BOOT
    Wed Feb  3 10:50:13 2016 [Z0][VMM][I]: Generating deployment file: /var/lib/one/vms/30901/deployment.0
    Wed Feb  3 10:50:13 2016 [Z0][VMM][I]: ExitCode: 0
    Wed Feb  3 10:50:13 2016 [Z0][VMM][I]: Successfully execute network driver operation: pre.
    Wed Feb  3 10:50:13 2016 [Z0][VMM][I]: Command execution fail: cat << EOT | /var/tmp/one/vmm/kvm/deploy '/var/lib/one//datastores/103/30901/deployment.0' 'nebula5' 30901 nebula5
    Wed Feb  3 10:50:13 2016 [Z0][VMM][I]: error: Failed to create domain from /var/lib/one//datastores/103/30901/deployment.0
    Wed Feb  3 10:50:13 2016 [Z0][VMM][I]: error: Failed to open file '/var/lib/one//datastores/103/30901/disk.0': No such file or directory
    Wed Feb  3 10:50:13 2016 [Z0][VMM][E]: Could not create domain from /var/lib/one//datastores/103/30901/deployment.0
    Wed Feb  3 10:50:13 2016 [Z0][VMM][I]: ExitCode: 255
    Wed Feb  3 10:50:13 2016 [Z0][VMM][I]: Failed to execute virtualization driver operation: deploy.
    Wed Feb  3 10:50:13 2016 [Z0][VMM][E]: Error deploying virtual machine: Could not create domain from /var/lib/one//datastores/103/30901/deployment.0
    Wed Feb  3 10:50:13 2016 [Z0][VM][I]: New LCM state is BOOT_FAILURE
    
  • libvirt XML definition of the VM: the <source file= should use /var/lib/one//datastores/<VDC>/103/30901/
    <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
        <name>one-30901</name>
        <vcpu>2</vcpu>
        <cputune>
            <shares>205</shares>
        </cputune>
        <memory>2097152</memory>
        <os>
            <type arch='x86_64'>hvm</type>
            <boot dev='hd'/>
        </os>
        <devices>
            <emulator>/usr/bin/qemu-system-x86_64</emulator>
            <disk type='file' device='disk'>
                <source file='/var/lib/one//datastores/103/30901/disk.0'/>
                <target dev='vda'/>
                <driver name='qemu' type='qcow2' cache='writeback' io='native'/>
            </disk>
            <disk type='file' device='cdrom'>
                <source file='/var/lib/one//datastores/103/30901/disk.1'/>
                <target dev='hda'/>
                <readonly/>
                <driver name='qemu' type='raw'/>
            </disk>
            <interface type='bridge'>
                <virtualport type='openvswitch'/>
                <source bridge='rectorat'/>
                <mac address='02:00:ac:1b:07:23'/>
                <model type='virtio'/>
            </interface>
            <graphics type='vnc' listen='0.0.0.0' port='36801' keymap='fr'/>
            <input type='tablet' bus='usb'/>
        </devices>
        <features>
            <acpi/>
            <apic/>
        </features>
        <metadata>
            <system_datastore>/var/lib/one//datastores/103/30901</system_datastore>
        </metadata>
    </domain>
    

To work around this, we created two simlinks:

  • /var/lib/one/datastores/103 pointing to /var/lib/one//datastores/<VDC>/103/
  • /var/lib/one/datastores/104 pointing to /var/lib/one//datastores/<VDC>/104/

Regards.

Associated revisions

Revision d0fb1f1c
Added by Ruben S. Montero almost 3 years ago

feature #4316: BASE_PATH of datastores is built from DATASTORE_LOCATION

Revision 3c7ec6fd
Added by Jaime Melis almost 3 years ago

Feature #4316: Remove BASE_PATH in upgrade

History

#1 Updated by Ruben S. Montero about 3 years ago

  • Target version set to Release 5.0

#2 Updated by EOLE Team about 3 years ago

We had an issue when mounting a GFS2 under another GFS2 when the second one crash.

We move our storage mounts flat under /var/lib/one/<DS>, with /var/lib/one/datastores the default one.

I change the BASE_PATH of the datastores 103 and 104 but the registered images are not updated, their SOURCE point to the old value.

I use this value in management scripts to remove images not registered in the database, when something went wrong:
  • an image can be registered without a real file in the datastore
  • a file can be created/uploaded but not registered in the database (for example #3214 and #3731)

Unfortunately, I can not change this value with oneimage update.

#3 Updated by OpenNebula Systems Support Team almost 3 years ago

BASE_PATH is only used for image operations, for example, adding a new image to the images datastore. If you want to change the path of datastores in the nodes you'll have to change the value DATASTORE_LOCATION. This can be done for the whole zone (in oned.conf) or per cluster.

http://docs.opennebula.org/4.14/administration/hosts_and_clusters/cluster_guide.html#cluster-properties

#4 Updated by EOLE Team almost 3 years ago

OpenNebula Systems Support Team wrote:

BASE_PATH is only used for image operations, for example, adding a new image to the images datastore. If you want to change the path of datastores in the nodes you'll have to change the value DATASTORE_LOCATION. This can be done for the whole zone (in oned.conf) or per cluster.

http://docs.opennebula.org/4.14/administration/hosts_and_clusters/cluster_guide.html#cluster-properties

That's unfortunate, the BASE_PATH is more flexible since it can be set per datastore, but DATASTORE_LOCATION is at best per cluster for all datastores.

Our base datastores are under DATASTORE_LOCATION and only two datastores needs to be somewhere else, so for now only the symlinks are usable.

You can change this issue to a request instead of a bug to make all datastores operations use the same values:

  1. BASE_PATH if set in datastore template
  2. ${DATASTORE_LOCATION}/<DS_ID> if ${DATASTORE_LOCATION} is set
  3. /var/lib/one/datastores/<DS_ID> by default.

Then, the images SOURCE should not be absolute path but relative to the datastore, the absolute path is calculated when necessary.

Regards.

#5 Updated by Ruben S. Montero almost 3 years ago

This seems to be a source of problems for many users, we think that the work-around proposed here could be a reasonable behavioral model. This is:

make all datastores operations use the same values*:

  1. ${DATASTORE_LOCATION}/<DS_ID> if ${DATASTORE_LOCATION} is set
  2. /var/lib/one/datastores/<DS_ID> by default.

We propose to remove the BASE_PATH setting per datastore, if needed a symlink can be created from any location. This will be probably clearer, datastores are alwasy in ${DATASTORE_LOCATION}/<DS_ID> being /var/lib/one/datastores the default value. DATASTORE_LOCATION is now only configurable system-wide in oned.conf.

EOLE Team wrote:

OpenNebula Systems Support Team wrote:

BASE_PATH is only used for image operations, for example, adding a new image to the images datastore. If you want to change the path of datastores in the nodes you'll have to change the value DATASTORE_LOCATION. This can be done for the whole zone (in oned.conf) or per cluster.

http://docs.opennebula.org/4.14/administration/hosts_and_clusters/cluster_guide.html#cluster-properties

That's unfortunate, the BASE_PATH is more flexible since it can be set per datastore, but DATASTORE_LOCATION is at best per cluster for all datastores.

Our base datastores are under DATASTORE_LOCATION and only two datastores needs to be somewhere else, so for now only the symlinks are usable.

You can change this issue to a request instead of a bug to make all datastores operations use the same values:

  1. BASE_PATH if set in datastore template
  2. ${DATASTORE_LOCATION}/<DS_ID> if ${DATASTORE_LOCATION} is set
  3. /var/lib/one/datastores/<DS_ID> by default.

Then, the images SOURCE should not be absolute path but relative to the datastore, the absolute path is calculated when necessary.

Regards.

#6 Updated by EOLE Team almost 3 years ago

Ruben S. Montero wrote:

This seems to be a source of problems for many users, we think that the work-around proposed here could be a reasonable behavioral model. This is:

We propose to remove the BASE_PATH setting per datastore, if needed a symlink can be created from any location. This will be probably clearer, datastores are alwasy in ${DATASTORE_LOCATION}/<DS_ID> being /var/lib/one/datastores the default value. DATASTORE_LOCATION is now only configurable system-wide in oned.conf.

We are OK with this modification, we already use symlink and revert the BASE_PATH setting from the datastores to /var/lib/one/datastores.

#7 Updated by Ruben S. Montero almost 3 years ago

  • Tracker changed from Bug to Feature

OK, will proceed that way then :)

THANKS!

#8 Updated by Ruben S. Montero almost 3 years ago

Change in master, documentation needed.

#9 Updated by Ruben S. Montero almost 3 years ago

  • Category changed from Drivers - Storage to Core & System

#10 Updated by Ruben S. Montero almost 3 years ago

  • Assignee set to Jaime Melis

#11 Updated by Ruben S. Montero almost 3 years ago

  • Status changed from Pending to Closed
  • Resolution set to fixed

Also available in: Atom PDF