OpenNebula

Totally agree, we first observed that when registering an image from the market place. I prefer the second option. But thinking about this, as we are always doing CP+STAT, why not merging both?. So CP would report back to OpenNebula both the SOURCE + SIZE. (If no size is reported back we can fail-back to that obtained from the first stat, I am thinking of custom datastore drivers). This way there is no need to execute other operation...

I like the idea of CP reporting the "final" size of an image. Making the second size parameter optional for drivers that don't report it (and when that's the case simply not update the size that was initially reported by STAT). Might look at this in 2-3 weeks.

This also happens for other formats like qcow2....

So we need to improve STAT and keep track of file size changes

Still there are problems with compressed qcow2 images, like hadoop from OpenNebula Marketplace.
Those are really of 10GB, but after import it is shown as 1.3GB.

Also when creating/registering image with only SOURCE attribute its size is always 0MB.
That is probably because PATH in this case is missing.

Rolandas Naujikas wrote:

Still there are problems with compressed qcow2 images, like hadoop from OpenNebula Marketplace.
Those are really of 10GB, but after import it is shown as 1.3GB.

This should be a problem of the market place image. Note that we are not decompressing the file to get the actual size. Market place images take their size from either the json document, or HTTP headers.

Locally we cannot support nested formats, so stat operation does not require decompressing a whole image.

Rolandas Naujikas wrote:

Also when creating/registering image with only SOURCE attribute its size is always 0MB.
That is probably because PATH in this case is missing.

Yes, when using SOURCE we are not calling the stat script. Note that using source should be a restricted operation that bypasses all OpenNebula checks. In fact it is in oned.conf as restricted:

IMAGE_RESTRICTED_ATTR = "SOURCE" 

But, you may need to put a size in the image. So,Probably the best option is to let the admin set the SIZE when using SOURCE. Like:

NAME=admin_image
SOURCE=/dev/hdb
SIZE=10240

What do you think?

I agree, setting SIZE for SOURCE would be solution.
But for the moment it doesn't work (I tried on one-4.4).

Ruben S. Montero wrote:

Rolandas Naujikas wrote:

Still there are problems with compressed qcow2 images, like hadoop from OpenNebula Marketplace.
Those are really of 10GB, but after import it is shown as 1.3GB.

This should be a problem of the market place image. Note that we are not decompressing the file to get the actual size. Market place images take their size from either the json document, or HTTP headers.

Locally we cannot support nested formats, so stat operation does not require decompressing a whole image.

So at least lets put right size in Marketplace metadata.

You are right this is not yet implemented, I've opened an issue for this #2489. This will be ready for 4.4.

Thanks for your feedback and testing!

I just got gzip'ed file http://www.cloudbase.it/ws2012r2/ with incorrect size registered:

$ gzip -l windows_server_2012_r2_standard_eval_kvm_20131117.qcow2.gz 
  compressed uncompressed  ratio uncompressed_name
  4831004561      2883584 -99.9% windows_server_2012_r2_standard_eval_kvm_20131117.qcow2

Rolandas Naujikas wrote:

I just got gzip'ed file http://www.cloudbase.it/ws2012r2/ with incorrect size registered:
[...]
OpenNebula registered it with SIZE=3MB.
I re gziped it (to check if it is problem on creator OS/gzip) and it shows the same.

Note that currently nested formats are not supported. If you register a qcow image it will use the the Virtual size, if you register a gzip image it will be used the uncompressed size. So if a gzip qcow image is registered the uncompressed qcow size (not virtual) will be used.

Problem is not in qcow2 format here, but invalid data from gzip -l output.

Rolandas Naujikas wrote:

Problem is not in qcow2 format here, but invalid data from gzip -l output.

OpenNebula uses gzip -l, and you are getting a different output running it directly?

$ gzip -l windows_server_2012_r2_standard_eval_kvm_20131117.qcow2.gz 
  compressed uncompressed  ratio uncompressed_name
  4831004561      2883584 -99.9% windows_server_2012_r2_standard_eval_kvm_20131117.qcow2

I'm finding this issue when importing a Microsoft HyperV image.

stuartl@rikishi ~ $ ls -lh /tmp/win2012de.vhd 
-rw-r--r-- 1 stuartl stuartl 7.5G Apr  2 12:20 /tmp/win2012de.vhd
stuartl@rikishi ~ $ qemu-img info /tmp/win2012de.vhd 
image: /tmp/win2012de.vhd
file format: vpc
virtual size: 40G (42949017600 bytes)
disk size: 7.5G

I'm using a custom datastore driver which I'm developing, using Ceph as the back-end. I've handled collecting the size from the raw image in my handler for the 'cp' action:

BYTES_PER_MB=1073741824
BYTES_PER_512KB=536870912
REGISTER_CMD=$(cat <<EOF
    set -e

    # determine size of image in bytes
    size=\$( $QEMU_IMG info --output=json "${TMP_DST}" | \
        grep virtual-size | cut -f 2 -d: | cut -f 1 -d , )
    # convert this to MB for ceph, rounding up to nearest MB
    size_mb=\$(( ( \$size + $BYTES_PER_512KB ) / $BYTES_PER_MB ))

    # create rbd
    $SUDO $RBD create --image-format 2 --size \${size_mb} $RBD_SOURCE
    # map rbd device
    $SUDO $RBD map $RBD_SOURCE
    # import image
    $SUDO $QEMU_IMG convert $TMP_DST /dev/rbd/$RBD_SOURCE
    # unmap rbd device
    $SUDO $RBD unmap /dev/rbd/$RBD_SOURCE

    # remove original
    $RM -f $TMP_DST
EOF
)

(And yes, things are a little … ugly.)

However I've got no idea how I tell OpenNebula that no, the image is actually 40GB not 7.5GB as originally thought.