OpenNebula

Sent by Sebastiaan Smit to the mailing list http://lists.opennebula.org/pipermail/users-opennebula.org/2014-November/046536.html:

We're testing the vCenter functionality in version 4.10 and see some strange behaviour while doing bulk actions.

Deleting VM's sometimes leave stray VM's on our cluster. We see the following in de VM log:

Sun Nov 9 15:51:34 2014 [Z0][LCM][I]: New VM state is RUNNING
Wed Nov 12 17:30:36 2014 [Z0][LCM][I]: New VM state is CLEANUP.
Wed Nov 12 17:30:36 2014 [Z0][VMM][I]: Driver command for 60 cancelled
Wed Nov 12 17:30:36 2014 [Z0][DiM][I]: New VM state is DONE
Wed Nov 12 17:30:41 2014 [Z0][VMM][W]: Ignored: LOG I 60 Command execution fail: /var/lib/one/remotes/vmm/vcenter/cancel '423cdcae-b6b3-07c1-def6-96b9f3f4b7b3' 'demo-01' 60 demo-01
Wed Nov 12 17:30:41 2014 [Z0][VMM][W]: Ignored: LOG I 60 Cancel of VM 423cdcae-b6b3-07c1-def6-96b9f3f4b7b3 on host demo-01 failed due to "ManagedObjectNotFound: The object has already been deleted or has not been completely created"
Wed Nov 12 17:30:41 2014 [Z0][VMM][W]: Ignored: LOG I 60 ExitCode: 255
Wed Nov 12 17:30:41 2014 [Z0][VMM][W]: Ignored: LOG I 60 Failed to execute virtualization driver operation: cancel.
Wed Nov 12 17:30:41 2014 [Z0][VMM][W]: Ignored: LOG I 60 Successfully execute network driver operation: clean.
Wed Nov 12 17:30:41 2014 [Z0][VMM][W]: Ignored: CLEANUP SUCCESS 60

We see it in a different manner while bulk creating VM's (20+ at a time):

Sun Nov 9 16:01:34 2014 [Z0][DiM][I]: New VM state is ACTIVE.
Sun Nov 9 16:01:34 2014 [Z0][LCM][I]: New VM state is PROLOG.
Sun Nov 9 16:01:34 2014 [Z0][LCM][I]: New VM state is BOOT
Sun Nov 9 16:01:34 2014 [Z0][VMM][I]: Generating deployment file: /var/lib/one/vms/81/deployment.0
Sun Nov 9 16:01:34 2014 [Z0][VMM][I]: Successfully execute network driver operation: pre.
Sun Nov 9 16:01:36 2014 [Z0][VMM][I]: Command execution fail: /var/lib/one/remotes/vmm/vcenter/deploy '/var/lib/one/vms/81/deployment.0' 'demo-01' 81 demo-01
Sun Nov 9 16:01:36 2014 [Z0][VMM][I]: Deploy of VM 81 on host demo-01 with /var/lib/one/vms/81/deployment.0 failed due to "undefined method `uuid' for nil:NilClass"
Sun Nov 9 16:01:36 2014 [Z0][VMM][I]: ExitCode: 255
Sun Nov 9 16:01:36 2014 [Z0][VMM][I]: Failed to execute virtualization driver operation: deploy.
Sun Nov 9 16:01:36 2014 [Z0][VMM][E]: Error deploying virtual machine
Sun Nov 9 16:01:36 2014 [Z0][DiM][I]: New VM state is FAILED
Wed Nov 12 17:30:19 2014 [Z0][DiM][I]: New VM state is DONE.
Wed Nov 12 17:30:19 2014 [Z0][LCM][E]: epilog_success_action, VM in a wrong state

I think these have two different root causes. The cluster is not under load.