Bug #4020

kvm polling fails when upgrading to 4.14

Added by Richard Stevenson almost 6 years ago. Updated almost 6 years ago.

Status:ClosedStart date:10/02/2015
Priority:NormalDue date:
Assignee:Javi Fontan% Done:

0%

Category:Drivers - Monitor
Target version:Release 4.14.2
Resolution:fixed Pull request:
Affected Versions:OpenNebula 4.14

Description

On upgrade to Opennebula 4.14 from 4.12, my Centos 7 KVM hosts no longer monitor successfully. /var/tmp/one/im/kvm-probes.d/poll.sh fails with the following output:

../../vmm/kvm/poll:368:in `block in get_disk_usage': undefined method `match' for nil:NilClass (NoMethodError)
from /usr/share/ruby/rexml/element.rb:905:in `block in each'
from /usr/share/ruby/rexml/xpath.rb:67:in `each'
from /usr/share/ruby/rexml/xpath.rb:67:in `each'
from /usr/share/ruby/rexml/element.rb:905:in `each'
from ../../vmm/kvm/poll:329:in `get_disk_usage'
from ../../vmm/kvm/poll:145:in `block in get_all_vm_info'
from ../../vmm/kvm/poll:129:in `each'
from ../../vmm/kvm/poll:129:in `get_all_vm_info'
from ../../vmm/kvm/poll:856:in `print_all_vm_template'
from ../../vmm/kvm/poll:908:in `<main>'

poll_xen_kvm.rb Magnifier (25.6 KB) Javi Fontan, 10/05/2015 10:32 AM

Associated revisions

Revision bdf1e54f
Added by Javi Fontan almost 6 years ago

bug #4020: bug monitoring KVM VM disks

Revision ec6ebe48
Added by Javi Fontan almost 6 years ago

bug #4020: bug monitoring KVM VM disks

(cherry picked from commit bdf1e54f74c78d6693690812b56385751131a0b9)

History

#1 Updated by Stefan Kooman almost 6 years ago

Just checking: have you done a "onehost sync --force" on the opennebula frontend (as oneadmin)?

#2 Updated by Richard Stevenson almost 6 years ago

Yes!

I'm presently working around this problem by commenting out lines 367-389 of /var/tmp/one/vmm/kvm/poll on the nodes. onehost sync --force re-breaks updating these nodes, as you would expect.

#3 Updated by Javi Fontan almost 6 years ago

Are you using gluster drivers? It may be related to this thread:

https://forum.opennebula.org/t/4-14-poll-sh-error-on-glusterfs-images/1282

We are working on a fix.

#4 Updated by Richard Stevenson almost 6 years ago

No, I'm using a shared datastores (GFS2 on a SAN) with the qcow2 driver.

#5 Updated by Richard Stevenson almost 6 years ago

Although I do observe that the traceback is very similar! Really, no gluster driver in sight.

#6 Updated by Javi Fontan almost 6 years ago

We need the output of this command for one of the VMS that can not be monitored:

virsh -c qemu:///system dumpxml one-<vmid>

#7 Updated by Anton Todorov almost 6 years ago

Richard Stevenson wrote:

Yes!

I'm presently working around this problem by commenting out lines 367-389 of /var/tmp/one/vmm/kvm/poll on the nodes. onehost sync --force re-breaks updating these nodes, as you would expect.

IMO if you edit /var/lib/one/remotes/vmm/kvm/poll (on the front end where oned is running) then

onehost sync --force
will spread the change to hosts /var/tmp/one/...

#8 Updated by Richard Stevenson almost 6 years ago

As requested by Javi

<domain type='kvm' id='121'>
  <name>one-172</name>
  <uuid>00a375d2-5b05-46c0-9ab6-9a8b78f1b96e</uuid>
  <memory unit='KiB'>1048576</memory>
  <currentMemory unit='KiB'>1048576</currentMemory>
  <vcpu placement='static'>2</vcpu>
  <cputune>
    <shares>2048</shares>
  </cputune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-i440fx-rhel7.0.0'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
  </features>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='block' device='disk'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source dev='/var/lib/one//datastores/0/172/disk.0'/>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/var/lib/one//datastores/0/172/disk.1'/>
      <backingStore/>
      <target dev='hda' bus='ide'/>
      <readonly/>
      <alias name='ide0-0-0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <controller type='usb' index='0'>
      <alias name='usb0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'>
      <alias name='pci.0'/>
    </controller>
    <controller type='ide' index='0'>
      <alias name='ide0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <alias name='virtio-serial0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='00:16:3e:0b:1c:ac'/>
      <source bridge='br0'/>
      <target dev='vnet1'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0' state='disconnected'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='spice' port='6072' autoport='no' listen='0.0.0.0'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <sound model='ich6'>
      <alias name='sound0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </sound>
    <video>
      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <redirdev bus='usb' type='spicevmc'>
      <alias name='redir0'/>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
      <alias name='redir1'/>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
      <alias name='redir2'/>
    </redirdev>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </memballoon>
  </devices>
</domain>

#9 Updated by Javi Fontan almost 6 years ago

There is a bug in the probe when the disks are not backed by Ceph or standard files. Can you try copying the attached to /var/lib/one/remotes/vmm/kvm/poll and executing onehost sync --force?

#10 Updated by Richard Stevenson almost 6 years ago

This appears to work.

#11 Updated by Javi Fontan almost 6 years ago

  • Category set to Drivers - Monitor
  • Status changed from Pending to Closed
  • Assignee set to Javi Fontan

The change disables disk monitoring for non Ceph or standard file disks.

#12 Updated by Richard Stevenson almost 6 years ago

Thanks!

#13 Updated by Javi Fontan almost 6 years ago

  • Target version set to Release 4.14.2

#14 Updated by Javi Fontan almost 6 years ago

  • Resolution set to fixed

Also available in: Atom PDF