Category Archives: XenServer News

Citrix Bugs, Citrix Product Enhancement Request, XenServer News

XenServer MetaData Corrupt Workaround!

Leave a comment

Here is the error you may get.

“SR_BACKEND_FAILURE_181”

“SR BACKEND FAILURE 181”

“Error in Metadata volume operation for SR”

I have seen this at a couple clients with XenServer 6.0 using ISCSI and Fiber Channel (FC), since these drives use LVM to manage the shared storage volumes.. In both situations I had to go through this little process to rename the management VDI (Virtual Disk Image) and unplug that PBD (Physical Block Device) for each SR (Storage Repository) that is affected and then plug it back in (In many instances where I have seen this happen it only happened to one of multiple LUNs and one client had it happen on 4 fiber channel LUNs and their 2 NFS LUNs were spared because they didn’t use LVM).

It is a known bug and this is the known\unknown workaround. This supposedly is a issue with the name of the XenServer “Name” and “Decritptions” or the actualy VMs VDI “Name” or “Descriptions”. You will see these errors when you are creating any new VM or cloneing any VM on a affected LUN or doing any other storage operation on that LUN.

Citrix Recently released this CTX Article about this.  Seems still whack that special characters would cause a problem because I have been using FC-LUN-133 or whatever since XenServer 4.0 with no issue.  http://support.citrix.com/article/CTX131660

Error snapshot from XenCenter (Ugh that helps)

“Error in Metadata volume operaiton for SR.”

Here is what was in the /var/log/messages what happened when I was trying to clone a VM.

/var/log/messages/ (Text)

Feb 20 10:25:39 UCS-LAB-XEN-01 xapi: [ warn|UCS-LAB-XEN-01.demolps.local|7833|Async.VM.clone R:80df72cd3a8a|xapi] VM Windows 7 (32-bit) could run on any of these hosts: [ UCS-LAB-XEN-02.demolps.local; UCS-LAB-XEN-01.demolps.local ]

Feb 20 10:25:40 UCS-LAB-XEN-01 xapi: [ warn|UCS-LAB-XEN-01.demolps.local|7840|Async.VM.provision R:2e17eb5bfd32|xapi] VM __gui__Win7Base could run on any of these hosts: [ UCS-LAB-XEN-02.demolps.local; UCS-LAB-XEN-01.demolps.local ]

/var/log/messages/ (Snapshot)

/var/log/SMLog (Text)

[29791] 2012-02-20 10:28:11.440819 [‘uuidgen’, ‘-r’]

[29791] 2012-02-20 10:28:11.449893 SUCCESS

[29791] 2012-02-20 10:28:11.453757 Setting LVM_DEVICE to /dev/disk/by-scsid/360060160049f1600305c6c647e50e111

[29791] 2012-02-20 10:28:11.460794 Setting LVM_DEVICE to /dev/disk/by-scsid/360060160049f1600305c6c647e50e111

[29791] 2012-02-20 10:28:11.496137 LVMCache created for VG_XenStorage-0d5365ea-6708-9930-bd14-57664aaf9ad7

[29791] 2012-02-20 10:28:11.509823 [‘/usr/sbin/vgs’, ‘VG_XenStorage-0d5365ea-6708-9930-bd14-57664aaf9ad7’]

[29791] 2012-02-20 10:28:11.580390 SUCCESS

[29791] 2012-02-20 10:28:11.580620 lock: acquired /var/lock/sm/0d5365ea-6708-9930-bd14-57664aaf9ad7/sr

[29791] 2012-02-20 10:28:11.580724 LVMCache: will initialize now

[29791] 2012-02-20 10:28:11.580795 LVMCache: refreshing

[29791] 2012-02-20 10:28:11.580895 [‘/usr/sbin/lvs’, ‘–noheadings’, ‘–units’, ‘b’, ‘-o’, ‘+lv_tags’, ‘/dev/VG_XenStorage-0d5365ea-6708-9930-bd14-57664aaf9ad7’]

[29791] 2012-02-20 10:28:11.605369 SUCCESS

[29791] 2012-02-20 10:28:11.605863 lock: released /var/lock/sm/0d5365ea-6708-9930-bd14-57664aaf9ad7/sr

[29791] 2012-02-20 10:28:11.605972 Entering _checkMetadataVolume

[29791] 2012-02-20 10:28:11.607546 lock: closed /var/lock/sm/0d5365ea-6708-9930-bd14-57664aaf9ad7/sr

[29791] 2012-02-20 10:28:11.607666 LVMCache created for VG_XenStorage-0d5365ea-6708-9930-bd14-57664aaf9ad7

[29791] 2012-02-20 10:28:11.624457 [‘/usr/sbin/vgs’, ‘VG_XenStorage-0d5365ea-6708-9930-bd14-57664aaf9ad7’]

[29791] 2012-02-20 10:28:11.645846 SUCCESS

[29791] 2012-02-20 10:28:11.646073 lock: acquired /var/lock/sm/0d5365ea-6708-9930-bd14-57664aaf9ad7/sr

[29791] 2012-02-20 10:28:11.646180 LVMCache: will initialize now

[29791] 2012-02-20 10:28:11.646252 LVMCache: refreshing

[29791] 2012-02-20 10:28:11.646357 [‘/usr/sbin/lvs’, ‘–noheadings’, ‘–units’, ‘b’, ‘-o’, ‘+lv_tags’, ‘/dev/VG_XenStorage-0d5365ea-6708-9930-bd14-57664aaf9ad7’]

[29791] 2012-02-20 10:28:11.670909 SUCCESS

[29791] 2012-02-20 10:28:11.671397 lock: released /var/lock/sm/0d5365ea-6708-9930-bd14-57664aaf9ad7/sr

[29791] 2012-02-20 10:28:11.671510 Entering _checkMetadataVolume

[29791] 2012-02-20 10:28:11.671922 lock: acquired /var/lock/sm/0d5365ea-6708-9930-bd14-57664aaf9ad7/sr

[29791] 2012-02-20 10:28:11.672226 [‘/usr/sbin/vgs’, ‘VG_XenStorage-0d5365ea-6708-9930-bd14-57664aaf9ad7’]

[29791] 2012-02-20 10:28:11.693721 SUCCESS

[29791] 2012-02-20 10:28:11.694269 vdi_create {‘sr_uuid’: ‘0d5365ea-6708-9930-bd14-57664aaf9ad7’, ‘subtask_of’: ‘OpaqueRef:aa2c634f-996e-1ee0-dd11-987b8d449174’, ‘vdi_type’: ‘system’, ‘args’: [‘37580963840’, ”, ‘Created by template provisioner’], ‘host_ref’: ‘OpaqueRef:922fc94c-6495-46d5-deba-2a1653dd09fb’, ‘session_ref’: ‘OpaqueRef:4fcaba07-47a9-511f-1583-3f94a1cae3ad’, ‘device_config’: {‘device’: ‘/dev/disk/mpInuse/360060160049f1600305c6c647e50e111’, ‘SCSIid’: ‘360060160049f1600305c6c647e50e111’, ‘SRmaster’: ‘true’}, ‘command’: ‘vdi_create’, ‘sr_ref’: ‘OpaqueRef:9a59047b-10ef-b77f-7499-6f1cdde1dcf3’, ‘local_cache_sr’: ‘b667c205-c68d-9347-23a0-7d216fea1e0b’, ‘vdi_sm_config’: {‘vmhint’: ‘0bcfa0f7-463e-c722-c25e-2aa3de2bea8d’}}

[29791] 2012-02-20 10:28:11.694425 LVHDVDI.create for 2f6f1534-c253-4464-86f8-42d6d65cb2fb

[29791] 2012-02-20 10:28:11.694524 LVHDVDI.create: type = vhd, /dev/VG_XenStorage-0d5365ea-6708-9930-bd14-57664aaf9ad7/VHD-2f6f1534-c253-4464-86f8-42d6d65cb2fb (size=37580963840)

[29791] 2012-02-20 10:28:11.694676 [‘/usr/sbin/vgs’, ‘–noheadings’, ‘–nosuffix’, ‘–units’, ‘b’, ‘VG_XenStorage-0d5365ea-6708-9930-bd14-57664aaf9ad7’]

[29791] 2012-02-20 10:28:11.715798 SUCCESS

[29791] 2012-02-20 10:28:11.716021 [‘/usr/sbin/lvcreate’, ‘-n’, ‘VHD-2f6f1534-c253-4464-86f8-42d6d65cb2fb’, ‘-L’, ‘35916’, ‘VG_XenStorage-0d5365ea-6708-9930-bd14-57664aaf9ad7’]

[29791] 2012-02-20 10:28:11.800691 SUCCESS

[29791] 2012-02-20 10:28:11.800901 [‘/usr/sbin/vhd-util’, ‘create’, ‘–debug’, ‘-n’, ‘/dev/VG_XenStorage-0d5365ea-6708-9930-bd14-57664aaf9ad7/VHD-2f6f1534-c253-4464-86f8-42d6d65cb2fb’, ‘-s’, ‘35840’, ‘-S’, ‘2097152’]

[29791] 2012-02-20 10:28:11.855321 SUCCESS

[29791] 2012-02-20 10:28:11.855549 [‘/usr/sbin/vhd-util’, ‘query’, ‘–debug’, ‘-v’, ‘-n’, ‘/dev/VG_XenStorage-0d5365ea-6708-9930-bd14-57664aaf9ad7/VHD-2f6f1534-c253-4464-86f8-42d6d65cb2fb’]

[29791] 2012-02-20 10:28:11.868041 SUCCESS

[29791] 2012-02-20 10:28:11.868305 [‘/usr/sbin/lvchange’, ‘-an’, ‘/dev/VG_XenStorage-0d5365ea-6708-9930-bd14-57664aaf9ad7/VHD-2f6f1534-c253-4464-86f8-42d6d65cb2fb’]

[29791] 2012-02-20 10:28:11.951169 SUCCESS

[29791] 2012-02-20 10:28:11.951372 [‘/sbin/dmsetup’, ‘status’, ‘VG_XenStorage–0d5365ea–6708–9930–bd14–57664aaf9ad7-VHD–2f6f1534–c253–4464–86f8–42d6d65cb2fb’]

[29791] 2012-02-20 10:28:11.965020 SUCCESS

[29791] 2012-02-20 10:28:11.965368 Checking if there is space in the metadata for 1 VDI.

[29791] 2012-02-20 10:28:11.965459 [‘uuidgen’, ‘-r’]

[29791] 2012-02-20 10:28:11.977072 SUCCESS

[29791] 2012-02-20 10:28:11.977219 Entering addVdiInternal

[29791] 2012-02-20 10:28:11.977432 Failed to read file with params [3, 0, 512, 512]. Error: Input/output error

[29791] 2012-02-20 10:28:11.977534 Return from read: result: -1, readString: , noOfBytesRead: 0, err: Input/output error

[29791] 2012-02-20 10:28:11.977663 Exception getting metadata length.Error: Failed to read file with params [3, 0, 512, 512]. Error: Input/output error

[29791] 2012-02-20 10:28:11.977795 Exception adding vdi with info: {‘read_only’: 0, ‘managed’: 0, ‘snapshot_time’: ”, ‘vdi_type’: ‘vhd’, ‘snapshot_of’: ”, ‘deleted’: ‘0’, ‘name_label’: ‘dummy vdi for space check’, ‘name_description’: ‘dummy vdi for space check’, ‘type’: ‘user’, ‘metadata_of_pool’: ”, ‘is_a_snapshot’: 0, ‘uuid’: ’76fc4f8d-c9b5-406c-9638-34ad68dcd775′}. Error: Failed to read file with params [3, 0, 512, 512]. Error: Input/output error

[29791] 2012-02-20 10:28:12.020326 Raising exception [181, Error in Metadata volume operation for SR. [opterr=Failed to read file with params [3, 0, 512, 512]. Error: Input/output error]]

[29791] 2012-02-20 10:28:12.020504 lock: released /var/lock/sm/0d5365ea-6708-9930-bd14-57664aaf9ad7/sr

[29791] 2012-02-20 10:28:12.022569 ***** vdi_create: EXCEPTION SR.SROSError, Error in Metadata volume operation for SR. [opterr=Failed to read file with params [3, 0, 512, 512]. Error: Input/output error]

File “/opt/xensource/sm/SRCommand.py”, line 94, in run

return self._run_locked(sr)

File “/opt/xensource/sm/SRCommand.py”, line 131, in _run_locked

return self._run(sr, target)

File “/opt/xensource/sm/SRCommand.py”, line 148, in _run

return target.create(self.params[‘sr_uuid’], self.vdi_uuid, long(self.params[‘args’][0]))

File “/opt/xensource/sm/LVHDSR.py”, line 1256, in create

LVMMetadataHandler(self.sr.mdpath).ensureSpaceIsAvailableForVdis(1)

File “/opt/xensource/sm/srmetadata.py”, line 355, in ensureSpaceIsAvailableForVdis

opterr=’%s’ % str(e))

File “/opt/xensource/sm/xs_errors.py”, line 49, in __init__

raise SR.SROSError(errorcode, errormessage)

[29791] 2012-02-20 10:28:12.022938 lock: closed /var/lock/sm/0d5365ea-6708-9930-bd14-57664aaf9ad7/sr

Trying to back up the machine VM MetaData from the XenServer console. Gets the official SR 181 error. (Text)

│ │Metadata Backup failed: (‘Using SR: FC-LUN-27\nCreating new backup VDI: │ │

│ │ Error code: SR_BACKEND_FAILURE_181\nError parameters: , Error in │ │

│ │ Metadata volume operation for SR. [opterr=Failed to read file with │ │

│ │ params [3, 0, 512, 512]. Error: Input/output error], \n\nThe uuid you │ │

│ │ supplied was invalid.\ntype: VDI\nuuid: \nThe uuid you supplied was │ │

│ │ invalid.\ntype: VDI\nuuid: \nerror creating VBD’,) │ │

│ │ │ │

│ │ <Enter> OK

Trying to back up the machine VM MetaData from the XenServer console. Gets the official SR 181 error. (Snapshot)


Workaround Start

  1. If you have a Backup solution or Snapshot Solution use it before starting this process. PHD Virtual is pretty much the best thing for XenServer so I would recommend it.
  2. First you need to get the Host UUID by running “xe host-list” (We will have to run a command on each XenServer in the pool)
    1. [root@UCS-LAB-XEN-01 ~]# xe host-list
    2. uuid ( RO) : 8d2dc3b3-82d1-4144-8b81-55e7cc5c06fc
    3. name-label ( RW): UCS-LAB-XEN-02.demolps.local
    4. name-description ( RW): 8d2dc3b3-82d1-4144-8b81-55e7cc5c06fc
    7. uuid ( RO) : d62e30a9-bd62-4253-ae0b-8a98c9bb8897
    8. name-label ( RW): UCS-LAB-XEN-01.demolps.local
    9. name-description ( RW): d62e30a9-bd62-4253-ae0b-8a98c9bb8897
    10. [root@UCS-LAB-XEN-01 ~]# xe host-list
    11. uuid ( RO) : 8d2dc3b3-82d1-4144-8b81-55e7cc5c06fc
    12. name-label ( RW): UCS-LAB-XEN-02.demolps.local
    13. name-description ( RW): 8d2dc3b3-82d1-4144-8b81-55e7cc5c06fc
    16. uuid ( RO) : d62e30a9-bd62-4253-ae0b-8a98c9bb8897
    17. name-label ( RW): UCS-LAB-XEN-01.demolps.local
    18. name-description ( RW): d62e30a9-bd62-4253-ae0b-8a98c9bb8897
  3. Find out your LVM SRs. “xe sr-list type=lvmohba” or “xe sr-list type=lvmoiscsi”
  5. Run this command to see what is in the LVM display. “lvdisplay | more” so you can find the MGT VDI.
  7. LVDisplay Result (Screenshot)

LVDisplay Result (Text)

— Logical volume —

LV Name /dev/VG_XenStorage-807d52ba-2cda-9984-6822-8a3fdc2f075a

/MGT

VG Name VG_XenStorage-807d52ba-2cda-9984-6822-8a3fdc2f075a

LV UUID LIk62p-Z97V-cUWr-LAOd-tTjh-b33r-iGTpvX

LV Write Access read/write

LV Status available

# open 0

LV Size 4.00 MB

Current LE 1

Segments 1

Allocation inherit

Read ahead sectors auto

– currently set to 1024

Block device 252:8

Your goal with this command is to find the UUID the PBD that is mounting the VDI to XenServer itself, which is a long process so get your light saber ready.

  1. Next we will need to get the PBD from each host XenServer by running “xe pbd-list sr-uuid=YOUR-SR-UUID host-uuid=YOUR-HOST-UUID?

Raw Commands and Outputs.

[root@UCS-LAB-XEN-01 ~]# xe pbd-list sr-uuid=807d52ba-2cda-9984-6822-8a3fdc2f075a host-uuid=d62e30a9-bd62-4253-ae0b-8a98c9bb8897

uuid ( RO) : d06c1ae8-ce88-818d-582b-8d211ba1b4b5

host-uuid ( RO): d62e30a9-bd62-4253-ae0b-8a98c9bb8897

sr-uuid ( RO): 807d52ba-2cda-9984-6822-8a3fdc2f075a

device-config (MRO): SCSIid: 360060160049f160028597eb8ba4fe111

currently-attached ( RO): true

[root@UCS-LAB-XEN-01 ~]# xe pbd-list sr-uuid=807d52ba-2cda-9984-6822-8a3fdc2f075a host-uuid=8d2dc3b3-82d1-4144-8b81-55e7cc5c06fc

uuid ( RO) : 97a522c2-016b-7876-cffd-7ac2a65cce13

host-uuid ( RO): 8d2dc3b3-82d1-4144-8b81-55e7cc5c06fc

sr-uuid ( RO): 807d52ba-2cda-9984-6822-8a3fdc2f075a

device-config (MRO): SCSIid: 360060160049f160028597eb8ba4fe111

currently-attached ( RO): true

  2. Power off all VMs on the Storage Repository and then unplug the PDB from each XenServer “xe pbd-unplug uuid=YOUR-PBD-UUID”. Once it has been unplugged then you can proceed with renaming the corrupt management VDI.
  3. Run the command “lvrename /dev/VG_XenStorage-YOUR-SR-UUID/MGT /dev/VG_XenStorage-YOUR-SR-UUID\OLDMT” This will rename it so that when you plug in the SR again it will automatically recreate the new one.
  5. Once this command is ran then you will plug that PBD back in “xe pbd-plug uuid=YOUR-PBD-UUID”
  6. Rerun the command “lvdisplay | more” so you can find the can make sure you see the new /MGT VDI and the old one /OLDMGT

This concludes the workaround I hope no one has this problem and maybe 6.0.2 will resolve this issue. Only time will tell.

Also on a side note make sure you always label your VDI’s within every storage repository. Even in our Lab Environment I had a fun time doing some UUID hunting because 3 of the VMs were not labeled and they were active during the PBD unplugs.

  1. The Error
  3. As a XenServer environment is being built from scratch or from templates the “Name” and “Description fields will stay generic. It will not be a problem until there is a problem with the VM Metadata or some other storage or VM related problem where you will not know what 60GB hard drive goes to what. As you can see we have 4 VMs built from Scratch and 3 made from a Template 2008 R2 VM. All the other VMs are labeled correctly. LPS highly recommends keeping up with this we have had some scenarios with customer outages that could have been prevented or mitigated with just labeling these VDI’s(Virtual Disk Images)
  5. Corrected VDI Drive Names