Lesson of the day: VMware ESXi and hot-swappable drives
17 January 2009 
8 Comments
This post was published 3 years 4 months 4 days ago, so if it contains anything technology-related, keep in mind that this information may have since been superseded or become irrelevant.I was beginning to think that I might have to deal with losing something important: the storage disk for my paravirtualized server. I’m telling this story mainly for myself; should I get into such a pickle again, I’ll have something to refer to for the solution. Also, through a few hours of searching the ‘net, I did not find this problem nor the solution. Perhaps someone else is a stupid as I am and needs to find this.
I once had a room full of servers. Soaring electric bills caused me to reduce the mass to just four machines. Some time after that, I virtualized all of the servers into one physical box. This means I have only one physical machine, but running inside are multiple virtual servers. The operating system on the physical server is VMware’s ESXi Hypervisor.
One of the virtual servers, Sydney, is a file server and it is also one of two web servers (this one is Windows, the other is Linux). Within the physical server (host) is a 750GB SATA drive dedicated to Sydney. For reasons which would be of a whole other story, I decided I wanted to remove this drive for a little maintenance (it’s a hot-swappable SATA drive which can be pulled out of a slot in the front of the server). I figured that all I needed to do would be to shut down Sydney and I would be allowed to remove the drive from running host. I believed I could then return the drive, VMware would re-register it, and I could then fire up Sydney and all would be well. Wrong.
Although the host did run fine with the drive removed, Sydney would not start back up. Infrastructure Client belched an error saying a file was missing. I checked, and found right away that the datastore that Sydney uses for storage did not exist. The missing datastore was located on the disk that I jacked with. This means VMware did not see that I had returned the drive. In Infrastructure Client, I navigated to Configuration -> Storage Adapters and saw that the disk was actually there. But when I navigated to Configuration -> Storage, the datastore was not in the list. Hmm… this is not good.
I opened an SSH terminal into the host and dug around. The VMFS partition should have been mounted under /vmfs/volumes, but it was not. However, I could see /dev/disks/vmhba1:1:0:1. Hmm… how do I mount this disk? I opened /etc/fstab, but quickly saw that this was not how VMFS partitions are mounted. So how? I knew if I could figure this out, I’d have my drive back.
Back in Infrastructure Client, I navigated to Configuration -> Storage and tried to ‘Add storage…’. I was allowed to select the missing disk, but it warned me that the existing partition would be destroyed. What? Why couldn’t I just mount it without formatting? Back to the SSH terminal…
I grep‘d my ass off trying to find where the disk mounting information was, but could find nothing. How is this possible? Is there some magic hole inside the server where this type of configuration information exists?
Well, to save you all from any more boring geek-speak, I’ll just get right to the solution: in Infrastructure Client, navigate to Configuration -> Advanced Settings -> LVM, and set “LVM.DisallowSnapshotLun” to “0″. Save the settings and refresh the Storage, and the lost disk will return!
I found that VMware will ignore VMFS partitions if they contain snapshots and appear foreign. Or something like that — I’m no expert, so that’s as far as I could figure.
Anyway, the datastore came back online and I was able to start Sydney. All websites and personal files are intact! And guess who’ll be fooling with hot-swappable drives in a live VMware hypervisor? Not me, that’s for damn sure. To put it plainly: do NOT remove hot-swappable drives from a running VMware ESXi host!
Author Info
You are a ledgend, thanks.
Many thanks !Your suggestion just saved my life.It solves my problem after a reboot of my ESXi server. It has lost my datastores despite any has been physically moved.I change the lvm parameter and launched a rescan and all came back again. Michael
thanks for this. Just saved me a load of time trying to work out what the hell happened after a simple update of the ESXi host.
Thank you for publicly documenting this. You just saved me some major heartache.
Thank you! Awesome info.
I had to pull a drive from a server to get a serial number (Don’t ask why). After that the Array Controller lost all the volume configurations and I had to setup the volumes again. After I rebooted the ESXi host I thought all is well until I launched the VI client. The main Data Store that all the VMs are on was gone!
Changed the setting and the disk was back!
Really awesome solution.
Werner
Glad to hear my article helped, and thanks for sharing!
Does this mean that hot swappable drives from a RAID 5 array cannot be hot swapped for ESXi host when it can be done on Windows?
I think a RAID array is different — as long as the RAID controller completely abstracts the individual drives and presents a virtual disk to the OS (i.e. ESXi), then the RAID controller itself is responsible for handling the drives (rebuilding the volume, etc), not the OS. Thus, if a member of the array fails, you should be able to hot-swap and rebuild without ESXi ever knowing about it. But I’m not entirely certain as I have never done it.
Thanks for stopping by!