Vsphere and Nexus 1000V-Removing Orphaned VEMs

A few days ago, I had been working with my lab on setting up the Cisco Nexus 1000V virtual distributed switches.  After finishing with my lab, I decided it was time to remove the Nexus 1000V-it doesnt work to great when you can only use it on one host.  One of my servers has disk drives, the other is diskless and uses a SAN. I only have 2 nics one of which is used by the SAN, so it appears impossible to migrate it over to the Nexus with only 1 nic, or vcenter loses connectivity during the process.  I found this out the hard way and had to reset my virtual networking a few times using DCUI..

So, I deleted my VSM VM's, before removing the VEM modules from the hosts.  I learned that this is definitely not the correct procedure for removing the Nexus 1000V.  The VEMs can't be deleted from the vsphere inventory, and re-installing the VSM's does not appear to be possible through the Cisco installer app as it detects the VEM presence on the host!  This was an incredibly frustrating process until I learned how to forcefully remove the VEM from the host.  While I was following the vmware KB's on the subject, I kept receiving an error that a cisco tar file was active, the vib could not be removed, and that a reboot was required.  Additionally, as I found out later, I ended up with an orphaned vds object in vCenter.  Deleting the VSM's first is not the way to go.

So, this procedure is for removing orphaned VEM's from ESXi hosts.  It requires use of the ESXi console.  First, make sure that all physical adapters are migrated off the orphaned VEM, and that there are no virtual machines still attached to it.  Put the host into maintenance mode.  Open up the DCUI on the ESXi host, and enable the command line/service console under troubleshooting options.  Once this is enabled, press ALT-F1 to open the console.

First, check the VEM status by entering in the command vem status

If you see that it is running, enter in vem stop

Now let's verify that there are no more vem processes running and using system resources-if there are, the next series of commands will fail and you'll get to reboot your ESXi host and start these all over again!

By making use of the linux lsof command, you can see if there are any vem processes using system files or network resources.  If you don't see anything on your screen, there are no more vem processes active, using system file resources.  If you do see any, its time to force kill these processes.

lsof | grep vem

kill -9 [vem pid]

Now, enter the command esxcli software vib list | grep Cisco

This command lists the Cisco vib files installed on the host for the VEM.  When you have the output, enter in the next command to remove it.

esxicli software vib remove -n [vib name]

This could take some time to complete, it will look like the screen has hung.  This should successfully finish.  Reboot the host and bring it out of maintenance mode.  You should no longer have the VEM installed on this specific host anymore.  After this process, I was able to run the Cisco installer app for the VSM and re-install the VSM and VEMs-however you will be greeted with a new problem.  The vds object in vCenter will not connect with the new VSM-see my next post on how to clear up this issue!

As a note, I am using vsphere 5.5 and had the Nexus 1000v version 4.2.1.SV2.2.1.a and vCenter is running as the VCSA.