Customization Wizard Not Yet Supported from VMware for Windows Server 2016

Windows Server 2016 released a few weeks ago.  I was interested in whether or not I was ready to run it.  

I'm unfortunately not in a great position to move to the latest version of VMware vSphere.  I'm on 5.5 Update 3.  6.5 has just been announced this week at VMworld Europe (though I didn't see it available for download yesterday).  I look forward to upgrading to the latest version shortly.

In the meantime, can I run Windows Server 2016 in my VMware environment?

The good news: yes.  Checking the VMware Compatability guide https://www.vmware.com/resources/compatibility/search.php?deviceCategory=software&testConfig=16) demonstrates that Windows Server 2016 is supported for vSphere 5.5 Update 3.  I'm on track.

(Note: When building virtual machines, Windows Server 2016 isn't an option.  But, that's OK, just pick Windows Server 2012.  https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2132789)

OK, I typically deploy servers from template.  I build my first VM, and performed my standard steps to prep it for templating.  Now, what about that Customization Wizard?  I ran through it, and it seemed to work.  But....

I couldn't find the answer on this one on my own.  I reached out to support.  I got my answer quickly enough: "As of now, the guest customization wizard for Windows Server 2016 is not supported."  And then, the matrix I hadn't managed to find to document the answer:http://partnerweb.vmware.com/programs/guestOS/guest-os-customization-matrix.pdf

I need to do some additional research.  Did the Customization Wizard actually properly SysPrep the server and it is safe to use?  Or, should I learn how to do the Microsoft method to clone the server?

Until I figure those answers out, I'll stick to manually building my Windows Server 2016 systems to avoid issues down the road.

Windows Convenience Rollup for Windows Server 2008 R2 and vmxnet3 on VMware ESXi = big trouble, lost network configuration, revert to DHCP

I'm a little hesitant to write this blog post as I don't yet have a complete resolution.  But this caused me big headaches and I found no one with the issue, so I thought posting could help others.

Microsoft releases hotfixes on the second Tuesday of each month.  I run a patch schedule where I patch test systems and low critical servers on the third Tuesday of each month, then more critical systems on the fourth Tuesday of each month and the first Tuesday of the following month.  This routine has worked well for me, and in my mind, gives me one to two weeks to avoid any patch headaches if a bad patch is released.  Granted, bad patches aren't very common, but not impossible either.  

May 2016, my patching on the third Tuesday went without issue.  Unfortunately, that trend did not continue.

As I patched on the fourth Tuesday of the month (to about 30 servers), I was horrified to discover TCP/IP configuration was disappearring from my VMware virtual Windows Server 2008 R2 servers upon reboot.  They would be reverted to DHCP, with their static configuration missing.  I was having to use the console to log on and reconfigure TCP/IP.  As I did so, Microsoft would prompt me: "The IP address <ipaddress> you have entered for this network adapter is already assigned to another adapter (<nic>) which is no longer present in the computer.  If the same address is assigned to both adapters and they both become active, only one of them will use this address.  This may result in incorrect system configuration."  This led me to presume that, somehow, my vmxnet3 virtual network adapters had been deleted and recreated.  How did this happen?

As I began investigating, I was stymied by the fact that I hadn't had this issue a week earlier.  When I reviewed the servers I had patched a week prior, I noticed they were missing a few patches.  Testing quickly helped me to identify which patch was causing me this headache: 

Convenience rollup update for Windows 7 SP1 and Windows Server 2008 R2 SP1 KB3125574.

My ESXi servers are at ESXi 5.5 update 3a.  My guests are running the latest version of tools to come with that version of ESXi.  

So at this point, I wanted to test the situation by building a fresh Windows Server 2008 R2.  I had to apply a prerequisite patch to KB3125574, which is April 2015 servicing stack update for Windows 7 and Windows Server 2008 R2 (KB3020369).  Then I could apply KB3125574.  I was a bit disappointed to find this did not reproduce my loss of TCP/IP information.  So at this point, my Windows Server 2008 R2 template that has been used to build 80 servers seems to have the bug.

I started with a ticket open to VMware support.  I got a capable support agent.  He asked if the Windows guest event logs yielded any relevant information.  They did not.  He had me collect logs from the ESXi host, and he determined the MAC address of my vmxnet3 did not change throughout this situation.  This led him to conclude that the network adapter was not deleted and recreated, but that the TCP/IP stack was being created new.  VMware advised me to take the issue to Microsoft (offering to help explain and offer their findings).

So, I'm currently working through the process via a ticket to Microsoft.  We're collecting log data after an impacted guest virtual server at the moment.  I'll be sure to update this post with more once I have it.  

Update 5/31/2016
My open ticket with Microsoft has progressed.  KB3125574 has been updated as follows.  So, basically be cautious with this rollup on vmxnet3 adapters!

Known issue in this convenience rollup

·       Known issue 1

Symptoms
After you install this rollup on VMWare virtual machines (VM), a new Ethernet vNIC that has default settings may replace the existing vNIC and, therefore, cause network issues. Any custom settings that were set on the previous vNIC are persisted in the registry but are not applied to the new vNIC. 

Resolution
To resolve this issue, uninstall the convenience rollup.

Status
Microsoft is researching this issue, and will work together with VMWare to determine the appropriate resolution. We will post more information in this article when the information becomes available.

Update 6/2/2016
VMware released a blog post yesterday on the matter:
http://blogs.vmware.com/apps/2016/06/rush-post-microsoft-convenience-update-and-vmware-vmxnet3-incompatibilities.html

VMware is aware of this issue and we are actively investigating the root causes and possible fixes. While this effort progresses, VMware is advising customers to delay applying the Microsoft “Convenience Update” to any virtual machine that uses the VMXNet3 vNIC type.

Difficulting Editing Network Card on vSphere 5.5

I have to confess, I've been "kickin' it old school" and haven't embraced some of the new ways of doing things with my VMware ESXi systems.

I stayed at version 5.0 on vCenter and ESX for awhile.  News of the challenges of the upgrade to 5.1 and the Single Sign On implementation gave me pause and I delayed.

So much so that I skipped 5.1 altogether and upgraded to 5.5 approximately 1 year ago.  (I haven't yet dipped my toe into 6.0.  We have other large scale upgrades and initiatives that haven't allowed me the time to do so.  I hope to investigate it in the coming months.)

One of the other new things I've been avoiding?  The vCenter Web Client.  I was used to being proficient and moving quickly through the vSphere fat client.  Certain features weren't there (hello, Update Manager), and so many things were in different places.

Related to that, I haven't taken my Virtual Machine Hardware versions above 8, so they could all be edited in the fat client.

Okay, with that background out of the way, this weekend I performed a sizeable P2V migration of a SQL server.  And, when the time came, I decided to abandon some of my older ways.  I built the new system as Virtual Machine Hardware version 10, current to my 5.5 infrastructure.

So, after the initial P2V work was done (by running vSphere Converter Standalone), I was working with my new virtual server (uninstalling hardware drivers as appropriate).  I noticed I had an Intel E1000 network adapter.  My standard is to use the VMXNET 3 adapter.  This allows my virtual systems to communicate amongst themselves at 10Gbps.

OK well this is easy enough to fix.  (Or is it?)  I powered down the virtual server for good measure, and attempted to Edit Settings.

Well, that was silly of me.  I knew that was coming.  OK time to sign into the vCenter Web Client to make the changes.

But wait!  Even with the VM powered off, I could not edit the adapter type--it was grayed out and unchangable.  What was going on?  I began to panic.

I attempted to add a second Network Adapter.  Again, I wound up with an E1000 I could not edit.  My panic level rose.

The trick, and how to accomplish this task, is to DELETE ALL NETWORK ADAPTERS.  Once there are none on the system, and you add one--then you can edit the type.  At that point, I selected my desired VMXNET 3 and was on my way.

Successfully Installing Windows 8 on VMware vSphere ESXi 5

Initial efforts to install Windows 8 Professional 64-bit were not successful for me.  I was pleasantly surprised to find my versions of vCenter (5.0.0 623373) and vSphere (5.0.0 623860 or update 1) had Microsoft Windows 8 32-bit and 64-bit in the guest operating system list.  However, my install was not completing successfully.  After initial reboot, all I was getting was an endless spinning circle (the modern day hourglass).

I didn't actually figure this solution out myself, but found the answer here:
http://tinkertry.com/windows-8-runs-fine-on-esxi-5-0-update-1-build-623860-and-build-768111/

Selecting the guest operating system as Windows 7 64-bit was the fix.  The above link mentions a UEFI BIOS but I didn't try that.

I also found information here:
http://kb.vmware.com/selfservice/microsites/microsite.do?cmd=displayKC&docType=kc&externalId=2006859&sliceId=2&docTypeID=DT_KB_1_1
indicating that vmxnet3 virtual NIC does not work in Windows 8.  Solution?  Stick with e1000e or e1000 NICs.

Completing VMware vSphere upgrade to ESX5i: Datastores

Today I begin tackling the last piece of my vSphere 5 upgrade project.

The project was a long time coming.  I was previously running ESX (no i) 4.0 update 1.  Sadly, this was a fairly buggy version.  Many support calls to VMware, some of which featured unpleasant downtimes, came back to blaming the version of my operating system.  I was unable to upgrade due to our previous backup software being incompatable.  Then once my colleague migrated us to a new backup system, other projects on my plate delayed things.

What did the upgrade consist of?  First, I had to tackle vCenter.  I had to migrate off Windows Server 2003 to Windows Server 2008 R2 64-bit, then I could go from vCenter 4 to vCenter 5.  At that point, the time came to upgrade each host from ESX4 to ESX5i.  That process was fairly straight forward, albeit quirky.  I frequently had to reconfigure the network settings or mess with NTP settings.

Now that my 9 hosts are upgraded, the last piece of the puzzle: datastores.

My datastores were running VMFS3.  The current version is VMFS5.  The wrinkle to the upgrade is that there are benefits to rebuilding the datastores fresh, not upgrading them.  One of the key features to VMFS5 is a unified 1MB block size.  An in-place upgrade would not give me that feature.

So, any system partitions on my hosts, or any local disks, I am not touching.  For my shared storage on my SAN, where I can migrate data off to alternate datastores, I am rebuilding.

The procedure is as follows:

1. Storage vMotion any VMs off a VMFS3 datastore by:
2. Right click VM in vCenter, Migrate.  
3. Change datastore, Next.
4. Select your datastore, virtual disk format (same format as source, or do you want to change: thick lazy, thick eager, thin).  Use the Advanced button if your server has data spread across multiple datastores.  Hit Next when you have things as needed.  Then Finish.
5. Confirm the datastore is empty.  In Datastores and Datastore Clusters view, on the Summary tab, note the Number of Hosts Connected.  Ensure Virtual Machines and Templates is ZERO!
6. Right click datastore, making note of it's original name if you want to reuse that later, and select Delete.  Confirm removal operation with Yes.
7. vCenter tasks will show the Remove datastore task and then all affected hosts will initiate a Rescan VMFS task.  Wait for these to complete.
8. In vCenter, go to Hosts and Clusters view.  Select a host connected to the storage, and go to the Configuration tab and then Storage.  Click Add Storage...
9. Storage type of Disk/LUN, Next.
10. Select the LUN (or name, if you're not working with a SAN).  Hopefully you only have the one you just deleted earlier.  Click Next.
11. File System Version, VMFS-5.  That is why we're doing this!  Next.
12. Review operation.  Next.
13. Enter desired datastore name and Next.
14. Maximum available space, Next.
15. Finish.
16. The datastore will be recreated, and the hosts will initiate another Rescan VMFS command.  Once those complete, you're good to go.  In DaDC view, you may want to relocate the new datastore back into any folder hierarchy you originally had.  Confirm all hosts reconnected (on the Summary tab) and that you are at VMFS5 (on the Configuration tab, mine lists at VMFS 5.54).
17. Repeat the above Storage vMotion operations (steps 2 thru 4) and you're good to go.
18. Work with your backup software to ensure any steps required for backups are completed.

I hope to plug away at this for a few days and have my 50ish datastores all at VMFS5.  Then onto other things!