Friday, October 21, 2016

Customization Wizard Not Yet Supported from VMware for Windows Server 2016

Windows Server 2016 released a few weeks ago.  I was interested in whether or not I was ready to run it.  

I'm unfortunately not in a great position to move to the latest version of VMware vSphere.  I'm on 5.5 Update 3.  6.5 has just been announced this week at VMworld Europe (though I didn't see it available for download yesterday).  I look forward to upgrading to the latest version shortly.

In the meantime, can I run Windows Server 2016 in my VMware environment?

The good news: yes.  Checking the VMware Compatability guide https://www.vmware.com/resources/compatibility/search.php?deviceCategory=software&testConfig=16) demonstrates that Windows Server 2016 is supported for vSphere 5.5 Update 3.  I'm on track.

(Note: When building virtual machines, Windows Server 2016 isn't an option.  But, that's OK, just pick Windows Server 2012.  https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2132789)

OK, I typically deploy servers from template.  I build my first VM, and performed my standard steps to prep it for templating.  Now, what about that Customization Wizard?  I ran through it, and it seemed to work.  But....

I couldn't find the answer on this one on my own.  I reached out to support.  I got my answer quickly enough: "As of now, the guest customization wizard for Windows Server 2016 is not supported."  And then, the matrix I hadn't managed to find to document the answer:http://partnerweb.vmware.com/programs/guestOS/guest-os-customization-matrix.pdf

I need to do some additional research.  Did the Customization Wizard actually properly SysPrep the server and it is safe to use?  Or, should I learn how to do the Microsoft method to clone the server?

Until I figure those answers out, I'll stick to manually building my Windows Server 2016 systems to avoid issues down the road.

Tuesday, September 13, 2016

False Positive Malwarebytes mdm.exe

On September 13, 2016, I discovered approximately 100 emails from my Malwarebytes Management Console, where I run Malwarebytes Anti-Malware and Anti-Exploit for my end users.  

They were entries such as the following:

9/13/2016 11:15:22 AM <client> <ip> Ransom.Petya Quarantined C:\Program Files (x86)\Common Files\microsoft shared\VS7DEBUG\mdm.exe

or


9/13/2016 10:40:22 AM <client> <ip> Ransom.Petya delete-on-reboot C:\Program Files (x86)\Common Files\microsoft shared\VS7DEBUG\mdm.exe

9/13/2016 10:40:22 AM <client> <ip> Ransom.Petya Quarantined HKLM\SYSTEM\CURRENTCONTROLSET\SERVICES\MDM

The entries began at 6:53AM ET and ended at 11:15am ET.

The volume and location within seemingly legitimate operating system locations made me think I had a false positive on my hands.  Some research promptly confirmed that.

By good fortune, one response already identified the resolution from Malwarebytes support.


Log onto the Malwarebytes Management Console
Click the Client Pane
Select the computer affected from the list of clients
Click the Security Logs Tab along the bottom
Right click the item you wish to restore, then click either: 
Restore this Object... to restore to the computer selected
Restore this object on all clients... to restore this item to all clients

Whew!  I was able to restore the object on all impacted systems with a few quick clicks from the console.  What a relief!  

As an aside, I was not overly familiar with mdm.exe.  It is apparently the Machine Debug Manager.  So it probably wouldn't affect end users terribly, but would thwart people in IT trying to do any troubleshooting on a system potentially.

I'll update this post with any fallout or any further issues.  I wanted to get it up quickly in case anyone else was similarly impacted and panicked like I was.

P.S. Please be careful with reports of viruses in mdm.exe.  It is apparently a known problem, but typically the file is not in its default location I outlined above.  

UPDATE 9/16/2016:
We had a SECOND round of this occur 9/15/2016 from 2:42pm ET to 4:31pm.  I called support myself shortly after 4pm (on the first go-round, above, I relied on the experience in that link provided).  

Support confirmed unofficially that it was a false positive, and collected data from me.  At 5:03pm they let me know it would be fixed in the next update.  

I connected remotely during the off hours,  I saw my definition time stamp had moved from 4:xxpm to 9:xxpm, so I again did the Restore this object on all clients and hopefully can put this to bed.

Friday, May 27, 2016

Windows Convenience Rollup for Windows Server 2008 R2 and vmxnet3 on VMware ESXi = big trouble, lost network configuration, revert to DHCP

I'm a little hesitant to write this blog post as I don't yet have a complete resolution.  But this caused me big headaches and I found no one with the issue, so I thought posting could help others.

Microsoft releases hotfixes on the second Tuesday of each month.  I run a patch schedule where I patch test systems and low critical servers on the third Tuesday of each month, then more critical systems on the fourth Tuesday of each month and the first Tuesday of the following month.  This routine has worked well for me, and in my mind, gives me one to two weeks to avoid any patch headaches if a bad patch is released.  Granted, bad patches aren't very common, but not impossible either.  

May 2016, my patching on the third Tuesday went without issue.  Unfortunately, that trend did not continue.

As I patched on the fourth Tuesday of the month (to about 30 servers), I was horrified to discover TCP/IP configuration was disappearring from my VMware virtual Windows Server 2008 R2 servers upon reboot.  They would be reverted to DHCP, with their static configuration missing.  I was having to use the console to log on and reconfigure TCP/IP.  As I did so, Microsoft would prompt me: "The IP address <ipaddress> you have entered for this network adapter is already assigned to another adapter (<nic>) which is no longer present in the computer.  If the same address is assigned to both adapters and they both become active, only one of them will use this address.  This may result in incorrect system configuration."  This led me to presume that, somehow, my vmxnet3 virtual network adapters had been deleted and recreated.  How did this happen?

As I began investigating, I was stymied by the fact that I hadn't had this issue a week earlier.  When I reviewed the servers I had patched a week prior, I noticed they were missing a few patches.  Testing quickly helped me to identify which patch was causing me this headache: 

Convenience rollup update for Windows 7 SP1 and Windows Server 2008 R2 SP1 KB3125574.


My ESXi servers are at ESXi 5.5 update 3a.  My guests are running the latest version of tools to come with that version of ESXi.  

So at this point, I wanted to test the situation by building a fresh Windows Server 2008 R2.  I had to apply a prerequisite patch to KB3125574, which is April 2015 servicing stack update for Windows 7 and Windows Server 2008 R2 (KB3020369).  Then I could apply KB3125574.  I was a bit disappointed to find this did not reproduce my loss of TCP/IP information.  So at this point, my Windows Server 2008 R2 template that has been used to build 80 servers seems to have the bug.

I started with a ticket open to VMware support.  I got a capable support agent.  He asked if the Windows guest event logs yielded any relevant information.  They did not.  He had me collect logs from the ESXi host, and he determined the MAC address of my vmxnet3 did not change throughout this situation.  This led him to conclude that the network adapter was not deleted and recreated, but that the TCP/IP stack was being created new.  VMware advised me to take the issue to Microsoft (offering to help explain and offer their findings).

So, I'm currently working through the process via a ticket to Microsoft.  We're collecting log data after an impacted guest virtual server at the moment.  I'll be sure to update this post with more once I have it.  

Update 5/31/2016
My open ticket with Microsoft has progressed.  KB3125574 has been updated as follows.  So, basically be cautious with this rollup on vmxnet3 adapters!


Known issue in this convenience rollup
·       Known issue 1
Symptoms
After you install this rollup on VMWare virtual machines (VM), a new Ethernet vNIC that has default settings may replace the existing vNIC and, therefore, cause network issues. Any custom settings that were set on the previous vNIC are persisted in the registry but are not applied to the new vNIC. 


Resolution
To resolve this issue, uninstall the convenience rollup.

Status
Microsoft is researching this issue, and will work together with VMWare to determine the appropriate resolution. We will post more information in this article when the information becomes available.


Update 6/2/2016
VMware released a blog post yesterday on the matter:
http://blogs.vmware.com/apps/2016/06/rush-post-microsoft-convenience-update-and-vmware-vmxnet3-incompatibilities.html

VMware is aware of this issue and we are actively investigating the root causes and possible fixes. While this effort progresses, VMware is advising customers to delay applying the Microsoft “Convenience Update” to any virtual machine that uses the VMXNet3 vNIC type.

Thursday, September 17, 2015

Head to Head Comparison: Fitbit One vs. Fitbit Charge HR

I'm a health-minded person, and I often spend a good portion of my free time on fitness-related pursuits.  Running is one of my favorite hobbies.

I've been a pedometer user for many years.  I used to wear a mechanical pedometer to track my daily activity.

In more recent years, I've moved onto fitness gadgets that keep track of all that for you.  I've had a Fitbit One since March 2013.  I enjoy tracking my daily activity with it and challenging my friends and family regularly via their leaderboard feature.

A few months ago, I set my One aside in favor of a Charge HR.  A lot of my exercise isn't easily read by a pedometer (best for running and walking activities).  I wanted the calories burned based on heart-rate readings.  I tie Fitbit in with MyFitnessPal and wanted accurate calorie data as I track what I eat in pursuit of weight loss goals.

After getting to know both gadgets, I could see definite pros and cons to each device.

  • Fitbit One
    • Pro
      • Inconspicuous
    • Cons
      • Easy to lose
      • Wristband for night wearing is inconvenient
  • Fitbit Charge HR
    • Pro
      • Heart rate data
    • Cons
      • Some people do not like the watch style
      • Skin rashes for some
      • Easy to see you are wearing it

I wanted to compare the accuracy between the two devices.  I was convinced that the Charge HR would give me loads of unearned steps from arm movements.  My experience: that was not the case!

Below is a brief overview of each day.  My activity (normally there'd be plenty of exercise, but I was suffering from a cold and didn't do as much as usual.  I also noted car trips.  I see lots of feedback from others about watching the watch-style Fitbits record steps while driving/riding in a car.

Results/Summary
It's a good reality check that these devices aren't the end-all, be-all for accuracy.  My father-in-law has done similar testing and gotten very different results.  He was finding larger discrepancies (say 2000 steps on a 12000 step day) with the Charge HR reporting more than the One.  

I've got half the week with close ending step counts (one was off by two steps, others off by a few hundred steps for a 3-4% discrepancy).  Those are promising results.

But I've got 3 days with 12-18% discrepancies representing step discrepancies of over 2000 steps.  Considering that's enough to be at least a mile, it adds up.

What does this mean for me?  Not tons.  I still love my Fitbit products and will continue to use them.  I just won't take the data as ironclad fact.  I find it interesting if nothing else.


Day 1: Saturday, September 5, 2015
Daily overview:
A fairly inactive day.  I had flown home from California to Michigan the day before.  On top of having to deal with the time change, I was sick with a cold.  The daily was fairly inactive and included a nap.  No time in the car.  I did take the dog on two short walks (1.75 miles total).
Charge HR: 9998 steps
One: 9996 steps
Result - very accurate

Day 2: Sunday, September 6, 2015
Daily overview:
Not extremely active.  Slept in very late due to my cold.
Charge HR: 11422 steps
One: 13425 steps
Result - 18% discrepancy, Charge HR lower

Day 3: Monday, September 7, 2015
Daily overview:
Fairly active around the house.  Walked the dog 1.25 miles.
Charge HR: 15213 steps
One: 17488 steps
Result - 15% discrepancy, Charge HR lower

Day 4: Tuesday, September 8, 2015
Daily overview:
Commute to work (15 miles each way) in the car.  Busy day at work, so not much time for walking.  Walked the dog 1.25 miles in the evening.
Charge HR: 7576 steps
One: 7873 steps
Result - 4% discrepancy, Charge HR lower

Day 5: Wednesday, September 9, 2015
Skipped audit on this day.  I had misplaced the charger for the Charge HR and its battery died.  I was unable to wear both trackers all day.

Day 6: Thursday, September 10, 2015
Brief car trips. Brief walk for dog. Walking desk for one hour completing 3 miles.
Charge HR: 12085 steps
One: 13523 steps
Result - 12% discrepancy, Charge HR lower

Day 7: Friday, September 11, 2015
Morning dog walk of 0.5 miles.  Commute to work in car (30 miles round trip).  Evening drive to Costco (30 miles round trip).  Plenty of walking in Costco
Charge HR: 11288 steps
One: 11746 steps
Result - 4% discrepancy, Charge HR lower

Day 8: Saturday, September 12, 2015
Completed a 3 mile jog and walked the dog times 2 for 1.4 miles. No car trips today.
Charge HR: 15386 steps
One: 14897 steps
Result - 3% discrepancy, Charge HR higher