Wednesday, December 28, 2016

Temporary Profile Problems with Active Directory to Blame

I recently received a ticket from my helpdesk escalating to my team.  It reported that two relatively new users were unable to get local user profiles on two computers.  They were receiving errors that a profile could not be created and a temporary profile was used.  

The full language of the error is: "You have been logged on with a temporary profile. You cannot access your files and files created in this profile will be deleted when you log off. To fix this, log off and try logging on later. Please see the event log for details or contact your system administrator."



The helpdesk staffer asked me to "delete or correct their AD profiles."  What?  I am certainly not deleting a user's Active Directory account.  I took the ticket, determined to demonstrate to the helpdesk that the problem was with these two computers and had nothing to do with Active Directory.

Sadly, that was not the case, oops!

My expectation was that the Default User profile on these computers was damaged, so new profiles were not being generated successfully.  I disproved that theory when I logged onto the system for the first time.  It generated a local profile for me without error or issue.  Hmmm.

At this point, I indeed had to go into Active Directory Users and Computers and I examined the two user accounts in question.  A little review, and the problem quickly came to light.  



Somehow, some info had been typed into the "Profile path" field.  We don't use roaming profiles, so this field should be left empty/default.  I typically copy user accounts (to preserve a number of account settings like group memberships for a similar job title), so I had copied this problem around to these two new users.  I also found this problem on the source user account.

By deleting the text from the Profile path field, the users confirmed their next logons were without issue.  They managed to create local profiles without error.

A simple problem, but it stumped me for a few moments, so it seemed worth sharing.

Friday, October 21, 2016

Customization Wizard Not Yet Supported from VMware for Windows Server 2016

Windows Server 2016 released a few weeks ago.  I was interested in whether or not I was ready to run it.  

I'm unfortunately not in a great position to move to the latest version of VMware vSphere.  I'm on 5.5 Update 3.  6.5 has just been announced this week at VMworld Europe (though I didn't see it available for download yesterday).  I look forward to upgrading to the latest version shortly.

In the meantime, can I run Windows Server 2016 in my VMware environment?

The good news: yes.  Checking the VMware Compatability guide https://www.vmware.com/resources/compatibility/search.php?deviceCategory=software&testConfig=16) demonstrates that Windows Server 2016 is supported for vSphere 5.5 Update 3.  I'm on track.

(Note: When building virtual machines, Windows Server 2016 isn't an option.  But, that's OK, just pick Windows Server 2012.  https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2132789)

OK, I typically deploy servers from template.  I build my first VM, and performed my standard steps to prep it for templating.  Now, what about that Customization Wizard?  I ran through it, and it seemed to work.  But....

I couldn't find the answer on this one on my own.  I reached out to support.  I got my answer quickly enough: "As of now, the guest customization wizard for Windows Server 2016 is not supported."  And then, the matrix I hadn't managed to find to document the answer:http://partnerweb.vmware.com/programs/guestOS/guest-os-customization-matrix.pdf

I need to do some additional research.  Did the Customization Wizard actually properly SysPrep the server and it is safe to use?  Or, should I learn how to do the Microsoft method to clone the server?

Until I figure those answers out, I'll stick to manually building my Windows Server 2016 systems to avoid issues down the road.

Tuesday, September 13, 2016

False Positive Malwarebytes mdm.exe

On September 13, 2016, I discovered approximately 100 emails from my Malwarebytes Management Console, where I run Malwarebytes Anti-Malware and Anti-Exploit for my end users.  

They were entries such as the following:

9/13/2016 11:15:22 AM <client> <ip> Ransom.Petya Quarantined C:\Program Files (x86)\Common Files\microsoft shared\VS7DEBUG\mdm.exe

or


9/13/2016 10:40:22 AM <client> <ip> Ransom.Petya delete-on-reboot C:\Program Files (x86)\Common Files\microsoft shared\VS7DEBUG\mdm.exe

9/13/2016 10:40:22 AM <client> <ip> Ransom.Petya Quarantined HKLM\SYSTEM\CURRENTCONTROLSET\SERVICES\MDM

The entries began at 6:53AM ET and ended at 11:15am ET.

The volume and location within seemingly legitimate operating system locations made me think I had a false positive on my hands.  Some research promptly confirmed that.

By good fortune, one response already identified the resolution from Malwarebytes support.


Log onto the Malwarebytes Management Console
Click the Client Pane
Select the computer affected from the list of clients
Click the Security Logs Tab along the bottom
Right click the item you wish to restore, then click either: 
Restore this Object... to restore to the computer selected
Restore this object on all clients... to restore this item to all clients

Whew!  I was able to restore the object on all impacted systems with a few quick clicks from the console.  What a relief!  

As an aside, I was not overly familiar with mdm.exe.  It is apparently the Machine Debug Manager.  So it probably wouldn't affect end users terribly, but would thwart people in IT trying to do any troubleshooting on a system potentially.

I'll update this post with any fallout or any further issues.  I wanted to get it up quickly in case anyone else was similarly impacted and panicked like I was.

P.S. Please be careful with reports of viruses in mdm.exe.  It is apparently a known problem, but typically the file is not in its default location I outlined above.  

UPDATE 9/16/2016:
We had a SECOND round of this occur 9/15/2016 from 2:42pm ET to 4:31pm.  I called support myself shortly after 4pm (on the first go-round, above, I relied on the experience in that link provided).  

Support confirmed unofficially that it was a false positive, and collected data from me.  At 5:03pm they let me know it would be fixed in the next update.  

I connected remotely during the off hours,  I saw my definition time stamp had moved from 4:xxpm to 9:xxpm, so I again did the Restore this object on all clients and hopefully can put this to bed.

Friday, May 27, 2016

Windows Convenience Rollup for Windows Server 2008 R2 and vmxnet3 on VMware ESXi = big trouble, lost network configuration, revert to DHCP

I'm a little hesitant to write this blog post as I don't yet have a complete resolution.  But this caused me big headaches and I found no one with the issue, so I thought posting could help others.

Microsoft releases hotfixes on the second Tuesday of each month.  I run a patch schedule where I patch test systems and low critical servers on the third Tuesday of each month, then more critical systems on the fourth Tuesday of each month and the first Tuesday of the following month.  This routine has worked well for me, and in my mind, gives me one to two weeks to avoid any patch headaches if a bad patch is released.  Granted, bad patches aren't very common, but not impossible either.  

May 2016, my patching on the third Tuesday went without issue.  Unfortunately, that trend did not continue.

As I patched on the fourth Tuesday of the month (to about 30 servers), I was horrified to discover TCP/IP configuration was disappearring from my VMware virtual Windows Server 2008 R2 servers upon reboot.  They would be reverted to DHCP, with their static configuration missing.  I was having to use the console to log on and reconfigure TCP/IP.  As I did so, Microsoft would prompt me: "The IP address <ipaddress> you have entered for this network adapter is already assigned to another adapter (<nic>) which is no longer present in the computer.  If the same address is assigned to both adapters and they both become active, only one of them will use this address.  This may result in incorrect system configuration."  This led me to presume that, somehow, my vmxnet3 virtual network adapters had been deleted and recreated.  How did this happen?

As I began investigating, I was stymied by the fact that I hadn't had this issue a week earlier.  When I reviewed the servers I had patched a week prior, I noticed they were missing a few patches.  Testing quickly helped me to identify which patch was causing me this headache: 

Convenience rollup update for Windows 7 SP1 and Windows Server 2008 R2 SP1 KB3125574.


My ESXi servers are at ESXi 5.5 update 3a.  My guests are running the latest version of tools to come with that version of ESXi.  

So at this point, I wanted to test the situation by building a fresh Windows Server 2008 R2.  I had to apply a prerequisite patch to KB3125574, which is April 2015 servicing stack update for Windows 7 and Windows Server 2008 R2 (KB3020369).  Then I could apply KB3125574.  I was a bit disappointed to find this did not reproduce my loss of TCP/IP information.  So at this point, my Windows Server 2008 R2 template that has been used to build 80 servers seems to have the bug.

I started with a ticket open to VMware support.  I got a capable support agent.  He asked if the Windows guest event logs yielded any relevant information.  They did not.  He had me collect logs from the ESXi host, and he determined the MAC address of my vmxnet3 did not change throughout this situation.  This led him to conclude that the network adapter was not deleted and recreated, but that the TCP/IP stack was being created new.  VMware advised me to take the issue to Microsoft (offering to help explain and offer their findings).

So, I'm currently working through the process via a ticket to Microsoft.  We're collecting log data after an impacted guest virtual server at the moment.  I'll be sure to update this post with more once I have it.  

Update 5/31/2016
My open ticket with Microsoft has progressed.  KB3125574 has been updated as follows.  So, basically be cautious with this rollup on vmxnet3 adapters!


Known issue in this convenience rollup
·       Known issue 1
Symptoms
After you install this rollup on VMWare virtual machines (VM), a new Ethernet vNIC that has default settings may replace the existing vNIC and, therefore, cause network issues. Any custom settings that were set on the previous vNIC are persisted in the registry but are not applied to the new vNIC. 


Resolution
To resolve this issue, uninstall the convenience rollup.

Status
Microsoft is researching this issue, and will work together with VMWare to determine the appropriate resolution. We will post more information in this article when the information becomes available.


Update 6/2/2016
VMware released a blog post yesterday on the matter:
http://blogs.vmware.com/apps/2016/06/rush-post-microsoft-convenience-update-and-vmware-vmxnet3-incompatibilities.html

VMware is aware of this issue and we are actively investigating the root causes and possible fixes. While this effort progresses, VMware is advising customers to delay applying the Microsoft “Convenience Update” to any virtual machine that uses the VMXNet3 vNIC type.