How To: Resolving a System Hang During Patching, Remotely!

Routine patching of systems and software is a crucial piece of any business’ information security strategy. Even so, many systems go unnoticed and unpatched for months, even years until an external threat forces the organization into action (e.g. the recent WannaCry ransomware outbreak).

When that happens, server administrators need to be prepared for irregularities they’re likely to encounter, such as a hang prior to reboot.

In this scenario, we’re going to assume that you’ve just finished patching and clicked the “Restart Now” button. You begin a continuous ping (ping -t [hostname/IP address]) and wait for the server to restart.

Let’s assume a normal reboot takes 5-10 minutes for this machine, and that 25+ minutes have passed.

You check the console, and are greeted by the “‘Preparing to Configure Windows. Do not turn off your computer” message. Time continues to pass while your maintenance window dwindles like falling grains in an hourglass… pressure is mounting, the business won’t wait. Time for action!

Logged in as an Administrator from your workstation check the Windows Module Installer service on the remote system…

  1. Run services.msc
  2. Right-click “Services (Local)” and select “Connect to another computer …”
  3. Make sure the “Another computer” radio button is selected and enter the hostname of the stuck server and click “OK”
  4. Search for “Windows Module Installer” service and verify its status. If it’s “stopping,” then you will need to force it to stop. This can’t be done here, so we’ll need to query its PID and use our old friend TaskKill to manually kill the service

Query the Process ID (PID) of the Windows Module Installer (TrustedInstaller) service…

  1. Open Command Prompt as an Administrator
  2. Run the following command:
sc \\[hostname of the server] queryex trustedinstaller

This will return (among other information) the PID of the stuck service, write it down as you’ll need it for the next step

Kill the hung service remotely using TaskKill…

  1. From the Command Prompt already opened, run the following
command:taskkill /s [hostname of the server] /pid [PIDFromAbove] /t

Congratulations, your system should now be unhung! Check your console or continuous ping to verify that the system is restarting and proceed to the next round of updates.

Windows 10 Woes

Like many others, when Microsoft told me I was entitled to a free upgrade from Windows 8.1 to 10, I decided to take them up on the offer.

I downloaded the installation media, and the upgrade went relatively smoothly. I had a few driver issues at first, but was eventually able to iron everything out.

After a week or two of using the new OS, I started to encounter strange UI bugs…

At first, my start menu tiles stopped accepting mouse clicks. I could still launch applications that were pinned to the taskbar, but could not click on any Windows UI menu elements.

This got progressively worse until no windows or applications would accept mouse input. I could still see the cursor and move it around, I could right click and drag on the desktop, but couldn’t interact with anything else using the mouse.

I could still use Keyboard shortcuts as a work around, but it made things awfully inconvenient. I searched for a fix, but the only thing I came up with was a powershell script that purportedly fixed “Start menu” issues:

Get-AppXPackage -AllUsers | Foreach {Add-AppxPackage -DisableDevelopmentMode -Register "$($_.InstallLocation)\AppXManifest.xml"}

This worked for me (after a reboot), but sure enough, the problem came back within a few days. In addition to everything else, it started crashing with a BSOD “Memory_Management” error.

I wasn’t thrilled about disabling all of my devices and enabling them back one-by-one until I found the faulting driver so a clean install started to look more appealing.

I did this, but discovered that I my newly installed Windows 10 wasn’t activated, nor could I activate it as Microsoft’s brilliant new system doesn’t give you an activation key on the free upgrade!

In theory, your PC is supposed to Automagically ™ active itself as soon as it’s connected to the internet. That is, unless there’s a problem with the activation server (as many have encountered) or some other issue…

Microsoft recommends doing a fresh re-installation of whatever previous OS you were on, then running the upgrade again, but that’s nonsense!

All you have to do to fix the activation issue (assuming, like me, you started with the upgrade) is reboot from Windows 10 installation media and select “Reset your PC” with the “Keep my files” option selected.

After several reboots, my Windows 10 installation was repaired – no more UI issues, and successfully activated WITHOUT having to reinstall Windows 8…

Don’t be an IT Order Taker!

Last year, the old Maytag Man (as portrayed by Gordon Jump) who sat bored in his dispatch office, waiting for a repair call that never came was replaced with a younger, more versatile model. Sad really, as I’m going to miss Ol’ Lonely, but I’d never emulate him, and you shouldn’t either!

Early on in my career, I worked in sales. I’ve carried that experience with me all through my career as it taught me the value of proactivity. There were those who sat under a tree of pre-qualified prospects, waiting for low hanging fruit to drop off the branch. These people rarely made quota, and eventually moved on or were let go.

Amazingly, many IT Managers (particularly in the Middle East) are the same way;  they perceive their job as merely order takers, putting out fires as they appeared, happy to go on maintaining the status quo.

Perhaps these IT Managers aren’t taking time to understand the business and it’s needs. It could also suggest that they lack confidence and/or initiative.

Whatever the case, salespeople and IT Managers alike who take  initiative will always have an advantage over those who don’t, and are less likely to be caught off-guard by issues that will [inevitably] arise.

So what can you do to be more proactive? Here are some suggestions that may help:

  1. “Don’t wait until it’s raining to mend your roof – do now, while the sun is shining.”Nothing is ever an IT emergency until it becomes one.The more time you spend preventing fires, the less time you’ll spend putting them out! (Backups, DRPs, documentation, etc.)
  2. “Make your rounds!”Often times, many easily-correctable issues go unreported because the employee(s) suffering from them can’t be bothered to submit a ticket, or don’t know how to articulate it. You can save a lot of time by visiting with people face-to-face to understand what their pains.Do this at every level of the organization! Be friendly, be approachable!
  3. “The map is NOT the territory, get out there once in a while and see it for yourself!”Don’t rely solely on documentation! It could be outdated, there may be human error, or other factors introduced in the course of maintaining your Asset Register/CMDB.
  4. “Focus on what’s important.”This cliché has been beaten to death, so I’ll try to make it simple and relevant – anything the business depends on to make money should be your highest priority, followed by the systems which support them and so on and so forth. 
  5. “Those who ignore history have no past – and no future!”Acknowledge those who have come before you. Study their mistakes, learn from them! It’s much less painful to side-step an avoidable pitfall than to climb out of it after-the-fact.

Microsoft CSAT Survey II: Someone Listened!

It’s not often something changes for the better, but I’m always pleased when they do!

Once again, I’ve found myself tasked with attaining Microsoft Gold level partnership for my employer. For those who have never had the pleasure, the process consists of attaining a combination of competencies (associated certified professionals, tested products etc.), customer references and the dreaded CSAT (Customer Satisfaction) survey.

In the past, the CSAT consisted of 30 questions, many of which applied to Microsoft product resale, which isn’t applicable to many would-be partners.While questions could be added (though I can’t imagine why anyone would want to), none could be removed.

Since then (about April of 2013 or so), the survey was reduced to only 5 questions that actually pertain to customer satisfaction – imagine that!

It’s hard enough asking for a customer’s time to fill out a survey, but if I must, I’d prefer it be short and to the point. I believe this iteration of the CSAT does just that.

So why the change? Did someone at Microsoft read my 2009 rant on the subject and act accordingly? Doubtful, but it’s a nice thought anyway :). Regardless, I’m happy it did, and hope this trend continues!

Blogging with Purpose

When I decided to create this blog, I did so with the expressed purpose of helping others (and myself) find answers that were difficult for me to come by. Either the information I wanted was scattered across several websites, buried deep in some forum, or worse still, phrased in an unclear or misleading way.

On very rare occasions, I’ve somehow managed to solve the problem and be the first (to best of my knowledge) to publish it! In my role, the emphasis is on proven technologies, so it isn’t often that I encounter an issue that’s so obscure that no one else has bothered to write about it.

Even after I’ve decided what to write about, I then will spend hours, sometimes days editing my article for spelling, grammar, readability and clarity. Not every article I write makes the cut, and I’ve got about a half dozen of these sitting in my queue that might never see the light of day.

As one might expect from this focus and model, I don’t publish many articles. I rationalize this by inferring that perhaps (content) quality is more important than quantity. But then how does an author define, “quality?”

That depends on the purpose of the blog. Some blogs are written to entertain, some for profit, others to inform while others still are a collection of seemingly random thoughts and ideas the author wanted to capture.

Does a Blog need an excuse?

For instance, suppose you’re an eccentric Welch change management consultant named ‘Rich’ with a blog whose URL suggests that it’s all about the intricacies of bovine partner-dance.

You could write about everything from social media, to technology to a badger you met on the way to the bathroom wearing a tiny badger-towel with “New Forest 1994” written on it (surprisingly, I’m not making this up – although you’d think I were given that that cheeky brit had since taken down the post it referred to)…

Whose Blog is it anyway?

Can we blog for the sake of writing? Why not!

Does every blog post have to contribute something useful to humanity? Certainly not. As Andy Leonard infers, one shouldn’t worry about what to write – time and practice will solve that for you – only that you write at all; Taking that first step.

So next time your Saudi-American friend’s eyebrow raises and suggests in an ever-so-slightly mocking tone, “You should blog about that…” Smile back and say, “Yes Yousef, I think I will!”

Disclaimer: No talking badgers, Welshmen or change management consultants were harmed in the writing of this post.

P.S. Rich, Stu and Adam – thanks for helping me retain my sanity and sense of humor in a time and place where both were hard to come by.