Powershell Scripting Part 1- Stupid little things nobody ever bothers to tell you

In my experience, there are two types of tutorials for everything: the most fundamentally basic and the most terrifyingly advanced.  When I was first learning powershell, I found a distinct lack of intermediate level tutorials.  So here's a blog post dedicated to telling you everything you ought to know but nobody ever bothers to tell you about powershell.

Variable Expansion

You probably know how to create variables in powershell, but in case you don't:

>>$PROCS=get-wmiobject -namespace root\cimv2 -query "Select * from win32_process"

$PROCS now contains an array of processes, and executing

>>$PROCS

displays all the information about every process on the system.  But what if you want just the CommandLine value for each of the processes?  Well, of course you could instantiate a new variable like

>>$COMMANDLINES=get-wmiobject -namespace root\cimv2 -query "Select CommandLine from win32_process"

but then you lose some informtion; you can't get the process id for the process with the command line you're looking for, for example.  Ok, so lets try to echo the command line and process ID properties:

>>foreach($PROC in $PROCS){echo $PROC.ProcessId : $PROC.CommandLine;}

Unfortunately, this echos each part on a seperate line.  The solution is to put quotes around the blocks you want to output on their own lines:

>>foreach($PROC in $PROCS){echo "$PROC.ProcessId : $PROC.CommandLine";}

But something unexpected happens!  instead of "804 : C:\Windows\System32\mspaint.exe", we end up with "<crasy looking string>.ProcessId : <equally crazy looking string>.CommandLine".  So what happened?  Well, variable expansion happened.

By placing the variables in quotes, we caused $PROC to get expaned into its full string value.  It turns out that when you expand a wmi object variable, it resolves to the __Path property of that variable.  So in essence, we just did the same thing as typing "<proc path>.ProcessId : <proc path>.CommandLine".  No surprise that these values don't get replaced by the values we actually wanted to put there, is it? 

So how do we fix it?  Well, when a quoted value has the '$' symbol in it, its a sign to powershell that you want to expand whatever the next token is.  So, we just have to make the next token "<proc path>.ProcessId" instead of $PROC.  It would look like this:

>>foreach($PROC in $PROCS){echo "$($PROC.ProcessId) : $($PROC.CommandLine)";}

In this case, powershell sees the '$', and tries to expand the next token. But it sees the (), and realizes that it has to expand whatever is inside the parenthesis first. So it expands $PROC.ProcessId into <proc path>.ProcessId, same as before, and then it goes ahead and expands <proc path>.ProcessId into the actual value we wanted to pass in the first place.

You can do as many of these nestings as you want. But just remember, any time you're trying to use a variable's property inside quotes, you need to do this- including inside query statements.  For example, if you want to refresh the properties of your win32_process object (remember, objects are a static snapshot of the system; if the process exits after you grab the wmi object, it won't be reflected in the object until you grab it again), you can do the following:

>>$PROC=get-wmiobject -namespace "root\cimv2" -query "Select * from win32_process where ProcessId=$($PROC.ProcessId))"

DbgEng.lib Part 2- Forcing a Debug Break

Today we had an issue where some drivers weren't getting installed correctly on a vm.  For whatever reason, they were taking upwards of 10 minutes to install, when in the past they've taken less than a minute.  We decided we needed to break into the debugger somewher in that 10 minute window and figure out what was going on.  Unfortunately, the problem didn't repro 100% of the time, and took a great deal of setup.  Fortunately, we already had it all automated- except for the breaking into the debugger part.  I thought to myself, given my last post, I had no choice but to use DbgEng.lib.

Anyway, I thought I'd post the source for a quick little program I wrote that takes the remote settings of the debugger as an argument, causes the debugger to break in, and then exits.  A perfect example of how DbgEng can be useful in everday life.

main.h:

#pragma once
#include <dbgeng.h>
#include <stdio.h>
#include <windows.h>



main.cpp:

#include "main.h"
void Usage()
{
    printf("Usage: BreakIntoDebugger.exe <connection string>\n");
    printf("\texample: BreakIntoDebugger.exe npipe:server=localhost,pipe=vmname");
}
int main(int argc, char* argv[])
{
    printf("BreakIntoDebugger Started\n");
    printf("number of args:%i\n", argc-1);
    for(int i = 1; i < argc; i++)
    {
        printf("argv[%i] = ", i);
        printf("%s\n", argv[i]);
    }
    if(argc < 2)
    {
        Usage();
        return 1;
    }
    HRESULT hr;
    IDebugControl* pControl = NULL;
    IDebugClient* pClient = NULL;
    ULONG status = 0;
    PCSTR pcRemoteOptions((PSTR)argv[1]);
    hr = DebugConnect(pcRemoteOptions, __uuidof(IDebugClient), (PVOID*)&pClient);
    if(FAILED(hr))
    {
        printf("DebugConnect failed");
        return 1;
    }
    hr = pClient->QueryInterface(__uuidof(IDebugControl), (PVOID*)&pControl);
    if(FAILED(hr))
    {
        printf("QueryInterface failed");
        return 1;
    }
  
    hr = pControl->GetExecutionStatus(&status);
    if(FAILED(hr))
    {
        printf("GetExecutionStatus failed");
        return 1;
    }
    if(status == DEBUG_STATUS_BREAK)
    {
        printf("Already Broken in!");
        return 0;
    }
  
    hr = pControl->SetInterrupt(DEBUG_INTERRUPT_ACTIVE);
    if(FAILED(hr))
    {
        printf("SetInterrupt failed");
        return 1;
    }
    printf("successfully broke into the debugger");
}



DbgEng.lib Part 1- Checking for Debug Breaks

DbgEng.lib- What is it?
dbgeng.lib is the library upon which WinDbg, CDB, and NTSD are all built.  It allows you to connect to a Windows operating system, insert breakpoints, catch bugchecks, dump memory, and anytyhing else you can do in a debugger.  Further, it allows you to connect to existing debugger instances and check their status.  You can get your hands on dbgeng.lib by downloading the debugging tools for windows here

Why would I want to use it?
Good question.  I used it to develop some automated stress testing for a component of windows.  We wanted to be careful that we weren’t rebooting VMs that were bugchecked (and losing all that juicy stack trace info), and the best way to do that was to check the attached debugger for a break before reseting the VM.  You could use it for driver development (seriously… test your drivers, or windows will bugcheck and everybody blames us), to implement your own debugger extensions, or to auto-triage failures based on the call stack.

What's the catch?
There’s always a catch.  It turns out that if your debugger gets out of sync with the client, some calls to dbgEng can hang indefinitely.  But we’ll get to that in a bit.

Getting Started
In this example, I’m going to stick with checking whether a debugger is broken in.  The more advanced case is implementing your own debugger extension, and maybe I’ll post about that later.
First, you’ll need to make sure you have a debugger running that’s connected to the target.  Pretty easy.  Say you have a VM named “vm1″, and you’ve set it’s com1 port to be a named pipe called “vm1_com1″, you would just run

kd -server npipe:pipe=vm1_debug_pipe,icfenable -k com:pipe,port=vm1_com1,resets=0,reconnect

from an elevated cmd prompt.  At this point, you’ll be able to connect to that remotely from any remote server (windbg.exe -remote npipe:server=servernamewherekdlives,pipe=vm1_debug_pipe)  If the usernames and passwords aren’t the same for the server your connecting from and the one you’re connecting to, you’ll get an access denied error.  Try the following from an elevated cmd prompt on the connecting machine to connect as a specific user:

net use \\servername /u:username password

Next, you’ll want to fire up your trusty C++ IDE.  Include dbgeng.h, and make sure you set your project to point to dbgeng.lib.

Codify Me!
Now Let’s take a look at some code:
#include dbgend.h
int main()
{
    HRESULT hr;
    BOOL BrokenIn = FALSE;
    HRESULT Status = 0;
    IDebugControl* pControl = NULL;
    IDebugClient* pClient = NULL;
    CHAR RemoteOptions[] = "npipe:server=localhost,pipe=vm_com1";
    PCSTR pcRemoteOptions((PSTR)RemoteOptions);
    hr = DebugConnect(pcRemoteOptions, __uuidof(IDebugClient), (PVOID*)&pClient);
    if(FAILED(hr))
    {
        TRACE_ERROR("DebugConnect failed with error %x", hr);
        goto Cleanup;
    }
    hr = pClient->QueryInterface(__uuidof(IDebugControl), (PVOID*)&pControl);
    if(FAILED(hr))
    {
        TRACE_ERROR("QueryInterface failed with error %x", hr);
        goto Cleanup;
    }
    hr = pControl->GetExecutionStatus(&status);
    if(FAILED(hr))
    {
        TRACE_ERROR("GetExecutionStatus failed with error %x", hr);
        goto Cleanup;
    }
    if(*status == DEBUG_STATUS_BREAK)
    {
        BrokenIn = TRUE;
    }
Cleanup:
    if(pControl)
    {
        pControl->Release();
    }
    if(pClient)
    {
        pClient->Release();
    }
    if(BrokenIn)
    {
        TRACE_ERROR(0, "Debugger is broken in");
        /*do something interesting*/
    }
    return 0;
}  

So what just happened?
Well, we called DebugConnect, a function provided by DbgEng, to connect to an already existing debugger.  If we wanted to create a debugger instead, we would use the DebugCreate() call.  We got ourselves an IDebugClient* out of the deal, which is a pointer to a COM interface object that we can use to query for an IDebugControl instance.  We could also query for other objects, like IDebugSymbols, IDebugDataSpaces, and IDebugRegisters.  These would be used to do interesting things in a debugger extension.

Once we had our IDebugControl instance, we just queried for the execution status of the debugger.  Viola, we know whether or now we’re broken in!  Now, you can do so much more with this interface- anything from executing debugger commands to disassembling processor commands to reading memory locations.

Now for the BUT...
As I mentioned before, the debuger and the debugee can get out of sync.  This leads to the debugger outputting the error “retry sending the same data packet 64 times” and becoming totally unresponsive.  Even worse, any call into the IDebugControl interface HANGS INDEFINITELY.  Horrible.  Just horrible.

In my experience, this seems to happen only on rebooting the debugee machine.  The workaround I came up with for dealing with this was to implement a ref counting scheme for each debug instance.  Every time a thread wanted to call into a debug instance, it would increment the ref count and set the LastCalledTime variable.  I then had a seperate thread that checked every few seconds if the ref count was > 0 and the LastCalledTime was greater than some threshold value.  Once you know the debugger is hung, you can either abort completely, closing down the debugger and starting up a new one, or you can reboot the debugee again.  Sometimes it takes several reboots before the debugee connects succesfully.

This is a terrible, hacky way to deal with it, but its better than the alternative.  It boggles my mind a that there isn’t a timeout implemented inside dbgeng.lib for exactly this case- that seems by far the best solution.

But alas, you can’t win them all.

Monkey out of the Cage

Hey All (by which I mean nobody),

I’m a Software Development Engineer in Test at Microsoft.  I’m starting a blog for one reason and one reason alone: I’m frustrated when I search for error codes, broken APIs, interfaces, and other such shenanigans on bing (read: google) and nothing comes up.  The most frustrating page in the world is right here.  I’m hoping (rather unrealistically) that this page eventually gets indexed and shows up on search results that led me to write the blog entries.

Anyways, I’m going to post one about the DebugConnect interface provided by dbgeng.h next time, since that’s the current burr in my saddle.