Is it possible to remotely debug a process started outside VB6?
The application is a VB6 application with quite a few dll/ocx resources. I am attempting to setup a ClickOnce deployment, using Registration-Free COM, of the VB6 app but have been getting errors when it executes.
My understanding of the way that VB6 redirects COM registerations will probably mean that this is not possible but I thought someone might have a better idea.
To support Darryl's answer suggesting Windbg - here's a 2006 blog post by a Microsoft guy about using Windbg with VB6, and 2004 blog post by another Microsoft guy with a brief introduction to Windbg.
EDIT: Just to make it totally clear. Windbg is a free standalone debugger from Microsoft. Compile your VB6 EXEs, DLLs and OCXs into native code with symbols (create PDB files) and you will be able to debug your ClickOnce application.
Key excerpt from the blog:
If you have limited access to the server machine then you can use the remote debugging facilities of WinDbg. Attach a copy of WinDbg to the process in the usual way and then turn it in to a debugging server (check out .server in the WinDbg help). You can then connect to it remotely from the File menu of WinDbg. It will be just like being there except for the lack of noise from the server room fans. When debugging a remote, your copy of WinDbg is just a very smart terminal so all extensions, symbols and so on have to be on the remote server. You set this up the exact same way for any DLL, VB6 or .NET.
The symbols for your component will not load until your component does and so you have to let the server run at least that long. You can put a break in early in your VB code if you want to stop the debugger at that point but if you do, remember that it will stop there every time through the code. Let’s assume that you let it run and then break in. If you list the loaded symbols for your module with "x MyModule!*" then you will see all of your functions together with a lot of symbols bundled in there for you. VB adds interfaces and symbols quite unashamedly but you don’t generally need to worry about those. One thing that will probably look strange is that all class/method syntax with the C++ double colon convention instead of the friendly little dot. WinDbg doesn’t understand that VB is different and it is treated just like any DLL with symbols.
From here, you can set breakpoints in the usual way (bp etc) and step through code. You can also open up VB source code modules and set breakpoints in them with F9 although the VB file extensions are not in the source file type dropdown. Stepping through the code is revealing but might be a little alarming if you have not seen the code that VB generates for you before. You will be stepping through the assembler and there is a lot of COM goo in there. Hresults get checked a lot. You will probably need to refer to the source often to work out where you are since it takes a bit of practice to be able to know what the source code looked like. Variants are especially challenging because VB does a lot of work for you there and what looks like a simple equation can result in a great deal of code. Optimised code is even harder because the order of execution is often very different from what you might expect and it is harder than usual to see the data.
Data is not easy to get at this way. When you look at local variables (dv is the command) then you may see that variables are simply listed as eclipsed which means that the memory is being used for something else as well within the function lifetime or that the name is not unique in this context. Enums just show as integers or longs and objects show as pointers. In fact, they always were exactly that but the VB IDE hides that from you. VB strings are COM BSTRs (and accordingly Unicode) under the covers and byte arrays are really char arrays. You might be surprised to discover that VB strings are Unicode as VB appears to have no support for anything but ANSI. That is because the Ruby forms engine was ANSI only. The runtime converts the Unicode strings to ANSI for Ruby and API calls although there are ways to pass Unicode if you want.
You are not going to be able to get at the Err, App or Printer objects since you would need to go through a lot of internal and completely undocumented structures to get at them. Even if you could get there, they would just be raw data without the accessor functions that you use in VB. If you need to look at any of those fields, your best bet is to embed debug code in the source code to copy their values to somewhere that you can get at.
You can step in to the VB runtime if you want but it probably won’t be very revealing if you are trying to debug your application. If you do, you will notice that VB’s internals are very COM influenced. The influence was actually two way since some COM ideas came from VB originally.
You may see exceptions when running your code. Null reference exceptions (i.e dereferencing a null pointer) are not uncommon or anything to worry about. They will show up as first chance C000005 exceptions with a 0 or almost 0 address. The runtime will sometimes do that if there are objects set to nothing but that is safe because the only possible values are null or a valid value. You will also see exceptions if your code does lookups in collections and the value is not there. Because exceptions are now so expensive, you probably want to avoid doing that if you can. Another exception that you will commonly see is c000008f. If you look the number up then you will find that it is a floating point inexact result exception. It is used in a different meaning here – since we don’t generate real floating point inexact result exceptions, they can safely be thrown to indicate VB errors of the normal trappable type.
Debugging hangs and crashes in VB components is done very much in the same way as with any other unmanaged component but it is just a little harder because of the compilations described above. If you have to try debugging VB code this way, I would strongly recommend that you start on a "Hello world" application and work your way up. All the things that may VB an easy language to code in make it a terrible language to debug.
I believe that when debugging in VB6, it does not attach to a running binary but instead interprets the code within it's own process. This is why the Task Manager and Win32 APIs show VB6.exe as the running app when debugging.
Also as you say, VB6 sometimes short-circuits calls to COM libraries so intercepting these calls is not always possible.
You're probably going to have to resort to intelligent logging (i.e. log the values of variables around the points where the errors you are getting occur in the hope of locating the line of code it occurs on, and/or the state of relevant variables.)
Good luck
