So it's now 4 am on the last day of the christmas holidays - well, at least if you include the weekend :-)
I spent almost all the time on the Debugger - mostly to "port" it to newer versions of Linux which are using NPTL. After reading its design paper, I was really impressed and anxious to get it working in the debugger. There are several improvements in the new 2.6.0 kernel which make life a lot easier from a debugger's point of view.
The most important change is the old pthread's helper thread is now gone and that you can now request debugging events being generated when the child forks or creates a new thread. This is done with a special waitpid() status code - and then you can call ptrace(PT_GETEVENTMSG) to get the new thread's pid.
When a new thread is created, the kernel now automatically sets the trace flag on it - this means you don't need to explicitly attach to it anymore. This was giving me a bit of a headache since it conflicted with debugger's event handling - but after redesigning the SingleSteppingEngine and the ThreadManager, I realized that the advantanges of this new model really justify the costs of the rewrite.
The debugger is now using one single thread with one single event loop for each child, ie. for all its threads. This will simplify a lot of things since all the locking and synchronization issues between the different SIngleSteppingEngines are now gone. It's also far more easy to stop/resume all of the target's threads - a simple SIGSTOP to any of its threads will do. This is also making things like stepping over another thread's breakpoint a lot easier.
We can now also reliably kill the target - a SIGKILL to any of its threads is always deadly for all of them. The same thing applies the the debugger itself: since there's just one event thread, calling Thread.Abort() on it will effectively and reliably shut it down.
Things are still very experimental at the moment - the new SingleSteppingEngine (which is now also our ThreadManager) is working perfectly fine, but things like support for non-NPTL systems (everything older than Red Hat 9) or debugging managed applications is currently broken, I need one or two more days to get this working again ....
I'm also planning to kill some old and obsolete code and remove features which were never really implemented (like storing a debugging session on disk, for instance).