Martin's Activity Log

MDB is learning ARM

2011-01-09T12:24:00-05:00

This week, I bought an HTC Desire HD. It's a really great Android phone (though the very little battery lifetime really sucks), I never want to miss a smartphone anymore. Having this cool new toy also made me very excited and enthusiastic about learning more about Android and the ARM architecture.

I only knew very little about ARM assembly language before I learned it this week - and how can you learn any better than by doing ?

So I started to hack again on the MDB and added ARM support to it. Adding support for a new hardware architecture was actually a lot easier than I expected. The biggest hassle was setting up some scripts to compile a 'mono' binary for armeabi.

Most of the architecture-specific stuff is implemented in managed code, in the backend/arch/ directory. The most important piece is the ARM instruction decoder, which is necessary for both single-stepping and stack unwinding. For single-stepping, we need to know whether the current instruction is a method call, a jump or a ret - speaking in x86's terms. On ARM, we treat BL as a method call and all other instructions that modify %pc as a jump.

Unlike the x86, ARM saves the return address of a method call in a special register called %lr - to return from that method, you simply set the program counter %pc to the return location that's stored in %lr - either by using a B or a MOV instruction.

While single-stepping, the debugger always steps by machine instructions using PTRACE_SINGLESTEP unless it's stepping over a method call - by setting a breakpoint on the next instruction and resuming execution with PTRACE_CONTINUE. Note that we also single-step by individual machine instructions if the user issued a next command.

A special case are managed trampolines and stepping over breakpoints.

Regarding stack unwinding, it pretty much looks like there is no "standard" prologue on the ARM like on the x86. where each method starts with push %ebp, mov %bp, %sp. On ARM, a method's prologue usually saves some registers on the stack and then sets either %fp or %ip to the current %sp.

In MDB, I wrote a basic instruction decoder which scans the entire prologue code and looks for instructions that save registers and setup %fp. At the moment, it aborts if it finds anything it doesn't recognize, but I did some tests with both managed and unmanaged frames and it seems to work just fine so far. I haven't done any LMF support yet, so crossing managed <-> native boundaries only works if all methods on the stack have debugging info; the stack unwinder needs to know the method's start address so it can scan its prologue.

Method invocations are already fully supported - this was surprisingly trivial thanks to the new generic invocation API, which moves most of the logic into architecture-independent code. In the old MDB, we had to write 5 different marshallers for different function signatures on each architecture. The new MDB only uses one - which takes a pointer as argument and returns a pointer. We always call the same C function and pass it a pointer to a data structure.

I just committed everything to my private forks, the code is in https://github.com/baulig/debugger/tree/last-hope and you need Mono from https://github.com/baulig/mono/tree/last-hope.

This currently works by copying a statically linked mdb-server, mono and mscorlib.dll to /data/data/martin in the Emulator and starting mdb-server from a shell, there is no Android integration yet. I'm only using the Android emulator as an emulator of the ARM architecture. Compiling Mono for the emulator is complicated and takes a lot of time and patience.

As a next step - whenever I find the time to work on this again - I want to look at Mono's xdebug and see how this can be integrated into MDB. The entire managed symbol file stuff in MDB is very old, so I'm planning to see whether we can reuse some of the newer code here - ideally, we could even use some code from SDB to get variables etc.

MDB's new backend

2010-09-01T14:50:00-05:00

Over the last couple of days, I did some extensive refactoring in MDB's backend and completely rewrote it.

The new backend now lives out-of-process and is written in C++ and there's a managed C# counterpart which talks to the C++ classes over a wire protocol.

One big mistake I made in MDB's old backend was abstracting things in the wrong way. The old backend was basically just wrapping C functions like ptrace() or waitpid and making them accessible from managed code. The big problem with this was that these functions have very specific calling semantics which needed to be replicated on the managed side to use them correctly.

Take ptrace() as an example - this function has to be called from one special thread and from this thread only. Or waitpid() as another example - interrupting it isn't trivial and there is no easy way of waiting for both the child process and user input. Because of this, more and more hacks where added to the managed part of MDB - hacks which were designed to work around specific calling semantics of these system calls. All this stuff made porting MDB very difficult. For instance, when you want to port it to Windows, WaitForDebugEvent must be called from the thread that created the child process - so the "wait event loop" has to be completely different than on Linux.

Primary goal of the new backend is wrapping functionality, not functions ... like "step one instruction, give me an event when done" - it doesn't matter how exactly we wait for the target to stop again, all we're interested in is getting that event.

This doesn't imply rewriting all of MDB in C++, but doing a little bit more in C++ than was previously done in C will make the C# code so much cleaner and so much more portable. As an added benefit, all the managed code will be fully debuggable by a managed debugger - something that was impossible in the old MDB :-)

The backend refactoring is now done, and we can do multi-thread single-stepping, so this afternoon, I cross-compiled Mono (trunk) from Linux to Windows, enabled mdb support in the runtime and started to play around with it ...

MDB getting ready for the next Millenium

2010-08-08T18:09:00-05:00

There's one really cool thing about GitHub which I really appreciate a lot: you can create a private fork of a project and put your private changes there. And you can keep them there - for weeks, months, even years - they won't ever get lost. Until recently, I wasn't a big fan of long-lived private weekend projects - they tend to bit-rot, get lost, etc. etc. etc.

This all has so much improved since we switched to GitHub, so I started a really cool weekend-project which involves some huge MDB refactoring and also some cool new stuff :-)

The first thing I did was make MDB's "server" part (the unmanaged "libmonodebuggerserver.la" library) actually a "server" (a stand-alone executable called "mdb-server.exe"). This has three main advantages:

It gives me complete control over the process - I deal with signal handlers, signal blocking masks, threads and such as I want and there's no Mono or user-code getting into my way. One major problem in the current debugger code is that I have to call waitpid (and also interrupt it somehow) without knowing what some other user-created thread might be doing at the moment. This was a major headache when embedding MDB and also set some restrictions you might not even know about (for instance, it's strictly forbidden to use fork(), sigprocmask(), waitpid() and some other system calls inside the MDB process - if you need any of these, or can't guarantee they're not used, then you need to wrap the Mono.Debugger.dll library and use it in an external process).
It makes it possible to debug MDB's C# code with an IDE such as MonoDevelop, Visual Studio / MonoVS or even MDB itself.
All operating systems that I can think of have this restriction of not letting you debug a debugger - this goes away by putting the "debugger" part of MDB into a separate process.
It makes it much easier to port MDB to other operating systems and/or hardware since you don't have to deploy the entire C# development environment to that machine - all you need is one statically linked binary ... which can even be cross-compiled from Linux.

I already set things up so that you can build the server independently from the C# pieces - my plan is to add some MonoDebugger.csproj in near future and then use xbuild to build the managed libraries. Long-term, I'd really like to use MonoDevelop to build, develop and debug MDB.

On the C# side, there's now a new abstract class called Mono.Debugger.Server.DebuggerServer which has two implementations - NativeDebuggerServer for the local [DllInvoke]'d library and RemoteDebuggerServer to talk to the external process. As part of this refactoring, I moved all [DllInvoke]s into the NativeDebuggerServer implementation and some other classes like ThreadManager have become abstract.

In this new server, we can also use a new event loop model for Linux. Since I now have full control about signal masks in all threads, I don't have to use hacks to interrupt waitpid() anymore. Instead, I now simply block SIGCHLD in all threads (there's only the main thread at the moment), sigwait() for it in one background thread and turn the signal into a select()-able event by using the "self-pipe trick".

The second big task is porting MDB to Windows and Android.

Windows is actually a bit more difficult on the server side because it requires me to build a different main event loop, whereas Android is Linux based and can thus reuse most of the Linux code.

The big difference between Windows and Linux is that the latter sends events for individual threads while the former always treats the entire process as being stopped or running.

From the C# code's perspective, there isn't so much of a difference since we now simply get an event from the remote server and don't need to worry about how it got it. There also isn't any managed wait loop anymore.

I'm really excited about this new code - I got basic single stepping almost working for all three platforms (breakpoints aren't done on ARM yet) and my next step is getting multi-threaded applications working.

It's certainly a very complex and time consuming challenge - but as I said earlier, there's no need to worry, it's all on GitHub in a private fork, no patches which might get lost over time ...

Hackweek

2010-06-06T18:08:00-05:00

This coming week is Hackweek V - and I picked a really cool project:

The ultimate goal is making MDB ready to debug moonlight and other "mixed-mode" applications.
Mixed mode means applications which are half unmanaged and half managed.

This requires quite a bit of work on mdb's unmanaged debugging support, but I definitely think it's worth it and it'll make mdb ready for the next century.

My battle plan looks like this:

Add support for the JIT's new dwarf debugging code
Integrate this into mdb's stack unwinding code
We're already using the dwarf frame info for unwinding unmanaged stack frames, but not for managed code and there are some bad hacks in place to support things like trampolines.
Fix attaching and debugging embedded applications.
Fix a thousand bugs in the unmanaged debugging code.
See whether we can improve the debugging information from the JIT a bit, cleanup the old debugging code and make it more dwarf-based.

12 Years Ago ....

2010-05-19T10:43:00-05:00

Twelve years ago, on May 19th 1998, I officially started to work on GNOME by importing LibGTop into GNOME CVS.

In the meantime, GNOME has first migrated to SVN and then to GIT, but that revision still exists:
commit 33cdc01c1db3a1cafaeb63e50247072b82ef49ea in GIT.

Time to celebrate the anniversary :-)

On Egypt

2010-01-05T05:07:00-05:00

Ok, so I got an email this morning from someone asking me to reword that last sentence of yesterday's blog post and warn people about high criminality in Egypt ...

Well sure, I wasn't saying Egypt is the paradise - crime rates are still very high there and, since our monthly income is so much higher than theirs, we tourists are their prime targets. Of course, these guys try to squeeze as much money as possible out of us - you see that for instance when trying to walk across a Bazar, or at the Pyramids of Giza, they try to sell you all kinds of worthless junk for your precious EUR / USD and they're really obtrusive. Our tour guide told us to never to walk alone and never talk to anyone.

Now I received this link to a german speaking website (www.cibev.de) this morning, which warns about tourists - and especially women - losing all their belongings in real estate deals or marriages. Ok, so I passed along the message ...

However, I also saw the other side - how the country cares about us tourists. All the increased security measures and high police presence to make it safer for us.

Because of this, I'd still recommend Egypt for everyone who's simply looking for a relaxing vacation - and it should be perfectly safe as long as you follow some simple security precautions:

Book a package holiday in a local travel agency - they'll pick you up at the airport, bring you to your hotel and look after you throughout your entire stay.
Inside your hotel, you should be perfectly safe - there's security at the entrance, so no locals can get inside. Hotels in Hurghada are huge - they usually have 10+ different buildings with gardens, pools, etc. in between.
Don't leave your hotel alone, stay together with other tourists or book some guided tours.
If you want to do some sightseeing, there are plenty of guided tours where you're in a larger group of other tourists together with a tour guide and sometimes even a security guard.

Back from Egypt

2010-01-04T11:36:00-05:00

Just came back from a wonderful vacation in Hurghada, Egypt :-)

I got really lucky to get some great last minute deal, just on the day prior to departure, and even more lucky that such a great destination as Egypt was about the only thing that was still available.

Had a really great hotel and beautiful weather - warm enough to swim in the Red Sea. This is just the perfect time to travel to Hurghada - it's their winter season; during their long summer it's just too hot in the sun.

They also offered some great daytrips, so I went to Cairo, Luxor and two snorkeling trips in the Red Sea. In Cairo, we visited the Pyramids of Giza - our tourguide was really cool, when we got there in the morning it was so foggy that you couldn't see anything, but then he offered to come back in the afternoon, which was really cool :-) We also visited the Museum of Cairo, had Lunch onboard a ship on Nil, went to a Bazar and a few more things.

In Luxor, we had a smaller group and visited the Valley of the Kings, the Temple of Hatshepsut and the Temple of Karnak - I wish I could have spent a lot more time there, the Valley of the Kings alone is worth spending an entire day.

The two snorkeling trips were also really cool, we were on the boat the whole day, stopping at different islands all the time to do snorkeling. They also offered free snorkeling instruction in the hotel pool for beginners like me, and our guide was also very patient with guys like me who weren't that good swimmers - I looked kinda silly in that oversized life jacket ;-)

All in all, it was really a great vacation, very relaxing and a lot of fun. I can really recommend Egypt to everyone.

Getting ready for "The Big Move" ....

2009-12-04T01:04:00-05:00

There's one thing about Windows Vista and Windows 7 which really sucks: unlike Windows XP, it cannot boot from an external USB hard disk.

To make things worse, running Vista or Windows 7 inside VMware also isn't such a great idea if you're planning on doing development. As soon as you're running some huge application like Visual Studio and then try to open something else in the background, it's starting to get really slow. It's not only CPU power, the external USD hard disk is also much slower than an internal one. I first experienced the problem by accident when I had Windows Update configured to automatically download and install updates in the background - and turning that off still didn't satisfy me.

Yesterday, I finally considered the external hard disk a failed experiment and called a local Lenovo shop to get the real problem fixed and replace the internal hard disk by a bigger model; I'll get it replaced on Tuesday.

Since I don't want to spend too much time reinstalling everything again, I'll move my entire Linux installation onto an empty partition on that external disk. Linux doesn't have any problems booting from such devices, so this move should go rather smoothly. Next weekend, I'll migrate everything back - I Google'd a bit, and it seems to be possible to migrate a Windows installation to another hard disk without reinstalling, so hopefully this'll save me a lot of time.

Adding Soft Debugger support to MDB

2009-11-30T10:51:00-05:00

Last week, I did some research about adding support for Zoltan's new Soft Debugger to MonoVS and / or MDB.

After playing around with it a little bit and looking at MonoDevelop's implementation, I decided to take a different path and kill two birds with one stone - by adding support to the debugger itself. Since this debugger already has a pretty good client / server abstraction, things are working out pretty well so far - I'm basically creating a new soft debugger "backend" and provide a new 'Language'.

This means we will not only be able to use the current cli debugger on the iPhone, but also get the MonoVS integration at no cost.

At the moment, I already got some very basic single-stepping working - we use some quick hack to stop on the first line of Main(), and then you can do some stepping. My next challenge is implementing the type system and then adding support for breakpoints.

Martin 2.1 - Fine-tuning the installation is a beast

2009-11-24T05:59:00-05:00

There is one thing about installing a new operating system which really sucks: fine-tuning the installation - which usually takes a lot more time than the installation itself. So I spent almost the entire weekend doing post-installation stuff and in the end, I installed everything three times before I was really satisfied with the result.

The first thing to consider was whether to use physical partitions or virtual disk images in VMware - I couldn't find much information about this on Google except that using physical partitions was considered an "expert" option. The only real advantage of using raw partition seems to be increased performance, but you can't take any snapshots of your VM with this option. I still decided to use them because disk space isn't really a problem for me anymore, I can easily use traditional Windows backup to backup the entire system to an unused hard disk partition, but getting maximum performance is very important for me.

Another important thing to decide was the amount of RAM to allocate to the VMs - the host has 4 GB, so I had the idea of running two VMs at the same time. After trying several options between 1-2 GB, I finally realized that my laptop isn't powerful enough to handle this, each time I tried the machine started swapping like hell and it took over an hour till I could do anything useful again (I didn't want to risk damaging the installation by doing a force shutdown, so I had to wait till it was done booting and let me shutdown cleanly). Now I'm allocating 2 GB to the VM, which means that both host and VM can run without any swapping.

Most of the work was fixing all these tiny but annoying little issues that came up while installing this stuff.

Last thing I had to do last night was running a backup of both installations.

Martin 2.0 - The Upgrade

2009-11-18T12:12:00-05:00

After our great MonoVS 1.0 Release, I finally got tired of doing disk surgery and repartitioning to gain more disk space. When I bought my laptop about a year ago, it was state of the art - 100 GB was the biggest hard disk that was available back then. Since the initial installation, I repartitioned at least 4-5 times to increase the Windows partition.

One thing that's really bad about Windows is if you run out of disk space while doing any kind of upgrade or software installation and you have system restore turned off - that was actually one of the first things I turned off to save disk space - it can leave your system in an inconsistent state where it'll fail to install any further upgrades. I run into this problem while trying to install ASP.NET MVC to fix some bug.

This week, I bought an external 1000 GB - and I refuse to call this a [I]terrabyte[/I] - USB hard drive and then completely reinstalling my system. My primary Vista partition is now 60 GB and after the clean install with Vista x64 SP2 and Visual Studio 2008, there's still plenty of space left, but I also won't install some huge software packages like OpenOffice on it.

In addition to this, I now have two VMware images of 200 GB each - one running Windows 7 with both Visual Studio 2008 and 2010 and the other one running Vista and Visual Studio 2008. I also created a 350 GB backup partition and reserved 200 GB for future extensions - for instance installing a different Linux version etc. - disk space is not something I need to worry about anymore ...

Now I'm basically running MonoVS "the other way around" - with Linux as the host and Windows inside the VM. This was necessary because there still isn't any support for installing Windows 7 onto an external USB hard disk, so you have to run this in some kind of a virtual machine. At the moment, I assigned 2 GB of RAM to the VM (my laptop has 4 GB of RAM) which means I can only run one of them at a time, but at maximum performance without any swapping.

Tomorrow, I'll do some more fine-tuning of the new system and then see whether we can run MonoVS on Windows 7 and/or with Visual Studio 2010. My guess is that the Windows 7 / Visual Studio 2008 setup will just work out-of-the box without any changes at all, but getting it working for Visual Studio 2010 most likely requires some minor tweaks to the registry code.

Welcome to the Universe

2009-08-02T09:22:00-05:00

Since all the cool guys are on Twitter these days, so am I - http://www.twitter.com/baulig/.

I'll also update my blog more often in near future.

Booting SuSE 11 inside VMware

2008-07-18T14:52:00-05:00

Two weeks ago, I got a new laptop and set it up to dual-boot both Windows Vista (which I need for work) and SuSE 11. Inside Vista, I also installed VMware and made it boot my existing Linux installation.

Using raw disk partitions in VMware is a bit dangerous and you have to be really careful when setting this up. Here's how I configured it - and it turned out to work really great:

First of all, make sure you select individual partitions in VMware and not the entire disk. Make sure you don't select the partition containing your host Windows operating system. This ensures that VMware doesn't let the guest operating system access that partition, so you can't accidentally boot or mount it - which'd result in severe data corruption and possible loss of all data on the disk.
Before booting the first time inside VMware, you need to enable SCPM and create two profiles - I'm using gondor (running natively) and isengard (running inside VMware). In yast, go to System / Profile Manager, active it there and make sure to set both Switch Mode and Boot Mode to Save Changes.
Then do
```
scpm copy default gondor
scpm switch gondor
scpm save
```
on the command line. This'll save your current configuration into the gondor profile. After that, do a scpm copy gondor isengard scpm switch isengard scpm save

Edit /boot/grub/menu.list, replace the SuSE 11 with Gondor, add PROFILE=gondor to the kernel command line and make sure you're using root=/dev/disk/by-uuid:

title Gondor
    root (hd0,2)
    kernel /boot/vmlinuz root=/dev/disk/by-uuid/b39b72d4-ac4e-4385-b9a3-08ca81b4638f splash=silent PROFILE=gondor showopts vga=791
    initrd /boot/initrd

title Mordor
    rootnoverify (hd0,2)
    chainloader (hd0,0)+1

Copy /boot/grub/stage2 to /boot/grub/state2-isgengard and also copy /boot/grub/menu.list to /boot/grub/menu-isengard.lst. Edit /boot/grub/menu-isengard.lst, remove the Mordor entry and replace Gondor with isengard. This configuration will be used when booting inside VMware:
```
title Isengard
    root (hd1,2)
    kernel /boot/vmlinuz root=/dev/disk/by-uuid/b39b72d4-ac4e-4385-b9a3-08ca81b4638f splash=silent showopts PROFILE=isengard
    initrd /boot/initrd
```

Edit /etc/fstab and use /dev/disk/by-uuid everywhere instead of /dev/disk/by-id or using the raw device. My /etc/fstab looks like this:

/dev/disk/by-uuid/b39b72d4-ac4e-4385-b9a3-08ca81b4638f /                    ext3       acl,user_xattr        1 1
/dev/mapper/cr_sda6  /home                ext3       acl,user_xattr,noauto 0 0
/dev/disk/by-uuid/13aee839-dccc-4999-a977-6168d109dae1 /work                ext3       acl,user_xattr        1 2

This is important since your hard disk will have different physical devices when running inside VMware, so /dev/disk/by-id will be different.

Now you can reboot your system into Windows.
In VMware, create a really small (10 MB is fine) virtual disk in addition to your physical hard disk. Double check that you're using individual partitions and the partition containing your Windows host operating system is not selected.
Boot inside VMware, press F2 to go to BIOS setup, select the virtual hard disk as primary boot device, save and exit. Press Esc to go to the boot menu, select CD-ROM as temporary boot device and boot off the SuSE 11 installation DVD, select Rescue System
Once in the rescue system, /dev/hda should be the virtual hard disk and /dev/hdb your physical one. Manually install grub into the MBR of the virtual disk and make it use /boot/grub/stage2-isengard and /boot/grub/menu-isengard.lst. You have to do that inside the grub shell, by using appropriate root and install commands - this is a bit dangerous since it could make your system unbootable, so check the manual.
Note that we only need the MBR in the virtual hard disk - the boot loader in it will load /boot/grub/stage2-isengard from your physical hard disk. Grub encodes the path of the configuration file (/boot/grub/menu-isengard.lst) in the stage2, that's why we're using a custom one.
This setup is a bit complicated, but it ensures that you don't accidentally boot Linux with the wrong profile.
Remove the SuSE 11 installation DVD and reboot.

With the two different boot loaders, you'll always boot your Linux with the correct SCPM profile. The SCPM profiles make sure you can have different configurations when running natively and inside VMware. For instance, your /etc/X11/xorg.conf will be different in each scenario and you may also have different network settings.

I'm even using different host names and different IP addresses for both configurations - the one "machine" is called gondor.trier.ximian.com (192.168.3.8) and the other one is called isengard.trier.ximian.com (192.168.3.9). You may also add additional files / services to SCPM to also save things like /etc/issue and /etc/motd, for instance.

SCPM also takes care of the different X11 configuration - when running natively, I'm using a multi-headed setup with an external monitor - and in Vista, I move the VMware window into the external monitor and use full-screen mode there. This means that I have Vista on my laptop's main display and my GNOME session on the external monitor - really awesome !

On Isengard, I'm also using Samba to mount /work/mordor from Windows - I'm using a separate host-only network (VMnet1) for that. To make this work on Vista, create a host-only network (in addition to bridging the ethernet card), assign it a static IP address (I'm using 192.168.3.0/24 for my "real" network and 192.168.8.0/24 for the host-only one) and set the default gateway to that address (address 192.168.8.1 / netmask 255.255.255.0 / gateway 192.168.8.1) - this'll fix the "Unidentified Network" problem in Vista.

Copying symbol files from Windows

2008-06-18T11:13:00-05:00

Today, I added a cool new feature to the debugger: you can now add "directory maps" to the configuration file.

This is very helpful if you copy a symbol file from Windows to Linux. Simply add something like this to your ~/.config/MonoDebugger/MonoDebugger.xml:

<?xml version="1.0"?>
<DebuggerConfiguration fileversion="1.0">
  <DirectoryMap>
    <Map from="c:/cygwin/home/ichotolot" to="/home/martin/work" />
  </DirectoryMap>
</DebuggerConfiguration>

The debugger now automatically detects whether a symbol file has been created on Windows and converts the file names if necessary. This also works when using pdb2mdb to convert a .pdb to .mdb.

You could, for instance, mount c:\cygwin\home\ichotolot on /work/mordor using samba and use the configuration file to tell the debugger where to look for the source files.

Back from Madrid

2007-12-04T19:00:00-05:00

After a really exciting week in Madrid, I'm now back at home and working on the debugger again.

We had a few really productive discussions about the debugger and sometimes, I got the impression that people didn't know that we actually have a working debugger before I did my little presentation at the meeting. Well, the debugger from SVN HEAD is considered stable and working - there will be an official release really soon.

Here are some pictures from our meeting:
http://www.flickr.com/photos/baulig/collections/

Ikea

2007-10-24T18:16:00-05:00

Today, I went to Ikea in Koblenz to buy an armchair. It's really amazing to see how one room can completely change its shape within one hour:

Before driving to Ikea:

And after, with my new armchair:

Debugger currently broken

2007-10-16T13:12:00-05:00

Unfortunately, I have some really bad news for all debugger users:
The debugger is currently broken and unusable because of recent changes in the Mono runtime. I can't give any ETA yet on how long it'll take to get it fixed since I need to do some research first. At the moment, I don't even know how to fix it, so I need to think about a solution first. Hopefully, I'll have more news after one or two days of research.

There's a Mono revision mentionend in the debugger's ChangeLog which is the last one which is guaranteed to work.

Debugger type system

2007-09-28T09:16:00-05:00

Yesterday, I finally finished the work I started last week on the new debugger's type system.

For each local variable and method parameter, the debugging info generated by the runtime now also contains a MonoType *. On the debugger's side, we read this MonoType * and create the Cecil type from it. There have been some extensive API changes in the debugger's type systems since we now need to do a target access to resolve a type - that's something which would have been neccessary for generics anyways.

As a side-effect, we can now also unload and free the debugging information about types once an assembly gets unloaded.

Next week, I'll start to work on "real" generics support.

Debugger and Generics

2007-09-20T10:07:00-05:00

About two weeks ago, I started to add generics support to the debugger.

After doing some research, I soon realized that things are a bit more complicated than I expected, and I need to modify some things in the debugger.

Let's assume we stopped inside some method Foo.Hello() and we want to print some local variable.

Without generics, the debugger simply used Cecil to get the type of that variable.

However, in order to actually do anything useful with the variable, the debugger needs to know a bit more than just its Cecil type. Precisely, it needs to know the exact layout of its fields and - if it wants to access a property - also the MonoMethod * addresses. Because of this, the debugger needs to read the type's MonoClass *.

At the moment, the debugger keeps an internal mapping between Cecil types and MonoClass * pointers. This works because a Cecil type could uniquely be identified by its image and type token - the debugger just read the image and type_token fields from the MonoClass * and then it had something to uniquely identify the corresponding Cecil type.

With generics, things become more complicated.

While Cecil knows about generic types etc., it obviously has no way of knowing the actual generic instantiation of a type - that's something which dynamically happens at run-time.

My first approach to the problem was to read the current generic context (MonoGenericContext *) and construct the Cecil type from it.

This worked, I could instantiate the type in the debugger and display it to the user - for instance IList<int>.

However, the debugger still needs to know the MonoClass * - and here comes the problem.

It's no longer enough just to read image and type_token, we also need to read the generic_class and interpret it. We can't get any unique identification of the class unless we fully read and interpret the generic_class. Of course, this is an expensive operation. We can't just fully read any single MonoClass * that's created just for the sake of inserting it into a hashtable.

Because of this, I had another idea.

Rather than reading the type of the local variable from the symbol file / Cecil and then get its MonoClass * from a hashtable, we directly read it from the target. The debugging info generated by the runtime will be extended by a MonoClass * field.

Key point is this:
It is easy for the debugger to construct a Cecil type from a MonoClass * - but it has no way of getting a MonoClass * from a Cecil type. So the thing the debugger really needs is the MonoClass *, not the Cecil type.

Initializing the debugging code

2007-08-23T15:22:00-05:00

With yesterday's changes, the unmanaged debugging API has changed. If you're embedding Mono and want to initialize the debugging API, you only need to call mono_debug_init () - the old mono_debug_init_1() etc. hacks are now all gone (caution: do not call mono_debug_init_corlib()).

mono_debug_init() needs to be called before the first appdomain is created - after that, everything will go automatically.

Domain unloading and identifying threads

2007-08-22T12:24:00-05:00

Today, I committed two important bug fixes in the debugger which both required a lot of code changes.

The most important user-visible change is that the debugging info is now stored on a per-appdomain basis - which means that it can be freed when the domain gets unloaded. We can now also unload symbol files. This required quite some changes - both in the runtime and in the debugger.

This wasn't only important to save memory, but also for correctness and robustness:
When a domain gets unloaded, all methods which were JITed in it are also freed - which means that the corresponding debugging info also isn't valid anymore. We now tell the debugger when an appdomain gets unloaded, so it can also free the corresponding symbol tables.

The second important change was a fix for a severe bug when attaching to a managed application. It is a bit hard to explain what this code exactly does, so this'll get a little bit technically.

On Linux, there is some kind of an API problem with the kernel and libc wrt. threads:

When you run a multi-threaded application, all threads share the same PID - so if you call getpid (), you'll get the same result in each thread. Internally each thread is a seperate `task' for the linux kernel and it also does have it's own pid - which is also called LWP - but there's no (portable) way of getting that from user-level code. That's why user-level code normally uses pthread_t (which is returned by pthread_self()) to identify a thread - we call that a TID.

When attaching to a managed application, the debugger needs to get some information about all the managed threads - precisely the LMF address, which is required to generate managed backtraces when we're stopped inside native code, and some other internal information. The only thing the runtime can provide is a TID to LMF mapping - but the debugger only knows about the LWP and not about the TID.

So I had to find a way of getting the TID from an LWP - the old code simply called pthread_self() in each thread, but of course this approach has several problems. The most severe one is that this obviously doesn't work for core files. It also doesn't scale very much to a large number of threads since invoking methods in the target is extremely expensive. Another important point is that I want to be able to generate a backtrace before executing any code in the target - that's very important if the target is not responding anymore.

After doing a lot of research, I finally decided to use glibc's thread_db library - this can be used to get information about all the threads, so after some work I finally had it working.

We still need to use some really bad hacks for core files, but they're now working as well.

So to summarize, almost two weeks of hard work - but now, attaching and domain unloading is finally working :-)

Tomorrow, I can finally move on to generics ...

Domain unloading and the debugging code

2007-08-08T16:09:00-05:00

Yesterday, I got an important and interesting bug report:

We currently do not support domain unloading in the new debugging code. This means that when a domain is unloaded, we do not free the debugging info. To make things worse, some internal hashtables also contain references to the MonoMethod *'s which will lead to crashes later on.

Freeing the debugging info is not so easy since the debugger may access it at any time, so we need to tell the debugger first. However, I got a really good idea this morning and am currently working on it.

Native line numbers

2007-07-26T09:16:00-05:00

After finally fixing the performance leak last week, I started to work on another rather difficult problem.

It all started with a rather trivial bug report:
Miguel sent me a small moonlight application and said that the debugger would crash when trying to get a backtrace from within an event handler.

Sounded like a rather trivial problem, so I started debugging ....

Things turned out to be a lot more complicated - after spending a lot of time doing some research on inline methods and extensive reading of the dwarf spec, I found a huge design problem in the code which reads the line numbers from the dwarf debugging info.

In DWARF, line numbers are stored in a separete section of the symbol file and not in any way related to methods. The debugging information entry for a method (DW_TAG_subprogram) only contains a start and end address - but no information about where the line numbers for that method are stored. The line number table, on the other hand, only contains a mapping from addresses to line numbers and contains no information about methods. To make things worse, the line number table does not need to be continuous and there is no separation whatsoever between different methods.

This makes it rather complicated for the debugger to read line numbers. It looks like the old code only worked by accident since older versions of gcc always created a continuous line number table - g++ doesn't (I could even create a test case where g++ produces a non-continuous line number table where the test case doesn't contain any c++ features at all).

To make things work, I had to do some changes in the debugger: rather than having on LineNumberTable per Method, each Method now contains a reference to a LineNumberTable but it doesn't "own" that table (ie. it could be shared between different methods). I had to move a lot of code around for this; for instance, the LineNumberTable now longer contains the method's start and end row etc.

While working on this, I soon began to realized that a change like this would have been required anyways when supporting compiler generated code. From the debugger's point of view, the difference between an anonymous method (in C#) and an inline method (in native code) is not so big at all.

If everything works out fine, I should be able to finish the new dwarf code till the end of the week and then start to work on compiler-generated code on Monday.

Debugger Performance

2007-07-19T04:08:00-05:00

Last night, I finally found the performance problem in the new debugger code - and it was really trivial:
We were missing a mono_debugger_unlock(), so we got a deadlock on exit. There's a 2 second timeout in mono_domain_finalize() when it's called from mini_cleanup() and we have 39 nunit tests - makes exactly 78 seconds in total.

Things are now fixed and the new debugger code is as fast as the old one :-)

More symbol table stuff

2007-07-11T16:24:00-05:00

My new breakpoint code is coming along really good :-)

Yesterday, I got all the required features working so I started testing. The good news is that this code actually works - multiple appdomains and generic instances are working just fine :-)

The longer I'm working on this, the more problems I find in the old symbol table code. There is, for instance, a piece of code which never worked for the past 2 years - but nobody ever noticed because it can only be triggered if you run a really, really huge application with ---debug. I only figured it out by accident while I decreased the limits to do some performance testing. There are also race conditions when we need to start a new symbol table since the debugger might currently be reading it.

However, this shouldn't be too bad - I already have some really good ideas which'll not only increase stability but also performance.

Method lookups, symbol files and breakpoints

2007-07-09T11:54:00-05:00

Last week, I started to work on true multi-appdomain support for the debugger. Soon, I realized that there's also another problem: generics.

Both appdomains and generics have one thing in common: a single method may be JITed multiple times. Because of that, I decided to use a common interface in the symbol table code for both. Now, each method in the source code may have multiple addresses.

To make things easier, I decided that in the case of generic methods, we may not insert a breakpoint on one particular instantiation - when inserting a breakpoint on a generic method (or any method in an instantiated generic class), it always affects all instantiations of that method. This doesn't only make the breakpoint code a lot easier - the fact that a method is JITed multiple times for multiple instantiations is also more or less an implementation detail of our current JIT.

When inserting a breakpoint, we first need to check which instances of that method have already been compiled and then physically insert a breakpoint on each of them. Each time, a new instance of the method is JITed, the debugger gets a notification, so it can also insert a breakpoint there.

One important thing when implementing the new code was that it should also scale well to a large number of appdomains and/or generic instantiations.

A first idea from me was to get a notification each time a new appdomain is created and then insert a breakpoint on each of them. However, since callbacks from the debugger to the JIT are expensive and reading a large chunk of data from it is rather cheap, I realized that this doesn't scale very well. Because of that, the JIT now gives the debugger a list of method addresses.

I've already got some preliminary code working, just need to do a few more tests. Before I finish this, I'd like to do some tests with a large number of appdomains and generic instances. Hopefully, I'll be able to finish this tomorrow or on Wednesday.

It can happen to the best of us .... and a long day with a lot of annoying paperwork is finally over

2007-07-03T12:35:00-05:00

Until yesterday afternoon, I thought things like EC card fraud can't happen to you if you're always careful with using your card - well, I was very wrong about that, it can happen to anyone :-(

Yesterday, after lunch I routinely checked the balance of my bank account using phone banking and was really shocked to hear that it was -4785 EUR - at the beginning of a month, something must be wrong. I immediately asked for customer support to check what was going on. First thing the customer service agent said to me was something like "Oh shit, f... we have a problem :-(" - someone was using my card to withdraw money from ATM machines in Bankog - there were a bunch of such withdrawals of 10.000 whatever-their-currency-is (which converts to about 220 EUR).

So today, I basically spent the whole day doing paperwork. First I had to go to my bank's local branch to get a detailed statement of all these withdrawals - then go to the police to file a report and then fax a copy of it to the bank.

The good news is, the bank already told me that I'll get my money back - since I still have the card, someone must have copied it when I was using an ATM or paying with it at one of these pay-at-pump gas stations.

It was just very frustrating to do all the paperwork and spend hours of waiting. The local police was extremely busy today and the officer I was supposed to talk to was called to several different operations while I was there, so I had to wait a lot.

Sometimes it's a little bit scary to wait that long at a police stations. They send you into a small room to wait till an officer has time for you and if you wait there for some time, a lot of other people come and go - most of them have problems like "Oh, I got my apartment searched yesterday, and you seized XYZ, where can I get it back?" or "Oh, I'm here to surrender my driver's license" or "Oh, I damaged XYZ while I was drunk and was asked to report here".

Live Free or Die Hard

2007-06-28T09:08:00-05:00

Yesterday, I watched Bruce Willis's new movie Live Free or Die Hard - which was really excellent :-)

This was one of the movies I really wanted to watch in the original version and also without any subtitles. The premier in Luxemburg is on July 4th and they also have these annoying subtitles, so I decided to drive to Düsseldorf and watch it there - a 300km drive from where I live, but it was really worth it.

One year and one day ...

2007-06-21T07:24:00-05:00

... have passed since I blogged last time.

That's really a very long time and a lot of things have happened in the meantime. Since about November, I'm back hacking on the debugger full-time and there some exciting news about it.

On May 9th, I released version 0.50 "Dublin" of the Mono Debugger. This is the latest stable version and also a milestone. It's one of the best releases ever and I'm really happy about it.

After that, I started some really large code cleanups and rewrites on a separate debugger-dublin branch. I had several goals with this project:

Support multiple appdomains
Rework the breakpoint code so we don't need to stop in Main() and we can also support static .cctors
Don't change the application's flow of execution as much as we're doing at the moment.

I'm now about one week away from implementing full multi-appdomain support, so I think I'll wait until that is done before merging my code back. I've posted a very detailed summary of my latest changes to our internal mailing list, which is also available in SVN.

Read the full story ...

The following detailed description of my recent work is also available on SVN in doc/debugger-dublin.txt in the /branches/martin/debugger-dublin branch.

Method, MethodSource and TargetFunctionType API changes:
The old SourceMethod is gone. We now have:
- MethodSource which is the source code of a method; it may or may not have a LineNumberTable associated with it' the LNT may or may not be loaded in memory.
- TargetFunctionType is a "high-level" representation of a method; each MethodSource has exactly one TargetFunctionType, but a TargetFunctionType may also describe a method without source code.
  A TargetFunctionType doesn't contain any information about how the method is currently loaded in memory; it's a symbol-file thing.
- Method is a low-level representation of a method and is domain-specific.
  When the application is running, each TargetFunctionType has one Method for each appdomain. In multi-appdomain applications, we create a separate Method in each domain.
  A Method may or may not be loaded (JITed in the appdomain).
This API change was done in preparation for full appdomain support and it was also required by the new breakpoint code.
The new breakpoint code - done and fully tested.
I made substancial changes to the breakpoint code which also affects the way how we deal with breakpoints in method which aren't JITed yet.
Key features:
- There's no technical requirement to stop in Main() anymore.
- Prepared multi-appdomain support:
  Each source code breakpoint location may now have multiple addresses.
- We don't need to compile a method anymore to insert a breakpoint in it.
Note: In the following, source method represents a method in the source code, identified by either its name or a filename and line number - it's basically a method in the symbol file. target method is a method in the target application; ie. a MonoMethod * in the JIT. In multi-appdomain scenarios there is more than one target method for each source method.
The long story:
Key component of the new breakpoint code is the new way how we insert a breakpoint on a method which isn't JITed yet.
Both the old and the new code have one fundamental problem in common: before we can insert a breakpoint, we need to know reliably whether that method has already been JITed or not.
The new code works like this:
- We acquire the metadata loader lock
- We lookup the method's address in the current domain's code hash.
- If it's not yet JITed, we register the JIT callback while still holding the loader lock.
- We release the metadata loader lock and tell the debugger the address or the callback ID.
This has to be done in one single callback.
The important thing is that we need to do both the address lookup and register the callback while holding the loader lock to avoid race conditions.
The old code explicitly triggered a JIT compilation of the method to get its address and then inserted the breakpoint on that address. This is bad as it has side-effects and modifies the application's flow of execution.
One key policy of the new code is not to change the application's flow of execution - the application shouldn't behave any differently when running inside the debugger.
As a side-effect of the new JIT interface, callbacks are now done per target method and not per source method anymore.
When the user requests a breakpoint on a source method, the debugger actually needs to insert multiple breakpoints since there is one target method for each appdomain.
After designing the new JIT breakpoint interface, I also needed to modify several things on the debugger side, especially in the session code.
There is no technical requirement to stop in Main() anymore. Previously, we had to stop in Main() to enable breakpoints - we now do that before initializing Main()'s class to make sure breakpoints are enabled before running any static .cctors.
We now automatically do this from inside the SingleSteppingEngine, the code has been removed from DebuggerSession.
When starting the application, the session code inserts a breakpoint on Main() - but this is just a regular breakpoint and it can be disabled and/or removed by the user.
I liked the idea that the debugger stops in Main() from a usability point of view, but now the user has the freedom to control this.
The biggest user-visible improvement is that we can now have breakpoints before Main() is executed - ie. in static .cctors.
Recursive callbacks
This is something debugger-internal which had to be done to support the next thing.
Basically, I improved the way how the debugger calls methods in the target application - we now support recursive callbacks and the stack unwinding code now also knows about callbacks, so we get correct stack traces.
Trampolines:
After the breakpoint code was fixed, I also needed to fix the way how we handle trampolines wrt stepping over breakpoints. It's a bit difficult to explain what this code is doing and why it is implemented in the way it is, but let me try .... Let's have a look at this little test case:
1 using System; public class Foo { 5 static Foo () { 7 Console.WriteLine ("STATIC CCTOR!"); } 10 public static void Hello () { 12 Console.WriteLine ("Hello World!"); 13 Console.WriteLine ("Second line"); } 15 } class X { static void Main () { 21 Foo.Hello (); } }
Looks trivial, right ? Well, it's not so trivial at all from the debugger's point of view.
Let's assume we're stopped a line 21 and the user issued a step command.
At that time, Foo.Hello() isn't JITed yet so we're actually stepping into a JIT trampoline. There's nothing special about that, we manually compile the method, get its address, insert a breakpoint on it and continue.
In the new code, the first thing we do here is manually initializing the class Foo - which'll execute the static .cctor. The new code has been designed in a way that the debugger basically "expects" to be interrupted while doing that, ie. that the user may have a breakpoint on that .cctor.
But that's not the problem here - let's assume we already initialized the class, we're done with any .cctors and already compiled the method.
The real problem here is that we have a breakpoint on line 21 and if we just continue, mono_magic_trampoline() would abort when attempting to patch the callsite because it get confused by the breakpoint instruction.
So, the code does the following:
- Initialize the class Foo (the debugger expects to be interrupted here and we correctly handle the case where the user has breakpoints on the .cctor).
- compile the method, insert a temporary breakpoint.
- acquire the thread lock
- remove the breakpoint instruction from line 21
- resume the target:
  mono_magic_trampoline() won't trigger any compilation here because we already compiled the method before; all it needs to do is patching the callsite for us.
  [There is still a very rare deadlock possible here:
  Although mono_magic_trampoline() will never actually compile the method, it may still block when trying to lookup its address if any other thread is holding the loader lock. We should find a way to explicitly pass it the address and just do the callsite patching. However, after extensive testing I couldn't trigger any deadlock here, so let's not worry unless we run into problems. The old code was way more problematic: we were also running the .cctor inside the thread lock.]
- re-insert the breakpoint instruction on line 21
- release the thread lock
The important difference to the old code is:
- We support static .cctors here and correctly handle breakpoints.
- We do not run the any managed code while holding the thread-lock
NOTE:
This is where recursive callbacks are used:
The debugger calls mono_runtime_class_init() and if that stops at a breakpoint, the debugger is still inside a callback. If the user does anything which triggers another callback, we have a recursive callback. There's a testcase in TestCCtors.cs for that.
I think we can call this done and working.

With the new breakpoint code in place, it isn't difficult anymore to add real multi-appdomain support.

What's missing is basically a way of notifying the debugger about appdomain loads/unloads and insert/remove breakpoints.

Shouldn't take more than a week to fully implement this.