Martin's Activity Log

MDB is learning ARM

This week, I bought an HTC Desire HD. It's a really great Android phone (though the very little battery lifetime really sucks), I never want to miss a smartphone anymore. Having this cool new toy also made me very excited and enthusiastic about learning more about Android and the ARM architecture.

I only knew very little about ARM assembly language before I learned it this week - and how can you learn any better than by doing ?

MDB's new backend

Over the last couple of days, I did some extensive refactoring in MDB's backend and completely rewrote it.

The new backend now lives out-of-process and is written in C++ and there's a managed C# counterpart which talks to the C++ classes over a wire protocol.

One big mistake I made in MDB's old backend was abstracting things in the wrong way. The old backend was basically just wrapping C functions like ptrace() or waitpid and making them accessible from managed code. The big problem with this was that these functions have very specific calling semantics which needed to be replicated on the managed side to use them correctly.

Take ptrace() as an example - this function has to be called from one special thread and from this thread only. Or waitpid() as another example - interrupting it isn't trivial and there is no easy way of waiting for both the child process and user input. Because of this, more and more hacks where added to the managed part of MDB - hacks which were designed to work around specific calling semantics of these system calls. All this stuff made porting MDB very difficult. For instance, when you want to port it to Windows, WaitForDebugEvent must be called from the thread that created the child process - so the "wait event loop" has to be completely different than on Linux.

Primary goal of the new backend is wrapping functionality, not functions ... like "step one instruction, give me an event when done" - it doesn't matter how exactly we wait for the target to stop again, all we're interested in is getting that event.

This doesn't imply rewriting all of MDB in C++, but doing a little bit more in C++ than was previously done in C will make the C# code so much cleaner and so much more portable. As an added benefit, all the managed code will be fully debuggable by a managed debugger - something that was impossible in the old MDB :-)

The backend refactoring is now done, and we can do multi-thread single-stepping, so this afternoon, I cross-compiled Mono (trunk) from Linux to Windows, enabled mdb support in the runtime and started to play around with it ...

MDB getting ready for the next Millenium

There's one really cool thing about GitHub which I really appreciate a lot: you can create a private fork of a project and put your private changes there. And you can keep them there - for weeks, months, even years - they won't ever get lost. Until recently, I wasn't a big fan of long-lived private weekend projects - they tend to bit-rot, get lost, etc. etc. etc.

This all has so much improved since we switched to GitHub, so I started a really cool weekend-project which involves some huge MDB refactoring and also some cool new stuff :-)

The first thing I did was make MDB's "server" part (the unmanaged "libmonodebuggerserver.la" library) actually a "server" (a stand-alone executable called "mdb-server.exe"). This has three main advantages:

• It gives me complete control over the process - I deal with signal handlers, signal blocking masks, threads and such as I want and there's no Mono or user-code getting into my way. One major problem in the current debugger code is that I have to call waitpid (and also interrupt it somehow) without knowing what some other user-created thread might be doing at the moment. This was a major headache when embedding MDB and also set some restrictions you might not even know about (for instance, it's strictly forbidden to use fork(), sigprocmask(), waitpid() and some other system calls inside the MDB process - if you need any of these, or can't guarantee they're not used, then you need to wrap the Mono.Debugger.dll library and use it in an external process).
• It makes it possible to debug MDB's C# code with an IDE such as MonoDevelop, Visual Studio / MonoVS or even MDB itself.
  All operating systems that I can think of have this restriction of not letting you debug a debugger - this goes away by putting the "debugger" part of MDB into a separate process.
• It makes it much easier to port MDB to other operating systems and/or hardware since you don't have to deploy the entire C# development environment to that machine - all you need is one statically linked binary ... which can even be cross-compiled from Linux.

I already set things up so that you can build the server independently from the C# pieces - my plan is to add some MonoDebugger.csproj in near future and then use xbuild to build the managed libraries. Long-term, I'd really like to use MonoDevelop to build, develop and debug MDB.

On the C# side, there's now a new abstract class called Mono.Debugger.Server.DebuggerServer which has two implementations - NativeDebuggerServer for the local [DllInvoke]'d library and RemoteDebuggerServer to talk to the external process. As part of this refactoring, I moved all [DllInvoke]s into the NativeDebuggerServer implementation and some other classes like ThreadManager have become abstract.

In this new server, we can also use a new event loop model for Linux. Since I now have full control about signal masks in all threads, I don't have to use hacks to interrupt waitpid() anymore. Instead, I now simply block SIGCHLD in all threads (there's only the main thread at the moment), sigwait() for it in one background thread and turn the signal into a select()-able event by using the "self-pipe trick".

The second big task is porting MDB to Windows and Android.

Windows is actually a bit more difficult on the server side because it requires me to build a different main event loop, whereas Android is Linux based and can thus reuse most of the Linux code.

The big difference between Windows and Linux is that the latter sends events for individual threads while the former always treats the entire process as being stopped or running.

From the C# code's perspective, there isn't so much of a difference since we now simply get an event from the remote server and don't need to worry about how it got it. There also isn't any managed wait loop anymore.

I'm really excited about this new code - I got basic single stepping almost working for all three platforms (breakpoints aren't done on ARM yet) and my next step is getting multi-threaded applications working.

It's certainly a very complex and time consuming challenge - but as I said earlier, there's no need to worry, it's all on GitHub in a private fork, no patches which might get lost over time ...

Hackweek

This coming week is Hackweek V - and I picked a really cool project:

The ultimate goal is making MDB ready to debug moonlight and other "mixed-mode" applications.
Mixed mode means applications which are half unmanaged and half managed.

This requires quite a bit of work on mdb's unmanaged debugging support, but I definitely think it's worth it and it'll make mdb ready for the next century.

My battle plan looks like this:

• Add support for the JIT's new dwarf debugging code
• Integrate this into mdb's stack unwinding code
  We're already using the dwarf frame info for unwinding unmanaged stack frames, but not for managed code and there are some bad hacks in place to support things like trampolines.
• Fix attaching and debugging embedded applications.
• Fix a thousand bugs in the unmanaged debugging code.
• See whether we can improve the debugging information from the JIT a bit, cleanup the old debugging code and make it more dwarf-based.

12 Years Ago ....

Twelve years ago, on May 19th 1998, I officially started to work on GNOME by importing LibGTop into GNOME CVS.

In the meantime, GNOME has first migrated to SVN and then to GIT, but that revision still exists:
commit 33cdc01c1db3a1cafaeb63e50247072b82ef49ea in GIT.

Time to celebrate the anniversary :-)

On Egypt

Ok, so I got an email this morning from someone asking me to reword that last sentence of yesterday's blog post and warn people about high criminality in Egypt ...

Well sure, I wasn't saying Egypt is the paradise - crime rates are still very high there and, since our monthly income is so much higher than theirs, we tourists are their prime targets. Of course, these guys try to squeeze as much money as possible out of us - you see that for instance when trying to walk across a Bazar, or at the Pyramids of Giza, they try to sell you all kinds of worthless junk for your precious EUR / USD and they're really obtrusive. Our tour guide told us to never to walk alone and never talk to anyone.

Now I received this link to a german speaking website (www.cibev.de) this morning, which warns about tourists - and especially women - losing all their belongings in real estate deals or marriages. Ok, so I passed along the message ...

However, I also saw the other side - how the country cares about us tourists. All the increased security measures and high police presence to make it safer for us.

Because of this, I'd still recommend Egypt for everyone who's simply looking for a relaxing vacation - and it should be perfectly safe as long as you follow some simple security precautions:

• Book a package holiday in a local travel agency - they'll pick you up at the airport, bring you to your hotel and look after you throughout your entire stay.
• Inside your hotel, you should be perfectly safe - there's security at the entrance, so no locals can get inside. Hotels in Hurghada are huge - they usually have 10+ different buildings with gardens, pools, etc. in between.
• Don't leave your hotel alone, stay together with other tourists or book some guided tours.
• If you want to do some sightseeing, there are plenty of guided tours where you're in a larger group of other tourists together with a tour guide and sometimes even a security guard.

Back from Egypt

Just came back from a wonderful vacation in Hurghada, Egypt :-)

I got really lucky to get some great last minute deal, just on the day prior to departure, and even more lucky that such a great destination as Egypt was about the only thing that was still available.

Had a really great hotel and beautiful weather - warm enough to swim in the Red Sea. This is just the perfect time to travel to Hurghada - it's their winter season; during their long summer it's just too hot in the sun.

They also offered some great daytrips, so I went to Cairo, Luxor and two snorkeling trips in the Red Sea. In Cairo, we visited the Pyramids of Giza - our tourguide was really cool, when we got there in the morning it was so foggy that you couldn't see anything, but then he offered to come back in the afternoon, which was really cool :-) We also visited the Museum of Cairo, had Lunch onboard a ship on Nil, went to a Bazar and a few more things.

In Luxor, we had a smaller group and visited the Valley of the Kings, the Temple of Hatshepsut and the Temple of Karnak - I wish I could have spent a lot more time there, the Valley of the Kings alone is worth spending an entire day.

The two snorkeling trips were also really cool, we were on the boat the whole day, stopping at different islands all the time to do snorkeling. They also offered free snorkeling instruction in the hotel pool for beginners like me, and our guide was also very patient with guys like me who weren't that good swimmers - I looked kinda silly in that oversized life jacket ;-)

All in all, it was really a great vacation, very relaxing and a lot of fun. I can really recommend Egypt to everyone.

Getting ready for "The Big Move" ....

There's one thing about Windows Vista and Windows 7 which really sucks: unlike Windows XP, it cannot boot from an external USB hard disk.

To make things worse, running Vista or Windows 7 inside VMware also isn't such a great idea if you're planning on doing development. As soon as you're running some huge application like Visual Studio and then try to open something else in the background, it's starting to get really slow. It's not only CPU power, the external USD hard disk is also much slower than an internal one. I first experienced the problem by accident when I had Windows Update configured to automatically download and install updates in the background - and turning that off still didn't satisfy me.

Yesterday, I finally considered the external hard disk a failed experiment and called a local Lenovo shop to get the real problem fixed and replace the internal hard disk by a bigger model; I'll get it replaced on Tuesday.

Since I don't want to spend too much time reinstalling everything again, I'll move my entire Linux installation onto an empty partition on that external disk. Linux doesn't have any problems booting from such devices, so this move should go rather smoothly. Next weekend, I'll migrate everything back - I Google'd a bit, and it seems to be possible to migrate a Windows installation to another hard disk without reinstalling, so hopefully this'll save me a lot of time.

Adding Soft Debugger support to MDB

Last week, I did some research about adding support for Zoltan's new Soft Debugger to MonoVS and / or MDB.

After playing around with it a little bit and looking at MonoDevelop's implementation, I decided to take a different path and kill two birds with one stone - by adding support to the debugger itself. Since this debugger already has a pretty good client / server abstraction, things are working out pretty well so far - I'm basically creating a new soft debugger "backend" and provide a new 'Language'.

This means we will not only be able to use the current cli debugger on the iPhone, but also get the MonoVS integration at no cost.

At the moment, I already got some very basic single-stepping working - we use some quick hack to stop on the first line of Main(), and then you can do some stepping. My next challenge is implementing the type system and then adding support for breakpoints.

Martin 2.1 - Fine-tuning the installation is a beast

There is one thing about installing a new operating system which really sucks: fine-tuning the installation - which usually takes a lot more time than the installation itself. So I spent almost the entire weekend doing post-installation stuff and in the end, I installed everything three times before I was really satisfied with the result.

The first thing to consider was whether to use physical partitions or virtual disk images in VMware - I couldn't find much information about this on Google except that using physical partitions was considered an "expert" option. The only real advantage of using raw partition seems to be increased performance, but you can't take any snapshots of your VM with this option. I still decided to use them because disk space isn't really a problem for me anymore, I can easily use traditional Windows backup to backup the entire system to an unused hard disk partition, but getting maximum performance is very important for me.

Another important thing to decide was the amount of RAM to allocate to the VMs - the host has 4 GB, so I had the idea of running two VMs at the same time. After trying several options between 1-2 GB, I finally realized that my laptop isn't powerful enough to handle this, each time I tried the machine started swapping like hell and it took over an hour till I could do anything useful again (I didn't want to risk damaging the installation by doing a force shutdown, so I had to wait till it was done booting and let me shutdown cleanly). Now I'm allocating 2 GB to the VM, which means that both host and VM can run without any swapping.

Most of the work was fixing all these tiny but annoying little issues that came up while installing this stuff.

Last thing I had to do last night was running a backup of both installations.

Martin 2.0 - The Upgrade

After our great MonoVS 1.0 Release, I finally got tired of doing disk surgery and repartitioning to gain more disk space. When I bought my laptop about a year ago, it was state of the art - 100 GB was the biggest hard disk that was available back then. Since the initial installation, I repartitioned at least 4-5 times to increase the Windows partition.

One thing that's really bad about Windows is if you run out of disk space while doing any kind of upgrade or software installation and you have system restore turned off - that was actually one of the first things I turned off to save disk space - it can leave your system in an inconsistent state where it'll fail to install any further upgrades. I run into this problem while trying to install ASP.NET MVC to fix some bug.

This week, I bought an external 1000 GB - and I refuse to call this a [I]terrabyte[/I] - USB hard drive and then completely reinstalling my system. My primary Vista partition is now 60 GB and after the clean install with Vista x64 SP2 and Visual Studio 2008, there's still plenty of space left, but I also won't install some huge software packages like OpenOffice on it.

In addition to this, I now have two VMware images of 200 GB each - one running Windows 7 with both Visual Studio 2008 and 2010 and the other one running Vista and Visual Studio 2008. I also created a 350 GB backup partition and reserved 200 GB for future extensions - for instance installing a different Linux version etc. - disk space is not something I need to worry about anymore ...

Now I'm basically running MonoVS "the other way around" - with Linux as the host and Windows inside the VM. This was necessary because there still isn't any support for installing Windows 7 onto an external USB hard disk, so you have to run this in some kind of a virtual machine. At the moment, I assigned 2 GB of RAM to the VM (my laptop has 4 GB of RAM) which means I can only run one of them at a time, but at maximum performance without any swapping.

Tomorrow, I'll do some more fine-tuning of the new system and then see whether we can run MonoVS on Windows 7 and/or with Visual Studio 2010. My guess is that the Windows 7 / Visual Studio 2008 setup will just work out-of-the box without any changes at all, but getting it working for Visual Studio 2010 most likely requires some minor tweaks to the registry code.

Welcome to the Universe

Since all the cool guys are on Twitter these days, so am I - http://www.twitter.com/baulig/.

I'll also update my blog more often in near future.

Booting SuSE 11 inside VMware

Two weeks ago, I got a new laptop and set it up to dual-boot both Windows Vista (which I need for work) and SuSE 11. Inside Vista, I also installed VMware and made it boot my existing Linux installation.

Using raw disk partitions in VMware is a bit dangerous and you have to be really careful when setting this up. Here's how I configured it - and it turned out to work really great:

• First of all, make sure you select individual partitions in VMware and not the entire disk. Make sure you don't select the partition containing your host Windows operating system. This ensures that VMware doesn't let the guest operating system access that partition, so you can't accidentally boot or mount it - which'd result in severe data corruption and possible loss of all data on the disk.

• Before booting the first time inside VMware, you need to enable SCPM and create two profiles - I'm using gondor (running natively) and isengard (running inside VMware). In yast, go to System / Profile Manager, active it there and make sure to set both Switch Mode and Boot Mode to Save Changes.

  Then do

  scpm copy default gondor
  scpm switch gondor
  scpm save
  

  on the command line. This'll save your current configuration into the gondor profile. After that, do a

  scpm copy gondor isengard
  scpm switch isengard
  scpm save
  

• Edit /boot/grub/menu.list, replace the SuSE 11 with Gondor, add PROFILE=gondor to the kernel command line and make sure you're using root=/dev/disk/by-uuid:

  title Gondor
      root (hd0,2)
      kernel /boot/vmlinuz root=/dev/disk/by-uuid/b39b72d4-ac4e-4385-b9a3-08ca81b4638f splash=silent PROFILE=gondor showopts vga=791
      initrd /boot/initrd
  
  title Mordor
      rootnoverify (hd0,2)
      chainloader (hd0,0)+1
  

• Copy /boot/grub/stage2 to /boot/grub/state2-isgengard and also copy /boot/grub/menu.list to /boot/grub/menu-isengard.lst. Edit /boot/grub/menu-isengard.lst, remove the Mordor entry and replace Gondor with isengard. This configuration will be used when booting inside VMware:

  title Isengard
      root (hd1,2)
      kernel /boot/vmlinuz root=/dev/disk/by-uuid/b39b72d4-ac4e-4385-b9a3-08ca81b4638f splash=silent showopts PROFILE=isengard
      initrd /boot/initrd
  

• Edit /etc/fstab and use /dev/disk/by-uuid everywhere instead of /dev/disk/by-id or using the raw device. My /etc/fstab looks like this:

  /dev/disk/by-uuid/b39b72d4-ac4e-4385-b9a3-08ca81b4638f /                    ext3       acl,user_xattr        1 1
  /dev/mapper/cr_sda6  /home                ext3       acl,user_xattr,noauto 0 0
  /dev/disk/by-uuid/13aee839-dccc-4999-a977-6168d109dae1 /work                ext3       acl,user_xattr        1 2
  

  This is important since your hard disk will have different physical devices when running inside VMware, so /dev/disk/by-id will be different.

• Now you can reboot your system into Windows.

• In VMware, create a really small (10 MB is fine) virtual disk in addition to your physical hard disk. Double check that you're using individual partitions and the partition containing your Windows host operating system is not selected.

• Boot inside VMware, press F2 to go to BIOS setup, select the virtual hard disk as primary boot device, save and exit. Press Esc to go to the boot menu, select CD-ROM as temporary boot device and boot off the SuSE 11 installation DVD, select Rescue System

• Once in the rescue system, /dev/hda should be the virtual hard disk and /dev/hdb your physical one. Manually install grub into the MBR of the virtual disk and make it use /boot/grub/stage2-isengard and /boot/grub/menu-isengard.lst. You have to do that inside the grub shell, by using appropriate root and install commands - this is a bit dangerous since it could make your system unbootable, so check the manual.

  Note that we only need the MBR in the virtual hard disk - the boot loader in it will load /boot/grub/stage2-isengard from your physical hard disk. Grub encodes the path of the configuration file (/boot/grub/menu-isengard.lst) in the stage2, that's why we're using a custom one.

  This setup is a bit complicated, but it ensures that you don't accidentally boot Linux with the wrong profile.

• Remove the SuSE 11 installation DVD and reboot.

With the two different boot loaders, you'll always boot your Linux with the correct SCPM profile. The SCPM profiles make sure you can have different configurations when running natively and inside VMware. For instance, your /etc/X11/xorg.conf will be different in each scenario and you may also have different network settings.

I'm even using different host names and different IP addresses for both configurations - the one "machine" is called gondor.trier.ximian.com (192.168.3.8) and the other one is called isengard.trier.ximian.com (192.168.3.9). You may also add additional files / services to SCPM to also save things like /etc/issue and /etc/motd, for instance.

SCPM also takes care of the different X11 configuration - when running natively, I'm using a multi-headed setup with an external monitor - and in Vista, I move the VMware window into the external monitor and use full-screen mode there. This means that I have Vista on my laptop's main display and my GNOME session on the external monitor - really awesome !

On Isengard, I'm also using Samba to mount /work/mordor from Windows - I'm using a separate host-only network (VMnet1) for that. To make this work on Vista, create a host-only network (in addition to bridging the ethernet card), assign it a static IP address (I'm using 192.168.3.0/24 for my "real" network and 192.168.8.0/24 for the host-only one) and set the default gateway to that address (address 192.168.8.1 / netmask 255.255.255.0 / gateway 192.168.8.1) - this'll fix the "Unidentified Network" problem in Vista.

Copying symbol files from Windows

Today, I added a cool new feature to the debugger: you can now add "directory maps" to the configuration file.

This is very helpful if you copy a symbol file from Windows to Linux. Simply add something like this to your ~/.config/MonoDebugger/MonoDebugger.xml:

<?xml version="1.0"?>
<DebuggerConfiguration fileversion="1.0">
  <DirectoryMap>
    <Map from="c:/cygwin/home/ichotolot" to="/home/martin/work" />
  </DirectoryMap>
</DebuggerConfiguration>

The debugger now automatically detects whether a symbol file has been created on Windows and converts the file names if necessary. This also works when using pdb2mdb to convert a .pdb to .mdb.

You could, for instance, mount c:\cygwin\home\ichotolot on /work/mordor using samba and use the configuration file to tell the debugger where to look for the source files.

Back from Madrid

After a really exciting week in Madrid, I'm now back at home and working on the debugger again.

We had a few really productive discussions about the debugger and sometimes, I got the impression that people didn't know that we actually have a working debugger before I did my little presentation at the meeting. Well, the debugger from SVN HEAD is considered stable and working - there will be an official release really soon.

Here are some pictures from our meeting:
http://www.flickr.com/photos/baulig/collections/

Ikea

Today, I went to Ikea in Koblenz to buy an armchair. It's really amazing to see how one room can completely change its shape within one hour:

Before driving to Ikea:

And after, with my new armchair:

Debugger currently broken

Unfortunately, I have some really bad news for all debugger users:
The debugger is currently broken and unusable because of recent changes in the Mono runtime. I can't give any ETA yet on how long it'll take to get it fixed since I need to do some research first. At the moment, I don't even know how to fix it, so I need to think about a solution first. Hopefully, I'll have more news after one or two days of research.

There's a Mono revision mentionend in the debugger's ChangeLog which is the last one which is guaranteed to work.

Debugger type system

Yesterday, I finally finished the work I started last week on the new debugger's type system.

For each local variable and method parameter, the debugging info generated by the runtime now also contains a MonoType *. On the debugger's side, we read this MonoType * and create the Cecil type from it. There have been some extensive API changes in the debugger's type systems since we now need to do a target access to resolve a type - that's something which would have been neccessary for generics anyways.

As a side-effect, we can now also unload and free the debugging information about types once an assembly gets unloaded.

Next week, I'll start to work on "real" generics support.

Debugger and Generics

About two weeks ago, I started to add generics support to the debugger.

After doing some research, I soon realized that things are a bit more complicated than I expected, and I need to modify some things in the debugger.

Initializing the debugging code

With yesterday's changes, the unmanaged debugging API has changed. If you're embedding Mono and want to initialize the debugging API, you only need to call mono_debug_init () - the old mono_debug_init_1() etc. hacks are now all gone (caution: do not call mono_debug_init_corlib()).

mono_debug_init() needs to be called before the first appdomain is created - after that, everything will go automatically.

Domain unloading and identifying threads

Today, I committed two important bug fixes in the debugger which both required a lot of code changes.

The most important user-visible change is that the debugging info is now stored on a per-appdomain basis - which means that it can be freed when the domain gets unloaded. We can now also unload symbol files. This required quite some changes - both in the runtime and in the debugger.

This wasn't only important to save memory, but also for correctness and robustness:
When a domain gets unloaded, all methods which were JITed in it are also freed - which means that the corresponding debugging info also isn't valid anymore. We now tell the debugger when an appdomain gets unloaded, so it can also free the corresponding symbol tables.

The second important change was a fix for a severe bug when attaching to a managed application. It is a bit hard to explain what this code exactly does, so this'll get a little bit technically.

On Linux, there is some kind of an API problem with the kernel and libc wrt. threads:

When you run a multi-threaded application, all threads share the same PID - so if you call getpid (), you'll get the same result in each thread. Internally each thread is a seperate `task' for the linux kernel and it also does have it's own pid - which is also called LWP - but there's no (portable) way of getting that from user-level code. That's why user-level code normally uses pthread_t (which is returned by pthread_self()) to identify a thread - we call that a TID.

When attaching to a managed application, the debugger needs to get some information about all the managed threads - precisely the LMF address, which is required to generate managed backtraces when we're stopped inside native code, and some other internal information. The only thing the runtime can provide is a TID to LMF mapping - but the debugger only knows about the LWP and not about the TID.

So I had to find a way of getting the TID from an LWP - the old code simply called pthread_self() in each thread, but of course this approach has several problems. The most severe one is that this obviously doesn't work for core files. It also doesn't scale very much to a large number of threads since invoking methods in the target is extremely expensive. Another important point is that I want to be able to generate a backtrace before executing any code in the target - that's very important if the target is not responding anymore.

After doing a lot of research, I finally decided to use glibc's thread_db library - this can be used to get information about all the threads, so after some work I finally had it working.

We still need to use some really bad hacks for core files, but they're now working as well.

So to summarize, almost two weeks of hard work - but now, attaching and domain unloading is finally working :-)

Tomorrow, I can finally move on to generics ...

Domain unloading and the debugging code

Yesterday, I got an important and interesting bug report:

We currently do not support domain unloading in the new debugging code. This means that when a domain is unloaded, we do not free the debugging info. To make things worse, some internal hashtables also contain references to the MonoMethod *'s which will lead to crashes later on.

Freeing the debugging info is not so easy since the debugger may access it at any time, so we need to tell the debugger first. However, I got a really good idea this morning and am currently working on it.

Native line numbers

After finally fixing the performance leak last week, I started to work on another rather difficult problem.

It all started with a rather trivial bug report:
Miguel sent me a small moonlight application and said that the debugger would crash when trying to get a backtrace from within an event handler.

Sounded like a rather trivial problem, so I started debugging ....

Things turned out to be a lot more complicated - after spending a lot of time doing some research on inline methods and extensive reading of the dwarf spec, I found a huge design problem in the code which reads the line numbers from the dwarf debugging info.

In DWARF, line numbers are stored in a separete section of the symbol file and not in any way related to methods. The debugging information entry for a method (DW_TAG_subprogram) only contains a start and end address - but no information about where the line numbers for that method are stored. The line number table, on the other hand, only contains a mapping from addresses to line numbers and contains no information about methods. To make things worse, the line number table does not need to be continuous and there is no separation whatsoever between different methods.

This makes it rather complicated for the debugger to read line numbers. It looks like the old code only worked by accident since older versions of gcc always created a continuous line number table - g++ doesn't (I could even create a test case where g++ produces a non-continuous line number table where the test case doesn't contain any c++ features at all).

To make things work, I had to do some changes in the debugger: rather than having on LineNumberTable per Method, each Method now contains a reference to a LineNumberTable but it doesn't "own" that table (ie. it could be shared between different methods). I had to move a lot of code around for this; for instance, the LineNumberTable now longer contains the method's start and end row etc.

While working on this, I soon began to realized that a change like this would have been required anyways when supporting compiler generated code. From the debugger's point of view, the difference between an anonymous method (in C#) and an inline method (in native code) is not so big at all.

If everything works out fine, I should be able to finish the new dwarf code till the end of the week and then start to work on compiler-generated code on Monday.

Debugger Performance

Last night, I finally found the performance problem in the new debugger code - and it was really trivial:
We were missing a mono_debugger_unlock(), so we got a deadlock on exit. There's a 2 second timeout in mono_domain_finalize() when it's called from mini_cleanup() and we have 39 nunit tests - makes exactly 78 seconds in total.

Things are now fixed and the new debugger code is as fast as the old one :-)

More performance problems - I feel like I'm seeing ghosts ....

On Saturday, I finally completed the new symbol table code.

Last week, I already noticed a really huge performance leak, so I spent some more time investigating. Unfortunately, I'm getting more and more confused the more I play with this.

At the moment, I feel like I'm seeing ghosts .....

Running the complete debugger test suite in the old debugger takes about 65-70 seconds. In the new debugger code, it takes about 150-160 seconds. So that's more than twice the time and we must have a really huge performance leak somewhere.

So I started investigating. To make sure it's not the new symbol table code in the runtime, I wrote a small script which compiles Mono.C5 100 times and uses mono --debug each time when invoking gmcs.

I was really surprised when I looked at the results:
It took 24 minutes 57.195 seconds with the old code and 24 minutes 34.012 seconds with the new one - which means we're now 23.183 seconds (or 1.55%) faster than before !

This means the real problem must be somewhere inside the debugger ....

More symbol table stuff

My new breakpoint code is coming along really good :-)

Yesterday, I got all the required features working so I started testing. The good news is that this code actually works - multiple appdomains and generic instances are working just fine :-)

The longer I'm working on this, the more problems I find in the old symbol table code. There is, for instance, a piece of code which never worked for the past 2 years - but nobody ever noticed because it can only be triggered if you run a really, really huge application with ---debug. I only figured it out by accident while I decreased the limits to do some performance testing. There are also race conditions when we need to start a new symbol table since the debugger might currently be reading it.

However, this shouldn't be too bad - I already have some really good ideas which'll not only increase stability but also performance.

Method lookups, symbol files and breakpoints

Last week, I started to work on true multi-appdomain support for the debugger. Soon, I realized that there's also another problem: generics.

Both appdomains and generics have one thing in common: a single method may be JITed multiple times. Because of that, I decided to use a common interface in the symbol table code for both. Now, each method in the source code may have multiple addresses.

To make things easier, I decided that in the case of generic methods, we may not insert a breakpoint on one particular instantiation - when inserting a breakpoint on a generic method (or any method in an instantiated generic class), it always affects all instantiations of that method. This doesn't only make the breakpoint code a lot easier - the fact that a method is JITed multiple times for multiple instantiations is also more or less an implementation detail of our current JIT.

When inserting a breakpoint, we first need to check which instances of that method have already been compiled and then physically insert a breakpoint on each of them. Each time, a new instance of the method is JITed, the debugger gets a notification, so it can also insert a breakpoint there.

One important thing when implementing the new code was that it should also scale well to a large number of appdomains and/or generic instantiations.

A first idea from me was to get a notification each time a new appdomain is created and then insert a breakpoint on each of them. However, since callbacks from the debugger to the JIT are expensive and reading a large chunk of data from it is rather cheap, I realized that this doesn't scale very well. Because of that, the JIT now gives the debugger a list of method addresses.

I've already got some preliminary code working, just need to do a few more tests. Before I finish this, I'd like to do some tests with a large number of appdomains and generic instances. Hopefully, I'll be able to finish this tomorrow or on Wednesday.

It can happen to the best of us .... and a long day with a lot of annoying paperwork is finally over

Until yesterday afternoon, I thought things like EC card fraud can't happen to you if you're always careful with using your card - well, I was very wrong about that, it can happen to anyone :-(

Yesterday, after lunch I routinely checked the balance of my bank account using phone banking and was really shocked to hear that it was -4785 EUR - at the beginning of a month, something must be wrong. I immediately asked for customer support to check what was going on. First thing the customer service agent said to me was something like "Oh shit, f... we have a problem :-(" - someone was using my card to withdraw money from ATM machines in Bankog - there were a bunch of such withdrawals of 10.000 whatever-their-currency-is (which converts to about 220 EUR).

So today, I basically spent the whole day doing paperwork. First I had to go to my bank's local branch to get a detailed statement of all these withdrawals - then go to the police to file a report and then fax a copy of it to the bank.

The good news is, the bank already told me that I'll get my money back - since I still have the card, someone must have copied it when I was using an ATM or paying with it at one of these pay-at-pump gas stations.

It was just very frustrating to do all the paperwork and spend hours of waiting. The local police was extremely busy today and the officer I was supposed to talk to was called to several different operations while I was there, so I had to wait a lot.

Sometimes it's a little bit scary to wait that long at a police stations. They send you into a small room to wait till an officer has time for you and if you wait there for some time, a lot of other people come and go - most of them have problems like "Oh, I got my apartment searched yesterday, and you seized XYZ, where can I get it back?" or "Oh, I'm here to surrender my driver's license" or "Oh, I damaged XYZ while I was drunk and was asked to report here".

Live Free or Die Hard

Yesterday, I watched Bruce Willis's new movie Live Free or Die Hard - which was really excellent :-)

This was one of the movies I really wanted to watch in the original version and also without any subtitles. The premier in Luxemburg is on July 4th and they also have these annoying subtitles, so I decided to drive to Düsseldorf and watch it there - a 300km drive from where I live, but it was really worth it.

One year and one day ...

... have passed since I blogged last time.

That's really a very long time and a lot of things have happened in the meantime. Since about November, I'm back hacking on the debugger full-time and there some exciting news about it.

On May 9th, I released version 0.50 "Dublin" of the Mono Debugger. This is the latest stable version and also a milestone. It's one of the best releases ever and I'm really happy about it.

After that, I started some really large code cleanups and rewrites on a separate debugger-dublin branch. I had several goals with this project:

• Support multiple appdomains
• Rework the breakpoint code so we don't need to stop in Main() and we can also support static .cctors
• Don't change the application's flow of execution as much as we're doing at the moment.

I'm now about one week away from implementing full multi-appdomain support, so I think I'll wait until that is done before merging my code back. I've posted a very detailed summary of my latest changes to our internal mailing list, which is also available in SVN.

Read the full story ...