Martin's Activity Log

Booting SuSE 11 inside VMware

2008-07-18T14:52:00-04:00

Two weeks ago, I got a new laptop and set it up to dual-boot both Windows Vista (which I need for work) and SuSE 11. Inside Vista, I also installed VMware and made it boot my existing Linux installation.

Using raw disk partitions in VMware is a bit dangerous and you have to be really careful when setting this up. Here's how I configured it - and it turned out to work really great:

First of all, make sure you select individual partitions in VMware and not the entire disk. Make sure you don't select the partition containing your host Windows operating system. This ensures that VMware doesn't let the guest operating system access that partition, so you can't accidentally boot or mount it - which'd result in severe data corruption and possible loss of all data on the disk.
Before booting the first time inside VMware, you need to enable SCPM and create two profiles - I'm using gondor (running natively) and isengard (running inside VMware). In yast, go to System / Profile Manager, active it there and make sure to set both Switch Mode and Boot Mode to Save Changes.
Then do
```
scpm copy default gondor
scpm switch gondor
scpm save
```
on the command line. This'll save your current configuration into the gondor profile. After that, do a scpm copy gondor isengard scpm switch isengard scpm save

Edit /boot/grub/menu.list, replace the SuSE 11 with Gondor, add PROFILE=gondor to the kernel command line and make sure you're using root=/dev/disk/by-uuid:

title Gondor
    root (hd0,2)
    kernel /boot/vmlinuz root=/dev/disk/by-uuid/b39b72d4-ac4e-4385-b9a3-08ca81b4638f splash=silent PROFILE=gondor showopts vga=791
    initrd /boot/initrd

title Mordor
    rootnoverify (hd0,2)
    chainloader (hd0,0)+1

Copy /boot/grub/stage2 to /boot/grub/state2-isgengard and also copy /boot/grub/menu.list to /boot/grub/menu-isengard.lst. Edit /boot/grub/menu-isengard.lst, remove the Mordor entry and replace Gondor with isengard. This configuration will be used when booting inside VMware:
```
title Isengard
    root (hd1,2)
    kernel /boot/vmlinuz root=/dev/disk/by-uuid/b39b72d4-ac4e-4385-b9a3-08ca81b4638f splash=silent showopts PROFILE=isengard
    initrd /boot/initrd
```

Edit /etc/fstab and use /dev/disk/by-uuid everywhere instead of /dev/disk/by-id or using the raw device. My /etc/fstab looks like this:

/dev/disk/by-uuid/b39b72d4-ac4e-4385-b9a3-08ca81b4638f /                    ext3       acl,user_xattr        1 1
/dev/mapper/cr_sda6  /home                ext3       acl,user_xattr,noauto 0 0
/dev/disk/by-uuid/13aee839-dccc-4999-a977-6168d109dae1 /work                ext3       acl,user_xattr        1 2

This is important since your hard disk will have different physical devices when running inside VMware, so /dev/disk/by-id will be different.

Now you can reboot your system into Windows.
In VMware, create a really small (10 MB is fine) virtual disk in addition to your physical hard disk. Double check that you're using individual partitions and the partition containing your Windows host operating system is not selected.
Boot inside VMware, press F2 to go to BIOS setup, select the virtual hard disk as primary boot device, save and exit. Press Esc to go to the boot menu, select CD-ROM as temporary boot device and boot off the SuSE 11 installation DVD, select Rescue System
Once in the rescue system, /dev/hda should be the virtual hard disk and /dev/hdb your physical one. Manually install grub into the MBR of the virtual disk and make it use /boot/grub/stage2-isengard and /boot/grub/menu-isengard.lst. You have to do that inside the grub shell, by using appropriate root and install commands - this is a bit dangerous since it could make your system unbootable, so check the manual.
Note that we only need the MBR in the virtual hard disk - the boot loader in it will load /boot/grub/stage2-isengard from your physical hard disk. Grub encodes the path of the configuration file (/boot/grub/menu-isengard.lst) in the stage2, that's why we're using a custom one.
This setup is a bit complicated, but it ensures that you don't accidentally boot Linux with the wrong profile.
Remove the SuSE 11 installation DVD and reboot.

With the two different boot loaders, you'll always boot your Linux with the correct SCPM profile. The SCPM profiles make sure you can have different configurations when running natively and inside VMware. For instance, your /etc/X11/xorg.conf will be different in each scenario and you may also have different network settings.

I'm even using different host names and different IP addresses for both configurations - the one "machine" is called gondor.trier.ximian.com (192.168.3.8) and the other one is called isengard.trier.ximian.com (192.168.3.9). You may also add additional files / services to SCPM to also save things like /etc/issue and /etc/motd, for instance.

SCPM also takes care of the different X11 configuration - when running natively, I'm using a multi-headed setup with an external monitor - and in Vista, I move the VMware window into the external monitor and use full-screen mode there. This means that I have Vista on my laptop's main display and my GNOME session on the external monitor - really awesome !

On Isengard, I'm also using Samba to mount /work/mordor from Windows - I'm using a separate host-only network (VMnet1) for that. To make this work on Vista, create a host-only network (in addition to bridging the ethernet card), assign it a static IP address (I'm using 192.168.3.0/24 for my "real" network and 192.168.8.0/24 for the host-only one) and set the default gateway to that address (address 192.168.8.1 / netmask 255.255.255.0 / gateway 192.168.8.1) - this'll fix the "Unidentified Network" problem in Vista.

Copying symbol files from Windows

2008-06-18T11:13:00-04:00

Today, I added a cool new feature to the debugger: you can now add "directory maps" to the configuration file.

This is very helpful if you copy a symbol file from Windows to Linux. Simply add something like this to your ~/.config/MonoDebugger/MonoDebugger.xml:

<?xml version="1.0"?>
<DebuggerConfiguration fileversion="1.0">
  <DirectoryMap>
    <Map from="c:/cygwin/home/ichotolot" to="/home/martin/work" />
  </DirectoryMap>
</DebuggerConfiguration>

The debugger now automatically detects whether a symbol file has been created on Windows and converts the file names if necessary. This also works when using pdb2mdb to convert a .pdb to .mdb.

You could, for instance, mount c:\cygwin\home\ichotolot on /work/mordor using samba and use the configuration file to tell the debugger where to look for the source files.

Back from Madrid

2007-12-04T19:00:00-04:00

After a really exciting week in Madrid, I'm now back at home and working on the debugger again.

We had a few really productive discussions about the debugger and sometimes, I got the impression that people didn't know that we actually have a working debugger before I did my little presentation at the meeting. Well, the debugger from SVN HEAD is considered stable and working - there will be an official release really soon.

Here are some pictures from our meeting:
http://www.flickr.com/photos/baulig/collections/

Ikea

2007-10-24T18:16:00-04:00

Today, I went to Ikea in Koblenz to buy an armchair. It's really amazing to see how one room can completely change its shape within one hour:

Before driving to Ikea:

And after, with my new armchair:

Debugger currently broken

2007-10-16T13:12:00-04:00

Unfortunately, I have some really bad news for all debugger users:
The debugger is currently broken and unusable because of recent changes in the Mono runtime. I can't give any ETA yet on how long it'll take to get it fixed since I need to do some research first. At the moment, I don't even know how to fix it, so I need to think about a solution first. Hopefully, I'll have more news after one or two days of research.

There's a Mono revision mentionend in the debugger's ChangeLog which is the last one which is guaranteed to work.

Debugger type system

2007-09-28T09:16:00-04:00

Yesterday, I finally finished the work I started last week on the new debugger's type system.

For each local variable and method parameter, the debugging info generated by the runtime now also contains a MonoType *. On the debugger's side, we read this MonoType * and create the Cecil type from it. There have been some extensive API changes in the debugger's type systems since we now need to do a target access to resolve a type - that's something which would have been neccessary for generics anyways.

As a side-effect, we can now also unload and free the debugging information about types once an assembly gets unloaded.

Next week, I'll start to work on "real" generics support.

Debugger and Generics

2007-09-20T10:07:00-04:00

About two weeks ago, I started to add generics support to the debugger.

After doing some research, I soon realized that things are a bit more complicated than I expected, and I need to modify some things in the debugger.

Let's assume we stopped inside some method Foo.Hello() and we want to print some local variable.

Without generics, the debugger simply used Cecil to get the type of that variable.

However, in order to actually do anything useful with the variable, the debugger needs to know a bit more than just its Cecil type. Precisely, it needs to know the exact layout of its fields and - if it wants to access a property - also the MonoMethod * addresses. Because of this, the debugger needs to read the type's MonoClass *.

At the moment, the debugger keeps an internal mapping between Cecil types and MonoClass * pointers. This works because a Cecil type could uniquely be identified by its image and type token - the debugger just read the image and type_token fields from the MonoClass * and then it had something to uniquely identify the corresponding Cecil type.

With generics, things become more complicated.

While Cecil knows about generic types etc., it obviously has no way of knowing the actual generic instantiation of a type - that's something which dynamically happens at run-time.

My first approach to the problem was to read the current generic context (MonoGenericContext *) and construct the Cecil type from it.

This worked, I could instantiate the type in the debugger and display it to the user - for instance IList<int>.

However, the debugger still needs to know the MonoClass * - and here comes the problem.

It's no longer enough just to read image and type_token, we also need to read the generic_class and interpret it. We can't get any unique identification of the class unless we fully read and interpret the generic_class. Of course, this is an expensive operation. We can't just fully read any single MonoClass * that's created just for the sake of inserting it into a hashtable.

Because of this, I had another idea.

Rather than reading the type of the local variable from the symbol file / Cecil and then get its MonoClass * from a hashtable, we directly read it from the target. The debugging info generated by the runtime will be extended by a MonoClass * field.

Key point is this:
It is easy for the debugger to construct a Cecil type from a MonoClass * - but it has no way of getting a MonoClass * from a Cecil type. So the thing the debugger really needs is the MonoClass *, not the Cecil type.

Initializing the debugging code

2007-08-23T15:22:00-04:00

With yesterday's changes, the unmanaged debugging API has changed. If you're embedding Mono and want to initialize the debugging API, you only need to call mono_debug_init () - the old mono_debug_init_1() etc. hacks are now all gone (caution: do not call mono_debug_init_corlib()).

mono_debug_init() needs to be called before the first appdomain is created - after that, everything will go automatically.

Domain unloading and identifying threads

2007-08-22T12:24:00-04:00

Today, I committed two important bug fixes in the debugger which both required a lot of code changes.

The most important user-visible change is that the debugging info is now stored on a per-appdomain basis - which means that it can be freed when the domain gets unloaded. We can now also unload symbol files. This required quite some changes - both in the runtime and in the debugger.

This wasn't only important to save memory, but also for correctness and robustness:
When a domain gets unloaded, all methods which were JITed in it are also freed - which means that the corresponding debugging info also isn't valid anymore. We now tell the debugger when an appdomain gets unloaded, so it can also free the corresponding symbol tables.

The second important change was a fix for a severe bug when attaching to a managed application. It is a bit hard to explain what this code exactly does, so this'll get a little bit technically.

On Linux, there is some kind of an API problem with the kernel and libc wrt. threads:

When you run a multi-threaded application, all threads share the same PID - so if you call getpid (), you'll get the same result in each thread. Internally each thread is a seperate `task' for the linux kernel and it also does have it's own pid - which is also called LWP - but there's no (portable) way of getting that from user-level code. That's why user-level code normally uses pthread_t (which is returned by pthread_self()) to identify a thread - we call that a TID.

When attaching to a managed application, the debugger needs to get some information about all the managed threads - precisely the LMF address, which is required to generate managed backtraces when we're stopped inside native code, and some other internal information. The only thing the runtime can provide is a TID to LMF mapping - but the debugger only knows about the LWP and not about the TID.

So I had to find a way of getting the TID from an LWP - the old code simply called pthread_self() in each thread, but of course this approach has several problems. The most severe one is that this obviously doesn't work for core files. It also doesn't scale very much to a large number of threads since invoking methods in the target is extremely expensive. Another important point is that I want to be able to generate a backtrace before executing any code in the target - that's very important if the target is not responding anymore.

After doing a lot of research, I finally decided to use glibc's thread_db library - this can be used to get information about all the threads, so after some work I finally had it working.

We still need to use some really bad hacks for core files, but they're now working as well.

So to summarize, almost two weeks of hard work - but now, attaching and domain unloading is finally working :-)

Tomorrow, I can finally move on to generics ...

Domain unloading and the debugging code

2007-08-08T16:09:00-04:00

Yesterday, I got an important and interesting bug report:

We currently do not support domain unloading in the new debugging code. This means that when a domain is unloaded, we do not free the debugging info. To make things worse, some internal hashtables also contain references to the MonoMethod *'s which will lead to crashes later on.

Freeing the debugging info is not so easy since the debugger may access it at any time, so we need to tell the debugger first. However, I got a really good idea this morning and am currently working on it.

Native line numbers

2007-07-26T09:16:00-04:00

After finally fixing the performance leak last week, I started to work on another rather difficult problem.

It all started with a rather trivial bug report:
Miguel sent me a small moonlight application and said that the debugger would crash when trying to get a backtrace from within an event handler.

Sounded like a rather trivial problem, so I started debugging ....

Things turned out to be a lot more complicated - after spending a lot of time doing some research on inline methods and extensive reading of the dwarf spec, I found a huge design problem in the code which reads the line numbers from the dwarf debugging info.

In DWARF, line numbers are stored in a separete section of the symbol file and not in any way related to methods. The debugging information entry for a method (DW_TAG_subprogram) only contains a start and end address - but no information about where the line numbers for that method are stored. The line number table, on the other hand, only contains a mapping from addresses to line numbers and contains no information about methods. To make things worse, the line number table does not need to be continuous and there is no separation whatsoever between different methods.

This makes it rather complicated for the debugger to read line numbers. It looks like the old code only worked by accident since older versions of gcc always created a continuous line number table - g++ doesn't (I could even create a test case where g++ produces a non-continuous line number table where the test case doesn't contain any c++ features at all).

To make things work, I had to do some changes in the debugger: rather than having on LineNumberTable per Method, each Method now contains a reference to a LineNumberTable but it doesn't "own" that table (ie. it could be shared between different methods). I had to move a lot of code around for this; for instance, the LineNumberTable now longer contains the method's start and end row etc.

While working on this, I soon began to realized that a change like this would have been required anyways when supporting compiler generated code. From the debugger's point of view, the difference between an anonymous method (in C#) and an inline method (in native code) is not so big at all.

If everything works out fine, I should be able to finish the new dwarf code till the end of the week and then start to work on compiler-generated code on Monday.

Debugger Performance

2007-07-19T04:08:00-04:00

Last night, I finally found the performance problem in the new debugger code - and it was really trivial:
We were missing a mono_debugger_unlock(), so we got a deadlock on exit. There's a 2 second timeout in mono_domain_finalize() when it's called from mini_cleanup() and we have 39 nunit tests - makes exactly 78 seconds in total.

Things are now fixed and the new debugger code is as fast as the old one :-)

More symbol table stuff

2007-07-11T16:24:00-04:00

My new breakpoint code is coming along really good :-)

Yesterday, I got all the required features working so I started testing. The good news is that this code actually works - multiple appdomains and generic instances are working just fine :-)

The longer I'm working on this, the more problems I find in the old symbol table code. There is, for instance, a piece of code which never worked for the past 2 years - but nobody ever noticed because it can only be triggered if you run a really, really huge application with ---debug. I only figured it out by accident while I decreased the limits to do some performance testing. There are also race conditions when we need to start a new symbol table since the debugger might currently be reading it.

However, this shouldn't be too bad - I already have some really good ideas which'll not only increase stability but also performance.

Method lookups, symbol files and breakpoints

2007-07-09T11:54:00-04:00

Last week, I started to work on true multi-appdomain support for the debugger. Soon, I realized that there's also another problem: generics.

Both appdomains and generics have one thing in common: a single method may be JITed multiple times. Because of that, I decided to use a common interface in the symbol table code for both. Now, each method in the source code may have multiple addresses.

To make things easier, I decided that in the case of generic methods, we may not insert a breakpoint on one particular instantiation - when inserting a breakpoint on a generic method (or any method in an instantiated generic class), it always affects all instantiations of that method. This doesn't only make the breakpoint code a lot easier - the fact that a method is JITed multiple times for multiple instantiations is also more or less an implementation detail of our current JIT.

When inserting a breakpoint, we first need to check which instances of that method have already been compiled and then physically insert a breakpoint on each of them. Each time, a new instance of the method is JITed, the debugger gets a notification, so it can also insert a breakpoint there.

One important thing when implementing the new code was that it should also scale well to a large number of appdomains and/or generic instantiations.

A first idea from me was to get a notification each time a new appdomain is created and then insert a breakpoint on each of them. However, since callbacks from the debugger to the JIT are expensive and reading a large chunk of data from it is rather cheap, I realized that this doesn't scale very well. Because of that, the JIT now gives the debugger a list of method addresses.

I've already got some preliminary code working, just need to do a few more tests. Before I finish this, I'd like to do some tests with a large number of appdomains and generic instances. Hopefully, I'll be able to finish this tomorrow or on Wednesday.

It can happen to the best of us .... and a long day with a lot of annoying paperwork is finally over

2007-07-03T12:35:00-04:00

Until yesterday afternoon, I thought things like EC card fraud can't happen to you if you're always careful with using your card - well, I was very wrong about that, it can happen to anyone :-(

Yesterday, after lunch I routinely checked the balance of my bank account using phone banking and was really shocked to hear that it was -4785 EUR - at the beginning of a month, something must be wrong. I immediately asked for customer support to check what was going on. First thing the customer service agent said to me was something like "Oh shit, f... we have a problem :-(" - someone was using my card to withdraw money from ATM machines in Bankog - there were a bunch of such withdrawals of 10.000 whatever-their-currency-is (which converts to about 220 EUR).

So today, I basically spent the whole day doing paperwork. First I had to go to my bank's local branch to get a detailed statement of all these withdrawals - then go to the police to file a report and then fax a copy of it to the bank.

The good news is, the bank already told me that I'll get my money back - since I still have the card, someone must have copied it when I was using an ATM or paying with it at one of these pay-at-pump gas stations.

It was just very frustrating to do all the paperwork and spend hours of waiting. The local police was extremely busy today and the officer I was supposed to talk to was called to several different operations while I was there, so I had to wait a lot.

Sometimes it's a little bit scary to wait that long at a police stations. They send you into a small room to wait till an officer has time for you and if you wait there for some time, a lot of other people come and go - most of them have problems like "Oh, I got my apartment searched yesterday, and you seized XYZ, where can I get it back?" or "Oh, I'm here to surrender my driver's license" or "Oh, I damaged XYZ while I was drunk and was asked to report here".

Live Free or Die Hard

2007-06-28T09:08:00-04:00

Yesterday, I watched Bruce Willis's new movie Live Free or Die Hard - which was really excellent :-)

This was one of the movies I really wanted to watch in the original version and also without any subtitles. The premier in Luxemburg is on July 4th and they also have these annoying subtitles, so I decided to drive to Düsseldorf and watch it there - a 300km drive from where I live, but it was really worth it.

One year and one day ...

2007-06-21T07:24:00-04:00

... have passed since I blogged last time.

That's really a very long time and a lot of things have happened in the meantime. Since about November, I'm back hacking on the debugger full-time and there some exciting news about it.

On May 9th, I released version 0.50 "Dublin" of the Mono Debugger. This is the latest stable version and also a milestone. It's one of the best releases ever and I'm really happy about it.

After that, I started some really large code cleanups and rewrites on a separate debugger-dublin branch. I had several goals with this project:

Support multiple appdomains
Rework the breakpoint code so we don't need to stop in Main() and we can also support static .cctors
Don't change the application's flow of execution as much as we're doing at the moment.

I'm now about one week away from implementing full multi-appdomain support, so I think I'll wait until that is done before merging my code back. I've posted a very detailed summary of my latest changes to our internal mailing list, which is also available in SVN.

Read the full story ...

The following detailed description of my recent work is also available on SVN in doc/debugger-dublin.txt in the /branches/martin/debugger-dublin branch.

Method, MethodSource and TargetFunctionType API changes:
The old SourceMethod is gone. We now have:
- MethodSource which is the source code of a method; it may or may not have a LineNumberTable associated with it' the LNT may or may not be loaded in memory.
- TargetFunctionType is a "high-level" representation of a method; each MethodSource has exactly one TargetFunctionType, but a TargetFunctionType may also describe a method without source code.
  A TargetFunctionType doesn't contain any information about how the method is currently loaded in memory; it's a symbol-file thing.
- Method is a low-level representation of a method and is domain-specific.
  When the application is running, each TargetFunctionType has one Method for each appdomain. In multi-appdomain applications, we create a separate Method in each domain.
  A Method may or may not be loaded (JITed in the appdomain).
This API change was done in preparation for full appdomain support and it was also required by the new breakpoint code.
The new breakpoint code - done and fully tested.
I made substancial changes to the breakpoint code which also affects the way how we deal with breakpoints in method which aren't JITed yet.
Key features:
- There's no technical requirement to stop in Main() anymore.
- Prepared multi-appdomain support:
  Each source code breakpoint location may now have multiple addresses.
- We don't need to compile a method anymore to insert a breakpoint in it.
Note: In the following, source method represents a method in the source code, identified by either its name or a filename and line number - it's basically a method in the symbol file. target method is a method in the target application; ie. a MonoMethod * in the JIT. In multi-appdomain scenarios there is more than one target method for each source method.
The long story:
Key component of the new breakpoint code is the new way how we insert a breakpoint on a method which isn't JITed yet.
Both the old and the new code have one fundamental problem in common: before we can insert a breakpoint, we need to know reliably whether that method has already been JITed or not.
The new code works like this:
- We acquire the metadata loader lock
- We lookup the method's address in the current domain's code hash.
- If it's not yet JITed, we register the JIT callback while still holding the loader lock.
- We release the metadata loader lock and tell the debugger the address or the callback ID.
This has to be done in one single callback.
The important thing is that we need to do both the address lookup and register the callback while holding the loader lock to avoid race conditions.
The old code explicitly triggered a JIT compilation of the method to get its address and then inserted the breakpoint on that address. This is bad as it has side-effects and modifies the application's flow of execution.
One key policy of the new code is not to change the application's flow of execution - the application shouldn't behave any differently when running inside the debugger.
As a side-effect of the new JIT interface, callbacks are now done per target method and not per source method anymore.
When the user requests a breakpoint on a source method, the debugger actually needs to insert multiple breakpoints since there is one target method for each appdomain.
After designing the new JIT breakpoint interface, I also needed to modify several things on the debugger side, especially in the session code.
There is no technical requirement to stop in Main() anymore. Previously, we had to stop in Main() to enable breakpoints - we now do that before initializing Main()'s class to make sure breakpoints are enabled before running any static .cctors.
We now automatically do this from inside the SingleSteppingEngine, the code has been removed from DebuggerSession.
When starting the application, the session code inserts a breakpoint on Main() - but this is just a regular breakpoint and it can be disabled and/or removed by the user.
I liked the idea that the debugger stops in Main() from a usability point of view, but now the user has the freedom to control this.
The biggest user-visible improvement is that we can now have breakpoints before Main() is executed - ie. in static .cctors.
Recursive callbacks
This is something debugger-internal which had to be done to support the next thing.
Basically, I improved the way how the debugger calls methods in the target application - we now support recursive callbacks and the stack unwinding code now also knows about callbacks, so we get correct stack traces.
Trampolines:
After the breakpoint code was fixed, I also needed to fix the way how we handle trampolines wrt stepping over breakpoints. It's a bit difficult to explain what this code is doing and why it is implemented in the way it is, but let me try .... Let's have a look at this little test case:
1 using System; public class Foo { 5 static Foo () { 7 Console.WriteLine ("STATIC CCTOR!"); } 10 public static void Hello () { 12 Console.WriteLine ("Hello World!"); 13 Console.WriteLine ("Second line"); } 15 } class X { static void Main () { 21 Foo.Hello (); } }
Looks trivial, right ? Well, it's not so trivial at all from the debugger's point of view.
Let's assume we're stopped a line 21 and the user issued a step command.
At that time, Foo.Hello() isn't JITed yet so we're actually stepping into a JIT trampoline. There's nothing special about that, we manually compile the method, get its address, insert a breakpoint on it and continue.
In the new code, the first thing we do here is manually initializing the class Foo - which'll execute the static .cctor. The new code has been designed in a way that the debugger basically "expects" to be interrupted while doing that, ie. that the user may have a breakpoint on that .cctor.
But that's not the problem here - let's assume we already initialized the class, we're done with any .cctors and already compiled the method.
The real problem here is that we have a breakpoint on line 21 and if we just continue, mono_magic_trampoline() would abort when attempting to patch the callsite because it get confused by the breakpoint instruction.
So, the code does the following:
- Initialize the class Foo (the debugger expects to be interrupted here and we correctly handle the case where the user has breakpoints on the .cctor).
- compile the method, insert a temporary breakpoint.
- acquire the thread lock
- remove the breakpoint instruction from line 21
- resume the target:
  mono_magic_trampoline() won't trigger any compilation here because we already compiled the method before; all it needs to do is patching the callsite for us.
  [There is still a very rare deadlock possible here:
  Although mono_magic_trampoline() will never actually compile the method, it may still block when trying to lookup its address if any other thread is holding the loader lock. We should find a way to explicitly pass it the address and just do the callsite patching. However, after extensive testing I couldn't trigger any deadlock here, so let's not worry unless we run into problems. The old code was way more problematic: we were also running the .cctor inside the thread lock.]
- re-insert the breakpoint instruction on line 21
- release the thread lock
The important difference to the old code is:
- We support static .cctors here and correctly handle breakpoints.
- We do not run the any managed code while holding the thread-lock
NOTE:
This is where recursive callbacks are used:
The debugger calls mono_runtime_class_init() and if that stops at a breakpoint, the debugger is still inside a callback. If the user does anything which triggers another callback, we have a recursive callback. There's a testcase in TestCCtors.cs for that.
I think we can call this done and working.

With the new breakpoint code in place, it isn't difficult anymore to add real multi-appdomain support.

What's missing is basically a way of notifying the debugger about appdomain loads/unloads and insert/remove breakpoints.

Shouldn't take more than a week to fully implement this.

Released version 0.30 "Koblenz" of the Mono Debugger

2006-07-20T11:57:00-04:00

It has been some time since the last debugger release, but now it's finally ready - version 0.30 "Koblenz" of the Mono Debugger has just been released :-)

The big news is that there are now binary packages available for a number of distros - they can be found at http://www.mono-project.com/Downloads. The source tarball can be found here.

There are a lot of stability improvements in this release and the debugger has also been tested on i386.

Btw. the release name is some kind of an "in-joke" - I'm reserving the name "Cologne" for a stable, 1.0-like, release - and Koblenz is about half the distance to Cologne from where I live.

Update: I have been asked about the binary packages since they are a bit hard to find. On http://www.mono-project.com/Downloads, select your distro - for instance "SuSE Linux Enterprise 9 x86_64" - and then follow the "latest" link. This'll lead you to http://www.go-mono.com/download-latest/sles-9-x86_64 where you find the debugger package in the "Development Tools" section.

Switching on Nullables

2006-06-16T08:34:00-04:00

Yesterday, I implemented a cool new feature in GMCS: switching on nullables - you can now write code like this:

int Test (int? a) { switch (a) { case 1: return 1; case null: return 2; default: return 3; } }

And for all you fans of the World Cup 2006 - I am very happy today:

I got two tickets for the Paraguay - Trididad & Tobago game in the Fritz Walter Stadium in Kaiserslautern (I know, the website is in German, click on Bildergalerie to see pictures). That stadium - also called "Betzenberg" - is the home stadium of my favorite team, the 1. FCK (unfortunately, they're not playing in first league anymore next season) - and I already watched a lot of games there in the past ....

Baile �tha Cliath

2006-06-09T09:08:00-04:00

I'm back from a wonderfull weekend in beautiful Dublin ("Baile �tha Cliath"), met a lot of nice people and had a lot of fun there :-)

Friday and Saturday, I spent most time going from one Pub to the next - mostly around Temple Bar - and tasting a lot of Irish Beer ...
The weather was absolutely awesome and so on Sunday, I did some sightseeing, walked around in the park and then finally found myself in another nice pub drinking even more Guineess and other Irish beers. I had to wake up really early on Monday morning, my plane was supposed to leave at 7:00 am - but somehow I still managed to miss it - it wasn't even that late, just about 6:40 am when I arrived at the airport, but they already closed checkin and I wasn't allowed to board anymore :-(

However, Air Lingus is really a very cool airline - they allowed me to change my ticket by just paying the fare difference plus a small fee. Unfortunately, the second flight on that day was already completely booked out as well as the 7:00 am one on the next morning.

So at about 6:55 am on Monday, when I realized that I was stuck in Ireland for another 34 hours, I decided to see more of this beautiful country and rented a car. It turned out that driving on the wrong side of the road wasn't even that complicated at all and the Irish people are really very nice and helpful - thanks a lot again for helping me out when I got stuck with a flat tire somewhere in the middle of nowhere, with nothing but cows and sheep around !

Galway and Cork were both really nice and Galway has some beautiful beaches. Unfortunately, it was very cold and rainy when I left Germany, so I didn't even think about swimming and didn't pack anything - so I just walked around the beach a little bit. I finally arrived at Cork late at night and was very happy to find a pub which was still open and they were also playing live music there :-)

The next day, I got up very early and started driving back to Dublin - I almost made it to the airport, just another 15 kilometers and plenty of time left - when I got stuck in a really bad traffic jam. For the first two hours, nothing was moving at all - I could have walked to the airport in that time, but I had to return that rental car ... - and then it started moving very slowly. I lost over 3 hours in that jam and when I finally arrived at the airport at 4:55 pm, I didn't even bother going to the checkin counter (my flight was supposed to leave at 5:00 pm) but run upstairs to the airport bar and just made it in time.

When I arrived, I saw my airplane already being pulled away from the gate and slowly moving towards the runway .... so I quickly ordered and when it took off, I waved goodbye with a Guinness in my hand ;-)

As I already said, Air Lingus is a really cool airline - changing my ticket a second time wasn't any problem at all and I got the 7:00 am flight the next morning.

Off to Dublin ....

2006-06-01T11:33:00-04:00

Tomorrow morning, I'm heading off to Dublin, the beautiful capital of Ireland for a long weekend of sightseeing, Guinness-drinking, partying and having fun :-)
I'm really looking forward to it - it has been a long time since I've been in Dublin last time ....

Just the right time to relax before a busy and stressfull week - next Thursday, I'm getting new windows in my apartment. In general, that's a very good idea since my old windows are probably older than I am and I'm really happy that the landlord wants to replace them. However, I'm a bit scared of the dirt and dust this operation will make. At least, I'll be some work moving my desk away from the window and covering all furniture with covering sheets. And then, on Friday, it's the big spring cleaning day for me ...

Ok, time to leave to the Irish Pub to get another pint of Kilkenney before leaving :-)

Released version 0.20 "Bruxelles" of the Mono Debugger

2006-05-24T10:10:00-04:00

After almost two months, I finally released a new stable version of the Mono Debugger - it's called 0.20 "Bruxelles" and can be found at http://primates.ximian.com/~martin/debugger/mono-debugger-0.20.tar.gz.

There are lots of improvements in this release:

This version of the debugger requires the latest Mono from SVN; it won't work with the Mono 1.1.15 release.
We now preserve breakpoints across different invocations of the same target.
Big API cleanups.
Cleaned up method lookups; there's a very complex test for them in test/testsuite/TestMethodLookup.cs.
Lots of stability improvements.
Fixed a very old GC bug which was preventing us from running xsp inside the debugger.
Fixed a very old race condition which was causing crashes at exit.
Fixed line numbers in stack traces.
Added experimental support for hardware watchpoints - I needed them to track down a race condition.
Unfortunately, a `make check' now requires even more memory than before; I'm still investigating the issue.
I'm currently using `ulimit -Sm 1000000 -Sv 1000000'.

Since this release contains so many stability improvements, I decided to move to a new versioning scheme - this release really deserves being called 0.20 rather than 0.14 ;-)

The next releases will be "Dublin", "Sidney" and "Cologne".

Released version 0.12 "Portland" of the Mono Debugger

2006-03-14T06:15:00-04:00

Version 0.12 "Portland" of the Mono Debugger has just been released - it can be found here :-)

This version of the Mono Debugger requires Mono 1.1.13.4.
For those of you using SVN, you need the latest Mono from either SVN HEAD or anything from the 1-1-13 branch which is at least Mono 1.1.13.4.
The debugger no longer uses a mono-debugger-mini-wrapper, it's now using the normal mono instead.
We can now attach to processes and examine core files - this still needs some testing, so bug reports are very welcome ...
We no longer run the backend in another process/appdomain; most of the remoting stuff is gone to make the debugger faster and more reliable.
This release is named after Portland, Oregon.

I'm still very busy fixing GMCS bugs this whole week, but next week I'll be back in debugger land - so if you have any problems or bugs, please lemme know.

C5 1.0 hits Mono SVN

2006-03-09T06:50:00-04:00

As Miguel already mentionend in his blog on February 1st:

Peter Sestoft released version 1.0 of his C5 generics library for C#, it is available here:
C5 provides functionality and data structures not provided by the standard .Net System.Collections.Generic namespace, such as tree data structures, heap based priority queues and hash indexed array lists and linked lists, making it more comprehensive than collection class libraries on similar platforms, such as Java. Unlike many other collection class libraries, C5 is designed with a strict policy of supporting "code to interface not implementation".

C5 is documented on his technical report and he also has the docs online.

Importing this version into Mono SVN was a real challenge since it exposed many bugs in GMCS and the runtime. Now, over a month later, I finally had the time to fix all these bugs and I'm really happy that we now have a working C5 in Mono SVN again :-)

Today, version 1.0 of C5 has finally hit Mono SVN in HEAD and the 1.1.13 branch.

The end of the `mono-debugger-mini-wrapper'

2006-01-21T11:06:00-04:00

As of today, the Mono Debugger is now using the normal `mono' to execute managed code rather than using its own `mono-debugger-mini-wrapper'.

This means that you need a recent SVN version of Mono from either HEAD or the 1-1-13 branch: everything after r55889 is ok.

There are two big advantages of the new setup:

Packaging: Since we no longer need to worry about finding the "correct" wrapper, packaging and relocating the debugger is now much easier.
Testing a different Mono: You can now use a different Mono to debug your applications.
This can be very helpful if you're trying to find a runtime or JIT bug: you can use a stable version of Mono to run the debugger and debug an application using a newly compiled, unstable one.

The required mini changes are both in HEAD and in the 1-1-13 branch, so these two versions are explicitly compatible with each other - you can, for instance, use the 1.1.13 Mono to run the debugger and use Mono from HEAD in the target application.

Back in cold Germany ....

2006-01-09T04:12:00-04:00

After two wonderful weeks of vacations in beautiful Guam, I'm finally back in cold Germany. Before I start looking at all these GMCS bugs in bugzilla, here are a few last pictures:

This is Orote Point, the westernpoint point of the United States (including territories)

See that little lagoon down there with this beautiful sand beach ? There's a deep and winding footpath climbing down that rock and then, after a short walk through the jungle ....
	.... you get here - to one of the most beautiful beaches on the island
We spent the whole afternoon there - swimming, snorkeling, having fun ...

Paradise is Chamorro

2005-12-29T07:20:00-04:00

After a very warm and funny Christmas, here are finally a few pictures ....

5 Things to Know about Guam

Where the hell is Guam ?
Guam is located at 13� 26′ 31″ N 144� 46′ 35″ E
How does Guam look like ?
Well, depends on how do you define paradise ....
Where do I get a map ?
It took me some time to figure this out when I came here. The best way of getting a map is either asking in one of the big hotels or at Two Lover's Point
Which language do they speak in Guam ?
Guam is an US territory, so the official language is english.
How are the native people of the island called ?
They're called Chamorros

5 Places you Must Go while you're in Guam

Tarague Beach
If you like a nice and quiet place to chill out and enjoy paradise ...
The Gapgab
Almost the westernmost point of the USA and a nice place to go snorkeling
Ypao Beach
Located right next to the big hotels. Unfortunately very crowded by tourists.
Ritidian Point
Another very nice beach which is located in the wildlife preserve.
Two Lovers Point
Very nice overview over the island and also a perfect place for your next wedding ....

5 Pubs and Bars

The Tower of London
A very cool english pub with all sorts of foreign beers.
Jimmy's Beach Bar
Having fun on the beach ...
Hard Rock Cafe
Of course, there's a Hard Rock Cafe in Guam.
Brewery
This isn't just a brewery, but a cool nightclub where you can dance and have a lot of fun.
Denial
If you want it more extreme ....

Some more pictures

A warm and sunny place ....

2005-12-15T11:21:00-04:00

Today was my last day of work before my christmas vacations. Now it's time for me to start packing my bags and get ready to leave - on Monday, I'm flying to Guam for 16 wonderful days :-)

At the moment, it's 81 F or 27 C there and the beach is waiting for me ....

Before I'm leaving, I also have a little Christmas present for you guys: after 18 months, there's finally another debugger release - it's called 0.11 "Guam" :-)

You can get the debugger at http://primates.ximian.com/~martin/debugger/mono-debugger-0.11.tar.gz. This release requires Mono 1.1.11, and - unlike most previous debugger releases - this one works with the released Mono and doesn't require you to use SVN. The debugger tarball also contains a pre-built version of Cecil, so you don't need to get that separately.

The documentation of the debugger can be found on the Mono website at http://www.mono-project.com/Guide:Debugger

Ok, now it's time for me wish you all Happy Christmas and a Happy New Year - I'll be back on January 9th - if I can get any internet access in Guam, I'll try to post some pictures ....

setXid() madness

2005-11-11T12:16:00-04:00

Yesterday, I stumbled upon a very weird thing in glibc when tying to figure out why applications using gnome-vfs were hanging in the debugger. In multi-threaded applications, all functions in the setXid familty (ie. setuid(), seteuid(), setgid(), setegid()) are not implemented using a system call, but instead send a signal to the process and have the signal handler do the system call in each thread.

The implementation of all these functions first checks entry 51 in the __libc_pthread_functions vtable: if it's NULL, a normal system call is made. Otherwise, the call is forwarded to that function - it's called __nptl_sigxid() and what it does is sending a signal to each of the process'es threads - so the signal handler is invoked in each thread. After that, it waits until all the signal handlers have finished and returns.

This is very bad when running in a debugger since one or more threads may be stopped - in which case it'd block forever. Sometimes the debugger also needs to do some "magic" which is normally invisible to the user, but causes some threads being stopped (when stepping over a breakpoint on the current instruction and when dealing with the garbage collector, for instance).

Unfortunately, gnome-vfs calls seteuid() a lot of times even if it's runinng as a normal user and not as root. It's explicitly forbidden to debug any setuid/setgid applications with mdb (the linux kernel doesn't allow this), so the only case where setuid() actually does something is when running as root.

In order to make gnome-vfs work reliably in mdb (debugging applications which use gnome-vfs is what a lot of people want), I added a really bad hack to the debugger: it sets a breakpoint on __nptl_setxid() and whenever it's called, it makes it return 0.

Of cource, this has the side-effect that if you're running as root, the application won't be able to drop privileges anymore and it'll always run as root. You can still run mdb as root to debug most applications, but you may run into problems if you have some deamon-like application which absolutely needs to change its UID for some reason.

However, the positive effect is that any application which is using gnome-vfs is now working reliably in mdb.

I really hope that some day the kernel people will implement the setXid() functionality in the kernel rather than having this weird signal-based approach in user-space.