Yesterday, I stumbled upon a very weird thing in glibc when tying to figure out why applications using gnome-vfs were hanging in the debugger. In multi-threaded applications, all functions in the setXid familty (ie. setuid(), seteuid(), setgid(), setegid()) are not implemented using a system call, but instead send a signal to the process and have the signal handler do the system call in each thread.
The implementation of all these functions first checks entry 51 in the __libc_pthread_functions vtable: if it's NULL, a normal system call is made. Otherwise, the call is forwarded to that function - it's called __nptl_sigxid() and what it does is sending a signal to each of the process'es threads - so the signal handler is invoked in each thread. After that, it waits until all the signal handlers have finished and returns.
This is very bad when running in a debugger since one or more threads may be stopped - in which case it'd block forever. Sometimes the debugger also needs to do some "magic" which is normally invisible to the user, but causes some threads being stopped (when stepping over a breakpoint on the current instruction and when dealing with the garbage collector, for instance).
Unfortunately, gnome-vfs calls seteuid() a lot of times even if it's runinng as a normal user and not as root. It's explicitly forbidden to debug any setuid/setgid applications with mdb (the linux kernel doesn't allow this), so the only case where setuid() actually does something is when running as root.
In order to make gnome-vfs work reliably in mdb (debugging applications which use gnome-vfs is what a lot of people want), I added a really bad hack to the debugger: it sets a breakpoint on __nptl_setxid() and whenever it's called, it makes it return 0.
Of cource, this has the side-effect that if you're running as root, the application won't be able to drop privileges anymore and it'll always run as root. You can still run mdb as root to debug most applications, but you may run into problems if you have some deamon-like application which absolutely needs to change its UID for some reason.
However, the positive effect is that any application which is using gnome-vfs is now working reliably in mdb.
I really hope that some day the kernel people will implement the setXid() functionality in the kernel rather than having this weird signal-based approach in user-space.