Massimiliano Mantione's Blog

Mono and oprofile profiler news

Lately I've been working on the Mono profiler.
Most of the work has been on implementing a new profiler, a prototype implementation can be found here (but beware, that patch no longer works in current svn).

To sum it up, this is not a revolution because the profiling API stays the same, it's just a reimplementation of the current default profiler.

This is motivated by a few issues we have with the current profiler:

• It stores all events in memory, and the events still point to the internal data structures (mainly MonoMethod* and MonoClass*) so that it cannot really work when dynamic methods are freed and-or appdomains are unloaded. For now a couple of gross hacks are in place (basically, if "mono_profiler_get_events () != 0" methods are never freed), but we should get rid of them.
• Since it stores everything in memory, its memory usage grows with time (in the statistical case we just allocate a fixed buffer in the default profiler, enough for most simple tests but not for a long running application).
• It processes all the data at the end of the process execution, so these things are not feasible-easy:
  • getting profiling data while the process is still running, and
  • having the data presented better by different front ends.

The main differences of this new profiler:

• It does not "process" the profiling events, it just logs them into a file. They are as packed as possible (a few bytes per event).
• Therefore:
  • it can "survive" when data structures are freed, and
  • one can write front ends that examine the log and show the data without being tied to the Mono runtime and without having to wait for the process to end,
  • but... it will produce large log files when running for a long time in non-statistical mode

These days I'm working on a managed implementation for the decoding library (which processes the files produced by the profiler), I will show it in a BOF at the Mono Summit 2007.

But the real topic of this post is another profiler related work: we can finally integrate oprofile with Mono!
Specifically, managed method names will show up in the opreport output, as shown in this screenshot of the oprofile-viewer GUI:

One nice aspect of oprofile is that it works automatically, without requiring instrumentation of the application code. And it can profile the whole system, including the kernel, so one can have a global view of how the application spends its time. Moreover, it can also profile things like cache misses, branch predictors behavior, and in general everything that's measured by the CPU performance counters.

The only thing is needs to provide a meaningful output is that the elf files of the profiled programs provide symbol names for their functions... which obviously is not true for jitted code.
However, there is interest is using oprofile also for jitted code, so to cope with this, the oprofile developers put in place a solution: generate fake elf files for the anonymous memory regions where jitted code resides ("anonymous" because they are not related to any file on disk), so that the oprofile reporting tools will find the symbols they need in these files, and work unmodified. Of course, these symbol names must come from the JIT itself. For this purpose, oprofile now exports a very simple API, and a library that contains the functions that the JIT must invoke to emit the symbols. Their work was mostly oriented to Java VMs, but the discussion on the oprofile mailing list kept it VM-neutral.

With time, this code in oprofile has matured, and at the beginning of October they created a CVS branch that tracks these efforts, so that now it has been very easy to fetch it and make it work with Mono. There were rough edges, but thanks to Philippe Elie and Maynard Johnson on the oprofile IRC channel we sorted them out quickly, so now it works!
The only downside is that call graphs are not supported for now. I won't enter in the details, but it will take a while to have them working (and BTW, they will likely work more for Mono than for Java, because due to the mixed JIT-interpreted nature of the JVM they don't really have complete stacks that oprofile can examine).

So, the oprofile code is already available in their "JIT_SUPPORT" cvs branch; I will likely commit the Mono code in the following days or during-after the summit... but for the truly impatient, here it is!

As you can see, it's really, really easy... enjoy :-)