Top 60 Oracle Blogs

Recent comments

Oracle memory troubleshooting, Part 3: Automatic top subheap dumping with heapdump

If you haven’t read them – here are the previous articles in Oracle memory troubleshooting series: Part 1, Part 2

In Oracle, the HEAPDUMP dump event in Oracle allows you to dump various heap contents to tracefile. With adding the “level” parameter to this dump event, you can specify which heaps to dump.

Julian Dyke has documented most of the levels here.

There are two little known, but useful level bits for heapdumps – bit 0x10000000 and 0x20000000. These allow Oracle to dump top-5 biggest subheaps in a heap recursively.

When bit 0x10000000 is enabled then Oracle dumps the top-5 subheaps inside any heap its instructed to dump.

When bit 0x20000000 is enabled then Oracle dumps the top-5 subheaps as mentioned above, but in addition Oracle recursively dumps top-5 subheaps inside any subheaps automatically dumped above. So instead of dumping 1 level of subheaps, Oracle recursively dumps 2 levels if such sub-sub-heaps exist.

This reduces the amount of manual work – as Oracle can drill down towards the root cause automatically and dump the relevant information.

Here’s a little test case:

I set the serverout buffer unlimited (Oracle 10.2+) so that Oracle would buffer unlimited amount of dbms_output data in UGA (this is also a “good” way for using up all the memory in your server so you use “unlimited” with care).

SQL> set serverout on size unlimited
SQL> exec for i in 1..1000000 loop dbms_output.put_line(lpad('x',10,'x')); end loop;

Without the recursive top subheap dumping we would see output like this (after processing the tracefile with heapdump_analyzer):

This is the usual way for dumping a target process private heap:

SQL> oradebug setorapid 35
Oracle pid: 35, Unix process pid: 26457, image: oracle@linux03
SQL> oradebug dump heapdump 1     <-- level 1 dumps all top level private heaps (top-level PGA,UGA and call heap)
Statement processed.

And the output is:

[oracle@linux03 trace]$ heapdump_analyzer LIN11G_ora_26486.trc

  -- Heapdump Analyzer v1.00 by Tanel Poder (  )

  Total_size #Chunks  Chunk_size,        From_heap,       Chunk_type,  Alloc_reason
  ---------- ------- ------------ ----------------- ----------------- -----------------
    55065316     841      65476 ,     top uga heap,         freeable,  session heap          <-- session heap is allocated from top uga heap
    41218392    2517      16376 ,     session heap,         freeable,  koh-kghu sessi      <-- koh-kghu session heap is allocated from session heap
    13650208     836      16328 ,     session heap,             free,
       65520       1      65520 ,         pga heap,             free,
       65476       1      65476 ,     top uga heap,         recreate,  session heap
       57736      14       4124 ,     session heap,         freeable,  kxsFrame4kPage

We would see that most of the uga memory (roughly 41MB of 55MB) is allocated for for some reason “koh-kghu sessi”. This is a session heap where from allocations for various objects like PL/SQL variables, records and collections are done. So when we’d want to drill down and see inside that heap we could use the HEAPDUMP_ADDR dump with that heap descriptors address as parameter to look inside it.

However with these extra bits mentioned above, Oracle can automatically dump us the contents of biggest subheaps inside the heaps we asked it to dump: