Latchprof allows you to drill down into your latching problems at session level (which V$LATCH, V$LATCH_PARENT and V$LATCH_CHILDREN can't do). It allows you to get valuable details about individual sessions who are holding a latch the most, therefore likely contributing to the latch contention problem the most.

However after you have discovered the troublemaking session, then what next? One way forward is looking into V$SESSTAT counters using Snapper tool. Depending on what latch is the problematic one, you would look for different stats like various buffer get stats for cache buffers chains latches and parsing/executing stats when looking into library cache latches. However if those stats look "normal", is there any other way do drill down further?

Yeah, there is and lets look into it!

A side comment: Nowadays I try to avoid using X$ tables in favor of V$ views (because in my experience in many client environments you don't always have easy SYS access or X$ table "proxy views" created). Also, if you can get your job done using simple V$ views, then why bother with more complex X$ tables? (yeah, it's much cooler, I know ;)

Anyway, here is an exception: LatchProfX script, which means Latch holder Profiler eXtended, because it provides some very interesting extended information about latch holders. It shows why the latch currently held was taken after all. It uses X$KSUPRLAT instead of V$LATCHHOLDER as the latter doesn't externalize all required information. I'm talking about latch Where statistics.

By "Where statistics" I mean that Oracle kernel's latching module coders have instrumented their code well enough - whenever they get a latch, they mark down the code location ID of the latch getter within the latch structure itself.

Let's not keep you waiting anymore and show some examples. The output is quite wide so you may need to size your browser window accordingly.

Normally you would run LatchProfX with specific enough parameters to monitor only one or few latches which you know are the troublemakers. However the first example run I do across all instance latches (the third parameter is %) to illustrate the usefulness of LatchProfX result data better.

LatchProfX syntax is similar to the original LatchProf script, with the addition of ksllwnam and ksllwlbl field names you can sample. These are exactly the fields (from X$KSLLW) which give us the descriptions of "where in kernel the latch was aquired from" opcodes we get from X$KSUPRLAT (see the script itself for syntax and internals).

So (finally), lets run it:

SQL> @latchprofx name,ksllwnam,ksllwlbl % % 100000

-- LatchProfX 1.06 by Tanel Poder ( http://www.tanelpoder.com )

NAME                                KSLLWNAM                                 KSLLWLBL                   Held       Gets  Held %     Held ms Avg hold ms
----------------------------------- ---------------------------------------- -------------------- ---------- ---------- ------- ----------- -----------
shared pool                         kghalo                                                              9705       5987    9.71     185.366        .031
ges resource hash list              kjuscl: lock close request                                          2429       2350    2.43      46.394        .020
kks stats                           kksAllocChildStat                                                   2001       1460    2.00      38.219        .026
ges resource hash list              kjrmas1: lookup master node                                         1780       1724    1.78      33.998        .020
shared pool                         kghupr1                                  Chunk Header               1618        891    1.62      30.904        .035
row cache objects                   kqrpre: find obj                                                     988        986     .99      18.871        .019
shared pool simulator               kglsim_unpin_simhp                                                   804         97     .80      15.356        .158
ges process parent latch            kjlalc: lock allocation from lmon                                    779        766     .78      14.879        .019
active service list                 kswsgetso: get service object                                        663        658     .66      12.663        .019
shared pool                         kghalp                                                               635        628     .64      12.129        .019
ges process parent latch            kjatioc                                                              518        518     .52       9.894        .019
shared pool simulator               kglsim_upd_newhp                                                     402        401     .40       7.678        .019
ges resource hash list              kjrref: find matched resource                                        374        371     .37       7.143        .019
row cache objects                   kqreqd                                                               360        360     .36       6.876        .019
ges process parent latch            kjlrem: detach lock from group                                       235        229     .24       4.489        .020
enqueues                            ksqdel                                                               162        162     .16       3.094        .019
enqueues                            ksqgel: create enqueue                   parent obj                  148        148     .15       2.827        .019
row cache objects                   kqreqd: reget                                                        108        105     .11       2.063        .020
enqueue hash chains                 ksqrcl                                   resource addr               101        101     .10       1.929        .019
shared pool                         kgh_heap_sizes                                                        99         99     .10       1.891        .019
enqueue hash chains                 ksqgtl3                                  acquiring session            93         93     .09       1.776        .019
ges enqueue table freelist          kjlalc: lock allocation                                               85         83     .09       1.624        .020
shared pool                         kghfre                                   Chunk Header                 74         65     .07       1.413        .022
shared pool simulator               kglsim_scan_lru: scan                                                 69         69     .07       1.318        .019
ges process parent latch            kjucll: closing lock                                                  68         61     .07       1.299        .021
shared pool simulator               kglsim_chg_simhp_free                                                 48         47     .05        .917        .020
ges domain table                    kjdmlad: add a lock to an xid-hashed loc                              42         41     .04        .802        .020
shared pool simulator               kglsim_chg_simhp_noob                                                 41         41     .04        .783        .019
shared pool simulator               kglsim_unpin_simhp: fast path                                         39         39     .04        .745        .019
ksuosstats global area              ksugetosstat                                                          29          1     .03        .554        .554
ges group table                     kjgdgll: move a lock from group lock lis                              13          6     .01        .248        .041
kokc descriptor allocation latch    kokcdlt: regular free                    latch                        11          4     .01        .210        .053
ges enqueue table freelist          kjlfr: remove lock from parent object                                 11         11     .01        .210        .019
kokc descriptor allocation latch    kokcdlt: allocation                      latch                         5          4     .01        .096        .024
cache buffers chains                kcbgtcr: fast path                                                     3          3     .00        .057        .019
ges caches resource lists           kjrchc: cached obj cleanup                                             2          2     .00        .038        .019
process allocation                  ksufap: active procs                                                   2          2     .00        .038        .019
ges group table                     kjgagll: move a lock from process lock l                               2          2     .00        .038        .019
ges domain table                    kjdmlrm: remove a lock from its xid-hash                               1          1     .00        .019        .019
ges caches resource lists           kjchc: clean resource cache                                            1          1     .00        .019        .019
ges resource hash list              kjruch: move resource from cache to free                               1          1     .00        .019        .019
name-service memory objects         kjxgsnep: get a value-block object                                     1          1     .00        .019        .019
active service list                 kswsite: service iterator                                              1          1     .00        .019        .019

43 rows selected.

Elapsed: 00:00:02.06

What do we see? While LatchProf would just have provided latch holder info about which (child) latch was held and by which SID, then LatchProfX will show you additional why this latch was gotten.

From above example you see that there were various reasons why the shared pool latch was gotten, but the reason which caused the SP latch to be held the most time, was a code location kghalo, which means something like Kernel Generic Heap Allocate. Which essentially is the function used fol searching and allocating free space from shared pool. This kind of makes sense as stuff like freeing a chunk and updating heap global stats are simple operations as Oracle already knows the memory location it needs to update, but searching for free chunks for allocation can require scanning through a lot of chunks if there are no suitable ones in freelists and a LRU scan is needed.

Next, there's a kghupr1 location which has caused the shared pool latches held for 1.62% of total sampling time. This location, in function kghupr (un-pin recreatable) needs to take shared pool latch for putting the unpinned chunk back to heap LRU list. An Oracle kernel coder has also specified a label (action description) as "Chunk Header" which is seen from KSLLWLBL column. Its in place to state that this kernel location gets shared pool latch as it needs to update a chunk header - putting a chunk back to LRU list means updating the chunk header (and its neighbor(s) headers).

There are also kgh_heap_sizes and kghfre locations, which update global heap statistics (the stuff you see from v$sgastat for example) and free a chunk in heap respectively.

Wherever you have estabilished which latch is the troublemaker (under too much contention) and you know the "why" reason for it, you probably also want to know the sessions contributing to the problem. And LatchProfX can show that:

SQL> @latchprofx sid,name,ksllwnam,ksllwlbl % "shared pool" 100000

-- LatchProfX 1.06 by Tanel Poder ( http://www.tanelpoder.com )

       SID NAME                                KSLLWNAM                                 KSLLWLBL                   Held       Gets  Held %     Held ms Avg hold ms
---------- ----------------------------------- ---------------------------------------- -------------------- ---------- ---------- ------- ----------- -----------
       113 shared pool                         kghalo                                                              5038       2787    5.04      92.699        .033
       112 shared pool                         kghalo                                                              4723       2820    4.72      86.903        .031
       113 shared pool                         kghupr1                                  Chunk Header                875        369     .88      16.100        .044
       113 shared pool simulator               kglsim_unpin_simhp                                                   728         48     .73      13.395        .279
       112 shared pool                         kghupr1                                  Chunk Header                623        438     .62      11.463        .026
       112 shared pool simulator               kglsim_unpin_simhp                                                   273         37     .27       5.023        .136
       112 shared pool                         kghalp                                                               232        232     .23       4.269        .018
       113 shared pool simulator               kglsim_upd_newhp                                                     230        230     .23       4.232        .018
       113 shared pool                         kghalp                                                               224        223     .22       4.122        .018
       112 shared pool simulator               kglsim_upd_newhp                                                     194        194     .19       3.570        .018
       112 shared pool                         kgh_heap_sizes                                                        49         49     .05        .902        .018
       113 shared pool                         kgh_heap_sizes                                                        49         49     .05        .902        .018
       112 shared pool                         kghfre                                   Chunk Header                 37         37     .04        .681        .018
       113 shared pool simulator               kglsim_scan_lru: scan                                                 32         32     .03        .589        .018
       112 shared pool simulator               kglsim_scan_lru: scan                                                 28         27     .03        .515        .019
       112 shared pool simulator               kglsim_chg_simhp_free                                                 27         27     .03        .497        .018
       113 shared pool simulator               kglsim_chg_simhp_free                                                 26         25     .03        .478        .019
       113 shared pool                         kghfre                                   Chunk Header                 24         23     .02        .442        .019
       112 shared pool simulator               kglsim_unpin_simhp: fast path                                         23         23     .02        .423        .018
       113 shared pool simulator               kglsim_chg_simhp_noob                                                 22         22     .02        .405        .018
       112 shared pool simulator               kglsim_chg_simhp_noob                                                 18         18     .02        .331        .018
       113 shared pool simulator               kglsim_unpin_simhp: fast path                                         18         18     .02        .331        .018
       112 shared pool sim alloc               kglsim_chk_objlist: alloc                                              1          1     .00        .018        .018
       122 shared pool                         kghfre                                   Chunk Header                  1          1     .00        .018        .018

24 rows selected.

Elapsed: 00:00:01.93
SQL>

From above you see are 2 main sessions contributing to shared pool latch contention and both are doing lots of kghalo calls, thus allocating memory. When V$SESSTAT for those sessions shows high "parse count (hard)" statistic it would be obvious that the heavy shared pool activity is due hard parsing (which is kind of common knowledge as excessive hard parsing very often the reason for shared pool latch contention). However check out another sample with same parameters:

SQL> @latchprofx sid,name,ksllwnam,ksllwlbl % "shared pool" 100000

-- LatchProfX 1.06 by Tanel Poder ( http://www.tanelpoder.com )

       SID NAME                                KSLLWNAM                                 KSLLWLBL                   Held       Gets  Held %     Held ms Avg hold ms
---------- ----------------------------------- ---------------------------------------- -------------------- ---------- ---------- ------- ----------- -----------
         0 shared pool                         kghalo                                                             10714          5   10.71     212.137      42.427
       122 shared pool                         kghalo                                                              7685         42    7.69     152.163       3.623
       112 shared pool                         kghalo                                                              3874       2305    3.87      76.705        .033
       113 shared pool                         kghalo                                                              3616       2318    3.62      71.597        .031
       112 shared pool                         kghupr1                                  Chunk Header                659        327     .66      13.048        .040
       113 shared pool                         kghupr1                                  Chunk Header                597        353     .60      11.821        .033
       112 shared pool simulator               kglsim_unpin_simhp                                                   416         29     .42       8.237        .284
[...snip...]
       122 shared pool simulator               kglsim_upd_newhp                                                       2          2     .00        .040        .020
       117 shared pool                         kghupr1                                  Chunk Header                  2          2     .00        .040        .020
       122 shared pool                         kghalp                                                                 2          2     .00        .040        .020
       117 shared pool                         kghalp                                                                 1          1     .00        .020        .020

38 rows selected.

Elapsed: 00:00:02.14

SQL>

What's this? Why is there a session with SID=0 which has called kghalo function and held the latch for much longer on average than any other sessions? LatchProfX has detected only 5 latch gets by that session from kghalo location which still have taken the most time in total. And average hold time for that latch is estimated over 42 milliseconds compared to most others which are in microseconds (even though the 3.6 milliseconds average hold time for a latch is quite long too).

One of the reason for such long hold times could be CPU preemption - a latch holder process was switched off the CPU and it took a while before it got back on it to complete its work and release the latch again. However on my 4 CPU test machine the was plenty of CPU time available so such preemption should have not been the problem. The SID 0 gives a clue. When a new session is started in Oracle, apparently it's reported as SID 0 during the session initialization sequence in some places, which includes the memory allocation for things like the v$sesstat and v$session_event arrays and also the session parameter values and set events in some cases. Considering that total size of these structures is quite large in normal shared pool usage context (tens of kB), then if freelists are empty and LRU scan must happen, this inevitably takes longer time - and shared pool latch is held all that time. (phew, I must write shorter statements).

Anyway, I recommend playing around with this script in a test environment, run different workloads like hard parses, soft parses, physical IOs and buffer gets and you should get quite a good overview of some Oracle internal workings that way. This hopefully also answers how do I know some of the Oracle function names and what's their purpose - the answer is that Oracle itself exposes quite a lot of this information in the X$ tables I'm using in my script. You just need to experiment, observe and try to figure out what are the real names behind function names like kghalo etc. Fun :)

Few more notes:

If you want to drill down to latch child level, then use LADDR as another field (among SID,NAME etc) in first parameter to LatchProfX (I've described this in AOT guide part 7).

Do not use LatchProf, LatchProfX and WaitProf when your server's CPUs are 100% utilized, for two reasons:

1. All those scripts never sleep during their sampling but spin on CPU - thus they would make the CPU starvation even worse
2. More importantly, if all your CPUs are running at 100% utilization, the latch contention you see is probably just a symptom of the CPU starvation itself! The same goes to wait events as well - when your CPU runqueues are long, some of the CPU runqueue waiting time can be attributed to wait events such IO instead (for example when IO completes, but the process can't get onto CPU to "end" the IO wait

I have updated both LatchProf and LatchProfX do detect whether they're running on Oracle 9i or 10g+ and dynamically run different code based on version (on 9i, there is no GETS column in V$LATCHHOLDER).

Pra entender como isso funciona, vou criar uma tabela com 10000 linhas e posteriormente deletar a maioria delas.

SQL> CREATE TABLE alex
2 AS SELECT rownum id, 'Rush' text
3 FROM dual
4 CONNECT BY level <=10000;

Tabela criada.

SQL> create index alex_id on alex(id);

Índice criado.

Temos então um indice com 10000 entradas. Vamos ver quanto leaf blocks nós temos

SQL> analyze index alex_id validate structure;

Índice analisado.
SQL> select lf_rows, lf_blks, del_lf_rows from index_stats;

LF_ROWS LF_BLKS DEL_LF_ROWS
---------- ---------- -----------
10000 21 0

Portanto neste momento temos 10000 LF_ROWS, 21 LF_BLKS e nenhuma linha indexada deletada.

Vamos deletar a maioria das linhas e consequentemente as entradas de indice.

SQL> delete alex where id <=9990;

9990 linhas deletadas.

SQL> commit;

Commit concluído.

Certo, agora nos temos um indice com a maioria de suas entradas deletadas e todos os blocos folha dos indice menos um vazios.

Vamos ver, inicialmente como o ANALYZE lida com estes blocos de indices vazios e com a deleções.

SQL> analyze index alex_id validate structure;

Índice analisado.

SQL> select lf_rows, lf_blks, del_lf_rows from index_stats;

LF_ROWS LF_BLKS DEL_LF_ROWS
---------- ---------- -----------
10000 21 9990

Como voce pode notar a estatistica LF_ROWS ainda mostra o valor 10000, ou seja, ainda conta as entradas dos indice apesar da deleção. LF_BLKS também continua 21, portanto os blocos vazios ainda são levados em consideração.

Vamos ver o que acontece quando utilizamos os DBMS_STATS

SQL> exec dbms_stats.gather_table_stats(ownname=>null, tabname=> 'ALEX', cascade => true, estimate_p
ercent=> null, method_opt=> 'FOR ALL COLUMNS SIZE 1');

Procedimento PL/SQL concluído com sucesso.

SQL> select index_name, num_rows, leaf_blocks from dba_indexes where index_name = 'ALEX_ID';

INDEX_NAME      NUM_ROWS LEAF_BLOCKS
------------- ---------- -----------
ALEX_ID                              10                    1

Notamos imediatamente duas diferenças.

Primeira : NUM_ROWS igual a 10. Mostrando que apenas as entradas não deletadas são levadas em consideração.

Segunda : LEAF_BLOCKS igual a 1. Mostrando que apenas os blocos que contem entradas não deletadas são levados em consideração. Embora a estrutura do indice possua outrso 20 "Leaf Blocks", estes não são considerados pelo CBO do Oracle quando utilizamos DBMS_STATS.

Vamos apenas executar uma consulta simples para ver seu plano de execução.

SQL> select * from alex where id between 1 and 10000;

ID TEXT
---------- ----
9991 Rush
9992 Rush
9993 Rush
9994 Rush
9995 Rush
9996 Rush
9997 Rush
9998 Rush
9999 Rush
10000 Rush

10 linhas selecionadas.

Plano de Execução
----------------------------------------------------------
Plan hash value: 1467085234

--------------------
| Id | Operation | Name |
--------------------
| 0 | SELECT STATEMENT |
| 1 | TABLE ACCESS BY INDEX ROWID| ALEX |
|* 2 | INDEX RANGE SCAN | ALEX_ID |
----------------------------------

O indice é utilizado por que o custo de é baixo.

Já se usarmos o ANALYZE

Vejamos se tem diferenças

SQL> analyze index alex_id COMPUTE STATISTICS;

Índice analisado.

SQL> select index_name, num_rows, leaf_blocks from dba_indexes where index_name = 'ALEX_ID'
2 /

INDEX_NAME NUM_ROWS LEAF_BLOCKS
--------------- ---------- -----------
ALEX_ID 10 21

Como voce pode notar o Analyze leva em consideração os blocos vazios. LEAF_BLOCKS 21 em vez de apenas 1 como anteriormente. Mas, blocos folhas (leaf blocks) são apenas uma das estatisticas que o CBO utiliza quando calcula o custo de se utilizar um determinado indice como acesso aos dados. E qual o impacto disso na nossa consulta.

SQL> select * from alex where id between 1 and 10000;

ID TEXT
---------- ----
9991 Rush
9992 Rush
9993 Rush
9994 Rush
9995 Rush
9996 Rush
9997 Rush
9998 Rush
9999 Rush
10000 Rush

10 linhas selecionadas.

Plano de Execução
-----------------------------------------------------
Plan hash value: 2452671568

---------------------------
| Id | Operation | Name|
---------------------------
| 0 | SELECT STATEMENT | |
|* 1 | TABLE ACCESS FULL| ALEX |
----------------------------

Agora o CBO decidiu fazer um full table scan, devido ao custo adicional calculado ao se utilizar o indice.

Portanto blocos de indices vazios tem um grande impacto, não apenas em como a consulta pode ser realizada, mas como o CBO calcula o custo associado, dependendo de como a estatistica tenha sido gerada.

Esta é apenas uma das razões pelas quais voce deve utilizar o DBMS_STATS ao invés do ANALYZE

As you probably know you can query DUAL table when database is not open, but in this case the actual query is made against X$DUAL as seen below:

SQL> select * from dual;

ADDR           INDX    INST_ID DUM
-------- ---------- ---------- ---
051ED14C          0          1 X

SQL>

When you have above fields when querying from DUAL then you know your database is probably not open.

So lets select something from dual:

SQL> select 'blah' x from dual;

X
------------
blah

It works.

Now lets run an equivalent query using subquery factoring:

SQL> with subquery as (select 'blah' x from dual) select * from subquery;
with subquery as (select 'blah' x from dual) select * from subquery
                                                           *
ERROR at line 1:
ORA-01219: database not open: queries allowed on fixed tables/views only

Hmm, even though I'm really accessing the same X$DUAL table which worked ok just before, I can't run that query.

I used to think that this kind of checking is done at database object level, so that when query would have resolved to base objects properly, Oracle would have realized it needs to access DUAL only and there is no such physical table like "subquery". However, by now I realize that one can't do database object level checking when database is closed as there is no means to access OBJ$ table itself. Chicken and egg problem (which is why the bootstrap segment exists btw).

Anyway, I decided to do a little test and to my surprise it worked!

SQL> with v$instance as (select 'blah' x from dual) select * from v$instance;

X
------------
blah

SQL>
 

Note that I am not selecting from v$instance v$ view but I just name my subquery alias to string "v$instance"! And apparently Oracle query execution engine is fine with it, as long as you select from an "object" which name matches one of the hardcoded ones...

As, I'm still busy, I make it short.

LatchProf is a script similar to WaitProf, only it samples latch holder statistics from V$LATCHHOLDER. As V$LATCHHOLDER contains a SID column (with session ID of a latch holder) it becomes possible to find who is hitting a latch the most (a way to prove that crappy monitoring tools which constantly scan through V$SQL DO cause library cache latch contention themselves).

Some day I will write longer about this (and check the script itself), but here's the syntax:

@latchprof <what> <sid|%> <latch|%|childaddr> <#samples>   

Here are the parameter definitions:

1) Columns to select actually map to V$LATCHHOLDER columns. So you can use SID,NAME,LADDR there for example.
2) SID is either a sid as number or % to see latches held by all SIDs
3) latch_name can be either % for all latches, a LIKE string, like %library% to see all latches with library in their name or the actual latch address! This allows us to monitor only specific child latches we want.
4) Number of samples to take. I normally use 100 000 samples as starting point.

As latchprof (and waitprof for that matter) never sleep, they burn as much CPU resource as they can. This is good for sampling accuracy (well, at least on multiprocessor systems with enough spare CPU capacity), but will consume one of your CPUs. So, you don't want to run this for too long at a time ( you could implement a short wait in there with a nice trick by Jonathan Lewis ).

100k samples is usually done in around a second on my test kit (giving 100+ kHz sampling rate :) This will vary by your hardware and Oracle version and what's your processes parameter (as V$LATCHHOLDER iterates through process state objects - the more processes you've got, the more time it takes).

Anyway, let say we have a performance issue and using wait interface we have determined that for a particular session, waits for cache buffers chains latches are unreasonably high.

Now, one way forward would be to identify the database blocks involved in this problem, by looking what blocks are protected by the latch (using X$BH.HLADDR = V$LATCH_CHILDREN.ADDR mapping) and trying to figure out the hottest block by X$BH.TCH touchcount. But where would that lead us? We would find out into which segment that block belongs, but it doesn't really help us to understand who's the troublemaker causing this issue.

Therefore we use another approach - sampling V$LATCHHOLDER to see which sessions hold the latches the most:

Just for demo purposes I start with listing general latchholder stats (not broken down by holding SID):

SQL> @latchprof name % % 100000

NAME                                 Held       Gets  Held %     Held ms Avg hold ms
------------------------------ ---------- ---------- ------- ----------- -----------
cache buffers chains                21248      17418   21.25     267.725        .015
simulator lru latch                  7192       7169    7.19      90.619        .013
simulator hash latch                   66         66     .07        .832        .013
enqueue hash chains                     4          4     .00        .050        .013
ges resource hash list                  3          3     .00        .038        .013
process allocation                      2          2     .00        .025        .013
library cache                           2          2     .00        .025        .013
name-service namespace bucket           2          2     .00        .025        .013
name-service memory objects             1          1     .00        .013        .013
ges caches resource lists               1          1     .00        .013        .013
library cache pin                       1          1     .00        .013        .013

11 rows selected.

Yes it looks that cache buffers chains latches are held the most. The Held column here means that during how many samples a latch was held. The Gets column shows roughly how many times that latch was gotten during sampling time.
As we took 100000 samples and during 21248 of them a cache buffers chains latch was held, we can conclude that this latch was busy (Held) 21.25% of the time. And as the sampling lasted about 1.3 seconds that time, we can extrapolate that of the 1.3 seconds, some cache buffers chains latches were in Held state for rougly 267 milliseconds. And as we know the number of gets, then we can compute the average latch hold time in milliseconds. This could help us detect very long latch holds like what can happen when someone is scanning through X$KSMSP or the server is CPU starved. The current estimate of 15 microseconds per latch hold is not that bad (note that with very few samples like 10000 or less, the avg hold time estimates become unreliable).

Anyway, let's break down the individual latch holding sessions now:

SQL> @latchprof sid,name % % 100000

       SID NAME                                 Held       Gets  Held %     Held ms Avg hold ms
---------- ------------------------------ ---------- ---------- ------- ----------- -----------
        81 cache buffers chains                14916      14915   14.92     187.942        .013
        81 simulator lru latch                  6797       6745    6.80      85.642        .013
        85 cache buffers chains                 6195       2284    6.20      78.057        .034
        85 simulator lru latch                   524        506     .52       6.602        .013
        81 simulator hash latch                  121         85     .12       1.525        .018
       112 name-service namespace bucket           2          2     .00        .025        .013
       100 enqueue hash chains                     2          2     .00        .025        .013
        98 library cache                           2          2     .00        .025        .013
       112 process allocation                      2          2     .00        .025        .013
       112 ges caches resource lists               1          1     .00        .013        .013
       112 enqueues                                1          1     .00        .013        .013
       104 messages                                1          1     .00        .013        .013
       100 cache buffers chains                    1          1     .00        .013        .013
       100 active service list                     1          1     .00        .013        .013
       100 enqueues                                1          1     .00        .013        .013
        85 simulator hash latch                    1          1     .00        .013        .013
       100 library cache                           1          1     .00        .013        .013

17 rows selected.

We see that sessions 81 and 85 are hitting CBC latches the most. Let say (for demo purposes again) that the SID 85 was the one which experienced bad performance and led us to investigate this issue.

As there are tens of thousands (or even more) of CBC latches in a decent Oracle database, then of course looking only at aggregated latch name level stats is not enough. Do the sessions compete for a single child latch? Or do they just use lots of different CBC latches over time and not collide at all? This is the question where LatchProf can help.

Let's include latch address (child latch address) into picture and query only "cache" related latches. This returns lots of rows, so I normally press CTRL+C after first pageful of results as they are most important (the results are ordered by most busy ones first):

SQL> @latchprof sid,name,laddr % cache 100000

       SID NAME                           LADDR          Held       Gets  Held %     Held ms Avg hold ms
---------- ------------------------------ -------- ---------- ---------- ------- ----------- -----------
        81 cache buffers chains           41AACEEC      15061      15017   15.06     186.756        .012
        85 cache buffers chains           41B2C37C         34         34     .03        .422        .012
        85 cache buffers chains           41B85D64         31         31     .03        .384        .012
        85 cache buffers chains           41BCF7FC         30         30     .03        .372        .012
        85 cache buffers chains           41B415EC         28         28     .03        .347        .012
        85 cache buffers chains           41BCA658         25         25     .03        .310        .012
        85 cache buffers chains           41B4E738         25         25     .03        .310        .012
        85 cache buffers chains           41AE4694         25         25     .03        .310        .012
        85 cache buffers chains           41AB02E0         25          9     .03        .310        .034
        85 cache buffers chains           427FA1F8         24         24     .02        .298        .012
        85 cache buffers chains           427F6E04         22         22     .02        .273        .012
        85 cache buffers chains           41B78D94         22         22     .02        .273        .012
        85 cache buffers chains           41BA2220         21         21     .02        .260        .012
        85 cache buffers chains           41BEDA68         21         21     .02        .260        .012
        85 cache buffers chains           41BC8D1C         21         21     .02        .260        .012
        85 cache buffers chains           41B486BC         20         20     .02        .248        .012
        85 cache buffers chains           42B99698         19         19     .02        .236        .012
        85 cache buffers chains           41AE6EA8         19         14     .02        .236        .017
        85 cache buffers chains           41B1F230         19         19     .02        .236        .012
        85 cache buffers chains           41B09E44         19         17     .02        .236        .014
        85 cache buffers chains           41AE7A88         19         19     .02        .236        .012
        85 cache buffers chains           41B69E98         18         18     .02        .223        .012
[...snip...]

And voila, we see SID 81 is hitting one single child latch in maniac fashion while SID 85 is behaving more normally, just hitting lots of different CBC latches (you see the number of gets of a single child during 100k sampling doesn't exceed 34). Interesting.
So now that we know the child latch address (LADDR), we can refine our query to only monitor that child latch to see if any other sessions could be affected:

SQL> @latchprof sid,name,laddr % 41AACEEC 100000

       SID NAME                           LADDR          Held       Gets  Held %     Held ms Avg hold ms
---------- ------------------------------ -------- ---------- ---------- ------- ----------- -----------
        81 cache buffers chains           41AACEEC      14058      14047   14.06     177.131        .013
        85 cache buffers chains           41AACEEC         18         18     .02        .227        .013

As of now it looks like noone else except the maniac SID 81 and victim SID 85 contend for the latch (otherwise one would expect to see at least few successful holds against that latch).

So, as SID 81 is so evidently the troublemaker here, let's check what it's doing:

SQL> select sql_text from v$sql where hash_value = (select sql_hash_value from v$session where sid = 81);

SQL_TEXT
---------------------------------------------------------------------------------------------------------
select count(*) X from kill_cpu connect by n > prior n start with n = 1

1 row selected.

This is Jonathan Lewis'es kill_cpu script in action, which hammered the same datablock in a tight loop using connect by ( you need to set _old_connect_by_enabled=true on 9i+ for this test to work :)

I will blog about some more use cases and an extended version of LatchProf script in (hopefully) near future.

Note once more that this script only detects latch holders. A latch held the most doesn't necessarily mean it's the latch waited for the most. You still need to start your diagnosis from wait interface, using Snapper or WaitProf for example. Only if these tools show significant latch waits, you should drill down into latches further. And as Anjo Kolk recently (very well) said, latch contention is a symptom, not the problem. The root cause is somewhere else (in the application likely), once you fix that, the latch contention will go away itself.

SQL> @sn 1 119

-- Session Snapper v1.07 by Tanel Poder ( http://www.tanelpoder.com )

---------------------------------------------------------------------------------------------------------------------------------------------
HEAD,     SID, SNAPSHOT START   ,  SECONDS, TYPE, STATISTIC                               ,         DELTA,  DELTA/SEC,     HDELTA, HDELTA/SEC
---------------------------------------------------------------------------------------------------------------------------------------------
DATA,     119, 20080621 05:22:05,        1, STAT, session logical reads                   ,         18284,      18284,     18.28k,     18.28k
DATA,     119, 20080621 05:22:05,        1, STAT, consistent gets                         ,         15301,      15301,      15.3k,      15.3k
DATA,     119, 20080621 05:22:05,        1, STAT, consistent gets from cache              ,         15228,      15228,     15.23k,     15.23k
DATA,     119, 20080621 05:22:05,        1, STAT, consistent gets from cache (fastpath)   ,         15136,      15136,     15.14k,     15.14k
DATA,     119, 20080621 05:22:05,        1, STAT, calls to get snapshot scn: kcmgss       ,            89,         89,         89,         89
DATA,     119, 20080621 05:22:05,        1, STAT, no work - consistent read gets          ,         14883,      14883,     14.88k,     14.88k
DATA,     119, 20080621 05:22:05,        1, STAT, table scans (short tables)              ,            21,         21,         21,         21
DATA,     119, 20080621 05:22:05,        1, STAT, table scan rows gotten                  ,       1429227,    1429227,      1.43M,      1.43M
DATA,     119, 20080621 05:22:05,        1, STAT, table scan blocks gotten                ,         17440,      17440,     17.44k,     17.44k
DATA,     119, 20080621 05:22:05,        1, WAIT, events in waitclass Other               ,        399831,     399831,   399.83ms,   399.83ms
--  End of snap 1

From Oracle 10g Oracle has consolidated lots of the events into "events in waitclass Other". This is because saving all 900+ wait event stats for every session (in V$SESSION_EVENT array) would waste too much memory with giving little benefit (normally there's only a handful of troublemaking events anyway). Therefore makes sense to aggregate the least likely happening events under some common category. Looks like Oracle kernel coders have set a threshold in event number above which all events are grouped under the "other" waitclass.

See below, this is from 11g:

SQL> select count(*) from v$event_name;

  COUNT(*)
----------
       961

SQL> select wait_class, count(*) from v$event_name group by wait_class;

WAIT_CLASS                                                         COUNT(*)
---------------------------------------------------------------- ----------
Concurrency                                                              26
System I/O                                                               23
User I/O                                                                 22
Administrative                                                           51
Other                                                                   632
Configuration                                                            21
Scheduler                                                                 3
Cluster                                                                  47
Application                                                              15
Queueing                                                                  4
Idle                                                                     80
Network                                                                  35
Commit                                                                    2

13 rows selected.

There is total 961 different wait events in this version of Oracle, but 632 of these are categorized under "other" name.

So, what to do when this "events in waitclass Other" wait becomes significant in the response time profile?

Actually it's simple. Only the V$SESSION_EVENT experiences this issue of event aggregation, V$SESSION_WAIT (and sql_tracing) do not. The reason (again) is that unlike V$SESSION_EVENT, the V$SESSION_WAIT view does not have to store the cumulative stats in somewhere memory, therefore aggregation for memory savings is not needed there.

So, let's use my WaitProf to check what's the real event my session is waiting for:

SQL> @waitprof print 119 e123 1000000

-- WaitProf 1.04 by Tanel Poder ( http://www.tanelpoder.com )

                                                                                                                          % Total  Total Event   Distinct   Avg time
    SID STATE   EVENT                               P1                         P2              P3                    SEQ#    Time      Time ms     Events   ms/Event
------- ------- ----------------------------------- -------------------------- --------------- --------------- ---------- ------- ------------ ---------- ----------
    119 WORKING On CPU / runqueue                                                                                           78.04    13056.393          1  13056.393
    119 WAITING latch free                          address= 000000005E60BE28  number= 148     tries= 0                     21.96     3673.607          1   3673.607

Elapsed: 00:00:16.73

See, it's actually waiting for a "latch free" event!

What is this latch?

SQL> @la 5E60BE28

ADDR         LATCH#   CHLD NAME                                           GETS      IGETS     MISSES    IMISSES     SLEEPS  WAIT_TIME
-------- ---------- ------ ---------------------------------------- ---------- ---------- ---------- ---------- ---------- ----------
5E60BE28        148      6 simulator lru latch                        16718933          3     144011          0       3456 1894137768

And we found that this is a buffer cache simulation (V$DB_CACHE_ADVICE) related latch!

To be honest, I kind of knew what the issue was already - I noticed that the servers CPUs were already 100% utilized and the CPU runqueues were quite lenghty. Once you have 100% CPU usage and runqueues start to build up, you will start seeing latch contention on solitary latches and latches with only few children. This is mostly due that if the unlucky event happens that a latch holder is scheduled off CPU then it may take tens or hundreds of milliseconds before the holder gets back to CPU to complete its work and release the latch. However it's likely that during that time many other processes try to grab the latch and start spinning on it. This raises the CPU usage even more. In this db there thousands of cache buffers chains latches, but only 16 of simulator lru latches. Therefore, even if a CBC latch holder went to sleep, the likelyhood of a collision was quite unlikely than in a similar case with simulator lru latch holder.

This brings me to point that whenever you see any latch contention (especially unusual one) in your performance tool output, first check the CPU utilization. High CPU usage (due execution plan gone bad, parallelism kicking in unexpectedly etc) can cause latch contention and in those conditions the "latch free" events are just a symptom of your real problem.

Anyway, I was writing about "events in waitclass Other", not latches today, so few more points on events:

I have a script which queries V$FIXED_VIEW_DEFINITION to see the contents of V$ views - and I check out on what X$ table is V$SESSION_EVENT based:

SQL> @v gv$session_event

VIEW_NAME                      TEXT
------------------------------ ----------------------------------------------------------------------------------------------------
GV$SESSION_EVENT               select s.inst_id, s.kslessid, d.kslednam, s.ksleswts, s.kslestmo, round(s.kslestim / 10000),
                               round(s.kslestim / (10000 * s.ksleswts), 2), round(s.kslesmxt / 10000),  s.kslestim, d.ksledhash,
                               d.ksledclassid, d.ksledclass#, d.ksledclass  from x$ksles s, x$ksled d  where s.ksleswts != 0 and
                               s.kslesenm = d.indx

SQL>

X$KSLES stores the session level wait event stats.
X$KSLED stores the event names (as X$KSLES doesn't store the event name itself but just its ID)

So, we check how many V$SESSION_EVENT records we have for session 119:

SQL> select count(*) from x$ksles where kslessid = 119;

  COUNT(*)
----------
       331

SQL>

331 records only for this session.

But, as I showed above, V$EVENT_NAME which is based on X$KSLED had 900+ event names in it.

So lets see what events do we see after event# 331:

SQL> select indx, kslednam from x$ksled where indx > 331;

      INDX KSLEDNAM
---------- ---------------------------------------------------------
       332 latch free
       333 unspecified wait event
       334 latch activity
       335 wait list latch activity
       336 wait list latch free
       337 kslwait unit test event 1
       338 kslwait unit test event 2
       339 kslwait unit test event 3
       340 global enqueue expand wait
       341 free process state object
       342 inactive session
       343 process terminate
       344 latch: session allocation
       345 check CPU wait times
       346 enq: CI - contention
       347 enq: PR - contention
       348 ksim generic wait event
       349 debugger command
       350 ksdxexeother
       351 ksdxexeotherwait
       352 enq: PE - contention
       353 enq: PG - contention
       354 ksbsrv
[...snipped...]

Most of the events are quite unknown ones, but it looks like latch free is also in one of the "other" ones now!
That explains why Snapper showed "events in wait class Other" instead of latch free.

Note that V$SYSTEM_EVENT doesn't have that event aggregation issue - as you need to store systemlevel wait stats only in one location in instance (and not for each session).

V$SYSTEM_EVENT is based on X$KSLEI:

SQL> select count(*) from x$kslei;

  COUNT(*)
----------
       961

That's pretty much all for today, but theres one more think I want to show, related to X$ table quirks. As V$ views and X$ tables are not normal tables (but rather structure definitions and pointers to underlying functions), they don't have to play according to normal table rules. Check this:

SQL> select min(indx),max(indx) from X$ksles where kslessid in (1);

 MIN(INDX)  MAX(INDX)
---------- ----------
         0        330

SQL> select min(indx),max(indx) from X$ksles where kslessid in (2);

 MIN(INDX)  MAX(INDX)
---------- ----------
         0        330

You see that regardless of the parameter I pass to the query, I do get min and max values of 0 and 330 in INDX column.

SQL> select min(indx),max(indx) from X$ksles where kslessid in (1,2);

 MIN(INDX)  MAX(INDX)
---------- ----------
         0        661

However if I pass both parameters together, then the max value reaches 661.

SQL> select min(indx),max(indx) from X$ksles where kslessid in (1,2,3);

 MIN(INDX)  MAX(INDX)
---------- ----------
         0        992

If I add third value, the max(indx) goes up to 992... however If I use only 3rd value (which caused the max to go up to 992), then the max goes down to 330:

SQL> select min(indx),max(indx) from X$ksles where kslessid in (3);

 MIN(INDX)  MAX(INDX)
---------- ----------
         0        330

This illustrates that the INDX column in X$ tables is quite like ROWNUM in normal tables. Actually, the INDX (as the name says) is the index into underlying X$ table array. However as this X$ table was a "dynamic" one which was not based on a fixed SGA structure directly but really called an intermediate X$ service function, which fetched only the required data from SGA and placed it into an array in UGA, the INDX represents the slot in the result UGA array, not SGA one ;)

I'll write about identifying different variations of X$ tables in a future post ;)

Anyway, when I wanted to drop that table involved, this happened:

SQL> drop table MYTABLE;
drop table MYTABLE
           *
ERROR at line 1:
ORA-00054: resource busy and acquire with NOWAIT specified

I can't drop a table as someone is holding a lock on it. Fair enough, this was a dev environment used only by me, so I used DBA_OBJECTS.OBJECT_ID to find out the object ID of that table:

SQL> @o MYTABLE

owner                     object_name                    object_type        CREATED           LAST_DDL_TIME     status           OID      D_OID
------------------------- ------------------------------ ------------------ ----------------- ----------------- --------- ---------- ----------
XYZ_DEV01_OWNER           MYTABLE                        TABLE              20080616 11:08:44 20080616 11:08:44 VALID          63764      63764

...and then I queried what enqueue locks were held on that object:

SQL> select * from v$lock where id1=63764;

ADDR     KADDR           SID TY        ID1        ID2      LMODE    REQUEST      CTIME      BLOCK
-------- -------- ---------- -- ---------- ---------- ---------- ---------- ---------- ----------
40034278 40034290        130 TM      63764          0          6          0       2662          0

Ok, I see session 130 holding a TM lock on that table. I queried the corresponding SERIAL# from v$session as well and killed the session:

SQL> alter system kill session '130,8764';
alter system kill session '130,8764'
*
ERROR at line 1:
ORA-00031: session marked for kill

SQL> select * from v$lock where id1=63764;

ADDR     KADDR           SID TY        ID1        ID2      LMODE    REQUEST      CTIME      BLOCK
-------- -------- ---------- -- ---------- ---------- ---------- ---------- ---------- ----------
40034278 40034290        130 TM      63764          0          6          0       2668          0

After hanging for 60 seconds, my kill command gave up (and marked my session for kill), but my lock was still not released... Now what?

This happens when the victim session is so stuck somewhore in an Oracle call that it never has a chance to receive the message it has been killed. And apparently some resources used can in that call can't be released (for whatever reason, it may be by design, it may just be a bug).

The below queries against V$SESSION and V$LOCK show that even though the session has been marked to be in killed status, it's still holding a lock:

SQL> @usid 130

USERNAME                SID                 AUDSID OSUSER           MACHINE            PROGRAM              SPID          HASH_VALUE   LASTCALL STATUS
----------------------- -------------- ----------- ---------------- ------------------ -------------------- ------------ ----------- ---------- --------
XYZ_DEV01_OWNER          '130,8764'          33533 1288249          \XYZHOST001        sqlplus.exe          3872          3564023715       4032 KILLED

SQL> select * from v$lock where id1=63764;

ADDR     KADDR           SID TY        ID1        ID2      LMODE    REQUEST      CTIME      BLOCK
-------- -------- ---------- -- ---------- ---------- ---------- ---------- ---------- ----------
40034278 40034290        130 TM      63764          0          6          0       2695          0

Ok, I tried various other options, like kill immediate and disconnect, which should have higher chance to clean up my session properly:

SQL> alter system kill session '130,8764' immediate;
alter system kill session '130,8764' immediate
*
ERROR at line 1:
ORA-00031: session marked for kill

SQL> alter system disconnect session '130,8764' immediate;
alter system disconnect session '130,8764' immediate
*
ERROR at line 1:
ORA-00031: session marked for kill

SQL> select * from v$lock where id1=63764;

ADDR     KADDR           SID TY        ID1        ID2      LMODE    REQUEST      CTIME      BLOCK
-------- -------- ---------- -- ---------- ---------- ---------- ---------- ---------- ----------
40034278 40034290        130 TM      63764          0          6          0       2710          0

Still no luck, lock is there.

So I found the OS PID of my server process (or actually OS Thread ID inside oracle.exe process as I was on Windows) and used oradebug short_stack to check where that process was stuck (the output is slightly formatted):

SQL> oradebug setospid 3872;
Oracle pid: 18, Windows thread id: 3872, image: ORACLE.EXE (SHAD)

SQL> oradebug short_stack;
_ksdxfstk+14<-_ksdxcb+1481<-_ssthreadsrgruncallback+428<-_OracleOradebugThreadStart@4+819
<-7C80B680<-00000000<-71A557C4<-71AB4376<-6298540C<-6298325E<-60A0D931<-609D005F<-609B073D<-609AF9
endExitAgent+202<-_hoxexit+188<-_hogmdis+890<-_hogdisc+8<-_xupidhs+137<-_upidhs+20<-_kpudtch+305
<-_OCIServerDetach+12<-_ncodncf+268<-_npidr0+2300<-_npifcc+46<-_qerrmFBu+457<-_qerrmFetch+1
+1291<-_opiodr+1099<-_rpidrus+178<-_rpidru+88<-_rpiswu2+426<-_rpidrv+1461<-_psddr0+449
<-_psdnal+283<-_pevm_EXIM+153<-_pfrinstr_EXIM+34<-_pfrrun_no_tool+56<-_pfrrun+781<-_plsql_run+738
<-_pr+1099<-_opidrv+819<-_sou2o+45<-_opimai_real+112<-_opimai+92<-_OracleThreadStart@4+726<-7C80B680

This terse stack shows (start reading from bottom right to left) this process is stuck somewhere "above" qerrmFetch (Remote Fetch). I guess the functions starting with "h" above that are heterogenous services functions. By the way, V$SESSION_WAIT didn't show any wait state changes either and the session itself was still constantly waiting for "HS message to agent" event. So, being stuck in a HS call was probably the reason why that session could not clean itself up.

Now, in such situations one normally would proceed with ORAKILL on Windows or just killing that server process at Unix level (after carefully thinking what you're about to do). Killing the process makes PMON to perform the cleanup and PMON usually does clean all resources up ok. However I didn't have access to that Windows server box, so OS kill was out of question.

So, I used another trick. While being connected to the victim process using oradebug, I issued this:

SQL> oradebug event immediate crash;
ORA-00072: process "Windows thread id: 3872, image: ORACLE.EXE (SHAD)" is not active
SQL>

After waiting for a while (crashdump occurred), sqlplus reported that the target process doesn't exist anymore. Thanks to the crash, PMON had woken up and performed the cleanup.

Let's see if it helped:

SQL> oradebug short_stack;
ORA-00072: process "Windows thread id: 3872, image: ORACLE.EXE (SHAD)" is not active

SQL> select * from v$lock where id1=63764;

no rows selected

SQL> @usid 130

no rows selected

SQL> drop table MYTABLE;

Table dropped.

Yippee, I finally got rid of that session, lock and could drop my table!

Note that I would still prefer killing the processes with Unix kill or Windows orakill, however there are some bugs with orakill (search in Metalink) that it doesn't always succeed killing the thread properly. And in my case I didn't have OS access anyway.

Ideally, the ALTER SYSTEM KILL session command should do all needed cleanup for us, but there are some corner cases involving loops, hangs and bugs where this approach won't work. There's a Metalink note 1020720.102 with a list of ways for killing Oracle processes/threads from OS level.

That's kind of good news, I hope this will eventually reduce the number of expert DBAs who set this parameter to true whenever they see any kind of shared pool / library cache latch contention.

On the other hand, spin_count was made an undocumented parameter long time ago, but is still heavily abused worldwide so I wouldn't be surprised if the same happens to future _cursor_space_for_time...

Before proceeding though, please read this post about safety of different stack sampling approaches.

I have had few non-trivial Oracle troubleshooting cases, related to query hangs and bad performance, where I've wanted to know where exactly in execution plan the current execution is.
Remember, Oracle is just another program executing instructions clustered in functions on your server, so stack sampling can help out here as well.

So, I was looking into the following stack trace taken from an Oracle 10.1 database on Solaris SPARC, running a SQL with this execution plan.

$ cat pstack.txt
 ------------------------------------------------------------------------
 0000000101eca6e4 kdirfrs (ffffffff78eca5d8, 1, 0, 2, 86521ee00, 0) + 24
 0000000102502adc qerixFetchFastFullScan (ffffffff78eca3e0, 10248ebc0, 0, 1, 86521ebf8, fffd) + 33c
 0000000102558720 qergiFetch (ffffffff78ecaca8, 10248ebc0, ffffffff7fff5ee8, 10449dba0, 1, 86bb6bb38) + 80
 000000010248edd8 qerjoFetch (45, 10248ebc0, ffffffff7fff5ee8, ffffffff78ecad60, 86bb6bc08, 114) + 118
 000000010248eeb8 qerjoFetch (7ffe, 10248ebc0, ffffffff7fff5fe8, ffffffff78ecb5e8, 7f399b6f8, 7ffe) + 1f8
 00000001025171b8 rwsfcd (8177747a8, 10248e1e0, ffffffff7fff6168, 7fff, f0, 10449dba0) + 78
 000000010248e4dc qeruaFetch (745c0ab40, 10248e1e0, ffffffff7fff6168, ffffffff78ecb620, 10449dba8, 8177747a8) + 11c
 000000010248d840 qervwFetch (1d, 1d, ffffffff7fff6238, 7fff, 7cfc87c50, 10449d000) + a0
 00000001025171b8 rwsfcd (7876e8d18, 10248e1e0, ffffffff7fff63b8, 7fff, c0, 10449dba0) + 78
 000000010248e4dc qeruaFetch (817d1e438, 10248e1e0, ffffffff7fff63b8, ffffffff78e7d318, 10449dba8, 7876e8d18) + 11c
 000000010248d840 qervwFetch (1d, 1d, ffffffff7fff6488, 7fff, 82355a348, 10449d000) + a0
 00000001025171b8 rwsfcd (7454b0408, 10249e4e0, ffffffff7fff6608, 7fff, c0, 10449dba0) + 78
 00000001024a4620 qerhjFetch (86b1d64d8, 0, 0, 1, ffffffff78e7df08, 0) + 300
 000000010248f99c qerjotFetch (78d8ea1d8, 0, 0, 1, ffffffff78e7e8d0, 10449dba0) + dc
 000000010248eeb8 qerjoFetch (1, 10248ebc0, ffffffff7fff6838, ffffffff78e7f2b0, 7d53bb730, 1) + 1f8
 000000010248eeb8 qerjoFetch (1, 10248ebc0, ffffffff7fff6938, ffffffff7b9b2f78, 74b306ba0, 7fff) + 1f8
 000000010248d840 qervwFetch (5, 5, ffffffff7fff6a08, 7fff, 80b0faa18, 10449d000) + a0
 00000001025171b8 rwsfcd (865e988f8, 10249e4e0, ffffffff7fff6b88, 7fff, c0, 10449dba0) + 78
 00000001024a4620 qerhjFetch (7a57ae350, 10249e4e0, ffffffff7fff6d88, 1, ffffffff7bb89f00, 0) + 300
 00000001025171b8 rwsfcd (823642298, 10249e4e0, ffffffff7fff6d88, 7fff, 6f0, 10449dba0) + 78
 00000001024a4620 qerhjFetch (751158588, 102517980, 7511587f0, 1, ffffffff7b9b5090, 0) + 300
 000000010251d1f0 qersoFetch (94, 10506adf8, 10449d000, ffffffff7b9b53b8, 799854d60, 7511587f0) + 350
 0000000101aa6f24 opifch2 (7, f, 150, 1, 104400, 1050685e8) + a64
 0000000101aa6384 opifch (5, 2, ffffffff7fff79f8, 105000, 0, 10434c0a8) + 44
 0000000101ad81ec opipls (104000, 10434c, 1, ffffffff7bba3e6a, 0, 140010) + f4c
 00000001002d0058 opiodr (6, 10506ae10, 10434cfb0, 10506a, 105000, 104000) + 598
 00000001002d4ec0 rpidrus (ffffffff7fff8820, 105067f18, 105068860, ffffffff7bb5f560, 4a8c, 200000) + a0
 0000000102f615e4 skgmstack (ffffffff7fff8a48, ffffffff7f87cf8f, ffffffff7fff898f, 1002d4e20, ffffffff7fff8a70, acc01800) + a4
 00000001002d504c rpidru (ffffffff7fff9140, 10422b000, 10422a918, 104229598, 410, 82) + ac
 00000001002d4808 rpiswu2 (0, 104556000, ffffffff7fff8b88, 2, 104556418, ffffffff7fff92c0) + 1a8
 00000001002d61cc rpidrv (a, ffffffff7fff9044, 9, ffffffff7fff90c0, ffffffff7fff9140, 1002d5180) + 62c
 0000000102797f90 psddr0 (104400, 86e2fc628, ffffffff7fff92c0, 9, 1050769d8, 1050769d8) + 1f0
 0000000102798e04 psdnal (ffffffff7fffa0b8, ffffffff7fffa258, 105000, ffffffff7bc5eb00, 7f7aa86d0, 40) + 184
 000000010376d268 pevm_BFTCHC (0, 7f7aa86d0, 50, ffffffff7bb5f560, ffffffff7bc5eb00, 0) + 188
 000000010373dff4 pfrinstr_FTCHC (10, 15d0000000000000, ffffffff7bb5f5c8, ffffffff7bb5f560, 4a8c, ffffffff7bb66330) + b4
 00000001037362c8 pfrrun_no_tool (ffffffff7bb5f560, 779953684, ffffffff7bb5f5c8, 10457c9d8, 2001, 2001) + 48
 00000001037372d0 pfrrun (ffffffff7bb5f5c8, 200000, 0, 200000, ffffffff7bb5f560, 779abb110) + 2f0
 0000000103783374 plsql_run (ffffffff7bb55408, 1, 0, ffffdfff, ffffffff7fffa0b8, 0) + 274
 0000000103722554 peicnt (ffffffff7fffa0b8, 105068860, 6, ffffffff7fff9f28, 41d8, 1050685e8) + d4
 000000010327b784 kkxexe (105000, 104000, 105068, 104296000, 1050685e8, ffffffff7bb5f560) + 284
 0000000101ad0228 opiexe (4, ffffffff7bc68ee8, ffffffff7fffab00, 0, 0, ffffffff7bc70480) + 33c8
 0000000101a4c0a8 kpoal8 (40008, 1, ffffffff7fffd890, 0, 0, 4) + 648
 00000001002d0058 opiodr (14, 10506ae10, 10434ce70, 10506a, 105000, 104000) + 598
 0000000102cded94 ttcpip (105071450, 18, ffffffff7fffd890, ffffffff7fffcb88, 104229c98, ffffffff7fffcb84) + 694
 00000001002cd3e8 opitsk (1002cf000, 1, 0, ffffffff7fffd9e8, 105071450, 105071458) + 428
 0000000101aaf564 opiino (105070000, 1050683c0, 0, 0, f5, 105070290) + 404
 00000001002d0058 opiodr (4, 10506ae10, 10434c920, 10000, 105071, 105000) + 598
 00000001002cc174 opidrv (0, 4, 10506a, 105071450, 0, 3c) + 354
 00000001002c9828 sou2o (ffffffff7fffe6b8, 3c, 4, ffffffff7fffe698, 104aa6000, 104aa6) + 48
 00000001002a7b34 main (2, ffffffff7fffe798, ffffffff7fffe7b0, 0, 0, 100000000) + 94
 00000001002a7a7c _start (0, 0, 0, 0, 0, 0) + 17c

Not too encouraging, huh?

So, let's run this stack trace through my os_explain script:

$ ./os_explain pstack.txt
   SELECT FETCH:
    SORT: Fetch
     HASH JOIN: Fetch
    * HASH JOIN: Fetch
     * VIEW: Fetch
        NESTED LOOP OUTER: Fetch
         NESTED LOOP OUTER: Fetch
          NESTED LOOP JOIN: Fetch
           HASH JOIN: Fetch
          * VIEW: Fetch
             UNION-ALL: Fetch
            * VIEW: Fetch
               UNION-ALL: Fetch
              * NESTED LOOP OUTER: Fetch
                 NESTED LOOP OUTER: Fetch
                  GRANULE ITERATOR: Fetch
                   INDEX: FetchFastFullScan
                    kdirfrs

Now this is much more understandable!

All my script did was:

• remove the bottom part of the stack not relevant to plan execution
• reverse the order of stack trace lines for better human readability
• translating Query Execution Rowsource (qer) function prefixes to their corresponding names using info provided in Metalink note 175982.1

Easy :)

So, how to read this?

First, by now it should be obvious that in Oracle, each rowsource operator (the different lines you see in SQL execution plans) is actually just a function inside Oracle kernel. These are the row source functions, starting with "qer". QER stands for Query Execution Rowsource as far as I know.

Whenever an Oracle process fetches data from a cursor, it calls opifch2() which in turn calls the root rowsource function in execution plan. In my case that function was qersoFetch and my os_explain script just substituted the "qerso" part with SORT (as per the Metalink note I mentioned above). The first child function of qersoFetch was qerhjFetch, which is a hash join rowsource, and so on. Note that os_explain prefixes some lines with an asterisk (*), this indicates that the output of given function is in turn filtered by a filter operation (the same filter ops what you normally see in the bottom of DBMS_XPLAN explained plans).

So, logically you can imagine an execution plan as a tree of Oracle functions:

• For a SELECT query the OPI fetch function (opifch2) would be the root.
• Various join and union functions like qerhjFetch (HASH JOIN) and qerjotFetch (NESTED LOOPS JOIN) would be branches.
• The leaves would always be some sort of access path functions like qertbFetch (TABLE ACCESS FULL) or qerixFetch ( INDEX UNIQUE / FULL / RANGE SCAN ).

But physically, an execution plan is just a memory structure in subheap 6 of a child cursor (x$kglcursor.kglobhd6), which has a bunch of rowsource function opcodes in it.
During plan execution Oracle process "just" traverses through those opcodes, looks up the corresponding rowsource function starting address using a lookup table and calls it. That function does its task (probably calls other rowsource functions recursively) and returns to caller.

Note that many rowsource functions are designed to be cascading, being able to do only the work needed for returning a small subset of rows and return only few rows at a time, as opposed to the whole resultset.
This is a very good thing as rows can be cascaded, or pipelined back to parent functions as rows become available. For example a table fetch only fetches a handful of rows (and not the whole table) at a time and returns these "up" for further processing. Also, a nested loop join is able to pass matching rows "up" from the moment first matches are found, again there's no need to perform the join on full dataset first before returning first rows.
This also means that there is no need to store the whole intermediate resultset somewhere in memory before passing it up to parent function; instead we just revisit that branch of execution plan tree whenever we need more rows from it. And the os_explain script shows you exactly in which execution branch the execution currently is.

Addition: I will elaborate on how to match the execution plan with stack trace in an upcoming post - it's too much material for an introductory post.

So, cascading rowsources allow us to "incrementally" execute plans involving large datasets, without need to keep the intermediate resultsets in memory. On the other hand, few rowsource operators in your execution plan, like SORT, can not return any rows up before all its children's rows are processed. With SORT (and aggregate operations which also use SORT) you just have to process all the source rows before returning any meaningful result back. You can't just go through only half of the source data, order it and start returning rows in hope that the rest of the rows should have not been returned as first in the order. This is where the SQL cursor workareas come into play for such operations.

SORT, HASH and BITMAP rowsources can allocate SQL workareas for them, while others can't. This can easily be identified from execution plan statistics of following sample query:

SQL> select /*+ gather_plan_statistics */ owner, count(*) from dba_source group by owner;

OWNER                            COUNT(*)
------------------------------ ----------
WKSYS                                8988
HR                                     34
[...some output snipped...]
SYS                                129299
WMSYS                                 704

25 rows selected.

SQL> select * from table(dbms_xplan.display_cursor(null,null,'MEMSTATS LAST'));

PLAN_TABLE_OUTPUT
----------------------------------------------------------------------------------------------------------------------------------
SQL_ID  dcp37kxt02m9f, child number 0
-------------------------------------
select /*+ gather_plan_statistics */ owner, count(*) from dba_source
group by owner

Plan hash value: 114136443

----------------------------------------------------------------------------------------------------------------------------
| Id  | Operation                     | Name            | Starts | E-Rows | A-Rows |   A-Time   |  OMem |  1Mem | Used-Mem |
----------------------------------------------------------------------------------------------------------------------------
|   1 |  HASH GROUP BY                |                 |      1 |    593K|     25 |00:00:18.24 |   821K|   821K| 5213K (0)|
|   2 |   VIEW                        | DBA_SOURCE      |      1 |    593K|    595K|00:00:16.84 |       |       |          |
|   3 |    UNION-ALL                  |                 |      1 |        |    595K|00:00:15.06 |       |       |          |
|*  4 |     FILTER                    |                 |      1 |        |    595K|00:00:10.88 |       |       |          |
|*  5 |      HASH JOIN                |                 |      1 |    595K|    595K|00:00:08.50 |   884K|   884K| 1316K (0)|
|*  6 |       HASH JOIN               |                 |      1 |   6527 |   6292 |00:00:00.12 |   870K|   870K| 1179K (0)|
|   7 |        TABLE ACCESS FULL      | USER$           |      1 |     98 |     98 |00:00:00.01 |       |       |          |
|*  8 |        HASH JOIN              |                 |      1 |   6527 |   6292 |00:00:00.09 |   909K|   909K| 1181K (0)|
|   9 |         INDEX FULL SCAN       | I_USER2         |      1 |     98 |     98 |00:00:00.01 |       |       |          |
|* 10 |         INDEX FAST FULL SCAN  | I_OBJ2          |      1 |   6527 |   6292 |00:00:00.04 |       |       |          |
|  11 |       INDEX FAST FULL SCAN    | I_SOURCE1       |      1 |    595K|    595K|00:00:01.56 |       |       |          |
|  12 |      NESTED LOOPS             |                 |      0 |      1 |      0 |00:00:00.01 |       |       |          |
|* 13 |       INDEX FULL SCAN         | I_USER2         |      0 |      1 |      0 |00:00:00.01 |       |       |          |
|* 14 |       INDEX RANGE SCAN        | I_OBJ4          |      0 |      1 |      0 |00:00:00.01 |       |       |          |
|  15 |     NESTED LOOPS              |                 |      1 |      1 |      0 |00:00:00.01 |       |       |          |
|  16 |      NESTED LOOPS             |                 |      1 |      1 |      0 |00:00:00.01 |       |       |          |
|  17 |       NESTED LOOPS            |                 |      1 |      1 |      0 |00:00:00.01 |       |       |          |
|* 18 |        INDEX FAST FULL SCAN   | I_OBJ2          |      1 |      6 |      0 |00:00:00.01 |       |       |          |
|* 19 |        FIXED TABLE FIXED INDEX| X$JOXFS (ind:1) |      0 |      1 |      0 |00:00:00.01 |       |       |          |
|* 20 |       INDEX RANGE SCAN        | I_USER2         |      0 |      1 |      0 |00:00:00.01 |       |       |          |
|  21 |      TABLE ACCESS CLUSTER     | USER$           |      0 |      1 |      0 |00:00:00.01 |       |       |          |
|* 22 |       INDEX UNIQUE SCAN       | I_USER#         |      0 |      1 |      0 |00:00:00.01 |       |       |          |
----------------------------------------------------------------------------------------------------------------------------

You see how the HASH operations do have their last Mem columns populated, therefore those rowsource functions did allocate a SQL workarea for them. Others like NESTED LOOPS joins and data access rowsources did not have any SQL workarea memory allocated as they are completely cascading.

Note that os_explain can also read the stack from STDIN as seen in example below. Also the -a option will tell os_explain to show all functions in the stack and not only the execution plan ones and their children.

Command:

SQL> alter session set statistics_level=typical;

Session altered.

SQL> select avg(length(text)) from dba_source where owner = 'SYS';

AVG(LENGTH(TEXT))
-----------------
       125.032127

Plan:

---------------------------------------------------------------------------------------------------
| Id  | Operation                           | Name            | E-Rows |  OMem |  1Mem | Used-Mem |
---------------------------------------------------------------------------------------------------
|   1 |  SORT AGGREGATE                     |                 |      1 |       |       |          |
|   2 |   VIEW                              | DBA_SOURCE      |  17972 |       |       |          |
|   3 |    UNION-ALL                        |                 |        |       |       |          |
|*  4 |     FILTER                          |                 |        |       |       |          |
|   5 |      NESTED LOOPS                   |                 |        |       |       |          |
|   6 |       NESTED LOOPS                  |                 |  18056 |       |       |          |
|*  7 |        HASH JOIN                    |                 |    198 |   909K|   909K| 1193K (0)|
|   8 |         INDEX FULL SCAN             | I_USER2         |     98 |       |       |          |
|   9 |         NESTED LOOPS                |                 |    198 |       |       |          |
|  10 |          TABLE ACCESS BY INDEX ROWID| USER$           |      1 |       |       |          |
|* 11 |           INDEX UNIQUE SCAN         | I_USER1         |      1 |       |       |          |
|* 12 |          INDEX RANGE SCAN           | I_OBJ5          |    198 |       |       |          |
|* 13 |        INDEX RANGE SCAN             | I_SOURCE1       |     93 |       |       |          |
|  14 |       TABLE ACCESS BY INDEX ROWID   | SOURCE$         |     91 |       |       |          |
|  15 |      NESTED LOOPS                   |                 |      1 |       |       |          |
|* 16 |       INDEX FULL SCAN               | I_USER2         |      1 |       |       |          |
|* 17 |       INDEX RANGE SCAN              | I_OBJ4          |      1 |       |       |          |
|  18 |     NESTED LOOPS                    |                 |      1 |       |       |          |
|  19 |      NESTED LOOPS                   |                 |      1 |       |       |          |
|  20 |       NESTED LOOPS                  |                 |      1 |       |       |          |
|  21 |        TABLE ACCESS BY INDEX ROWID  | USER$           |      1 |       |       |          |
|* 22 |         INDEX UNIQUE SCAN           | I_USER1         |      1 |       |       |          |
|* 23 |        INDEX RANGE SCAN             | I_OBJ5          |      1 |       |       |          |
|* 24 |       FIXED TABLE FIXED INDEX       | X$JOXFS (ind:1) |      1 |       |       |          |
|* 25 |      INDEX RANGE SCAN               | I_USER2         |      1 |       |       |          |
---------------------------------------------------------------------------------------------------

Stack:

$ pstack 23740 | ./os_explain -a
   main
    ssthrdmain
     opimai_real
      sou2o
       opidrv
        opiodr
         opiino
          opitsk
           ttcpip
            opiodr
             kpoal8
              SELECT FETCH:
               GROUP BY SORT: Fetch
                VIEW: Fetch
                 UNION-ALL: Fetch
                * FILTER DEFINITION: FetchOutside
                   UNION-ALL: RowProcedure
                    VIEW: RowProcedure
                     qesaAggNonDistSS.
                      evaopn2
                       evalen
                        lxsCntChar

The FILTER rowsources (not talking about filter predicates on normal rowsources here) can make things more complicated though as they can introduce their own logic and loops into the execution plan (for running correlated subqueries etc).

By the way, see what happens when I run exactly the same query with rowsource level statistics collection enabled:

SQL> alter session set statistics_level=all;

Session altered.

SQL> select avg(length(text)) from dba_source where owner = 'SYS';

AVG(LENGTH(TEXT))
-----------------
       125.032127


$ pstack 23740 | ./os_explain -a
   main
    ssthrdmain
     opimai_real
      sou2o
       opidrv
        opiodr
         opiino
          opitsk
           ttcpip
            opiodr
             kpoal8
              SELECT FETCH:
               QUERY EXECUTION STATISTICS: Fetch
                GROUP BY SORT: Fetch
                 QUERY EXECUTION STATISTICS: Fetch
                  VIEW: Fetch
                   QUERY EXECUTION STATISTICS: Fetch
                    UNION-ALL: Fetch
                     QUERY EXECUTION STATISTICS: Fetch
                    * QUERY EXECUTION STATISTICS: Fetch
                       FILTER DEFINITION: FetchOutside
                        QUERY EXECUTION STATISTICS: Fetch
                         NESTED LOOP JOIN: Fetch
                          QUERY EXECUTION STATISTICS: Fetch
                           QUERY EXECUTION STATISTICS: SnapStats
                            sltrgftime64
                             gettimeofday
                              __kernel_vsyscall

Every rowsource is wrapped into a QUERY EXECUTION STATISTICS wrapper, which' task is just to count the number of rows sent "up" to parents in the tree, logical IOs and also rowsource timing info whenever an internal counter (set by _rowsource_statistics_sampfreq parameter) wraps.

This is just an intro to Oracle execution plan internals and troubleshooting. Hopefully you don't need this technique too often, however it has helped me to successfully pinpoint the root cause of a problem in few non-trivial database problems.

Note that in Oracle 11g there's an excellent new feature called Real-time SQL Monitoring. It allows you to monitor the progress of currently running SQL statements. For serially running statements the monitoring kicks in for a statement after it's used total 5 seconds of CPU or IO time (this time is controlled by _sqlmon_threshold parameter, but for PX the monitoring is always enabled). After that you can query V$SQL_MONITOR and V$SQL_PLAN_MONITOR for seeing how much time/rows/LIOs that session has spent executing a statement. You can see these details even at the SQL execution plan line level. Alternatively you can use DBMS_SQLTUNE. REPORT_SQL_MONITOR function to get this info nicely formatted. Greg Rahn has written a good blog entry about it.

However it's important to note that both DBMS_SQLTUNE and V$SQL_MONITOR/V$SQL_PLAN_MONITOR use requires you to have Oracle Tuning Pack license which in turn requires Oracle Diagnostic Pack license. Details are in Oracle 11g Licensing Guide.

So another, low level approach for real-time monitoring will still be handy even after 11g becomes mainstream. In a future post I will be showing how to measure the progress and get execution profile of your plan by aggregating multiple stack traces and also some cool opportunities with DTrace.

In Oracle, latches and mutexes are different things and managed using different modules. KSL* modules for latches and KGX* for mutexes.

General mutex operatins require less CPU instructions than latch operations (as they aren't as sophisticated as latches and don't maintain get/miss counts as latches do).

But the main scalability benefit comes from that there's a mutex structure in each child cursor handle and the mutex itself acts as cursor pin structure. So if you have a cursor open (or cached in session cursor cache) you don't need to get the library cache latch (which was previously needed for changing cursor pin status), but you can modify the cursor's mutex refcount directly (with help of pointers in open cursor state area in sessions UGA).

Therefore you have much higher scalability when pinning/unpinning cursors (no library cache/library cache pin latching needed, virtually no false contention) and no separate pin structures need to be allocated/maintained.

Few notes:

1. library cache latching is still needed for parsing etc, the mutexes address only the pinning issue in library cache
2. mutexes are currently used for library cache cursors (not other objects like PL/SQL stored procs, table defs etc)
3. As mutexes are a generic mechanism (not library cache specific) they're used in V$SQLSTATS underlying structures too
4. When mutexes are enabled, you won't see cursor pins from X$KGLPN anymore (as X$KGLPN is a fixed table based on the KGL pin array - which wouldn't be used for cursors anymore)

Strongly in C, C++ and should preferably know Java. Should be very comfortable with Operating System Internals (preferably Linux/Unix flavours)

Salary : Excellent compensation.

Event will be held in Delhi and Bangalore.

Job Locations : Bangalore

Note: Locations are subject to change.

Last Date to Apply 30/06/2008

Click Here for More Details

Nope!

Direct SGA access has excellent troubleshooting potential (as long as you know the shared memory data structures), but it has one major drawback - very few companies have such tools already in place in their production systems.

I have occasionally been called in to solve an urgent performance problem, happening right now and it needs solving immediately! And did I mention, these are critical production systems. Where you can't just install binary executables freshly downloaded off internet. In fact you would want to diagnose the issue with minimal impact and changes required to those production environments (and that leaves sql tracing out the first round troubleshooting tools for me as well!)

So, I've developed myself a toolset for such purpose, Snapper and sw.sql and some process stack reading techniques I already have introduced in my blog.

Next in line is waitprof.sql which is a high-frequency V$SESSION_WAIT sampler - implemented in plain SQL (not PL/SQL).

Waitprof is basically a sampling session wait profiler. It's like running a select against V$SESSION_WAIT in a very tight loop and aggregating results - but I have used a trick to do all this in plain SQL, which gives me performance advantage over PL/SQL based loops. Waitprof is able to sample V$SESSION_WAIT for a session up to 100 000 times per second!

This depends on your hardware of course and Oracle version too, but normally you'll get 50-70kHz sampling rate with it.

Ok, you want to see an example? ;-)

I will run my lotspios.sql script in one session (SID=142), which, as the name says, generates lots of physical IOs.

SQL> @lotspios 100

Now, let say I'm interested what this session is doing over time (is it on CPU or waiting and if waiting, on what). For that I run waitprof.sql on that session. The syntax is following:

@waitprof    

And lets run waitprof now, on SID 142, gather only events waited on and take 100000 samples:

SQL> @waitprof noprint 142 e 100000

-- WaitProf 1.03 by Tanel Poder ( http://www.tanelpoder.com )

                                                                               % Total  Total Event   Distinct   Avg time
    SID STATE   EVENT                               P1                            Time      Time ms     Events   ms/Event
------- ------- ----------------------------------- -------------------------- ------- ------------ ---------- ----------
    142 WAITING db file scattered read                                           39.75      580.306        721       .805
    142 WAITING gc buffer busy                                                   30.90      451.198        485       .930
    142 WORKING On CPU / runqueue                                                17.42      254.347       1786       .142
    142 WAITING read by other session                                            10.35      151.081        313       .483
    142 WAITING db file sequential read                                           1.56       22.834        244       .094
    142 WAITING latch: cache buffers chains                                        .02         .234          9       .026

6 rows selected.

Elapsed: 00:00:01.48

From the %Total Time column you see that during sampling that session stats, most of the time (39.75%) it was waiting for db file scattered read event. This roughly accounts for 580 milliseconds.

The Distinct Events column shows the number of distinct occurrences of that event, so during the 1.48 second sampling period 721 waits on db file scattered read was done. This column is based on V$SESSION_WAIT.SEQ# so it's reasonably accurate over short periods of time. And the Avg time ms/Event column shows average duration for a single event, so an db file scattered read in this example took 805 microseconds.

This is the simplest usage of waitprof, however this is not why I wrote it. I want to know the details of those waits too!

The what_to_sample parameter can accept any combination of following values:

• e - sample event names (this is always done automatically)
• 1 - sample P1 values
• 2 - sample P2 values
• 3 - sample P3 values
• s - sample SEQ# values

So, I can run waitprof with e1 to get all wait event parameter details:

SQL> @waitprof noprint 142 e1 100000

-- WaitProf 1.03 by Tanel Poder ( http://www.tanelpoder.com )

                                                                               % Total  Total Event   Distinct   Avg time
    SID STATE   EVENT                               P1                            Time      Time ms     Events   ms/Event
------- ------- ----------------------------------- -------------------------- ------- ------------ ---------- ----------
    142 WAITING db file scattered read              file#= 6                     49.07      814.496       1057       .771
    142 WAITING gc buffer busy                      file#= 6                     23.99      398.151        437       .911
    142 WORKING On CPU / runqueue                                                20.38      338.358       1895       .179
    142 WAITING read by other session               file#= 6                      5.72       94.985        210       .452
    142 WAITING db file sequential read             file#= 6                       .83       13.712        159       .086
    142 WAITING latch: cache buffers chains         address= 422E843C              .01         .083          1       .083
    142 WAITING latch: cache buffers chains         address= 41F8E900              .00         .066          1       .066
    142 WAITING latch: cache buffers chains         address= 437FCEC4              .00         .017          1       .017
    142 WAITING latch: cache buffers chains         address= 4238EE2C              .00         .017          1       .017
    142 WAITING latch: cache buffers chains         address= 42342298              .00         .017          1       .017
    142 WAITING latch: cache buffers chains         address= 41FBF44C              .00         .017          1       .017
    142 WAITING latch: cache buffers chains         address= 41FB4998              .00         .017          1       .017
    142 WAITING latch: cache buffers chains         address= 41FACB6C              .00         .017          1       .017
    142 WAITING latch: cache buffers chains         address= 41F7D074              .00         .017          1       .017
    142 WAITING latch: cache buffers chains         address= 00000006              .00         .017          1       .017
    142 WAITING latch: cache buffers chains         address= 41FA98F4              .00         .017          1       .017

16 rows selected.

Elapsed: 00:00:01.70

So now I have the wait profile breakdown by event and its P1 value. For buffer related events we see that all IOs happened against datafile 6.

For latch waits, the P1 value is the latch address. This is especially helpful in pre-10g databases as it helps us to translate this address back to the actual latch name (and even to individual child latch) using V$LATCH_PARENT and V$LATCH_CHILDREN.

I have written a script la.sql for this:

SQL> @la 422E843C

ADDR         LATCH#   CHLD NAME                                           GETS      IGETS     MISSES    IMISSES     SLEEPS  WAIT_TIME
-------- ---------- ------ ---------------------------------------- ---------- ---------- ---------- ---------- ---------- ----------
422E843C        122   1735 cache buffers chains                         164637      64387        329          0          6        167

Waitprof allows you to use the what_to_sample parameters in any combination, for example, if I only want to know the breakdown of scattered reads by number of blocks read at a time (P3), I can use "3" in the parameter as seen below. Note that as P2, P3 and SEQ# are not printed with noprint option (for shorter line width), then now we need to use print option:

SQL> @waitprof print 142 e3 100000

-- WaitProf 1.03 by Tanel Poder ( http://www.tanelpoder.com )

                                                                                                                          % Total  Total Event   Distinct   Avg time
    SID STATE   EVENT                               P1                         P2              P3                    SEQ#    Time      Time ms     Events   ms/Event
------- ------- ----------------------------------- -------------------------- --------------- --------------- ---------- ------- ------------ ---------- ----------
    142 WORKING On CPU / runqueue                                                                                           36.03      544.053       1211       .449
    142 WAITING gc buffer busy                                                                 id#= 65537                   28.73      433.838        393      1.104
    142 WAITING db file scattered read                                                         blocks= 16                   22.39      338.149        270      1.252
    142 WAITING read by other session                                                          class#= 1                     7.61      114.866        340       .338
    142 WAITING db file scattered read                                                         blocks= 13                    1.16       17.546         20       .877
    142 WAITING db file scattered read                                                         blocks= 12                    1.03       15.568         16       .973
    142 WAITING db file scattered read                                                         blocks= 14                     .49        7.459          7      1.066
    142 WAITING db file scattered read                                                         blocks= 11                     .46        6.976          9       .775
    142 WAITING db file scattered read                                                         blocks= 10                     .36        5.361          7       .766
    142 WAITING db file scattered read                                                         blocks= 8                      .31        4.696          7       .671
    142 WAITING db file sequential read                                                        blocks= 1                      .27        4.077         47       .087
    142 WAITING db file scattered read                                                         blocks= 9                      .27        4.062          6       .677
    142 WAITING db file scattered read                                                         blocks= 3                      .16        2.341         11       .213
    142 WAITING db file scattered read                                                         blocks= 2                      .15        2.235         14       .160
    142 WAITING db file scattered read                                                         blocks= 15                     .15        2.235          2      1.117
    142 WAITING db file scattered read                                                         blocks= 4                      .13        1.933          6       .322
    142 WAITING db file scattered read                                                         blocks= 6                      .12        1.737          5       .347
    142 WAITING db file scattered read                                                         blocks= 5                      .10        1.495          4       .374
    142 WAITING db file scattered read                                                         blocks= 7                      .07        1.027          2       .513
    142 WAITING read by other session                                                          class#= 65537                  .01         .136          9       .015
    142 WAITING gc buffer busy                                                                 id#= 1                         .01         .106          7       .015
    142 WAITING latch: cache buffers chains                                                    tries= 0                       .00         .045          2       .023
    142 WAITING read by other session                                                          class#= 0                      .00         .015          1       .015
    142 WAITING db file sequential read                                                        blocks= 2                      .00         .015          1       .015
    142 WAITING gc buffer busy                                                                 id#= 5                         .00         .015          1       .015
    142 WAITING db file scattered read                                                         blocks= 1                      .00         .015          1       .015

26 rows selected.

Elapsed: 00:00:01.89

From above we see that as far as IO operations are concerned, most of the response time during sampling was spent during 16/13/12 block reads. Other read sizes took less time.

Of course we can also make Waitprof show all P1,P2,P3 colums, but this may display lots of lines for an active session:

SQL> @waitprof print 142 e123 100000

-- WaitProf 1.03 by Tanel Poder ( http://www.tanelpoder.com )

                                                                                                                          % Total  Total Event   Distinct   Avg time
    SID STATE   EVENT                               P1                         P2              P3                    SEQ#    Time      Time ms     Events   ms/Event
------- ------- ----------------------------------- -------------------------- --------------- --------------- ---------- ------- ------------ ---------- ----------
    142 WORKING On CPU / runqueue                                                                                           21.63      424.007       1970       .215
    142 WAITING db file scattered read              file#= 1                   block#= 43525   blocks= 16                     .16        3.175          1      3.175
    142 WAITING db file scattered read              file#= 1                   block#= 43541   blocks= 12                     .13        2.587          1      2.587
    142 WAITING db file scattered read              file#= 1                   block#= 43493   blocks= 16                     .11        2.234          1      2.234
    142 WAITING db file scattered read              file#= 1                   block#= 44167   blocks= 16                     .10        2.019          1      2.019
    142 WAITING db file scattered read              file#= 1                   block#= 55466   blocks= 16                     .10        1.901          2       .951
    142 WAITING db file scattered read              file#= 1                   block#= 44041   blocks= 16                     .10        1.882          1      1.882
    142 WAITING db file scattered read              file#= 1                   block#= 43877   blocks= 16                     .09        1.705          1      1.705
    142 WAITING db file scattered read              file#= 1                   block#= 44067   blocks= 16                     .09        1.686          1      1.686
    142 WAITING db file scattered read              file#= 1                   block#= 44103   blocks= 16                     .08        1.646          1      1.646
    142 WAITING db file scattered read              file#= 1                   block#= 44135   blocks= 16                     .08        1.568          1      1.568
    142 WAITING db file scattered read              file#= 1                   block#= 43597   blocks= 16                     .08        1.509          1      1.509
    142 WAITING db file scattered read              file#= 1                   block#= 8696    blocks= 16                     .08        1.490          1      1.490
[...lots of lines snipped...]
    142 WAITING db file sequential read             file#= 1                   block#= 35630   blocks= 1                      .00         .020          1       .020
    142 WAITING db file sequential read             file#= 1                   block#= 35634   blocks= 1                      .00         .020          1       .020
    142 WAITING db file sequential read             file#= 1                   block#= 35636   blocks= 1                      .00         .020          1       .020
    142 WAITING db file sequential read             file#= 1                   block#= 35638   blocks= 1                      .00         .020          1       .020
    142 WAITING db file sequential read             file#= 1                   block#= 35640   blocks= 1                      .00         .020          1       .020
    142 WAITING db file sequential read             file#= 1                   block#= 35644   blocks= 1                      .00         .020          1       .020
    142 WAITING db file sequential read             file#= 1                   block#= 38097   blocks= 16                     .00         .020          1       .020
    142 WAITING db file sequential read             file#= 1                   block#= 43562   blocks= 1                      .00         .020          1       .020
    142 WAITING db file scattered read              file#= 1                   block#= 35529   blocks= 16                     .00         .020          1       .020

1979 rows selected.

Even though the CPU usage is in the top of the list, it has only used 21.63% of total response time, so the rest 78% of the IO time is the "problem" here. The reason is that all CPU time is aggregated together (the time between waits), while most events are brought out separately due differences in P1,P2,P3 values.

If you want to see events in chronological order, then you can include SEQ# (which is the wait state sequence identifier) also in the output:

SQL> @waitprof print 142 e123s 1000

-- WaitProf 1.03 by Tanel Poder ( http://www.tanelpoder.com )

                                                                                                                          % Total  Total Event   Distinct   Avg time
    SID STATE   EVENT                               P1                         P2              P3                    SEQ#    Time      Time ms     Events   ms/Event
------- ------- ----------------------------------- -------------------------- --------------- --------------- ---------- ------- ------------ ---------- ----------
    142 WAITING db file scattered read              file#= 6                   block#= 43897   blocks= 13           26036    1.80         .360          1       .360
    142 WORKING On CPU / runqueue                                                                                   26036    2.40         .480          1       .480
    142 WAITING db file scattered read              file#= 6                   block#= 44105   blocks= 13           26037    5.30        1.060          1      1.060
    142 WORKING On CPU / runqueue                                                                                   26037    1.40         .280          1       .280
    142 WAITING db file scattered read              file#= 6                   block#= 44314   blocks= 12           26038    5.90        1.180          1      1.180
    142 WORKING On CPU / runqueue                                                                                   26038    1.70         .340          1       .340
    142 WORKING On CPU / runqueue                                                                                   26039    1.80         .360          1       .360
    142 WAITING db file scattered read              file#= 6                   block#= 44534   blocks= 13           26039    6.40        1.280          1      1.280
    142 WORKING On CPU / runqueue                                                                                   26040    1.80         .360          1       .360
    142 WAITING db file scattered read              file#= 6                   block#= 44742   blocks= 13           26040    6.20        1.240          1      1.240
    142 WORKING On CPU / runqueue                                                                                   26041    1.70         .340          1       .340
    142 WAITING db file scattered read              file#= 6                   block#= 44950   blocks= 13           26041    6.00        1.200          1      1.200
    142 WAITING db file scattered read              file#= 6                   block#= 45159   blocks= 12           26042    5.60        1.120          1      1.120
    142 WORKING On CPU / runqueue                                                                                   26042    2.10         .420          1       .420
    142 WAITING db file scattered read              file#= 6                   block#= 45366   blocks= 13           26043    6.00        1.200          1      1.200
    142 WORKING On CPU / runqueue                                                                                   26043    1.70         .340          1       .340
    142 WAITING db file scattered read              file#= 6                   block#= 45574   blocks= 13           26044    5.90        1.180          1      1.180
    142 WORKING On CPU / runqueue                                                                                   26044    1.80         .360          1       .360
    142 WAITING db file scattered read              file#= 6                   block#= 45769   blocks= 13           26045    5.90        1.180          1      1.180
    142 WORKING On CPU / runqueue                                                                                   26045    1.60         .320          1       .320
    142 WORKING On CPU / runqueue                                                                                   26046    1.60         .320          1       .320
    142 WAITING db file scattered read              file#= 6                   block#= 45978   blocks= 12           26046    5.50        1.100          1      1.100
    142 WAITING db file scattered read              file#= 6                   block#= 46185   blocks= 13           26047    6.00        1.200          1      1.200
    142 WORKING On CPU / runqueue                                                                                   26047    1.80         .360          1       .360
    142 WAITING db file scattered read              file#= 6                   block#= 46380   blocks= 13           26048    6.10        1.220          1      1.220
    142 WORKING On CPU / runqueue                                                                                   26048    1.70         .340          1       .340
    142 WAITING db file scattered read              file#= 6                   block#= 46601   blocks= 13           26049    4.30         .860          1       .860

27 rows selected.

Here you see a detailed chronological view of events, ordered by their SEQ# number. There are few gotchas like SEQ# wrapping to zero after reaching 65535 and potentially missed events due sampling granularity, but more about this in an upcoming post. Btw, you can get more details about script usage from the script header and by reading the single SQL statement in there :)

One might ask why use such sampling script when you can enable extended SQL trace which also gives you chronological ordering and all wait parameters. One answer is that it's more convenient to use a script for first-round diagnosis of performance problems. Another answer is that sometimes, in change-controlled production environments it may take quite some time to get the change for enabling trace through.

In an upcoming post I'll show more (very cool) uses of this powerful sampling technique, a hint is that the script is called Latchprof ;-)

If you haven't read my other Advanced Oracle Troubleshooting posts, here they are:

http://blog.tanelpoder.com/2007/06/18/advanced-oracle-troubleshooting-guide-when-the-wait-interface-is-not-enough-part-1/
http://blog.tanelpoder.com/2007/08/27/advanced-oracle-troubleshooting-guide-part-2-no-magic-is-needed-systematic-approach-will-do/
http://blog.tanelpoder.com/2007/09/06/advanced-oracle-troubleshooting-guide-part-3-more-adventures-in-process-stack/
http://blog.tanelpoder.com/2008/06/03/advanced-oracle-troubleshooting-guide-part-4-diagnosing-a-long-parsing-issue/

Meanwhile, can someone explain how my script can sample the same table (V$SESSION_WAIT) hundreds of thousands of times by running a single SQL statement only once? ;-)

Classic tracing + tkprof techniques didn't show much (just some recursive queries consuming insignificant amounts of time).

The proven V$SESSION_WAIT sampling technique didn't reveal anything as it showed the session being constantly on CPU (the wait state = 'WAITED KNOWN TIME' which means session is on CPU) and SEQ# didn't increase (which means that wait state did not change over time).

Due the symptoms described above I was well prepared to troubleshoot this issue. This looks exactly like one of the troubleshooting use cases I demonstrate in of my Advanced Oracle Troubleshooting class (nice embedded advertisment, huh? ;)

In such a case where tracing and V$ views don't provide any useful information about what the session is doing, I normally look into few stack traces of the server process. In this case I asked the poster to do this and here is the result:

21820: oracleXXXXXX (DESCRIPT