Search

Top 60 Oracle Blogs

Recent comments

Oakies Blog Aggregator

How Does An Execution Plan Suddenly Change When The Statistics (And Everything Else) Remains The Same ? (In Limbo)

I’ve slipped this post in as there have been a number of discussions recently on how execution plans have changed while nothing else appears to have changed in the database. How can an execution plan suddenly change when no one has made any changes to the database ?   By no changes, it means that there [...]

New versions of LatchProf and LatchProfX for latch contention troubleshooting and tuning

The LatchProf and LatchProfX scripts allow you to be more systematic with latch contention troubleshooting and tuning. No more guesswork is needed as these scripts give you exact session IDs and in this version also SQLIDs of the troublemaking applications.

You can download the new versions here:

Example output (with SQLID info) is below:

SQL> @latchprof name,sid,sqlid % % 100000

-- LatchProf 1.21 by Tanel Poder ( http://www.tanelpoder.com )

NAME                                       SID SQLID               Held       Gets  Held %     Held ms Avg hold ms
----------------------------------- ---------- ------------- ---------- ---------- ------- ----------- -----------
cache buffers chains                       133 3jbwa65aqmkvm        349        348     .35      14.169        .041
simulator lru latch                        133 3jbwa65aqmkvm         51         51     .05       2.071        .041
row cache objects                          133 3jbwa65aqmkvm          5          5     .01        .203        .041
cache buffers chains                        24                        5          5     .01        .203        .041
cache buffers chains                       149 3jbwa65aqmkvm          3          3     .00        .122        .041
resmgr group change latch                   33 147a57cxq3w5y          2          2     .00        .081        .041
cache buffers chains                         9 5raw2bzx227wp          2          1     .00        .081        .081
In memory undo latch                       149 f3y38zthh270n          2          1     .00        .081        .081
active checkpoint queue latch                5                        2          1     .00        .081        .081
cache buffers chains                       149 75621g9y3xmvd          2          2     .00        .081        .041
cache buffers chains                         9 gvgdv2v90wfa7          2          2     .00        .081        .041
cache buffers chains                        33 75621g9y3xmvd          2          2     .00        .081        .041
checkpoint queue latch                       5                        1          1     .00        .041        .041
ksuosstats global area                       8                        1          1     .00        .041        .041
cache buffers lru chain                    133 3jbwa65aqmkvm          1          1     .00        .041        .041
multiblock read objects                    155 75ju2gn3s8009          1          1     .00        .041        .041
resmgr group change latch                    9 0w2qpuc6u2zsp          1          1     .00        .041        .041
resmgr group change latch                   33 apgb2g9q2zjh1          1          1     .00        .041        .041
resmgr group change latch                  133 apgb2g9q2zjh1          1          1     .00        .041        .041
space background task latch                 17                        1          1     .00        .041        .041
cache buffers chains                       149 5raw2bzx227wp          1          1     .00        .041        .041
cache buffers chains                        33 5raw2bzx227wp          1          1     .00        .041        .041
cache buffers chains                        33 05s4vdwsf5802          1          1     .00        .041        .041
cache buffers chains                        31 0yas01u2p9ch4          1          1     .00        .041        .041
cache buffers chains                         9 41zu158rqf4kf          1          1     .00        .041        .041
In memory undo latch                        33 0bzhqhhj9mpaa          1          1     .00        .041        .041
In memory undo latch                        31 gvgdv2v90wfa7          1          1     .00        .041        .041
In memory undo latch                         9 gvgdv2v90wfa7          1          1     .00        .041        .041
simulator lru latch                        149 3jbwa65aqmkvm          1          1     .00        .041        .041
row cache objects                          133 5yq51dtyc6qf2          1          1     .00        .041        .041
SQL memory manager workarea list la        133 3jbwa65aqmkvm          1          1     .00        .041        .041
enqueues                                   141                        1          1     .00        .041        .041
row cache objects                          132                        1          1     .00        .041        .041

33 rows selected.

LatchProf scripts allow you to easily identify which session and SQLID (or sqlhash in 9i) cause the latch(es) to be held the most.

Let’s check what’s the most “latch-holding” SQL reported by LatchProf:

SQL> @sqlid 3jbwa65aqmkvm

HASH_VALUE  CH# SQL_TEXT
---------- ---- ------------------------------------------------------------------------------------------------------------------------------------------------------
1432996723    0 SELECT O.ORDER_ID, LINE_ITEM_ID, PRODUCT_ID, UNIT_PRICE, QUANTITY, ORDER_MODE, ORDER_STATUS, ORDER_TOTAL, SALES_REP_ID, PROMOTION_ID, C.CUSTOMER_ID,
 CUST_FIRST_NAME, CUST_LAST_NAME, CREDIT_LIMIT, CUST_EMAIL, ORDER_DATE FROM ORDERS O , ORDER_ITEMS OI, CUSTOMERS C WHERE O.ORDER_ID = OI.ORDER_ID AND
 O.CUSTOMER_ID = C.CUSTOMER_ID AND O.ORDER_STATUS <= 4

If you want to read more about the capabilities of LatchProf and LatchProfX, go here:

Note that the latest version (v1.21) also fixes a problem with Oracle 11.2 where the script execution plan order was wrong, causing the sampling to not happen in correct order. I added a NO_TRANSFORM_DISTINCT_AGG hint to disable a new transformation happening in 11.2 to make the scripts behave correctly…

Share/Bookmark

Oracle Troubleshooting: How to read Oracle ERRORSTACK output?!

I have written the first article to the troubleshooting section of my new website tech.E2SN.com:

It’s about a very valuable Oracle troubleshooting tool -> ERRORSTACK trace.

I cover 4 frequently asked questions there:

  1. Reading the current executing SQL statement text from errorstack trace
  2. Reading the current executing PL/SQL package and PL/SQL source code line number from errorstack trace
  3. Reading the current bind variable values from errostack trace
  4. Identifying how much private (UGA) memory a cursor is using

You can read it here:

By the way, if you like my new website, feel free to link to it !!! ;-)

Share/Bookmark

Hello world!

Welcome to my personal blog.  Dr. Paranoid is not a medical site nor a site for those of us with conspiracy theories.  I know there are many DrParanoids out there but in the Oracle Data Guard world, it’s what I get called a lot so it stuck.

I will discuss all things related to Oracle’s Data Guard and Maximum Availability Architecture with forays into High Availability, Disaster Recovery and Business Continuity.

To get you introduced to Data Guard, what it is and how it works,  I was going to start writing but then I thought, why not show you all Chapter One of our book.    Once you’ve read that you’ll be ready to read some more.  Yes, this is a shameless plug for the book!

Larry

Future appearances, conferences and seminars

Just to let you know where I’ll be in the near future :)

Seminars

Advanced Oracle Troubleshooting 2.0

I have rearranged and adjusted the material so it flows smoother, has even more practical tools and scripts and describes some internals even deeper ;-)

These dates are close, last chance to register ;-) Note that after these I won’t be doing an Advanced Oracle Troubleshooting class in US for a while…

Conferences

Hotsos Symposium, Dallas, TX, 8-11 March

I will deliver 2 presentations at the Hotsos Symposium 2010

  • Back to Basics: Choosing the Starting Point of Performance Tuning and Troubleshooting Wisely
  • Understanding LGWR, log file sync Waits and Commit Performance

Also, I will be the Training Day speaker after the conference (I feel quite honored about this btw ;)

  • Scripts and Tools for Oracle Troubleshooting and Advanced Performance Analysis

Note that I will show some very cool (and of course, useful) stuff at the Training Day! Some things are gonna be so cool that I almost want to sit in the audience myself! One of the things is called MOATS – the Mother Of All Tuning Scripts for example ;-)

Miracle OpenWorld 2010, Denmark, 14-16 April

I will deliver two sessions at MOW

  • Understanding LGWR, Log file sync Waits and Commit Performance
  • Oracle Wait Interface is Useless (sometimes) – together with  James Morle

It will be fun! :)

IOUG Collaborate, Las Vegas, NV, 19-22 April

Does “What happens in Vegas…” also apply to all the new stuff learnt at a conference? ;-)

EMEA Harmony 2010, Tallinn, Estonia, 20-21 May

This event will be a blast – awesome speakers (Steven Feuerstein, Tom Kyte, C.J. Date among others) and for the first time the Estonian,Latvian,Finnish and Russian OUG organize such event together (I’m from Estonia originally who don’t know that)

I will deliver a keynote there and one more session:

  • Back to Basics: Choosing the Starting Point of Performance Tuning and Troubleshooting Wisely (keynote)
  • Understanding LGWR, log file sync Waits and Commit Performance

Michigan OakTable Symposium, Ann Arbor, MI, 16-17 September

This will be an awesome event, lots of OakTable people there, including me!

  • Back to Basics: Choosing the Starting Point of Performance Tuning and Troubleshooting Wisely
  • Understanding LGWR, log file sync Waits and Commit Performance

I think we’ll even have a chance to see some football there…

Share/Bookmark

Using Process Memory Matrix script for calculating Oracle process memory usage on Solaris

I just published a new script and article about calculating the real Oracle process memory usage on Solaris.

The problem with V$PROCESS* views (and the V$SESSTAT) is that they will tell you what Oracle thinks it’s using, not the real amount of memory used. There will be discrepancies due how memory is actually allocated in OS, libraries out of Oracle’s control, the static memory areas inside Oracle binary and of course bugs.

I was working on one of such problems and decided to make my life easier by writing the script. It’s not so much about calculating the exact figures (they will never be 100% correct), but more about presenting the memory usage data in a better and more convenient fashion.

The script is called procmm and stands for Process Memory Matrix as it shows the memory usage in a matrix grid.

Here’s an example output to show what I’m talking about:

oracle@solaris02:~/research/memory$ ./procmm.sh -t `pgrep -f ora_.*SOL102`

-- procmm.sh: Process Memory Matrix v1.01 by Tanel Poder ( http://tech.e2sn.com )
-- All numbers are shown in kilobytes

Total PIDs 17, working: .................

PID            SEGMENT_TYPE      VIRTUAL          RSS         ANON       LOCKED    SWAP_RSVD
------ -------------------- ------------ ------------ ------------ ------------ ------------
0                       lib       389844       388796        13180            0        17816
0                    oracle      1629064      1628908         3336            0        42012
0            ism_shmid=0x1d      6963336      6963336            0      6963336            0
0             hc_SOL102.dat           48           48            0            0            0
0                      anon        32936        15936        15452            0        32868
0                     stack         1660         1628         1592            0         1660
0                      heap        37004        18016        16844            0        37004
------ -------------------- ------------ ------------ ------------ ------------ ------------
0                 TOTAL(kB)      9053892      9016668        50404      6963336       131360

And here’s the full article and in there a link to the script:

Comments are welcome here, as I haven’t set up commenting on my other site yet…

Share/Bookmark

Oracle Flash Cache on Linux with Exadata

cool stuff:
Guy Harrison explains how to set up flash caching on Linux for Oracle without Exatata
Late he give some analysis on performance of no flash, flash cache and flash tablespae

http://guyharrison.squarespace.com/blog/2010/1/24/flash-tablespace-vs-db-flash-cache.html

SQL Plan visualization by Tanel Poder

Check this out - SQL Plan visualization by Tanel Poder:


Thinking Clearly About Performance

I’ve posted a new paper at method-r.com called “Thinking Clearly About Performance.” It’s a topic I’ll be presenting this year at:

The paper is only 13 pages long, and I think you’ll be pleased with its information density. Here is the table of contents:

  1. An Axiomatic Approach
  2. What is Performance?
  3. Response Time vs Throughput
  4. Percentile Specifications
  5. Problem Diagnosis
  6. The Sequence Diagram
  7. The Profile
  8. Amdahl’s Law
  9. Minimizing Risk
  10. Load
  11. Queueing Delay
  12. The Knee
  13. Relevance of the Knee
  14. Capacity Planning
  15. Random Arrivals
  16. Coherency Delay
  17. Performance Testing
  18. Measuring
  19. Performance is a Feature

As usual, I learned a lot writing it. I hope you’ll find it to be a useful distillation of how performance works.

Deliberate Practice

Recently I did some soul searching about my expertise as a DBA. I am not talking about my knowledge, my talents and my work style. I’m talking about which things I’m really comfortable doing. The commands I know by heart, the issues I ran into so often that I can diagnose with the tiniest clues.

There are definitely things I’m very good at. Diagnosing why RAC crashed or wouldn’t start. Solving a range of different problems with Streams. User managed recoveries. Netapp. BASH. Top. sar. vmstat. Redo log mining. Datapump. ASH and its relatives AWR and ADDM. Using Snapper to work with wait event interface. SQL coding, network diagnosis, Patching.

These are mostly things I do every day or close enough to it that the commands, the techniques, the traps and the limitations are always clear in my mind. But there are things that I do rarely or even never. This are important DBA skills, some are even very basic, which I do not have because they are not very useful in my specific position.

These include RMAN, ASM, Dataguard, AWK, perl, python, PL/SQL, tracing, SQL tuning, upgrade testing, benchmarks, many Linux administration tools, hadoop and those new NoSQL things, MySQL, Amazon’s cloud databases, RAT, partitions, scheduler.

These are all things that I know something about, that I’ve read about – but I can’t say I’m confident with any of these because I simply haven’t played with them all that much. After all, you learn by doing and running into issues – not by reading people say how everything works perfectly when they use it.

In order to widen my skill set a bit, I’m planning to take time this year to deliberately practice some of the technologies I didn’t use much last year. I’m thinking of taking anything from few weeks to few month per technology. I’ll invent and look up “lab exercises” for the specific topic and then proceed to spend the month practicing. Think of it as a “poor DBA’s Oracle University”.

This is the opposite of what I’ve been doing until now which can be described as “read Oracle’s Concepts book”. Reading the concepts book is great, but I’m at a point where I feel that I know tons of theory and need to spend some quality time grappling with its various applications. There seems to be a lot of research that shoes that best way to become an expert is with deliberate practice, so this year – I’ll practice.

Unless I’ll run into something really exciting, I don’t expect to blog about this adventure. After all, if I start posting scores of trivial AWK scripts, I doubt if anyone will keep reading my blog. But I thought that maybe some of my readers will enjoy joining me in my practice. So I opened this mailing list.

If you are also interested in practicing with me, feel free to join the mailing list. I’ll announce a “topic of the month”. This month its AWK, next month probably RMAN, we’ll see how this goes. I’ll send to the list the questions I’m planning to solve. If you join, you can send in your own questions. We’ll send our answers to our own and others questions with a week delay (so everyone will have time to practice on his own first). We’ll discuss and compare our answers. We’ll cheer each other as we improve our skills. I will not try to sell you anything.

This is an obvious ploy. I need people on the list so I’ll be accountable to this practice. Otherwise I’ll probably forget it by next week. Feel free to join just to watch me stumble and remind me that this is what learning looks like and it will be worth it.

Here’s to a year of practice and hopefully better skills that will follow.