Oracle DB Development

The case for CASE

I've just uploaded my latest article. This time it is a look at the SQL keyword CASE. It's power is often not fully appreciated. It is much more flexible and powerful than you may expect from just taking a quick glance in the official documentation.

The article is located here.

Please post comments and questions here.

Supported version of skip locked?

As Pawel showed in his post some details about how the row scn technology works in specific situations isn't as well documented as you'd wish.

I asked Oracle development to clarify the questions I had and the answer back was both interesting and useful. Every kind of update and lock on a row will make the rowscn for the row/block NULL until the updating/locking transaction completes. That is, any kind of lock and update can be identified when retrieving data.

More interesting is that this can be used to implement "skip locked" which is a feature oracle AQ uses, but it is not supported for use by end users. For a table where rowdependencies is defined, this can now be implemented with "where ora_rowscn is NOT NULL". I don't know exactly how and when I'll use it, but it is good to have this tool when the situation arises. I'm sure it'll come in handy soon enough as discussions about the skip locked feature seems to occur almost on a monthly basis.

I'm on a long vacation so I cannot test this until I come home again. I had some time over this mornings, so I figured it would be a good time to show that the blog isn't completely dead, it's just not too active when I have to choose between talking with friends and family I haven't seen in a couple of years and researching Oracle technologies.

SQL Challenge I - Solution

Both Gary and Philip postd working versions. Gary's didn't work for me when I had the same letter more than one time in the string. Philips did work and the need to deal with each letter as if it is unique can of course be questioned. The intent was for a solution that could handle that and as Philip showed adjusting that is as easy as to just add a distinct.

My version is based on not using analytical functions of which I thought connect by was a part. Looking at it in another way, I wanted something that could be adopted to other databases.

My starting version of the solution would be:

   with v    as (select 'ABC' v from dual)
   ,base as (select substr(v.v,1,1) nv, 1 nn from v
             union all
             select substr(v.v,2,1) nv, 2 nn  from v
             union all
             select substr(v.v,3,1) nv, 3 nn  from v)
  select a.nv||b.nv||c.nv
    from base a, base b, base c
   where a.nn not in(b.nn, c.nn)
     and b.nn not in (c.nn)

What's not to like? I get to use one with construct that takes the output from another with construct as it's input.
The "base" with gives us a result table that has one letter from the input string on each row. I'm then joining that with itself three times (as I have three letter in the string) to get a cartesian product that holds all cominations. It has the valid combinations (where all three letters are used) and invalid ones (where some letters are used more than one time). The last thing is to remove the invalid combinations. I do this by checking if the number I assign to each value (nn) occurs more than one time in the result. For AAA we would have nn for table alias A, B, and C be 1. I only have to check if each value occurs later on in the column list as if it was equal to an earlier one, the test for that earlier one would detect that the same was used.

So it works… Interesting and satisfying at the same time. However, the union is not too lovable. How can we avoid it? Well, the easy way would of course be to use the connect by trick, but as I intended to not use it I'll have to cheat and use anoter unrelated table (view).

   with v    as (select 'ABC' v from dual)
      ,base as (select substr( v.v,rownum,1) nv
                      ,rownum nn from v, all_objects
                 where rownum <= length(v.v))

    select a.nv||b.nv||c.nv

     from base a

         ,base b

         ,base c

    where a.nn not in (b.nn, c.nn)

     and b.nn not in (c.nn) 

I'm usig all_objects which should have more rows than the length of a string you want all combinations. It works, but I still prefer the connect by version if I were to use it. This way it does satisfy the limitation I put up for this.

So while this works, it is not as dynamic as one could want it to be. You need a different SQL if you have a four letter string. It ieasy to change this for that, but it is still annoying to have to have many different ones. An alternative would be to write PL*SQL to generate the SQL and then execute it. While that may be nicer, I wanted a solution that was a single statement.

I did find a way to do it. I'm sure it can be done easier and nicer. Maybe analytic functions would make the SQL faster and easier to understand. However, This way combines XML, with clauses, and dynamic in-line queries. For fancy use of the Oracle database, it's almost optimal… How practical it would be to use this very often and with long strings would have to be tested before getting it into a production scenario.

   with v as (select 'ABC' v from dual)
      ,base as (select substr(v.v,rownum,1) nv
                      ,rownum nn
                  from v
                      ,all_objects
                 where rownum <= length(v.v))       ,sel  as (select 'select ' ||                        substr(max(sys_connect_by_path('a'||
                       nn||'.nv', '||')),3)||' data' a
                  from base
               connect by prior nn = nn - 1
                 start with nn = 1)
     ,frm  as (select 'from ' ||
                      substr(max(sys_connect_by_path(
                      'base a'||nn, ',')),2) a
                 from base
              connect by prior nn = nn - 1
                start with nn = 1)
     ,whr2  as (select 'a'||(nn-1)||'.nn not in('||
                       substr(max(sys_connect_by_path(
                       'a'||nn||'.nn', ', ')),2) ||
                       ')' cond
                      ,max(rownum) rn
                  from base
                      ,v
                 where nn > 1
               connect by prior nn = nn + 1
                 start with nn = length( v.v)
                 group by nn
                 order by nn)
     ,whr  as (select 'where ' || substr(max(
                      sys_connect_by_path(cond, ' and '))
                      ,5) a
                 from whr2
                     ,v
                where not rn = length(v.v)
              connect by prior rn = rn + 1
                start with rn = length( v.v) - 1)
  select extractvalue(t.column_value,'/DATA')
    from sel a
        ,frm b
        ,whr c
        ,xmltable(
          'for $root in $vals
           return $root/ROWSET/ROW/DATA'
           passing xmltype(dbms_xmlgen.getxml(
          'with v    as (select ''ABC'' v
                           from dual)
               ,base as (select substr(v.v,rownum,1) nv
                               ,rownum nn
                           from v
                               ,all_objects
                          where rownum <= length(v.v))'                              ||a.a||' '                              ||b.a||' '                              ||c.a)) as "vals") t; 

The formatting is a bit forced here as I need it to fit in the line size google allows on this blog. I will not explain in detail what I have tried to achieve in this SQL. If you would want me to talk about what I'm doing in a post, leave a comment and I'll try to write it up in a post soon.

In short, I'm generating different pieces of the SQL statement needed in the with clauses and them I'm using that to create the XML document with all these cominations ina single XML document. I'm then using some XML function and XMLTable with an XML Query to get it back out form one row (one XML doc) to multiple rows to get it back to relational data.

The threaad that started me thinking about this was this thread. There are some permutations things there that solves this with pure analytics. But why do this the easy way when you can make it more complicated. Laurent's solution is probably the only one that scales well with longer strings.

I hope you enjoyed this and hopefully even learned something. I know I did, both that Cartesian products can be used in ways I hadn't really thought of and that I need to learn more about analytical functions.

I also learned more about XML function and how to move data in and out of these dynamic in-line queries.

Dynamic in-line queries

I don't know of a better name for this feature. I think this name descibes the function well, even if it may sound overly ambitious. This whole post will be based on a post by Laurent Schneider where he uses XML features to do a select count(*) from different tables and return the result in one result table. It is like a auto generated number of selects that are unioned together.

You can find Laurent's post here. It's a wonderful piece of SQL to do something most of us would probably not have thought of using a single SQL statement for.

The version Laurent uses is:

   select table_name
        ,to_number(extractvalue(xmltype(
            dbms_xmlgen.getxml(
              'select count(*) c from '||table_name))
                  ,'/ROWSET/ROW/C')) count
    from user_tables;

Let's first review what happens in this query. The inner most function is a call to dbms.getxml with a query that will return one row and one value. This sql stateement is created on the fly based on the table name the outer select retrieves from user_tables. The function will of course turn the query result into XML text. The xmltype that wraps the function will convert the XML text into an XML object so XPATH functions can be used. The extractvalue function pulls out the node ROWSET/ROW/C where the two first are given as that is how getxml will return the data. C is the name of the column or in this case the alias assigned in the select statement as the resulting "column" from the count(*) function. The last thing that happens is that the extracted value is converted to a number and it is then returned to the outer select as the result from this function.

The query returns a result set like:

TABLE_NAME                      COUNT

Creating a feed with ORA-00600 errors

Instead of having to look at your different alert logs for ORA-00600 errors frequently, why not have them come to you? This post will, as promised, show the Perl code needed to publish a feed with ORA-00600 errors. This is building on the previous two posts that shows how to create an external table for the alert log and how to pull out all information for an ORA-00600 so each fetch retrieves one the data you're interested in.

The feed I'm using is a RSS 2.0 format. All feeds are fairly closely related so you can change it or provide different ones if you want to. You'll have to look up the different formats and then make the changes you want to this code.

The script I have here is a simplified version that removes all the usual dynamic nature and flexibility with parameters I usually have. This is done for brevity and to focus the script on the task at hand - delivering the ORA-00600 errors to your newsreader soon after they occur.

The script is not pretty and it could be written much nicer with use of modules such as XML::RSS. Unfortunately that module has a chain of dependencies that I didn't want to satisfy, so I'm creating the tags the old fashioned hard coded way. If you are to do a lot feeds with Perl, you will want to get that module installed.

!/usr/bin/perl
require v5.6;
use strict;
use CGI qw(:standard :html3 :netscape);
use CGI::Carp;
use DBI;
use DBD::Oracle qw(:ora_session_modes);

my ($ORACLE_HOME, $DBH);

if ( $ENV{TWO_TASK} ) { undef $ENV{TWO_TASK}; }

$ENV{"ORACLE_HOME"}='/path/to/oracle/install/dir';

my $statusRec;

print 'Content-Type: text/xml' . "\n\n";

print '<?xml version="1.0"?>' . "\n";

print '<rss version="2.0">' . "\n";

print '  <channel>' . "\n";

print '    <title>Oracle ORA-00600</title>' . "\n";

print '    <description>ORA 600 errors</description>' . "\n";

print '    <language>en-us</language>' . "\n";

print '    <copyright>Copyright 2007</copyright>' . "\n";

print '    <managingEditor>email@domain</managingEditor>' . "\n";

print '    <webMaster>email@domain</webMaster>' . "\n";

print '    <generator>script name</generator>';

# Create db connection

$DBH = DBI->connect("DBI:Oracle:host=zzz;SID=zzz;port=nnnn", "alertlog", "alertlog",

    { RaiseError => 1, AutoCommit => 0 })

    || die "Could not connect\n\tError: $DBI::errstr";

my $STH = $DBH->prepare("with alert as

               (select rownum rn

                      ,text_line

                  from alert_log)

            select a.text_line date_msg

                  ,b.text_line file_msg

                  ,c.text_line err_msg

              from alert a inner join alert b

                on a.rn = b.rn - 1

                           inner join alert c

                on b.rn = c.rn - 1

             where c.text_line like '%ORA-00600%'");

$STH->execute() || die "Couldn't execute statement: $STH->errstr";

while($statusRec = $STH->fetchrow_hashref())

{

  print "     <item>\n";

  print "       <title>ORA-600 Occured</title>\n";

  print "       <description>An ORA-00600 has been detected.\n";

  print "<p>\n";

  print " \n";

  print "The following information was logged about the event:\n</p><p>";

  print $statusRec->{"DATE_MSG"}   . "</p>\n<p>";

  print $statusRec->{"FILE_MSG"} . "</p>\n<p>";

  print $statusRec->{"ERR_MSG"}  . "</p>\n" . "</description>\n";

  print "       <author>your name</author>\n";

  print "     </item>\n";

}

$STH->finish();

$DBH->disconnect();

print "  </channel>\n";

print "</rss>";

That is all it takes. This is obviously a CGI script that would run in your web server. If you have one with CGI capabilities configured, you'd place the script there and call it with an url like "http://your.domain/cgi-bin/script_name.cgi". Ask your web admin if you don' know where and how to place this script. If you have your own web server and don't know how to do this, leave a comment and I'll try to give you a few quick hints about how to get it working.

I'm afraid the code above will be truncated by google's new blog interface. The code will still be there so just cut and paste it into your favorite code editor and you'll have all the needed code.

You will probably want to change these lines to something different:

$ENV{"ORACLE_HOME"}='/path/to/oracle/install/dir';

$DBH = DBI->connect("DBI:Oracle:host=...snip...,

...and the following nodes...

copyright

managingEditor

webMaster

generator

author

The first one should point to where you have Oracle installed and the second will be your connection string. If you don't know the port ask your DBA or try the default of 1521. It is the port you have your listener configured to listen on.

There is no guarantee that the code works as is. Every environment is different and the code is clearly not environment proof. If you have problems with it, check your error log for the web server and leave a message if you want my input on what the problem might be.

Enough alert log stuff? Probably. Sorry if you feel I've overdone it and got stuck on this subject for too long. Next post will not even mention the alert log.

If you have APEX or want to wait for 11G where it will be a supported option, you will want to use APEX and pure SQL for this rather than Perl. A good intro to how to do it can be found on asktom.oracle.com. It is written by Sean Dillon and looks well written and easy to follow, it can be found here. I've not tried it myself, but it sounds straight forward and anything from the people behind asktom and APEX at Oracle is usually both easy to understand and easy to implement (relative to the subject's complexity).

Querying the alert log for ORA-00600

Time to expand on the last post. As promised this will be about how to get the ORA-00600 information out of an external table for the alert log. We will use the same user and setup as we had in that post.

Lets first review how an ORA-00600 is represented in the alert log.

   Tue Feb  1 02:09:44 2005
   Errors in file /path/to/trace/file:
   ORA-00600: internal error code, arguments: [1100],
     [0x388F0D990], [0x382253888], [], [], [], [], []

The actual arguments does not matter, I took them from a forum post to just have an example. What we care about for what we're doing here is that the format is one line with a date, one line with a reference to a trace file, and one line specifying the error code (ORA-00600) and the needed arguments (for Oracle support).

Thus, we'll need to find the row with the ORA-00600 and match it up with the two previous rows. To do this we'll return to my favorite SQL key word, the with clause. It lets us get all rows, get a generated unique id per row, and know that the rows we look at will be identical as we're now using the same source. In theory, we could end up with the rows in different order if we joined the actual table with itself. Oracle does not guarantee the order you retrieve data in unless you order it, end we have nothing to order on here.

    with alert as (select rownum rn
                       ,text_line
                   from alert_log)
 select a.text_line date_msg
       ,b.text_line file_msg
       ,c.text_line err_msg
   from alert a inner join alert b
     on a.rn = b.rn - 1
                inner join alert c
     on b.rn = c.rn - 1
  where c.text_line like '%ORA-00600%';

This gives us the alert log message in three columns. We can now process it any way we want. It could be used to create a report or generate an email with your ORA-00600 messages.

However, the actual ORA-00600 shows that we made a mistake when we created the table. It cuts off the text_line after a comma. That is not what we want, we want all data up to the end of the line. So we need to change the definition of the table to make sure we get all of that.

   conn alertlog/alertlog
   drop table alert_log;
   create table alert_log
     (text_line varchar2(255))
     organization external
     (type oracle_loader
     default directory bdump
     access parameters(
     fields terminated by x'10')
     location ('alert_ORCL.log'));

That's better, the default delimiter of comma has been replaced with linefeed (ASCII 10) for UNIX platforms. Replace it with what you need. As long as it is a charcter that will never occur in a text line in your alert log, this will work fine.

Now that we have a select that gets all data for an ORA-00600 in each row it returns, it is easy to operate on that data. This model can of course be used to list other pieces of data from the alert log. One such use could be to report how long time it takes between log switches. That is left as an exercise for the reader.

My next post will show how to use this to create a RSS feed for your ORA-00600. That allows you to within minutes know when you have had an ORA-00600 occur and you get it delivered as a separate feed to your news/blog reader.

SQL Access to the alert log

I'm going to discuss how to get SQL access to your alert log in this entry. I will then build on that to make it even more useful in future posts.

Browsing the alert log can be convenient if you're already on the server. However, it is often not possible or at least not overly convenient. You may find that as the developer DBA you do not have access to the production server, or you have to log in to many different servers to check all your alert logs. Since SQL is the main tool for DBAs and one of the most critical for developers of database driven applications, it would make sense to access the key log for database operations and issues via the same interface and be able to use your SQL tool of choice.

The way to be able to access the alert log with SQL is to define an external table for it. So I'll use this need to write a little about how to create an external table for text file. It's pretty easy and can often be very useful.

create directory bdump as '/path/to/bdump';
create user alertlog identified by alertlog;
alter user alertlog quota unlimited on users;
grant create session, resource to alertlog;
grant read,write on directory bdump to alertlog;

The directory path needs to be what you have "background_dump_dest" set to. You can look the current value up in v$parameter. To be able to do this with one script across databases, you'll want to make this more dynamic with pl*sql such that the directory is retrieved from the database and used to build a dynamic DDL statement.

Now that we have a user with access to our bdump directory, we'll create our alert_log table.

create table alert_log
 (text_line varchar2(255))
 organization external
 (type oracle_loader
  default directory bdump
  location ('alert_ORCL.log'));

This is a minimal create external table statement. There are many more options to use, but we'll leave those alone in this blog post.

The create table statement is pretty straight forward, here's what it will do:

• Creates a table with data that is located external to the database
• Take all data in each line into the column text_line
• Use the normal oracle_loader (SQL*Loader) driver to access the data
• It uses the bdump directory object to specify where the alert log file is located.
• Get data from the file alert_ORCL.log.

That is all it takes. We can now access this table with SQL and use our normal techniques for filtering and aggregating information.

To look at the snapshot too old messages we generated in a previous blog post, we'd use this statement:

select text_line
 from alert_log
where text_line like '%ORA-01555%';

TEXT_LINE

Multitable insert

Mutlitable insert is a feature I've read about many times, but I've never taken the time to read up on it or play with the way this SQL statement works. I figure making a post about it will give me the reason I need to learn more about it.

It is not so much a new statement as an extension of the existing INSERT statement. It allows you to insert into different tables and with different conditions from a select statement. That is, it is a variant of the common "insert into select from" version of the insert statement.

There are of course many uses of this. Splitting a table into multiple tables seems like an obvious use, but more common may be when data is added to more than one table at the same time. One such situation could be a situation where a blob is placed in it's own table or where a parent table needs to be maintained while loading the dependent table with new data.

In our test we'll read all rows from all_ind_columns and insert into the tables test_tb and test_col. The two tables will of course hold tables and indexed columns in tables respectively.

Lets first set up the user and the tables.

conn system
create user multiins identified by multiins;
alter user multiins quota unlimited on users;
grant create session, resource to multiins;
conn multiins/multiins
create table test_tb as
select table_name
  from user_ind_columns
 where 1 = 0;
create table test_col as
select table_name, column_name
  from user_ind_columns
 where 1 = 0;

We now have a table test_tb that holds just a table_name and a table test_col that holds table_name and column_name. This is a traditional parent child table relationship. The task is now to in one statement load both tables with data from all_ind_columns so we get one row per table in test_tb and one per indexed column in test_col.

insert
  when cn = (select min(column_name)
               from all_ind_columns a
              where a.table_name = tn) then
    into test_tb values (tn)
  when 1=1 then
    into test_col values (tn, cn)
select table_name  tn
      ,column_name cn
  from all_ind_columns;

Pretty nice. This can be used to denormalize tables or just load tables in parallel when the data is in one single data source. It can even be used to have one join pull data and then put the data into different tables. The number of conditions and tables is virtually unlimited. I believe the limit is set to 127 and I'm sure I'll reach the limit for a single statement (64,000 bytes) before I reach that limit.

The subselect used in the first condition is to make sure that we only insert the table name once. In each table, there will just be one column that has the lowest value as column names are unique within a table.

You'd of course want to add primary keys and a foreign key if these tables were to now be used in a real application.

The WITH clause

To mix the last post with something that may be easier to digest, I'll make this one about one of my favorite keywords in SQL. It is the WITH clause. I can only think of one bad thing with it; it's name. It's virtually impossible to search for information on something with a name that results in over 4 billion hits on a Google search.

One reason for why I like it is for how I learned about it's existence. It was a few years ago and my manager at the time found it during leisure reading about Oracle and asked me what I thought about it. I probably answered something stupid like I'm not sure that feature even exists. We broke out the book and read about it, and I knew this was going to be an important feature for some problems. It was one of those things where you know that you have no immediate need for it, but that you have a new tool to solve problems with and it will come handy soon enough. Every DBA should be so lucky as to have a group manager that cares enough about technology that they read about it on their spare time.

Lets leave memory lane and return to reality so we can look a little on how this feature works. The technical name for it is "subquery refactoring", but I've never heard anyone refer to it as anything other than "the with clause".

It allows you to create one or more result tables you can refer to in the real select clause. Inline views can sometimes be used to achieve the same result, but the really cool thing with this feature is that you can refer to it as a table and refer to it in more than one place in your query.

Lets consider a situation where you have one table with products and one with services. Both of these tables have a category id field and that links to a category table. Lets assume that you want to retrive only the products and services that are in priority 1 categories. This is easy to do with normal SQL, but WITH can probably make it more effective and easier to maintain.

Let's first create the user and the tables we'll use in the test cases. Log on with a DBA user and execute the following commands.

drop user withtest cascade;
create user withtest identified by withtest;
alter user withtest quota unlimited on users;
grant create session, resource to withtest;
conn withtest/withtest
create table products
  (prod_id     number(2)    not null
  ,prod_name   varchar2(20) not null
  ,category_id number(2)    not null);
insert into products
  (prod_id, prod_name, category_id)
  values(10,'Chair',20);
insert into products
  (prod_id, prod_name, category_id)
  values(11,'Sofa',21);
insert into products
  (prod_id, prod_name, category_id)
  values(12,'Bench',21);
insert into products
  (prod_id, prod_name, category_id)
  values(13,'Stool',22);
insert into products
  (prod_id, prod_name, category_id)
  values(14,'Armchair',21);
insert into products
  (prod_id, prod_name, category_id)
  values(15,'Ottoman',20);
create table services
  (serv_id     number(2)    not null
  ,serv_name   varchar2(20) not null
  ,category_id number(2)    not null);
insert into services
  (serv_id, serv_name, category_id)
  values(30,'Watering',20);
insert into services
  (serv_id, serv_name, category_id)
  values(31,'Cooking',21);
insert into services
  (serv_id, serv_name, category_id)
  values(32,'Cleaning',21);
insert into services
  (serv_id, serv_name, category_id)
  values(33,'Shopping',22);
insert into services
  (serv_id, serv_name, category_id)
  values(34,'Washing',21);
insert into services
  (serv_id, serv_name, category_id)
  values(35,'Decorating',20);
create table categories
  (category_id   number(2)    not null
  ,category_name varchar2(20) not null
  ,priority      number(1)    not null);
insert into categories
  (category_id, category_name, priority)
  values(20,'Unimportant',3);
insert into categories
  (category_id, category_name, priority)
  values(21,'Very Important',1);
insert into categories
  (category_id, category_name, priority)
  values(22,'A little important',2);
commit;

We now have tables and data to allow us to write the SQL we normally would have used.

select prod_name
  from products a inner join categories b
    on a.category_id = b.category_id
   and b.priority = 1
union all
select serv_name
  from services a inner join categories b
    on a.category_id = b.category_id
   and b.priority = 1;

PROD_NAME