Oracle DB Development

Rant about partitioning licensing

I'll give myself a quota for no more than one rant a month. However, one of my pet peeves is how Oracle licenses the database. I'l probably return to why I dislike the general model in a future rant, but this on e will be specific to partitioning.

To use partitioning, you have to pay for the Enterprise Edition of the database at $40,000 per CPU and then pay an additional $10,000 for the partitioning option. That is a lot of money if all I really need is SE One (at $5,000 per CPU) with Partitioning. Unfortunately, that is not an option.

I know partitioning used to be a fairly exotic feature. That was however over 10 years ago, when few datbases were really large and few SQL statements would benefit from parallel execution. As the average database has grown to be really large and most database servers now are multi CPU (and core) servers, this should now be treated as core database functionality.

In many cases it doesn't matter as large companies has site wide use of Oracle DB technology. It does, however, matter for smaller companies and startup companies. If you were to build a new solution today and market it initially to small companies or even run it as your own ASP, you would not be able to design with partitioning. Later on, your solution will be created and the cost of rearchitecting for partitioning is unacceptable.

Even worse is when you're taking an existing application and trying to move it downstream where SE One has to be used to meet the budgets those implementations have. Why spend a lot of time of a lot of good resources to implement "manual" partitioning?

I think Oracle would benefit from making partitioning included in all licenses. One thing it would do would tie customers closer to Oracle as the cost of converting partitioning from one DB to another is high due to the lack of standard even for the DDL used. I think it would also remove another reason for customers to consider other databases. I think licensing should be such that customers can scale with the database and get started on what they need. The situation today is such that building small and needing partitioning always requires a discussion of "can we replace Oracle in our technology stack".

By having partitioning as an included feature, companies could design for it when they begin designing new applications and get the use of it as they grow. That helps Oracle attract small companies that hopefully grow into large customers over time. It also helps in not getting people convinced that Oracle doesn't scale as using partitioning was too expensive when they started and now it's too costly (or politically impossible) to change it.

Designing manual partitioning schemes is really a waste of my and other Oracle professionals time. So… Please Oracle, it is time to revisit how licensing partitioning is done. It is a feature we need to put in the core toolbox and get people to make better use of the database with it. I hope 11G comes with a new licensing strategy for this. I believe Oracle would make a lot of money on making this an included feature in all editions of the database.

That's the partitioning rant. I know others have similar ideas on this as I've read some recent blogs on partitioning. I love Oracle's partitioning, but I hate having to use it only for the largest customers. I really dislike having to think of and implement manual partitioning, I know it will n ot scale as well as database driven partitioning and I still have to waste my time on it for financial reasons. Even worse is having to discuss if we should avoid Oracle altogether as we cannot use partitioning and then maybe a cheaper database will work just as well for us. I don't want to have to avoid the best database over financial details.

If one database is the best for the job, then I want a licensing model that allows me to start small and pay more as my need grows (meaning using SE One to start with and grow to EE while I design for and with partitioning the whole time).

Optimistic Locking with ORA_ROWSCN

Are you using 9i/10G and still implement optimistic locking with your own column rather than through Oracle's pseudo column? So am I, but I couldn't really explain why. My main (defensive) argument would be that the system was built long before 9i. Still, it would make sense for us to change it. Let's look at a, hopefully, quick example.

Let's first create a user with two tables and add the same data to both tables.

conn system
create user rowscn identified by rowscn;
alter user rowscn default tablespace users;
alter user rowscn quota unlimited on users;
grant resource, create session to rowscn;
conn rowscn/rowscn
create table t1 (id number, total number);
insert into t1 (id, total) values(1,100);
insert into t1  (id, total) values(2,200);
insert into t1  (id, total) values(3,300);
create table t2 (id number, total number) rowdependencies;
insert into t2 (id, total) values(1,100);
insert into t2  (id, total) values(2,200);
insert into t2  (id, total) values(3,300);
commit;

Two tables with just one "small" difference. We'll soon see see the difference it makes. To see how this works, we'll use a simple update of these three rows.

update t1 set total = total + 1 where id = 1;
commit;
update t1 set total = total + 1 where id = 2;
commit;
update t1 set total = total + 1 where id = 3;
commit;

Each row was updated in a different transaction as we committed between each update. If we now look at the pseudo column, ORA_ROWSCN, we will see something like:

select id, total, ora_rowscn from t1;
ID  TOTAL ORA_ROWSCN
---

Who reads blogs

Reading this post about why the average Oracle IT worker doesn't read blogs got me thinking. Does the average IT worker read blogs? I'm not sure. In what I have observed, it is something a segment does. I'd contend that the average blog reader is more advanced and more thirsty for more knowledge that the average person in the same field. I think it is also something the more internet aware do. Everyone knows about the internet, but how many non blog readers can explain the technologies involved? I have a feeling that most that can are on the blogosphere as readers today.

This is not to say that there aren't really good Oracle Specialists (or IT Specialists for that matter) that doesn't read blogs at all. Some has never looked at newsreaders and stay away because they don't want the hassle of learning one more product, other feel it is a waste of time as most blogs are useless to them, yet others feel that there are good blogs - but they are too hard to find.

I think most will improve and learn interesting things if they start reading blogs, I know I have learned many things I'd never have thought of or heard about if it weren't for blogs. Participating on and reading the best blogs is like an informal study group where you can opt in to learn about the subjects you are interested in.

I think it is that some like reading and learning small segments, while others are happy just doing what they know and hopefully hitting the Oracle docs from time to time when they get stuck. It may be similar to how it is natural for some to look up concepts and syntax in the documentation while others try to avoid the documentation like the plague. Maybe most blog readers are among those who like to read the documentation to learn more. There is a difference between looking things up because you have to and enjoying it because may always learn something new.

What do you think? Is this an awareness issue or is this just one of those things where some people will never learn to like the blog format for their reading and learning?

SQL Challenge I - Solution

Both Gary and Philip postd working versions. Gary's didn't work for me when I had the same letter more than one time in the string. Philips did work and the need to deal with each letter as if it is unique can of course be questioned. The intent was for a solution that could handle that and as Philip showed adjusting that is as easy as to just add a distinct.

My version is based on not using analytical functions of which I thought connect by was a part. Looking at it in another way, I wanted something that could be adopted to other databases.

My starting version of the solution would be:

   with v    as (select 'ABC' v from dual)
   ,base as (select substr(v.v,1,1) nv, 1 nn from v
             union all
             select substr(v.v,2,1) nv, 2 nn  from v
             union all
             select substr(v.v,3,1) nv, 3 nn  from v)
  select a.nv||b.nv||c.nv
    from base a, base b, base c
   where a.nn not in(b.nn, c.nn)
     and b.nn not in (c.nn)

What's not to like? I get to use one with construct that takes the output from another with construct as it's input.
The "base" with gives us a result table that has one letter from the input string on each row. I'm then joining that with itself three times (as I have three letter in the string) to get a cartesian product that holds all cominations. It has the valid combinations (where all three letters are used) and invalid ones (where some letters are used more than one time). The last thing is to remove the invalid combinations. I do this by checking if the number I assign to each value (nn) occurs more than one time in the result. For AAA we would have nn for table alias A, B, and C be 1. I only have to check if each value occurs later on in the column list as if it was equal to an earlier one, the test for that earlier one would detect that the same was used.

So it works… Interesting and satisfying at the same time. However, the union is not too lovable. How can we avoid it? Well, the easy way would of course be to use the connect by trick, but as I intended to not use it I'll have to cheat and use anoter unrelated table (view).

   with v    as (select 'ABC' v from dual)
      ,base as (select substr( v.v,rownum,1) nv
                      ,rownum nn from v, all_objects
                 where rownum <= length(v.v))

    select a.nv||b.nv||c.nv

     from base a

         ,base b

         ,base c

    where a.nn not in (b.nn, c.nn)

     and b.nn not in (c.nn) 

I'm usig all_objects which should have more rows than the length of a string you want all combinations. It works, but I still prefer the connect by version if I were to use it. This way it does satisfy the limitation I put up for this.

So while this works, it is not as dynamic as one could want it to be. You need a different SQL if you have a four letter string. It ieasy to change this for that, but it is still annoying to have to have many different ones. An alternative would be to write PL*SQL to generate the SQL and then execute it. While that may be nicer, I wanted a solution that was a single statement.

I did find a way to do it. I'm sure it can be done easier and nicer. Maybe analytic functions would make the SQL faster and easier to understand. However, This way combines XML, with clauses, and dynamic in-line queries. For fancy use of the Oracle database, it's almost optimal… How practical it would be to use this very often and with long strings would have to be tested before getting it into a production scenario.

   with v as (select 'ABC' v from dual)
      ,base as (select substr(v.v,rownum,1) nv
                      ,rownum nn
                  from v
                      ,all_objects
                 where rownum <= length(v.v))       ,sel  as (select 'select ' ||                        substr(max(sys_connect_by_path('a'||
                       nn||'.nv', '||')),3)||' data' a
                  from base
               connect by prior nn = nn - 1
                 start with nn = 1)
     ,frm  as (select 'from ' ||
                      substr(max(sys_connect_by_path(
                      'base a'||nn, ',')),2) a
                 from base
              connect by prior nn = nn - 1
                start with nn = 1)
     ,whr2  as (select 'a'||(nn-1)||'.nn not in('||
                       substr(max(sys_connect_by_path(
                       'a'||nn||'.nn', ', ')),2) ||
                       ')' cond
                      ,max(rownum) rn
                  from base
                      ,v
                 where nn > 1
               connect by prior nn = nn + 1
                 start with nn = length( v.v)
                 group by nn
                 order by nn)
     ,whr  as (select 'where ' || substr(max(
                      sys_connect_by_path(cond, ' and '))
                      ,5) a
                 from whr2
                     ,v
                where not rn = length(v.v)
              connect by prior rn = rn + 1
                start with rn = length( v.v) - 1)
  select extractvalue(t.column_value,'/DATA')
    from sel a
        ,frm b
        ,whr c
        ,xmltable(
          'for $root in $vals
           return $root/ROWSET/ROW/DATA'
           passing xmltype(dbms_xmlgen.getxml(
          'with v    as (select ''ABC'' v
                           from dual)
               ,base as (select substr(v.v,rownum,1) nv
                               ,rownum nn
                           from v
                               ,all_objects
                          where rownum <= length(v.v))'                              ||a.a||' '                              ||b.a||' '                              ||c.a)) as "vals") t; 

The formatting is a bit forced here as I need it to fit in the line size google allows on this blog. I will not explain in detail what I have tried to achieve in this SQL. If you would want me to talk about what I'm doing in a post, leave a comment and I'll try to write it up in a post soon.

In short, I'm generating different pieces of the SQL statement needed in the with clauses and them I'm using that to create the XML document with all these cominations ina single XML document. I'm then using some XML function and XMLTable with an XML Query to get it back out form one row (one XML doc) to multiple rows to get it back to relational data.

The threaad that started me thinking about this was this thread. There are some permutations things there that solves this with pure analytics. But why do this the easy way when you can make it more complicated. Laurent's solution is probably the only one that scales well with longer strings.

I hope you enjoyed this and hopefully even learned something. I know I did, both that Cartesian products can be used in ways I hadn't really thought of and that I need to learn more about analytical functions.

I also learned more about XML function and how to move data in and out of these dynamic in-line queries.

SQL Challenge I

While maybe a pretentious title, I've always enjoyed getting and giving what I call SQL Challenges. It is basically a short explanation for what you should do with pure SQL and some limitations to how you can do it and then you get to try to come up with a solution. It is frequently more about being able to do it, than it is that one would actually implement it that way. The idea is that you lear new ways to work with SQL or think about SQL and that will help you write better SQL for things you'd actually do in your application.

The "I" is in case this is something someone likes and I think of other ones in the future. It the first one isn't I, then the second one cannot really be II, right?

To the challenge. I thought of this after reading a question on a forum a while back. The final more performant solution was not SQL, but I thought it was interesting to actually do this in SQL. In case anyone recognizes this from the forum, please don't link to it. I will link to it when I show my version of a solution to this challenge. The challenge is about challenging yourself to learn something, it's not about winning over everyone else. Learning is a personal adventure, not some form of competition.

The problem this challenge poses is to find all combinations of the letters in a string. Lets say that it is always three letters in this string to reduce complexity and make it more possible to solve it with pure SQL.

If the string contains ABC, you'd expect the query to return:
ABC
ACB
BAC
BCA
CAB
CBA

If it returns more or less, then you have not managed to get just the combinations. AAA, for example, is not a valid answer as there is just one A in the input.

You should limit your SQL to not use analytical functions at all, and not PL*SQL. The SQL should be written such that you only need one SQL statement and no other objects are created to support your single SQL statement.

Also consider that you will want a solution that is possible to rewrite for at least 8 letters instead of the three you're writing the SQL for. Several ways of solving this will make the statement too large soon after three letters.

Hopefully this is complicated enough to be a challenge, I know it took me quite some time to figure out how I could solve it.

Dynamic in-line queries

I don't know of a better name for this feature. I think this name descibes the function well, even if it may sound overly ambitious. This whole post will be based on a post by Laurent Schneider where he uses XML features to do a select count(*) from different tables and return the result in one result table. It is like a auto generated number of selects that are unioned together.

You can find Laurent's post here. It's a wonderful piece of SQL to do something most of us would probably not have thought of using a single SQL statement for.

The version Laurent uses is:

   select table_name
        ,to_number(extractvalue(xmltype(
            dbms_xmlgen.getxml(
              'select count(*) c from '||table_name))
                  ,'/ROWSET/ROW/C')) count
    from user_tables;

Let's first review what happens in this query. The inner most function is a call to dbms.getxml with a query that will return one row and one value. This sql stateement is created on the fly based on the table name the outer select retrieves from user_tables. The function will of course turn the query result into XML text. The xmltype that wraps the function will convert the XML text into an XML object so XPATH functions can be used. The extractvalue function pulls out the node ROWSET/ROW/C where the two first are given as that is how getxml will return the data. C is the name of the column or in this case the alias assigned in the select statement as the resulting "column" from the count(*) function. The last thing that happens is that the extracted value is converted to a number and it is then returned to the outer select as the result from this function.

The query returns a result set like:

TABLE_NAME                      COUNT

Creating a feed with ORA-00600 errors

Instead of having to look at your different alert logs for ORA-00600 errors frequently, why not have them come to you? This post will, as promised, show the Perl code needed to publish a feed with ORA-00600 errors. This is building on the previous two posts that shows how to create an external table for the alert log and how to pull out all information for an ORA-00600 so each fetch retrieves one the data you're interested in.

The feed I'm using is a RSS 2.0 format. All feeds are fairly closely related so you can change it or provide different ones if you want to. You'll have to look up the different formats and then make the changes you want to this code.

The script I have here is a simplified version that removes all the usual dynamic nature and flexibility with parameters I usually have. This is done for brevity and to focus the script on the task at hand - delivering the ORA-00600 errors to your newsreader soon after they occur.

The script is not pretty and it could be written much nicer with use of modules such as XML::RSS. Unfortunately that module has a chain of dependencies that I didn't want to satisfy, so I'm creating the tags the old fashioned hard coded way. If you are to do a lot feeds with Perl, you will want to get that module installed.

!/usr/bin/perl
require v5.6;
use strict;
use CGI qw(:standard :html3 :netscape);
use CGI::Carp;
use DBI;
use DBD::Oracle qw(:ora_session_modes);

my ($ORACLE_HOME, $DBH);

if ( $ENV{TWO_TASK} ) { undef $ENV{TWO_TASK}; }

$ENV{"ORACLE_HOME"}='/path/to/oracle/install/dir';

my $statusRec;

print 'Content-Type: text/xml' . "\n\n";

print '<?xml version="1.0"?>' . "\n";

print '<rss version="2.0">' . "\n";

print '  <channel>' . "\n";

print '    <title>Oracle ORA-00600</title>' . "\n";

print '    <description>ORA 600 errors</description>' . "\n";

print '    <language>en-us</language>' . "\n";

print '    <copyright>Copyright 2007</copyright>' . "\n";

print '    <managingEditor>email@domain</managingEditor>' . "\n";

print '    <webMaster>email@domain</webMaster>' . "\n";

print '    <generator>script name</generator>';

# Create db connection

$DBH = DBI->connect("DBI:Oracle:host=zzz;SID=zzz;port=nnnn", "alertlog", "alertlog",

    { RaiseError => 1, AutoCommit => 0 })

    || die "Could not connect\n\tError: $DBI::errstr";

my $STH = $DBH->prepare("with alert as

               (select rownum rn

                      ,text_line

                  from alert_log)

            select a.text_line date_msg

                  ,b.text_line file_msg

                  ,c.text_line err_msg

              from alert a inner join alert b

                on a.rn = b.rn - 1

                           inner join alert c

                on b.rn = c.rn - 1

             where c.text_line like '%ORA-00600%'");

$STH->execute() || die "Couldn't execute statement: $STH->errstr";

while($statusRec = $STH->fetchrow_hashref())

{

  print "     <item>\n";

  print "       <title>ORA-600 Occured</title>\n";

  print "       <description>An ORA-00600 has been detected.\n";

  print "<p>\n";

  print " \n";

  print "The following information was logged about the event:\n</p><p>";

  print $statusRec->{"DATE_MSG"}   . "</p>\n<p>";

  print $statusRec->{"FILE_MSG"} . "</p>\n<p>";

  print $statusRec->{"ERR_MSG"}  . "</p>\n" . "</description>\n";

  print "       <author>your name</author>\n";

  print "     </item>\n";

}

$STH->finish();

$DBH->disconnect();

print "  </channel>\n";

print "</rss>";

That is all it takes. This is obviously a CGI script that would run in your web server. If you have one with CGI capabilities configured, you'd place the script there and call it with an url like "http://your.domain/cgi-bin/script_name.cgi". Ask your web admin if you don' know where and how to place this script. If you have your own web server and don't know how to do this, leave a comment and I'll try to give you a few quick hints about how to get it working.

I'm afraid the code above will be truncated by google's new blog interface. The code will still be there so just cut and paste it into your favorite code editor and you'll have all the needed code.

You will probably want to change these lines to something different:

$ENV{"ORACLE_HOME"}='/path/to/oracle/install/dir';

$DBH = DBI->connect("DBI:Oracle:host=...snip...,

...and the following nodes...

copyright

managingEditor

webMaster

generator

author

The first one should point to where you have Oracle installed and the second will be your connection string. If you don't know the port ask your DBA or try the default of 1521. It is the port you have your listener configured to listen on.

There is no guarantee that the code works as is. Every environment is different and the code is clearly not environment proof. If you have problems with it, check your error log for the web server and leave a message if you want my input on what the problem might be.

Enough alert log stuff? Probably. Sorry if you feel I've overdone it and got stuck on this subject for too long. Next post will not even mention the alert log.

If you have APEX or want to wait for 11G where it will be a supported option, you will want to use APEX and pure SQL for this rather than Perl. A good intro to how to do it can be found on asktom.oracle.com. It is written by Sean Dillon and looks well written and easy to follow, it can be found here. I've not tried it myself, but it sounds straight forward and anything from the people behind asktom and APEX at Oracle is usually both easy to understand and easy to implement (relative to the subject's complexity).

Querying the alert log for ORA-00600

Time to expand on the last post. As promised this will be about how to get the ORA-00600 information out of an external table for the alert log. We will use the same user and setup as we had in that post.

Lets first review how an ORA-00600 is represented in the alert log.

   Tue Feb  1 02:09:44 2005
   Errors in file /path/to/trace/file:
   ORA-00600: internal error code, arguments: [1100],
     [0x388F0D990], [0x382253888], [], [], [], [], []

The actual arguments does not matter, I took them from a forum post to just have an example. What we care about for what we're doing here is that the format is one line with a date, one line with a reference to a trace file, and one line specifying the error code (ORA-00600) and the needed arguments (for Oracle support).

Thus, we'll need to find the row with the ORA-00600 and match it up with the two previous rows. To do this we'll return to my favorite SQL key word, the with clause. It lets us get all rows, get a generated unique id per row, and know that the rows we look at will be identical as we're now using the same source. In theory, we could end up with the rows in different order if we joined the actual table with itself. Oracle does not guarantee the order you retrieve data in unless you order it, end we have nothing to order on here.

    with alert as (select rownum rn
                       ,text_line
                   from alert_log)
 select a.text_line date_msg
       ,b.text_line file_msg
       ,c.text_line err_msg
   from alert a inner join alert b
     on a.rn = b.rn - 1
                inner join alert c
     on b.rn = c.rn - 1
  where c.text_line like '%ORA-00600%';

This gives us the alert log message in three columns. We can now process it any way we want. It could be used to create a report or generate an email with your ORA-00600 messages.

However, the actual ORA-00600 shows that we made a mistake when we created the table. It cuts off the text_line after a comma. That is not what we want, we want all data up to the end of the line. So we need to change the definition of the table to make sure we get all of that.

   conn alertlog/alertlog
   drop table alert_log;
   create table alert_log
     (text_line varchar2(255))
     organization external
     (type oracle_loader
     default directory bdump
     access parameters(
     fields terminated by x'10')
     location ('alert_ORCL.log'));

That's better, the default delimiter of comma has been replaced with linefeed (ASCII 10) for UNIX platforms. Replace it with what you need. As long as it is a charcter that will never occur in a text line in your alert log, this will work fine.

Now that we have a select that gets all data for an ORA-00600 in each row it returns, it is easy to operate on that data. This model can of course be used to list other pieces of data from the alert log. One such use could be to report how long time it takes between log switches. That is left as an exercise for the reader.

My next post will show how to use this to create a RSS feed for your ORA-00600. That allows you to within minutes know when you have had an ORA-00600 occur and you get it delivered as a separate feed to your news/blog reader.

SQL Access to the alert log

I'm going to discuss how to get SQL access to your alert log in this entry. I will then build on that to make it even more useful in future posts.

Browsing the alert log can be convenient if you're already on the server. However, it is often not possible or at least not overly convenient. You may find that as the developer DBA you do not have access to the production server, or you have to log in to many different servers to check all your alert logs. Since SQL is the main tool for DBAs and one of the most critical for developers of database driven applications, it would make sense to access the key log for database operations and issues via the same interface and be able to use your SQL tool of choice.

The way to be able to access the alert log with SQL is to define an external table for it. So I'll use this need to write a little about how to create an external table for text file. It's pretty easy and can often be very useful.

create directory bdump as '/path/to/bdump';
create user alertlog identified by alertlog;
alter user alertlog quota unlimited on users;
grant create session, resource to alertlog;
grant read,write on directory bdump to alertlog;

The directory path needs to be what you have "background_dump_dest" set to. You can look the current value up in v$parameter. To be able to do this with one script across databases, you'll want to make this more dynamic with pl*sql such that the directory is retrieved from the database and used to build a dynamic DDL statement.

Now that we have a user with access to our bdump directory, we'll create our alert_log table.

create table alert_log
 (text_line varchar2(255))
 organization external
 (type oracle_loader
  default directory bdump
  location ('alert_ORCL.log'));

This is a minimal create external table statement. There are many more options to use, but we'll leave those alone in this blog post.

The create table statement is pretty straight forward, here's what it will do:

• Creates a table with data that is located external to the database
• Take all data in each line into the column text_line
• Use the normal oracle_loader (SQL*Loader) driver to access the data
• It uses the bdump directory object to specify where the alert log file is located.
• Get data from the file alert_ORCL.log.

That is all it takes. We can now access this table with SQL and use our normal techniques for filtering and aggregating information.

To look at the snapshot too old messages we generated in a previous blog post, we'd use this statement:

select text_line
 from alert_log
where text_line like '%ORA-01555%';

TEXT_LINE