Improving data move on EXADATA II

22 05 2013

Writing log records

The last post in this series introduced the problem briefly. You find that post here.

In this post I’ll talk about the changes made to make that writing of log records fast enough. There were 50 million records that was written. Each of them pretty much in its own transaction. Of course the commit activity caused problem, as did log buffer issues. Some of this could be somewhat remedied with configuration.

The big issue though was that the writes themselves took too much time and too often many session ended up in long contention chains. Yes, it would have been great to have the luxury of redesigning the whole logging situation from the ground up. But, as is often the case, the solution was built such that all systems connecting were implemented in such a way that redesigning was not an option. Fixing the performance of this had to be done without requiring code changes to the systems performing the logging. Oh, joy.

So what caused the problem then? For the inserts it was pretty straight forward. Too many transactions making an insert and a commit. This caused indexes to be hotspots where all processes wanted to write at the same spot. Hash-partitioning had been introduced and that had led to less contention but slower performance. As the partitions existed on different parts on the disks the write head had to be constantly moved and that caused slower service times.

What could we do to make a big improvement while not affecting the code? We’re not talking about just 10-20% of improvement on any area in this case, and even more important was to make the performance stable. That is, the most important thing was to ensure that there were no spikes where an insert suddenly to 20 times longer than usual. The contention chains that was occurring made performance spike such that the whole system became unusable.

The solution here turned out to be something so far from advanced technologies as questioning assumptions. The first time I asked “why do we have these indexes”, most people in the room thought I was just joking around. Eventually they realised that I was serious. After an amusing period of silence where I could see them thinking “Do we need to inform him that indexes are needed to enforce uniqueness and to support referential integrity?”, someone went ahead and did just that. OK, now we were on to a productive discussion, as of course that wasn’t what I meant. The followup discussion about why we needed referential integrity and uniqueness for this set of data was very enlightening for everyone. To make a long story short, it was not needed at all. It was there because it had always been there and nobody had questioned the need before.

How come we didn’t need data to be unique? Well, this is log-data. That is it tells us what actions has been performed by the system. If some activity would be reported twice, it really wouldn’t be the end of the world. The possible problem that some activity isn’t logged cannot be handled with defining unique constraints. That is pure system design and nothing I could improve or worsen by removing some indexes.

Thus, the indexes was removed together with the foreign keys (referential integrity).

Sounds simple enough, but did it help? Did it ever! In one month after making the change, there has not been one report of one transaction that was anywhere close to take too long. This simple solution made the logging so fast that it is no longer a concern.

The next post in this series will discuss the solution for moving data to the history tables. This process took around 16 hours and it had to become at least three times as fast. As you’ll see, moving all these rows can be done much faster than so.





Faster data move on EXADATA I

15 05 2013

Introduction

In my work among other things I tune and tweak solutions for EXADATA. Today I’ll write about a big improvement we achieved with a process that moves data from the operational tables to the ones where the history is stored.

This will not be a technical post. While I talk about using advanced technologies, I will not discuss code or deep details of them in this post.

And yes, when I say post, I mean a series of posts. This will be too long to be a single post. I’ll break it up into an introduction and then a post on each area of improvement.

Let’s first discuss the before situation. This set of tables are logged to during the day. These log records are needed both to investigate how transactions were executed as well as to satisfy legal requirements. It is in a highly regulated industry and for good reason as mistakes could put someone’s life in danger.

In this situation the solutions were writing around 50 million log records per day to five tables. These tables all had a primary key based on a sequence and there was also referential integrity set up. This means that for the indexes, all processes were writing to the same place on disk. The lookup on the referential integrity was also looking at the same place. An attempt to remedy some of this had been made by hash partitioning the tables. The write activity was intense enough during the day that most of the logging had to be turned off as the solution otherwise was too slow. This of course has legal as well as diagnostic implications.

What’s worse is that once all that data was written, it had to be moved to another database where the history is kept. This process was even slower and the estimate for how long it would take to move one days worth of data was 16 hours. It never did run for that long as it was not allowed to run during the day, it had to start after midnight and finish before 7 am. As a result the volume built up every night until logging was turned off for a while and the move then caught up a little every night.

This series will have the following parts:

  1. Introduction (this post)
  2. Writing log records
  3. Moving to history tables
  4. Reducing storage requirements
  5. Wrap-up and summary

The plan is to publish one part each week. Hopefully I’ll have time to publish some more technical posts between the posts in this series.





The power of using records in APEX III

15 03 2013

In this post I’ll finish up the CRUD implementation using records, procedures and views. This series of blog posts started with this post which was followed by this.

At this point we have a working report that links to a form. The report is based on a view and the form is based on a procedure. At this point the form is only loading the record in using a procedure that uses a record in its signature. In this post we’ll complete the functionality by using the same form for insert, update, and delete functionality.

Let’s start with adding a mode page item to the form. We will use this to know if the form is invoked in insert or in update/delete mode.

  • Right-click on “Form on Stored Procedure” on the page where your form is (page 4 in my example)
  • Select “Create Page Item”
  • Select Hidden
  • Give the page item a name, like P4_MODE.
  • Click “Next”
  • Click “Next” (again)
  • Select “Create Item”

You now have a new page item which is not displayed, but whose value you can reference on this page.

To make this example easy, we’ll start with just implementing update functionality. But first we’ll go to the report and change this link column to pass in a “U” for the mode page item when it is clicked.

In the page with the report, perform the following steps.

  • Right-click “List of Employees”
  • Click “Edit Report Attributes”
  • Scroll down to “Link Column”
  • For Item 2, enter the name and value we want the newly created page item to be set to. P4_MODE and U in my case.
  • Apply Changes

Now with that done, let’s head back to the page with the form. But before we make any changes there, we need to change the package to add a procedure for updating a row in the emp table.

Here is the code for the procedure we’ll use.

create or replace package tb_access as
  type t_emp_rec is record (empno emp.empno%TYPE
                           ,ename emp.ename%TYPE);

  procedure read_emp(p_in_empno     in emp.empno%type
                    ,p_out_emp_rec out t_emp_rec);

  procedure write_emp(p_in_empno    in emp.empno%type
                     ,p_in_emp_rec  in t_emp_rec);
end tb_access;
create or replace package body tb_access as
  procedure read_emp(p_in_empno in emp.empno%type
                    ,p_out_emp_rec out t_emp_rec) is
    begin
      select empno
            ,ename
        into p_out_emp_rec.empno
            ,p_out_emp_rec.ename
        from emp
       where empno = p_in_empno;
  end read_emp;
----------------------------------------------------
  procedure write_emp(p_in_empno   in emp.empno%type
                     ,p_in_emp_rec in t_emp_rec) is
    begin
      update emp
         set ename = p_in_emp_rec.ename
       where empno = p_in_empno;
  end write_emp;
end tb_access;

This code adds the procedure write_emp wich also takes empno and the record used for read_emp, only this time they are both in parameters as we’ll not return any data from the procedure that writes to the table. The actual code in the procedure is a very simple update that sets the name to the ename in the record for the row that has the empno passed in.

With this in place we’re ready to change the form to call this procedure upon submit. We do this by taking the code used for reading in the record (The process in page rendering) and put it into the “Run Stored Procedure” process we commented out in page processing.

The code we copy looks like this:

declare
  in_emp_rec tb_access.t_emp_rec;

begin
  tb_access.read_emp(p_in_empno    => :p4_empno
                    ,p_out_emp_rec => in_emp_rec);

  :p4_ename := in_emp_rec.ename;
end;

Replace the code in “Run Stored Procedure” with the above and change it to look like this:

declare
  in_emp_rec tb_access.t_emp_rec;

begin
  in_emp_rec.ename := :p4_ename;

  tb_access.write_emp(p_in_empno    => :p4_empno
                     ,p_out_emp_rec => in_emp_rec);
end;

In the “Run Stored Procedure” change the condition to “- Select condition type -” to not have a condition for when the process is executed. It is currently conditioned on running only when the save button is clicked, leave that condition on for now.

With that in place, it is time to test the functionality of the forms new functionality. Run the report, click on the link and update the name of an employee. Assuming you followed the above and got the changes made in the right places, the form will now update the name of the employee.

The next step in this will be to add insert functionality to the form. First off is to set up the package to support inserting an employee. As we’re now going to have both insert and update through the same procedure, we need to include the mode in the signature of the procedure.

Let’s first update the package specification so the write_emp looks like this:

  procedure write_emp(p_in_empno    in emp.empno%type
                     ,p_in_mode     in varchar2
                     ,p_in_emp_rec  in t_emp_rec);

The procedure in the package needs to be updated to look like this:

  procedure write_emp(p_in_empno   in emp.empno%type
                     ,p_in_mode    in varchar2
                     ,p_in_emp_rec in t_emp_rec) is
    begin
      case p_in_mode
        when 'U' then
          update emp
             set ename = p_in_emp_rec.ename
           where empno = p_in_empno;
        when 'I' then
          insert into emp
             (empno
             ,ename)
           values((select max(empno) + 1 from emp) 
                 ,p_in_emp_rec.ename);
      end case;
  end write_emp;

Before the form is functional again, we need to change the call in “Run Stored Procedure” to match the new signature of write_emp.

declare
  in_emp_rec tb_access.t_emp_rec;

begin
  in_emp_rec.ename := :p4_ename;

  tb_access.write_emp(p_in_empno    => :p4_empno
                     ,p_in_mode     => :p4_mode
                     ,p_in_emp_rec  => in_emp_rec);
end;

With that change in place the form now works and the mode we set on the link column is now passed to the procedure to make sure an update is performed and not an insert.

To introduce insert functionality, we want to add a new button on the page with the report that invokes the form in insert mode. To add the button follow these steps on the page with the report.

  • Right-click on “List of Employees”
  • Select “Create Region Button”
  • Set name and label to “Add”
  • Click Next
  • Set position to “Right of Interactive Report Search Bar”
  • Set Alignment to “Left”
  • Click Next.
  • Select action “Redirect to Page in this Application”
  • Set the page to your form-page (4 in my example)
  • Set Request to INSERT
  • Set “Clear Cache” to 4
  • Set “Set these items” to P4_MODE
  • Set “With these values” to I.
  • Click Next.
  • Click “Create Button”

If you test the page, you’ll see that it ends up immediately to the right of the search bar. Do not click on the button just yet.

Now go to the page with the form and open the process in rendering (Read Emp).

  • Set “Condition Type” to “Value of item / column in expression 1 = expression 2″.
  • Set Expression 1 to P4_MODE.
  • Set Expression 2 to U
  • Apply Changes

This ensures that we’re only reading a row from emp when updating data. When inserting, there is no data to be read and displayed.

You may also want to set the same condition on the page item for empno to hide it when inserting as there is no empno to display then.

Test the function now by running the report page and click on the “Add” button. It adds a row with the name you enter on the form. The rest of the columns will have null as their value. Adding more columns in the procedures and on the form is an exercise left to the reader.

With the function now supporting insert and update, it is time to add the delete function so we can get rid of some of the test employees we’ve created. This function will be implemented only on the form and by adding supporting code in the package. Thus, the delete will be performed by clicking on the link and then clicking on a delete button on the form.

To implement this we’ll update the implementation of the write_emp procedure once more.

  procedure write_emp(p_in_empno   in emp.empno%type
                     ,p_in_mode    in varchar2
                     ,p_in_emp_rec in t_emp_rec) is
    begin
      case p_in_mode
        when 'I' then
          insert into emp
             (empno
             ,ename)
           values((select max(empno) + 1 from emp) 
                 ,p_in_emp_rec.ename);
        when 'U' then
          update emp
             set ename = p_in_emp_rec.ename
           where empno = p_in_empno;
        when 'D' then
          delete from emp
           where empno = p_in_empno;
      end case;
  end write_emp;

With the package updated, our next step is to change the code in the “Run Stored Procedure” process on the form page.

declare
  in_emp_rec tb_access.t_emp_rec;
  v_mode     varchar2(1);
begin
  in_emp_rec.ename := :p4_ename;
  v_mode           := :p4_mode;

  if :REQUEST = 'DELETE' then
    v_mode := 'D';
  end if;

  tb_access.write_emp(p_in_empno    => :p4_empno
                     ,p_in_mode     => v_mode
                     ,p_in_emp_rec  => in_emp_rec);
end;

The code is now changed such that a click on the delete button (not implemented yet) will set the mode to D. This is done by checking the REQUEST which we will set to DELETE for the button we’ll create now. REQUEST is a standard attribute available on buttons and branches in APEX. By setting it to a unique value, we can check for different invocation methods in code. This allows us to deal with different scenarios with just one block of code.

To add the button, follow these steps.

  • On the form-page, right-click on “Form on Stored Procedure”.
  • Select “Create Region Button”
  • Set “Button Name” and “Label” to “Delete”.
  • Click Next
  • Set “Position” to “Top of Region”
  • Set Alignment to “Left”
  • Click Next
  • Click Next (again)
  • Click Create Button

The value of “Button Name” is used to set REQUEST when the page is submitted. With this in place our form should now support deleting rows also. Update the branch on the page to not have a condition on the button pressed. Take it out for a spin.

There is one thing left to do, hide the delete button when the form is invoked to add a row.

  • Right-click the delete-button under rendering.
  • Select “Edit”.
  • Scroll down to conditions.
  • Set “Condition Type” to “Value of item / column in expression 1 = expression 2″.
  • Set Expression 1 to P4_MODE.
  • Set Expression 2 to U
  • Apply Changes

Now you’ve got a CRUD solution based on an interactive report with a form that supports insert, Update, and Delete. The total amount of code required is very small and you still have isolated the APEX-application from changes to the table structure by using a view, even the record can have fields added to it without impacting the functionality of your application. Developing using the demonstrated methods here allows you to create these applications almost as fast as with making them table based, but you have much more control and can make a lot of changes without touching the application,





Previous post hidden

19 02 2013

I have hidden my previous post on the explicit request by Oracle Security Team. If you did read it or have it in your RSS, please do not forward or talk about it online until further notice. I’m not sure it is the right thing to do, but I have promised Oracle to not publish it for now. Please help me keep that promise.





The power of using records in APEX II

7 11 2012

In this post I follow up on the post where I started talking about records in APEX. In this post I’ll show how to add a page in front of the form created in that post. Thus this post will show a report and let you navigate to the form where the selected record is displayed. Pretty basic stuff, but it allows better testing of the form and it sets the stage for showing how to make the form used for insert, update and delete in the third post on this subject.

An interactive report is based on a report. As using procedures to read data for the form to avoid having hardcoded SQL in the application, we’ll use a similar technique here. By having a view, we can change the table and columns with no impact on the application. So we start with creating a view specific for this report.

create view apex_vw_emp as
  select empno
        ,ename
        ,job
        ,mgr
        ,hiredate
        ,sal
        ,comm
        ,deptno
    from emp;

Now that we have the view, we just create a new page with an interactive report.

Go ahead and create a page with an interactive report that uses the SQL “select * from apex_vw_emp“. Choose to not have a link to single row view.

When the page is created, edit the report attributes and add a link to a custom target.

  • Target = Page in this application
  • Page = 4 (the page where the form created in the last post is located, happens to be 4 in my application)
  • Clear cache = 4 (same as Page)
  • Item 1 Name = p4_empno (the item on the form used to retrieve the row)
  • Item 1 Value = #empno#

It is worth noting that #empno# is a reference to the empno on the row in the report that was clicked.

After you apply those changes, the report now has a pen icon to the left of the rows. When you click it, it will take you to the form where the row you clicked is loaded into the form (showing the ename of the empno).

This makes it easier to test the form, but more importantly it will make it possible to set up and test full CRUD functionality in a coming post.





Is blogging another word for bragging?

31 10 2012

That was essentially reader Gabriel posted on my post Suppressing repeating values in SQL.

At first my reaction was just to ignore it as I felt it was too odd. But then use FarMan posted a complete opposite “Dude, you just saved my life. Thanks so much!”. Could the same post really get so different reactions?

Gabriel’s question was:
“The above is a neat trick, but only for the purpose of showing off. What’s the use? Please enlighten me, because I’m totally missing the point why you would waste your time and talent on this.”

At first I didn’t think much of it and just took it as a comment about some feature not being what Gabriel needed. But while it was probably not what Gabriel was asking, there may be a reason to question if blogging is just a way to show off?

After thinking about it, I’d say that yes it probably is a little bit. Anyone writing a blog does it out of interest to attract readers and to share their knowledge or experience in some way. But is an interest in writing and sharing ones knowledge necessarily the same thing as bragging?

I dont think it is, but I think there are blogs that does just that. You know the ones, I know the ones. However, I do not intend to point them out by name. The ones that does not share knowledge. A blog that just claims a point and states that the author can fix it by applying their special magic. Especially when claiming things other people rarely see or even think is a problem, but with no test case, no data, and no shared approach it is hard to prove otherwise. Even if such a blog would be correct, there is nothing anyone else can learn from it. Such posts clearly serves to boost the ego of the writer without letting other people learn something.

However, blogs written following Tom Kyte’s teachings on “Do not trust anyone, prove it to yourself” where all claims must have an appropriate testcase with setup, expected result, and explanation for how to fix the problem are not bragging. those are the ones that allows everyone to take the tests and run them and if the conclusions are faulty an engaging debate about how to better evaluate and troubleshoot the issue. Such posts rather puts the author at risk of being humbled by the masters of the field in case their claim is based on a misunderstanding.

I also object against the idea that showing a way to use a technology in a way that has no apparent immediate application to be bragging or just useless. I think that surpressing values has a great application in “some” cases. But even if you do not, the suppression testcase shows how you can relate one row to the last one and base operations in the SQL on it. Clearly that can be useful if the data has a relationship that is useful to the intended result of the query. Seeing a way today for how a certain feature works or some obscure way to use SQL may very well solve a problem you didn’t know that you’ll have tomorrow.

That is exactly why reading blogs is useful. When you need a feature, you may have read about it in a blog a few months earlier. It may even have been mine and you could have thought “what a waste, there is no need whatsoever for this feature”. Then one day you realize that your problem can be solved by using that obscure feature you read about a while back. Hopefully Gabriel will one day find a use for suppressing values and may then remember the post he read about it. Or maybe someone finds a much better way to do it and then through a comment here we’ll all learn of that way to implement suppression.





The power of using records in APEX

24 10 2012

Do you use forms based on procedures in APEX? If not, why not?
Do you use records in the procedures you use for forms in APEX? If not, why not?

I like procedures and records and with APEX I like them even more. Why you say? Should you not just point forms against tables and let the APEX magic take care of it all? No, I do not think you should.

The reason is that I love that feature for prototyping, but not for production quality code. The reason is that I think the classic design pattern of separating presentation logic from business logic and data layer is as important with APEX as it is with any other technology.

That is, APEX applications should contain as much logic for presenting data as possible, and as little data about the data layer as possible. Thus, I think any table should be able to be renamed without making any change to the APEX application.

One key to that is to use procedures to hide what tables are updated or read. That allows the tables to be changed with just updating the procedure. No change to the APEX application would be needed and thus no new deploy of it would be required.

But what if a column is added or removed from the table, does that not require a change to the signature of the procedure? Sure it does, if every column in the table is reflected in the signature as individual fields. This is why using records and procedures together is such a powerful combination.

Lets take a look at a basic implementation using a form based on a procedure that uses a record.

Here is the code for the procedure we’ll use.

create or replace package tb_access as
  type t_emp_rec is record (empno emp.empno%TYPE
                           ,ename emp.ename%TYPE);

  procedure read_emp(p_in_empno     in emp.empno%type
                    ,p_out_emp_rec out t_emp_rec);
end tb_access;
create or replace package body tb_access as
  procedure read_emp(p_in_empno    in emp.empno%type
                    ,p_out_emp_rec out t_emp_rec) is
    begin
      select empno
            ,ename
       into p_out_emp_rec.empno
           ,p_out_emp_rec.ename
       from emp
      where empno = p_in_empno;
  end read_emp;
end tb_access;

This is a procedure that just returns a row of a certain emp_no from the emp table. The columns emp_no and ename are returned in the record.

To create a page in APEX using this procedure, follow these steps.

  1. Create a new page
  2. Select Form
  3. Select Form on Procedure
  4. Select the procedure (tb_access-read_emp) + next
  5. Change page and region names if you want to. + next
  6. Next (Do not use tabs)
  7. Next (No invoking page)
  8. Select a page to brach to on cancel and submit. + next
  9. Select to include only the columns that are in the record (empno and ename).
  10. Select display only as the display type for empno. +next
  11. Create

That was pretty easy, pretty much the same amount of work as to create it based on a table. However, there is one more thing we need to do. The reason for this is a bug. Hopefully this step is not needed in future versions. You know you have encountered this if the page items on the new page has names such as p4_pno and p4_ame. That is the first two letters in the name of the fields in the record were lost when the page items were created. This does not happen for fields in the signature, only for fields in a record that is in the signature.

It is easy enough to change the names of the page items. It is of course not needed technically speaking, but you will save yourself a lot of grief by having page items you know what they are used for.

The next step is to update the call to the procedure. In the “page processing” section there is a “run stored procedure”. The code will look something like this.

#OWNER#.TB_ACCESS.READ_EMP(
EMPNO => :p4_PNO,
ENAME => :p4_AME);

Yes, the bug affects the name of the page items used to provide values to the call. Not only is that wrong, the call is not using a record so the call will not work. A third problem is that the call is in the page processing section, but we want it to read data to be displayed.

As we will not update any data in this post, just comment out this code and copy it. Set the condition to never to avoid the call to even be attempted. Now create a process in “before regions” in page rendering. Paste the code you copied and update it to look like this.

declare
  in_emp_rec tb_access.t_emp_rec;

begin
  tb_access.read_emp(p_in_empno    => :p4_empno
                    ,p_out_emp_rec => in_emp_rec);

  :p4_ename := in_emp_rec.ename;
end;

This code fetches a row from the emp table based on an empno that the page item p4_empno has been set to. This is of course assumed to be done by a page that branches to this page.

This code is of course something that is very much a template that once you have it set up the way you want in one place it is just a matter of adjusting it for the needs of different pages. It really does not take much more time than widget based page creation using a table. This however gives much more flexibility in changing the underlaying datamodel as well as having some people focus on developing the presentation logic and others focused on writing data access logic.

This form is now just a read-only form, but it takes very little work to make it a full fledged crud by supporting insert, update, and delete in the same form. I may build on this form in a future post to show one way of doing that.

As a last thing, let’s just show that this page is now immune to changes to the record. Lets change the package to add another field to the record and populate it in the SQL.

create or replace package tb_access as
  type t_emp_rec is record ( 
        empno  emp.empno%type
       ,ename  emp.ename%type
       ,sal    emp.ename%type);

    procedure read_emp(p_in_empno     in emp.empno%type
                      ,p_out_emp_rec out t_emp_rec);
end tb_access;
create or replace package body tb_access as
    procedure read_emp(p_in_empno     in emp.empno%type
                      ,p_out_emp_rec out t_emp_rec)  is
      begin
         select empno
               ,ename
               ,sal
           into p_out_emp_rec.emp_no
               ,p_out_emp_rec.ename
               ,p_out_emp_rec.sal
           from emp
          where empno = p_in_empno;
      end read_emp;
end tb_access;

This change is now available to any user of the procedure, while the existing page just keeps working. It does not even need a resave/compile. The signature of the procedure is the same and so all callers will keep working. This allows a change to be made without being forced to also change code or recompile all places where a call to the procedure is made.