Tuesday, February 18, 2014

PostgreSQL WAL vs. Oracle Redo Log

Write-Ahead Logging (WAL) in PostgreSQL and Redo Logs in Oracle have some things in common. Here I want to compare both with respect to database configuration.
WAL and Redo Logs both have the same purpose: to guarantee data integrity the database management system must write changes to some kind of stable storage. But instead of writing data directly to the location where it belongs and from where it will be retrieved later on, records describing the changes (in Oracle sometimes called change vectors) are written to dedicated log files. After the log has been flushed to disk the DBMS has enough information available to recreate the original data changes after a crash. The data eventually will be written to the correct location by an operation called checkpoint. When the checkpoint is completed the log is no longer needed for crash recovery (but it could still be valuable for log based replication or disaster recovery).
The invention of these logs is probably the consequence of plain old rotational disks being the prevailing kind of stable storage for most of the Information Age. The throughput of writing sequentially to a disk is much larger than writing single blocks to different spots on the disk. Therefore it will in general be much faster to write the log with a change description than to write the changed data itself to the database files.
The most visible difference between WAL and redo logs is their configuration. An Oracle DBA has to decide on the number of redo logs (two or more; to simplify things I will ignore mirrored redo logs here) and their size. So the disk space used by the redo logs is fixed and will only change if the DBA modifies the configuration. For PostgreSQL on the other hand the size of a WAL file is always 16 megabytes (unless the server software has been recompiled with a different value). The exact number of WAL files depends on a couple of configuration parameters (checkpoint_completion_target, checkpoint_segments & wal_keep_segments) and the current load profile.
So what does this mean for the database operation? Both DBMS will reuse a log when the contained data is no longer needed for recovery. That implies that the checkpoint for the relevant data blocks must have finished. The log must also have been archived successfully if archiving is enabled.
Oracle can only use the pre-configured logs and will emit warnings or error messages when a log cannot be reused. The message “checkpoint not complete” in the alert log indicates that Oracle has to postpone database changes because switching to the next redo log is not yet possible due to the still running checkpoint. Obviously this reduces the performance of the application. It gets worse when the archiver is unable to archive a log. In this case the error message “ORA-00257: archiver error. Connect internal only, until freed” comes up and the database freezes until the issue is resolved by the DBA.
PostgreSQL on the other hand will start to add more WAL files to the directory pg_xlog when it can not reuse an existing log. This prevents the database from stalling if checkpoints do not finish in time or the archiver process can't keep up with the work. Of course an adequate amount of disk space must be reserved for PostgreSQL to make this possible without failures.
Both DBMS offer parameters to control the frequency of checkpoints. In PostgreSQL the parameter checkpoint_timeout is used to define the maximum time interval between two checkpoints. Oracle has a similar parameter called log_checkpoint_timeout. But nowadays the checkpoint frequency is often automatically adjusted to satisfy the crash recovery time. The parameter fast_start_mttr_target sets the target for the duration of a crash recovery and Oracle triggers checkpoints to achieve that goal. So in this case the configuration of Oracle uses a more business aligned number.
In both systems the amount of changes done also triggers a checkpoint. PostgreSQL has the parameter checkpoint_segments to set the number of segments written to the WAL files before a checkpoint happens. As said above, the segments normally have a size of 16 megabytes. So this gives at least 16 megabytes written to WAL until this type of checkpoint is triggered. For Oracle the parameter log_checkpoint_interval specifies the number of OS blocks (containing 512 bytes on most platforms) between checkpoints. A checkpoint is also triggered by a redo log switch and therefore the amount of changes is also defined by the size of the redo logs. A DBA can create redo logs much smaller than 16 megabytes so this could lead to very frequent checkpoints in Oracle. In this case the DBA will have to increase the size of the redo logs to reduce the frequency of these checkpoints. In PostgreSQL the same can by achieved by increasing the parameter checkpoint_segments.
The first step when looking into checkpoint activity is to write the relevant details to a logfile. PostgreSQL uses the parameter log_checkpoints to write details of every checkpoint into the server log. The log even shows if the checkpoint has been triggered by the timeout (indicated by checkpoint starting: time) or by the amount of changes (indicated by checkpoint starting: xlog). Oracle uses the parameter log_checkpoints_to_alert to write details into the alert log.

How to send SMS using SMSLib over bluetooth link on windows platform

Hello readers,
purpose of this blog is to explain 'How to send SMS using SMSLib '.

Assumption:
1. You have java1.5 or letter installed on your Computer
2. You are on WinXP
3. Your java installtion directory JAVA_HOME = c:\java\jdk1.5
4. You have no separate jre installed on your pc ( you are yousing jre that is loacated at
JAVA_HOME\jre)
5. You hava any java supported mobile phone ( I have tested on Nokia 7210, N97, N72)

Here We will connect PC with any java supported mobile via "Standard Serial Over blue tooth link". So you need BlueTooth Devise ( Blue tooth radio) .

STEP 1. Download SMSLib binary distribution from here
STEP 1. Download SMSLib src from here
STEP 2. Download RXTX from here.

STEP 3. Download CommonsNet from here.
STEP4. Download javacomm20-win32 from here.
STEP5. Download slf4j-1.5.6 from here.
STEP6. Download log4j-1.2.8.jar from here

STEP7. Unzip all the files in suitable location

STEP8. Copy following files to JAVA_HOME\jre\lib\ext
1. commons-net-2.0.jar from CommonsNet\commons-net-2.0.jar
2. comm.jar from javacomm20-win32\commapi\comm.jar
3. javax.comm.properties from javacomm20-win32\commapi\javax.comm.properties
4.RXTXcomm.jar from rxtx-2.1-7-bins-r2\RXTXcomm.jar
5. All the jar from slf4j-1.5.6\*.jar
6. smslib-3.4.1.jar from smslib-v3.4.1-bin\dist\lib\smslib-3.4.1.jar

STEP9. Copy following files to JAVA_HOME\jre\bin
1.win32com.dll from javacomm20-win32\commapi\win32com.dll
2. rxtxSerial.dll from rxtx-2.1-7-bins-r2\Windows\i368-mingw32\rxtxSerial.dll
3. rxtxParallel.dll from rxtx-2.1-7-bins-r2\Windows\i368-mingw32\rxtxParallel.dll

STEP10. Copy following files to C:\Program Files\WINDOWS\system32
1.win32com.dll from javacomm20-win32\commapi\win32com.dll
2. rxtxSerial.dll from rxtx-2.1-7-bins-r2\Windows\i368-mingw32\rxtxSerial.dll
3. rxtxParallel.dll from rxtx-2.1-7-bins-r2\Windows\i368-mingw32\rxtxParallel.dll

STEP11. If you are using any java IDE (i.e netbeans) then create new project,
add source packages smslib-v3.4.1-src\src\java.
You need to have all the file of STEP 8 in your class path.( to compile the project) and build project. Set misc.CommTest as main class of the project

STEP12.
Connect Bluetooth devise to your computer's USB port. If PC promps you for driver installation
then install the driver. If your PC can not find any suitable driver the get the
driver from someware for Standard Serial Link Over BuleTooth

STEP 13. Once the driver is installed start bluetooth of mobile phone
STEP14

Open control panel/BueToothDevice
click add
check 'select passkey for me'
click next
system will display pass key
accept connection from mobile phone using that pass key( allow connections without prompt)

STEP15.
Open control panel/Phone and modem
You should see one modem attached to port like 'COM5' in modem list.

STEP16.
Run misc.CommTest and wait for programm to complete.
When programm completes , Note-down for which values of boude rate and comm port
programm printed text like

Getting Info :
(some text)(model name of us phone) i.e (Some text)Nokia7210 Suppernova.
You have to note values for the com port attached to your modem in modem list ( comtrol panel /phone and modem)

STEP17
Open examples.modem.SendMessage.java file in your IDE
Modefy following line
SerialModemGateway gateway = new SerialModemGateway("modem.com1", "COM1", 57600, "Nokia", "6310i");
according to your environment.
1. First arg = here first argument of the SerialModemGateway() constructor can be any String
2. 2nd Arg = com port ( port attached to your modem in modem list ( comtrol panel /phone and modem))
3. 3rd arg = oude rate for which commtest got info( refer to step16)
4. 4rth arg = manufacturer of your mobile phone i.e Nokia , SAMSUNG
5. 5th arg = Model name of your mobile phone ( refer to step16)


Modify following line
msg = new OutboundMessage("+306948494037", "Hello from SMSLib!");
here replace +306948494037 with +<> i.e +919275901456

add following line someware aftwre instantiation of SerialModemGateway
gateway.setSMSCNumber("+919825068000")
This smsc number can be found from you mobile's 'Message'/'Message Settings' depending on your mobile phone software

STEP18. Run examples.modem.SendMessage

Best of luck

Tuesday, February 11, 2014

ETL : Moving Data From Oracle To PostgreSQL

  1. Install and Configure Oracle Instant Client for linux
  2. I'm going to cover this step in quite a bit of detail; even though there are pre-built .rpm packages available from oracle for both i386 and x86_64 platforms, getting everything in running can be quite a mission if you omit a few vital steps...
    1. Download the Oracle Instant Client .rpm's from Oracle here
      Make sure you get the following :
      • oracle-instantclient11.2-basic-11.2.0.3.0-1.x86_64.rpm
      • oracle-instantclient11.2-sqlplus-11.2.0.3.0-1.x86_64.rpm
      • oracle-instantclient11.2-devel-11.2.0.3.0-1.x86_64.rpm

    2. Once you've downloaded the .rpm's, you can install them either with rpm or yum (I generally prefer yum as it will automatically handle resolving and installing dependencies for you).

      $ sudo yum -y localinstall oracle-instantclient*.rpm
    3. Update the ld library cache

    4. $ cat > /etc/ld.so.conf.d/oracle.conf <<EOF
      /usr/lib/oracle/11.2/client64/lib
      EOF

      $ ldconfig

    5. Set the Oracle environment variable

      $ cat >> /etc/bashrc <<EOF
      export TNS_ADMIN=$HOME
      export EDITOR=vim
      export ORACLE_HOME=/usr/lib/oracle/11.2/client64
      EOF

      $ source /etc/bashrc

    6. Place your tnsnames.ora file in your home directory
    7. You can test your oracle client installation by running sqlplus

      $ sqlplus64 /nolog
  3. Install Perl-DBD-Oracle

    1. Before we start ensure that you have the following packages installed
      • perl
      • perl-CPAN
      • perl-DBD
      • perl-DBI
      • gcc
      • libaio
      • libaio-devel

    2. Download the Perl-DBD-Oracle Source with CPAN


    $ perl -MCPAN -e shell
    CPAN> get DBD::Oracle
    CPAN> exit

    $ cd ~/.cpan/build/DBD-Oracle*
    $ perl Makefile.PL
    $ make
    $ make install

  4. Install and Configure ora2pg

    1. Download ora2pg here
    2. Uncompress the tar archive

      $ tar xjvf ora2pg*.tar.bz2 && rm ora2pg*.tar.bz2
    3. Build ora2pg

    4. $ cd ora2pg*
      $ perl Makefile.PL
      $ make && make install

    5. Configure ora2pg

    6. $ gedit /etc/ora2pg/ora2pg.conf
      Look out for the following configuration properties
      • ORACLE_HOME - you can set this to $ORACLE_HOME
      • ORACLE_DSN - set this to the desired entry in you tnsnames.ora
      • ORACLE_USER - the oracle username
      • ORACLE_PWD - the oracle password
      • USER_GRANTS - set this to 1 if you're not connecting as a DBA role
      • TYPE - set this to DATA if you only want to export data and ignore structure
    7. run ora2pg
    8. Simply run ora2pg from the command line, by default your data will be dumped to a file called output.sql in the current directory.

Wednesday, February 5, 2014

Not using Enterprise Library

I have resolved problem by not using Enterprise Library. Instead my .NET functions RequestLock and ReleaseLock were modified to use dotConnect for Oracle classes directly. i.e OracleConnection and OracleCommand.

The problem with Enterprise Library was how those classes were managing connections.

The code you see below now works. The _DatabaseConnection object is created when the application starts up. It opens up a connection (OracleConnection object) that is subsequently referenced by these functions. The key is the connection does not go away and therefore the locks are maintained.

Public Function RequestLock(ByVal lockName As String) As Boolean
Dim isLocked As Boolean = True

Try
Dim conn As OracleConnection
Dim cmd As OracleCommand
Dim param As OracleParameter

conn = _DatabaseConnection.GetConnection()

cmd = conn.CreateCommand()
cmd.CommandText = "REQUEST_LOCK"
cmd.CommandType = CommandType.StoredProcedure

param = cmd.CreateParameter()
param.Direction = ParameterDirection.Input
param.OracleDbType = Devart.Data.Oracle.OracleDbType.VarChar
param.Value = lockName
param.ParameterName = "i_lock_name"

cmd.Parameters.Add(param)
cmd.ExecuteNonQuery()

Catch ex As Exception
isLocked = False
End Try

Return isLocked
End Function

Public Function ReleaseLock(ByVal lockName As String) As Boolean
Dim isReleased As Boolean = True

Try
Dim conn As OracleConnection
Dim cmd As OracleCommand
Dim param As OracleParameter

conn = _DatabaseConnection.GetConnection()

cmd = conn.CreateCommand()
cmd.CommandText = "RELEASE_LOCK"
cmd.CommandType = CommandType.StoredProcedure

param = cmd.CreateParameter()
param.Direction = ParameterDirection.Input
param.OracleDbType = Devart.Data.Oracle.OracleDbType.VarChar
param.Value = lockName
param.ParameterName = "i_lock_name"

cmd.Parameters.Add(param)
cmd.ExecuteNonQuery()

Catch ex As Exception
isReleased = False
End Try

Return isReleased
End Function

Oracle Locking Using DBMS_LOCK

We are using the folling Oracle functions to perform row level locking. These functions are called from our WinForms application using dotConnect for Oracle and Enterprise Library 4.1.

-- internal function to get a lock handle
-- (private for use by REQUEST_LOCK and RELEASE_LOCK)
CREATE OR REPLACE FUNCTION GET_HANDLE (i_lock_name IN VARCHAR2) RETURN VARCHAR2 IS
PRAGMA AUTONOMOUS_TRANSACTION;
v_lock_handle VARCHAR2(128);
BEGIN
DBMS_LOCK.ALLOCATE_UNIQUE (
lockname => i_lock_name,
lockhandle => v_lock_handle,
expiration_secs => 864000); -- 10 days
RETURN v_lock_handle;
END GET_HANDLE;

CREATE OR REPLACE PROCEDURE REQUEST_LOCK (i_lock_name IN VARCHAR2) IS
v_lock_status NUMBER;
BEGIN
v_lock_status := DBMS_LOCK.REQUEST(
lockhandle => GET_HANDLE(i_lock_name),
lockmode => DBMS_LOCK.X_MODE, -- eXclusive
timeout => 0, -- do not wait
release_on_commit => FALSE);
CASE v_lock_status
WHEN 0 THEN NULL;
WHEN 2 THEN RAISE_APPLICATION_ERROR(-20000,'deadlock detected');
WHEN 4 THEN RAISE_APPLICATION_ERROR(-20000,'lock already obtained');
ELSE RAISE_APPLICATION_ERROR(-20000,'request lock failed - ' || TO_CHAR(v_lock_status));
END CASE;
END REQUEST_LOCK;

-- wrapper to release a lock
CREATE OR REPLACE PROCEDURE RELEASE_LOCK (i_lock_name IN VARCHAR2) IS
v_lock_status NUMBER;
BEGIN
v_lock_status := DBMS_LOCK.RELEASE(
lockhandle => GET_HANDLE(i_lock_name));
IF v_lock_status > 0 THEN
RAISE_APPLICATION_ERROR(-20000,'release lock failed - ' || TO_CHAR(v_lock_status));
END IF;
END RELEASE_LOCK;


These are the .Net Fuctions that call the Oracle lock functions mentioned above:

Public Function RequestLock(ByVal lockName As String) As Boolean
Dim isLocked As Boolean = True

Try
Dim db As Database
Dim cmd As DbCommand

db = DatabaseFactory.CreateDatabase()
cmd = db.GetStoredProcCommand("REQUEST_LOCK", lockName)
cmd.Connection = _DatabaseConnection.GetConnection()
db.ExecuteNonQuery(cmd)

Catch ex As Exception
isLocked = False
End Try

Return isLocked
End Function

Public Function ReleaseLock(ByVal lockName As String) As Boolean
Dim isReleased As Boolean = True

Try
Dim db As Database
Dim cmd As DbCommand

db = DatabaseFactory.CreateDatabase()
cmd = db.GetStoredProcCommand("RELEASE_LOCK", lockName)
cmd.Connection = _DatabaseConnection.GetConnection()

db.ExecuteNonQuery(cmd)

Catch ex As Exception
isReleased = False
End Try

Return isReleased
End Function


The problem I am having is creating a lock and having that lock persist. I have tried Direct and OCI modes and used the pooling and non-pooling options in connection string and nothing I try will work for us. The Direct/Pooling options works for a period of time but if we monitor the Oracle sessions the connections that hold the locks are closed. Obviously when the connections are dropping the locks disappear as well.

Sample connect strings we have tried.

providerName="dotConnect for Oracle" />
providerName="dotConnect for Oracle" />
providerName="dotConnect for Oracle" />
providerName="dotConnect for Oracle" />