6 Troubleshooting AWT Cache Groups

Enable AWT performance monitoring

Use the ttCacheAwtMonitorConfig built-in procedure to enable monitoring for AWT cache groups. Ensure that the replication agent is running before calling ttCacheAwtMonitorConfig.

The following example enables monitoring and sets the sampling frequency to 16. A sampling factor of 16 is recommended for accuracy and performance.

Command> CALL ttCacheAwtMonitorConfig('ON',16);
< ON, 16 >
1 row found.

Display AWT performance results with the ttRepAdmin utility

Use the ttRepAdmin utility with the -awtmoninfo and -showstatus commands to display the monitoring results. The AWT monitoring statistics include the following:

• TimesTen processing time: The total number of milliseconds spent in processing AWT transaction data since monitoring was enabled.

• Oracle bookmark management time: The total number of milliseconds spent in managing AWT metadata on Oracle since monitoring was enabled.

• Oracle execute time: The total number of milliseconds spent in OCI preparation, binding and execution for AWT SQL operations since monitoring was enabled. This statistic includes network latency between TimesTen and Oracle.

• Oracle commit time: The total number of milliseconds spent in committing AWT updates on Oracle since monitoring was enabled. This statistic includes network latency between TimesTen and Oracle.

• Time since monitoring was started.

• Total number of TimesTen row operations: The total number of rows updated in AWT cache groups since monitoring was enabled.

• Total number of TimesTen transactions: The total number of transactions in AWT cache groups since monitoring was enabled.

• Total number of flushes to Oracle: The total number of times that TimesTen data has been sent to Oracle.

• The percentage of time spent on TimesTen processing, Oracle bookmark management, Oracle execution and Oracle commits.

For example:

ttRepAdmin -showstatus -awtmoninfo myDSN

[other -showstatus output]
...
AWT Monitoring statistics
-------------------------
 TimesTen processing time : 8443.424000 millisecs (44.064958 )
   Oracle execute  (SQL execution) time : 8930.320000 millisecs (46.605994 )
   Oracle execute (PL/SQL execution) time : 0.000000 millisecs (0 )
   Oracle commit time : 1787.568000 millisecs (9.329048 )
   Time since monitoring was started: 21954.410000 millisecs
   CacheAwtMethod mode : 0
   Cache-connect Operational Stats :
      SQL Operations sent to Oracle : 143556
           Number of update operations : 58
           Number of update batches    : 58
           Number of insert operations : 143498
           Number of insert batches    : 1146
           Number of delete operations : 0
           Number of delete batches    : 0
           Total number of batches sent: 1204
           Number of bytes sent : 4769094
      Number of TimesTen Transactions sent to Oracle (includes retries) :
            143556
      Number of retries on TimesTen due to errors on Oracle : 0
      Number of round trips to Oracle (includes executes, commits and
            rollbacks) : 2290
      Number of commits on Oracle : 1086
      Number of rollbacks on Oracle : 0
      Number of rxbatches: 244
      Number of rxskips: 0
 
Peer name         Host name                 Port    State  Proto
----------------  ------------------------ ------  ------- -----
_ORACLE           MYHOST                    Auto   Start      27
 
Last Msg Sent Last Msg Recv Latency TPS     RecordsPS Logs
------------- ------------- ------- ------- --------- ----
00:00:00      -                2.35    6311      6311    1

Using system tables to monitor AWT operations

To determine whether asynchronous writethrough operations are keeping up with the rate of updates on cache tables in AWT cache groups, query the LAST_LOG_FILE, REPHOLD_LOG_FILE and REPHOLD_LOG_OFF columns of the SYS.MONITOR system table. If the difference between the value in the LAST_LOG_FILE column and the value in the REPHOLD_LOG_FILE column is increasing over time, and the value in the REPHOLD_LOG_OFF column is increasing slowly or not changing, then the tables are being updated at a faster rate than the updates are being replicated.

Then run a ttRepAdmin -receiver -list utility command and find the row where _ORACLE is in the Peer name field. View the values in the Last Msg Sent, Last Msg Recv, Latency and TPS fields within the same row to determine if the replication activity that is falling behind is asynchronous writethrough operations.

For more information about monitoring log files, see "Monitoring accumulation of transaction log files" in Oracle TimesTen In-Memory Database Operations Guide.

The SYS.SYSTEMSTATS system table has a variety of statistics about AWT performance. See "SYS.SYSTEMSTATS" in Oracle TimesTen In-Memory Database System Tables and Views Reference.

Problems on the Oracle Database

If an AWT cache group cannot propate updates to the Oracle database, the causes can include network failure, an Oracle database failure, I/O bottlenecks and CPU bottlenecks. See Oracle Database Performance Tuning Guide for information about analyzing Oracle Database performance.

AWT propagation to the Oracle database can also fail because the replication agent is stopped. See "Unable to start or stop replication agent" and "Replication does not work".

UNLOAD CACHE GROUP requires row updates to complete propagation

Execution of the UNLOAD CACHE GROUP statement for an AWT cache group waits until updates on the rows have been propagated to the Oracle database. However, the UNLOAD CACHE GROUP statement may fail if data propagation to Oracle is stopped for any reason, such as network failure, the Oracle database is down, or the replication agent is stopped.

Permanent Oracle errors reported by TimesTen

Insert, update, or delete errors that occur while applying changes to Oracle are saved in an error file located in the database directory with the following name:

<DataSourceName>.awterrs

Errors reported to this file are permanent errors. TimesTen does not retry the transaction. The errors may be reported in the AWT error file long after the commit to TimesTen occurs.

The format of the messages in the AWT error file is similar to those generated for conflict and transaction errors in replication, as shown in Example 6-1. Oracle error messages are also reported in the support log and the user log. The Oracle errors are prefixed with 'ORA-.' The TimesTen error messages are prefixed with 'TT.' For example, in Example 6-1, TimesTen errors 5210 and 5025 are shown as TT5210 and TT5025. These can be looked up in the Oracle TimesTen In-Memory Database Error Messages and SNMP Traps guide.

Error occurred 14:48:55 on 03-22-2011
Datastore: c:\temp\cgDSN
Oracle Id: system1
Transmitting name: cgDSN
Error message: 
  TT5210: Oracle unique constraint violation error in OCIStmtExecute(): 
ORA-00001: unique constraint (GUSER.SYS_C00357240) violated rc = -1 -- file
 "bdbTblH.c", lineno 1205, procedure "ttBDbStmtForce()"
  TT5025: Commit failure in Oracle. Transaction must be rolled back in TimesTen. 
-- file "bdbConnect.c", lineno 885, procedure "ttBDbXact()"

Operation that caused the error:
Insert into table TESTUSER.T1 <9,1000>

Failed transaction:
Insert into table TESTUSER.T1 <9, 1000>
End of failed transaction

May 04 18:12:36 HOST1 TimesTen Replication 11.2[2136]: 
[Err ] DEFAULT:meta.c(639): 
TT16062: Failed to compile command:
select p.commit_timestamp, p.commit_seqnum, p.protocol from owner1.TT_06_REPPEERS 
p where p.replication_name = :rname and p.replication_owner = :rowner and 
p.tt_store_id = :oid and p.subscriber_id = :sid

May 04 18:12:36 HOST1 TimesTen Replication 11.2[2136]: 
[Err ] DEFAULT:meta.c(639): 
TT5221: TT5221: Oracle syntax error in OCIStmtExecute(): 
ORA-00942: table or view does not exist rc = -1 -- file "bdbStmt.c", lineno 1041, 
procedure "getOraOutTypesNLengths()"

1. Stop the replication agent.

2. Drop and re-create the cache group.

3. Restart the replication agent.

Transient Oracle errors reported by TimesTen

The support log for databases with AWT cache groups may contain Oracle errors if the replication agent encounters a problem on the Oracle database. If the replication agent encounters one of these errors, AWT rolls back the transaction and retries it. If the support log becomes full, the oldest messages are deleted and replaced by new messages.

The Oracle errors in the support log are considered transient because AWT retries the transaction.

Some transient errors indicate an underlying problem on the Oracle database must be solved before AWT operations can continue. For example:

ORA-01536: space quota exceeded for tablespace
ORA-01034: ORACLE not available

After the underlying problem has been fixed, AWT retries the operation.

For more information about the Oracle errors, see Oracle Database Error Messages for the Oracle release you are using.

The following Oracle errors are transient:

ORA-00018: maximum number of sessions exceeded
ORA-00019: maximum number of session licenses exceeded
ORA-00020: maximum number of processes (%s) exceeded
ORA-00025: failed to allocate %s
ORA-00028: your session has been killed
ORA-00038: Cannot create session: server group belongs to another user
ORA-00051: timeout occurred while waiting for a resource
ORA-00052: maximum number of enqueue resources (%s) exceeded
ORA-00053: maximum number of enqueues exceeded
ORA-00054: resource busy and acquire with NOWAIT specified
ORA-00055: maximum number of DML locks exceeded
ORA-00057: maximum number of temporary table locks exceeded
ORA-00058: DB_BLOCK_SIZE must be %s to mount this database (not %s)
ORA-00059: maximum number of DB_FILES exceeded
ORA-00060: deadlock detected while waiting for resource
ORA-00063: maximum number of LOG_FILES exceeded
ORA-00064: object is too large to allocate on this O/S (%s,%s)
ORA-00099: timed out while waiting for resource, potential PDML deadlock
ORA-00104: deadlock detected; all public servers blocked waiting for resources
ORA-00107: failed to connect to ORACLE listener process
ORA-00115: connection refused; dispatcher connection table is full
ORA-00125: connection refused; invalid presentation
ORA-00126: connection refused; invalid duplicity
ORA-00284: recovery session still in progress
ORA-00370: potential deadlock during kcbchange operation
ORA-00371: not enough shared pool memory
ORA-00376: file %s cannot be read at this time
ORA-00379: no free buffers available in buffer pool %s for block size %sK
ORA-00384: Insufficient memory to grow cache
ORA-00568: Maximum number of interrupt handlers exceeded
ORA-00579: osndnt: server received malformed connection request
ORA-00600: internal error code, arguments: [%s], [%s], [%s], [%s], [%s], [%s], [%s], [%s]
ORA-00603: ORACLE server session terminated by fatal error
ORA-01000: maximum open cursors exceeded
ORA-01012: not logged on
ORA-01014: ORACLE shutdown in progress
ORA-01019: unable to allocate memory in the user side
ORA-01031: insufficient privileges
ORA-01033: ORACLE initialization or shutdown in progress
ORA-01034: ORACLE not available
ORA-01035: ORACLE only available to users with RESTRICTED SESSION privilege
ORA-01037: maximum cursor memory exceeded
ORA-01046: cannot acquire space to extend context area
ORA-01073: fatal connection error: unrecognized call type
ORA-01089: immediate shutdown in progress - no operations are permitted
ORA-01090: shutdown in progress - connection is not permitted
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-01094: ALTER DATABASE CLOSE in progress. Connections not permitted
ORA-01109: database not open
ORA-01147: SYSTEM tablespace file %s is offline
ORA-01154: database busy. Open, close, mount, and dismount not allowed now
ORA-01155: the database is being opened, closed, mounted or dismounted
ORA-01219: database not open: queries allowed on fixed tables/views only
ORA-01237: cannot extend datafile %s
ORA-01456: may not perform insert/delete/update operation inside a READ ONLY transaction
ORA-01536: space quota exceeded for tablespace '%s'
ORA-01539: tablespace '%s' is not online
ORA-01542: tablespace '%s' is offline, cannot allocate space in it
ORA-01562: failed to extend rollback segment number %s
ORA-01573: shutting down instance, no further change allowed
ORA-01628: max # extents (%s) reached for rollback segment %s
ORA-01629: max # extents (%s) reached saving undo for tablespace %s
ORA-01630: max # extents (%s) reached in temp segment in tablespace %s
ORA-01631: max # extents (%s) reached in table %s.%s
ORA-01632: max # extents (%s) reached in index %s.%s
ORA-01650: unable to extend rollback segment %s by %s in tablespace %s
ORA-01651: unable to extend save undo segment by %s for tablespace %s
ORA-01652: unable to extend temp segment by %s in tablespace %s
ORA-01653: unable to extend table %s.%s by %s in tablespace %s
ORA-01654: unable to extend index %s.%s by %s in tablespace %s
ORA-01655: unable to extend cluster %s.%s by %s in tablespace %s
ORA-01656: max # extents (%s) reached in cluster %s.%s
ORA-01658: unable to create INITIAL extent for segment in tablespace %s
ORA-01659: unable to allocate MINEXTENTS beyond %s in tablespace %s
ORA-01680: unable to extend LOB segment by %s in tablespace %s
ORA-01681: max # extents (%s) reached in LOB segment in tablespace %s
ORA-01683: unable to extend index %s.%s partition %s by %s in tablespace %s
ORA-01684: max # extents (%s) reached in table %s.%s partition %s
ORA-01685: max # extents (%s) reached in index %s.%s partition %s
ORA-01686: max # files (%s) reached for the tablespace %s
ORA-01688: unable to extend table %s.%s partition %s by %s in tablespace %s
ORA-01691: unable to extend lob segment %s.%s by %s in tablespace %s
ORA-01692: unable to extend lob segment %s.%s partition %s by %s in tablespace %s
ORA-01693: max # extents (%s) reached in lob segment %s.%s
ORA-01694: max # extents (%s) reached in lob segment %s.%s partition %s
ORA-03113: end-of-file on communication channel
ORA-03114: not connected to ORACLE
ORA-03134: Connections to this server version are no longer supported.
ORA-03135: connection lost contact
ORA-03136: inbound connection timed out
ORA-03232: unable to allocate an extent of %s blocks from tablespace %s
ORA-03233: unable to extend table %s.%s subpartition %s by %s in tablespace %s
ORA-03234: unable to extend index %s.%s subpartition %s by %s in tablespace %s
ORA-03235: max # extents (%s) reached in table %s.%s subpartition %s
ORA-03236: max # extents (%s) reached in index %s.%s subpartition %s
ORA-03237: Initial Extent of specified size cannot be allocated
ORA-03238: unable to extend LOB segment %s.%s subpartition %s by %s in tablespace %s
ORA-03239: maxextents (%s) reached in LOB segment %s.%s subpartition %s
ORA-04020: deadlock detected while trying to lock object %s%s%s%s%s
ORA-06019: NETASY: invalid login (connect) string
ORA-06021: NETASY: connect failed
ORA-06030: NETDNT: connect failed, unrecognized node name
ORA-06031: NETDNT: connect failed, unrecognized object name
ORA-06032: NETDNT: connect failed, access control data rejected
ORA-06033: NETDNT: connect failed, partner rejected connection
ORA-06034: NETDNT: connect failed, partner exited unexpectedly
ORA-06035: NETDNT: connect failed, insufficient resources
ORA-06036: NETDNT: connect failed, no response from object
ORA-06037: NETDNT: connect failed, node unreachable
ORA-06039: NETDNT: connect failed
ORA-06040: NETDNT: invalid login (connect) string
ORA-06108: NETTCP: connect to host failed
ORA-06113: NETTCP: Too many connections
ORA-06114: NETTCP: SID lookup failure
ORA-06143: NETTCP: maximum connections exceeded
ORA-06315: IPA: Invalid connect string
ORA-06316: IPA: Invalid database SID
ORA-06317: IPA: Local maximum number of users exceeded
ORA-06318: IPA: Local maximum number of connections exceeded
ORA-06319: IPA: Remote maximum number of users exceeded
ORA-06320: IPA: Remote maximum number of connections exceeded
ORA-06404: NETCMN: invalid login (connect) string
ORA-06413: Connection not open.
ORA-10435: enable tracing of global enqueue service deadlock detetction
ORA-10626: specify timeout for online index rebuild to wait for DML
ORA-10906: Unable to extend segment after insert direct load
ORA-12150: TNS:unable to send data
ORA-12151: TNS:received bad packet type from network layer
ORA-12152: TNS:unable to send break message
ORA-12153: TNS:not connected
ORA-12154: TNS:could not resolve service name
ORA-12155: TNS:received bad datatype in NSWMARKER packet
ORA-12156: TNS:tried to reset line from incorrect state
ORA-12157: TNS:internal network communication error
ORA-12158: TNS:could not initialize parameter subsystem
ORA-12159: TNS:trace file not writeable
ORA-12160: TNS:internal error: Bad error number
ORA-12161: TNS:internal error: partial data received
ORA-12162: TNS:service name is incorrectly specified
ORA-12163: TNS:connect descriptor is too long
ORA-12166: TNS:Client can not connect to HO agent.
ORA-12168: TNS:Unable to contact Directory Server.
ORA-12169: TNS:Net service name given as connect identifier is too long
ORA-12170: TNS:Connect timeout occurred
ORA-12171: TNS:could not resolve connect identifier: %s
ORA-12196: TNS:received an error from TNS
ORA-12197: TNS:keyword-value resolution error
ORA-12198: TNS:could not find path to destination
ORA-12200: TNS:could not allocate memory
ORA-12201: TNS:encountered too small a connection buffer
ORA-12202: TNS:internal navigation error
ORA-12203: TNS:unable to connect to destination
ORA-12204: TNS:received data refused from an application
ORA-12205: TNS:could not get failed addresses
ORA-12206: TNS:received a TNS error during navigation
ORA-12207: TNS:unable to perform navigation
ORA-12208: TNS:could not find the TNSNAV.ORA file
ORA-12209: TNS:encountered uninitialized global
ORA-12210: TNS:error in finding Navigator data
ORA-12211: TNS:needs PREFERRED_CMANAGERS entry in TNSNAV.ORA
ORA-12212: TNS:incomplete PREFERRED_CMANAGERS binding in TNSNAV.ORA
ORA-12213: TNS:incomplete PREFERRED_CMANAGERS binding in TNSNAV.ORA
ORA-12214: TNS:missing local communities entry in TNSNAV.ORA
ORA-12215: TNS:poorly formed PREFERRED_NAVIGATORS Addresses in TNSNAV.ORA
ORA-12216: TNS:poorly formed PREFERRED_CMANAGERS addresses in TNSNAV.ORA
ORA-12217: TNS:could not contact PREFERRED_CMANAGERS in TNSNAV.ORA
ORA-12218: TNS:unacceptable network configuration data
ORA-12219: TNS:missing community name from address in ADDRESS_LIST
ORA-12221: TNS:illegal ADDRESS parameters
ORA-12222: TNS:no such protocol adapter
ORA-12223: TNS:internal limit restriction exceeded
ORA-12224: TNS:no listener
ORA-12225: TNS:destination host unreachable
ORA-12226: TNS:operating system resource quota exceeded
ORA-12227: TNS:syntax error
ORA-12228: TNS:protocol adapter not loadable
ORA-12229: TNS:Interchange has no more free connections
ORA-12230: TNS:Severe Network error ocurred in making this connection
ORA-12231: TNS:No connection possible to destination
ORA-12232: TNS:No path available to destination
ORA-12233: TNS:Failure to accept a connection
ORA-12235: TNS:Failure to redirect to destination
ORA-12236: TNS:protocol adapter not loaded
ORA-12316: syntax error in database link's connect string
ORA-12326: database %s is closing immediately;  no operations are permitted
ORA-12329: database %s is closed;  no operations are permitted
ORA-12500: TNS:listener failed to start a dedicated server process
ORA-12501: TNS:listener failed to spawn process
ORA-12502: TNS:listener received no CONNECT_DATA from client
ORA-12504: TNS:listener was not given the SID in CONNECT_DATA
ORA-12505: TNS:listener could not resolve SID given in connect descriptor
ORA-12506: TNS:listener was not given the ALIAS in CONNECT_DATA
ORA-12507: TNS:listener could not resolve ALIAS given
ORA-12508: TNS:listener could not resolve the COMMAND given
ORA-12509: TNS:listener failed to redirect client to service handler
ORA-12510: TNS:database temporarily lacks resources to handle the request
ORA-12511: TNS:service handler found but it is not accepting connections
ORA-12512: TNS:service handler found but it has not registered a 
redirect address
ORA-12513: TNS:service handler found but it has registered for a 
different protocol
ORA-12514: TNS:listener could not resolve SERVICE_NAME given in 
connect descriptor
ORA-12515: TNS:listener could not find a handler for this presentation
ORA-12516: TNS:listener could not find available handler with matching 
protocol stack
ORA-12517: TNS:listener could not find service handler supporting direct handoff
ORA-12518: TNS:listener could not hand off client connection
ORA-12519: TNS:no appropriate service handler found
ORA-12520: TNS:listener could not find available handler for requested 
type of server
ORA-12521: TNS:listener could not resolve INSTANCE_NAME given in 
connect descriptor
ORA-12522: TNS:listener could not find available instance with given 
INSTANCE_ROLE
ORA-12523: TNS:listener could not find instance appropriate for the 
client connection
ORA-12524: TNS:listener could not resolve HANDLER_NAME given in 
connect descriptor
ORA-12525: TNS:listener has not received client's request in time allowed
ORA-12526: TNS:listener: all appropriate instances are in restricted mode
ORA-12527: TNS:listener: all instances are in restricted mode or blocking 
new connections
ORA-12528: TNS:listener: all appropriate instances are blocking new connections
ORA-12529: TNS:connect request rejected based on current filtering rules
ORA-12531: TNS:cannot allocate memory
ORA-12532: TNS:invalid argument
ORA-12533: TNS:illegal ADDRESS parameters
ORA-12534: TNS:operation not supported
ORA-12535: TNS:operation timed out
ORA-12536: TNS:operation would block
ORA-12537: TNS:connection closed
ORA-12538: TNS:no such protocol adapter
ORA-12539: TNS:buffer over- or under-flow
ORA-12540: TNS:internal limit restriction exceeded
ORA-12541: TNS:no listener
ORA-12542: TNS:address already in use
ORA-12543: TNS:destination host unreachable
ORA-12544: TNS:contexts have different wait/test functions
ORA-12545: Connect failed because target host or object does not exist
ORA-12546: TNS:permission denied
ORA-12547: TNS:lost contact
ORA-12549: TNS:operating system resource quota exceeded
ORA-12550: TNS:syntax error
ORA-12551: TNS:missing keyword
ORA-12552: TNS:operation was interrupted
ORA-12554: TNS:current operation is still in progress
ORA-12555: TNS:permission denied
ORA-12556: TNS:no caller
ORA-12557: TNS:protocol adapter not loadable
ORA-12558: TNS:protocol adapter not loaded
ORA-12560: TNS:protocol adapter error
ORA-12561: TNS:unknown error
ORA-12562: TNS:bad global handle
ORA-12564: TNS:connection refused
ORA-12566: TNS:protocol error
ORA-12569: TNS:packet checksum failure
ORA-12570: TNS:packet reader failure
ORA-12571: TNS:packet writer failure
ORA-12574: TNS:redirection denied
ORA-12582: TNS:invalid operation
ORA-12583: TNS:no reader
ORA-12585: TNS:data truncation
ORA-12589: TNS:connection not bequeathable
ORA-12590: TNS:no I/O buffer
ORA-12591: TNS:event signal failure
ORA-12592: TNS:bad packet
ORA-12593: TNS:no registered connection
ORA-12595: TNS:no confirmation
ORA-12596: TNS:internal inconsistency
ORA-12600: TNS: string open failed
ORA-12602: TNS: Connection Pooling limit reached
ORA-12606: TNS: Application timeout occurred
ORA-12607: TNS: Connect timeout occurred
ORA-12608: TNS: Send timeout occurred
ORA-12609: TNS: Receive timeout occurred
ORA-12612: TNS:connection is busy
ORA-12615: TNS:preempt error
ORA-12623: TNS:operation is illegal in this state
ORA-12624: TNS:connection is already registered
ORA-12636: Packet send failed
ORA-12637: Packet receive failed
ORA-12829: Deadlock - itls occupied by siblings at block %s of file %s
ORA-12993: tablespace '%s' is offline, cannot drop column
ORA-14117: partition resides in offlined tablespace
ORA-14268: subpartition '%s' of the partition resides in offlined tablespace
ORA-16000: database open for read-only access
ORA-16003: standby database is restricted to read-only access
ORA-16403: shutdown in progress - remote connection is not permitted
ORA-16724: the intended state for resource has been set to OFFLINE
ORA-16903: Unable to connect to database
ORA-16914: Missing connect string. Try \"help\"
ORA-18014: deadlock detected while waiting for resource %s
ORA-21521: exceeded maximum number of connections in OCI (object mode only)
ORA-21522: attempted to use an invalid connection in OCI (object mode only)
ORA-23317: a communication failure has occurred
ORA-24401: cannot open further connections
ORA-24418: Cannot open further sessions.
ORA-24778: cannot open connections
ORA-25400: must replay fetch
ORA-25401: can not continue fetches
ORA-25402: transaction must roll back
ORA-25403: could not reconnect
ORA-25405: transaction status unknown
ORA-25407: connection terminated
ORA-25408: can not safely replay call
ORA-25409: failover happened during the network operation,cannot continue
ORA-25425: connection lost during rollback
ORA-29306: datafile %s is not online
ORA-30006: resource busy; acquire with WAIT timeout expired
ORA-30036: unable to extend segment by %s in undo tablespace '%s'
ORA-30040: Undo tablespace is offline
ORA-31443: deadlock detected while acquiring lock on %s
ORA-37013: (XSACQUIRE_DEADLOCK) Cannot wait to acquire object %j since doing so 
would cause a deadlock.
ORA-44317: database open read-only