15 Message Delivery Properties

This chapter explains the options available for configuration of the property files for user exit properties and Java application properties.

This chapter includes the following sections:

15.1 User Exit Properties

The following properties set the log files and the type of logging.

15.1.1 Logging Properties

Logging is controlled by the following properties.

15.1.1.1 gg.log

Specifies the type of logging that is to be used. The default implementation for the Oracle GoldenGate Adapters is the JDK option. This is the built-in Java logging called java.util.logging (JUL). The other logging options are log4j or logback.

For example, to set the type of logging to log4j:

gg.log=log4j 

The log file is created in the report subdirectory of the installation. The default log file name includes the group name of the associated Extract and the file extension is .log.

15.1.1.2 gg.log.level

Specifies the overall log level for all modules. The syntax is:

gg.log.level={ERROR|WARN|INFO|DEBUG}

The log levels are defined as follows:

  • ERROR – Only write messages if errors occur

  • WARN – Write error and warning messages

  • INFO – Write error, warning and informational messages

  • DEBUG – Write all messages, including debug ones.

The default logging level is INFO. The messages in this case will be produced on startup, shutdown and periodically during operation. If the level is switched to DEBUG, large volumes of messages may occur which could impact performance. For example, the following sets the global logging level to INFO:

# global logging level
gg.log.level=INFO

15.1.1.3 gg.log.file

Specifies the path to the log file. The syntax is:

gg.log.file=path_to_file

Where the path_to_file is the fully defined location of the log file. This allows a change to the name of the log, but you must include the Extract name if you have more than one Extract to avoid one overwriting the log of the other.

15.1.1.4 gg.log.classpath

Specifies the class path to the jars used to implement logging.

gg.log.classpath=path_to_jars

15.1.2 General Properties

The following properties apply to all writer type user exits and are not specific to the user exit.

15.1.2.1 goldengate.userexit.writers

Specifies the name of the writer. This is always jvm and should not be modified.

For example:

goldengate.userexit.writers=jvm

All other properties in the file should be prefixed by the writer name, jvm.

15.1.2.2 goldengate.userexit.chkptprefix

Specifies a string value for the prefix added to the checkpoint file name. For example:

goldengate.userexit.chkptprefix=javaue_

15.1.2.3 goldengate.userexit.nochkpt

Disables or enables the user exit checkpoint file. The default is false, the checkpoint file is enabled. Set this property to true if transactions are supported and enabled on the target.

For exampe, if JMS is the target and JMS local transactions are enabled (the default), set goldengate.userexit.nochkpt=true to disable the user exit checkpoint file. If JMS transactions are disabled by setting localTx=false on the handler, the user exit checkpoint file should be enabled by setting goldengate.userexit.nochkpt=false.

goldengate.userexit.nochkpt={true|false}

15.1.2.4 goldengate.userexit.usetargetcols

Specifies whether or not mapping to target columns is allowed. The default is false, no target mapping.

goldengate.userexit.usetargetcols={true|false}

15.1.3 JVM boot Options

The following options configure the Java Runtime Environment. In particular, this is where the maximum memory the JVM can use is specified; if you see Java out-of-memory errors, edit these settings.

15.1.3.1 jvm.bootoptions

Specifies the classpath and boot options that will be applied when the user exit starts up the JVM. The path needs colon (:) separators for UNIX/Linux and semicolons (;) for Windows. This is where to specify various options for the JVM, such as heap size and class path; for example:

  • -Xms: initial java heap size

  • -Xmx: maximum java heap size

  • -Djava.class.path: class path specifying location of at least the main application jar, ggjava.jar. Other jars, such as JMS provider jars, may also be specified here as well; alternatively, these may be specified in the Java application properties file.

  • -verbose:jni: run in verbose mode (for JNI)

For example (all on a single line):

jvm.bootoptions= -Djava.class.path=ggjava/ggjava.jar 
-Dlog4j.configuration=my-log4j.properties -Xmx512m

The log4j.configuration property can be a fully qualified URL to a log4j properties file; by default this file is searched for in the class path. You may use your own log4j configuration, or one of the preconfigured log4j settings: log4j.properties (default level of logging), debug-log4j.properties (debug logging) or trace-log4j.properties (very verbose logging).

15.1.4 Statistics and Reporting

The use of the user exit causes Extract to assume that the records handled by the exit are ignored. This causes the standard Oracle GoldenGate reporting to be incomplete. Oracle GoldenGate for Java adds its own reporting to handle this issue.

Statistics can be reported every t seconds or every n records - or if both are specified, whichever criteria is met first.

There are two sets of statistics recorded: those maintained by the User Exit shared library (on the C side) and those obtained from the Java libraries. The reports received from the Java side are formatted and returned by the individual handlers.

The User Exit statistics include the total number of operations, transactions and corresponding rates.

15.1.4.1 jvm.stats.display

Controls the output of statistics to the Oracle GoldenGate report file and to the user exit log files.

The following example outputs these statistics.

jvm.stats.display=true

15.1.4.2 jvm.stats.full

Controls the output of statistics from the Java side, in addition to the statistics from the C side.

Java side statistics are more detailed but also involve some additional overhead, so if statistics are reported often and a less detailed summary is adequate, it is recommended that stats.full property is set to false.

The following example will output Java statistics in addition to C.

jvm.stats.full=true

15.1.4.3 jvm.stats.time | jvm.stats.numrecs

Specifies a time interval, in seconds or a number of records, after which statistics will be reported. The default is to report statistics every hour or every 10000 records (which ever occurs first).

For example, to report ever 10 minutes or every 1000 records, specify:

jvm.stats.time=600
jvm.stats.numrecs=1000

The Java application statistics are handler-dependent:

  • For the all handlers, there is at least the total elapsed time, processing time, number of operations, transactions;

  • For the JMS handler, there is additionally the total number of bytes received and sent.

  • The report can be customized using a template.

15.2 Java Application Properties

The following defines the properties which may be set in the Java application property file.

15.2.1 Properties for All Handlers

The following properties apply to all handlers.

15.2.1.1 gg.handlerlist

The handler list is a list of active handlers separated by commas. These values are used in the rest of the property file to configure the individual handlers. For example:

gg.handlerlist=name1, name2
gg.handler.name1.propertyA=value1
gg.handler.name1.propertyB=value2
gg.handler.name1.propertyC=value3
gg.handler.name2.propertyA=value1
gg.handler.name2.propertyB=value2
gg.handler.name2.propertyC=value3

Using the handlerlist property, you may include completely configured handlers in the property file and just disable them by removing them from the handlerlist.

15.2.1.2 gg.handler.name.type

This type of handler. This is either a predefined value for built-in handlers, or a fully qualified Java class name. The syntax is:

gg.handler.name.type={jms|jms_map|aq|singlefile|rollingfile|custom_java_class}

Where:

All but the last are pre-defined handlers:

  • jms – Sends transactions, operations, and metadata as formatted messages to a JMS provider

  • aq – Sends transactions, operations, and metadata as formatted messages to Oracle Advanced Queuing (AQ)

  • jms_map – Sends JMS map messages

  • singlefile – Writes to a single file on disk, but does not roll the file

  • rollingfile – Writes transactions, operations, and metadata to a file on disk, rolling the file over after a certain size or after a certain amount of time

  • custom_java_class – Any class that extends the Oracle GoldenGate for Java AbstractHandler class and can handle transaction, operation, or metadata events

15.2.2 Properties for Formatted Output

The following properties apply to all handlers capable of producing formatted output; this includes:

  • The jms_text handler (but not the jms_map handler)

  • The aq handler

  • The singlefile and rolling handlers, for writing formatted output to files

15.2.2.1 gg.handler.name.format

Specifies the format used to transform operations and transactions into messages sent to JMS, to the Big Data target or to a file. The format is specified uniquely for each handler. The value may be:

  • Velocity template

  • Java class name (fully qualified - the class specified must be a type of formatter)

  • csv for delimited values (such as comma separated values; the delimiter can be customized)

  • fixed for fixed-length fields

  • Built-in formatter, such as:

    • xml – demo XML format

    • xml2 – internal XML format

For example, to specify a custom Java class:

gg.handlerlist=abc
gg.handler.abc.format=com.mycompany.MyFormat

Or, for a Velocity template:

gg.handlerlist=xyz
gg.handler.xyz.format=path/to/sample.vm

If using templates, the file is found relative to some directory or JAR that is in the classpath. By default, the Oracle GoldenGate installation directory is in the classpath, so the preceding template could be placed in the dirprm directory of the Oracle GoldenGate installation location.

The default format is to use the built-in XML formatter.

15.2.2.2 gg.handler.name.includeTables

Specifies a list of tables this handler will include. If the schema (or owner) of the table is specified, then only that schema matches the table name; otherwise, the table name matches any schema. A comma separated list of tables can be specified.

For example, to have the handler only process tables foo.customer and bar.orders:

gg.handler.myhandler.includeTables=foo.customer, bar.orders

Note:

In order to selectively process operations on a table by table basis, the handler must be processing in operation mode. If the handler is processing in transaction mode, then when a single transaction contains several operations spanning several tables, if any table matches the include list of tables, the transaction will be included.

15.2.2.3 gg.handler.name.excludeTables

Specifies a list of tables this handler will exclude. If the schema (or owner) of the table is specified, then only that schema matches the table name; otherwise, the table name matches any schema. A list of tables may be specified, comma-separated. For example, to have the handler process all operations on all tables except table date_modified in all schemas:

gg.handler.myhandler.excludeTables=date_modified

15.2.2.4 gg.handler.name.mode, gg.handler.name.format.mode

Specifies whether to output one operation per message (op) or one transaction per message (tx). The default is op. Use gg.handler.name.format.mode when you have a custom formatter.

Note:

This property must be set to one transaction per message (tx) if you are using group transaction properties. If it is set to one operation per message (op), gg.handler.name.minGroupSize and gg.handler.name.maxGroupSize will be ignored

15.2.3 Properties for CSV and Fixed Format Output

If the handler is set to use either CSV or fixed format output, the following properties may also be set.

15.2.3.1 gg.handler.name.format.delim

Specifies the delimiter to use between fields. Set this to no value to have no delimiter used. For example:

gg.handler.handler1.format.delim=,

15.2.3.2 gg.handler.name.format.quote

Specifies the quote character to be used if column values are quoted. For example:

gg.handler.handler1.format.quote='

15.2.3.3 gg.handler.name.format.metacols

Specifies the metadata column values to appear at the beginning of the record, before any column data. Specify any of the following, in the order they should appear:

  • position – unique position indicator of records in a trail

  • opcodeI, U, or D for insert, update, or delete records (see: insertChar, updateChar, deleteChar)

  • txind – transaction indicator – such as 0=begin, 1=middle, 2=end, 3=whole tx (see beginTxChar, middleTxChar, endTxChar, wholeTxChar)

  • opcount – position of a record in a transaction, starting from 0

  • schema – schema/owner of the table for the record

  • tableonly – just table (no schema/owner)

  • table – full name of table, schema.table

  • timestamp – commit timestamp of record

For example:

gg.handler.handler1.format.metacols=opcode, table, txind, position

15.2.3.4 gg.handler.name.format.missingColumnChar

Specifies a special column prefix for a column value that was not captured from the source database transaction log. The column value is not in trail and it is unknown if it has a value or is NULL

The character used to represent the missing state of the column value can be customized. For example:

gg.handler.handler1.format.missingColumnChar=M

By default, the missing column value is set to an empty string and does not show.

15.2.3.5 gg.handler.name.format.presentColumnChar

Specifies a special column prefix for a column value that exists in the trail and is not NULL.

The character used to represent the state of the column can be customized. For example:

gg.handler.handler1.format.presentColumnChar=P

By default, the present column value is set to an empty string and does not show.

15.2.3.6 gg.handler.name.format.nullColumnChar

Specifies a special column prefix for a column value that exists in the trail and is set to NULL.

The character used to represent the state of the column can be customized. For example:

gg.handler.handler1.format.nullColumnChar=N

By default, the null column value is set to an empty string and does not show.

15.2.3.7 gg.handler.name.format.beginTxChar

Specifies the header metadata character (see metacols) used to identify a record as the begin of a transaction. For example:

gg.handler.handler1.format.beginTxChar=B

15.2.3.8 gg.handler.name.format.middleTxChar

Specifies the header metadata characters (see metacols) used to identify a record as the middle of a transaction. For example:

gg.handler.handler1.format.middleTxChar=M

15.2.3.9 gg.handler.name.format.endTxChar

Specifies the header metadata characters (see metacols) used to identify a record as the end of a transaction. For example:

gg.handler.handler1.format.endTxChar=E

15.2.3.10 gg.handler.name.format.wholeTxChar

Specifies the header metadata characters (see metacols) used to identify a record as a complete transaction; referred to as a whole transaction. For example:

gg.handler.handler1.format.wholeTxChar=W

15.2.3.11 gg.handler.name.format.insertChar

Specifies the character to identify an insert operation. The default I.

For example, to use INS instead of I for insert operations:

gg.handler.handler1.format.insertChar=INS

15.2.3.12 gg.handler.name.format.updateChar

Specifies the character to identify an update operation. The default is U.

For example, to use UPD instead of U for update operations:

gg.handler.handler1.format.updateChar=UPD

15.2.3.13 gg.handler.name.format.deleteChar

Specifies the character to identify a delete operation. The default is D.

For example, to use DEL instead of D for delete operations:

gg.handler.handler1.format.deleteChar=DEL

15.2.3.14 gg.handler.name.format.endOfLine

Specifies the end-of-line character as:

  • EOL - Native platform

  • CR - Neutral (UNIX-style \n)

  • CRLF - Windows (\r\n)

For example:

gg.handler.handler1.format.endOfLine=CR

15.2.3.15 gg.handler.name.format.justify

Specifies the left or right justification of fixed fields. For example:

gg.handler.handler1.format.justify=left

15.2.3.16 gg.handler.name.format.includeBefores

Controls whether before images should be included in the output. There must be before images in the trail. For example:

gg.handler.handler1.format.includeBefores=false

15.2.4 File Writer Properties

The following properties only apply to handlers that write their output to files: the file handler and the singlefile handler.

15.2.4.1 gg.handler.name.file

Specifies the name of the output file for the given handler. If the handler is a rolling file, this name is used to derive the rolled file names. The default file name is output.xml.

15.2.4.2 gg.handler.name.append

Controls whether the file should be appended to (true) or overwritten upon restart (false).

15.2.4.3 gg.handler.name.rolloverSize

If using the file handler, this specifies the size of the file before a rollover should be attempted. The file size will be at least this size, but will most likely be larger. Operations and transactions are not broken across files. The size is specified in bytes, but a suffix may be given to identify MB or KB. For example:

gg.handler.myfile.rolloverSize=5MB

The default rollover size is 10 MB.

15.2.5 JMS Handler Properties

The following properties apply to the JMS handlers. Some of these values may be defined in the Java application properties file using the name of the handler. Other properties may be placed into a separate JMS properties file, which is useful if using more than one JMS handler at a time. For example:

gg.handler.myjms.type=jms_text
gg.handler.myjms.format=xml
gg.handler.myjms.properties=weblogic.properties

Just as with Velocity templates and formatting property files, this additional JMS properties file is found in the classpath. The above properties file weblogic.properties would be found in {gg_install_dir}/dirprm/weblogic.properties, since the dirprm directory is included by default in the class path.

Settings that can be made in the Java application properties file will override the corresponding value set in the supplemental JMS properties file (weblogic.properties in the example above). In the following example, the destination property is specified in the Java application properties file. This allows the same default connection information for the two handlers myjms1 and myjms2, but customizes the target destination queue.

gg.handlerlist=myjms1,myjms2
gg.handler.myjms1.type=jms_text
gg.handler.myjms1.destination=queue.sampleA
gg.handler.myjms1.format=sample.vm
gg.handler.myjms1.properties=tibco-default.properties
gg.handler.myjms2.type=jms_map
gg.handler.myjms2.destination=queue.sampleB
gg.handler.myjms2.properties=tibco-default.properties

To set a property, specify the handler name as a prefix; for example:

gg.handlerlist=sample,sample2
gg.handler.sample.type=jms_text
gg.handler.sample.format=my_template.vm
gg.handler.sample.destination=gg.myqueue
gg.handler.sample.queueortopic=queue
gg.handler.sample.connectionUrl=tcp://host:61616?jms.useAsyncSend=true
gg.handler.sample.useJndi=false
gg.handler.sample.connectionFactory=ConnectionFactory
gg.handler.sample.connectionFactoryClass=\
    org.apache.activemq.ActiveMQConnectionFactory
gg.handler.sample.connectionUrl=\
    tcp://localhost:61616?jms.useAsyncSend=true
gg.handler.sample.timeToLive=50000

15.2.5.1 Standard JMS Settings

The following outlines the JMS properties which may be set, and the accepted values. These apply for both JMS handler types: jms_text (TextMessage) and jms_map (MapMessage).

15.2.5.1.1 gg.handler.name.destination

The queue or topic to which the message is sent. This must be correctly configured on the JMS server. Typical values may be: queue/A, queue.Test, example.MyTopic, etc.

gg.handler.name.destination=queue_or_topic
15.2.5.1.2 gg.handler.name.user

(Optional) User name required to send messages to the JMS server.

gg.handler.name.user=user_name
15.2.5.1.3 gg.handler.name.password

(Optional) Password required to send messages to the JMS server

gg.handler.name.password=password
15.2.5.1.4 gg.handler.name.queueOrTopic

Whether the handler is sending to a queue (a single receiver) or a topic (publish / subscribe). This must be correctly configured in the JMS provider. This property is an alias of gg.handler.name.destination. The syntax is:

gg.handler.name.queueOrTopic={queue|topic}

Where:

  • queue – a message is removed from the queue once it has been read. This is the default.

  • topic – messages are published and may be delivered to multiple subscribers.

15.2.5.1.5 gg.handler.name.persistent

If the delivery mode is set to persistent or not. If the messages are to be persistent, the JMS provider must be configured to log the message to stable storage as part of the client's send operation. The syntax is:

gg.handler.name.persistent={true|false}
15.2.5.1.6 gg.handler.name.priority

JMS defines a 10 level priority value, with 0 as the lowest and 9 as the highest. Priority is set to 4 by default. The syntax is:

gg.handler.name.priority=integer

For example:

gg.handler.name.priority=5
15.2.5.1.7 gg.handler.name.timeToLive

The length of time in milliseconds from its dispatch time that a produced message should be retained by the message system. A value of zero specifies the time is unlimited. The default is zero. The syntax is:

gg.handler.name.timeToLive=milliseconds

For example:

gg.handler.name.timeToLive= 36000
15.2.5.1.8 gg.handler.name.connectionFactory

Name of the connection factory to lookup via JNDI. ConnectionFactoryJNDIName is an alias. The syntax is:

gg.handler.name.connectionFactory=JNDI_name
15.2.5.1.9 gg.handler.name.useJndi

If gg.handler.name.usejndi is false, then JNDI is not used to configure the JMS client. Instead, factories and connections are explicitly constructed. The syntax is:

gg.handler.name.useJndi={true|false}
15.2.5.1.10 gg.handler.name.connectionUrl

Connection URL is used only when not using JNDI to explicitly create the connection. The syntax is:

gg.handler.name.connectionUrl=url
15.2.5.1.11 gg.handler.name.connectionFactoryClass

The Connection Factory Class is used to access a factory only when not using JNDI. The value of this property is the Java class name to instantiate; constructing a factory object explicitly.

gg.handler.name.connectionFactoryClass=java_class_name
15.2.5.1.12 gg.handler.name.localTX

Specifies whether or not local transactions are used. The default is true, local transactions are used. The syntax is:

gg.handler.name.localTX={true|false}
15.2.5.1.13 gg.handlerlist.nop

Disables the sending of JMS messages to allow testing of message generation. This is a global property used only for testing. The events are still generated and handled and the message is constructed. The default is false; do not disable message send. The syntax is:

gg.handlerlist.nop={true|false}

15.2.5.2 Group Transaction Properties

These properties set limits for grouping transactions.

Note:

When you use group transaction properties, you must:

  • Ensure that gg.handler.name.mode is set to one transaction per message (tx). Otherwise the group transaction properties will be ignored.

  • Ensure that the goldengate.userexit.nochkpt property is set to false.

  • Ignore the transaction indicator on the operations and not use it to determine transaction boundaries.

  • Use only one named handler per installation.

15.2.5.2.1 gg.handler.name.minGroupSize

Specifies the minimum number of operations that must accumulate before the transaction will be sent.

The syntax is:

gg.handler.name.minGroupSize=number_ops

Where:

  • number_ops specifies the minimum number of operations that must be accumulated before the transaction is sent.

The maximum value allowed is integer.MAX.VALUE or 2147483647. The minimum value is one.

If you use both properties, the value set for gg.handler.name.minGroupSize should be less than or equal to the value set for gg.handler.name.maxGroupSize.

The following example will test for a minimum of 50 operations before a send.

gg.handler.name.minGroupSize=50
15.2.5.2.2 gg.handler.name.maxGroupSize

Specifies the maximum number of operations that will rigger the transaction send.

The syntax is:

gg.handler.name.maxGroupSize=number_ops 

Where:

  • number_ops specifies the maximum number of operations that will be accumulated before the transaction is sent.

The maximum value allowed is integer.MAX.VALUE or 2147483647. The minimum value is one.

If you use both properties, the value set for gg.handler.name.minGroupSize should be less than or equal to the value set for gg.handler.name.maxGroupSize.

The following example will send when the maximum of 50 operations is reached.

gg.handler.name.maxGroupSize=50

15.2.6 JNDI Properties

These JNDI properties are required for connection to an Initial Context to look up the connection factory and initial destination.

java.naming.provider.url=url
java.naming.factory.initial=java-class-name

If JNDI security is enabled, the following properties may be set:

java.naming.security.principal=user-name
java.naming.security.credentials=password-or-other-authenticator

For example:

java.naming.provider.url= t3://localhost:7001
java.naming.factory.initial=weblogic.jndi.WLInitialContextFactory
java.naming.security.principal=jndiuser
java.naming.security.credentials=jndipw

15.2.7 General Properties

The following are general properties that are used for the user exit Java framework.

15.2.7.1 gg.classpath

Specifies a comma delimited list of additional paths to directories or jars to add to the class path. Optionally, the list can be delimited by semicolons for Windows systems or by colons for UNIX. For example:

gg.classpath=C:\Program Files\MyProgram\bin;C:\Program Files\ProgramB\app\bin;

15.2.7.2 gg.report.time

Specifies how often statistics are calculated and sent to Extract for the processing report. If Extract is configured to print a report, these statistics are included. The syntax is:

gg.report.time=report_interval{s|m|h}

Where:

  • report_interval is an integer

  • The valid time units are:

    • s - seconds

    • m - minutes

    • h - hours

If no value is entered, the default is to calculate and send every 24 hours.