3 Introducing the Java Adapter

This chapter describes the Oracle GoldenGate Adapter for Java. The Oracle GoldenGate Adapter for Java implements 1) the capture of Java Message Service (JMS) messages to send for processing into Oracle GoldenGate trail data, and 2) the processing of transactional data captured by Oracle GoldenGate to be delivered as JMS messages.

This chapter includes the following sections:

3.1 Oracle GoldenGate VAM Message Capture

Oracle GoldenGate message capture connects to JMS messaging to parse messages and send them through a VAM interface to an Oracle GoldenGate Extract that builds an Oracle GoldenGate trail of message data. This allows JMS messages to be delivered to an Oracle GoldenGate system running for a target database.

Using Oracle GoldenGate JMS message capture requires two components:

  • The dynamically linked shared VAM library that is attached to the Oracle GoldenGate Extract process.

  • A separate utility, Gendef, that uses the message capture properties file and parser-specific data definitions to create an Oracle GoldenGate source definitions file.

3.1.1 Message Capture Configuration Options

The options for configuring the three parts of message capture are:

  • Message connectivity: Values in the property file set connection properties such as the Java class path for the JMS client, the JMS source destination name, JNDI connection properties, and security information.

  • Parsing: Values in the property file set parsing rules for fixed width, comma delimited, or XML messages. This includes settings such as the delimiter to be used, values for the beginning and end of transactions and the date format.

  • VAM interface: Parameters that identify the VAM, dll, or so library and a property file are set for the Oracle GoldenGate core Extract process.

3.1.2 Typical Configuration

The following diagram shows a typical configuration for capturing JMS messages.

In this configuration, JMS messages are picked up by the Oracle GoldenGate Adapter JMS Handler and transferred using the adapter's message capture VAM to an Extract process. The Extract writes the data to a trail which is sent over the network by a Data Pump Extract to an Oracle GoldenGate target instance. The target Replicat then uses the trail to update the target database.

Figure 3-1 Configuration for JMS Message Capture

Description of Figure 3-1 follows
Description of "Figure 3-1 Configuration for JMS Message Capture"

3.2 Oracle GoldenGate Java User Exit

Through the Oracle GoldenGate Java API, transactional data captured by Oracle GoldenGate can be delivered to targets other than a relational database, such as a JMS (Java Message Service), files written to disk, or an integration with a custom application Java API.

Oracle GoldenGate for Java provides the ability to execute Java code from an Oracle GoldenGate Extract process or Replicat process. Replicat is the recommended way to integrate Java delivery because of the transaction grouping functionality and improved checkpointing provided by Replicat. Using Oracle GoldenGate for Java requires two components:

  • A dynamically linked or shared library implemented in C or C++, integrating as a User Exit (UE). The user exit shared libraries are different for Extract and Replicat. For Extract the library is libggjava_ue.so on Linux and UNIX; it is gjava_ue.dll for Windows. For Replicat, the library is libggjava.so for Linux and UNIX; it is ggjava.dll for Windows.

  • A set of Java libraries (JARS) that comprise the Oracle GoldenGate Java API. This Java framework communicates with the user exit through the Java Native Interface (JNI). The interface of the Java layer is identical for running with either the Extract process or the Replicat process.

Figure 3-2 Configuration for Delivering JMS Messages

Description of Figure 3-2 follows
Description of "Figure 3-2 Configuration for Delivering JMS Messages"

3.2.1 Delivery Configuration Options

The dynamically linked library is configurable using a simple properties file. The Java framework is loaded by the user exit and is also initialized by a properties file. Application behavior can be customized by:

  • Editing the property files; for example to:

    • Set host names, port numbers, output file names, JMS connection settings;

    • Add/remove targets (such as JMS or files) by listing any number of active handlers to which the transactions should be sent;

    • Turn on/off debug-level logging, etc.

    • Identify which message format should be used.

  • Customizing the format of messages sent to JMS or files. Message formats can be custom tailored by:

    • Setting properties for the pre-existing format process (for fixed-length or field-delimited message formats);

    • Customizing message templates, using the Velocity template macro language;

    • (Optional) Writing custom Java code.

  • (Optional) Writing custom Java code to provide custom handling of transactions and operations, do filtering, or implementing custom message formats.

There are existing implementations (handlers) for sending messages via JMS and for writing out files to disk. There are several predefined message formats for sending the messages (e.g. XML or field-delimited); or custom formats can be implemented using templates. Each handler has documentation that describes its configuration properties; for example, a file name can be specified for a file writer, and a JMS queue name can be specified for the JMS handler. Some properties apply to more than one handler; for example, the same message format can be used for JMS and files.

3.3 Running with Extract

This section explains how to run Java Adapter with the Oracle GoldenGate Extract process.

3.3.1 Extract Configuration

The following

EXTRACT hdfs
discardfile ./dirrpt/avro1.dsc, purge 
--SOURCEDEFS ./dirdef/dbo.def
CUSEREXIT libjavaue.so CUSEREXIT PASSTHRU, INCLUDEUPDATEBEFORES, PARAMS "dirprm/hdfs.props" 
GETUPDATEBEFORES 
TABLE dbo.*;

The following is explanation of the Replicat configuration entries:

EXTRACT hdfs - The Extract process name.

discardfile ./dirrpt/avro1.dsc, purge - Set the discard file

--SOURCEDEFS ./dirdef/dbo.def - Source definitions are not required for 12.2 trial files.

CUSEREXIT libjavaue.so CUSEREXIT PASSTHRU, INCLUDEUPDATEBEFORES, PARAMS "dirprm/hdfs.props" - Set you exit shared library, and point to the Java Adapter Properties file

GETUPDATEBEFORES - Get update before images.

TABLE dbo.*; - Select which tables to replicate or exclude to filter.

3.3.2 Adding the Extract Process

ADD EXTRACT hdfs, EXTTRAILSOURCE ./dirdat/gg
START hdfs

3.3.3 Extract Grouping

The Extract process provides no functionality for transaction grouping. However, transaction grouping is still possible when integrating Java Delivery with the Extract process. The Java Delivery layer enables transaction grouping with configuration in the Java Adapter properties file.

  1. gg.handler.name.mode

    To enable grouping, the value of this property must be set to tx.

  2. gg.handler.name.maxGroupSize

    Controls the maximum number of operations that can be held by an operation group - irrespective of whether the operation group holds operations from a single transaction or multiple transactions.

    The operation group will send a transaction commit and end the group as soon as this number of operations is reached. This property leads to splitting of transactions across multiple operation groups.

  3. gg.handler.name.minGroupSize

    This is the minimum number of operations that must exist in a group before the group can end.

    This property helps to avoid groups that are too small by grouping multiple small transactions into one operation group so that it can be more efficiently processed.

    Note:

    maxGroupSize should always be greater than or equal to minGroupSize; that is, maxGroupSize >= minGroupSize.

Note:

It is not recommended to use the Java layer transaction grouping when running Java Delivery with the Replicat process. If running with the Replicat process, you should use Replicat transaction grouping controlled by the GROUPTRANSOPS Replicat property.

3.4 Running with Replicat

This section explains how to run the Java Adapter with the Oracle GoldenGate Replicat process.

3.4.1 Replicat Configuration

The following is an example of a Replicat process properties file for Java Adapter.

REPLICAT hdfs
TARGETDB LIBFILE libggjava.so SET property=dirprm/hdfs.properties 
--SOURCEDEFS ./dirdef/dbo.def 
DDL INCLUDE ALL
GROUPTRANSOPS 1000
MAPEXCLUDE dbo.excludetable
MAP dbo.*, TARGET dbo.*;

The following is explanation of the Replicat configuration entries:

REPLICAT hdfs - The name of the Replicat process.

TARGETDB LIBFILE libggjava.so SET property=dirprm/hdfs.properties - Names the target database as you exit libggjava.so and sets the Java Adapters Property file to dirprm/hdfs.properties

--SOURCEDEFS ./dirdef/dbo.def - Sets a source database definitions file. Commented out because Oracle GoldenGate 12.2.0.1 trail files provide metadata in trail.

GROUPTRANSOPS 1000 - To group 1000 transactions from the source trail files into a single target transaction. This is the default and improves the performance of Big Data integrations.

MAPEXCLUDE dbo.excludetable - To identify tables to exclude.

MAP dbo.*, TARGET dbo.*; - Shows the mapping of input to output tables.

3.4.2 Adding the Replicat Process

The command to add and start the Replicat process in ggsci is the following:

ADD REPLICAT hdfs, EXTTRAIL ./dirdat/gg
START hdfs

3.4.3 Replicat Grouping

The Replicat process provides the Replicat configuration property GROUPTRANSOPS to control transaction grouping. By default, the Replicat process implements transaction grouping of 1000 source transactions into a single target transaction. If you want to turn off transaction grouping then the GROUPTRANSOPS Replicat property should be set to 1.

3.4.4 Replicat Checkpointing

CHECKPOINTTABLE and NODBCHECKPOINT are not applicable for Java Delivery with Replicat. Beside Replicat checkpoint file (.cpr), additional checkpoint file (dirchk/<group>.cpj) will be created that contains information similar to CHECKPOINTTABLE in Replicat for RDBMS.

3.4.5 Unsupported Replicat Features

The following Replicat features are not supported in this release:

  • BATCHSQL

  • SQLEXEC

  • Stored procedure

  • Conflict resolution and detection (CDR)

  • REPERROR

3.4.6 Mapping Functionality

The Oracle GoldenGate Replicat process supports mapping functionality to custom target schemas. This functionality is not available using the Oracle GoldenGate Extract process. You must use the Metadata Provider functionality to define a target schema or schemas and then use the standard Replicat mapping syntax in the Replicat configuration file to define the mapping. Refer to the Oracle GoldenGate Replicat documentation to understand the Replicat mapping syntax in the Replication configuration file.