1/33

Contents

Title and Copyright Information

Preface

• Audience
• Documentation Accessibility
• Related Information
• Conventions

1 Introducing Oracle GoldenGate for Big Data

• 1.1 Understanding What’s Supported
  • 1.1.1 Verifying Certification and System Requirements
  • 1.1.2 What are the Additional Support Considerations?
• 1.2 Setting Up Oracle GoldenGate for Big Data
  • 1.2.1 About Oracle GoldenGate Properties Files
  • 1.2.2 Setting Up the Java Runtime Environment
  • 1.2.3 Configuring Java Virtual Machine Memory
  • 1.2.4 Grouping Transactions
• 1.3 Configuring GoldenGate for Big Data
  • 1.3.1 Running with Replicat
    • 1.3.1.1 Configuring Replicat
    • 1.3.1.2 Adding the Replicat Process
    • 1.3.1.3 Replicat Grouping
    • 1.3.1.4 About Replicat Checkpointing
    • 1.3.1.5 About Initial Load Support
    • 1.3.1.6 About the Unsupported Replicat Features
    • 1.3.1.7 How the Mapping Functionality Works
  • 1.3.2 Overview of Logging
    • 1.3.2.1 About Replicat Process Logging
    • 1.3.2.2 About Java Layer Logging
  • 1.3.3 About Schema Evolution and Metadata Change Events
  • 1.3.4 About Configuration Property CDATA[] Wrapping
  • 1.3.5 Using Regular Expression Search and Replace
    • 1.3.5.1 Using Schema Data Replace
    • 1.3.5.2 Using Content Data Replace
  • 1.3.6 Scaling Oracle GoldenGate for Big Data Delivery
  • 1.3.7 Using Identities in Oracle GoldenGate Credential Store
    • 1.3.7.1 Creating a Credential Store
    • 1.3.7.2 Adding Users to a Credential Store
    • 1.3.7.3 Configuring Properties to Access the Credential Store

2 Using the BigQuery Handler

• 2.1 Detailing the Functionality
  • 2.1.1 Data Types
  • 2.1.2 Operation Modes
  • 2.1.3 Operation Processing Support
• 2.2 Setting Up and Running the BigQuery Handler
  • 2.2.1 Understanding the BigQuery Handler Configuration
  • 2.2.2 Review a Sample Configuration
  • 2.2.3 Proxy Settings
  • 2.2.4 Configuring Handler Authentication

3 Using the Cassandra Handler

• 3.1 Overview
• 3.2 Detailing the Functionality
  • 3.2.1 About the Cassandra Data Types
  • 3.2.2 About Catalog, Schema, Table, and Column Name Mapping
  • 3.2.3 About DDL Functionality
    • 3.2.3.1 About the Keyspaces
    • 3.2.3.2 About the Tables
    • 3.2.3.3 Addng Column Functionality
    • 3.2.3.4 Dropping Column Functionality
  • 3.2.4 How Operations are Processed
  • 3.2.5 About Compressed Updates vs. Full Image Updates
  • 3.2.6 About Primary Key Updates
• 3.3 Setting Up and Running the Cassandra Handler
  • 3.3.1 Understanding the Cassandra Handler Configuration
  • 3.3.2 Review a Sample Configuration
  • 3.3.3 Configuring Security
• 3.4 About Automated DDL Handling
  • 3.4.1 About the Table Check and Reconciliation Process
  • 3.4.2 Capturing New Change Data
• 3.5 Performance Considerations
• 3.6 Additional Considerations
• 3.7 Troubleshooting
  • 3.7.1 Java Classpath
  • 3.7.2 Logging
  • 3.7.3 Write Timeout Exception
  • 3.7.4 Logging
  • 3.7.5 Datastax Driver Error

4 Using the Elasticsearch Handler

• 4.1 Overview
• 4.2 Detailing the Functionality
  • 4.2.1 About the Elasticsearch Version Property
  • 4.2.2 About the Index and Type
  • 4.2.3 About the Document
  • 4.2.4 About the Primary Key Update
  • 4.2.5 About the Data Types
  • 4.2.6 Operation Mode
  • 4.2.7 Operation Processing Support
  • 4.2.8 About the Connection
• 4.3 Setting Up and Running the Elasticsearch Handler
  • 4.3.1 Configuring the Elasticsearch Handler
  • 4.3.2 About the Transport Client Settings Properties File
• 4.4 Performance Consideration
• 4.5 About the Shield Plug-In Support
• 4.6 About DDL Handling
• 4.7 Troubleshooting
  • 4.7.1 Incorrect Java Classpath
  • 4.7.2 Elasticsearch Version Mismatch
  • 4.7.3 Transport Client Properties File Not Found
  • 4.7.4 Cluster Connection Problem
  • 4.7.5 Unsupported Truncate Operation
  • 4.7.6 Bulk Execute Errors
• 4.8 Logging
• 4.9 Known Issues in the Elasticsearch Handler

5 Using the File Writer Handler

• 5.1 Overview
  • 5.1.1 Detailing the Functionality
    • 5.1.1.1 Using File Roll Events
    • 5.1.1.2 Automatic Directory Creation
    • 5.1.1.3 About the Active Write Suffix
    • 5.1.1.4 Maintenance of State
    • 5.1.1.5 Using Templated Strings
  • 5.1.2 Configuring the File Writer Handler
  • 5.1.3 Review a Sample Configuration
• 5.2 Using the HDFS Event Handler
  • 5.2.1 Detailing the Functionality
    • 5.2.1.1 Configuring the Handler
    • 5.2.1.2 Using Templated Strings
    • 5.2.1.3 Configuring the HDFS Event Handler
• 5.3 Using the Optimized Row Columnar Event Handler
  • 5.3.1 Overview
  • 5.3.2 Detailing the Functionality
    • 5.3.2.1 About the Upstream Data Format
    • 5.3.2.2 About the Library Dependencies
    • 5.3.2.3 Requirements
    • 5.3.2.4 Using Templated Strings
  • 5.3.3 Configuring the ORC Event Handler
• 5.4 Using the Oracle Cloud Infrastructure Event Handler
  • 5.4.1 Overview
  • 5.4.2 Detailing the Functionality
  • 5.4.3 Configuring the Oracle Cloud Infrastructure Event Handler
  • 5.4.4 Configuring Credentials for Oracle Cloud Infrastructure
  • 5.4.5 Using Templated Strings
  • 5.4.6 Troubleshooting
• 5.5 Using the Oracle Cloud Infrastructure Classic Event Handler
  • 5.5.1 Overview
  • 5.5.2 Detailing the Functionality
  • 5.5.3 Configuring the Oracle Cloud Infrastructure Classic Event Handler
  • 5.5.4 Using Templated Strings
  • 5.5.5 Troubleshooting
• 5.6 Using the Parquet Event Handler
  • 5.6.1 Overview
  • 5.6.2 Detailing the Functionality
    • 5.6.2.1 Configuring the Parquet Event Handler to Write to HDFS
    • 5.6.2.2 About the Upstream Data Format
    • 5.6.2.3 Using Templated Strings
  • 5.6.3 Configuring the Parquet Event Handler
• 5.7 Using the S3 Event Handler
  • 5.7.1 Overview
  • 5.7.2 Detailing Functionality
    • 5.7.2.1 Configuring the Client ID and Secret
    • 5.7.2.2 About the AWS S3 Buckets
    • 5.7.2.3 Using Templated Strings
    • 5.7.2.4 Troubleshooting
  • 5.7.3 Configuring the S3 Event Handler

6 Using the Flume Handler

• 6.1 Overview
• 6.2 Setting Up and Running the Flume Handler
  • 6.2.1 Classpath Configuration
  • 6.2.2 Flume Handler Configuration
  • 6.2.3 Review a Sample Configuration
• 6.3 Data Mapping of Operations to Flume Events
  • 6.3.1 Operation Mode
  • 6.3.2 Transaction Mode and EventMapsTo Operation
  • 6.3.3 Transaction Mode and EventMapsTo Transaction
• 6.4 Performance Considerations
• 6.5 Metadata Change Events
• 6.6 Example Flume Source Configuration
  • 6.6.1 Avro Flume Source
  • 6.6.2 Thrift Flume Source
• 6.7 Advanced Features
  • 6.7.1 Schema Propagation
  • 6.7.2 Security
  • 6.7.3 Fail Over Functionality
  • 6.7.4 Load Balancing Functionality
• 6.8 Troubleshooting the Flume Handler
  • 6.8.1 Java Classpath
  • 6.8.2 Flume Flow Control Issues
  • 6.8.3 Flume Agent Configuration File Not Found
  • 6.8.4 Flume Connection Exception
  • 6.8.5 Other Failures

7 Using the HBase Handler

• 7.1 Overview
• 7.2 Detailed Functionality
• 7.3 Setting Up and Running the HBase Handler
  • 7.3.1 Classpath Configuration
  • 7.3.2 HBase Handler Configuration
  • 7.3.3 Sample Configuration
  • 7.3.4 Performance Considerations
• 7.4 Security
• 7.5 Metadata Change Events
• 7.6 Additional Considerations
• 7.7 Troubleshooting the HBase Handler
  • 7.7.1 Java Classpath
  • 7.7.2 HBase Connection Properties
  • 7.7.3 Logging of Handler Configuration
  • 7.7.4 HBase Handler Delete-Insert Problem
  • 7.7.5 Cloudera CDH HBase Compatibility

8 Using the HDFS Handler

• 8.1 Overview
• 8.2 Writing into HDFS in SequenceFile Format
  • 8.2.1 Integrating with Hive
  • 8.2.2 Understanding the Data Format
• 8.3 Setting Up and Running the HDFS Handler
  • 8.3.1 Classpath Configuration
  • 8.3.2 HDFS Handler Configuration
  • 8.3.3 Review a Sample Configuration
  • 8.3.4 Performance Considerations
  • 8.3.5 Security
• 8.4 Writing in HDFS in Avro Object Container File Format
• 8.5 Generating HDFS File Names Using Template Strings
• 8.6 Metadata Change Events
• 8.7 Partitioning
• 8.8 HDFS Additional Considerations
• 8.9 Best Practices
• 8.10 Troubleshooting the HDFS Handler
  • 8.10.1 Java Classpath
  • 8.10.2 HDFS Connection Properties
  • 8.10.3 Handler and Formatter Configuration

9 Using the Java Database Connectivity Handler

• 9.1 Overview
• 9.2 Detailed Functionality
  • 9.2.1 Single Operation Mode
  • 9.2.2 Oracle Database Data Types
  • 9.2.3 MySQL Database Data Types
  • 9.2.4 Netezza Database Data Types
  • 9.2.5 Redshift Database Data Types
• 9.3 Setting Up and Running the JDBC Handler
  • 9.3.1 Java Classpath
  • 9.3.2 Handler Configuration
  • 9.3.3 Statement Caching
  • 9.3.4 Setting Up Error Handling
• 9.4 Sample Configurations
  • 9.4.1 Sample Oracle Database Target
  • 9.4.2 Sample Oracle Database Target with JDBC Metadata Provider
  • 9.4.3 Sample MySQL Database Target
  • 9.4.4 Sample MySQL Database Target with JDBC Metadata Provider

10 Using the Kafka Handler

• 10.1 Overview
• 10.2 Detailed Functionality
• 10.3 Setting Up and Running the Kafka Handler
  • 10.3.1 Classpath Configuration
  • 10.3.2 Kafka Handler Configuration
  • 10.3.3 Java Adapter Properties File
  • 10.3.4 Kafka Producer Configuration File
  • 10.3.5 Using Templates to Resolve the Topic Name and Message Key
  • 10.3.6 Kafka Configuring with Kerberos on a Hadoop Platform
• 10.4 Schema Propagation
• 10.5 Performance Considerations
• 10.6 About Security
• 10.7 Metadata Change Events
• 10.8 Snappy Considerations
• 10.9 Troubleshooting
  • 10.9.1 Verify the Kafka Setup
  • 10.9.2 Classpath Issues
  • 10.9.3 Invalid Kafka Version
  • 10.9.4 Kafka Producer Properties File Not Found
  • 10.9.5 Kafka Connection Problem

11 Using the Kafka Connect Handler

• 11.1 Overview
• 11.2 Detailed Functionality
• 11.3 Setting Up and Running the Kafka Connect Handler
  • 11.3.1 Kafka Connect Handler Configuration
  • 11.3.2 Using Templates to Resolve the Topic Name and Message Key
  • 11.3.3 Configuring Security in the Kafka Connect Handler
• 11.4 Kafka Connect Handler Performance Considerations
• 11.5 Troubleshooting the Kafka Connect Handler
  • 11.5.1 Java Classpath for Kafka Connect Handler
  • 11.5.2 Invalid Kafka Version
  • 11.5.3 Kafka Producer Properties File Not Found
  • 11.5.4 Kafka Connection Problem

12 Using the Kafka REST Proxy Handler

• 12.1 Overview
• 12.2 Setting Up and Starting the Kafka REST Proxy Handler Services
  • 12.2.1 Using the Kafka REST Proxy Handler
  • 12.2.2 Kafka REST Proxy Handler Configuration
  • 12.2.3 Security
  • 12.2.4 Generating a Keystore
  • 12.2.5 Using Templates to Resolve the Topic Name and Message Key
  • 12.2.6 Kafka REST Proxy Handler Formatter Properties
  • 12.2.7 Setting Metacolumn Output
• 12.3 Consuming the Records
• 12.4 Performance Considerations
• 12.5 Kafka REST Proxy Handler Metacolumns Template Property

13 Using the Kinesis Streams Handler

• 13.1 Overview
• 13.2 Detailed Functionality
  • 13.2.1 Amazon Kinesis Java SDK
  • 13.2.2 Kinesis Streams Input Limits
• 13.3 Setting Up and Running the Kinesis Streams Handler
  • 13.3.1 Set the Classpath in Kinesis Streams Handler
  • 13.3.2 Kinesis Streams Handler Configuration
  • 13.3.3 Using Templates to Resolve the Stream Name and Partition Name
  • 13.3.4 Configuring the Client ID and Secret in Kinesis Handler
  • 13.3.5 Configuring the Proxy Server for Kinesis Streams Handler
  • 13.3.6 Configuring Security in Kinesis Streams Handler
• 13.4 Kinesis Handler Performance Considerations
  • 13.4.1 Kinesis Streams Input Limitations
  • 13.4.2 Transaction Batching
  • 13.4.3 Deferring Flush at Transaction Commit
• 13.5 Troubleshooting
  • 13.5.1 Java Classpath
  • 13.5.2 Kinesis Handler Connectivity Issues
  • 13.5.3 Logging

14 Using the MongoDB Handler

• 14.1 Overview
• 14.2 Detailed Functionality
  • 14.2.1 Document Key Column
  • 14.2.2 Primary Key Update Operation
  • 14.2.3 MongoDB Trail Data Types
• 14.3 Setting Up and Running the MongoDB Handler
  • 14.3.1 Classpath Configuration
  • 14.3.2 MongoDB Handler Configuration
  • 14.3.3 Connecting and Authenticating
  • 14.3.4 Using Bulk Write
  • 14.3.5 Using Write Concern
  • 14.3.6 Using Three-Part Table Names
  • 14.3.7 Using Undo Handling
• 14.4 Review a Sample Configuration

15 Using the Metadata Providers

• 15.1 About the Metadata Providers
• 15.2 Avro Metadata Provider
  • 15.2.1 Detailed Functionality
  • 15.2.2 Runtime Prerequisites
  • 15.2.3 Classpath Configuration
  • 15.2.4 Avro Metadata Provider Configuration
  • 15.2.5 Review a Sample Configuration
  • 15.2.6 Metadata Change Events
  • 15.2.7 Limitations
  • 15.2.8 Troubleshooting
    • 15.2.8.1 Invalid Schema Files Location
    • 15.2.8.2 Invalid Schema File Name
    • 15.2.8.3 Invalid Namespace in Schema File
    • 15.2.8.4 Invalid Table Name in Schema File
• 15.3 Java Database Connectivity Metadata Provider
  • 15.3.1 JDBC Detailed Functionality
  • 15.3.2 Java Classpath
  • 15.3.3 JDBC Metadata Provider Configuration
  • 15.3.4 Review a Sample Configuration
• 15.4 Hive Metadata Provider
  • 15.4.1 Detailed Functionality
  • 15.4.2 Configuring Hive with a Remote Metastore Database
  • 15.4.3 Classpath Configuration
  • 15.4.4 Hive Metadata Provider Configuration Properties
  • 15.4.5 Review a Sample Configuration
  • 15.4.6 Security
  • 15.4.7 Metadata Change Event
  • 15.4.8 Limitations
  • 15.4.9 Additional Considerations
  • 15.4.10 Troubleshooting

16 Using the Oracle NoSQL Handler

• 16.1 Overview
• 16.2 Detailed Functionality
  • 16.2.1 Oracle NoSQL Data Types
  • 16.2.2 Performance Considerations
  • 16.2.3 Operation Processing Support
  • 16.2.4 Column Processing
  • 16.2.5 Table Check and Reconciliation Process
  • 16.2.6 Security
• 16.3 Oracle NoSQL Handler Configuration
• 16.4 Review a Sample Configuration
• 16.5 Performance Considerations
• 16.6 Full Image Data Requirements

17 Using the Pluggable Formatters

• 17.1 Using the Avro Formatter
  • 17.1.1 Avro Row Formatter
    • 17.1.1.1 Operation Metadata Formatting Details
    • 17.1.1.2 Operation Data Formatting Details
    • 17.1.1.3 Sample Avro Row Messages
      • 17.1.1.3.1 Sample Insert Message
      • 17.1.1.3.2 Sample Update Message
      • 17.1.1.3.3 Sample Delete Message
      • 17.1.1.3.4 Sample Truncate Message
    • 17.1.1.4 Avro Schemas
    • 17.1.1.5 Avro Row Configuration Properties
    • 17.1.1.6 Review a Sample Configuration
    • 17.1.1.7 Metadata Change Events
    • 17.1.1.8 Special Considerations
      • 17.1.1.8.1 Troubleshooting
      • 17.1.1.8.2 Primary Key Updates
      • 17.1.1.8.3 Generic Wrapper Functionality
  • 17.1.2 The Avro Operation Formatter
    • 17.1.2.1 Operation Metadata Formatting Details
    • 17.1.2.2 Operation Data Formatting Details
    • 17.1.2.3 Sample Avro Operation Messages
      • 17.1.2.3.1 Sample Insert Message
      • 17.1.2.3.2 Sample Update Message
      • 17.1.2.3.3 Sample Delete Message
      • 17.1.2.3.4 Sample Truncate Message
    • 17.1.2.4 Avro Schema
    • 17.1.2.5 Avro Operation Formatter Configuration Properties
    • 17.1.2.6 Review a Sample Configuration
    • 17.1.2.7 Metadata Change Events
    • 17.1.2.8 Special Considerations
      • 17.1.2.8.1 Troubleshooting
      • 17.1.2.8.2 Primary Key Updates
      • 17.1.2.8.3 Generic Wrapper Message
  • 17.1.3 Avro Object Container File Formatter
    • 17.1.3.1 Avro OCF Formatter Configuration Properties
  • 17.1.4 Setting Metacolumn Output
• 17.2 Using the Delimited Text Formatter
  • 17.2.1 Message Formatting Details
  • 17.2.2 Sample Formatted Messages
    • 17.2.2.1 Sample Insert Message
    • 17.2.2.2 Sample Update Message
    • 17.2.2.3 Sample Delete Message
    • 17.2.2.4 Sample Truncate Message
  • 17.2.3 Output Format Summary Log
  • 17.2.4 Delimited Text Formatter Configuration Properties
  • 17.2.5 Review a Sample Configuration
  • 17.2.6 Metadata Change Events
  • 17.2.7 Setting Metacolumn Output
  • 17.2.8 Additional Considerations
    • 17.2.8.1 Primary Key Updates
    • 17.2.8.2 Data Consolidation
• 17.3 Using the JSON Formatter
  • 17.3.1 Operation Metadata Formatting Details
  • 17.3.2 Operation Data Formatting Details
  • 17.3.3 Row Data Formatting Details
  • 17.3.4 Sample JSON Messages
    • 17.3.4.1 Sample Operation Modeled JSON Messages
    • 17.3.4.2 Sample Flattened Operation Modeled JSON Messages
    • 17.3.4.3 Sample Row Modeled JSON Messages
    • 17.3.4.4 Sample Primary Key Output JSON Message
  • 17.3.5 JSON Schemas
  • 17.3.6 JSON Formatter Configuration Properties
  • 17.3.7 Review a Sample Configuration
  • 17.3.8 Metadata Change Events
  • 17.3.9 Setting Metacolumn Output
  • 17.3.10 JSON Primary Key Updates
  • 17.3.11 Integrating Oracle Stream Analytics
• 17.4 Using the Length Delimited Value Formatter
  • 17.4.1 Formatting Message Details
  • 17.4.2 Sample Formatted Messages
  • 17.4.3 LDV Formatter Configuration Properties
  • 17.4.4 Additional Considerations
• 17.5 Using Operation-Based versus Row-Based Formatting
  • 17.5.1 Operation Formatters
  • 17.5.2 Row Formatters
  • 17.5.3 Table Row or Column Value States
• 17.6 Using the XML Formatter
  • 17.6.1 Message Formatting Details
  • 17.6.2 Sample XML Messages
    • 17.6.2.1 Sample Insert Message
    • 17.6.2.2 Sample Update Message
    • 17.6.2.3 Sample Delete Message
    • 17.6.2.4 Sample Truncate Message
  • 17.6.3 XML Schema
  • 17.6.4 XML Formatter Configuration Properties
  • 17.6.5 Review a Sample Configuration
  • 17.6.6 Metadata Change Events
  • 17.6.7 Setting Metacolumn Output
  • 17.6.8 Primary Key Updates

18 Using Oracle GoldenGate Capture for Cassandra

• 18.1 Overview
• 18.2 Setting Up Cassandra Change Data Capture
  • 18.2.1 Data Types
  • 18.2.2 Cassandra Database Operations
• 18.3 Deduplication
• 18.4 Topology Changes
• 18.5 Data Availability in the CDC Logs
• 18.6 Using Extract Initial Load
• 18.7 Using Change Data Capture Extract
• 18.8 Replicating to RDMBS Targets
• 18.9 Partition Update or Insert of Static Columns
• 18.10 Partition Delete
• 18.11 Security and Authentication
  • 18.11.1 Configuring SSL
• 18.12 Multiple Extract Support
• 18.13 CDC Configuration Reference
• 18.14 Troubleshooting

19 Connecting to Microsoft Azure Data Lake

A Cassandra Handler Client Dependencies

• A.1 Cassandra Datastax Java Driver 3.1.0

B Cassandra Capture Client Dependencies

C Elasticsearch Handler Client Dependencies

• C.1 Elasticsearch 2.4.4 and Shield Plugin 2.2.2
• C.2 Elasticsearch 5.1.2 with X-Pack 5.1.2

D Flume Handler Client Dependencies

• D.1 Flume 1.7.0
• D.2 Flume 1.6.0
• D.3 Flume 1.5.2
• D.4 Flume 1.4.0

E HBase Handler Client Dependencies

• E.1 HBase 1.2.5
• E.2 HBase 1.1.1
• E.3 HBase 1.0.1.1

F HDFS Handler Client Dependencies

• F.1 Hadoop Client Dependencies
  • F.1.1 HDFS 2.8.0
  • F.1.2 HDFS 2.7.1
  • F.1.3 HDFS 2.6.0
  • F.1.4 HDFS 2.5.2
  • F.1.5 HDFS 2.4.1
  • F.1.6 HDFS 2.3.0
  • F.1.7 HDFS 2.2.0

G Kafka Handler Client Dependencies

• G.1 Kafka 1.1.0
• G.2 Kafka 1.0.0
• G.3 Kafka 0.11.0.0
• G.4 Kafka 0.10.2.0
• G.5 Kafka 0.10.1.1
• G.6 Kafka 0.10.0.1
• G.7 Kafka 0.9.0.1

H Kafka Connect Handler Client Dependencies

• H.1 Kafka  0.11.0.0
• H.2 Kafka 0.10.2.0
• H.3 Kafka 0.10.2.0
• H.4 Kafka 0.10.0.0
• H.5 Kafka 0.9.0.1
• H.6 Confluent 4.1.2
• H.7 Confluent 4.0.0
• H.8 Confluent 3.2.1
• H.9 Confluent 3.2.0
• H.10 Confluent 3.2.1
• H.11 Confluent 3.1.1
• H.12 Confluent 3.0.1
• H.13 Confluent 2.0.1
• H.14 Confluent 2.0.1

I MongoDB Handler Client Dependencies

• I.1 MongoDB Java Driver 3.4.3

J Optimized Row Columnar Event Handler Client Dependencies

• J.1 ORC Client Dependencies

K Parquet Event Handler Client Dependencies

• K.1 Parquet Client Dependencies