org.apache.hadoop.hive.ql.exec.spark

Class SparkUtilities

java.lang.Object

org.apache.hadoop.hive.ql.exec.spark.SparkUtilities

public class SparkUtilities
extends Object

Contains utilities methods used as part of Spark tasks.

Constructor Summary

Constructors

Constructor and Description
SparkUtilities()

Method Summary

Methods

Modifier and Type	Method and Description
static org.apache.hadoop.io.BytesWritable	copyBytesWritable(org.apache.hadoop.io.BytesWritable bw)
static HiveKey	copyHiveKey(HiveKey key)
static SparkSession	getSparkSession(HiveConf conf, SparkSessionManager sparkSessionManager)
static URI	getURI(String path)
static boolean	needUploadToHDFS(URI source, org.apache.spark.SparkConf sparkConf)
static URI	uploadToHDFS(URI source, HiveConf conf) Uploads a local file to HDFS

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SparkUtilities

public SparkUtilities()

Method Detail

copyHiveKey

public static HiveKey copyHiveKey(HiveKey key)

copyBytesWritable

public static org.apache.hadoop.io.BytesWritable copyBytesWritable(org.apache.hadoop.io.BytesWritable bw)

getURI

public static URI getURI(String path)
                  throws URISyntaxException

Throws:

URISyntaxException

uploadToHDFS

public static URI uploadToHDFS(URI source,
               HiveConf conf)
                        throws IOException

Uploads a local file to HDFS

Parameters:

source -

conf -

Returns:

Throws:

IOException

needUploadToHDFS

public static boolean needUploadToHDFS(URI source,
                       org.apache.spark.SparkConf sparkConf)

getSparkSession

public static SparkSession getSparkSession(HiveConf conf,
                           SparkSessionManager sparkSessionManager)
                                    throws HiveException

Throws:

HiveException