| 限定符和类型 | 方法和说明 |
|---|---|
Configuration |
Configured.getConf() |
Configuration |
Configurable.getConf()
Return the configuration used by this object.
|
| 限定符和类型 | 方法和说明 |
|---|---|
static void |
Configuration.dumpConfiguration(Configuration config,
Writer out)
Writes out all the parameters and their properties (final and resource) to
the given
Writer
The format of the output would be
{ "properties" : [ {key1,value1,key1.isFinal,key1.resource}, {key2,value2,
key2.isFinal,key2.resource}... ] }
It does not output the parameters of the configuration object which is
loaded from an input stream. |
void |
Configured.setConf(Configuration conf) |
void |
Configurable.setConf(Configuration conf)
Set the configuration to be used by this object.
|
| 构造器和说明 |
|---|
Configuration(Configuration other)
A new configuration with the same settings cloned from another.
|
Configured(Configuration conf)
Construct a Configured.
|
| 构造器和说明 |
|---|
LineDocRecordReader(Configuration job,
FileSplit split)
Constructor
|
| 构造器和说明 |
|---|
FileSystemDirectory(FileSystem fs,
Path directory,
boolean create,
Configuration conf)
Constructor
|
| 限定符和类型 | 方法和说明 |
|---|---|
Configuration |
IndexUpdateConfiguration.getConfiguration()
Get the underlying configuration object.
|
| 限定符和类型 | 方法和说明 |
|---|---|
void |
IndexUpdater.run(Configuration conf,
Path[] inputPaths,
Path outputPath,
int numMapTasks,
Shard[] shards) |
void |
IIndexUpdater.run(Configuration conf,
Path[] inputPaths,
Path outputPath,
int numMapTasks,
Shard[] shards)
Create a Map/Reduce job configuration and run the Map/Reduce job to
analyze documents and update Lucene instances in parallel.
|
| 构造器和说明 |
|---|
IndexUpdateConfiguration(Configuration conf)
Constructor
|
| 构造器和说明 |
|---|
MultiFileWordCount.MultiFileLineRecordReader(Configuration conf,
MultiFileSplit split) |
| 限定符和类型 | 方法和说明 |
|---|---|
static void |
DistributedCache.addArchiveToClassPath(Path archive,
Configuration conf)
已过时。
Please use
DistributedCache.addArchiveToClassPath(Path, Configuration, FileSystem)
instead. The FileSystem should be obtained within an
appropriate doAs. |
static void |
DistributedCache.addArchiveToClassPath(Path archive,
Configuration conf,
FileSystem fs)
Add an archive path to the current set of classpath entries.
|
static void |
DistributedCache.addCacheArchive(URI uri,
Configuration conf)
Add a archives to be localized to the conf.
|
static void |
DistributedCache.addCacheFile(URI uri,
Configuration conf)
Add a file to be localized to the conf.
|
static void |
DistributedCache.addFileToClassPath(Path file,
Configuration conf)
已过时。
Please use
DistributedCache.addFileToClassPath(Path, Configuration, FileSystem)
instead. The FileSystem should be obtained within an
appropriate doAs. |
static void |
DistributedCache.addFileToClassPath(Path file,
Configuration conf,
FileSystem fs)
Add a file path to the current set of classpath entries.
|
static void |
DistributedCache.addLocalArchives(Configuration conf,
String str)
Add a archive that has been localized to the conf.
|
static void |
DistributedCache.addLocalFiles(Configuration conf,
String str)
Add a file that has been localized to the conf..
|
static void |
TrackerDistributedCacheManager.createAllSymlink(Configuration conf,
File jobCacheDir,
File workDir)
This method create symlinks for all files in a given dir in another
directory.
|
static void |
DistributedCache.createAllSymlink(Configuration conf,
File jobCacheDir,
File workDir)
已过时。
Internal to MapReduce framework. Use DistributedCacheManager
instead.
|
static void |
DistributedCache.createSymlink(Configuration conf)
This method allows you to create symlinks in the current working directory
of the task to all the cache files/archives.
|
static void |
TrackerDistributedCacheManager.determineTimestampsAndCacheVisibilities(Configuration job)
Determines timestamps of files to be cached, and stores those
in the configuration.
|
static long |
TrackerDistributedCacheManager.downloadCacheObject(Configuration conf,
URI source,
Path destination,
long desiredTimestamp,
boolean isArchive,
FsPermission permission)
Download a given path to the local file system.
|
static Path[] |
DistributedCache.getArchiveClassPaths(Configuration conf)
Get the archive entries in classpath as an array of Path.
|
static long[] |
DistributedCache.getArchiveTimestamps(Configuration conf)
Get the timestamps of the archives.
|
static boolean[] |
TrackerDistributedCacheManager.getArchiveVisibilities(Configuration conf)
Get the booleans on whether the archives are public or not.
|
static URI[] |
DistributedCache.getCacheArchives(Configuration conf)
Get cache archives set in the Configuration.
|
static URI[] |
DistributedCache.getCacheFiles(Configuration conf)
Get cache files set in the Configuration.
|
static void |
TrackerDistributedCacheManager.getDelegationTokens(Configuration job,
Credentials credentials)
For each archive or cache file - get the corresponding delegation token
|
static Path[] |
DistributedCache.getFileClassPaths(Configuration conf)
Get the file entries in classpath as an array of Path.
|
static FileStatus |
DistributedCache.getFileStatus(Configuration conf,
URI cache)
Returns
FileStatus of a given cache file on hdfs. |
static long[] |
DistributedCache.getFileTimestamps(Configuration conf)
Get the timestamps of the files.
|
static boolean[] |
TrackerDistributedCacheManager.getFileVisibilities(Configuration conf)
Get the booleans on whether the files are public or not.
|
static Path[] |
DistributedCache.getLocalCacheArchives(Configuration conf)
Return the path array of the localized caches.
|
static Path[] |
DistributedCache.getLocalCacheFiles(Configuration conf)
Return the path array of the localized files.
|
static boolean |
DistributedCache.getSymlink(Configuration conf)
This method checks to see if symlinks are to be create for the
localized cache files in the current working directory
Used by internal DistributedCache code.
|
static long |
DistributedCache.getTimestamp(Configuration conf,
URI cache)
Returns mtime of a given cache file on hdfs.
|
TaskDistributedCacheManager |
TrackerDistributedCacheManager.newTaskDistributedCacheManager(JobID jobId,
Configuration taskConf) |
static void |
DistributedCache.setArchiveTimestamps(Configuration conf,
String timestamps)
This is to check the timestamp of the archives to be localized.
|
static void |
DistributedCache.setCacheArchives(URI[] archives,
Configuration conf)
Set the configuration with the given set of archives.
|
static void |
DistributedCache.setCacheFiles(URI[] files,
Configuration conf)
Set the configuration with the given set of files.
|
static void |
DistributedCache.setFileTimestamps(Configuration conf,
String timestamps)
This is to check the timestamp of the files to be localized.
|
static void |
DistributedCache.setLocalArchives(Configuration conf,
String str)
Set the conf to contain the location for localized archives.
|
static void |
DistributedCache.setLocalFiles(Configuration conf,
String str)
Set the conf to contain the location for localized files.
|
void |
TaskDistributedCacheManager.setupCache(Configuration taskConf,
String publicCacheSubdir,
String privateCacheSubdir)
Retrieve public distributed cache files into the local cache and updates
the task configuration (which has been passed in via the constructor).
|
static void |
TrackerDistributedCacheManager.validate(Configuration conf)
This is part of the framework API.
|
| 构造器和说明 |
|---|
TrackerDistributedCacheManager.CleanupThread(Configuration conf) |
TrackerDistributedCacheManager(Configuration conf,
TaskController controller) |
| 限定符和类型 | 方法和说明 |
|---|---|
Configuration |
FilterFileSystem.getConf() |
| 限定符和类型 | 方法和说明 |
|---|---|
static void |
FileSystem.addFileSystemForTesting(URI uri,
Configuration conf,
FileSystem fs)
This method adds a file system for testing so that we can find it later.
|
static boolean |
FileUtil.copy(File src,
FileSystem dstFS,
Path dst,
boolean deleteSource,
Configuration conf)
Copy local files to a FileSystem.
|
static boolean |
FileUtil.copy(FileSystem srcFS,
Path[] srcs,
FileSystem dstFS,
Path dst,
boolean deleteSource,
boolean overwrite,
Configuration conf) |
static boolean |
FileUtil.copy(FileSystem srcFS,
Path src,
File dst,
boolean deleteSource,
Configuration conf)
Copy FileSystem files to local files.
|
static boolean |
FileUtil.copy(FileSystem srcFS,
Path src,
FileSystem dstFS,
Path dst,
boolean deleteSource,
boolean overwrite,
Configuration conf)
Copy files between FileSystems.
|
static boolean |
FileUtil.copy(FileSystem srcFS,
Path src,
FileSystem dstFS,
Path dst,
boolean deleteSource,
Configuration conf)
Copy files between FileSystems.
|
static boolean |
FileUtil.copyMerge(FileSystem srcFS,
Path srcDir,
FileSystem dstFS,
Path dstFile,
boolean deleteSource,
Configuration conf,
String addString)
Copy all files in a directory to one output file (merge).
|
File |
LocalDirAllocator.createTmpFileForWrite(String pathStr,
long size,
Configuration conf)
Creates a temporary file in the local FS.
|
static FileSystem |
FileSystem.get(Configuration conf)
Returns the configured filesystem implementation.
|
static FileSystem |
FileSystem.get(URI uri,
Configuration conf)
Returns the FileSystem for this URI's scheme and authority.
|
static FileSystem |
FileSystem.get(URI uri,
Configuration conf,
String user) |
Iterable<Path> |
LocalDirAllocator.getAllLocalPathsToRead(String pathStr,
Configuration conf)
Get all of the paths that currently exist in the working directories.
|
static URI |
FileSystem.getDefaultUri(Configuration conf)
Get the default filesystem URI from a configuration.
|
FileSystem |
Path.getFileSystem(Configuration conf)
Return the FileSystem that owns this Path.
|
static LocalFileSystem |
FileSystem.getLocal(Configuration conf)
Get the local file syste
|
Path |
LocalDirAllocator.getLocalPathForWrite(String pathStr,
Configuration conf)
Get a path from the local FS.
|
Path |
LocalDirAllocator.getLocalPathForWrite(String pathStr,
long size,
Configuration conf)
Get a path from the local FS.
|
Path |
LocalDirAllocator.getLocalPathForWrite(String pathStr,
long size,
Configuration conf,
boolean checkWrite)
Get a path from the local FS.
|
Path |
LocalDirAllocator.getLocalPathToRead(String pathStr,
Configuration conf)
Get a path from the local FS for reading.
|
static FileSystem |
FileSystem.getNamed(String name,
Configuration conf)
已过时。
call #get(URI,Configuration) instead.
|
boolean |
LocalDirAllocator.ifExists(String pathStr,
Configuration conf)
We search through all the configured dirs for the file's existence
and return true when we find
|
void |
RawLocalFileSystem.initialize(URI uri,
Configuration conf) |
void |
HarFileSystem.initialize(URI name,
Configuration conf)
Initialize a Har filesystem per har archive.
|
void |
FilterFileSystem.initialize(URI name,
Configuration conf)
Called after a new FileSystem instance is constructed.
|
void |
FileSystem.initialize(URI name,
Configuration conf)
Called after a new FileSystem instance is constructed.
|
void |
ChecksumFileSystem.setConf(Configuration conf) |
static void |
FileSystem.setDefaultUri(Configuration conf,
String uri)
Set the default filesystem URI in a configuration.
|
static void |
FileSystem.setDefaultUri(Configuration conf,
URI uri)
Set the default filesystem URI in a configuration.
|
| 构造器和说明 |
|---|
DF(File path,
Configuration conf) |
DU(File path,
Configuration conf)
Keeps track of disk usage.
|
FsShell(Configuration conf) |
FsUrlStreamHandlerFactory(Configuration conf) |
InMemoryFileSystem(URI uri,
Configuration conf)
已过时。
|
Trash(Configuration conf)
Construct a trash can accessor.
|
Trash(FileSystem fs,
Configuration conf)
Construct a trash can accessor for the FileSystem provided.
|
| 限定符和类型 | 方法和说明 |
|---|---|
void |
FTPFileSystem.initialize(URI uri,
Configuration conf) |
| 限定符和类型 | 方法和说明 |
|---|---|
void |
KosmosFileSystem.initialize(URI uri,
Configuration conf) |
| 限定符和类型 | 方法和说明 |
|---|---|
static FsPermission |
FsPermission.getUMask(Configuration conf)
Get the user file creation mask (umask)
UMASK_LABEL config param has umask value that is either symbolic
or octal. |
static void |
FsPermission.setUMask(Configuration conf,
FsPermission umask)
Set the user file creation mask (umask)
|
| 限定符和类型 | 方法和说明 |
|---|---|
void |
S3FileSystem.initialize(URI uri,
Configuration conf) |
void |
S3Credentials.initialize(URI uri,
Configuration conf) |
void |
FileSystemStore.initialize(URI uri,
Configuration conf) |
| 限定符和类型 | 方法和说明 |
|---|---|
void |
NativeS3FileSystem.initialize(URI uri,
Configuration conf) |
| 构造器和说明 |
|---|
Command(Configuration conf)
Constructor
|
Count(String[] cmd,
int pos,
Configuration conf)
Constructor
|
| 限定符和类型 | 方法和说明 |
|---|---|
void |
HftpFileSystem.TokenManager.cancel(Token<?> token,
Configuration conf) |
void |
DFSClient.Renewer.cancel(Token<?> token,
Configuration conf) |
static ClientProtocol |
DFSClient.createNamenode(Configuration conf)
Create a
NameNode proxy |
static ClientProtocol |
DFSClient.createNamenode(InetSocketAddress nameNodeAddr,
Configuration conf) |
static float |
DFSUtil.getInvalidateWorkPctPerIteration(Configuration conf)
Get DFS_NAMENODE_INVALIDATE_WORK_PCT_PER_ITERATION from configuration.
|
static int |
DFSUtil.getReplWorkMultiplier(Configuration conf)
Get DFS_NAMENODE_REPLICATION_WORK_MULTIPLIER_PER_ITERATION
from configuration.
|
void |
HsftpFileSystem.initialize(URI name,
Configuration conf) |
void |
HftpFileSystem.initialize(URI name,
Configuration conf) |
void |
DistributedFileSystem.initialize(URI uri,
Configuration conf) |
long |
HftpFileSystem.TokenManager.renew(Token<?> token,
Configuration conf) |
long |
DFSClient.Renewer.renew(Token<?> token,
Configuration conf) |
| 构造器和说明 |
|---|
ChecksumDistributedFileSystem(InetSocketAddress namenode,
Configuration conf)
已过时。
|
DFSClient(Configuration conf)
Same as this(NameNode.getAddress(conf), conf);
|
DFSClient(InetSocketAddress nameNodeAddr,
Configuration conf)
Same as this(nameNodeAddr, conf, null);
|
DFSClient(InetSocketAddress nameNodeAddr,
Configuration conf,
FileSystem.Statistics stats)
Same as this(nameNodeAddr, null, conf, stats);
|
DistributedFileSystem(InetSocketAddress namenode,
Configuration conf)
已过时。
|
| 限定符和类型 | 方法和说明 |
|---|---|
static Token<DelegationTokenIdentifier> |
DelegationTokenSelector.selectHdfsDelegationToken(InetSocketAddress nnAddr,
UserGroupInformation ugi,
Configuration conf)
Select the delegation token for hdfs from the ugi.
|
| 限定符和类型 | 方法和说明 |
|---|---|
Configuration |
Balancer.getConf()
return this balancer's configuration
|
| 限定符和类型 | 方法和说明 |
|---|---|
void |
Balancer.setConf(Configuration conf)
set this balancer's configuration
|
| 限定符和类型 | 方法和说明 |
|---|---|
static DataNode |
DataNode.createDataNode(String[] args,
Configuration conf)
Instantiate & Start a single datanode daemon and wait for it to finish.
|
static DataNode |
DataNode.createDataNode(String[] args,
Configuration conf,
SecureDataNodeStarter.SecureResources resources)
Instantiate & Start a single datanode daemon and wait for it to finish.
|
static InterDatanodeProtocol |
DataNode.createInterDataNodeProtocolProxy(DatanodeInfo info,
Configuration conf,
int socketTimeout,
boolean connectToDnViaHostname) |
static InetSocketAddress |
DataNode.getInfoAddr(Configuration conf)
Determine the http server's effective addr
|
static InetSocketAddress |
DataNode.getStreamingAddr(Configuration conf) |
static DataNode |
DataNode.instantiateDataNode(String[] args,
Configuration conf)
Instantiate a single datanode object.
|
static DataNode |
DataNode.instantiateDataNode(String[] args,
Configuration conf,
SecureDataNodeStarter.SecureResources resources)
Instantiate a single datanode object.
|
static DataNode |
DataNode.makeInstance(String[] dataDirs,
Configuration conf,
SecureDataNodeStarter.SecureResources resources)
Make an instance of DataNode after ensuring that at least one of the
given data directories (and their parent directories, if necessary)
can be created.
|
| 构造器和说明 |
|---|
FSDataset(DataStorage storage,
Configuration conf)
An FSDataset has a directory where it loads its data files.
|
| 限定符和类型 | 方法和说明 |
|---|---|
static DataNodeInstrumentation |
DataNodeInstrumentation.create(Configuration conf,
String storageID) |
static DataNodeInstrumentation |
DataNodeInstrumentation.create(Configuration conf,
String storageID,
MetricsSystem ms) |
| 构造器和说明 |
|---|
DataNodeInstrumentation(Configuration conf,
String storageId) |
| 限定符和类型 | 方法和说明 |
|---|---|
static NameNode |
NameNode.createNameNode(String[] argv,
Configuration conf) |
static void |
NameNode.format(Configuration conf)
Format a new filesystem.
|
static InetSocketAddress |
NameNode.getAddress(Configuration conf) |
static int |
JspHelper.getDefaultChunkSize(Configuration conf)
Get the default chunk size.
|
static UserGroupInformation |
JspHelper.getDefaultWebUser(Configuration conf)
If security is turned off, what is the default web user?
|
static DFSClient |
JspHelper.getDFSClient(UserGroupInformation user,
InetSocketAddress addr,
Configuration conf) |
static InetSocketAddress |
SecondaryNameNode.getHttpAddress(Configuration conf) |
static String |
NameNode.getInfoServer(Configuration conf) |
static BlockPlacementPolicy |
BlockPlacementPolicy.getInstance(Configuration conf,
FSClusterStats stats,
NetworkTopology clusterMap)
Get an instance of the configured Block Placement Policy based on the
value of the configuration paramater dfs.block.replicator.classname.
|
static Collection<File> |
FSNamesystem.getNamespaceDirs(Configuration conf) |
static Collection<File> |
FSNamesystem.getNamespaceEditsDirs(Configuration conf) |
static InetSocketAddress |
NameNode.getServiceAddress(Configuration conf,
boolean fallback)
Fetches the address for services to use when connecting to namenode
based on the value of fallback returns null if the special
address is not specified or returns the default namenode address
to be used by both clients and services.
|
protected InetSocketAddress |
NameNode.getServiceRpcServerAddress(Configuration conf)
Given a configuration get the address of the service rpc server
If the service rpc is not configured returns null
|
static UserGroupInformation |
JspHelper.getUGI(javax.servlet.http.HttpServletRequest request,
Configuration conf)
Same as getUGI(null, request, conf).
|
static UserGroupInformation |
JspHelper.getUGI(javax.servlet.ServletContext context,
javax.servlet.http.HttpServletRequest request,
Configuration conf)
Get
UserGroupInformation and possibly the delegation token out of
the request. |
static UserGroupInformation |
JspHelper.getUGI(javax.servlet.ServletContext context,
javax.servlet.http.HttpServletRequest request,
Configuration conf,
UserGroupInformation.AuthenticationMethod secureAuthMethod,
boolean tryUgiParameter)
Get
UserGroupInformation and possibly the delegation token out of
the request. |
void |
BlockPlacementPolicyWithNodeGroup.initialize(Configuration conf,
FSClusterStats stats,
NetworkTopology clusterMap) |
void |
BlockPlacementPolicyDefault.initialize(Configuration conf,
FSClusterStats stats,
NetworkTopology clusterMap)
Used to setup a BlockPlacementPolicy object.
|
protected abstract void |
BlockPlacementPolicy.initialize(Configuration conf,
FSClusterStats stats,
NetworkTopology clusterMap)
Used to setup a BlockPlacementPolicy object.
|
void |
FSNamesystem.refreshNodes(Configuration conf)
Rereads the config to get hosts and exclude list file names.
|
protected void |
NameNode.setRpcServiceServerAddress(Configuration conf)
Modifies the configuration passed to contain the service rpc address setting
|
static void |
NameNode.setServiceAddress(Configuration conf,
String address)
Set the configuration property for the service rpc address
to address
|
void |
JspHelper.streamBlockInAscii(InetSocketAddress addr,
long blockId,
Token<BlockTokenIdentifier> accessToken,
long genStamp,
long blockSize,
long offsetIntoBlock,
long chunkSizeToView,
javax.servlet.jsp.JspWriter out,
Configuration conf) |
| 构造器和说明 |
|---|
NameNode(Configuration conf)
Start NameNode.
|
SecondaryNameNode(Configuration conf)
Create a connection to the primary namenode.
|
| 限定符和类型 | 方法和说明 |
|---|---|
static NameNodeInstrumentation |
NameNodeInstrumentation.create(Configuration conf) |
static NameNodeInstrumentation |
NameNodeInstrumentation.create(Configuration conf,
MetricsSystem ms)
Create a v2 metrics instrumentation
|
| 限定符和类型 | 方法和说明 |
|---|---|
static void |
DelegationTokenFetcher.cancelDelegationToken(String protocol,
InetSocketAddress addr,
Token<DelegationTokenIdentifier> tok,
Configuration conf)
Cancel a Delegation Token.
|
static Credentials |
DelegationTokenFetcher.getDTfromRemote(String protocol,
InetSocketAddress nnAddr,
String renewer,
Configuration conf)
Utility method to obtain a delegation token over http
|
protected static String |
DelegationTokenFetcher.getRenewAddress(String protocol,
InetSocketAddress addr,
Configuration conf)
Get the URI that we use for getting, renewing, and cancelling the
delegation token.
|
static long |
DelegationTokenFetcher.renewDelegationToken(String protocol,
InetSocketAddress addr,
Token<DelegationTokenIdentifier> tok,
Configuration conf)
Renew a Delegation Token.
|
static void |
DelegationTokenFetcher.setupSsl(Configuration conf)
Set up SSL resources
|
| 构造器和说明 |
|---|
DFSAdmin(Configuration conf)
Construct a DFSAdmin object.
|
DFSck(Configuration conf)
Filesystem checker.
|
| 限定符和类型 | 方法和说明 |
|---|---|
void |
WebHdfsFileSystem.DtRenewer.cancel(Token<?> token,
Configuration conf) |
void |
WebHdfsFileSystem.initialize(URI uri,
Configuration conf) |
static boolean |
WebHdfsFileSystem.isEnabled(Configuration conf,
org.apache.commons.logging.Log log)
Is WebHDFS enabled in conf?
|
long |
WebHdfsFileSystem.DtRenewer.renew(Token<?> token,
Configuration conf) |
| 限定符和类型 | 方法和说明 |
|---|---|
short |
ReplicationParam.getValue(Configuration conf) |
int |
BufferSizeParam.getValue(Configuration conf) |
long |
BlockSizeParam.getValue(Configuration conf) |
| 限定符和类型 | 方法和说明 |
|---|---|
void |
HttpServer.addSslListener(InetSocketAddress addr,
Configuration sslConf,
boolean needClientAuth)
Configure an ssl listener on the server.
|
void |
HttpServer.addSslListener(InetSocketAddress addr,
Configuration sslConf,
boolean needCertsAuth,
boolean needKrbAuth)
Configure an ssl listener on the server.
|
org.mortbay.jetty.Connector |
HttpServer.createBaseListener(Configuration conf)
Create a required listener for the Jetty instance listening on the port
provided.
|
abstract void |
FilterInitializer.initFilter(FilterContainer container,
Configuration conf)
Initialize a Filter to a FilterContainer.
|
| 构造器和说明 |
|---|
HttpServer(String name,
String bindAddress,
int port,
boolean findPort,
Configuration conf) |
HttpServer(String name,
String bindAddress,
int port,
boolean findPort,
Configuration conf,
AccessControlList adminsAcl)
Create a status server on the given port.
|
HttpServer(String name,
String bindAddress,
int port,
boolean findPort,
Configuration conf,
AccessControlList adminsAcl,
org.mortbay.jetty.Connector connector) |
HttpServer(String name,
String bindAddress,
int port,
boolean findPort,
Configuration conf,
org.mortbay.jetty.Connector connector) |
| 限定符和类型 | 方法和说明 |
|---|---|
void |
StaticUserWebFilter.initFilter(FilterContainer container,
Configuration conf) |
| 限定符和类型 | 方法和说明 |
|---|---|
Configuration |
ObjectWritable.getConf() |
Configuration |
GenericWritable.getConf() |
Configuration |
AbstractMapWritable.getConf() |
| 限定符和类型 | 方法和说明 |
|---|---|
static <T extends Writable> |
WritableUtils.clone(T orig,
Configuration conf)
Make a copy of a writable object using serialization to a buffer.
|
static void |
IOUtils.copyBytes(InputStream in,
OutputStream out,
Configuration conf)
Copies from one stream to another.
|
static void |
IOUtils.copyBytes(InputStream in,
OutputStream out,
Configuration conf,
boolean close)
Copies from one stream to another.
|
protected SequenceFile.Reader |
MapFile.Reader.createDataFileReader(FileSystem fs,
Path dataFile,
Configuration conf)
Override this method to specialize the type of
SequenceFile.Reader returned. |
static SequenceFile.Writer |
SequenceFile.createWriter(Configuration conf,
FSDataOutputStream out,
Class keyClass,
Class valClass,
SequenceFile.CompressionType compressionType,
CompressionCodec codec)
Construct the preferred type of 'raw' SequenceFile Writer.
|
static SequenceFile.Writer |
SequenceFile.createWriter(Configuration conf,
FSDataOutputStream out,
Class keyClass,
Class valClass,
SequenceFile.CompressionType compressionType,
CompressionCodec codec,
SequenceFile.Metadata metadata)
Construct the preferred type of 'raw' SequenceFile Writer.
|
static SequenceFile.Writer |
SequenceFile.createWriter(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass)
Construct the preferred type of SequenceFile Writer.
|
static SequenceFile.Writer |
SequenceFile.createWriter(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass,
int bufferSize,
short replication,
long blockSize,
boolean createParent,
SequenceFile.CompressionType compressionType,
CompressionCodec codec,
SequenceFile.Metadata metadata)
Construct the preferred type of SequenceFile Writer.
|
static SequenceFile.Writer |
SequenceFile.createWriter(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass,
int bufferSize,
short replication,
long blockSize,
SequenceFile.CompressionType compressionType,
CompressionCodec codec,
Progressable progress,
SequenceFile.Metadata metadata)
Construct the preferred type of SequenceFile Writer.
|
static SequenceFile.Writer |
SequenceFile.createWriter(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass,
SequenceFile.CompressionType compressionType)
Construct the preferred type of SequenceFile Writer.
|
static SequenceFile.Writer |
SequenceFile.createWriter(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass,
SequenceFile.CompressionType compressionType,
CompressionCodec codec)
Construct the preferred type of SequenceFile Writer.
|
static SequenceFile.Writer |
SequenceFile.createWriter(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass,
SequenceFile.CompressionType compressionType,
CompressionCodec codec,
Progressable progress)
Construct the preferred type of SequenceFile Writer.
|
static SequenceFile.Writer |
SequenceFile.createWriter(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass,
SequenceFile.CompressionType compressionType,
CompressionCodec codec,
Progressable progress,
SequenceFile.Metadata metadata)
Construct the preferred type of SequenceFile Writer.
|
static SequenceFile.Writer |
SequenceFile.createWriter(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass,
SequenceFile.CompressionType compressionType,
Progressable progress)
Construct the preferred type of SequenceFile Writer.
|
static long |
MapFile.fix(FileSystem fs,
Path dir,
Class<? extends Writable> keyClass,
Class<? extends Writable> valueClass,
boolean dryrun,
Configuration conf)
This method attempts to fix a corrupt MapFile by re-creating its index.
|
static Class<?> |
WritableName.getClass(String name,
Configuration conf)
Return the class for a name.
|
static SequenceFile.CompressionType |
SequenceFile.getCompressionType(Configuration job)
已过时。
Use
SequenceFileOutputFormat.getOutputCompressionType(org.apache.hadoop.mapred.JobConf)
to get SequenceFile.CompressionType for job-outputs. |
static <K> K |
DefaultStringifier.load(Configuration conf,
String keyName,
Class<K> itemClass)
Restores the object from the configuration.
|
static <K> K[] |
DefaultStringifier.loadArray(Configuration conf,
String keyName,
Class<K> itemClass)
Restores the array of objects from the configuration.
|
static Writable |
WritableFactories.newInstance(Class<? extends Writable> c,
Configuration conf)
Create a new instance of a class with a defined factory.
|
protected void |
MapFile.Reader.open(FileSystem fs,
String dirName,
WritableComparator comparator,
Configuration conf) |
static Object |
ObjectWritable.readObject(DataInput in,
Configuration conf)
|
static Object |
ObjectWritable.readObject(DataInput in,
ObjectWritable objectWritable,
Configuration conf)
|
static void |
SequenceFile.setCompressionType(Configuration job,
SequenceFile.CompressionType val)
已过时。
Use the one of the many SequenceFile.createWriter methods to specify
the
SequenceFile.CompressionType while creating the SequenceFile or
SequenceFileOutputFormat.setOutputCompressionType(org.apache.hadoop.mapred.JobConf, org.apache.hadoop.io.SequenceFile.CompressionType)
to specify the SequenceFile.CompressionType for job-outputs.
or |
void |
ObjectWritable.setConf(Configuration conf) |
void |
GenericWritable.setConf(Configuration conf) |
void |
AbstractMapWritable.setConf(Configuration conf) |
static void |
MapFile.Writer.setIndexInterval(Configuration conf,
int interval)
Sets the index interval and stores it in conf
|
static <K> void |
DefaultStringifier.store(Configuration conf,
K item,
String keyName)
Stores the item in the configuration with the given keyName.
|
static <K> void |
DefaultStringifier.storeArray(Configuration conf,
K[] items,
String keyName)
Stores the array of items in the configuration with the given keyName.
|
static void |
ObjectWritable.writeObject(DataOutput out,
Object instance,
Class declaredClass,
Configuration conf)
|
| 构造器和说明 |
|---|
ArrayFile.Reader(FileSystem fs,
String file,
Configuration conf)
Construct an array reader for the named file.
|
ArrayFile.Writer(Configuration conf,
FileSystem fs,
String file,
Class<? extends Writable> valClass)
Create the named file for values of the named class.
|
ArrayFile.Writer(Configuration conf,
FileSystem fs,
String file,
Class<? extends Writable> valClass,
SequenceFile.CompressionType compress,
Progressable progress)
Create the named file for values of the named class.
|
BloomMapFile.Reader(FileSystem fs,
String dirName,
Configuration conf) |
BloomMapFile.Reader(FileSystem fs,
String dirName,
WritableComparator comparator,
Configuration conf) |
BloomMapFile.Reader(FileSystem fs,
String dirName,
WritableComparator comparator,
Configuration conf,
boolean open) |
BloomMapFile.Writer(Configuration conf,
FileSystem fs,
String dirName,
Class<? extends WritableComparable> keyClass,
Class valClass) |
BloomMapFile.Writer(Configuration conf,
FileSystem fs,
String dirName,
Class<? extends WritableComparable> keyClass,
Class<? extends Writable> valClass,
SequenceFile.CompressionType compress,
CompressionCodec codec,
Progressable progress) |
BloomMapFile.Writer(Configuration conf,
FileSystem fs,
String dirName,
Class<? extends WritableComparable> keyClass,
Class valClass,
SequenceFile.CompressionType compress) |
BloomMapFile.Writer(Configuration conf,
FileSystem fs,
String dirName,
Class<? extends WritableComparable> keyClass,
Class valClass,
SequenceFile.CompressionType compress,
Progressable progress) |
BloomMapFile.Writer(Configuration conf,
FileSystem fs,
String dirName,
WritableComparator comparator,
Class valClass) |
BloomMapFile.Writer(Configuration conf,
FileSystem fs,
String dirName,
WritableComparator comparator,
Class valClass,
SequenceFile.CompressionType compress) |
BloomMapFile.Writer(Configuration conf,
FileSystem fs,
String dirName,
WritableComparator comparator,
Class valClass,
SequenceFile.CompressionType compress,
CompressionCodec codec,
Progressable progress) |
BloomMapFile.Writer(Configuration conf,
FileSystem fs,
String dirName,
WritableComparator comparator,
Class valClass,
SequenceFile.CompressionType compress,
Progressable progress) |
DefaultStringifier(Configuration conf,
Class<T> c) |
MapFile.Reader(FileSystem fs,
String dirName,
Configuration conf)
Construct a map reader for the named map.
|
MapFile.Reader(FileSystem fs,
String dirName,
WritableComparator comparator,
Configuration conf)
Construct a map reader for the named map using the named comparator.
|
MapFile.Reader(FileSystem fs,
String dirName,
WritableComparator comparator,
Configuration conf,
boolean open)
Hook to allow subclasses to defer opening streams until further
initialization is complete.
|
MapFile.Writer(Configuration conf,
FileSystem fs,
String dirName,
Class<? extends WritableComparable> keyClass,
Class valClass)
Create the named map for keys of the named class.
|
MapFile.Writer(Configuration conf,
FileSystem fs,
String dirName,
Class<? extends WritableComparable> keyClass,
Class valClass,
SequenceFile.CompressionType compress)
Create the named map for keys of the named class.
|
MapFile.Writer(Configuration conf,
FileSystem fs,
String dirName,
Class<? extends WritableComparable> keyClass,
Class valClass,
SequenceFile.CompressionType compress,
CompressionCodec codec,
Progressable progress)
Create the named map for keys of the named class.
|
MapFile.Writer(Configuration conf,
FileSystem fs,
String dirName,
Class<? extends WritableComparable> keyClass,
Class valClass,
SequenceFile.CompressionType compress,
Progressable progress)
Create the named map for keys of the named class.
|
MapFile.Writer(Configuration conf,
FileSystem fs,
String dirName,
WritableComparator comparator,
Class valClass)
Create the named map using the named key comparator.
|
MapFile.Writer(Configuration conf,
FileSystem fs,
String dirName,
WritableComparator comparator,
Class valClass,
SequenceFile.CompressionType compress)
Create the named map using the named key comparator.
|
MapFile.Writer(Configuration conf,
FileSystem fs,
String dirName,
WritableComparator comparator,
Class valClass,
SequenceFile.CompressionType compress,
CompressionCodec codec,
Progressable progress)
Create the named map using the named key comparator.
|
MapFile.Writer(Configuration conf,
FileSystem fs,
String dirName,
WritableComparator comparator,
Class valClass,
SequenceFile.CompressionType compress,
Progressable progress)
Create the named map using the named key comparator.
|
SequenceFile.Reader(FileSystem fs,
Path file,
Configuration conf)
Open the named file.
|
SequenceFile.Sorter(FileSystem fs,
Class<? extends WritableComparable> keyClass,
Class valClass,
Configuration conf)
Sort and merge files containing the named classes.
|
SequenceFile.Sorter(FileSystem fs,
RawComparator comparator,
Class keyClass,
Class valClass,
Configuration conf)
Sort and merge using an arbitrary
RawComparator. |
SequenceFile.Writer(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass)
Create the named file.
|
SequenceFile.Writer(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass,
int bufferSize,
short replication,
long blockSize,
Progressable progress,
SequenceFile.Metadata metadata)
Create the named file with write-progress reporter.
|
SequenceFile.Writer(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass,
Progressable progress,
SequenceFile.Metadata metadata)
Create the named file with write-progress reporter.
|
SetFile.Reader(FileSystem fs,
String dirName,
Configuration conf)
Construct a set reader for the named set.
|
SetFile.Reader(FileSystem fs,
String dirName,
WritableComparator comparator,
Configuration conf)
Construct a set reader for the named set using the named comparator.
|
SetFile.Writer(Configuration conf,
FileSystem fs,
String dirName,
Class<? extends WritableComparable> keyClass,
SequenceFile.CompressionType compress)
Create a set naming the element class and compression type.
|
SetFile.Writer(Configuration conf,
FileSystem fs,
String dirName,
WritableComparator comparator,
SequenceFile.CompressionType compress)
Create a set naming the element comparator and compression type.
|
| 限定符和类型 | 方法和说明 |
|---|---|
Configuration |
SnappyCodec.getConf()
Return the configuration used by this object.
|
Configuration |
DefaultCodec.getConf() |
| 限定符和类型 | 方法和说明 |
|---|---|
static List<Class<? extends CompressionCodec>> |
CompressionCodecFactory.getCodecClasses(Configuration conf)
Get the list of codecs listed in the configuration
|
static Compressor |
CodecPool.getCompressor(CompressionCodec codec,
Configuration conf)
Get a
Compressor for the given CompressionCodec from the
pool or a new one. |
static boolean |
SnappyCodec.isNativeSnappyLoaded(Configuration conf)
Are the native snappy libraries loaded & initialized?
|
void |
Compressor.reinit(Configuration conf)
Prepare the compressor to be used in a new stream with settings defined in
the given Configuration
|
static void |
CompressionCodecFactory.setCodecClasses(Configuration conf,
List<Class> classes)
Sets a list of codec classes in the configuration.
|
void |
SnappyCodec.setConf(Configuration conf)
Set the configuration to be used by this object.
|
void |
DefaultCodec.setConf(Configuration conf) |
| 构造器和说明 |
|---|
CompressionCodecFactory(Configuration conf)
Find the codecs specified in the config value io.compression.codecs
and register them.
|
| 限定符和类型 | 方法和说明 |
|---|---|
void |
BZip2DummyCompressor.reinit(Configuration conf) |
| 限定符和类型 | 方法和说明 |
|---|---|
void |
SnappyCompressor.reinit(Configuration conf)
Prepare the compressor to be used in a new stream with settings defined in
the given Configuration
|
| 限定符和类型 | 方法和说明 |
|---|---|
static ZlibCompressor.CompressionLevel |
ZlibFactory.getCompressionLevel(Configuration conf) |
static ZlibCompressor.CompressionStrategy |
ZlibFactory.getCompressionStrategy(Configuration conf) |
static Compressor |
ZlibFactory.getZlibCompressor(Configuration conf)
Return the appropriate implementation of the zlib compressor.
|
static Class<? extends Compressor> |
ZlibFactory.getZlibCompressorType(Configuration conf)
Return the appropriate type of the zlib compressor.
|
static Decompressor |
ZlibFactory.getZlibDecompressor(Configuration conf)
Return the appropriate implementation of the zlib decompressor.
|
static Class<? extends Decompressor> |
ZlibFactory.getZlibDecompressorType(Configuration conf)
Return the appropriate type of the zlib decompressor.
|
static boolean |
ZlibFactory.isNativeZlibLoaded(Configuration conf)
Check if native-zlib code is loaded & initialized correctly and
can be loaded for this job.
|
void |
ZlibCompressor.reinit(Configuration conf)
Prepare the compressor to be used in a new stream with settings defined in
the given Configuration.
|
void |
BuiltInZlibDeflater.reinit(Configuration conf)
reinit the compressor with the given configuration.
|
static void |
ZlibFactory.setCompressionLevel(Configuration conf,
ZlibCompressor.CompressionLevel level) |
static void |
ZlibFactory.setCompressionStrategy(Configuration conf,
ZlibCompressor.CompressionStrategy strategy) |
| 构造器和说明 |
|---|
ZlibCompressor(Configuration conf)
Creates a new compressor, taking settings from the configuration.
|
| 构造器和说明 |
|---|
TFile.Reader(FSDataInputStream fsdis,
long fileLength,
Configuration conf)
Constructor
|
TFile.Writer(FSDataOutputStream fsdos,
int minBlockSize,
String compressName,
String comparator,
Configuration conf)
Constructor
|
| 限定符和类型 | 方法和说明 |
|---|---|
static RetryPolicy |
RetryUtils.getDefaultRetryPolicy(Configuration conf,
String retryPolicyEnabledKey,
boolean defaultRetryPolicyEnabled,
String retryPolicySpecKey,
String defaultRetryPolicySpec,
Class<? extends Exception>... remoteExceptionsToRetry)
Return the default retry policy set in conf.
|
static RetryPolicy |
RetryUtils.getMultipleLinearRandomRetry(Configuration conf,
String retryPolicyEnabledKey,
boolean defaultRetryPolicyEnabled,
String retryPolicySpecKey,
String defaultRetryPolicySpec)
Return the MultipleLinearRandomRetry policy specified in the conf,
or null if the feature is disabled.
|
| 构造器和说明 |
|---|
SerializationFactory(Configuration conf)
Serializations are found by reading the
io.serializations
property from conf, which is a comma-delimited list of
classnames. |
| 限定符和类型 | 方法和说明 |
|---|---|
static Object[] |
RPC.call(Method method,
Object[][] params,
InetSocketAddress[] addrs,
Configuration conf)
|
static Object[] |
RPC.call(Method method,
Object[][] params,
InetSocketAddress[] addrs,
UserGroupInformation ticket,
Configuration conf)
Expert: Make multiple, parallel calls to a set of servers.
|
Writable[] |
Client.call(Writable[] params,
InetSocketAddress[] addresses,
Class<?> protocol,
UserGroupInformation ticket,
Configuration conf)
Makes a set of calls in parallel.
|
Writable |
Client.call(Writable param,
InetSocketAddress addr,
Class<?> protocol,
UserGroupInformation ticket,
int rpcTimeout,
Configuration conf)
Make a call, passing
param, to the IPC server running at
address which is servicing the protocol protocol,
with the ticket credentials, rpcTimeout as timeout
and conf as configuration for this connection, returning the
value. |
static VersionedProtocol |
RPC.getProxy(Class<? extends VersionedProtocol> protocol,
long clientVersion,
InetSocketAddress addr,
Configuration conf)
Construct a client-side proxy object with the default SocketFactory
|
static VersionedProtocol |
RPC.getProxy(Class<? extends VersionedProtocol> protocol,
long clientVersion,
InetSocketAddress addr,
Configuration conf,
int rpcTimeout) |
static VersionedProtocol |
RPC.getProxy(Class<? extends VersionedProtocol> protocol,
long clientVersion,
InetSocketAddress addr,
Configuration conf,
SocketFactory factory)
Construct a client-side proxy object that implements the named protocol,
talking to a server at the named address.
|
static VersionedProtocol |
RPC.getProxy(Class<? extends VersionedProtocol> protocol,
long clientVersion,
InetSocketAddress addr,
Configuration conf,
SocketFactory factory,
int rpcTimeout)
Construct a client-side proxy object that implements the named protocol,
talking to a server at the named address.
|
static VersionedProtocol |
RPC.getProxy(Class<? extends VersionedProtocol> protocol,
long clientVersion,
InetSocketAddress addr,
UserGroupInformation ticket,
Configuration conf,
SocketFactory factory)
Construct a client-side proxy object that implements the named protocol,
talking to a server at the named address.
|
static VersionedProtocol |
RPC.getProxy(Class<? extends VersionedProtocol> protocol,
long clientVersion,
InetSocketAddress addr,
UserGroupInformation ticket,
Configuration conf,
SocketFactory factory,
int rpcTimeout)
Construct a client-side proxy object that implements the named protocol,
talking to a server at the named address.
|
static VersionedProtocol |
RPC.getProxy(Class<? extends VersionedProtocol> protocol,
long clientVersion,
InetSocketAddress addr,
UserGroupInformation ticket,
Configuration conf,
SocketFactory factory,
int rpcTimeout,
RetryPolicy connectionRetryPolicy,
boolean checkVersion)
Construct a client-side proxy object that implements the named protocol,
talking to a server at the named address.
|
static RPC.Server |
RPC.getServer(Object instance,
String bindAddress,
int port,
Configuration conf)
Construct a server for a protocol implementation instance listening on a
port and address.
|
static RPC.Server |
RPC.getServer(Object instance,
String bindAddress,
int port,
int numHandlers,
boolean verbose,
Configuration conf)
Construct a server for a protocol implementation instance listening on a
port and address.
|
static RPC.Server |
RPC.getServer(Object instance,
String bindAddress,
int port,
int numHandlers,
boolean verbose,
Configuration conf,
SecretManager<? extends TokenIdentifier> secretManager)
Construct a server for a protocol implementation instance listening on a
port and address, with a secret manager.
|
static int |
Client.getTimeout(Configuration conf)
The time after which a RPC will timeout.
|
static void |
Client.setPingInterval(Configuration conf,
int pingInterval)
set the ping interval value in configuration
|
static VersionedProtocol |
RPC.waitForProxy(Class<? extends VersionedProtocol> protocol,
long clientVersion,
InetSocketAddress addr,
Configuration conf) |
| 构造器和说明 |
|---|
Client(Class<? extends Writable> valueClass,
Configuration conf)
Construct an IPC client with the default SocketFactory
|
Client(Class<? extends Writable> valueClass,
Configuration conf,
SocketFactory factory)
Construct an IPC client whose values are of the given
Writable
class. |
RPC.Server(Object instance,
Configuration conf,
String bindAddress,
int port)
Construct an RPC server.
|
RPC.Server(Object instance,
Configuration conf,
String bindAddress,
int port,
int numHandlers,
boolean verbose,
SecretManager<? extends TokenIdentifier> secretManager)
Construct an RPC server.
|
Server(String bindAddress,
int port,
Class<? extends Writable> paramClass,
int handlerCount,
Configuration conf) |
Server(String bindAddress,
int port,
Class<? extends Writable> paramClass,
int handlerCount,
Configuration conf,
String serverName) |
Server(String bindAddress,
int port,
Class<? extends Writable> paramClass,
int handlerCount,
Configuration conf,
String serverName,
SecretManager<? extends TokenIdentifier> secretManager)
Constructs a server listening on the named port and address.
|
| 限定符和类型 | 类和说明 |
|---|---|
class |
JobConf
A map/reduce job configuration.
|
| 限定符和类型 | 字段和说明 |
|---|---|
protected Configuration |
SequenceFileRecordReader.conf |
| 限定符和类型 | 方法和说明 |
|---|---|
Configuration |
TaskController.getConf() |
Configuration |
Task.getConf() |
Configuration |
SequenceFileInputFilter.FilterBase.getConf() |
| 限定符和类型 | 方法和说明 |
|---|---|
void |
UserLogCleaner.addOldUserLogsForDeletion(File loc,
Configuration conf)
Adds the job log directories for deletion with default retain hours.
|
void |
JobClient.Renewer.cancel(Token<?> token,
Configuration conf) |
void |
UserLogCleaner.clearOldUserLogs(Configuration conf)
Clears all the logs in userlogs directory.
|
protected static <INKEY,INVALUE,OUTKEY,OUTVALUE> |
Task.createReduceContext(Reducer<INKEY,INVALUE,OUTKEY,OUTVALUE> reducer,
Configuration job,
TaskAttemptID taskId,
RawKeyValueIterator rIter,
Counter inputKeyCounter,
Counter inputValueCounter,
RecordWriter<OUTKEY,OUTVALUE> output,
OutputCommitter committer,
StatusReporter reporter,
RawComparator<INKEY> comparator,
Class<INKEY> keyClass,
Class<INVALUE> valueClass) |
static long[] |
JobLocalizer.downloadPrivateCache(Configuration conf)
Download the parts of the distributed cache that are private.
|
static InetSocketAddress |
JobTracker.getAddress(Configuration conf) |
static int |
SkipBadRecords.getAttemptsToStartSkipping(Configuration conf)
Get the number of Task attempts AFTER which skip mode
will be kicked off.
|
static boolean |
SkipBadRecords.getAutoIncrMapperProcCount(Configuration conf)
Get the flag which if set to true,
SkipBadRecords.COUNTER_MAP_PROCESSED_RECORDS is incremented
by MapRunner after invoking the map function. |
static boolean |
SkipBadRecords.getAutoIncrReducerProcCount(Configuration conf)
Get the flag which if set to true,
SkipBadRecords.COUNTER_REDUCE_PROCESSED_GROUPS is incremented
by framework after invoking the reduce function. |
protected static FileSystem.Statistics |
Task.getFsStatistics(Path path,
Configuration conf)
Gets a handle to the Statistics instance based on the scheme associated
with path.
|
static long |
SkipBadRecords.getMapperMaxSkipRecords(Configuration conf)
Get the number of acceptable skip records surrounding the bad record PER
bad record in mapper.
|
static SequenceFile.Reader[] |
SequenceFileOutputFormat.getReaders(Configuration conf,
Path dir)
Open the output generated by this format.
|
static MapFile.Reader[] |
MapFileOutputFormat.getReaders(FileSystem ignored,
Path dir,
Configuration conf)
Open the output generated by this format.
|
static long |
SkipBadRecords.getReducerMaxSkipGroups(Configuration conf)
Get the number of acceptable skip groups surrounding the bad group PER
bad group in reducer.
|
static Path |
SkipBadRecords.getSkipOutputPath(Configuration conf)
Get the directory to which skipped records are written.
|
static String |
JobHistory.JobInfo.getWorkflowAdjacencies(Configuration conf)
Get the workflow adjacencies from the job conf
The string returned is of the form "key"="value" "key"="value" ...
|
long |
JobClient.Renewer.renew(Token<?> token,
Configuration conf) |
static void |
SkipBadRecords.setAttemptsToStartSkipping(Configuration conf,
int attemptsToStartSkipping)
Set the number of Task attempts AFTER which skip mode
will be kicked off.
|
static void |
SkipBadRecords.setAutoIncrMapperProcCount(Configuration conf,
boolean autoIncr)
Set the flag which if set to true,
SkipBadRecords.COUNTER_MAP_PROCESSED_RECORDS is incremented
by MapRunner after invoking the map function. |
static void |
SkipBadRecords.setAutoIncrReducerProcCount(Configuration conf,
boolean autoIncr)
Set the flag which if set to true,
SkipBadRecords.COUNTER_REDUCE_PROCESSED_GROUPS is incremented
by framework after invoking the reduce function. |
void |
TaskController.setConf(Configuration conf) |
void |
Task.setConf(Configuration conf) |
void |
SequenceFileInputFilter.RegexFilter.setConf(Configuration conf)
configure the Filter by checking the configuration
|
void |
SequenceFileInputFilter.PercentFilter.setConf(Configuration conf)
configure the filter by checking the configuration
|
void |
SequenceFileInputFilter.MD5Filter.setConf(Configuration conf)
configure the filter according to configuration
|
void |
DefaultTaskController.setConf(Configuration conf) |
static void |
SequenceFileInputFilter.setFilterClass(Configuration conf,
Class filterClass)
set the filter class
|
static void |
SequenceFileInputFilter.PercentFilter.setFrequency(Configuration conf,
int frequency)
set the frequency and stores it in conf
|
static void |
SequenceFileInputFilter.MD5Filter.setFrequency(Configuration conf,
int frequency)
set the filtering frequency in configuration
|
static void |
SkipBadRecords.setMapperMaxSkipRecords(Configuration conf,
long maxSkipRecs)
Set the number of acceptable skip records surrounding the bad record PER
bad record in mapper.
|
static void |
SequenceFileInputFilter.RegexFilter.setPattern(Configuration conf,
String regex)
Define the filtering regex and stores it in conf
|
static void |
SkipBadRecords.setReducerMaxSkipGroups(Configuration conf,
long maxSkipGrps)
Set the number of acceptable skip groups surrounding the bad group PER
bad group in reducer.
|
| 限定符和类型 | 方法和说明 |
|---|---|
Configuration |
CompositeRecordReader.getConf()
Return the configuration used by this object.
|
| 限定符和类型 | 方法和说明 |
|---|---|
void |
CompositeRecordReader.setConf(Configuration conf)
Set the configuration to be used by this object.
|
| 限定符和类型 | 方法和说明 |
|---|---|
Configuration |
InputSampler.getConf() |
| 限定符和类型 | 方法和说明 |
|---|---|
void |
InputSampler.setConf(Configuration conf) |
| 构造器和说明 |
|---|
Submitter(Configuration conf) |
| 构造器和说明 |
|---|
MRAdmin(Configuration conf) |
| 限定符和类型 | 方法和说明 |
|---|---|
Configuration |
JobContext.getConfiguration()
Return the configuration for the job.
|
| 限定符和类型 | 方法和说明 |
|---|---|
static Job |
Job.getInstance(Configuration conf)
Creates a new
Job with a given Configuration. |
static Job |
Job.getInstance(Configuration conf,
String jobName)
Creates a new
Job with a given Configuration
and a given jobName. |
static Path |
JobSubmissionFiles.getStagingDir(JobClient client,
Configuration conf)
Initializes the staging directory and returns the path.
|
| 限定符和类型 | 方法和说明 |
|---|---|
Configuration |
DBInputFormat.getConf() |
Configuration |
DBConfiguration.getConf() |
| 限定符和类型 | 方法和说明 |
|---|---|
static void |
DBConfiguration.configureDB(Configuration job,
String driverClass,
String dbUrl)
Sets the DB access related fields in the JobConf.
|
static void |
DBConfiguration.configureDB(Configuration conf,
String driverClass,
String dbUrl,
String userName,
String passwd)
Sets the DB access related fields in the
Configuration. |
protected RecordReader<LongWritable,T> |
OracleDataDrivenDBInputFormat.createDBRecordReader(DBInputFormat.DBInputSplit split,
Configuration conf) |
protected RecordReader<LongWritable,T> |
DBInputFormat.createDBRecordReader(DBInputFormat.DBInputSplit split,
Configuration conf) |
protected RecordReader<LongWritable,T> |
DataDrivenDBInputFormat.createDBRecordReader(DBInputFormat.DBInputSplit split,
Configuration conf) |
static void |
DataDrivenDBInputFormat.setBoundingQuery(Configuration conf,
String query)
Set the user-defined bounding query to use with a user-defined query.
|
void |
DBInputFormat.setConf(Configuration conf)
Set the configuration to be used by this object.
|
static void |
OracleDBRecordReader.setSessionTimeZone(Configuration conf,
Connection conn)
Set session time zone
|
List<InputSplit> |
TextSplitter.split(Configuration conf,
ResultSet results,
String colName)
This method needs to determine the splits between two user-provided strings.
|
List<InputSplit> |
IntegerSplitter.split(Configuration conf,
ResultSet results,
String colName) |
List<InputSplit> |
FloatSplitter.split(Configuration conf,
ResultSet results,
String colName) |
List<InputSplit> |
DBSplitter.split(Configuration conf,
ResultSet results,
String colName)
Given a ResultSet containing one record (and already advanced to that record)
with two columns (a low value, and a high value, both of the same type), determine
a set of splits that span the given values.
|
List<InputSplit> |
DateSplitter.split(Configuration conf,
ResultSet results,
String colName) |
List<InputSplit> |
BooleanSplitter.split(Configuration conf,
ResultSet results,
String colName) |
List<InputSplit> |
BigDecimalSplitter.split(Configuration conf,
ResultSet results,
String colName) |
| 限定符和类型 | 字段和说明 |
|---|---|
protected Configuration |
SequenceFileRecordReader.conf |
| 限定符和类型 | 方法和说明 |
|---|---|
Configuration |
SequenceFileInputFilter.FilterBase.getConf() |
| 限定符和类型 | 方法和说明 |
|---|---|
static List<FileSplit> |
NLineInputFormat.getSplitsForFile(FileStatus status,
Configuration conf,
int numLinesPerSplit) |
void |
SequenceFileInputFilter.RegexFilter.setConf(Configuration conf)
configure the Filter by checking the configuration
|
void |
SequenceFileInputFilter.PercentFilter.setConf(Configuration conf)
configure the filter by checking the configuration
|
void |
SequenceFileInputFilter.MD5Filter.setConf(Configuration conf)
configure the filter according to configuration
|
static void |
SequenceFileInputFilter.PercentFilter.setFrequency(Configuration conf,
int frequency)
set the frequency and stores it in conf
|
static void |
SequenceFileInputFilter.MD5Filter.setFrequency(Configuration conf,
int frequency)
set the filtering frequency in configuration
|
static void |
SequenceFileInputFilter.RegexFilter.setPattern(Configuration conf,
String regex)
Define the filtering regex and stores it in conf
|
| 构造器和说明 |
|---|
KeyValueLineRecordReader(Configuration conf) |
| 构造器和说明 |
|---|
ControlledJob(Configuration conf)
Construct a job.
|
| 限定符和类型 | 方法和说明 |
|---|---|
Configuration |
TotalOrderPartitioner.getConf() |
Configuration |
KeyFieldBasedPartitioner.getConf() |
Configuration |
KeyFieldBasedComparator.getConf() |
Configuration |
BinaryPartitioner.getConf() |
| 限定符和类型 | 方法和说明 |
|---|---|
static String |
TotalOrderPartitioner.getPartitionFile(Configuration conf)
Get the path to the SequenceFile storing the sorted partition keyset.
|
void |
TotalOrderPartitioner.setConf(Configuration conf)
Read in the partition file and build indexing data structures.
|
void |
KeyFieldBasedPartitioner.setConf(Configuration conf) |
void |
KeyFieldBasedComparator.setConf(Configuration conf) |
void |
BinaryPartitioner.setConf(Configuration conf) |
static void |
BinaryPartitioner.setLeftOffset(Configuration conf,
int offset)
Set the subarray to be used for partitioning to
bytes[offset:] in Python syntax. |
static void |
BinaryPartitioner.setOffsets(Configuration conf,
int left,
int right)
Set the subarray to be used for partitioning to
bytes[left:(right+1)] in Python syntax. |
static void |
TotalOrderPartitioner.setPartitionFile(Configuration conf,
Path p)
Set the path to the SequenceFile storing the sorted partition keyset.
|
static void |
BinaryPartitioner.setRightOffset(Configuration conf,
int offset)
Set the subarray to be used for partitioning to
bytes[:(offset+1)] in Python syntax. |
| 构造器和说明 |
|---|
InputSampler(Configuration conf) |
| 限定符和类型 | 方法和说明 |
|---|---|
static void |
TokenCache.cleanUpTokenReferral(Configuration conf)
Remove jobtoken referrals which don't make sense in the context
of the task execution.
|
static Credentials |
TokenCache.loadTokens(String jobTokenFile,
Configuration conf)
load job token from a file
|
static void |
TokenCache.obtainTokensForNamenodes(Credentials credentials,
Path[] ps,
Configuration conf)
Convenience method to obtain delegation tokens from namenodes
corresponding to the paths passed.
|
| 限定符和类型 | 方法和说明 |
|---|---|
static void |
DelegationTokenRenewal.registerDelegationTokensForRenewal(JobID jobId,
Credentials ts,
Configuration conf) |
| 限定符和类型 | 方法和说明 |
|---|---|
void |
UserLogManager.clearOldUserLogs(Configuration conf)
Called during TaskTracker restart/re-init.
|
| 构造器和说明 |
|---|
UserLogManager(Configuration conf)
Create the user log manager to manage user logs on
TaskTracker. |
UserLogManager(Configuration conf,
TaskController taskController)
Create the user log manager to manage user logs on
TaskTracker. |
| 限定符和类型 | 方法和说明 |
|---|---|
static void |
JobSplitWriter.createSplitFiles(Path jobSubmitDir,
Configuration conf,
FileSystem fs,
InputSplit[] splits) |
static <T extends InputSplit> |
JobSplitWriter.createSplitFiles(Path jobSubmitDir,
Configuration conf,
FileSystem fs,
List<InputSplit> splits) |
static <T extends InputSplit> |
JobSplitWriter.createSplitFiles(Path jobSubmitDir,
Configuration conf,
FileSystem fs,
T[] splits) |
static JobSplit.TaskSplitMetaInfo[] |
SplitMetaInfoReader.readSplitMetaInfo(JobID jobId,
FileSystem fs,
Configuration conf,
Path jobSubmitDir) |
| 限定符和类型 | 方法和说明 |
|---|---|
Configuration |
SocksSocketFactory.getConf() |
Configuration |
ScriptBasedMapping.getConf() |
| 限定符和类型 | 方法和说明 |
|---|---|
static SocketFactory |
NetUtils.getDefaultSocketFactory(Configuration conf)
Get the default socket factory as specified by the configuration
parameter hadoop.rpc.socket.factory.default
|
static NetworkTopology |
NetworkTopology.getInstance(Configuration conf)
Get an instance of NetworkTopology based on the value of the configuration
parameter net.topology.impl.
|
static String |
NetUtils.getServerAddress(Configuration conf,
String oldBindAddressName,
String oldPortName,
String newBindAddressName)
已过时。
|
static SocketFactory |
NetUtils.getSocketFactory(Configuration conf,
Class<?> clazz)
Get the socket factory for the given class according to its
configuration parameter
hadoop.rpc.socket.factory.class.
|
static SocketFactory |
NetUtils.getSocketFactoryFromProperty(Configuration conf,
String propValue)
Get the socket factory corresponding to the given proxy URI.
|
void |
SocksSocketFactory.setConf(Configuration conf) |
void |
ScriptBasedMapping.setConf(Configuration conf) |
| 构造器和说明 |
|---|
ScriptBasedMapping(Configuration conf) |
| 限定符和类型 | 方法和说明 |
|---|---|
static AccessControlList |
SecurityUtil.getAdminAcls(Configuration conf,
String configKey)
Get the ACL object representing the cluster administrators
The user who starts the daemon is automatically added as an admin
|
static Groups |
Groups.getUserToGroupsMappingService(Configuration conf) |
static void |
SaslRpcServer.init(Configuration conf) |
void |
AuthenticationFilterInitializer.initFilter(FilterContainer container,
Configuration conf)
Initializes Alfredo AuthenticationFilter.
|
static void |
SecurityUtil.login(Configuration conf,
String keytabFileKey,
String userNameKey)
If a keytab has been provided, login as that user.
|
static void |
SecurityUtil.login(Configuration conf,
String keytabFileKey,
String userNameKey,
String hostname)
If a keytab has been provided, login as that user.
|
static Credentials |
Credentials.readTokenStorageFile(Path filename,
Configuration conf)
Convenience method for reading a token storage file, and loading the Tokens
therein in the passed UGI
|
static void |
UserGroupInformation.setConfiguration(Configuration conf)
Set the static configuration for UGI.
|
static void |
KerberosName.setConfiguration(Configuration conf)
Set the static configuration to get the rules.
|
void |
Credentials.writeTokenStorageFile(Path filename,
Configuration conf) |
| 构造器和说明 |
|---|
Groups(Configuration conf) |
| 限定符和类型 | 方法和说明 |
|---|---|
static void |
ServiceAuthorizationManager.authorize(UserGroupInformation user,
Class<?> protocol,
Configuration conf,
InetAddress addr)
Authorize the user to access the protocol being used.
|
static void |
ProxyUsers.authorize(UserGroupInformation user,
String remoteAddress,
Configuration newConf)
Authorize the superuser which is doing doAs
|
static void |
ServiceAuthorizationManager.refresh(Configuration conf,
PolicyProvider provider) |
static void |
ProxyUsers.refreshSuperUserGroupsConfiguration(Configuration conf)
refresh configuration
|
| 限定符和类型 | 方法和说明 |
|---|---|
void |
Token.cancel(Configuration conf)
Cancel this delegation token
|
abstract void |
TokenRenewer.cancel(Token<?> token,
Configuration conf)
Cancel the given token
|
void |
Token.TrivialRenewer.cancel(Token<?> token,
Configuration conf) |
long |
Token.renew(Configuration conf)
Renew this delegation token
|
abstract long |
TokenRenewer.renew(Token<?> token,
Configuration conf)
Renew the given token.
|
long |
Token.TrivialRenewer.renew(Token<?> token,
Configuration conf) |
| 限定符和类型 | 字段和说明 |
|---|---|
protected Configuration |
StreamJob.config_ |
| 限定符和类型 | 方法和说明 |
|---|---|
Configuration |
StreamJob.getConf() |
Configuration |
LoadTypedBytes.getConf() |
Configuration |
DumpTypedBytes.getConf() |
Configuration |
PipeMapRed.getConfiguration()
Returns the Configuration.
|
| 限定符和类型 | 方法和说明 |
|---|---|
static Class |
StreamUtil.goodClassOrNull(Configuration conf,
String className,
String defaultPackage)
It may seem strange to silently switch behaviour when a String
is not a classname; the reason is simplified Usage:
-mapper [classname | program ]
instead of the explicit Usage:
[-mapper program | -javamapper classname], -mapper and -javamapper are mutually exclusive.
|
void |
StreamJob.setConf(Configuration conf) |
void |
LoadTypedBytes.setConf(Configuration conf) |
void |
DumpTypedBytes.setConf(Configuration conf) |
| 构造器和说明 |
|---|
DumpTypedBytes(Configuration conf) |
LoadTypedBytes(Configuration conf) |
| 限定符和类型 | 方法和说明 |
|---|---|
Configuration |
Logalyzer.LogComparator.getConf() |
Configuration |
HadoopArchives.getConf() |
Configuration |
DistCp.getConf() |
| 限定符和类型 | 方法和说明 |
|---|---|
static void |
DistCp.copy(Configuration conf,
String srcPath,
String destPath,
Path logPath,
boolean srcAsList,
boolean ignoreReadFailures)
已过时。
|
void |
Logalyzer.LogComparator.setConf(Configuration conf) |
void |
HadoopArchives.setConf(Configuration conf) |
void |
DistCp.setConf(Configuration conf) |
| 构造器和说明 |
|---|
DistCp(Configuration conf) |
HadoopArchives(Configuration conf) |
| 限定符和类型 | 方法和说明 |
|---|---|
static void |
DistCpOptionSwitch.addToConf(Configuration conf,
DistCpOptionSwitch option)
Helper function to set an option to hadoop configuration object
|
static void |
DistCpOptionSwitch.addToConf(Configuration conf,
DistCpOptionSwitch option,
String value)
Helper function to add an option to hadoop configuration object
|
void |
DistCpOptions.appendToConf(Configuration conf)
Add options to configuration.
|
static CopyListing |
CopyListing.getCopyListing(Configuration configuration,
Credentials credentials,
DistCpOptions options)
Public Factory method with which the appropriate CopyListing implementation may be retrieved.
|
| 构造器和说明 |
|---|
CopyListing(Configuration configuration,
Credentials credentials)
Protected constructor, to initialize configuration.
|
DistCp(Configuration configuration,
DistCpOptions inputOptions)
Public Constructor.
|
FileBasedCopyListing(Configuration configuration,
Credentials credentials)
Constructor, to initialize base-class.
|
GlobbedCopyListing(Configuration configuration,
Credentials credentials)
Constructor, to initialize the configuration.
|
SimpleCopyListing(Configuration configuration,
Credentials credentials)
Protected constructor, to initialize configuration.
|
| 限定符和类型 | 方法和说明 |
|---|---|
static long |
DistCpUtils.getFileSize(Path path,
Configuration configuration)
Retrieves size of the file at the specified path.
|
static int |
DistCpUtils.getInt(Configuration configuration,
String label)
Utility to retrieve a specified key from a Configuration.
|
static long |
DistCpUtils.getLong(Configuration configuration,
String label)
Utility to retrieve a specified key from a Configuration.
|
static int |
DistCpUtils.getNumMapTasks(Configuration configuration) |
static Class<? extends InputFormat> |
DistCpUtils.getStrategy(Configuration conf,
DistCpOptions options)
Returns the class that implements a copy strategy.
|
static <T> void |
DistCpUtils.publish(Configuration configuration,
String label,
T value)
Utility to publish a value to a configuration.
|
static Path |
DistCpUtils.sortListing(FileSystem fs,
Configuration conf,
Path sourceListing)
Sort sequence file containing FileStatus and Text as key and value respecitvely
|
| 限定符和类型 | 方法和说明 |
|---|---|
void |
InputDemuxer.bindTo(Path path,
Configuration conf)
Bind the
InputDemuxer to a particular file. |
void |
DefaultInputDemuxer.bindTo(Path path,
Configuration conf) |
void |
Outputter.init(Path path,
Configuration conf)
Initialize the
Outputter to a specific path. |
void |
DefaultOutputter.init(Path path,
Configuration conf) |
| 构造器和说明 |
|---|
ClusterTopologyReader(Path path,
Configuration conf)
Constructor.
|
JobTraceReader(Path path,
Configuration conf)
Constructor.
|
ZombieCluster(Path path,
MachineNode defaultNode,
Configuration conf)
Construct a homogeneous cluster.
|
ZombieJobProducer(Path path,
ZombieCluster cluster,
Configuration conf)
Constructor
|
ZombieJobProducer(Path path,
ZombieCluster cluster,
Configuration conf,
long randomSeed)
Constructor
|
| 限定符和类型 | 方法和说明 |
|---|---|
Configuration |
TypedBytesWritableInput.getConf() |
| 限定符和类型 | 方法和说明 |
|---|---|
void |
TypedBytesWritableInput.setConf(Configuration conf) |
| 限定符和类型 | 方法和说明 |
|---|---|
Configuration |
GenericOptionsParser.getConfiguration()
Get the modified configuration
|
| 限定符和类型 | 方法和说明 |
|---|---|
static <T> T |
ReflectionUtils.copy(Configuration conf,
T src,
T dst)
Make a copy of the writable object using serialization to a buffer
|
static URL[] |
GenericOptionsParser.getLibJars(Configuration conf)
If libjars are set in the conf, parse the libjars.
|
boolean |
NativeCodeLoader.getLoadNativeLibraries(Configuration conf)
Return if native hadoop libraries, if present, can be used for this job.
|
static MemoryCalculatorPlugin |
MemoryCalculatorPlugin.getMemoryCalculatorPlugin(Class<? extends MemoryCalculatorPlugin> clazz,
Configuration conf)
已过时。
Get the MemoryCalculatorPlugin from the class name and configure it.
|
static ResourceCalculatorPlugin |
ResourceCalculatorPlugin.getResourceCalculatorPlugin(Class<? extends ResourceCalculatorPlugin> clazz,
Configuration conf)
Get the ResourceCalculatorPlugin from the class name and configure it.
|
static String[] |
Shell.getUlimitMemoryCommand(Configuration conf)
已过时。
|
static <T> T |
ReflectionUtils.newInstance(Class<T> theClass,
Configuration conf)
Create an object for the given class and initialize it from conf
|
static int |
ToolRunner.run(Configuration conf,
Tool tool,
String[] args)
Runs the given
Tool by Tool.run(String[]), after
parsing with the given generic arguments. |
static void |
ReflectionUtils.setConf(Object theObject,
Configuration conf)
Check and set 'configuration' if necessary.
|
void |
NativeCodeLoader.setLoadNativeLibraries(Configuration conf,
boolean loadNativeLibraries)
Set if native hadoop libraries, if present, can be used for this job.
|
| 构造器和说明 |
|---|
GenericOptionsParser(Configuration conf,
org.apache.commons.cli.Options options,
String[] args)
Create a
GenericOptionsParser to parse given options as well
as generic Hadoop options. |
GenericOptionsParser(Configuration conf,
String[] args)
Create a
GenericOptionsParser |
LineReader(InputStream in,
Configuration conf)
Create a line reader that reads from the given stream using the
io.file.buffer.size specified in the given
Configuration. |
LineReader(InputStream in,
Configuration conf,
byte[] recordDelimiterBytes)
Create a line reader that reads from the given stream using the
io.file.buffer.size specified in the given
Configuration, and using a custom delimiter of array of
bytes. |
| 限定符和类型 | 方法和说明 |
|---|---|
static int |
Hash.getHashType(Configuration conf)
This utility method converts the name of the configured
hash type to a symbolic constant.
|
static Hash |
Hash.getInstance(Configuration conf)
Get a singleton instance of hash function of a type
defined in the configuration.
|
Copyright © 2009 The Apache Software Foundation