org.apache.hadoop.hdfs

类 DFSClient

java.lang.Object

org.apache.hadoop.hdfs.DFSClient

所有已实现的接口:

Closeable, AutoCloseable, FSConstants

public class DFSClient
extends Object
implements FSConstants, Closeable

DFSClient can connect to a Hadoop Filesystem and perform basic file tasks. It uses the ClientProtocol to communicate with a NameNode daemon, and connects directly to DataNodes to read/write block data. Hadoop DFS users should obtain an instance of DistributedFileSystem, which uses DFSClient to handle filesystem tasks.

嵌套类概要

嵌套类

限定符和类型	类和说明
static class	DFSClient.DFSDataInputStream The Hdfs implementation of FSDataInputStream
class	DFSClient.DFSInputStream DFSInputStream provides bytes from a named file.
static class	DFSClient.RemoteBlockReader This is a wrapper around connection to datadone and understands checksum, offset etc
static class	DFSClient.Renewer

从接口继承的嵌套类/接口 org.apache.hadoop.hdfs.protocol.FSConstants

FSConstants.DatanodeReportType, FSConstants.SafeModeAction, FSConstants.UpgradeAction

字段概要

字段

限定符和类型	字段和说明
static org.apache.commons.logging.Log	LOG
static int	MAX_BLOCK_ACQUIRE_FAILURES
ClientProtocol	namenode

从接口继承的字段 org.apache.hadoop.hdfs.protocol.FSConstants

BLOCKREPORT_INITIAL_DELAY, BLOCKREPORT_INTERVAL, BUFFER_SIZE, DEFAULT_BLOCK_SIZE, DEFAULT_DATA_SOCKET_SIZE, HEARTBEAT_INTERVAL, LAYOUT_VERSION, LEASE_HARDLIMIT_PERIOD, LEASE_RECOVER_PERIOD, LEASE_SOFTLIMIT_PERIOD, MAX_PATH_DEPTH, MAX_PATH_LENGTH, MIN_BLOCKS_FOR_WRITE, QUOTA_DONT_SET, QUOTA_RESET, SIZE_OF_INTEGER, SMALL_BUFFER_SIZE

构造器概要

构造器

构造器和说明
DFSClient(Configuration conf) Same as this(NameNode.getAddress(conf), conf);
DFSClient(InetSocketAddress nameNodeAddr, Configuration conf) Same as this(nameNodeAddr, conf, null);
DFSClient(InetSocketAddress nameNodeAddr, Configuration conf, FileSystem.Statistics stats) Same as this(nameNodeAddr, null, conf, stats);

方法概要

方法

限定符和类型	方法和说明
FSDataOutputStream	append(String src, int buffersize, Progressable progress, FileSystem.Statistics statistics) Append to an existing HDFS file.
void	cancelDelegationToken(Token<DelegationTokenIdentifier> token) 已过时。 Use Token.cancel instead.
void	close() Close the file system, abandoning all of the leases and files being created and close connections to the namenode.
void	concat(String trg, String[] srcs) Move blocks from src to trg and delete src See ClientProtocol.concat(String, String []).
OutputStream	create(String src, boolean overwrite) Create a new dfs file and return an output stream for writing into it.
OutputStream	create(String src, boolean overwrite, Progressable progress) Create a new dfs file and return an output stream for writing into it with write-progress reporting.
OutputStream	create(String src, boolean overwrite, short replication, long blockSize) Create a new dfs file with the specified block replication and return an output stream for writing into the file.
OutputStream	create(String src, boolean overwrite, short replication, long blockSize, Progressable progress) Create a new dfs file with the specified block replication with write-progress reporting and return an output stream for writing into the file.
OutputStream	create(String src, boolean overwrite, short replication, long blockSize, Progressable progress, int buffersize) Call create(String,FsPermission,boolean,short,long,Progressable,int) with default permission.
OutputStream	create(String src, FsPermission permission, boolean overwrite, boolean createParent, short replication, long blockSize, Progressable progress, int buffersize) Create a new dfs file with the specified block replication with write-progress reporting and return an output stream for writing into the file.
OutputStream	create(String src, FsPermission permission, boolean overwrite, short replication, long blockSize, Progressable progress, int buffersize) Call create(String,FsPermission,boolean,boolean,short,long,Progressable,int) with createParent set to true.
static ClientProtocol	createNamenode(Configuration conf) Create a NameNode proxy
static ClientProtocol	createNamenode(InetSocketAddress nameNodeAddr, Configuration conf)
DatanodeInfo[]	datanodeReport(FSConstants.DatanodeReportType type)
boolean	delete(String src) 已过时。
boolean	delete(String src, boolean recursive) delete file or directory.
UpgradeStatusReport	distributedUpgradeProgress(FSConstants.UpgradeAction action)
boolean	exists(String src) Implemented using getFileInfo(src)
void	finalizeUpgrade()
BlockLocation[]	getBlockLocations(String src, long start, long length) Get block location info about file getBlockLocations() returns a list of hostnames that store data for a specific file region.
long	getBlockSize(String f)
long	getCorruptBlocksCount() Returns count of blocks with at least one replica marked corrupt.
long	getDefaultBlockSize() Get the default block size for this cluster
short	getDefaultReplication()
Token<DelegationTokenIdentifier>	getDelegationToken(Text renewer)
DistributedFileSystem.DiskStatus	getDiskStatus()
MD5MD5CRC32FileChecksum	getFileChecksum(String src) Get the checksum of a file.
static MD5MD5CRC32FileChecksum	getFileChecksum(String src, ClientProtocol namenode, SocketFactory socketFactory, int socketTimeout) Get the checksum of a file.
HdfsFileStatus	getFileInfo(String src)
String[][]	getHints(String src, long start, long length) 已过时。 Use getBlockLocations instead Get hints about the location of the indicated block(s). getHints() returns a list of hostnames that store data for a specific file region. It returns a set of hostnames for every block within the indicated region. This function is very useful when writing code that considers data-placement when performing operations. For example, the MapReduce system tries to schedule tasks on the same machines as the data-block the task processes.
org.apache.hadoop.hdfs.LeaseRenewer	getLeaseRenewer() Return the lease renewer instance.
long	getMissingBlocksCount() Returns count of blocks with no good replicas left.
long	getUnderReplicatedBlocksCount() Returns count of blocks with one of more replica missing.
boolean	isDirectory(String src) 已过时。 Use getHdfsFileStatus() instead
boolean	isFileClosed(String src) Close status of a file
DirectoryListing	listPaths(String src, byte[] startAfter) Get a partial listing of the indicated directory Recommend to use HdfsFileStatus.EMPTY_NAME as startAfter if the application wants to fetch a listing starting from the first entry in the directory
void	metaSave(String pathname) Dumps DFS data structures into specified file.
boolean	mkdirs(String src)
boolean	mkdirs(String src, FsPermission permission) Create a directory (or hierarchy of directories) with the given name and permission.
DFSClient.DFSInputStream	open(String src)
DFSClient.DFSInputStream	open(String src, int buffersize, boolean verifyChecksum, FileSystem.Statistics stats) Create an input stream that obtains a nodelist from the namenode, and then reads from all the right places.
void	refreshNodes() Refresh the hosts and exclude files.
boolean	rename(String src, String dst) Rename file or directory.
long	renewDelegationToken(Token<DelegationTokenIdentifier> token) 已过时。 Use Token.renew instead.
void	reportBadBlocks(LocatedBlock[] blocks) Report corrupt blocks that were discovered by the client.
void	setBalancerBandwidth(long bandwidth) Requests the namenode to tell all datanodes to use a new, non-persistent bandwidth value for dfs.balance.bandwidthPerSec.
void	setOwner(String src, String username, String groupname) Set file or directory owner.
void	setPermission(String src, FsPermission permission) Set permissions to a file or directory.
boolean	setReplication(String src, short replication) Set replication for an existing file.
boolean	setSafeMode(FSConstants.SafeModeAction action) Enter, leave or get safe mode.
void	setTimes(String src, long mtime, long atime) set the modification and access time of a file
static String	stringifyToken(Token<DelegationTokenIdentifier> token) A test method for printing out tokens
String	toString()
long	totalRawCapacity()
long	totalRawUsed()

从类继承的方法 java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

字段详细资料

LOG

public static final org.apache.commons.logging.Log LOG

MAX_BLOCK_ACQUIRE_FAILURES

public static final int MAX_BLOCK_ACQUIRE_FAILURES

另请参阅:

常量字段值

namenode

public final ClientProtocol namenode

构造器详细资料

DFSClient

public DFSClient(Configuration conf)
          throws IOException

Same as this(NameNode.getAddress(conf), conf);

抛出:

IOException

另请参阅:

DFSClient(InetSocketAddress, Configuration)

DFSClient

public DFSClient(InetSocketAddress nameNodeAddr,
         Configuration conf)
          throws IOException

Same as this(nameNodeAddr, conf, null);

抛出:

IOException

另请参阅:

DFSClient(InetSocketAddress, Configuration, org.apache.hadoop.fs.FileSystem.Statistics)

DFSClient

public DFSClient(InetSocketAddress nameNodeAddr,
         Configuration conf,
         FileSystem.Statistics stats)
          throws IOException

Same as this(nameNodeAddr, null, conf, stats);

抛出:

IOException

另请参阅:

DFSClient(InetSocketAddress, ClientProtocol, Configuration, org.apache.hadoop.fs.FileSystem.Statistics)

方法详细资料

createNamenode

public static ClientProtocol createNamenode(Configuration conf)
                                     throws IOException

Create a NameNode proxy

抛出:

IOException

createNamenode

public static ClientProtocol createNamenode(InetSocketAddress nameNodeAddr,
                            Configuration conf)
                                     throws IOException

抛出:

IOException

getLeaseRenewer

public org.apache.hadoop.hdfs.LeaseRenewer getLeaseRenewer()
                                                    throws IOException

Return the lease renewer instance. The renewer thread won't start until the first output stream is created. The same instance will be returned until all output streams are closed.

抛出:

IOException

close

public void close()
           throws IOException

Close the file system, abandoning all of the leases and files being created and close connections to the namenode.

指定者:

close 在接口中 Closeable

指定者:

close 在接口中 AutoCloseable

抛出:

IOException

getDefaultBlockSize

public long getDefaultBlockSize()

Get the default block size for this cluster

返回:

the default block size in bytes

getBlockSize

public long getBlockSize(String f)
                  throws IOException

抛出:

IOException

stringifyToken

public static String stringifyToken(Token<DelegationTokenIdentifier> token)
                             throws IOException

A test method for printing out tokens

抛出:

IOException

getDelegationToken

public Token<DelegationTokenIdentifier> getDelegationToken(Text renewer)
                                                    throws IOException

抛出:

IOException

renewDelegationToken

public long renewDelegationToken(Token<DelegationTokenIdentifier> token)
                          throws SecretManager.InvalidToken,
                                 IOException

已过时。 Use Token.renew instead.

Renew a delegation token

参数:

token - the token to renew

返回:

the new expiration time

抛出:

SecretManager.InvalidToken

IOException

cancelDelegationToken

public void cancelDelegationToken(Token<DelegationTokenIdentifier> token)
                           throws SecretManager.InvalidToken,
                                  IOException

已过时。 Use Token.cancel instead.

Cancel a delegation token

参数:

token - the token to cancel

抛出:

SecretManager.InvalidToken

IOException

reportBadBlocks

public void reportBadBlocks(LocatedBlock[] blocks)
                     throws IOException

Report corrupt blocks that were discovered by the client.

抛出:

IOException

getDefaultReplication

public short getDefaultReplication()

getHints

@Deprecated
public String[][] getHints(String src,
                             long start,
                             long length)
                    throws IOException

已过时。 Use getBlockLocations instead Get hints about the location of the indicated block(s). getHints() returns a list of hostnames that store data for a specific file region. It returns a set of hostnames for every block within the indicated region. This function is very useful when writing code that considers data-placement when performing operations. For example, the MapReduce system tries to schedule tasks on the same machines as the data-block the task processes.

抛出:

IOException

getBlockLocations

public BlockLocation[] getBlockLocations(String src,
                                long start,
                                long length)
                                  throws IOException

Get block location info about file getBlockLocations() returns a list of hostnames that store data for a specific file region. It returns a set of hostnames for every block within the indicated region. This function is very useful when writing code that considers data-placement when performing operations. For example, the MapReduce system tries to schedule tasks on the same machines as the data-block the task processes.

抛出:

IOException

open

public DFSClient.DFSInputStream open(String src)
                              throws IOException

抛出:

IOException

open

public DFSClient.DFSInputStream open(String src,
                            int buffersize,
                            boolean verifyChecksum,
                            FileSystem.Statistics stats)
                              throws IOException

Create an input stream that obtains a nodelist from the namenode, and then reads from all the right places. Creates inner subclass of InputStream that does the right out-of-band work.

抛出:

IOException

create

public OutputStream create(String src,
                  boolean overwrite)
                    throws IOException

Create a new dfs file and return an output stream for writing into it.

参数:

src - stream name

overwrite - do not check for file existence if true

返回:

output stream

抛出:

IOException

create

public OutputStream create(String src,
                  boolean overwrite,
                  Progressable progress)
                    throws IOException

Create a new dfs file and return an output stream for writing into it with write-progress reporting.

参数:

src - stream name

overwrite - do not check for file existence if true

返回:

output stream

抛出:

IOException

create

public OutputStream create(String src,
                  boolean overwrite,
                  short replication,
                  long blockSize)
                    throws IOException

Create a new dfs file with the specified block replication and return an output stream for writing into the file.

参数:

src - stream name

overwrite - do not check for file existence if true

replication - block replication

返回:

output stream

抛出:

IOException

create

public OutputStream create(String src,
                  boolean overwrite,
                  short replication,
                  long blockSize,
                  Progressable progress)
                    throws IOException

Create a new dfs file with the specified block replication with write-progress reporting and return an output stream for writing into the file.

参数:

src - stream name

overwrite - do not check for file existence if true

replication - block replication

返回:

output stream

抛出:

IOException

create

public OutputStream create(String src,
                  boolean overwrite,
                  short replication,
                  long blockSize,
                  Progressable progress,
                  int buffersize)
                    throws IOException

Call create(String,FsPermission,boolean,short,long,Progressable,int) with default permission.

抛出:

IOException

另请参阅:

FsPermission.getDefault()

create

public OutputStream create(String src,
                  FsPermission permission,
                  boolean overwrite,
                  short replication,
                  long blockSize,
                  Progressable progress,
                  int buffersize)
                    throws IOException

Call create(String,FsPermission,boolean,boolean,short,long,Progressable,int) with createParent set to true.

抛出:

IOException

create

public OutputStream create(String src,
                  FsPermission permission,
                  boolean overwrite,
                  boolean createParent,
                  short replication,
                  long blockSize,
                  Progressable progress,
                  int buffersize)
                    throws IOException

Create a new dfs file with the specified block replication with write-progress reporting and return an output stream for writing into the file.

参数:

src - stream name

permission - The permission of the directory being created. If permission == null, use FsPermission.getDefault().

overwrite - do not check for file existence if true

createParent - create missing parent directory if true

replication - block replication

返回:

output stream

抛出:

IOException

另请参阅:

ClientProtocol.create(String, FsPermission, String, boolean, short, long)

isFileClosed

public boolean isFileClosed(String src)
                     throws IOException

Close status of a file

返回:

true if file is already closed

抛出:

IOException

append

public FSDataOutputStream append(String src,
                        int buffersize,
                        Progressable progress,
                        FileSystem.Statistics statistics)
                          throws IOException

Append to an existing HDFS file.

参数:

src - file name

buffersize - buffer size

progress - for reporting write-progress; null is acceptable.

statistics - file system statistics; null is acceptable.

返回:

an output stream for writing into the file

抛出:

IOException

另请参阅:

ClientProtocol.append(String, String)

setReplication

public boolean setReplication(String src,
                     short replication)
                       throws IOException

Set replication for an existing file.

参数:

replication -

返回:

true is successful or false if file does not exist

抛出:

IOException

另请参阅:

ClientProtocol.setReplication(String, short)

concat

public void concat(String trg,
          String[] srcs)
            throws IOException

Move blocks from src to trg and delete src See ClientProtocol.concat(String, String []).

抛出:

IOException

rename

public boolean rename(String src,
             String dst)
               throws IOException

Rename file or directory. See ClientProtocol.rename(String, String).

抛出:

IOException

delete

@Deprecated
public boolean delete(String src)
               throws IOException

已过时。

Delete file or directory. See ClientProtocol.delete(String).

抛出:

IOException

delete

public boolean delete(String src,
             boolean recursive)
               throws IOException

delete file or directory. delete contents of the directory if non empty and recursive set to true

抛出:

IOException

exists

public boolean exists(String src)
               throws IOException

Implemented using getFileInfo(src)

抛出:

IOException

isDirectory

@Deprecated
public boolean isDirectory(String src)
                    throws IOException

已过时。 Use getHdfsFileStatus() instead

抛出:

IOException

listPaths

public DirectoryListing listPaths(String src,
                         byte[] startAfter)
                           throws IOException

Get a partial listing of the indicated directory Recommend to use HdfsFileStatus.EMPTY_NAME as startAfter if the application wants to fetch a listing starting from the first entry in the directory

参数:

src - the directory name

startAfter - the name to start listing after

返回:

a partial listing starting after startAfter

抛出:

IOException

getFileInfo

public HdfsFileStatus getFileInfo(String src)
                           throws IOException

抛出:

IOException

getFileChecksum

public MD5MD5CRC32FileChecksum getFileChecksum(String src)
                                        throws IOException

Get the checksum of a file.

参数:

src - The file path

返回:

The checksum

抛出:

IOException

另请参阅:

DistributedFileSystem.getFileChecksum(Path)

getFileChecksum

public static MD5MD5CRC32FileChecksum getFileChecksum(String src,
                                      ClientProtocol namenode,
                                      SocketFactory socketFactory,
                                      int socketTimeout)
                                               throws IOException

Get the checksum of a file.

参数:

src - The file path

返回:

The checksum

抛出:

IOException

setPermission

public void setPermission(String src,
                 FsPermission permission)
                   throws IOException

Set permissions to a file or directory.

参数:

src - path name.

permission -

抛出:

FileNotFoundException - is file does not exist.

IOException

setOwner

public void setOwner(String src,
            String username,
            String groupname)
              throws IOException

Set file or directory owner.

参数:

src - path name.

username - user id.

groupname - user group.

抛出:

FileNotFoundException - is file does not exist.

IOException

getDiskStatus

public DistributedFileSystem.DiskStatus getDiskStatus()
                                               throws IOException

抛出:

IOException

totalRawCapacity

public long totalRawCapacity()
                      throws IOException

抛出:

IOException

totalRawUsed

public long totalRawUsed()
                  throws IOException

抛出:

IOException

getMissingBlocksCount

public long getMissingBlocksCount()
                           throws IOException

Returns count of blocks with no good replicas left. Normally should be zero.

抛出:

IOException

getUnderReplicatedBlocksCount

public long getUnderReplicatedBlocksCount()
                                   throws IOException

Returns count of blocks with one of more replica missing.

抛出:

IOException

getCorruptBlocksCount

public long getCorruptBlocksCount()
                           throws IOException

Returns count of blocks with at least one replica marked corrupt.

抛出:

IOException

datanodeReport

public DatanodeInfo[] datanodeReport(FSConstants.DatanodeReportType type)
                              throws IOException

抛出:

IOException

setSafeMode

public boolean setSafeMode(FSConstants.SafeModeAction action)
                    throws IOException

Enter, leave or get safe mode. See ClientProtocol.setSafeMode(FSConstants.SafeModeAction) for more details.

抛出:

IOException

另请参阅:

ClientProtocol.setSafeMode(FSConstants.SafeModeAction)

refreshNodes

public void refreshNodes()
                  throws IOException

Refresh the hosts and exclude files. (Rereads them.) See ClientProtocol.refreshNodes() for more details.

抛出:

IOException

另请参阅:

ClientProtocol.refreshNodes()

metaSave

public void metaSave(String pathname)
              throws IOException

Dumps DFS data structures into specified file. See ClientProtocol.metaSave(String) for more details.

抛出:

IOException

另请参阅:

ClientProtocol.metaSave(String)

setBalancerBandwidth

public void setBalancerBandwidth(long bandwidth)
                          throws IOException

Requests the namenode to tell all datanodes to use a new, non-persistent bandwidth value for dfs.balance.bandwidthPerSec. See ClientProtocol.setBalancerBandwidth(long) for more details.

抛出:

IOException

另请参阅:

ClientProtocol.setBalancerBandwidth(long)

finalizeUpgrade

public void finalizeUpgrade()
                     throws IOException

抛出:

IOException

另请参阅:

ClientProtocol.finalizeUpgrade()

distributedUpgradeProgress

public UpgradeStatusReport distributedUpgradeProgress(FSConstants.UpgradeAction action)
                                               throws IOException

抛出:

IOException

另请参阅:

ClientProtocol.distributedUpgradeProgress(FSConstants.UpgradeAction)

mkdirs

public boolean mkdirs(String src)
               throws IOException

抛出:

IOException

mkdirs

public boolean mkdirs(String src,
             FsPermission permission)
               throws IOException

Create a directory (or hierarchy of directories) with the given name and permission.

参数:

src - The path of the directory being created

permission - The permission of the directory being created. If permission == null, use FsPermission.getDefault().

返回:

True if the operation success.

抛出:

IOException

另请参阅:

ClientProtocol.mkdirs(String, FsPermission)

setTimes

public void setTimes(String src,
            long mtime,
            long atime)
              throws IOException

set the modification and access time of a file

抛出:

FileNotFoundException - if the path is not a file

IOException

toString

public String toString()

覆盖:

toString 在类中 Object