org.apache.hadoop.hbase.tool

Class BulkLoadHFilesTool

java.lang.Object

org.apache.hadoop.conf.Configured

org.apache.hadoop.hbase.tool.BulkLoadHFilesTool

All Implemented Interfaces:

org.apache.hadoop.conf.Configurable, BulkLoadHFiles, org.apache.hadoop.util.Tool

@InterfaceAudience.LimitedPrivate(value="Tools")
public class BulkLoadHFilesTool
extends org.apache.hadoop.conf.Configured
implements BulkLoadHFiles, org.apache.hadoop.util.Tool

The implementation for BulkLoadHFiles, and also can be executed from command line as a tool.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.hadoop.hbase.tool.BulkLoadHFiles

BulkLoadHFiles.LoadQueueItem

Field Summary

Fields

Modifier and Type	Field and Description
static String	BULK_LOAD_HFILES_BY_FAMILY HBASE-24221 Support bulkLoadHFile by family to avoid long time waiting of bulkLoadHFile because of compacting at server side
static String	NAME

Fields inherited from interface org.apache.hadoop.hbase.tool.BulkLoadHFiles

ALWAYS_COPY_FILES, ASSIGN_SEQ_IDS, CREATE_TABLE_CONF_KEY, IGNORE_UNMATCHED_CF_CONF_KEY, MAX_FILES_PER_REGION_PER_FAMILY, RETRY_ON_IO_EXCEPTION

Constructor Summary

Constructors

Constructor and Description
BulkLoadHFilesTool(org.apache.hadoop.conf.Configuration conf)

Method Summary

Modifier and Type	Method and Description
Map<BulkLoadHFiles.LoadQueueItem,ByteBuffer>	bulkLoad(TableName tableName, Map<byte[],List<org.apache.hadoop.fs.Path>> family2Files) Perform a bulk load of the given directory into the given pre-existing table.
Map<BulkLoadHFiles.LoadQueueItem,ByteBuffer>	bulkLoad(TableName tableName, org.apache.hadoop.fs.Path dir) Perform a bulk load of the given directory into the given pre-existing table.
protected void	bulkLoadPhase(AsyncClusterConnection conn, TableName tableName, Deque<BulkLoadHFiles.LoadQueueItem> queue, org.apache.hbase.thirdparty.com.google.common.collect.Multimap<ByteBuffer,BulkLoadHFiles.LoadQueueItem> regionGroups, boolean copyFiles, Map<BulkLoadHFiles.LoadQueueItem,ByteBuffer> item2RegionMap) This takes the LQI's grouped by likely regions and attempts to bulk load them.
void	disableReplication() Disables replication for all bulkloads done via this instance, when bulkload replication is configured.
protected Pair<List<BulkLoadHFiles.LoadQueueItem>,String>	groupOrSplit(AsyncClusterConnection conn, TableName tableName, org.apache.hbase.thirdparty.com.google.common.collect.Multimap<ByteBuffer,BulkLoadHFiles.LoadQueueItem> regionGroups, BulkLoadHFiles.LoadQueueItem item, List<Pair<byte[],byte[]>> startEndKeys) Attempt to assign the given load queue item into its target region group.
static byte[][]	inferBoundaries(SortedMap<byte[],Integer> bdryMap) Infers region boundaries for a new table.
void	initialize()
boolean	isReplicationDisabled()
void	loadHFileQueue(AsyncClusterConnection conn, TableName tableName, Deque<BulkLoadHFiles.LoadQueueItem> queue, boolean copyFiles) Used by the replication sink to load the hfiles from the source cluster.
static void	main(String[] args)
static void	prepareHFileQueue(AsyncClusterConnection conn, TableName tableName, Map<byte[],List<org.apache.hadoop.fs.Path>> map, Deque<BulkLoadHFiles.LoadQueueItem> queue, boolean silence) Prepare a collection of LoadQueueItem from list of source hfiles contained in the passed directory and validates whether the prepared queue has all the valid table column families in it.
static void	prepareHFileQueue(org.apache.hadoop.conf.Configuration conf, AsyncClusterConnection conn, TableName tableName, org.apache.hadoop.fs.Path hfilesDir, Deque<BulkLoadHFiles.LoadQueueItem> queue, boolean validateHFile, boolean silence) Prepare a collection of LoadQueueItem from list of source hfiles contained in the passed directory and validates whether the prepared queue has all the valid table column families in it.
int	run(String[] args)
void	setBulkToken(String bulkToken)
void	setClusterIds(List<String> clusterIds)
protected CompletableFuture<Collection<BulkLoadHFiles.LoadQueueItem>>	tryAtomicRegionLoad(AsyncClusterConnection conn, TableName tableName, boolean copyFiles, byte[] first, Collection<BulkLoadHFiles.LoadQueueItem> lqis) Attempts to do an atomic load of many hfiles into a region.

Methods inherited from class org.apache.hadoop.conf.Configured

getConf, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hadoop.hbase.tool.BulkLoadHFiles

create

Methods inherited from interface org.apache.hadoop.conf.Configurable

getConf, setConf

Field Detail

NAME

public static final String NAME

See Also:

Constant Field Values

BULK_LOAD_HFILES_BY_FAMILY

public static final String BULK_LOAD_HFILES_BY_FAMILY

HBASE-24221 Support bulkLoadHFile by family to avoid long time waiting of bulkLoadHFile because of compacting at server side

See Also:

Constant Field Values

Constructor Detail

BulkLoadHFilesTool

public BulkLoadHFilesTool(org.apache.hadoop.conf.Configuration conf)

Method Detail

initialize

public void initialize()

prepareHFileQueue

public static void prepareHFileQueue(AsyncClusterConnection conn,
                                     TableName tableName,
                                     Map<byte[],List<org.apache.hadoop.fs.Path>> map,
                                     Deque<BulkLoadHFiles.LoadQueueItem> queue,
                                     boolean silence)
                              throws IOException

Prepare a collection of LoadQueueItem from list of source hfiles contained in the passed directory and validates whether the prepared queue has all the valid table column families in it.

Parameters:

map - map of family to List of hfiles

tableName - table to which hfiles should be loaded

queue - queue which needs to be loaded into the table

silence - true to ignore unmatched column families

Throws:

IOException - If any I/O or network error occurred

prepareHFileQueue

public static void prepareHFileQueue(org.apache.hadoop.conf.Configuration conf,
                                     AsyncClusterConnection conn,
                                     TableName tableName,
                                     org.apache.hadoop.fs.Path hfilesDir,
                                     Deque<BulkLoadHFiles.LoadQueueItem> queue,
                                     boolean validateHFile,
                                     boolean silence)
                              throws IOException

Prepare a collection of LoadQueueItem from list of source hfiles contained in the passed directory and validates whether the prepared queue has all the valid table column families in it.

Parameters:

hfilesDir - directory containing list of hfiles to be loaded into the table

queue - queue which needs to be loaded into the table

validateHFile - if true hfiles will be validated for its format

silence - true to ignore unmatched column families

Throws:

IOException - If any I/O or network error occurred

loadHFileQueue

public void loadHFileQueue(AsyncClusterConnection conn,
                           TableName tableName,
                           Deque<BulkLoadHFiles.LoadQueueItem> queue,
                           boolean copyFiles)
                    throws IOException

Used by the replication sink to load the hfiles from the source cluster. It does the following,

1. groupOrSplitPhase(AsyncClusterConnection, TableName, ExecutorService, Deque, List)
2. bulkLoadPhase(AsyncClusterConnection, TableName, Deque, Multimap, boolean, Map)

Parameters:

conn - Connection to use

tableName - Table to which these hfiles should be loaded to

queue - LoadQueueItem has hfiles yet to be loaded

Throws:

IOException

tryAtomicRegionLoad

@InterfaceAudience.Private
protected CompletableFuture<Collection<BulkLoadHFiles.LoadQueueItem>> tryAtomicRegionLoad(AsyncClusterConnection conn,
                                                                                                                     TableName tableName,
                                                                                                                     boolean copyFiles,
                                                                                                                     byte[] first,
                                                                                                                     Collection<BulkLoadHFiles.LoadQueueItem> lqis)

Attempts to do an atomic load of many hfiles into a region. If it fails, it returns a list of hfiles that need to be retried. If it is successful it will return an empty list. NOTE: To maintain row atomicity guarantees, region server side should succeed atomically and fails atomically.

Parameters:

conn - Connection to use

tableName - Table to which these hfiles should be loaded to

copyFiles - whether replicate to peer cluster while bulkloading

first - the start key of region

lqis - hfiles should be loaded

Returns:

empty list if success, list of items to retry on recoverable failure

bulkLoadPhase

@InterfaceAudience.Private
protected void bulkLoadPhase(AsyncClusterConnection conn,
                                                        TableName tableName,
                                                        Deque<BulkLoadHFiles.LoadQueueItem> queue,
                                                        org.apache.hbase.thirdparty.com.google.common.collect.Multimap<ByteBuffer,BulkLoadHFiles.LoadQueueItem> regionGroups,
                                                        boolean copyFiles,
                                                        Map<BulkLoadHFiles.LoadQueueItem,ByteBuffer> item2RegionMap)
                                                 throws IOException

This takes the LQI's grouped by likely regions and attempts to bulk load them. Any failures are re-queued for another pass with the groupOrSplitPhase.

protected for testing.

Throws:

IOException

groupOrSplit

@InterfaceAudience.Private
protected Pair<List<BulkLoadHFiles.LoadQueueItem>,String> groupOrSplit(AsyncClusterConnection conn,
                                                                                                  TableName tableName,
                                                                                                  org.apache.hbase.thirdparty.com.google.common.collect.Multimap<ByteBuffer,BulkLoadHFiles.LoadQueueItem> regionGroups,
                                                                                                  BulkLoadHFiles.LoadQueueItem item,
                                                                                                  List<Pair<byte[],byte[]>> startEndKeys)
                                                                                           throws IOException

Attempt to assign the given load queue item into its target region group. If the hfile boundary no longer fits into a region, physically splits the hfile such that the new bottom half will fit and returns the list of LQI's corresponding to the resultant hfiles.

protected for testing

Throws:

IOException - if an IO failure is encountered

inferBoundaries

public static byte[][] inferBoundaries(SortedMap<byte[],Integer> bdryMap)

Infers region boundaries for a new table.

Parameter:
bdryMap is a map between keys to an integer belonging to {+1, -1}

• If a key is a start key of a file, then it maps to +1
• If a key is an end key of a file, then it maps to -1

Algo:

1. Poll on the keys in order:
   1. Keep adding the mapped values to these keys (runningSum)
   2. Each time runningSum reaches 0, add the start Key from when the runningSum had started to a boundary list.
2. Return the boundary list.

bulkLoad

public Map<BulkLoadHFiles.LoadQueueItem,ByteBuffer> bulkLoad(TableName tableName,
                                                             Map<byte[],List<org.apache.hadoop.fs.Path>> family2Files)
                                                      throws IOException

Description copied from interface: BulkLoadHFiles

Perform a bulk load of the given directory into the given pre-existing table.

Specified by:

bulkLoad in interface BulkLoadHFiles

Parameters:

tableName - the table to load into

family2Files - map of family to List of hfiles

Throws:

TableNotFoundException - if table does not yet exist

IOException

bulkLoad

public Map<BulkLoadHFiles.LoadQueueItem,ByteBuffer> bulkLoad(TableName tableName,
                                                             org.apache.hadoop.fs.Path dir)
                                                      throws IOException

Description copied from interface: BulkLoadHFiles

Perform a bulk load of the given directory into the given pre-existing table.

Specified by:

bulkLoad in interface BulkLoadHFiles

Parameters:

tableName - the table to load into

dir - the directory that was provided as the output path of a job using HFileOutputFormat

Throws:

TableNotFoundException - if table does not yet exist

IOException

setBulkToken

public void setBulkToken(String bulkToken)

setClusterIds

public void setClusterIds(List<String> clusterIds)

run

public int run(String[] args)
        throws Exception

Specified by:

run in interface org.apache.hadoop.util.Tool

Throws:

Exception

main

public static void main(String[] args)
                 throws Exception

Throws:

Exception

disableReplication

public void disableReplication()

Description copied from interface: BulkLoadHFiles

Disables replication for all bulkloads done via this instance, when bulkload replication is configured.

Specified by:

disableReplication in interface BulkLoadHFiles

isReplicationDisabled

public boolean isReplicationDisabled()

Specified by:

isReplicationDisabled in interface BulkLoadHFiles

Returns:

true if replication has been disabled.