org.apache.hadoop.hive.ql.exec.persistence

Class BytesBytesMultiHashMap

java.lang.Object

org.apache.hadoop.hive.ql.exec.persistence.BytesBytesMultiHashMap

public final class BytesBytesMultiHashMap
extends Object

HashMap that maps byte arrays to byte arrays with limited functionality necessary for MapJoin hash tables, with small memory overhead. Supports multiple values for single key. Values can be added for key (but cannot be removed); values can be gotten for the key. Some things (like entrySet) are easy to add; some e.g. deletion are pretty hard to do well. Additionally, for each key it contains a magic "state byte" which is not part of the key and can be updated on every put for that key. That is really silly, we use it to store aliasFilter. Magic byte could be removed for generality. Initially inspired by HPPC LongLongOpenHashMap; however, the code is almost completely reworked and there's very little in common left save for quadratic probing (and that with some changes).

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static interface	BytesBytesMultiHashMap.KvSource The source of keys and values to put into hashtable; avoids byte copying.
static class	BytesBytesMultiHashMap.Result The result of looking up a key in the multi-hash map.

Field Summary

Fields

Modifier and Type	Field and Description
static org.apache.commons.logging.Log	LOG

Constructor Summary

Constructors

Constructor and Description
BytesBytesMultiHashMap(int initialCapacity, float loadFactor, int wbSize, long memUsage)

Method Summary

Methods

Modifier and Type	Method and Description
void	clear()
void	debugDumpMetrics()
void	debugDumpTable() Writes the debug dump of the table into logs.
void	expandAndRehashToTarget(int estimateNewRowCount)
int	getNumValues() Number of values in the hashmap This is equal to or bigger than number of keys, since some values may share the same key
byte	getValueResult(byte[] key, int offset, int length, BytesBytesMultiHashMap.Result hashMapResult) Finds a key.
long	memorySize() Number of bytes used by the hashmap There are two main components that take most memory: writeBuffers and refs Others include instance fields: 100
void	populateValue(WriteBuffers.ByteSegmentRef valueRef) Take the segment reference from #getValueRefs(byte[], int, List) result and makes it self-contained - adds byte array where the value is stored, and updates the offset from "global" write buffers offset to offset within that array.
void	put(BytesBytesMultiHashMap.KvSource kv, int keyHashCode)
void	seal()
int	size() Number of keys in the hashmap

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

public static final org.apache.commons.logging.Log LOG

Constructor Detail

BytesBytesMultiHashMap

public BytesBytesMultiHashMap(int initialCapacity,
                      float loadFactor,
                      int wbSize,
                      long memUsage)

Method Detail

put

public void put(BytesBytesMultiHashMap.KvSource kv,
       int keyHashCode)
         throws SerDeException

Throws:

SerDeException

getValueResult

public byte getValueResult(byte[] key,
                  int offset,
                  int length,
                  BytesBytesMultiHashMap.Result hashMapResult)

Finds a key. Values can be read with the supplied result object. Important Note: The caller is expected to pre-allocate the hashMapResult and not share it among other threads.

Parameters:

key - Key buffer.

offset - the offset to the key in the buffer

hashMapResult - The object to fill in that can read the values.

Returns:

The state byte.

populateValue

public void populateValue(WriteBuffers.ByteSegmentRef valueRef)

Take the segment reference from #getValueRefs(byte[], int, List) result and makes it self-contained - adds byte array where the value is stored, and updates the offset from "global" write buffers offset to offset within that array.

size

public int size()

Number of keys in the hashmap

Returns:

number of keys

getNumValues

public int getNumValues()

Number of values in the hashmap This is equal to or bigger than number of keys, since some values may share the same key

Returns:

number of values

memorySize

public long memorySize()

Number of bytes used by the hashmap There are two main components that take most memory: writeBuffers and refs Others include instance fields: 100

Returns:

number of bytes

seal

public void seal()

clear

public void clear()

expandAndRehashToTarget

public void expandAndRehashToTarget(int estimateNewRowCount)

debugDumpTable

public void debugDumpTable()

Writes the debug dump of the table into logs. Not thread-safe.

debugDumpMetrics

public void debugDumpMetrics()