PREHOOK: query: CREATE TABLE bucket5_1(key string, value string) CLUSTERED BY (key) INTO 2 BUCKETS PREHOOK: type: CREATETABLE PREHOOK: Output: database:default PREHOOK: Output: default@bucket5_1 POSTHOOK: query: CREATE TABLE bucket5_1(key string, value string) CLUSTERED BY (key) INTO 2 BUCKETS POSTHOOK: type: CREATETABLE POSTHOOK: Output: database:default POSTHOOK: Output: default@bucket5_1 PREHOOK: query: explain extended insert overwrite table bucket5_1 select * from src cluster by key PREHOOK: type: QUERY POSTHOOK: query: explain extended insert overwrite table bucket5_1 select * from src cluster by key POSTHOOK: type: QUERY ABSTRACT SYNTAX TREE: TOK_QUERY TOK_FROM TOK_TABREF TOK_TABNAME src TOK_INSERT TOK_DESTINATION TOK_TAB TOK_TABNAME bucket5_1 TOK_SELECT TOK_SELEXPR TOK_ALLCOLREF TOK_CLUSTERBY TOK_TABLE_OR_COL key STAGE DEPENDENCIES: Stage-1 is a root stage Stage-0 depends on stages: Stage-1 Stage-2 depends on stages: Stage-0 STAGE PLANS: Stage: Stage-1 Map Reduce Map Operator Tree: TableScan alias: src Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE GatherStats: false Select Operator expressions: key (type: string), value (type: string) outputColumnNames: _col0, _col1 Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: string) sort order: + Map-reduce partition columns: _col0 (type: string) Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE tag: -1 value expressions: _col1 (type: string) auto parallelism: false Path -> Alias: #### A masked pattern was here #### Path -> Partition: #### A masked pattern was here #### Partition base file name: src input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat properties: COLUMN_STATS_ACCURATE true bucket_count -1 columns key,value columns.comments 'default','default' columns.types string:string #### A masked pattern was here #### name default.src numFiles 1 numRows 500 rawDataSize 5312 serialization.ddl struct src { string key, string value} serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe totalSize 5812 #### A masked pattern was here #### serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat properties: COLUMN_STATS_ACCURATE true bucket_count -1 columns key,value columns.comments 'default','default' columns.types string:string #### A masked pattern was here #### name default.src numFiles 1 numRows 500 rawDataSize 5312 serialization.ddl struct src { string key, string value} serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe totalSize 5812 #### A masked pattern was here #### serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe name: default.src name: default.src Truncated Path -> Alias: /src [src] Needs Tagging: false Reduce Operator Tree: Select Operator expressions: KEY.reducesinkkey0 (type: string), VALUE._col0 (type: string) outputColumnNames: _col0, _col1 Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false GlobalTableId: 1 #### A masked pattern was here #### NumFilesPerFileSink: 2 Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE #### A masked pattern was here #### table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat properties: bucket_count 2 bucket_field_name key columns key,value columns.comments columns.types string:string #### A masked pattern was here #### name default.bucket5_1 serialization.ddl struct bucket5_1 { string key, string value} serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe #### A masked pattern was here #### serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe name: default.bucket5_1 TotalFiles: 2 GatherStats: true MultiFileSpray: true Stage: Stage-0 Move Operator tables: replace: true #### A masked pattern was here #### table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat properties: bucket_count 2 bucket_field_name key columns key,value columns.comments columns.types string:string #### A masked pattern was here #### name default.bucket5_1 serialization.ddl struct bucket5_1 { string key, string value} serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe #### A masked pattern was here #### serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe name: default.bucket5_1 Stage: Stage-2 Stats-Aggr Operator #### A masked pattern was here #### PREHOOK: query: insert overwrite table bucket5_1 select * from src cluster by key PREHOOK: type: QUERY PREHOOK: Input: default@src PREHOOK: Output: default@bucket5_1 POSTHOOK: query: insert overwrite table bucket5_1 select * from src cluster by key POSTHOOK: type: QUERY POSTHOOK: Input: default@src POSTHOOK: Output: default@bucket5_1 POSTHOOK: Lineage: bucket5_1.key SIMPLE [(src)src.FieldSchema(name:key, type:string, comment:default), ] POSTHOOK: Lineage: bucket5_1.value SIMPLE [(src)src.FieldSchema(name:value, type:string, comment:default), ] PREHOOK: query: select sum(hash(key)),sum(hash(value)) from bucket5_1 PREHOOK: type: QUERY PREHOOK: Input: default@bucket5_1 #### A masked pattern was here #### POSTHOOK: query: select sum(hash(key)),sum(hash(value)) from bucket5_1 POSTHOOK: type: QUERY POSTHOOK: Input: default@bucket5_1 #### A masked pattern was here #### 21025334 36210398070 PREHOOK: query: select sum(hash(key)),sum(hash(value)) from src PREHOOK: type: QUERY PREHOOK: Input: default@src #### A masked pattern was here #### POSTHOOK: query: select sum(hash(key)),sum(hash(value)) from src POSTHOOK: type: QUERY POSTHOOK: Input: default@src #### A masked pattern was here #### 21025334 36210398070 PREHOOK: query: create table complex_tbl_1(aid string, bid string, t int, ctime string, etime bigint, l string, et string) partitioned by (ds string) PREHOOK: type: CREATETABLE PREHOOK: Output: database:default PREHOOK: Output: default@complex_tbl_1 POSTHOOK: query: create table complex_tbl_1(aid string, bid string, t int, ctime string, etime bigint, l string, et string) partitioned by (ds string) POSTHOOK: type: CREATETABLE POSTHOOK: Output: database:default POSTHOOK: Output: default@complex_tbl_1 PREHOOK: query: create table complex_tbl_2(aet string, aes string) partitioned by (ds string) PREHOOK: type: CREATETABLE PREHOOK: Output: database:default PREHOOK: Output: default@complex_tbl_2 POSTHOOK: query: create table complex_tbl_2(aet string, aes string) partitioned by (ds string) POSTHOOK: type: CREATETABLE POSTHOOK: Output: database:default POSTHOOK: Output: default@complex_tbl_2 PREHOOK: query: explain extended insert overwrite table complex_tbl_1 partition (ds='2010-03-29') select s2.* from ( select TRANSFORM (aid,bid,t,ctime,etime,l,et) USING 'cat' AS (aid string, bid string, t int, ctime string, etime bigint, l string, et string) from ( select transform(aet,aes) using 'cat' as (aid string, bid string, t int, ctime string, etime bigint, l string, et string) from complex_tbl_2 where ds ='2010-03-29' cluster by bid )s )s2 PREHOOK: type: QUERY POSTHOOK: query: explain extended insert overwrite table complex_tbl_1 partition (ds='2010-03-29') select s2.* from ( select TRANSFORM (aid,bid,t,ctime,etime,l,et) USING 'cat' AS (aid string, bid string, t int, ctime string, etime bigint, l string, et string) from ( select transform(aet,aes) using 'cat' as (aid string, bid string, t int, ctime string, etime bigint, l string, et string) from complex_tbl_2 where ds ='2010-03-29' cluster by bid )s )s2 POSTHOOK: type: QUERY ABSTRACT SYNTAX TREE: TOK_QUERY TOK_FROM TOK_SUBQUERY TOK_QUERY TOK_FROM TOK_SUBQUERY TOK_QUERY TOK_FROM TOK_TABREF TOK_TABNAME complex_tbl_2 TOK_INSERT TOK_DESTINATION TOK_DIR TOK_TMP_FILE TOK_SELECT TOK_SELEXPR TOK_TRANSFORM TOK_EXPLIST TOK_TABLE_OR_COL aet TOK_TABLE_OR_COL aes TOK_SERDE TOK_RECORDWRITER 'cat' TOK_SERDE TOK_RECORDREADER TOK_TABCOLLIST TOK_TABCOL aid TOK_STRING TOK_TABCOL bid TOK_STRING TOK_TABCOL t TOK_INT TOK_TABCOL ctime TOK_STRING TOK_TABCOL etime TOK_BIGINT TOK_TABCOL l TOK_STRING TOK_TABCOL et TOK_STRING TOK_WHERE = TOK_TABLE_OR_COL ds '2010-03-29' TOK_CLUSTERBY TOK_TABLE_OR_COL bid s TOK_INSERT TOK_DESTINATION TOK_DIR TOK_TMP_FILE TOK_SELECT TOK_SELEXPR TOK_TRANSFORM TOK_EXPLIST TOK_TABLE_OR_COL aid TOK_TABLE_OR_COL bid TOK_TABLE_OR_COL t TOK_TABLE_OR_COL ctime TOK_TABLE_OR_COL etime TOK_TABLE_OR_COL l TOK_TABLE_OR_COL et TOK_SERDE TOK_RECORDWRITER 'cat' TOK_SERDE TOK_RECORDREADER TOK_TABCOLLIST TOK_TABCOL aid TOK_STRING TOK_TABCOL bid TOK_STRING TOK_TABCOL t TOK_INT TOK_TABCOL ctime TOK_STRING TOK_TABCOL etime TOK_BIGINT TOK_TABCOL l TOK_STRING TOK_TABCOL et TOK_STRING s2 TOK_INSERT TOK_DESTINATION TOK_TAB TOK_TABNAME complex_tbl_1 TOK_PARTSPEC TOK_PARTVAL ds '2010-03-29' TOK_SELECT TOK_SELEXPR TOK_ALLCOLREF TOK_TABNAME s2 STAGE DEPENDENCIES: Stage-1 is a root stage Stage-0 depends on stages: Stage-1 Stage-2 depends on stages: Stage-0 STAGE PLANS: Stage: Stage-1 Map Reduce Map Operator Tree: TableScan alias: complex_tbl_2 Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE GatherStats: false Filter Operator isSamplingPred: false predicate: (ds = '2010-03-29') (type: boolean) Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE Select Operator expressions: aet (type: string), aes (type: string) outputColumnNames: _col0, _col1 Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE Transform Operator command: cat output info: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat properties: columns _col0,_col1,_col2,_col3,_col4,_col5,_col6 columns.types string,string,int,string,bigint,string,string field.delim 9 serialization.format 9 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE Reduce Output Operator key expressions: _col1 (type: string) sort order: + Map-reduce partition columns: _col1 (type: string) Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE tag: -1 value expressions: _col0 (type: string), _col1 (type: string), _col2 (type: int), _col3 (type: string), _col4 (type: bigint), _col5 (type: string), _col6 (type: string) auto parallelism: false Needs Tagging: false Reduce Operator Tree: Select Operator expressions: VALUE._col0 (type: string), VALUE._col1 (type: string), VALUE._col2 (type: int), VALUE._col3 (type: string), VALUE._col4 (type: bigint), VALUE._col5 (type: string), VALUE._col6 (type: string) outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6 Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE Transform Operator command: cat output info: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat properties: columns _col0,_col1,_col2,_col3,_col4,_col5,_col6 columns.types string,string,int,string,bigint,string,string field.delim 9 serialization.format 9 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE File Output Operator compressed: false GlobalTableId: 1 #### A masked pattern was here #### NumFilesPerFileSink: 1 Static Partition Specification: ds=2010-03-29/ Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE #### A masked pattern was here #### table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat properties: bucket_count -1 columns aid,bid,t,ctime,etime,l,et columns.comments columns.types string:string:int:string:bigint:string:string #### A masked pattern was here #### name default.complex_tbl_1 partition_columns ds partition_columns.types string serialization.ddl struct complex_tbl_1 { string aid, string bid, i32 t, string ctime, i64 etime, string l, string et} serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe #### A masked pattern was here #### serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe name: default.complex_tbl_1 TotalFiles: 1 GatherStats: true MultiFileSpray: false Stage: Stage-0 Move Operator tables: partition: ds 2010-03-29 replace: true #### A masked pattern was here #### table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat properties: bucket_count -1 columns aid,bid,t,ctime,etime,l,et columns.comments columns.types string:string:int:string:bigint:string:string #### A masked pattern was here #### name default.complex_tbl_1 partition_columns ds partition_columns.types string serialization.ddl struct complex_tbl_1 { string aid, string bid, i32 t, string ctime, i64 etime, string l, string et} serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe #### A masked pattern was here #### serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe name: default.complex_tbl_1 Stage: Stage-2 Stats-Aggr Operator #### A masked pattern was here ####