PREHOOK: query: drop table bucket5_1 PREHOOK: type: DROPTABLE POSTHOOK: query: drop table bucket5_1 POSTHOOK: type: DROPTABLE PREHOOK: query: CREATE TABLE bucket5_1(key string, value string) CLUSTERED BY (key) INTO 2 BUCKETS PREHOOK: type: CREATETABLE POSTHOOK: query: CREATE TABLE bucket5_1(key string, value string) CLUSTERED BY (key) INTO 2 BUCKETS POSTHOOK: type: CREATETABLE POSTHOOK: Output: default@bucket5_1 PREHOOK: query: explain extended insert overwrite table bucket5_1 select * from src cluster by key PREHOOK: type: QUERY POSTHOOK: query: explain extended insert overwrite table bucket5_1 select * from src cluster by key POSTHOOK: type: QUERY ABSTRACT SYNTAX TREE: (TOK_QUERY (TOK_FROM (TOK_TABREF src)) (TOK_INSERT (TOK_DESTINATION (TOK_TAB bucket5_1)) (TOK_SELECT (TOK_SELEXPR TOK_ALLCOLREF)) (TOK_CLUSTERBY (TOK_TABLE_OR_COL key)))) STAGE DEPENDENCIES: Stage-1 is a root stage Stage-0 depends on stages: Stage-1 STAGE PLANS: Stage: Stage-1 Map Reduce Alias -> Map Operator Tree: src TableScan alias: src Select Operator expressions: expr: key type: string expr: value type: string outputColumnNames: _col0, _col1 Reduce Output Operator key expressions: expr: _col0 type: string sort order: + Map-reduce partition columns: expr: _col0 type: string tag: -1 value expressions: expr: _col0 type: string expr: _col1 type: string Needs Tagging: false Path -> Alias: file:/data/users/njain/hive_commit1/hive_commit1/build/ql/test/data/warehouse/src [src] Path -> Partition: file:/data/users/njain/hive_commit1/hive_commit1/build/ql/test/data/warehouse/src Partition base file name: src input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat properties: bucket_count -1 columns key,value columns.types string:string file.inputformat org.apache.hadoop.mapred.TextInputFormat file.outputformat org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat location file:/data/users/njain/hive_commit1/hive_commit1/build/ql/test/data/warehouse/src name src serialization.ddl struct src { string key, string value} serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe transient_lastDdlTime 1270584980 serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat properties: bucket_count -1 columns key,value columns.types string:string file.inputformat org.apache.hadoop.mapred.TextInputFormat file.outputformat org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat location file:/data/users/njain/hive_commit1/hive_commit1/build/ql/test/data/warehouse/src name src serialization.ddl struct src { string key, string value} serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe transient_lastDdlTime 1270584980 serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe name: src name: src Reduce Operator Tree: Extract Select Operator expressions: expr: _col0 type: string expr: _col1 type: string outputColumnNames: _col0, _col1 File Output Operator compressed: false GlobalTableId: 1 directory: file:/data/users/njain/hive_commit1/hive_commit1/build/ql/scratchdir/hive_2010-04-06_13-16-21_146_8698358277578653299/10000 NumFilesPerFileSink: 2 table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat properties: bucket_count 2 bucket_field_name key columns key,value columns.types string:string file.inputformat org.apache.hadoop.mapred.TextInputFormat file.outputformat org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat location file:/data/users/njain/hive_commit1/hive_commit1/build/ql/test/data/warehouse/bucket5_1 name bucket5_1 serialization.ddl struct bucket5_1 { string key, string value} serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe transient_lastDdlTime 1270584981 serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe name: bucket5_1 TotalFiles: 2 MultiFileSpray: true Stage: Stage-0 Move Operator tables: replace: true source: file:/data/users/njain/hive_commit1/hive_commit1/build/ql/scratchdir/hive_2010-04-06_13-16-21_146_8698358277578653299/10000 table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat properties: bucket_count 2 bucket_field_name key columns key,value columns.types string:string file.inputformat org.apache.hadoop.mapred.TextInputFormat file.outputformat org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat location file:/data/users/njain/hive_commit1/hive_commit1/build/ql/test/data/warehouse/bucket5_1 name bucket5_1 serialization.ddl struct bucket5_1 { string key, string value} serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe transient_lastDdlTime 1270584981 serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe name: bucket5_1 tmp directory: file:/data/users/njain/hive_commit1/hive_commit1/build/ql/scratchdir/hive_2010-04-06_13-16-21_146_8698358277578653299/10001 PREHOOK: query: insert overwrite table bucket5_1 select * from src cluster by key PREHOOK: type: QUERY PREHOOK: Input: default@src PREHOOK: Output: default@bucket5_1 POSTHOOK: query: insert overwrite table bucket5_1 select * from src cluster by key POSTHOOK: type: QUERY POSTHOOK: Input: default@src POSTHOOK: Output: default@bucket5_1 POSTHOOK: Lineage: bucket5_1.key SIMPLE [(src)src.FieldSchema(name:key, type:string, comment:default), ] POSTHOOK: Lineage: bucket5_1.value SIMPLE [(src)src.FieldSchema(name:value, type:string, comment:default), ] PREHOOK: query: select sum(hash(key)),sum(hash(value)) from bucket5_1 PREHOOK: type: QUERY PREHOOK: Input: default@bucket5_1 PREHOOK: Output: file:/data/users/njain/hive_commit1/hive_commit1/build/ql/scratchdir/hive_2010-04-06_13-16-24_929_6291705567058449432/10000 POSTHOOK: query: select sum(hash(key)),sum(hash(value)) from bucket5_1 POSTHOOK: type: QUERY POSTHOOK: Input: default@bucket5_1 POSTHOOK: Output: file:/data/users/njain/hive_commit1/hive_commit1/build/ql/scratchdir/hive_2010-04-06_13-16-24_929_6291705567058449432/10000 POSTHOOK: Lineage: bucket5_1.key SIMPLE [(src)src.FieldSchema(name:key, type:string, comment:default), ] POSTHOOK: Lineage: bucket5_1.value SIMPLE [(src)src.FieldSchema(name:value, type:string, comment:default), ] 21025334 36210398070 PREHOOK: query: select sum(hash(key)),sum(hash(value)) from src PREHOOK: type: QUERY PREHOOK: Input: default@src PREHOOK: Output: file:/data/users/njain/hive_commit1/hive_commit1/build/ql/scratchdir/hive_2010-04-06_13-16-28_771_1329782713419507109/10000 POSTHOOK: query: select sum(hash(key)),sum(hash(value)) from src POSTHOOK: type: QUERY POSTHOOK: Input: default@src POSTHOOK: Output: file:/data/users/njain/hive_commit1/hive_commit1/build/ql/scratchdir/hive_2010-04-06_13-16-28_771_1329782713419507109/10000 POSTHOOK: Lineage: bucket5_1.key SIMPLE [(src)src.FieldSchema(name:key, type:string, comment:default), ] POSTHOOK: Lineage: bucket5_1.value SIMPLE [(src)src.FieldSchema(name:value, type:string, comment:default), ] 21025334 36210398070 PREHOOK: query: drop table complex_tbl_1 PREHOOK: type: DROPTABLE POSTHOOK: query: drop table complex_tbl_1 POSTHOOK: type: DROPTABLE POSTHOOK: Lineage: bucket5_1.key SIMPLE [(src)src.FieldSchema(name:key, type:string, comment:default), ] POSTHOOK: Lineage: bucket5_1.value SIMPLE [(src)src.FieldSchema(name:value, type:string, comment:default), ] PREHOOK: query: create table complex_tbl_1(aid string, bid string, t int, ctime string, etime bigint, l string, et string) partitioned by (ds string) PREHOOK: type: CREATETABLE POSTHOOK: query: create table complex_tbl_1(aid string, bid string, t int, ctime string, etime bigint, l string, et string) partitioned by (ds string) POSTHOOK: type: CREATETABLE POSTHOOK: Output: default@complex_tbl_1 POSTHOOK: Lineage: bucket5_1.key SIMPLE [(src)src.FieldSchema(name:key, type:string, comment:default), ] POSTHOOK: Lineage: bucket5_1.value SIMPLE [(src)src.FieldSchema(name:value, type:string, comment:default), ] PREHOOK: query: drop table complex_tbl_2 PREHOOK: type: DROPTABLE POSTHOOK: query: drop table complex_tbl_2 POSTHOOK: type: DROPTABLE POSTHOOK: Lineage: bucket5_1.key SIMPLE [(src)src.FieldSchema(name:key, type:string, comment:default), ] POSTHOOK: Lineage: bucket5_1.value SIMPLE [(src)src.FieldSchema(name:value, type:string, comment:default), ] PREHOOK: query: create table complex_tbl_2(aet string, aes string) partitioned by (ds string) PREHOOK: type: CREATETABLE POSTHOOK: query: create table complex_tbl_2(aet string, aes string) partitioned by (ds string) POSTHOOK: type: CREATETABLE POSTHOOK: Output: default@complex_tbl_2 POSTHOOK: Lineage: bucket5_1.key SIMPLE [(src)src.FieldSchema(name:key, type:string, comment:default), ] POSTHOOK: Lineage: bucket5_1.value SIMPLE [(src)src.FieldSchema(name:value, type:string, comment:default), ] PREHOOK: query: explain extended insert overwrite table complex_tbl_1 partition (ds='2010-03-29') select s2.* from ( select TRANSFORM (aid,bid,t,ctime,etime,l,et) USING 'cat' AS (aid string, bid string, t int, ctime string, etime bigint, l string, et string) from ( select transform(aet,aes) using 'cat' as (aid string, bid string, t int, ctime string, etime bigint, l string, et string) from complex_tbl_2 where ds ='2010-03-29' cluster by bid )s )s2 PREHOOK: type: QUERY POSTHOOK: query: explain extended insert overwrite table complex_tbl_1 partition (ds='2010-03-29') select s2.* from ( select TRANSFORM (aid,bid,t,ctime,etime,l,et) USING 'cat' AS (aid string, bid string, t int, ctime string, etime bigint, l string, et string) from ( select transform(aet,aes) using 'cat' as (aid string, bid string, t int, ctime string, etime bigint, l string, et string) from complex_tbl_2 where ds ='2010-03-29' cluster by bid )s )s2 POSTHOOK: type: QUERY POSTHOOK: Lineage: bucket5_1.key SIMPLE [(src)src.FieldSchema(name:key, type:string, comment:default), ] POSTHOOK: Lineage: bucket5_1.value SIMPLE [(src)src.FieldSchema(name:value, type:string, comment:default), ] ABSTRACT SYNTAX TREE: (TOK_QUERY (TOK_FROM (TOK_SUBQUERY (TOK_QUERY (TOK_FROM (TOK_SUBQUERY (TOK_QUERY (TOK_FROM (TOK_TABREF complex_tbl_2)) (TOK_INSERT (TOK_DESTINATION (TOK_DIR TOK_TMP_FILE)) (TOK_SELECT (TOK_SELEXPR (TOK_TRANSFORM (TOK_EXPLIST (TOK_TABLE_OR_COL aet) (TOK_TABLE_OR_COL aes)) TOK_SERDE TOK_RECORDWRITER 'cat' TOK_SERDE TOK_RECORDREADER (TOK_TABCOLLIST (TOK_TABCOL aid TOK_STRING) (TOK_TABCOL bid TOK_STRING) (TOK_TABCOL t TOK_INT) (TOK_TABCOL ctime TOK_STRING) (TOK_TABCOL etime TOK_BIGINT) (TOK_TABCOL l TOK_STRING) (TOK_TABCOL et TOK_STRING))))) (TOK_WHERE (= (TOK_TABLE_OR_COL ds) '2010-03-29')) (TOK_CLUSTERBY (TOK_TABLE_OR_COL bid)))) s)) (TOK_INSERT (TOK_DESTINATION (TOK_DIR TOK_TMP_FILE)) (TOK_SELECT (TOK_SELEXPR (TOK_TRANSFORM (TOK_EXPLIST (TOK_TABLE_OR_COL aid) (TOK_TABLE_OR_COL bid) (TOK_TABLE_OR_COL t) (TOK_TABLE_OR_COL ctime) (TOK_TABLE_OR_COL etime) (TOK_TABLE_OR_COL l) (TOK_TABLE_OR_COL et)) TOK_SERDE TOK_RECORDWRITER 'cat' TOK_SERDE TOK_RECORDREADER (TOK_TABCOLLIST (TOK_TABCOL aid TOK_STRING) (TOK_TABCOL bid TOK_STRING) (TOK_TABCOL t TOK_INT) (TOK_TABCOL ctime TOK_STRING) (TOK_TABCOL etime TOK_BIGINT) (TOK_TABCOL l TOK_STRING) (TOK_TABCOL et TOK_STRING))))))) s2)) (TOK_INSERT (TOK_DESTINATION (TOK_TAB complex_tbl_1 (TOK_PARTSPEC (TOK_PARTVAL ds '2010-03-29')))) (TOK_SELECT (TOK_SELEXPR (TOK_ALLCOLREF s2))))) STAGE DEPENDENCIES: Stage-1 is a root stage Stage-0 depends on stages: Stage-1 STAGE PLANS: Stage: Stage-1 Map Reduce Alias -> Map Operator Tree: s2:s:complex_tbl_2 TableScan alias: complex_tbl_2 Filter Operator isSamplingPred: false predicate: expr: (ds = '2010-03-29') type: boolean Filter Operator isSamplingPred: false predicate: expr: (ds = '2010-03-29') type: boolean Select Operator expressions: expr: aet type: string expr: aes type: string outputColumnNames: _col0, _col1 Transform Operator command: cat output info: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat properties: columns _col0,_col1,_col2,_col3,_col4,_col5,_col6 columns.types string,string,int,string,bigint,string,string field.delim 9 serialization.format 9 Reduce Output Operator key expressions: expr: _col1 type: string sort order: + Map-reduce partition columns: expr: _col1 type: string tag: -1 value expressions: expr: _col0 type: string expr: _col1 type: string expr: _col2 type: int expr: _col3 type: string expr: _col4 type: bigint expr: _col5 type: string expr: _col6 type: string Needs Tagging: false Reduce Operator Tree: Extract Select Operator expressions: expr: _col0 type: string expr: _col1 type: string expr: _col2 type: int expr: _col3 type: string expr: _col4 type: bigint expr: _col5 type: string expr: _col6 type: string outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6 Transform Operator command: cat output info: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat properties: columns _col0,_col1,_col2,_col3,_col4,_col5,_col6 columns.types string,string,int,string,bigint,string,string field.delim 9 serialization.format 9 Select Operator expressions: expr: _col0 type: string expr: _col1 type: string expr: _col2 type: int expr: _col3 type: string expr: _col4 type: bigint expr: _col5 type: string expr: _col6 type: string outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6 File Output Operator compressed: false GlobalTableId: 1 directory: file:/data/users/njain/hive_commit1/hive_commit1/build/ql/scratchdir/hive_2010-04-06_13-16-32_432_8813792273595897349/10000 NumFilesPerFileSink: 1 table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat properties: bucket_count -1 columns aid,bid,t,ctime,etime,l,et columns.types string:string:int:string:bigint:string:string file.inputformat org.apache.hadoop.mapred.TextInputFormat file.outputformat org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat location file:/data/users/njain/hive_commit1/hive_commit1/build/ql/test/data/warehouse/complex_tbl_1 name complex_tbl_1 partition_columns ds serialization.ddl struct complex_tbl_1 { string aid, string bid, i32 t, string ctime, i64 etime, string l, string et} serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe transient_lastDdlTime 1270584992 serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe name: complex_tbl_1 TotalFiles: 1 MultiFileSpray: false Stage: Stage-0 Move Operator tables: partition: ds 2010-03-29 replace: true source: file:/data/users/njain/hive_commit1/hive_commit1/build/ql/scratchdir/hive_2010-04-06_13-16-32_432_8813792273595897349/10000 table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat properties: bucket_count -1 columns aid,bid,t,ctime,etime,l,et columns.types string:string:int:string:bigint:string:string file.inputformat org.apache.hadoop.mapred.TextInputFormat file.outputformat org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat location file:/data/users/njain/hive_commit1/hive_commit1/build/ql/test/data/warehouse/complex_tbl_1 name complex_tbl_1 partition_columns ds serialization.ddl struct complex_tbl_1 { string aid, string bid, i32 t, string ctime, i64 etime, string l, string et} serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe transient_lastDdlTime 1270584992 serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe name: complex_tbl_1 tmp directory: file:/data/users/njain/hive_commit1/hive_commit1/build/ql/scratchdir/hive_2010-04-06_13-16-32_432_8813792273595897349/10001 PREHOOK: query: drop table complex_tbl_2 PREHOOK: type: DROPTABLE POSTHOOK: query: drop table complex_tbl_2 POSTHOOK: type: DROPTABLE POSTHOOK: Output: default@complex_tbl_2 POSTHOOK: Lineage: bucket5_1.key SIMPLE [(src)src.FieldSchema(name:key, type:string, comment:default), ] POSTHOOK: Lineage: bucket5_1.value SIMPLE [(src)src.FieldSchema(name:value, type:string, comment:default), ] PREHOOK: query: drop table complex_tbl_1 PREHOOK: type: DROPTABLE POSTHOOK: query: drop table complex_tbl_1 POSTHOOK: type: DROPTABLE POSTHOOK: Output: default@complex_tbl_1 POSTHOOK: Lineage: bucket5_1.key SIMPLE [(src)src.FieldSchema(name:key, type:string, comment:default), ] POSTHOOK: Lineage: bucket5_1.value SIMPLE [(src)src.FieldSchema(name:value, type:string, comment:default), ] PREHOOK: query: drop table bucket5_1 PREHOOK: type: DROPTABLE POSTHOOK: query: drop table bucket5_1 POSTHOOK: type: DROPTABLE POSTHOOK: Output: default@bucket5_1 POSTHOOK: Lineage: bucket5_1.key SIMPLE [(src)src.FieldSchema(name:key, type:string, comment:default), ] POSTHOOK: Lineage: bucket5_1.value SIMPLE [(src)src.FieldSchema(name:value, type:string, comment:default), ]