PREHOOK: query: CREATE TABLE T1(key STRING, value STRING) STORED AS TEXTFILE PREHOOK: type: CREATETABLE PREHOOK: Output: database:default PREHOOK: Output: default@T1 POSTHOOK: query: CREATE TABLE T1(key STRING, value STRING) STORED AS TEXTFILE POSTHOOK: type: CREATETABLE POSTHOOK: Output: database:default POSTHOOK: Output: default@T1 PREHOOK: query: LOAD DATA LOCAL INPATH '../../data/files/T1.txt' INTO TABLE T1 PREHOOK: type: LOAD #### A masked pattern was here #### PREHOOK: Output: default@t1 POSTHOOK: query: LOAD DATA LOCAL INPATH '../../data/files/T1.txt' INTO TABLE T1 POSTHOOK: type: LOAD #### A masked pattern was here #### POSTHOOK: Output: default@t1 PREHOOK: query: drop table array_valued_T1 PREHOOK: type: DROPTABLE POSTHOOK: query: drop table array_valued_T1 POSTHOOK: type: DROPTABLE PREHOOK: query: create table array_valued_T1 (key string, value array) SKEWED BY (key) ON ((8)) PREHOOK: type: CREATETABLE PREHOOK: Output: database:default PREHOOK: Output: default@array_valued_T1 POSTHOOK: query: create table array_valued_T1 (key string, value array) SKEWED BY (key) ON ((8)) POSTHOOK: type: CREATETABLE POSTHOOK: Output: database:default POSTHOOK: Output: default@array_valued_T1 PREHOOK: query: insert overwrite table array_valued_T1 select key, array(value) from T1 PREHOOK: type: QUERY PREHOOK: Input: default@t1 PREHOOK: Output: default@array_valued_t1 POSTHOOK: query: insert overwrite table array_valued_T1 select key, array(value) from T1 POSTHOOK: type: QUERY POSTHOOK: Input: default@t1 POSTHOOK: Output: default@array_valued_t1 POSTHOOK: Lineage: array_valued_t1.key SIMPLE [(t1)t1.FieldSchema(name:key, type:string, comment:null), ] POSTHOOK: Lineage: array_valued_t1.value EXPRESSION [(t1)t1.FieldSchema(name:value, type:string, comment:null), ] PREHOOK: query: -- copy from skewjoinopt10 -- test compile time skew join and auto map join -- This test is to verify the skew join compile optimization when the join is followed by a lateral view -- adding a order by at the end to make the results deterministic explain select * from (select a.key as key, b.value as array_val from T1 a join array_valued_T1 b on a.key=b.key) i lateral view explode (array_val) c as val PREHOOK: type: QUERY POSTHOOK: query: -- copy from skewjoinopt10 -- test compile time skew join and auto map join -- This test is to verify the skew join compile optimization when the join is followed by a lateral view -- adding a order by at the end to make the results deterministic explain select * from (select a.key as key, b.value as array_val from T1 a join array_valued_T1 b on a.key=b.key) i lateral view explode (array_val) c as val POSTHOOK: type: QUERY STAGE DEPENDENCIES: Stage-8 is a root stage Stage-2 depends on stages: Stage-8 Stage-0 depends on stages: Stage-2 STAGE PLANS: Stage: Stage-8 Map Reduce Local Work Alias -> Map Local Tables: i:a Fetch Operator limit: -1 subquery1:a Fetch Operator limit: -1 Alias -> Map Local Operator Tree: i:a TableScan alias: a Statistics: Num rows: 0 Data size: 30 Basic stats: PARTIAL Column stats: NONE Filter Operator predicate: (key is not null and (key = '8')) (type: boolean) Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE HashTable Sink Operator keys: 0 key (type: string) 1 key (type: string) subquery1:a TableScan alias: a Statistics: Num rows: 0 Data size: 30 Basic stats: PARTIAL Column stats: NONE Filter Operator predicate: (key is not null and (not (key = '8'))) (type: boolean) Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE HashTable Sink Operator keys: 0 key (type: string) 1 key (type: string) Stage: Stage-2 Map Reduce Map Operator Tree: TableScan alias: b Statistics: Num rows: 6 Data size: 24 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: (key is not null and (key = '8')) (type: boolean) Statistics: Num rows: 1 Data size: 4 Basic stats: COMPLETE Column stats: NONE Map Join Operator condition map: Inner Join 0 to 1 keys: 0 key (type: string) 1 key (type: string) outputColumnNames: _col0, _col6 Statistics: Num rows: 1 Data size: 4 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col0 (type: string), _col6 (type: array) outputColumnNames: _col0, _col1 Statistics: Num rows: 1 Data size: 4 Basic stats: COMPLETE Column stats: NONE Union Statistics: Num rows: 3 Data size: 12 Basic stats: COMPLETE Column stats: NONE Lateral View Forward Statistics: Num rows: 3 Data size: 12 Basic stats: COMPLETE Column stats: NONE Select Operator SELECT * : (no compute) expressions: _col0 (type: string), _col1 (type: array) outputColumnNames: org.apache.hadoop.hive.ql.plan.ExprNodeColumnDesc, org.apache.hadoop.hive.ql.plan.ExprNodeColumnDesc Statistics: Num rows: 3 Data size: 12 Basic stats: COMPLETE Column stats: NONE Lateral View Join Operator outputColumnNames: _col0, _col1, _col2 Statistics: Num rows: 6 Data size: 24 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col0 (type: string), _col1 (type: array), _col2 (type: string) outputColumnNames: _col0, _col1, _col2 Statistics: Num rows: 6 Data size: 24 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false Statistics: Num rows: 6 Data size: 24 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Select Operator expressions: _col1 (type: array) outputColumnNames: _col0 Statistics: Num rows: 3 Data size: 12 Basic stats: COMPLETE Column stats: NONE UDTF Operator Statistics: Num rows: 3 Data size: 12 Basic stats: COMPLETE Column stats: NONE function name: explode Lateral View Join Operator outputColumnNames: _col0, _col1, _col2 Statistics: Num rows: 6 Data size: 24 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col0 (type: string), _col1 (type: array), _col2 (type: string) outputColumnNames: _col0, _col1, _col2 Statistics: Num rows: 6 Data size: 24 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false Statistics: Num rows: 6 Data size: 24 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe TableScan alias: b Statistics: Num rows: 6 Data size: 24 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: (key is not null and (not (key = '8'))) (type: boolean) Statistics: Num rows: 2 Data size: 8 Basic stats: COMPLETE Column stats: NONE Map Join Operator condition map: Inner Join 0 to 1 keys: 0 key (type: string) 1 key (type: string) outputColumnNames: _col0, _col6 Statistics: Num rows: 2 Data size: 8 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col0 (type: string), _col6 (type: array) outputColumnNames: _col0, _col1 Statistics: Num rows: 2 Data size: 8 Basic stats: COMPLETE Column stats: NONE Union Statistics: Num rows: 3 Data size: 12 Basic stats: COMPLETE Column stats: NONE Lateral View Forward Statistics: Num rows: 3 Data size: 12 Basic stats: COMPLETE Column stats: NONE Select Operator SELECT * : (no compute) expressions: _col0 (type: string), _col1 (type: array) outputColumnNames: org.apache.hadoop.hive.ql.plan.ExprNodeColumnDesc, org.apache.hadoop.hive.ql.plan.ExprNodeColumnDesc Statistics: Num rows: 3 Data size: 12 Basic stats: COMPLETE Column stats: NONE Lateral View Join Operator outputColumnNames: _col0, _col1, _col2 Statistics: Num rows: 6 Data size: 24 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col0 (type: string), _col1 (type: array), _col2 (type: string) outputColumnNames: _col0, _col1, _col2 Statistics: Num rows: 6 Data size: 24 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false Statistics: Num rows: 6 Data size: 24 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Select Operator expressions: _col1 (type: array) outputColumnNames: _col0 Statistics: Num rows: 3 Data size: 12 Basic stats: COMPLETE Column stats: NONE UDTF Operator Statistics: Num rows: 3 Data size: 12 Basic stats: COMPLETE Column stats: NONE function name: explode Lateral View Join Operator outputColumnNames: _col0, _col1, _col2 Statistics: Num rows: 6 Data size: 24 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col0 (type: string), _col1 (type: array), _col2 (type: string) outputColumnNames: _col0, _col1, _col2 Statistics: Num rows: 6 Data size: 24 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false Statistics: Num rows: 6 Data size: 24 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Local Work: Map Reduce Local Work Stage: Stage-0 Fetch Operator limit: -1 Processor Tree: ListSink PREHOOK: query: select * from (select a.key as key, b.value as array_val from T1 a join array_valued_T1 b on a.key=b.key) i lateral view explode (array_val) c as val ORDER BY key, val PREHOOK: type: QUERY PREHOOK: Input: default@array_valued_t1 PREHOOK: Input: default@t1 #### A masked pattern was here #### POSTHOOK: query: select * from (select a.key as key, b.value as array_val from T1 a join array_valued_T1 b on a.key=b.key) i lateral view explode (array_val) c as val ORDER BY key, val POSTHOOK: type: QUERY POSTHOOK: Input: default@array_valued_t1 POSTHOOK: Input: default@t1 #### A masked pattern was here #### 1 ["11"] 11 2 ["12"] 12 3 ["13"] 13 7 ["17"] 17 8 ["18"] 18 8 ["18"] 18 8 ["28"] 28 8 ["28"] 28