PREHOOK: query: -- negative, RS + join explain select * from src a join src b on a.key=b.key limit 20 PREHOOK: type: QUERY POSTHOOK: query: -- negative, RS + join explain select * from src a join src b on a.key=b.key limit 20 POSTHOOK: type: QUERY ABSTRACT SYNTAX TREE: (TOK_QUERY (TOK_FROM (TOK_JOIN (TOK_TABREF (TOK_TABNAME src) a) (TOK_TABREF (TOK_TABNAME src) b) (= (. (TOK_TABLE_OR_COL a) key) (. (TOK_TABLE_OR_COL b) key)))) (TOK_INSERT (TOK_DESTINATION (TOK_DIR TOK_TMP_FILE)) (TOK_SELECT (TOK_SELEXPR TOK_ALLCOLREF)) (TOK_LIMIT 20))) STAGE DEPENDENCIES: Stage-1 is a root stage Stage-0 is a root stage STAGE PLANS: Stage: Stage-1 Map Reduce Alias -> Map Operator Tree: a TableScan alias: a Reduce Output Operator key expressions: expr: key type: string sort order: + Map-reduce partition columns: expr: key type: string tag: 0 value expressions: expr: key type: string expr: value type: string b TableScan alias: b Reduce Output Operator key expressions: expr: key type: string sort order: + Map-reduce partition columns: expr: key type: string tag: 1 value expressions: expr: key type: string expr: value type: string Reduce Operator Tree: Join Operator condition map: Inner Join 0 to 1 condition expressions: 0 {VALUE._col0} {VALUE._col1} 1 {VALUE._col0} {VALUE._col1} handleSkewJoin: false outputColumnNames: _col0, _col1, _col4, _col5 Select Operator expressions: expr: _col0 type: string expr: _col1 type: string expr: _col4 type: string expr: _col5 type: string outputColumnNames: _col0, _col1, _col2, _col3 Limit File Output Operator compressed: false GlobalTableId: 0 table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat Stage: Stage-0 Fetch Operator limit: 20 PREHOOK: query: -- negative, RS + filter explain select value, sum(key) as sum from src group by value having sum > 100 limit 20 PREHOOK: type: QUERY POSTHOOK: query: -- negative, RS + filter explain select value, sum(key) as sum from src group by value having sum > 100 limit 20 POSTHOOK: type: QUERY ABSTRACT SYNTAX TREE: (TOK_QUERY (TOK_FROM (TOK_TABREF (TOK_TABNAME src))) (TOK_INSERT (TOK_DESTINATION (TOK_DIR TOK_TMP_FILE)) (TOK_SELECT (TOK_SELEXPR (TOK_TABLE_OR_COL value)) (TOK_SELEXPR (TOK_FUNCTION sum (TOK_TABLE_OR_COL key)) sum)) (TOK_GROUPBY (TOK_TABLE_OR_COL value)) (TOK_HAVING (> (TOK_TABLE_OR_COL sum) 100)) (TOK_LIMIT 20))) STAGE DEPENDENCIES: Stage-1 is a root stage Stage-0 is a root stage STAGE PLANS: Stage: Stage-1 Map Reduce Alias -> Map Operator Tree: src TableScan alias: src Select Operator expressions: expr: value type: string expr: key type: string outputColumnNames: value, key Group By Operator aggregations: expr: sum(key) bucketGroup: false keys: expr: value type: string mode: hash outputColumnNames: _col0, _col1 Reduce Output Operator key expressions: expr: _col0 type: string sort order: + Map-reduce partition columns: expr: _col0 type: string tag: -1 value expressions: expr: _col1 type: double Reduce Operator Tree: Group By Operator aggregations: expr: sum(VALUE._col0) bucketGroup: false keys: expr: KEY._col0 type: string mode: mergepartial outputColumnNames: _col0, _col1 Filter Operator predicate: expr: (_col1 > 100.0) type: boolean Select Operator expressions: expr: _col0 type: string expr: _col1 type: double outputColumnNames: _col0, _col1 Limit File Output Operator compressed: false GlobalTableId: 0 table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat Stage: Stage-0 Fetch Operator limit: 20 PREHOOK: query: -- negative, RS + lateral view explain select key, L.* from (select * from src order by key) a lateral view explode(array(value, value)) L as v limit 10 PREHOOK: type: QUERY POSTHOOK: query: -- negative, RS + lateral view explain select key, L.* from (select * from src order by key) a lateral view explode(array(value, value)) L as v limit 10 POSTHOOK: type: QUERY ABSTRACT SYNTAX TREE: (TOK_QUERY (TOK_FROM (TOK_LATERAL_VIEW (TOK_SELECT (TOK_SELEXPR (TOK_FUNCTION explode (TOK_FUNCTION array (TOK_TABLE_OR_COL value) (TOK_TABLE_OR_COL value))) v (TOK_TABALIAS L))) (TOK_SUBQUERY (TOK_QUERY (TOK_FROM (TOK_TABREF (TOK_TABNAME src))) (TOK_INSERT (TOK_DESTINATION (TOK_DIR TOK_TMP_FILE)) (TOK_SELECT (TOK_SELEXPR TOK_ALLCOLREF)) (TOK_ORDERBY (TOK_TABSORTCOLNAMEASC (TOK_TABLE_OR_COL key))))) a))) (TOK_INSERT (TOK_DESTINATION (TOK_DIR TOK_TMP_FILE)) (TOK_SELECT (TOK_SELEXPR (TOK_TABLE_OR_COL key)) (TOK_SELEXPR (TOK_ALLCOLREF (TOK_TABNAME L)))) (TOK_LIMIT 10))) STAGE DEPENDENCIES: Stage-1 is a root stage Stage-0 is a root stage STAGE PLANS: Stage: Stage-1 Map Reduce Alias -> Map Operator Tree: a:src TableScan alias: src Select Operator expressions: expr: key type: string expr: value type: string outputColumnNames: _col0, _col1 Reduce Output Operator key expressions: expr: _col0 type: string sort order: + tag: -1 value expressions: expr: _col0 type: string expr: _col1 type: string Reduce Operator Tree: Extract Lateral View Forward Select Operator expressions: expr: _col0 type: string outputColumnNames: _col0 Lateral View Join Operator outputColumnNames: _col0, _col2 Select Operator expressions: expr: _col0 type: string expr: _col2 type: string outputColumnNames: _col0, _col1 Limit File Output Operator compressed: false GlobalTableId: 0 table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat Select Operator expressions: expr: array(_col1,_col1) type: array outputColumnNames: _col0 UDTF Operator function name: explode Lateral View Join Operator outputColumnNames: _col0, _col2 Select Operator expressions: expr: _col0 type: string expr: _col2 type: string outputColumnNames: _col0, _col1 Limit File Output Operator compressed: false GlobalTableId: 0 table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat Stage: Stage-0 Fetch Operator limit: 10 PREHOOK: query: -- negative, RS + forward + multi-groupby CREATE TABLE dest_2(key STRING, c1 INT) PREHOOK: type: CREATETABLE POSTHOOK: query: -- negative, RS + forward + multi-groupby CREATE TABLE dest_2(key STRING, c1 INT) POSTHOOK: type: CREATETABLE POSTHOOK: Output: default@dest_2 PREHOOK: query: CREATE TABLE dest_3(key STRING, c1 INT) PREHOOK: type: CREATETABLE POSTHOOK: query: CREATE TABLE dest_3(key STRING, c1 INT) POSTHOOK: type: CREATETABLE POSTHOOK: Output: default@dest_3 PREHOOK: query: EXPLAIN FROM src INSERT OVERWRITE TABLE dest_2 SELECT value, sum(key) GROUP BY value INSERT OVERWRITE TABLE dest_3 SELECT value, sum(key) GROUP BY value limit 20 PREHOOK: type: QUERY POSTHOOK: query: EXPLAIN FROM src INSERT OVERWRITE TABLE dest_2 SELECT value, sum(key) GROUP BY value INSERT OVERWRITE TABLE dest_3 SELECT value, sum(key) GROUP BY value limit 20 POSTHOOK: type: QUERY ABSTRACT SYNTAX TREE: (TOK_QUERY (TOK_FROM (TOK_TABREF (TOK_TABNAME src))) (TOK_INSERT (TOK_DESTINATION (TOK_TAB (TOK_TABNAME dest_2))) (TOK_SELECT (TOK_SELEXPR (TOK_TABLE_OR_COL value)) (TOK_SELEXPR (TOK_FUNCTION sum (TOK_TABLE_OR_COL key)))) (TOK_GROUPBY (TOK_TABLE_OR_COL value))) (TOK_INSERT (TOK_DESTINATION (TOK_TAB (TOK_TABNAME dest_3))) (TOK_SELECT (TOK_SELEXPR (TOK_TABLE_OR_COL value)) (TOK_SELEXPR (TOK_FUNCTION sum (TOK_TABLE_OR_COL key)))) (TOK_GROUPBY (TOK_TABLE_OR_COL value)) (TOK_LIMIT 20))) STAGE DEPENDENCIES: Stage-2 is a root stage Stage-0 depends on stages: Stage-2 Stage-3 depends on stages: Stage-0 Stage-4 depends on stages: Stage-2 Stage-1 depends on stages: Stage-4 Stage-5 depends on stages: Stage-1 STAGE PLANS: Stage: Stage-2 Map Reduce Alias -> Map Operator Tree: src TableScan alias: src Select Operator expressions: expr: value type: string expr: key type: string outputColumnNames: value, key Reduce Output Operator key expressions: expr: value type: string sort order: + Map-reduce partition columns: expr: value type: string tag: -1 value expressions: expr: key type: string Reduce Operator Tree: Forward Group By Operator aggregations: expr: sum(VALUE._col0) bucketGroup: false keys: expr: KEY._col0 type: string mode: complete outputColumnNames: _col0, _col1 Select Operator expressions: expr: _col0 type: string expr: UDFToInteger(_col1) type: int outputColumnNames: _col0, _col1 File Output Operator compressed: false GlobalTableId: 1 table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe name: default.dest_2 Group By Operator aggregations: expr: sum(VALUE._col0) bucketGroup: false keys: expr: KEY._col0 type: string mode: complete outputColumnNames: _col0, _col1 Select Operator expressions: expr: _col0 type: string expr: _col1 type: double outputColumnNames: _col0, _col1 Limit File Output Operator compressed: false GlobalTableId: 0 table: input format: org.apache.hadoop.mapred.SequenceFileInputFormat output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat Stage: Stage-0 Move Operator tables: replace: true table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe name: default.dest_2 Stage: Stage-3 Stats-Aggr Operator Stage: Stage-4 Map Reduce Alias -> Map Operator Tree: #### A masked pattern was here #### Reduce Output Operator sort order: tag: -1 TopN: 20 TopN Hash Memory Usage: 0.3 value expressions: expr: _col0 type: string expr: _col1 type: double Reduce Operator Tree: Extract Limit Select Operator expressions: expr: _col0 type: string expr: UDFToInteger(_col1) type: int outputColumnNames: _col0, _col1 File Output Operator compressed: false GlobalTableId: 2 table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe name: default.dest_3 Stage: Stage-1 Move Operator tables: replace: true table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe name: default.dest_3 Stage: Stage-5 Stats-Aggr Operator PREHOOK: query: -- nagative, multi distinct explain select count(distinct key)+count(distinct value) from src limit 20 PREHOOK: type: QUERY POSTHOOK: query: -- nagative, multi distinct explain select count(distinct key)+count(distinct value) from src limit 20 POSTHOOK: type: QUERY ABSTRACT SYNTAX TREE: (TOK_QUERY (TOK_FROM (TOK_TABREF (TOK_TABNAME src))) (TOK_INSERT (TOK_DESTINATION (TOK_DIR TOK_TMP_FILE)) (TOK_SELECT (TOK_SELEXPR (+ (TOK_FUNCTIONDI count (TOK_TABLE_OR_COL key)) (TOK_FUNCTIONDI count (TOK_TABLE_OR_COL value))))) (TOK_LIMIT 20))) STAGE DEPENDENCIES: Stage-1 is a root stage Stage-0 is a root stage STAGE PLANS: Stage: Stage-1 Map Reduce Alias -> Map Operator Tree: src TableScan alias: src Select Operator expressions: expr: key type: string expr: value type: string outputColumnNames: key, value Group By Operator aggregations: expr: count(DISTINCT key) expr: count(DISTINCT value) bucketGroup: false keys: expr: key type: string expr: value type: string mode: hash outputColumnNames: _col0, _col1, _col2, _col3 Reduce Output Operator key expressions: expr: _col0 type: string expr: _col1 type: string sort order: ++ tag: -1 value expressions: expr: _col2 type: bigint expr: _col3 type: bigint Reduce Operator Tree: Group By Operator aggregations: expr: count(DISTINCT KEY._col0:0._col0) expr: count(DISTINCT KEY._col0:1._col0) bucketGroup: false mode: mergepartial outputColumnNames: _col0, _col1 Select Operator expressions: expr: (_col0 + _col1) type: bigint outputColumnNames: _col0 Limit File Output Operator compressed: false GlobalTableId: 0 table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat Stage: Stage-0 Fetch Operator limit: 20