PREHOOK: query: -- we will generate one MR job. EXPLAIN SELECT tmp.key FROM (SELECT x1.key AS key FROM src x1 JOIN src1 y1 ON (x1.key = y1.key) UNION ALL SELECT x2.key AS key FROM src x2 JOIN src1 y2 ON (x2.key = y2.key)) tmp ORDER BY tmp.key PREHOOK: type: QUERY POSTHOOK: query: -- we will generate one MR job. EXPLAIN SELECT tmp.key FROM (SELECT x1.key AS key FROM src x1 JOIN src1 y1 ON (x1.key = y1.key) UNION ALL SELECT x2.key AS key FROM src x2 JOIN src1 y2 ON (x2.key = y2.key)) tmp ORDER BY tmp.key POSTHOOK: type: QUERY STAGE DEPENDENCIES: Stage-8 is a root stage Stage-2 depends on stages: Stage-8 Stage-0 depends on stages: Stage-2 STAGE PLANS: Stage: Stage-8 Map Reduce Local Work Alias -> Map Local Tables: null-subquery1:$hdt$_0-subquery1:$hdt$_1:y1 Fetch Operator limit: -1 null-subquery2:$hdt$_0-subquery2:$hdt$_1:y1 Fetch Operator limit: -1 Alias -> Map Local Operator Tree: null-subquery1:$hdt$_0-subquery1:$hdt$_1:y1 TableScan alias: y1 Statistics: Num rows: 25 Data size: 191 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: key is not null (type: boolean) Statistics: Num rows: 13 Data size: 99 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: string) outputColumnNames: _col0 Statistics: Num rows: 13 Data size: 99 Basic stats: COMPLETE Column stats: NONE HashTable Sink Operator keys: 0 _col0 (type: string) 1 _col0 (type: string) null-subquery2:$hdt$_0-subquery2:$hdt$_1:y1 TableScan alias: y1 Statistics: Num rows: 25 Data size: 191 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: key is not null (type: boolean) Statistics: Num rows: 13 Data size: 99 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: string) outputColumnNames: _col0 Statistics: Num rows: 13 Data size: 99 Basic stats: COMPLETE Column stats: NONE HashTable Sink Operator keys: 0 _col0 (type: string) 1 _col0 (type: string) Stage: Stage-2 Map Reduce Map Operator Tree: TableScan alias: x1 Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: key is not null (type: boolean) Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: string) outputColumnNames: _col0 Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE Map Join Operator condition map: Inner Join 0 to 1 keys: 0 _col0 (type: string) 1 _col0 (type: string) outputColumnNames: _col0 Statistics: Num rows: 275 Data size: 2921 Basic stats: COMPLETE Column stats: NONE Union Statistics: Num rows: 550 Data size: 5842 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: string) sort order: + Statistics: Num rows: 550 Data size: 5842 Basic stats: COMPLETE Column stats: NONE TableScan alias: x1 Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: key is not null (type: boolean) Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: string) outputColumnNames: _col0 Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE Map Join Operator condition map: Inner Join 0 to 1 keys: 0 _col0 (type: string) 1 _col0 (type: string) outputColumnNames: _col0 Statistics: Num rows: 275 Data size: 2921 Basic stats: COMPLETE Column stats: NONE Union Statistics: Num rows: 550 Data size: 5842 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: string) sort order: + Statistics: Num rows: 550 Data size: 5842 Basic stats: COMPLETE Column stats: NONE Local Work: Map Reduce Local Work Reduce Operator Tree: Select Operator expressions: KEY.reducesinkkey0 (type: string) outputColumnNames: _col0 Statistics: Num rows: 550 Data size: 5842 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false Statistics: Num rows: 550 Data size: 5842 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Stage: Stage-0 Fetch Operator limit: -1 Processor Tree: ListSink PREHOOK: query: SELECT tmp.key FROM (SELECT x1.key AS key FROM src x1 JOIN src1 y1 ON (x1.key = y1.key) UNION ALL SELECT x2.key AS key FROM src x2 JOIN src1 y2 ON (x2.key = y2.key)) tmp ORDER BY tmp.key PREHOOK: type: QUERY PREHOOK: Input: default@src PREHOOK: Input: default@src1 #### A masked pattern was here #### POSTHOOK: query: SELECT tmp.key FROM (SELECT x1.key AS key FROM src x1 JOIN src1 y1 ON (x1.key = y1.key) UNION ALL SELECT x2.key AS key FROM src x2 JOIN src1 y2 ON (x2.key = y2.key)) tmp ORDER BY tmp.key POSTHOOK: type: QUERY POSTHOOK: Input: default@src POSTHOOK: Input: default@src1 #### A masked pattern was here #### RUN: Stage-8:MAPREDLOCAL RUN: Stage-2:MAPRED 128 128 128 128 128 128 146 146 146 146 150 150 213 213 213 213 224 224 224 224 238 238 238 238 255 255 255 255 273 273 273 273 273 273 278 278 278 278 311 311 311 311 311 311 369 369 369 369 369 369 401 401 401 401 401 401 401 401 401 401 406 406 406 406 406 406 406 406 66 66 98 98 98 98 PREHOOK: query: -- Check if the total size of local tables will be -- larger than the limit that -- we set through hive.auto.convert.join.noconditionaltask.size (right now, it is -- 400 bytes). If so, do not merge. -- For this query, we will merge the MapJoin of x2 and y2 into the MR job -- for UNION ALL and ORDER BY. But, the MapJoin of x1 and y2 will not be merged -- into that MR job. EXPLAIN SELECT tmp.key FROM (SELECT x1.key AS key FROM src x1 JOIN src1 y1 ON (x1.key = y1.key) UNION ALL SELECT x2.key AS key FROM src x2 JOIN src1 y2 ON (x2.key = y2.key)) tmp ORDER BY tmp.key PREHOOK: type: QUERY POSTHOOK: query: -- Check if the total size of local tables will be -- larger than the limit that -- we set through hive.auto.convert.join.noconditionaltask.size (right now, it is -- 400 bytes). If so, do not merge. -- For this query, we will merge the MapJoin of x2 and y2 into the MR job -- for UNION ALL and ORDER BY. But, the MapJoin of x1 and y2 will not be merged -- into that MR job. EXPLAIN SELECT tmp.key FROM (SELECT x1.key AS key FROM src x1 JOIN src1 y1 ON (x1.key = y1.key) UNION ALL SELECT x2.key AS key FROM src x2 JOIN src1 y2 ON (x2.key = y2.key)) tmp ORDER BY tmp.key POSTHOOK: type: QUERY STAGE DEPENDENCIES: Stage-9 is a root stage Stage-7 depends on stages: Stage-9 Stage-8 depends on stages: Stage-7 Stage-2 depends on stages: Stage-8 Stage-0 depends on stages: Stage-2 STAGE PLANS: Stage: Stage-9 Map Reduce Local Work Alias -> Map Local Tables: null-subquery2:$hdt$_0-subquery2:$hdt$_1:y1 Fetch Operator limit: -1 Alias -> Map Local Operator Tree: null-subquery2:$hdt$_0-subquery2:$hdt$_1:y1 TableScan alias: y1 Statistics: Num rows: 25 Data size: 191 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: key is not null (type: boolean) Statistics: Num rows: 13 Data size: 99 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: string) outputColumnNames: _col0 Statistics: Num rows: 13 Data size: 99 Basic stats: COMPLETE Column stats: NONE HashTable Sink Operator keys: 0 _col0 (type: string) 1 _col0 (type: string) Stage: Stage-7 Map Reduce Map Operator Tree: TableScan alias: x1 Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: key is not null (type: boolean) Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: string) outputColumnNames: _col0 Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE Map Join Operator condition map: Inner Join 0 to 1 keys: 0 _col0 (type: string) 1 _col0 (type: string) outputColumnNames: _col0 Statistics: Num rows: 275 Data size: 2921 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false table: input format: org.apache.hadoop.mapred.SequenceFileInputFormat output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe Local Work: Map Reduce Local Work Stage: Stage-8 Map Reduce Local Work Alias -> Map Local Tables: null-subquery1:$hdt$_0-subquery1:$hdt$_1:y1 Fetch Operator limit: -1 Alias -> Map Local Operator Tree: null-subquery1:$hdt$_0-subquery1:$hdt$_1:y1 TableScan alias: y1 Statistics: Num rows: 25 Data size: 191 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: key is not null (type: boolean) Statistics: Num rows: 13 Data size: 99 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: string) outputColumnNames: _col0 Statistics: Num rows: 13 Data size: 99 Basic stats: COMPLETE Column stats: NONE HashTable Sink Operator keys: 0 _col0 (type: string) 1 _col0 (type: string) Stage: Stage-2 Map Reduce Map Operator Tree: TableScan Union Statistics: Num rows: 550 Data size: 5842 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: string) sort order: + Statistics: Num rows: 550 Data size: 5842 Basic stats: COMPLETE Column stats: NONE TableScan alias: x1 Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: key is not null (type: boolean) Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: string) outputColumnNames: _col0 Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE Map Join Operator condition map: Inner Join 0 to 1 keys: 0 _col0 (type: string) 1 _col0 (type: string) outputColumnNames: _col0 Statistics: Num rows: 275 Data size: 2921 Basic stats: COMPLETE Column stats: NONE Union Statistics: Num rows: 550 Data size: 5842 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: string) sort order: + Statistics: Num rows: 550 Data size: 5842 Basic stats: COMPLETE Column stats: NONE Local Work: Map Reduce Local Work Reduce Operator Tree: Select Operator expressions: KEY.reducesinkkey0 (type: string) outputColumnNames: _col0 Statistics: Num rows: 550 Data size: 5842 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false Statistics: Num rows: 550 Data size: 5842 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Stage: Stage-0 Fetch Operator limit: -1 Processor Tree: ListSink PREHOOK: query: SELECT tmp.key FROM (SELECT x1.key AS key FROM src x1 JOIN src1 y1 ON (x1.key = y1.key) UNION ALL SELECT x2.key AS key FROM src x2 JOIN src1 y2 ON (x2.key = y2.key)) tmp ORDER BY tmp.key PREHOOK: type: QUERY PREHOOK: Input: default@src PREHOOK: Input: default@src1 #### A masked pattern was here #### POSTHOOK: query: SELECT tmp.key FROM (SELECT x1.key AS key FROM src x1 JOIN src1 y1 ON (x1.key = y1.key) UNION ALL SELECT x2.key AS key FROM src x2 JOIN src1 y2 ON (x2.key = y2.key)) tmp ORDER BY tmp.key POSTHOOK: type: QUERY POSTHOOK: Input: default@src POSTHOOK: Input: default@src1 #### A masked pattern was here #### RUN: Stage-9:MAPREDLOCAL RUN: Stage-7:MAPRED RUN: Stage-8:MAPREDLOCAL RUN: Stage-2:MAPRED 128 128 128 128 128 128 146 146 146 146 150 150 213 213 213 213 224 224 224 224 238 238 238 238 255 255 255 255 273 273 273 273 273 273 278 278 278 278 311 311 311 311 311 311 369 369 369 369 369 369 401 401 401 401 401 401 401 401 401 401 406 406 406 406 406 406 406 406 66 66 98 98 98 98 PREHOOK: query: -- We will use two jobs. -- We will generate one MR job for GROUP BY -- on x1, one MR job for both the MapJoin of x2 and y2, the UNION ALL, and the -- ORDER BY. EXPLAIN SELECT tmp.key FROM (SELECT x1.key AS key FROM src1 x1 GROUP BY x1.key UNION ALL SELECT x2.key AS key FROM src x2 JOIN src1 y2 ON (x2.key = y2.key)) tmp ORDER BY tmp.key PREHOOK: type: QUERY POSTHOOK: query: -- We will use two jobs. -- We will generate one MR job for GROUP BY -- on x1, one MR job for both the MapJoin of x2 and y2, the UNION ALL, and the -- ORDER BY. EXPLAIN SELECT tmp.key FROM (SELECT x1.key AS key FROM src1 x1 GROUP BY x1.key UNION ALL SELECT x2.key AS key FROM src x2 JOIN src1 y2 ON (x2.key = y2.key)) tmp ORDER BY tmp.key POSTHOOK: type: QUERY STAGE DEPENDENCIES: Stage-1 is a root stage Stage-6 depends on stages: Stage-1 Stage-2 depends on stages: Stage-6 Stage-0 depends on stages: Stage-2 STAGE PLANS: Stage: Stage-1 Map Reduce Map Operator Tree: TableScan alias: x1 Statistics: Num rows: 25 Data size: 191 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: string) outputColumnNames: _col0 Statistics: Num rows: 25 Data size: 191 Basic stats: COMPLETE Column stats: NONE Group By Operator keys: _col0 (type: string) mode: hash outputColumnNames: _col0 Statistics: Num rows: 25 Data size: 191 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: string) sort order: + Map-reduce partition columns: _col0 (type: string) Statistics: Num rows: 25 Data size: 191 Basic stats: COMPLETE Column stats: NONE Reduce Operator Tree: Group By Operator keys: KEY._col0 (type: string) mode: mergepartial outputColumnNames: _col0 Statistics: Num rows: 12 Data size: 91 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false table: input format: org.apache.hadoop.mapred.SequenceFileInputFormat output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe Stage: Stage-6 Map Reduce Local Work Alias -> Map Local Tables: null-subquery2:$hdt$_0-subquery2:$hdt$_1:x1 Fetch Operator limit: -1 Alias -> Map Local Operator Tree: null-subquery2:$hdt$_0-subquery2:$hdt$_1:x1 TableScan alias: x1 Statistics: Num rows: 25 Data size: 191 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: key is not null (type: boolean) Statistics: Num rows: 13 Data size: 99 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: string) outputColumnNames: _col0 Statistics: Num rows: 13 Data size: 99 Basic stats: COMPLETE Column stats: NONE HashTable Sink Operator keys: 0 _col0 (type: string) 1 _col0 (type: string) Stage: Stage-2 Map Reduce Map Operator Tree: TableScan Union Statistics: Num rows: 287 Data size: 3012 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: string) sort order: + Statistics: Num rows: 287 Data size: 3012 Basic stats: COMPLETE Column stats: NONE TableScan alias: x2 Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: key is not null (type: boolean) Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: string) outputColumnNames: _col0 Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE Map Join Operator condition map: Inner Join 0 to 1 keys: 0 _col0 (type: string) 1 _col0 (type: string) outputColumnNames: _col0 Statistics: Num rows: 275 Data size: 2921 Basic stats: COMPLETE Column stats: NONE Union Statistics: Num rows: 287 Data size: 3012 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: string) sort order: + Statistics: Num rows: 287 Data size: 3012 Basic stats: COMPLETE Column stats: NONE Local Work: Map Reduce Local Work Reduce Operator Tree: Select Operator expressions: KEY.reducesinkkey0 (type: string) outputColumnNames: _col0 Statistics: Num rows: 287 Data size: 3012 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false Statistics: Num rows: 287 Data size: 3012 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Stage: Stage-0 Fetch Operator limit: -1 Processor Tree: ListSink PREHOOK: query: SELECT tmp.key FROM (SELECT x1.key AS key FROM src1 x1 GROUP BY x1.key UNION ALL SELECT x2.key AS key FROM src x2 JOIN src1 y2 ON (x2.key = y2.key)) tmp ORDER BY tmp.key PREHOOK: type: QUERY PREHOOK: Input: default@src PREHOOK: Input: default@src1 #### A masked pattern was here #### POSTHOOK: query: SELECT tmp.key FROM (SELECT x1.key AS key FROM src1 x1 GROUP BY x1.key UNION ALL SELECT x2.key AS key FROM src x2 JOIN src1 y2 ON (x2.key = y2.key)) tmp ORDER BY tmp.key POSTHOOK: type: QUERY POSTHOOK: Input: default@src POSTHOOK: Input: default@src1 #### A masked pattern was here #### RUN: Stage-1:MAPRED RUN: Stage-6:MAPREDLOCAL RUN: Stage-2:MAPRED 128 128 128 128 146 146 146 150 150 213 213 213 224 224 224 238 238 238 255 255 255 273 273 273 273 278 278 278 311 311 311 311 369 369 369 369 401 401 401 401 401 401 406 406 406 406 406 66 66 98 98 98 PREHOOK: query: -- When Correlation Optimizer is disabled, -- we will use 5 jobs. -- We will generate one MR job to evaluate the sub-query tmp1, -- one MR job to evaluate the sub-query tmp2, -- one MR job for the Join of tmp1 and tmp2, -- one MR job for aggregation on the result of the Join of tmp1 and tmp2, -- and one MR job for the ORDER BY. EXPLAIN SELECT tmp1.key as key, count(*) as cnt FROM (SELECT x1.key AS key FROM src x1 JOIN src1 y1 ON (x1.key = y1.key) GROUP BY x1.key) tmp1 JOIN (SELECT x2.key AS key FROM src x2 JOIN src1 y2 ON (x2.key = y2.key) GROUP BY x2.key) tmp2 ON (tmp1.key = tmp2.key) GROUP BY tmp1.key ORDER BY key, cnt PREHOOK: type: QUERY POSTHOOK: query: -- When Correlation Optimizer is disabled, -- we will use 5 jobs. -- We will generate one MR job to evaluate the sub-query tmp1, -- one MR job to evaluate the sub-query tmp2, -- one MR job for the Join of tmp1 and tmp2, -- one MR job for aggregation on the result of the Join of tmp1 and tmp2, -- and one MR job for the ORDER BY. EXPLAIN SELECT tmp1.key as key, count(*) as cnt FROM (SELECT x1.key AS key FROM src x1 JOIN src1 y1 ON (x1.key = y1.key) GROUP BY x1.key) tmp1 JOIN (SELECT x2.key AS key FROM src x2 JOIN src1 y2 ON (x2.key = y2.key) GROUP BY x2.key) tmp2 ON (tmp1.key = tmp2.key) GROUP BY tmp1.key ORDER BY key, cnt POSTHOOK: type: QUERY STAGE DEPENDENCIES: Stage-17 is a root stage Stage-2 depends on stages: Stage-17 Stage-12 depends on stages: Stage-2, Stage-8 , consists of Stage-15, Stage-16, Stage-3 Stage-15 has a backup stage: Stage-3 Stage-10 depends on stages: Stage-15 Stage-4 depends on stages: Stage-3, Stage-10, Stage-11 Stage-5 depends on stages: Stage-4 Stage-16 has a backup stage: Stage-3 Stage-11 depends on stages: Stage-16 Stage-3 Stage-18 is a root stage Stage-8 depends on stages: Stage-18 Stage-0 depends on stages: Stage-5 STAGE PLANS: Stage: Stage-17 Map Reduce Local Work Alias -> Map Local Tables: $hdt$_0:$hdt$_0:$hdt$_0:$hdt$_0:$hdt$_1:y1 Fetch Operator limit: -1 Alias -> Map Local Operator Tree: $hdt$_0:$hdt$_0:$hdt$_0:$hdt$_0:$hdt$_1:y1 TableScan alias: y1 Statistics: Num rows: 25 Data size: 191 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: key is not null (type: boolean) Statistics: Num rows: 13 Data size: 99 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: string) outputColumnNames: _col0 Statistics: Num rows: 13 Data size: 99 Basic stats: COMPLETE Column stats: NONE HashTable Sink Operator keys: 0 _col0 (type: string) 1 _col0 (type: string) Stage: Stage-2 Map Reduce Map Operator Tree: TableScan alias: x1 Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: key is not null (type: boolean) Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: string) outputColumnNames: _col0 Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE Map Join Operator condition map: Inner Join 0 to 1 keys: 0 _col0 (type: string) 1 _col0 (type: string) outputColumnNames: _col0 Statistics: Num rows: 275 Data size: 2921 Basic stats: COMPLETE Column stats: NONE Group By Operator keys: _col0 (type: string) mode: hash outputColumnNames: _col0 Statistics: Num rows: 275 Data size: 2921 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: string) sort order: + Map-reduce partition columns: _col0 (type: string) Statistics: Num rows: 275 Data size: 2921 Basic stats: COMPLETE Column stats: NONE Local Work: Map Reduce Local Work Reduce Operator Tree: Group By Operator keys: KEY._col0 (type: string) mode: mergepartial outputColumnNames: _col0 Statistics: Num rows: 137 Data size: 1455 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false table: input format: org.apache.hadoop.mapred.SequenceFileInputFormat output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe Stage: Stage-12 Conditional Operator Stage: Stage-15 Map Reduce Local Work Alias -> Map Local Tables: $hdt$_0:$hdt$_0:$INTNAME1 Fetch Operator limit: -1 Alias -> Map Local Operator Tree: $hdt$_0:$hdt$_0:$INTNAME1 TableScan HashTable Sink Operator keys: 0 _col0 (type: string) 1 _col0 (type: string) Stage: Stage-10 Map Reduce Map Operator Tree: TableScan Map Join Operator condition map: Inner Join 0 to 1 keys: 0 _col0 (type: string) 1 _col0 (type: string) outputColumnNames: _col0 Group By Operator aggregations: count() keys: _col0 (type: string) mode: hash outputColumnNames: _col0, _col1 File Output Operator compressed: false table: input format: org.apache.hadoop.mapred.SequenceFileInputFormat output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe Local Work: Map Reduce Local Work Stage: Stage-4 Map Reduce Map Operator Tree: TableScan Reduce Output Operator key expressions: _col0 (type: string) sort order: + Map-reduce partition columns: _col0 (type: string) Statistics: Num rows: 150 Data size: 1600 Basic stats: COMPLETE Column stats: NONE value expressions: _col1 (type: bigint) Reduce Operator Tree: Group By Operator aggregations: count(VALUE._col0) keys: KEY._col0 (type: string) mode: mergepartial outputColumnNames: _col0, _col1 Statistics: Num rows: 75 Data size: 800 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false table: input format: org.apache.hadoop.mapred.SequenceFileInputFormat output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe Stage: Stage-5 Map Reduce Map Operator Tree: TableScan Reduce Output Operator key expressions: _col0 (type: string), _col1 (type: bigint) sort order: ++ Statistics: Num rows: 75 Data size: 800 Basic stats: COMPLETE Column stats: NONE Reduce Operator Tree: Select Operator expressions: KEY.reducesinkkey0 (type: string), KEY.reducesinkkey1 (type: bigint) outputColumnNames: _col0, _col1 Statistics: Num rows: 75 Data size: 800 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false Statistics: Num rows: 75 Data size: 800 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Stage: Stage-16 Map Reduce Local Work Alias -> Map Local Tables: $hdt$_0:$hdt$_0:$INTNAME Fetch Operator limit: -1 Alias -> Map Local Operator Tree: $hdt$_0:$hdt$_0:$INTNAME TableScan HashTable Sink Operator keys: 0 _col0 (type: string) 1 _col0 (type: string) Stage: Stage-11 Map Reduce Map Operator Tree: TableScan Map Join Operator condition map: Inner Join 0 to 1 keys: 0 _col0 (type: string) 1 _col0 (type: string) outputColumnNames: _col0 Group By Operator aggregations: count() keys: _col0 (type: string) mode: hash outputColumnNames: _col0, _col1 File Output Operator compressed: false table: input format: org.apache.hadoop.mapred.SequenceFileInputFormat output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe Local Work: Map Reduce Local Work Stage: Stage-3 Map Reduce Map Operator Tree: TableScan Reduce Output Operator key expressions: _col0 (type: string) sort order: + Map-reduce partition columns: _col0 (type: string) Statistics: Num rows: 137 Data size: 1455 Basic stats: COMPLETE Column stats: NONE TableScan Reduce Output Operator key expressions: _col0 (type: string) sort order: + Map-reduce partition columns: _col0 (type: string) Statistics: Num rows: 137 Data size: 1455 Basic stats: COMPLETE Column stats: NONE Reduce Operator Tree: Join Operator condition map: Inner Join 0 to 1 keys: 0 _col0 (type: string) 1 _col0 (type: string) outputColumnNames: _col0 Statistics: Num rows: 150 Data size: 1600 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: count() keys: _col0 (type: string) mode: hash outputColumnNames: _col0, _col1 Statistics: Num rows: 150 Data size: 1600 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false table: input format: org.apache.hadoop.mapred.SequenceFileInputFormat output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe Stage: Stage-18 Map Reduce Local Work Alias -> Map Local Tables: $hdt$_0:$hdt$_0:$hdt$_1:$hdt$_1:$hdt$_2:y1 Fetch Operator limit: -1 Alias -> Map Local Operator Tree: $hdt$_0:$hdt$_0:$hdt$_1:$hdt$_1:$hdt$_2:y1 TableScan alias: y1 Statistics: Num rows: 25 Data size: 191 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: key is not null (type: boolean) Statistics: Num rows: 13 Data size: 99 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: string) outputColumnNames: _col0 Statistics: Num rows: 13 Data size: 99 Basic stats: COMPLETE Column stats: NONE HashTable Sink Operator keys: 0 _col0 (type: string) 1 _col0 (type: string) Stage: Stage-8 Map Reduce Map Operator Tree: TableScan alias: x1 Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: key is not null (type: boolean) Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: string) outputColumnNames: _col0 Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE Map Join Operator condition map: Inner Join 0 to 1 keys: 0 _col0 (type: string) 1 _col0 (type: string) outputColumnNames: _col0 Statistics: Num rows: 275 Data size: 2921 Basic stats: COMPLETE Column stats: NONE Group By Operator keys: _col0 (type: string) mode: hash outputColumnNames: _col0 Statistics: Num rows: 275 Data size: 2921 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: string) sort order: + Map-reduce partition columns: _col0 (type: string) Statistics: Num rows: 275 Data size: 2921 Basic stats: COMPLETE Column stats: NONE Local Work: Map Reduce Local Work Reduce Operator Tree: Group By Operator keys: KEY._col0 (type: string) mode: mergepartial outputColumnNames: _col0 Statistics: Num rows: 137 Data size: 1455 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false table: input format: org.apache.hadoop.mapred.SequenceFileInputFormat output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe Stage: Stage-0 Fetch Operator limit: -1 Processor Tree: ListSink PREHOOK: query: SELECT tmp1.key as key, count(*) as cnt FROM (SELECT x1.key AS key FROM src x1 JOIN src1 y1 ON (x1.key = y1.key) GROUP BY x1.key) tmp1 JOIN (SELECT x2.key AS key FROM src x2 JOIN src1 y2 ON (x2.key = y2.key) GROUP BY x2.key) tmp2 ON (tmp1.key = tmp2.key) GROUP BY tmp1.key ORDER BY key, cnt PREHOOK: type: QUERY PREHOOK: Input: default@src PREHOOK: Input: default@src1 #### A masked pattern was here #### POSTHOOK: query: SELECT tmp1.key as key, count(*) as cnt FROM (SELECT x1.key AS key FROM src x1 JOIN src1 y1 ON (x1.key = y1.key) GROUP BY x1.key) tmp1 JOIN (SELECT x2.key AS key FROM src x2 JOIN src1 y2 ON (x2.key = y2.key) GROUP BY x2.key) tmp2 ON (tmp1.key = tmp2.key) GROUP BY tmp1.key ORDER BY key, cnt POSTHOOK: type: QUERY POSTHOOK: Input: default@src POSTHOOK: Input: default@src1 #### A masked pattern was here #### RUN: Stage-17:MAPREDLOCAL RUN: Stage-18:MAPREDLOCAL RUN: Stage-2:MAPRED RUN: Stage-8:MAPRED RUN: Stage-12:CONDITIONAL RUN: Stage-15:MAPREDLOCAL RUN: Stage-10:MAPRED RUN: Stage-4:MAPRED RUN: Stage-5:MAPRED 128 1 146 1 150 1 213 1 224 1 238 1 255 1 273 1 278 1 311 1 369 1 401 1 406 1 66 1 98 1 PREHOOK: query: -- When Correlation Optimizer is enabled, -- we will use two jobs. This first MR job will evaluate sub-queries of tmp1, tmp2, -- the Join of tmp1 and tmp2, and the aggregation on the result of the Join of -- tmp1 and tmp2. The second job will do the ORDER BY. EXPLAIN SELECT tmp1.key as key, count(*) as cnt FROM (SELECT x1.key AS key FROM src x1 JOIN src1 y1 ON (x1.key = y1.key) GROUP BY x1.key) tmp1 JOIN (SELECT x2.key AS key FROM src x2 JOIN src1 y2 ON (x2.key = y2.key) GROUP BY x2.key) tmp2 ON (tmp1.key = tmp2.key) GROUP BY tmp1.key ORDER BY key, cnt PREHOOK: type: QUERY POSTHOOK: query: -- When Correlation Optimizer is enabled, -- we will use two jobs. This first MR job will evaluate sub-queries of tmp1, tmp2, -- the Join of tmp1 and tmp2, and the aggregation on the result of the Join of -- tmp1 and tmp2. The second job will do the ORDER BY. EXPLAIN SELECT tmp1.key as key, count(*) as cnt FROM (SELECT x1.key AS key FROM src x1 JOIN src1 y1 ON (x1.key = y1.key) GROUP BY x1.key) tmp1 JOIN (SELECT x2.key AS key FROM src x2 JOIN src1 y2 ON (x2.key = y2.key) GROUP BY x2.key) tmp2 ON (tmp1.key = tmp2.key) GROUP BY tmp1.key ORDER BY key, cnt POSTHOOK: type: QUERY STAGE DEPENDENCIES: Stage-9 is a root stage Stage-2 depends on stages: Stage-9 Stage-3 depends on stages: Stage-2 Stage-0 depends on stages: Stage-3 STAGE PLANS: Stage: Stage-9 Map Reduce Local Work Alias -> Map Local Tables: $hdt$_0:$hdt$_0:$hdt$_0:$hdt$_0:$hdt$_1:y1 Fetch Operator limit: -1 $hdt$_0:$hdt$_0:$hdt$_1:$hdt$_1:$hdt$_2:y1 Fetch Operator limit: -1 Alias -> Map Local Operator Tree: $hdt$_0:$hdt$_0:$hdt$_0:$hdt$_0:$hdt$_1:y1 TableScan alias: y1 Statistics: Num rows: 25 Data size: 191 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: key is not null (type: boolean) Statistics: Num rows: 13 Data size: 99 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: string) outputColumnNames: _col0 Statistics: Num rows: 13 Data size: 99 Basic stats: COMPLETE Column stats: NONE HashTable Sink Operator keys: 0 _col0 (type: string) 1 _col0 (type: string) $hdt$_0:$hdt$_0:$hdt$_1:$hdt$_1:$hdt$_2:y1 TableScan alias: y1 Statistics: Num rows: 25 Data size: 191 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: key is not null (type: boolean) Statistics: Num rows: 13 Data size: 99 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: string) outputColumnNames: _col0 Statistics: Num rows: 13 Data size: 99 Basic stats: COMPLETE Column stats: NONE HashTable Sink Operator keys: 0 _col0 (type: string) 1 _col0 (type: string) Stage: Stage-2 Map Reduce Map Operator Tree: TableScan alias: x1 Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: key is not null (type: boolean) Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: string) outputColumnNames: _col0 Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE Map Join Operator condition map: Inner Join 0 to 1 keys: 0 _col0 (type: string) 1 _col0 (type: string) outputColumnNames: _col0 Statistics: Num rows: 275 Data size: 2921 Basic stats: COMPLETE Column stats: NONE Group By Operator keys: _col0 (type: string) mode: hash outputColumnNames: _col0 Statistics: Num rows: 275 Data size: 2921 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: string) sort order: + Map-reduce partition columns: _col0 (type: string) Statistics: Num rows: 275 Data size: 2921 Basic stats: COMPLETE Column stats: NONE TableScan alias: x1 Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: key is not null (type: boolean) Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: string) outputColumnNames: _col0 Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE Map Join Operator condition map: Inner Join 0 to 1 keys: 0 _col0 (type: string) 1 _col0 (type: string) outputColumnNames: _col0 Statistics: Num rows: 275 Data size: 2921 Basic stats: COMPLETE Column stats: NONE Group By Operator keys: _col0 (type: string) mode: hash outputColumnNames: _col0 Statistics: Num rows: 275 Data size: 2921 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: string) sort order: + Map-reduce partition columns: _col0 (type: string) Statistics: Num rows: 275 Data size: 2921 Basic stats: COMPLETE Column stats: NONE Local Work: Map Reduce Local Work Reduce Operator Tree: Demux Operator Statistics: Num rows: 550 Data size: 5842 Basic stats: COMPLETE Column stats: NONE Group By Operator keys: KEY._col0 (type: string) mode: mergepartial outputColumnNames: _col0 Statistics: Num rows: 275 Data size: 2921 Basic stats: COMPLETE Column stats: NONE Mux Operator Statistics: Num rows: 550 Data size: 5842 Basic stats: COMPLETE Column stats: NONE Join Operator condition map: Inner Join 0 to 1 keys: 0 _col0 (type: string) 1 _col0 (type: string) outputColumnNames: _col0 Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE Mux Operator Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE Group By Operator aggregations: count() keys: _col0 (type: string) mode: complete outputColumnNames: _col0, _col1 Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE File Output Operator compressed: false table: input format: org.apache.hadoop.mapred.SequenceFileInputFormat output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe Group By Operator keys: KEY._col0 (type: string) mode: mergepartial outputColumnNames: _col0 Statistics: Num rows: 275 Data size: 2921 Basic stats: COMPLETE Column stats: NONE Mux Operator Statistics: Num rows: 550 Data size: 5842 Basic stats: COMPLETE Column stats: NONE Join Operator condition map: Inner Join 0 to 1 keys: 0 _col0 (type: string) 1 _col0 (type: string) outputColumnNames: _col0 Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE Mux Operator Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE Group By Operator aggregations: count() keys: _col0 (type: string) mode: complete outputColumnNames: _col0, _col1 Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE File Output Operator compressed: false table: input format: org.apache.hadoop.mapred.SequenceFileInputFormat output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe Stage: Stage-3 Map Reduce Map Operator Tree: TableScan Reduce Output Operator key expressions: _col0 (type: string), _col1 (type: bigint) sort order: ++ Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE Reduce Operator Tree: Select Operator expressions: KEY.reducesinkkey0 (type: string), KEY.reducesinkkey1 (type: bigint) outputColumnNames: _col0, _col1 Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE File Output Operator compressed: false Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Stage: Stage-0 Fetch Operator limit: -1 Processor Tree: ListSink PREHOOK: query: SELECT tmp1.key as key, count(*) as cnt FROM (SELECT x1.key AS key FROM src x1 JOIN src1 y1 ON (x1.key = y1.key) GROUP BY x1.key) tmp1 JOIN (SELECT x2.key AS key FROM src x2 JOIN src1 y2 ON (x2.key = y2.key) GROUP BY x2.key) tmp2 ON (tmp1.key = tmp2.key) GROUP BY tmp1.key ORDER BY key, cnt PREHOOK: type: QUERY PREHOOK: Input: default@src PREHOOK: Input: default@src1 #### A masked pattern was here #### POSTHOOK: query: SELECT tmp1.key as key, count(*) as cnt FROM (SELECT x1.key AS key FROM src x1 JOIN src1 y1 ON (x1.key = y1.key) GROUP BY x1.key) tmp1 JOIN (SELECT x2.key AS key FROM src x2 JOIN src1 y2 ON (x2.key = y2.key) GROUP BY x2.key) tmp2 ON (tmp1.key = tmp2.key) GROUP BY tmp1.key ORDER BY key, cnt POSTHOOK: type: QUERY POSTHOOK: Input: default@src POSTHOOK: Input: default@src1 #### A masked pattern was here #### RUN: Stage-9:MAPREDLOCAL RUN: Stage-2:MAPRED RUN: Stage-3:MAPRED 128 1 146 1 150 1 213 1 224 1 238 1 255 1 273 1 278 1 311 1 369 1 401 1 406 1 66 1 98 1 PREHOOK: query: -- When Correlation Optimizer is disabled, -- we will use five jobs. -- We will generate one MR job to evaluate the sub-query tmp1, -- one MR job to evaluate the sub-query tmp2, -- one MR job for the Join of tmp1 and tmp2, -- one MR job for aggregation on the result of the Join of tmp1 and tmp2, -- and one MR job for the ORDER BY. EXPLAIN SELECT tmp1.key as key, count(*) as cnt FROM (SELECT x1.key AS key FROM src1 x1 GROUP BY x1.key) tmp1 JOIN (SELECT x2.key AS key FROM src x2 JOIN src1 y2 ON (x2.key = y2.key) GROUP BY x2.key) tmp2 ON (tmp1.key = tmp2.key) GROUP BY tmp1.key ORDER BY key, cnt PREHOOK: type: QUERY POSTHOOK: query: -- When Correlation Optimizer is disabled, -- we will use five jobs. -- We will generate one MR job to evaluate the sub-query tmp1, -- one MR job to evaluate the sub-query tmp2, -- one MR job for the Join of tmp1 and tmp2, -- one MR job for aggregation on the result of the Join of tmp1 and tmp2, -- and one MR job for the ORDER BY. EXPLAIN SELECT tmp1.key as key, count(*) as cnt FROM (SELECT x1.key AS key FROM src1 x1 GROUP BY x1.key) tmp1 JOIN (SELECT x2.key AS key FROM src x2 JOIN src1 y2 ON (x2.key = y2.key) GROUP BY x2.key) tmp2 ON (tmp1.key = tmp2.key) GROUP BY tmp1.key ORDER BY key, cnt POSTHOOK: type: QUERY STAGE DEPENDENCIES: Stage-7 is a root stage Stage-10 depends on stages: Stage-2, Stage-7 , consists of Stage-12, Stage-13, Stage-3 Stage-12 has a backup stage: Stage-3 Stage-8 depends on stages: Stage-12 Stage-4 depends on stages: Stage-3, Stage-8, Stage-9 Stage-5 depends on stages: Stage-4 Stage-13 has a backup stage: Stage-3 Stage-9 depends on stages: Stage-13 Stage-3 Stage-14 is a root stage Stage-2 depends on stages: Stage-14 Stage-0 depends on stages: Stage-5 STAGE PLANS: Stage: Stage-7 Map Reduce Map Operator Tree: TableScan alias: x1 Statistics: Num rows: 25 Data size: 191 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: key is not null (type: boolean) Statistics: Num rows: 13 Data size: 99 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: string) outputColumnNames: _col0 Statistics: Num rows: 13 Data size: 99 Basic stats: COMPLETE Column stats: NONE Group By Operator keys: _col0 (type: string) mode: hash outputColumnNames: _col0 Statistics: Num rows: 13 Data size: 99 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: string) sort order: + Map-reduce partition columns: _col0 (type: string) Statistics: Num rows: 13 Data size: 99 Basic stats: COMPLETE Column stats: NONE Reduce Operator Tree: Group By Operator keys: KEY._col0 (type: string) mode: mergepartial outputColumnNames: _col0 Statistics: Num rows: 6 Data size: 45 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false table: input format: org.apache.hadoop.mapred.SequenceFileInputFormat output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe Stage: Stage-10 Conditional Operator Stage: Stage-12 Map Reduce Local Work Alias -> Map Local Tables: $hdt$_0:$hdt$_0:$INTNAME1 Fetch Operator limit: -1 Alias -> Map Local Operator Tree: $hdt$_0:$hdt$_0:$INTNAME1 TableScan HashTable Sink Operator keys: 0 _col0 (type: string) 1 _col0 (type: string) Stage: Stage-8 Map Reduce Map Operator Tree: TableScan Map Join Operator condition map: Inner Join 0 to 1 keys: 0 _col0 (type: string) 1 _col0 (type: string) outputColumnNames: _col1 Select Operator expressions: _col1 (type: string) outputColumnNames: _col0 Group By Operator aggregations: count() keys: _col0 (type: string) mode: hash outputColumnNames: _col0, _col1 File Output Operator compressed: false table: input format: org.apache.hadoop.mapred.SequenceFileInputFormat output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe Local Work: Map Reduce Local Work Stage: Stage-4 Map Reduce Map Operator Tree: TableScan Reduce Output Operator key expressions: _col0 (type: string) sort order: + Map-reduce partition columns: _col0 (type: string) Statistics: Num rows: 150 Data size: 1600 Basic stats: COMPLETE Column stats: NONE value expressions: _col1 (type: bigint) Reduce Operator Tree: Group By Operator aggregations: count(VALUE._col0) keys: KEY._col0 (type: string) mode: mergepartial outputColumnNames: _col0, _col1 Statistics: Num rows: 75 Data size: 800 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false table: input format: org.apache.hadoop.mapred.SequenceFileInputFormat output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe Stage: Stage-5 Map Reduce Map Operator Tree: TableScan Reduce Output Operator key expressions: _col0 (type: string), _col1 (type: bigint) sort order: ++ Statistics: Num rows: 75 Data size: 800 Basic stats: COMPLETE Column stats: NONE Reduce Operator Tree: Select Operator expressions: KEY.reducesinkkey0 (type: string), KEY.reducesinkkey1 (type: bigint) outputColumnNames: _col0, _col1 Statistics: Num rows: 75 Data size: 800 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false Statistics: Num rows: 75 Data size: 800 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Stage: Stage-13 Map Reduce Local Work Alias -> Map Local Tables: $hdt$_0:$hdt$_0:$INTNAME Fetch Operator limit: -1 Alias -> Map Local Operator Tree: $hdt$_0:$hdt$_0:$INTNAME TableScan HashTable Sink Operator keys: 0 _col0 (type: string) 1 _col0 (type: string) Stage: Stage-9 Map Reduce Map Operator Tree: TableScan Map Join Operator condition map: Inner Join 0 to 1 keys: 0 _col0 (type: string) 1 _col0 (type: string) outputColumnNames: _col1 Select Operator expressions: _col1 (type: string) outputColumnNames: _col0 Group By Operator aggregations: count() keys: _col0 (type: string) mode: hash outputColumnNames: _col0, _col1 File Output Operator compressed: false table: input format: org.apache.hadoop.mapred.SequenceFileInputFormat output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe Local Work: Map Reduce Local Work Stage: Stage-3 Map Reduce Map Operator Tree: TableScan Reduce Output Operator key expressions: _col0 (type: string) sort order: + Map-reduce partition columns: _col0 (type: string) Statistics: Num rows: 137 Data size: 1455 Basic stats: COMPLETE Column stats: NONE TableScan Reduce Output Operator key expressions: _col0 (type: string) sort order: + Map-reduce partition columns: _col0 (type: string) Statistics: Num rows: 6 Data size: 45 Basic stats: COMPLETE Column stats: NONE Reduce Operator Tree: Join Operator condition map: Inner Join 0 to 1 keys: 0 _col0 (type: string) 1 _col0 (type: string) outputColumnNames: _col1 Statistics: Num rows: 150 Data size: 1600 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col1 (type: string) outputColumnNames: _col0 Statistics: Num rows: 150 Data size: 1600 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: count() keys: _col0 (type: string) mode: hash outputColumnNames: _col0, _col1 Statistics: Num rows: 150 Data size: 1600 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false table: input format: org.apache.hadoop.mapred.SequenceFileInputFormat output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe Stage: Stage-14 Map Reduce Local Work Alias -> Map Local Tables: $hdt$_0:$hdt$_0:$hdt$_0:$hdt$_0:$hdt$_1:x1 Fetch Operator limit: -1 Alias -> Map Local Operator Tree: $hdt$_0:$hdt$_0:$hdt$_0:$hdt$_0:$hdt$_1:x1 TableScan alias: x1 Statistics: Num rows: 25 Data size: 191 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: key is not null (type: boolean) Statistics: Num rows: 13 Data size: 99 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: string) outputColumnNames: _col0 Statistics: Num rows: 13 Data size: 99 Basic stats: COMPLETE Column stats: NONE HashTable Sink Operator keys: 0 _col0 (type: string) 1 _col0 (type: string) Stage: Stage-2 Map Reduce Map Operator Tree: TableScan alias: x2 Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: key is not null (type: boolean) Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: string) outputColumnNames: _col0 Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE Map Join Operator condition map: Inner Join 0 to 1 keys: 0 _col0 (type: string) 1 _col0 (type: string) outputColumnNames: _col0 Statistics: Num rows: 275 Data size: 2921 Basic stats: COMPLETE Column stats: NONE Group By Operator keys: _col0 (type: string) mode: hash outputColumnNames: _col0 Statistics: Num rows: 275 Data size: 2921 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: string) sort order: + Map-reduce partition columns: _col0 (type: string) Statistics: Num rows: 275 Data size: 2921 Basic stats: COMPLETE Column stats: NONE Local Work: Map Reduce Local Work Reduce Operator Tree: Group By Operator keys: KEY._col0 (type: string) mode: mergepartial outputColumnNames: _col0 Statistics: Num rows: 137 Data size: 1455 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false table: input format: org.apache.hadoop.mapred.SequenceFileInputFormat output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe Stage: Stage-0 Fetch Operator limit: -1 Processor Tree: ListSink PREHOOK: query: SELECT tmp1.key as key, count(*) as cnt FROM (SELECT x1.key AS key FROM src1 x1 GROUP BY x1.key) tmp1 JOIN (SELECT x2.key AS key FROM src x2 JOIN src1 y2 ON (x2.key = y2.key) GROUP BY x2.key) tmp2 ON (tmp1.key = tmp2.key) GROUP BY tmp1.key ORDER BY key, cnt PREHOOK: type: QUERY PREHOOK: Input: default@src PREHOOK: Input: default@src1 #### A masked pattern was here #### POSTHOOK: query: SELECT tmp1.key as key, count(*) as cnt FROM (SELECT x1.key AS key FROM src1 x1 GROUP BY x1.key) tmp1 JOIN (SELECT x2.key AS key FROM src x2 JOIN src1 y2 ON (x2.key = y2.key) GROUP BY x2.key) tmp2 ON (tmp1.key = tmp2.key) GROUP BY tmp1.key ORDER BY key, cnt POSTHOOK: type: QUERY POSTHOOK: Input: default@src POSTHOOK: Input: default@src1 #### A masked pattern was here #### RUN: Stage-7:MAPRED RUN: Stage-14:MAPREDLOCAL RUN: Stage-2:MAPRED RUN: Stage-10:CONDITIONAL RUN: Stage-13:MAPREDLOCAL RUN: Stage-9:MAPRED RUN: Stage-4:MAPRED RUN: Stage-5:MAPRED 128 1 146 1 150 1 213 1 224 1 238 1 255 1 273 1 278 1 311 1 369 1 401 1 406 1 66 1 98 1 PREHOOK: query: -- When Correlation Optimizer is enabled, -- we will use two job. This first MR job will evaluate sub-queries of tmp1, tmp2, -- the Join of tmp1 and tmp2, and the aggregation on the result of the Join of -- tmp1 and tmp2. The second job will do the ORDER BY. EXPLAIN SELECT tmp1.key as key, count(*) as cnt FROM (SELECT x1.key AS key FROM src1 x1 GROUP BY x1.key) tmp1 JOIN (SELECT x2.key AS key FROM src x2 JOIN src1 y2 ON (x2.key = y2.key) GROUP BY x2.key) tmp2 ON (tmp1.key = tmp2.key) GROUP BY tmp1.key ORDER BY key, cnt PREHOOK: type: QUERY POSTHOOK: query: -- When Correlation Optimizer is enabled, -- we will use two job. This first MR job will evaluate sub-queries of tmp1, tmp2, -- the Join of tmp1 and tmp2, and the aggregation on the result of the Join of -- tmp1 and tmp2. The second job will do the ORDER BY. EXPLAIN SELECT tmp1.key as key, count(*) as cnt FROM (SELECT x1.key AS key FROM src1 x1 GROUP BY x1.key) tmp1 JOIN (SELECT x2.key AS key FROM src x2 JOIN src1 y2 ON (x2.key = y2.key) GROUP BY x2.key) tmp2 ON (tmp1.key = tmp2.key) GROUP BY tmp1.key ORDER BY key, cnt POSTHOOK: type: QUERY STAGE DEPENDENCIES: Stage-7 is a root stage Stage-2 depends on stages: Stage-7 Stage-3 depends on stages: Stage-2 Stage-0 depends on stages: Stage-3 STAGE PLANS: Stage: Stage-7 Map Reduce Local Work Alias -> Map Local Tables: $hdt$_0:$hdt$_0:$hdt$_0:$hdt$_0:$hdt$_1:x1 Fetch Operator limit: -1 Alias -> Map Local Operator Tree: $hdt$_0:$hdt$_0:$hdt$_0:$hdt$_0:$hdt$_1:x1 TableScan alias: x1 Statistics: Num rows: 25 Data size: 191 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: key is not null (type: boolean) Statistics: Num rows: 13 Data size: 99 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: string) outputColumnNames: _col0 Statistics: Num rows: 13 Data size: 99 Basic stats: COMPLETE Column stats: NONE HashTable Sink Operator keys: 0 _col0 (type: string) 1 _col0 (type: string) Stage: Stage-2 Map Reduce Map Operator Tree: TableScan alias: x1 Statistics: Num rows: 25 Data size: 191 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: key is not null (type: boolean) Statistics: Num rows: 13 Data size: 99 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: string) outputColumnNames: _col0 Statistics: Num rows: 13 Data size: 99 Basic stats: COMPLETE Column stats: NONE Group By Operator keys: _col0 (type: string) mode: hash outputColumnNames: _col0 Statistics: Num rows: 13 Data size: 99 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: string) sort order: + Map-reduce partition columns: _col0 (type: string) Statistics: Num rows: 13 Data size: 99 Basic stats: COMPLETE Column stats: NONE TableScan alias: x2 Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: key is not null (type: boolean) Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: string) outputColumnNames: _col0 Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE Map Join Operator condition map: Inner Join 0 to 1 keys: 0 _col0 (type: string) 1 _col0 (type: string) outputColumnNames: _col0 Statistics: Num rows: 275 Data size: 2921 Basic stats: COMPLETE Column stats: NONE Group By Operator keys: _col0 (type: string) mode: hash outputColumnNames: _col0 Statistics: Num rows: 275 Data size: 2921 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: string) sort order: + Map-reduce partition columns: _col0 (type: string) Statistics: Num rows: 275 Data size: 2921 Basic stats: COMPLETE Column stats: NONE Local Work: Map Reduce Local Work Reduce Operator Tree: Demux Operator Statistics: Num rows: 288 Data size: 3020 Basic stats: COMPLETE Column stats: NONE Group By Operator keys: KEY._col0 (type: string) mode: mergepartial outputColumnNames: _col0 Statistics: Num rows: 144 Data size: 1510 Basic stats: COMPLETE Column stats: NONE Mux Operator Statistics: Num rows: 288 Data size: 3020 Basic stats: COMPLETE Column stats: NONE Join Operator condition map: Inner Join 0 to 1 keys: 0 _col0 (type: string) 1 _col0 (type: string) outputColumnNames: _col1 Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE Select Operator expressions: _col1 (type: string) outputColumnNames: _col0 Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE Mux Operator Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE Group By Operator aggregations: count() keys: _col0 (type: string) mode: complete outputColumnNames: _col0, _col1 Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE File Output Operator compressed: false table: input format: org.apache.hadoop.mapred.SequenceFileInputFormat output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe Group By Operator keys: KEY._col0 (type: string) mode: mergepartial outputColumnNames: _col0 Statistics: Num rows: 144 Data size: 1510 Basic stats: COMPLETE Column stats: NONE Mux Operator Statistics: Num rows: 288 Data size: 3020 Basic stats: COMPLETE Column stats: NONE Join Operator condition map: Inner Join 0 to 1 keys: 0 _col0 (type: string) 1 _col0 (type: string) outputColumnNames: _col1 Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE Select Operator expressions: _col1 (type: string) outputColumnNames: _col0 Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE Mux Operator Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE Group By Operator aggregations: count() keys: _col0 (type: string) mode: complete outputColumnNames: _col0, _col1 Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE File Output Operator compressed: false table: input format: org.apache.hadoop.mapred.SequenceFileInputFormat output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe Stage: Stage-3 Map Reduce Map Operator Tree: TableScan Reduce Output Operator key expressions: _col0 (type: string), _col1 (type: bigint) sort order: ++ Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE Reduce Operator Tree: Select Operator expressions: KEY.reducesinkkey0 (type: string), KEY.reducesinkkey1 (type: bigint) outputColumnNames: _col0, _col1 Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE File Output Operator compressed: false Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Stage: Stage-0 Fetch Operator limit: -1 Processor Tree: ListSink PREHOOK: query: SELECT tmp1.key as key, count(*) as cnt FROM (SELECT x1.key AS key FROM src1 x1 GROUP BY x1.key) tmp1 JOIN (SELECT x2.key AS key FROM src x2 JOIN src1 y2 ON (x2.key = y2.key) GROUP BY x2.key) tmp2 ON (tmp1.key = tmp2.key) GROUP BY tmp1.key ORDER BY key, cnt PREHOOK: type: QUERY PREHOOK: Input: default@src PREHOOK: Input: default@src1 #### A masked pattern was here #### POSTHOOK: query: SELECT tmp1.key as key, count(*) as cnt FROM (SELECT x1.key AS key FROM src1 x1 GROUP BY x1.key) tmp1 JOIN (SELECT x2.key AS key FROM src x2 JOIN src1 y2 ON (x2.key = y2.key) GROUP BY x2.key) tmp2 ON (tmp1.key = tmp2.key) GROUP BY tmp1.key ORDER BY key, cnt POSTHOOK: type: QUERY POSTHOOK: Input: default@src POSTHOOK: Input: default@src1 #### A masked pattern was here #### RUN: Stage-7:MAPREDLOCAL RUN: Stage-2:MAPRED RUN: Stage-3:MAPRED 128 1 146 1 150 1 213 1 224 1 238 1 255 1 273 1 278 1 311 1 369 1 401 1 406 1 66 1 98 1 PREHOOK: query: -- Check if we can correctly handle partitioned table. CREATE TABLE part_table(key string, value string) PARTITIONED BY (partitionId int) PREHOOK: type: CREATETABLE PREHOOK: Output: database:default PREHOOK: Output: default@part_table POSTHOOK: query: -- Check if we can correctly handle partitioned table. CREATE TABLE part_table(key string, value string) PARTITIONED BY (partitionId int) POSTHOOK: type: CREATETABLE POSTHOOK: Output: database:default POSTHOOK: Output: default@part_table RUN: Stage-0:DDL PREHOOK: query: INSERT OVERWRITE TABLE part_table PARTITION (partitionId=1) SELECT key, value FROM src ORDER BY key, value LIMIT 100 PREHOOK: type: QUERY PREHOOK: Input: default@src PREHOOK: Output: default@part_table@partitionid=1 POSTHOOK: query: INSERT OVERWRITE TABLE part_table PARTITION (partitionId=1) SELECT key, value FROM src ORDER BY key, value LIMIT 100 POSTHOOK: type: QUERY POSTHOOK: Input: default@src POSTHOOK: Output: default@part_table@partitionid=1 POSTHOOK: Lineage: part_table PARTITION(partitionid=1).key SIMPLE [(src)src.FieldSchema(name:key, type:string, comment:default), ] POSTHOOK: Lineage: part_table PARTITION(partitionid=1).value SIMPLE [(src)src.FieldSchema(name:value, type:string, comment:default), ] RUN: Stage-1:MAPRED RUN: Stage-0:MOVE RUN: Stage-2:STATS PREHOOK: query: INSERT OVERWRITE TABLE part_table PARTITION (partitionId=2) SELECT key, value FROM src1 ORDER BY key, value PREHOOK: type: QUERY PREHOOK: Input: default@src1 PREHOOK: Output: default@part_table@partitionid=2 POSTHOOK: query: INSERT OVERWRITE TABLE part_table PARTITION (partitionId=2) SELECT key, value FROM src1 ORDER BY key, value POSTHOOK: type: QUERY POSTHOOK: Input: default@src1 POSTHOOK: Output: default@part_table@partitionid=2 POSTHOOK: Lineage: part_table PARTITION(partitionid=2).key SIMPLE [(src1)src1.FieldSchema(name:key, type:string, comment:default), ] POSTHOOK: Lineage: part_table PARTITION(partitionid=2).value SIMPLE [(src1)src1.FieldSchema(name:value, type:string, comment:default), ] RUN: Stage-1:MAPRED RUN: Stage-0:MOVE RUN: Stage-2:STATS PREHOOK: query: EXPLAIN SELECT count(*) FROM part_table x JOIN src1 y ON (x.key = y.key) PREHOOK: type: QUERY POSTHOOK: query: EXPLAIN SELECT count(*) FROM part_table x JOIN src1 y ON (x.key = y.key) POSTHOOK: type: QUERY STAGE DEPENDENCIES: Stage-5 is a root stage Stage-2 depends on stages: Stage-5 Stage-0 depends on stages: Stage-2 STAGE PLANS: Stage: Stage-5 Map Reduce Local Work Alias -> Map Local Tables: y Fetch Operator limit: -1 Alias -> Map Local Operator Tree: y TableScan alias: y Statistics: Num rows: 25 Data size: 191 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: key is not null (type: boolean) Statistics: Num rows: 13 Data size: 99 Basic stats: COMPLETE Column stats: NONE HashTable Sink Operator keys: 0 key (type: string) 1 key (type: string) Stage: Stage-2 Map Reduce Map Operator Tree: TableScan alias: x Statistics: Num rows: 125 Data size: 1261 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: key is not null (type: boolean) Statistics: Num rows: 63 Data size: 635 Basic stats: COMPLETE Column stats: NONE Map Join Operator condition map: Inner Join 0 to 1 keys: 0 key (type: string) 1 key (type: string) Statistics: Num rows: 69 Data size: 698 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: count() mode: hash outputColumnNames: _col0 Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator sort order: Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE value expressions: _col0 (type: bigint) Local Work: Map Reduce Local Work Reduce Operator Tree: Group By Operator aggregations: count(VALUE._col0) mode: mergepartial outputColumnNames: _col0 Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col0 (type: bigint) outputColumnNames: _col0 Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Stage: Stage-0 Fetch Operator limit: -1 Processor Tree: ListSink PREHOOK: query: SELECT count(*) FROM part_table x JOIN src1 y ON (x.key = y.key) PREHOOK: type: QUERY PREHOOK: Input: default@part_table PREHOOK: Input: default@part_table@partitionid=1 PREHOOK: Input: default@part_table@partitionid=2 PREHOOK: Input: default@src1 #### A masked pattern was here #### POSTHOOK: query: SELECT count(*) FROM part_table x JOIN src1 y ON (x.key = y.key) POSTHOOK: type: QUERY POSTHOOK: Input: default@part_table POSTHOOK: Input: default@part_table@partitionid=1 POSTHOOK: Input: default@part_table@partitionid=2 POSTHOOK: Input: default@src1 #### A masked pattern was here #### RUN: Stage-5:MAPREDLOCAL RUN: Stage-2:MAPRED 121 PREHOOK: query: -- HIVE-5891 Alias conflict when merging multiple mapjoin tasks into their common -- child mapred task EXPLAIN SELECT * FROM ( SELECT c.key FROM (SELECT a.key FROM src a JOIN src b ON a.key=b.key GROUP BY a.key) tmp JOIN src c ON tmp.key=c.key UNION ALL SELECT c.key FROM (SELECT a.key FROM src a JOIN src b ON a.key=b.key GROUP BY a.key) tmp JOIN src c ON tmp.key=c.key ) x PREHOOK: type: QUERY POSTHOOK: query: -- HIVE-5891 Alias conflict when merging multiple mapjoin tasks into their common -- child mapred task EXPLAIN SELECT * FROM ( SELECT c.key FROM (SELECT a.key FROM src a JOIN src b ON a.key=b.key GROUP BY a.key) tmp JOIN src c ON tmp.key=c.key UNION ALL SELECT c.key FROM (SELECT a.key FROM src a JOIN src b ON a.key=b.key GROUP BY a.key) tmp JOIN src c ON tmp.key=c.key ) x POSTHOOK: type: QUERY STAGE DEPENDENCIES: Stage-15 is a root stage Stage-4 depends on stages: Stage-15 Stage-14 depends on stages: Stage-4, Stage-8 Stage-2 depends on stages: Stage-14 Stage-16 is a root stage Stage-8 depends on stages: Stage-16 Stage-0 depends on stages: Stage-2 STAGE PLANS: Stage: Stage-15 Map Reduce Local Work Alias -> Map Local Tables: null-subquery1:$hdt$_0-subquery1:$hdt$_0:$hdt$_1:$hdt$_1:$hdt$_1:a Fetch Operator limit: -1 Alias -> Map Local Operator Tree: null-subquery1:$hdt$_0-subquery1:$hdt$_0:$hdt$_1:$hdt$_1:$hdt$_1:a TableScan alias: a Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: key is not null (type: boolean) Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: string) outputColumnNames: _col0 Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE HashTable Sink Operator keys: 0 _col0 (type: string) 1 _col0 (type: string) Stage: Stage-4 Map Reduce Map Operator Tree: TableScan alias: a Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: key is not null (type: boolean) Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: string) outputColumnNames: _col0 Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE Map Join Operator condition map: Inner Join 0 to 1 keys: 0 _col0 (type: string) 1 _col0 (type: string) outputColumnNames: _col0 Statistics: Num rows: 275 Data size: 2921 Basic stats: COMPLETE Column stats: NONE Group By Operator keys: _col0 (type: string) mode: hash outputColumnNames: _col0 Statistics: Num rows: 275 Data size: 2921 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: string) sort order: + Map-reduce partition columns: _col0 (type: string) Statistics: Num rows: 275 Data size: 2921 Basic stats: COMPLETE Column stats: NONE Local Work: Map Reduce Local Work Reduce Operator Tree: Group By Operator keys: KEY._col0 (type: string) mode: mergepartial outputColumnNames: _col0 Statistics: Num rows: 137 Data size: 1455 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false table: input format: org.apache.hadoop.mapred.SequenceFileInputFormat output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe Stage: Stage-14 Map Reduce Local Work Alias -> Map Local Tables: null-subquery1:$hdt$_0-subquery1:$hdt$_0:$hdt$_0:a Fetch Operator limit: -1 null-subquery2:$hdt$_0-subquery2:$hdt$_0:$hdt$_0:a Fetch Operator limit: -1 Alias -> Map Local Operator Tree: null-subquery1:$hdt$_0-subquery1:$hdt$_0:$hdt$_0:a TableScan alias: a Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: key is not null (type: boolean) Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: string) outputColumnNames: _col0 Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE HashTable Sink Operator keys: 0 _col0 (type: string) 1 _col0 (type: string) null-subquery2:$hdt$_0-subquery2:$hdt$_0:$hdt$_0:a TableScan alias: a Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: key is not null (type: boolean) Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: string) outputColumnNames: _col0 Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE HashTable Sink Operator keys: 0 _col0 (type: string) 1 _col0 (type: string) Stage: Stage-2 Map Reduce Map Operator Tree: TableScan Map Join Operator condition map: Inner Join 0 to 1 keys: 0 _col0 (type: string) 1 _col0 (type: string) outputColumnNames: _col0 Statistics: Num rows: 275 Data size: 2921 Basic stats: COMPLETE Column stats: NONE Union Statistics: Num rows: 550 Data size: 5842 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col0 (type: string) outputColumnNames: _col0 Statistics: Num rows: 550 Data size: 5842 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false Statistics: Num rows: 550 Data size: 5842 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe TableScan Map Join Operator condition map: Inner Join 0 to 1 keys: 0 _col0 (type: string) 1 _col0 (type: string) outputColumnNames: _col0 Statistics: Num rows: 275 Data size: 2921 Basic stats: COMPLETE Column stats: NONE Union Statistics: Num rows: 550 Data size: 5842 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col0 (type: string) outputColumnNames: _col0 Statistics: Num rows: 550 Data size: 5842 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false Statistics: Num rows: 550 Data size: 5842 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Local Work: Map Reduce Local Work Stage: Stage-16 Map Reduce Local Work Alias -> Map Local Tables: null-subquery2:$hdt$_0-subquery2:$hdt$_0:$hdt$_1:$hdt$_1:$hdt$_1:a Fetch Operator limit: -1 Alias -> Map Local Operator Tree: null-subquery2:$hdt$_0-subquery2:$hdt$_0:$hdt$_1:$hdt$_1:$hdt$_1:a TableScan alias: a Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: key is not null (type: boolean) Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: string) outputColumnNames: _col0 Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE HashTable Sink Operator keys: 0 _col0 (type: string) 1 _col0 (type: string) Stage: Stage-8 Map Reduce Map Operator Tree: TableScan alias: a Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: key is not null (type: boolean) Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: string) outputColumnNames: _col0 Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE Map Join Operator condition map: Inner Join 0 to 1 keys: 0 _col0 (type: string) 1 _col0 (type: string) outputColumnNames: _col0 Statistics: Num rows: 275 Data size: 2921 Basic stats: COMPLETE Column stats: NONE Group By Operator keys: _col0 (type: string) mode: hash outputColumnNames: _col0 Statistics: Num rows: 275 Data size: 2921 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: string) sort order: + Map-reduce partition columns: _col0 (type: string) Statistics: Num rows: 275 Data size: 2921 Basic stats: COMPLETE Column stats: NONE Local Work: Map Reduce Local Work Reduce Operator Tree: Group By Operator keys: KEY._col0 (type: string) mode: mergepartial outputColumnNames: _col0 Statistics: Num rows: 137 Data size: 1455 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false table: input format: org.apache.hadoop.mapred.SequenceFileInputFormat output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe Stage: Stage-0 Fetch Operator limit: -1 Processor Tree: ListSink PREHOOK: query: SELECT * FROM ( SELECT c.key FROM (SELECT a.key FROM src a JOIN src b ON a.key=b.key GROUP BY a.key) tmp JOIN src c ON tmp.key=c.key UNION ALL SELECT c.key FROM (SELECT a.key FROM src a JOIN src b ON a.key=b.key GROUP BY a.key) tmp JOIN src c ON tmp.key=c.key ) x PREHOOK: type: QUERY PREHOOK: Input: default@src #### A masked pattern was here #### POSTHOOK: query: SELECT * FROM ( SELECT c.key FROM (SELECT a.key FROM src a JOIN src b ON a.key=b.key GROUP BY a.key) tmp JOIN src c ON tmp.key=c.key UNION ALL SELECT c.key FROM (SELECT a.key FROM src a JOIN src b ON a.key=b.key GROUP BY a.key) tmp JOIN src c ON tmp.key=c.key ) x POSTHOOK: type: QUERY POSTHOOK: Input: default@src #### A masked pattern was here #### RUN: Stage-15:MAPREDLOCAL RUN: Stage-16:MAPREDLOCAL RUN: Stage-4:MAPRED RUN: Stage-8:MAPRED RUN: Stage-14:MAPREDLOCAL RUN: Stage-2:MAPRED 0 0 0 10 100 100 103 103 104 104 105 11 111 113 113 114 116 118 118 119 119 119 12 12 120 120 125 125 126 128 128 128 129 129 131 133 134 134 136 137 137 138 138 138 138 143 145 146 146 149 149 15 15 150 152 152 153 155 156 157 158 160 162 163 164 164 165 165 166 167 167 167 168 169 169 169 169 17 170 172 172 174 174 175 175 176 176 177 178 179 179 18 18 180 181 183 186 187 187 187 189 19 190 191 191 192 193 193 193 194 195 195 196 197 197 199 199 199 2 20 200 200 201 202 203 203 205 205 207 207 208 208 208 209 209 213 213 214 216 216 217 217 218 219 219 221 221 222 223 223 224 224 226 228 229 229 230 230 230 230 230 233 233 235 237 237 238 238 239 239 24 24 241 242 242 244 247 248 249 252 255 255 256 256 257 258 26 26 260 262 263 265 265 266 27 272 272 273 273 273 274 275 277 277 277 277 278 278 28 280 280 281 281 282 282 283 284 285 286 287 288 288 289 291 292 296 298 298 298 30 302 305 306 307 307 308 309 309 310 311 311 311 315 316 316 316 317 317 318 318 318 321 321 322 322 323 325 325 327 327 327 33 331 331 332 333 333 335 336 338 339 34 341 342 342 344 344 345 348 348 348 348 348 35 35 35 351 353 353 356 360 362 364 365 366 367 367 368 369 369 369 37 37 373 374 375 377 378 379 382 382 384 384 384 386 389 392 393 394 395 395 396 396 396 397 397 399 399 4 400 401 401 401 401 401 402 403 403 403 404 404 406 406 406 406 407 409 409 409 41 411 413 413 414 414 417 417 417 418 419 42 42 421 424 424 427 429 429 43 430 430 430 431 431 431 432 435 436 437 438 438 438 439 439 44 443 444 446 448 449 452 453 454 454 454 455 457 458 458 459 459 460 462 462 463 463 466 466 466 467 468 468 468 468 469 469 469 469 469 47 470 472 475 477 478 478 479 480 480 480 481 482 483 484 485 487 489 489 489 489 490 491 492 492 493 494 495 496 497 498 498 498 5 5 5 51 51 53 54 57 58 58 64 65 66 67 67 69 70 70 70 72 72 74 76 76 77 78 8 80 82 83 83 84 84 85 86 87 9 90 90 90 92 95 95 96 97 97 98 98 0 0 0 10 100 100 103 103 104 104 105 11 111 113 113 114 116 118 118 119 119 119 12 12 120 120 125 125 126 128 128 128 129 129 131 133 134 134 136 137 137 138 138 138 138 143 145 146 146 149 149 15 15 150 152 152 153 155 156 157 158 160 162 163 164 164 165 165 166 167 167 167 168 169 169 169 169 17 170 172 172 174 174 175 175 176 176 177 178 179 179 18 18 180 181 183 186 187 187 187 189 19 190 191 191 192 193 193 193 194 195 195 196 197 197 199 199 199 2 20 200 200 201 202 203 203 205 205 207 207 208 208 208 209 209 213 213 214 216 216 217 217 218 219 219 221 221 222 223 223 224 224 226 228 229 229 230 230 230 230 230 233 233 235 237 237 238 238 239 239 24 24 241 242 242 244 247 248 249 252 255 255 256 256 257 258 26 26 260 262 263 265 265 266 27 272 272 273 273 273 274 275 277 277 277 277 278 278 28 280 280 281 281 282 282 283 284 285 286 287 288 288 289 291 292 296 298 298 298 30 302 305 306 307 307 308 309 309 310 311 311 311 315 316 316 316 317 317 318 318 318 321 321 322 322 323 325 325 327 327 327 33 331 331 332 333 333 335 336 338 339 34 341 342 342 344 344 345 348 348 348 348 348 35 35 35 351 353 353 356 360 362 364 365 366 367 367 368 369 369 369 37 37 373 374 375 377 378 379 382 382 384 384 384 386 389 392 393 394 395 395 396 396 396 397 397 399 399 4 400 401 401 401 401 401 402 403 403 403 404 404 406 406 406 406 407 409 409 409 41 411 413 413 414 414 417 417 417 418 419 42 42 421 424 424 427 429 429 43 430 430 430 431 431 431 432 435 436 437 438 438 438 439 439 44 443 444 446 448 449 452 453 454 454 454 455 457 458 458 459 459 460 462 462 463 463 466 466 466 467 468 468 468 468 469 469 469 469 469 47 470 472 475 477 478 478 479 480 480 480 481 482 483 484 485 487 489 489 489 489 490 491 492 492 493 494 495 496 497 498 498 498 5 5 5 51 51 53 54 57 58 58 64 65 66 67 67 69 70 70 70 72 72 74 76 76 77 78 8 80 82 83 83 84 84 85 86 87 9 90 90 90 92 95 95 96 97 97 98 98