PREHOOK: query: create table if not exists loc_staging ( state string, locid int, zip bigint, year string ) row format delimited fields terminated by '|' stored as textfile PREHOOK: type: CREATETABLE PREHOOK: Output: database:default PREHOOK: Output: default@loc_staging POSTHOOK: query: create table if not exists loc_staging ( state string, locid int, zip bigint, year string ) row format delimited fields terminated by '|' stored as textfile POSTHOOK: type: CREATETABLE POSTHOOK: Output: database:default POSTHOOK: Output: default@loc_staging PREHOOK: query: LOAD DATA LOCAL INPATH '../../data/files/loc.txt' OVERWRITE INTO TABLE loc_staging PREHOOK: type: LOAD #### A masked pattern was here #### PREHOOK: Output: default@loc_staging POSTHOOK: query: LOAD DATA LOCAL INPATH '../../data/files/loc.txt' OVERWRITE INTO TABLE loc_staging POSTHOOK: type: LOAD #### A masked pattern was here #### POSTHOOK: Output: default@loc_staging PREHOOK: query: create table if not exists loc_orc ( state string, locid int, zip bigint ) partitioned by(year string) stored as orc PREHOOK: type: CREATETABLE PREHOOK: Output: database:default PREHOOK: Output: default@loc_orc POSTHOOK: query: create table if not exists loc_orc ( state string, locid int, zip bigint ) partitioned by(year string) stored as orc POSTHOOK: type: CREATETABLE POSTHOOK: Output: database:default POSTHOOK: Output: default@loc_orc PREHOOK: query: -- basicStatState: NONE colStatState: NONE explain select * from loc_orc PREHOOK: type: QUERY POSTHOOK: query: -- basicStatState: NONE colStatState: NONE explain select * from loc_orc POSTHOOK: type: QUERY STAGE DEPENDENCIES: Stage-0 is a root stage STAGE PLANS: Stage: Stage-0 Fetch Operator limit: -1 Processor Tree: TableScan alias: loc_orc Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: PARTIAL Select Operator expressions: state (type: string), locid (type: int), zip (type: bigint), year (type: string) outputColumnNames: _col0, _col1, _col2, _col3 Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: PARTIAL ListSink PREHOOK: query: insert overwrite table loc_orc partition(year) select * from loc_staging PREHOOK: type: QUERY PREHOOK: Input: default@loc_staging PREHOOK: Output: default@loc_orc POSTHOOK: query: insert overwrite table loc_orc partition(year) select * from loc_staging POSTHOOK: type: QUERY POSTHOOK: Input: default@loc_staging POSTHOOK: Output: default@loc_orc@year=2001 POSTHOOK: Output: default@loc_orc@year=__HIVE_DEFAULT_PARTITION__ POSTHOOK: Lineage: loc_orc PARTITION(year=2001).locid SIMPLE [(loc_staging)loc_staging.FieldSchema(name:locid, type:int, comment:null), ] POSTHOOK: Lineage: loc_orc PARTITION(year=2001).state SIMPLE [(loc_staging)loc_staging.FieldSchema(name:state, type:string, comment:null), ] POSTHOOK: Lineage: loc_orc PARTITION(year=2001).zip SIMPLE [(loc_staging)loc_staging.FieldSchema(name:zip, type:bigint, comment:null), ] POSTHOOK: Lineage: loc_orc PARTITION(year=__HIVE_DEFAULT_PARTITION__).locid SIMPLE [(loc_staging)loc_staging.FieldSchema(name:locid, type:int, comment:null), ] POSTHOOK: Lineage: loc_orc PARTITION(year=__HIVE_DEFAULT_PARTITION__).state SIMPLE [(loc_staging)loc_staging.FieldSchema(name:state, type:string, comment:null), ] POSTHOOK: Lineage: loc_orc PARTITION(year=__HIVE_DEFAULT_PARTITION__).zip SIMPLE [(loc_staging)loc_staging.FieldSchema(name:zip, type:bigint, comment:null), ] PREHOOK: query: -- stats are disabled. basic stats will report the file size but not raw data size. so initial statistics will be PARTIAL -- basicStatState: PARTIAL colStatState: NONE explain select * from loc_orc PREHOOK: type: QUERY POSTHOOK: query: -- stats are disabled. basic stats will report the file size but not raw data size. so initial statistics will be PARTIAL -- basicStatState: PARTIAL colStatState: NONE explain select * from loc_orc POSTHOOK: type: QUERY STAGE DEPENDENCIES: Stage-0 is a root stage STAGE PLANS: Stage: Stage-0 Fetch Operator limit: -1 Processor Tree: TableScan alias: loc_orc Statistics: Num rows: 6 Data size: 767 Basic stats: COMPLETE Column stats: PARTIAL Select Operator expressions: state (type: string), locid (type: int), zip (type: bigint), year (type: string) outputColumnNames: _col0, _col1, _col2, _col3 Statistics: Num rows: 6 Data size: 1104 Basic stats: COMPLETE Column stats: PARTIAL ListSink PREHOOK: query: -- partition level analyze statistics for specific parition analyze table loc_orc partition(year='2001') compute statistics PREHOOK: type: QUERY PREHOOK: Input: default@loc_orc PREHOOK: Input: default@loc_orc@year=2001 PREHOOK: Output: default@loc_orc PREHOOK: Output: default@loc_orc@year=2001 POSTHOOK: query: -- partition level analyze statistics for specific parition analyze table loc_orc partition(year='2001') compute statistics POSTHOOK: type: QUERY POSTHOOK: Input: default@loc_orc POSTHOOK: Input: default@loc_orc@year=2001 POSTHOOK: Output: default@loc_orc POSTHOOK: Output: default@loc_orc@year=2001 PREHOOK: query: -- basicStatState: PARTIAL colStatState: NONE explain select * from loc_orc where year='__HIVE_DEFAULT_PARTITION__' PREHOOK: type: QUERY POSTHOOK: query: -- basicStatState: PARTIAL colStatState: NONE explain select * from loc_orc where year='__HIVE_DEFAULT_PARTITION__' POSTHOOK: type: QUERY STAGE DEPENDENCIES: Stage-0 is a root stage STAGE PLANS: Stage: Stage-0 Fetch Operator limit: -1 Processor Tree: TableScan alias: loc_orc Statistics: Num rows: 3 Data size: 342 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: state (type: string), locid (type: int), zip (type: bigint), '__HIVE_DEFAULT_PARTITION__' (type: string) outputColumnNames: _col0, _col1, _col2, _col3 Statistics: Num rows: 3 Data size: 342 Basic stats: COMPLETE Column stats: NONE ListSink PREHOOK: query: -- basicStatState: PARTIAL colStatState: NONE explain select * from loc_orc PREHOOK: type: QUERY POSTHOOK: query: -- basicStatState: PARTIAL colStatState: NONE explain select * from loc_orc POSTHOOK: type: QUERY STAGE DEPENDENCIES: Stage-0 is a root stage STAGE PLANS: Stage: Stage-0 Fetch Operator limit: -1 Processor Tree: TableScan alias: loc_orc Statistics: Num rows: 10 Data size: 767 Basic stats: COMPLETE Column stats: PARTIAL Select Operator expressions: state (type: string), locid (type: int), zip (type: bigint), year (type: string) outputColumnNames: _col0, _col1, _col2, _col3 Statistics: Num rows: 10 Data size: 1840 Basic stats: COMPLETE Column stats: PARTIAL ListSink PREHOOK: query: -- basicStatState: COMPLETE colStatState: NONE explain select * from loc_orc where year='2001' PREHOOK: type: QUERY POSTHOOK: query: -- basicStatState: COMPLETE colStatState: NONE explain select * from loc_orc where year='2001' POSTHOOK: type: QUERY STAGE DEPENDENCIES: Stage-0 is a root stage STAGE PLANS: Stage: Stage-0 Fetch Operator limit: -1 Processor Tree: TableScan alias: loc_orc Statistics: Num rows: 7 Data size: 425 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: state (type: string), locid (type: int), zip (type: bigint), '2001' (type: string) outputColumnNames: _col0, _col1, _col2, _col3 Statistics: Num rows: 7 Data size: 425 Basic stats: COMPLETE Column stats: NONE ListSink PREHOOK: query: -- partition level analyze statistics for all partitions analyze table loc_orc partition(year) compute statistics PREHOOK: type: QUERY PREHOOK: Input: default@loc_orc PREHOOK: Input: default@loc_orc@year=2001 PREHOOK: Input: default@loc_orc@year=__HIVE_DEFAULT_PARTITION__ PREHOOK: Output: default@loc_orc PREHOOK: Output: default@loc_orc@year=2001 PREHOOK: Output: default@loc_orc@year=__HIVE_DEFAULT_PARTITION__ POSTHOOK: query: -- partition level analyze statistics for all partitions analyze table loc_orc partition(year) compute statistics POSTHOOK: type: QUERY POSTHOOK: Input: default@loc_orc POSTHOOK: Input: default@loc_orc@year=2001 POSTHOOK: Input: default@loc_orc@year=__HIVE_DEFAULT_PARTITION__ POSTHOOK: Output: default@loc_orc POSTHOOK: Output: default@loc_orc@year=2001 POSTHOOK: Output: default@loc_orc@year=__HIVE_DEFAULT_PARTITION__ PREHOOK: query: -- basicStatState: COMPLETE colStatState: NONE explain select * from loc_orc where year='__HIVE_DEFAULT_PARTITION__' PREHOOK: type: QUERY POSTHOOK: query: -- basicStatState: COMPLETE colStatState: NONE explain select * from loc_orc where year='__HIVE_DEFAULT_PARTITION__' POSTHOOK: type: QUERY STAGE DEPENDENCIES: Stage-0 is a root stage STAGE PLANS: Stage: Stage-0 Fetch Operator limit: -1 Processor Tree: TableScan alias: loc_orc Statistics: Num rows: 1 Data size: 342 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: state (type: string), locid (type: int), zip (type: bigint), '__HIVE_DEFAULT_PARTITION__' (type: string) outputColumnNames: _col0, _col1, _col2, _col3 Statistics: Num rows: 1 Data size: 342 Basic stats: COMPLETE Column stats: NONE ListSink PREHOOK: query: -- basicStatState: COMPLETE colStatState: NONE explain select * from loc_orc PREHOOK: type: QUERY POSTHOOK: query: -- basicStatState: COMPLETE colStatState: NONE explain select * from loc_orc POSTHOOK: type: QUERY STAGE DEPENDENCIES: Stage-0 is a root stage STAGE PLANS: Stage: Stage-0 Fetch Operator limit: -1 Processor Tree: TableScan alias: loc_orc Statistics: Num rows: 8 Data size: 767 Basic stats: COMPLETE Column stats: PARTIAL Select Operator expressions: state (type: string), locid (type: int), zip (type: bigint), year (type: string) outputColumnNames: _col0, _col1, _col2, _col3 Statistics: Num rows: 8 Data size: 1472 Basic stats: COMPLETE Column stats: PARTIAL ListSink PREHOOK: query: -- basicStatState: COMPLETE colStatState: NONE explain select * from loc_orc where year='2001' or year='__HIVE_DEFAULT_PARTITION__' PREHOOK: type: QUERY POSTHOOK: query: -- basicStatState: COMPLETE colStatState: NONE explain select * from loc_orc where year='2001' or year='__HIVE_DEFAULT_PARTITION__' POSTHOOK: type: QUERY STAGE DEPENDENCIES: Stage-0 is a root stage STAGE PLANS: Stage: Stage-0 Fetch Operator limit: -1 Processor Tree: TableScan alias: loc_orc Statistics: Num rows: 8 Data size: 767 Basic stats: COMPLETE Column stats: PARTIAL Select Operator expressions: state (type: string), locid (type: int), zip (type: bigint), year (type: string) outputColumnNames: _col0, _col1, _col2, _col3 Statistics: Num rows: 8 Data size: 1472 Basic stats: COMPLETE Column stats: PARTIAL ListSink PREHOOK: query: -- both partitions will be pruned -- basicStatState: NONE colStatState: NONE explain select * from loc_orc where year='2001' and year='__HIVE_DEFAULT_PARTITION__' PREHOOK: type: QUERY POSTHOOK: query: -- both partitions will be pruned -- basicStatState: NONE colStatState: NONE explain select * from loc_orc where year='2001' and year='__HIVE_DEFAULT_PARTITION__' POSTHOOK: type: QUERY STAGE DEPENDENCIES: Stage-0 is a root stage STAGE PLANS: Stage: Stage-0 Fetch Operator limit: -1 Processor Tree: TableScan alias: loc_orc Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: PARTIAL Filter Operator predicate: ((year = '2001') and (year = '__HIVE_DEFAULT_PARTITION__')) (type: boolean) Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: PARTIAL Select Operator expressions: state (type: string), locid (type: int), zip (type: bigint), '__HIVE_DEFAULT_PARTITION__' (type: string) outputColumnNames: _col0, _col1, _col2, _col3 Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: PARTIAL ListSink PREHOOK: query: -- partition level partial column statistics analyze table loc_orc partition(year='2001') compute statistics for columns state,locid PREHOOK: type: QUERY PREHOOK: Input: default@loc_orc PREHOOK: Input: default@loc_orc@year=2001 #### A masked pattern was here #### POSTHOOK: query: -- partition level partial column statistics analyze table loc_orc partition(year='2001') compute statistics for columns state,locid POSTHOOK: type: QUERY POSTHOOK: Input: default@loc_orc POSTHOOK: Input: default@loc_orc@year=2001 #### A masked pattern was here #### PREHOOK: query: -- basicStatState: COMPLETE colStatState: NONE explain select zip from loc_orc PREHOOK: type: QUERY POSTHOOK: query: -- basicStatState: COMPLETE colStatState: NONE explain select zip from loc_orc POSTHOOK: type: QUERY STAGE DEPENDENCIES: Stage-0 is a root stage STAGE PLANS: Stage: Stage-0 Fetch Operator limit: -1 Processor Tree: TableScan alias: loc_orc Statistics: Num rows: 8 Data size: 767 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: zip (type: bigint) outputColumnNames: _col0 Statistics: Num rows: 8 Data size: 767 Basic stats: COMPLETE Column stats: NONE ListSink PREHOOK: query: -- basicStatState: COMPLETE colStatState: PARTIAL explain select state from loc_orc PREHOOK: type: QUERY POSTHOOK: query: -- basicStatState: COMPLETE colStatState: PARTIAL explain select state from loc_orc POSTHOOK: type: QUERY STAGE DEPENDENCIES: Stage-0 is a root stage STAGE PLANS: Stage: Stage-0 Fetch Operator limit: -1 Processor Tree: TableScan alias: loc_orc Statistics: Num rows: 8 Data size: 767 Basic stats: COMPLETE Column stats: PARTIAL Select Operator expressions: state (type: string) outputColumnNames: _col0 Statistics: Num rows: 8 Data size: 688 Basic stats: COMPLETE Column stats: PARTIAL ListSink PREHOOK: query: -- basicStatState: COMPLETE colStatState: COMPLETE explain select year from loc_orc PREHOOK: type: QUERY POSTHOOK: query: -- basicStatState: COMPLETE colStatState: COMPLETE explain select year from loc_orc POSTHOOK: type: QUERY STAGE DEPENDENCIES: Stage-0 is a root stage STAGE PLANS: Stage: Stage-0 Fetch Operator limit: -1 Processor Tree: TableScan alias: loc_orc Statistics: Num rows: 8 Data size: 767 Basic stats: COMPLETE Column stats: COMPLETE Select Operator expressions: year (type: string) outputColumnNames: _col0 Statistics: Num rows: 8 Data size: 1472 Basic stats: COMPLETE Column stats: COMPLETE ListSink PREHOOK: query: -- column statistics for __HIVE_DEFAULT_PARTITION__ is not supported yet. Hence colStatState reports PARTIAL -- basicStatState: COMPLETE colStatState: PARTIAL explain select state,locid from loc_orc PREHOOK: type: QUERY POSTHOOK: query: -- column statistics for __HIVE_DEFAULT_PARTITION__ is not supported yet. Hence colStatState reports PARTIAL -- basicStatState: COMPLETE colStatState: PARTIAL explain select state,locid from loc_orc POSTHOOK: type: QUERY STAGE DEPENDENCIES: Stage-0 is a root stage STAGE PLANS: Stage: Stage-0 Fetch Operator limit: -1 Processor Tree: TableScan alias: loc_orc Statistics: Num rows: 8 Data size: 767 Basic stats: COMPLETE Column stats: PARTIAL Select Operator expressions: state (type: string), locid (type: int) outputColumnNames: _col0, _col1 Statistics: Num rows: 8 Data size: 720 Basic stats: COMPLETE Column stats: PARTIAL ListSink PREHOOK: query: -- basicStatState: COMPLETE colStatState: COMPLETE explain select state,locid from loc_orc where year='2001' PREHOOK: type: QUERY POSTHOOK: query: -- basicStatState: COMPLETE colStatState: COMPLETE explain select state,locid from loc_orc where year='2001' POSTHOOK: type: QUERY STAGE DEPENDENCIES: Stage-0 is a root stage STAGE PLANS: Stage: Stage-0 Fetch Operator limit: -1 Processor Tree: TableScan alias: loc_orc Statistics: Num rows: 7 Data size: 425 Basic stats: COMPLETE Column stats: COMPLETE Select Operator expressions: state (type: string), locid (type: int) outputColumnNames: _col0, _col1 Statistics: Num rows: 7 Data size: 630 Basic stats: COMPLETE Column stats: COMPLETE ListSink PREHOOK: query: -- basicStatState: COMPLETE colStatState: NONE explain select state,locid from loc_orc where year!='2001' PREHOOK: type: QUERY POSTHOOK: query: -- basicStatState: COMPLETE colStatState: NONE explain select state,locid from loc_orc where year!='2001' POSTHOOK: type: QUERY STAGE DEPENDENCIES: Stage-0 is a root stage STAGE PLANS: Stage: Stage-0 Fetch Operator limit: -1 Processor Tree: TableScan alias: loc_orc Statistics: Num rows: 1 Data size: 342 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: state (type: string), locid (type: int) outputColumnNames: _col0, _col1 Statistics: Num rows: 1 Data size: 342 Basic stats: COMPLETE Column stats: NONE ListSink PREHOOK: query: -- basicStatState: COMPLETE colStatState: PARTIAL explain select * from loc_orc PREHOOK: type: QUERY POSTHOOK: query: -- basicStatState: COMPLETE colStatState: PARTIAL explain select * from loc_orc POSTHOOK: type: QUERY STAGE DEPENDENCIES: Stage-0 is a root stage STAGE PLANS: Stage: Stage-0 Fetch Operator limit: -1 Processor Tree: TableScan alias: loc_orc Statistics: Num rows: 8 Data size: 767 Basic stats: COMPLETE Column stats: PARTIAL Select Operator expressions: state (type: string), locid (type: int), zip (type: bigint), year (type: string) outputColumnNames: _col0, _col1, _col2, _col3 Statistics: Num rows: 8 Data size: 2192 Basic stats: COMPLETE Column stats: PARTIAL ListSink PREHOOK: query: -- This is to test filter expression evaluation on partition column -- numRows: 2 dataSize: 8 basicStatState: COMPLETE colStatState: COMPLETE explain select locid from loc_orc where locid>0 and year='2001' PREHOOK: type: QUERY POSTHOOK: query: -- This is to test filter expression evaluation on partition column -- numRows: 2 dataSize: 8 basicStatState: COMPLETE colStatState: COMPLETE explain select locid from loc_orc where locid>0 and year='2001' POSTHOOK: type: QUERY STAGE DEPENDENCIES: Stage-1 is a root stage Stage-0 depends on stages: Stage-1 STAGE PLANS: Stage: Stage-1 Map Reduce Map Operator Tree: TableScan alias: loc_orc Statistics: Num rows: 7 Data size: 425 Basic stats: COMPLETE Column stats: COMPLETE Filter Operator predicate: (locid > 0) (type: boolean) Statistics: Num rows: 2 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE Select Operator expressions: locid (type: int) outputColumnNames: _col0 Statistics: Num rows: 2 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE File Output Operator compressed: false Statistics: Num rows: 2 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Stage: Stage-0 Fetch Operator limit: -1 Processor Tree: ListSink PREHOOK: query: explain select locid,year from loc_orc where locid>0 and year='2001' PREHOOK: type: QUERY POSTHOOK: query: explain select locid,year from loc_orc where locid>0 and year='2001' POSTHOOK: type: QUERY STAGE DEPENDENCIES: Stage-1 is a root stage Stage-0 depends on stages: Stage-1 STAGE PLANS: Stage: Stage-1 Map Reduce Map Operator Tree: TableScan alias: loc_orc Statistics: Num rows: 7 Data size: 425 Basic stats: COMPLETE Column stats: COMPLETE Filter Operator predicate: (locid > 0) (type: boolean) Statistics: Num rows: 2 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE Select Operator expressions: locid (type: int), '2001' (type: string) outputColumnNames: _col0, _col1 Statistics: Num rows: 2 Data size: 184 Basic stats: COMPLETE Column stats: COMPLETE File Output Operator compressed: false Statistics: Num rows: 2 Data size: 184 Basic stats: COMPLETE Column stats: COMPLETE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Stage: Stage-0 Fetch Operator limit: -1 Processor Tree: ListSink PREHOOK: query: explain select * from (select locid,year from loc_orc) test where locid>0 and year='2001' PREHOOK: type: QUERY POSTHOOK: query: explain select * from (select locid,year from loc_orc) test where locid>0 and year='2001' POSTHOOK: type: QUERY STAGE DEPENDENCIES: Stage-1 is a root stage Stage-0 depends on stages: Stage-1 STAGE PLANS: Stage: Stage-1 Map Reduce Map Operator Tree: TableScan alias: loc_orc Statistics: Num rows: 7 Data size: 425 Basic stats: COMPLETE Column stats: COMPLETE Filter Operator predicate: (locid > 0) (type: boolean) Statistics: Num rows: 2 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE Select Operator expressions: locid (type: int), '2001' (type: string) outputColumnNames: _col0, _col1 Statistics: Num rows: 2 Data size: 184 Basic stats: COMPLETE Column stats: COMPLETE File Output Operator compressed: false Statistics: Num rows: 2 Data size: 184 Basic stats: COMPLETE Column stats: COMPLETE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Stage: Stage-0 Fetch Operator limit: -1 Processor Tree: ListSink