PREHOOK: query: CREATE TABLE hbase_pushdown(key string, value string) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,cf:string") PREHOOK: type: CREATETABLE PREHOOK: Output: database:default PREHOOK: Output: default@hbase_pushdown POSTHOOK: query: CREATE TABLE hbase_pushdown(key string, value string) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,cf:string") POSTHOOK: type: CREATETABLE POSTHOOK: Output: database:default POSTHOOK: Output: default@hbase_pushdown PREHOOK: query: INSERT OVERWRITE TABLE hbase_pushdown SELECT cast(key as string), value FROM src PREHOOK: type: QUERY PREHOOK: Input: default@src PREHOOK: Output: default@hbase_pushdown POSTHOOK: query: INSERT OVERWRITE TABLE hbase_pushdown SELECT cast(key as string), value FROM src POSTHOOK: type: QUERY POSTHOOK: Input: default@src POSTHOOK: Output: default@hbase_pushdown PREHOOK: query: -- with full pushdown explain select * from hbase_pushdown where key>'90' PREHOOK: type: QUERY POSTHOOK: query: -- with full pushdown explain select * from hbase_pushdown where key>'90' POSTHOOK: type: QUERY STAGE DEPENDENCIES: Stage-1 is a root stage Stage-0 depends on stages: Stage-1 STAGE PLANS: Stage: Stage-1 Map Reduce Map Operator Tree: TableScan alias: hbase_pushdown filterExpr: (key > '90') (type: boolean) Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE Filter Operator predicate: (key > '90') (type: boolean) Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE Select Operator expressions: key (type: string), value (type: string) outputColumnNames: _col0, _col1 Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE File Output Operator compressed: false Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Stage: Stage-0 Fetch Operator limit: -1 Processor Tree: ListSink PREHOOK: query: select * from hbase_pushdown where key>'90' PREHOOK: type: QUERY PREHOOK: Input: default@hbase_pushdown #### A masked pattern was here #### POSTHOOK: query: select * from hbase_pushdown where key>'90' POSTHOOK: type: QUERY POSTHOOK: Input: default@hbase_pushdown #### A masked pattern was here #### 92 val_92 95 val_95 96 val_96 97 val_97 98 val_98 PREHOOK: query: select * from hbase_pushdown where key<'1' PREHOOK: type: QUERY PREHOOK: Input: default@hbase_pushdown #### A masked pattern was here #### POSTHOOK: query: select * from hbase_pushdown where key<'1' POSTHOOK: type: QUERY POSTHOOK: Input: default@hbase_pushdown #### A masked pattern was here #### 0 val_0 PREHOOK: query: select * from hbase_pushdown where key<='2' PREHOOK: type: QUERY PREHOOK: Input: default@hbase_pushdown #### A masked pattern was here #### POSTHOOK: query: select * from hbase_pushdown where key<='2' POSTHOOK: type: QUERY POSTHOOK: Input: default@hbase_pushdown #### A masked pattern was here #### 0 val_0 10 val_10 100 val_100 103 val_103 104 val_104 105 val_105 11 val_11 111 val_111 113 val_113 114 val_114 116 val_116 118 val_118 119 val_119 12 val_12 120 val_120 125 val_125 126 val_126 128 val_128 129 val_129 131 val_131 133 val_133 134 val_134 136 val_136 137 val_137 138 val_138 143 val_143 145 val_145 146 val_146 149 val_149 15 val_15 150 val_150 152 val_152 153 val_153 155 val_155 156 val_156 157 val_157 158 val_158 160 val_160 162 val_162 163 val_163 164 val_164 165 val_165 166 val_166 167 val_167 168 val_168 169 val_169 17 val_17 170 val_170 172 val_172 174 val_174 175 val_175 176 val_176 177 val_177 178 val_178 179 val_179 18 val_18 180 val_180 181 val_181 183 val_183 186 val_186 187 val_187 189 val_189 19 val_19 190 val_190 191 val_191 192 val_192 193 val_193 194 val_194 195 val_195 196 val_196 197 val_197 199 val_199 2 val_2 PREHOOK: query: select * from hbase_pushdown where key>='90' PREHOOK: type: QUERY PREHOOK: Input: default@hbase_pushdown #### A masked pattern was here #### POSTHOOK: query: select * from hbase_pushdown where key>='90' POSTHOOK: type: QUERY POSTHOOK: Input: default@hbase_pushdown #### A masked pattern was here #### 90 val_90 92 val_92 95 val_95 96 val_96 97 val_97 98 val_98 PREHOOK: query: -- with cnostant expressinon explain select * from hbase_pushdown where key>=cast(40 + 50 as string) PREHOOK: type: QUERY POSTHOOK: query: -- with cnostant expressinon explain select * from hbase_pushdown where key>=cast(40 + 50 as string) POSTHOOK: type: QUERY STAGE DEPENDENCIES: Stage-1 is a root stage Stage-0 depends on stages: Stage-1 STAGE PLANS: Stage: Stage-1 Map Reduce Map Operator Tree: TableScan alias: hbase_pushdown filterExpr: (key >= '90') (type: boolean) Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE Filter Operator predicate: (key >= '90') (type: boolean) Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE Select Operator expressions: key (type: string), value (type: string) outputColumnNames: _col0, _col1 Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE File Output Operator compressed: false Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Stage: Stage-0 Fetch Operator limit: -1 Processor Tree: ListSink PREHOOK: query: select * from hbase_pushdown where key>=cast(40 + 50 as string) PREHOOK: type: QUERY PREHOOK: Input: default@hbase_pushdown #### A masked pattern was here #### POSTHOOK: query: select * from hbase_pushdown where key>=cast(40 + 50 as string) POSTHOOK: type: QUERY POSTHOOK: Input: default@hbase_pushdown #### A masked pattern was here #### 90 val_90 92 val_92 95 val_95 96 val_96 97 val_97 98 val_98 PREHOOK: query: -- with partial pushdown explain select * from hbase_pushdown where key>'90' and value like '%9%' PREHOOK: type: QUERY POSTHOOK: query: -- with partial pushdown explain select * from hbase_pushdown where key>'90' and value like '%9%' POSTHOOK: type: QUERY STAGE DEPENDENCIES: Stage-1 is a root stage Stage-0 depends on stages: Stage-1 STAGE PLANS: Stage: Stage-1 Map Reduce Map Operator Tree: TableScan alias: hbase_pushdown filterExpr: (key > '90') (type: boolean) Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE Filter Operator predicate: (value like '%9%') (type: boolean) Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE Select Operator expressions: key (type: string), value (type: string) outputColumnNames: _col0, _col1 Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE File Output Operator compressed: false Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Stage: Stage-0 Fetch Operator limit: -1 Processor Tree: ListSink PREHOOK: query: select * from hbase_pushdown where key>'90' and value like '%9%' PREHOOK: type: QUERY PREHOOK: Input: default@hbase_pushdown #### A masked pattern was here #### POSTHOOK: query: select * from hbase_pushdown where key>'90' and value like '%9%' POSTHOOK: type: QUERY POSTHOOK: Input: default@hbase_pushdown #### A masked pattern was here #### 92 val_92 95 val_95 96 val_96 97 val_97 98 val_98 PREHOOK: query: -- with two residuals explain select * from hbase_pushdown where key>='90' and value like '%9%' and key=cast(value as int) PREHOOK: type: QUERY POSTHOOK: query: -- with two residuals explain select * from hbase_pushdown where key>='90' and value like '%9%' and key=cast(value as int) POSTHOOK: type: QUERY STAGE DEPENDENCIES: Stage-1 is a root stage Stage-0 depends on stages: Stage-1 STAGE PLANS: Stage: Stage-1 Map Reduce Map Operator Tree: TableScan alias: hbase_pushdown filterExpr: (key >= '90') (type: boolean) Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE Filter Operator predicate: ((value like '%9%') and (UDFToDouble(key) = UDFToDouble(UDFToInteger(value)))) (type: boolean) Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE Select Operator expressions: key (type: string), value (type: string) outputColumnNames: _col0, _col1 Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE File Output Operator compressed: false Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Stage: Stage-0 Fetch Operator limit: -1 Processor Tree: ListSink PREHOOK: query: select * from hbase_pushdown where key>='90' and value like '%9%' and key=cast(value as int) PREHOOK: type: QUERY PREHOOK: Input: default@hbase_pushdown #### A masked pattern was here #### POSTHOOK: query: select * from hbase_pushdown where key>='90' and value like '%9%' and key=cast(value as int) POSTHOOK: type: QUERY POSTHOOK: Input: default@hbase_pushdown #### A masked pattern was here #### PREHOOK: query: -- with contradictory pushdowns explain select * from hbase_pushdown where key<'80' and key>'90' and value like '%90%' PREHOOK: type: QUERY POSTHOOK: query: -- with contradictory pushdowns explain select * from hbase_pushdown where key<'80' and key>'90' and value like '%90%' POSTHOOK: type: QUERY STAGE DEPENDENCIES: Stage-1 is a root stage Stage-0 depends on stages: Stage-1 STAGE PLANS: Stage: Stage-1 Map Reduce Map Operator Tree: TableScan alias: hbase_pushdown filterExpr: ((key < '80') and (key > '90')) (type: boolean) Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE Filter Operator predicate: (value like '%90%') (type: boolean) Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE Select Operator expressions: key (type: string), value (type: string) outputColumnNames: _col0, _col1 Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE File Output Operator compressed: false Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Stage: Stage-0 Fetch Operator limit: -1 Processor Tree: ListSink PREHOOK: query: select * from hbase_pushdown where key<'80' and key>'90' and value like '%90%' PREHOOK: type: QUERY PREHOOK: Input: default@hbase_pushdown #### A masked pattern was here #### POSTHOOK: query: select * from hbase_pushdown where key<'80' and key>'90' and value like '%90%' POSTHOOK: type: QUERY POSTHOOK: Input: default@hbase_pushdown #### A masked pattern was here #### PREHOOK: query: -- with nothing to push down explain select * from hbase_pushdown PREHOOK: type: QUERY POSTHOOK: query: -- with nothing to push down explain select * from hbase_pushdown POSTHOOK: type: QUERY STAGE DEPENDENCIES: Stage-0 is a root stage STAGE PLANS: Stage: Stage-0 Fetch Operator limit: -1 Processor Tree: TableScan alias: hbase_pushdown Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE Select Operator expressions: key (type: string), value (type: string) outputColumnNames: _col0, _col1 Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE ListSink PREHOOK: query: -- with a predicate which is not actually part of the filter, so -- it should be ignored by pushdown explain select * from hbase_pushdown where (case when key<'90' then 2 else 4 end) > 3 PREHOOK: type: QUERY POSTHOOK: query: -- with a predicate which is not actually part of the filter, so -- it should be ignored by pushdown explain select * from hbase_pushdown where (case when key<'90' then 2 else 4 end) > 3 POSTHOOK: type: QUERY STAGE DEPENDENCIES: Stage-1 is a root stage Stage-0 depends on stages: Stage-1 STAGE PLANS: Stage: Stage-1 Map Reduce Map Operator Tree: TableScan alias: hbase_pushdown Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE Filter Operator predicate: (CASE WHEN ((key < '90')) THEN (2) ELSE (4) END > 3) (type: boolean) Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE Select Operator expressions: key (type: string), value (type: string) outputColumnNames: _col0, _col1 Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE File Output Operator compressed: false Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Stage: Stage-0 Fetch Operator limit: -1 Processor Tree: ListSink PREHOOK: query: -- with a predicate which is under an OR, so it should -- be ignored by pushdown explain select * from hbase_pushdown where key<='80' or value like '%90%' PREHOOK: type: QUERY POSTHOOK: query: -- with a predicate which is under an OR, so it should -- be ignored by pushdown explain select * from hbase_pushdown where key<='80' or value like '%90%' POSTHOOK: type: QUERY STAGE DEPENDENCIES: Stage-1 is a root stage Stage-0 depends on stages: Stage-1 STAGE PLANS: Stage: Stage-1 Map Reduce Map Operator Tree: TableScan alias: hbase_pushdown Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE Filter Operator predicate: ((key <= '80') or (value like '%90%')) (type: boolean) Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE Select Operator expressions: key (type: string), value (type: string) outputColumnNames: _col0, _col1 Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE File Output Operator compressed: false Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Stage: Stage-0 Fetch Operator limit: -1 Processor Tree: ListSink PREHOOK: query: -- following will get pushed into hbase after HIVE-2819 explain select * from hbase_pushdown where key > '281' and key < '287' PREHOOK: type: QUERY POSTHOOK: query: -- following will get pushed into hbase after HIVE-2819 explain select * from hbase_pushdown where key > '281' and key < '287' POSTHOOK: type: QUERY STAGE DEPENDENCIES: Stage-1 is a root stage Stage-0 depends on stages: Stage-1 STAGE PLANS: Stage: Stage-1 Map Reduce Map Operator Tree: TableScan alias: hbase_pushdown filterExpr: ((key > '281') and (key < '287')) (type: boolean) Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE Filter Operator predicate: ((key > '281') and (key < '287')) (type: boolean) Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE Select Operator expressions: key (type: string), value (type: string) outputColumnNames: _col0, _col1 Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE File Output Operator compressed: false Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Stage: Stage-0 Fetch Operator limit: -1 Processor Tree: ListSink PREHOOK: query: select * from hbase_pushdown where key > '281' and key < '287' PREHOOK: type: QUERY PREHOOK: Input: default@hbase_pushdown #### A masked pattern was here #### POSTHOOK: query: select * from hbase_pushdown where key > '281' and key < '287' POSTHOOK: type: QUERY POSTHOOK: Input: default@hbase_pushdown #### A masked pattern was here #### 282 val_282 283 val_283 284 val_284 285 val_285 286 val_286 PREHOOK: query: -- with pushdown disabled explain select * from hbase_pushdown where key<='90' PREHOOK: type: QUERY POSTHOOK: query: -- with pushdown disabled explain select * from hbase_pushdown where key<='90' POSTHOOK: type: QUERY STAGE DEPENDENCIES: Stage-1 is a root stage Stage-0 depends on stages: Stage-1 STAGE PLANS: Stage: Stage-1 Map Reduce Map Operator Tree: TableScan alias: hbase_pushdown Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE Filter Operator predicate: (key <= '90') (type: boolean) Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE Select Operator expressions: key (type: string), value (type: string) outputColumnNames: _col0, _col1 Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE File Output Operator compressed: false Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Stage: Stage-0 Fetch Operator limit: -1 Processor Tree: ListSink