Binary Download

Download the source code from http://tajo.apache.org/downloads.html.

Source Download

Download the source code and build Tajo as follows:

$ git clone https://git-wip-us.apache.org/repos/asf/tajo.git tajo

Unpack tarball

You should unpack the tarball (refer to build instruction).

$ tar xzvf tajo-0.2.0-SNAPSHOT.tar.gz

This will result in the creation of subdirectory named tajo-x.y.z-SNAPSHOT. You MUST copy this directory into the same directory on all cluster nodes.

Setup a Tajo cluster

First of all, you need to add the environment variables to conf/tajo-env.sh.

# Hadoop home. Required
export HADOOP_HOME= ...

# The java implementation to use.  Required.
export JAVA_HOME= ...

catalog-site.xml and tajo-site.xml

Tajo’s configuration is based on Hadoop’s configuration system. Tajo uses two config files:

• catalog-site.xml - configuration for the catalog server.
• tajo-site.xml - configuration for other tajo modules.

Each config consists of a pair of a name and a value. If you want to set the config name a.b.c with the value 123, add the following element to an appropriate file.

  <property>
    <name>a.b.c</name>
    <value>123</value>
  </property>

Tajo has a variety of internal configs. If you don’t set some config explicitly, the default config will be used for for that config. Tajo is designed to use only a few of configs in usual cases. You may not be concerned with the configuration.

In default, there is no tajo-site.xml in ${TAJO}/conf directory. If you set some configs, first copy $TAJO_HOME/conf/tajo-site.xml.templete to tajo-site.xml. Then, add the configs to your tajo-site.

tajo-env.sh

tajo-env.sh is a shell script file. The main purpose of this file is to set shell environment variables for TajoMaster and TajoWorker java program. So, you can set some variable as follows:

VARIABLE=value

If a value is a literal string, type this as follows:

VARIABLE='value'

TajoMaster Configuration

  <property>
    <name>tajo.rootdir</name>
    <value>hdfs://namenode_hostname:port/path</value>
  </property>

TAJO_MASTER_HEAPSIZE=2000

Worker Heap Memory Size

The environment variable TAJO_WORKER_HEAPSIZE in conf/tajo-env.sh allow Tajo Worker to use the specified heap memory size.

If you want to adjust heap memory size, set TAJO_WORKER_HEAPSIZE variable in conf/tajo-env.sh with a proper size as follows:

TAJO_WORKER_HEAPSIZE=8000

The default size is 1000 (1GB).

Temporary Data Directory

TajoWorker stores temporary data on local file system due to out-of-core algorithms. It is possible to specify one or more temporary data directories where temporary data will be stored.

tajo-site.xml

  <property>
    <name>tajo.worker.tmpdir.locations</name>
    <value>/disk1/tmpdir,/disk2/tmpdir,/disk3/tmpdir</value>
  </property>

Maximum number of parallel running tasks for each worker

Each worker can execute multiple tasks at a time. Tajo allows users to specify the maximum number of parallel running tasks for each worker.

tajo-site.xml

  <property>
    <name>tajo.worker.parallel-execution.max-num</name>
    <value>12</value>
  </property>

Tajo Master

Worker

Command Line Interface (tsql)

Synopsis

bin/tsql [options]

Options

• -c "quoted sql" : Execute quoted sql statements, and then the shell will exist.

• -f filename (--file filename) : Use the file named filename as the source of commands instead of interactive shell.

• -h hostname (--host hostname) : Specifies the host name of the machine on which the Tajo master is running.

• -p port (--port port) : Specifies the TCP port. If it is not set, the port will be 26002 in default.

Using real number value (real and double)

The real and double data types are mapped to float and double of java primitives respectively. Java primitives float and double follows the IEEE 754 specification. So, these types are correctly matched to SQL standard data types.

• float[( n )] is mapped to either float or double according to a given length n. If n is specified, it must be bewtween 1 and 53. The default value of n is 53.
• If 1 <= n <= 24, a value is mapped to float (6 decimal digits precision).
• If 25 <= n <= 53, a value is mapped to double (15 decimal digits precision).

• Do not use approximate real number columns in WHERE clause in order to compare some exact matches, especially the = and <> operators. The > or < comparisons work well.

The SQL Language

CREATE TABLE

Synopsis

CREATE TABLE <table_name> [(<column_name> <data_type>, ... )]
  [using <storage_type> [with (<key> = <value>, ...)]] [AS <select_statement>]

CREATE EXTERNAL TABLE

CREATE EXTERNAL TABLE <table_name> (<column_name> <data_type>, ... )
  using <storage_type> [with (<key> = <value>, ...)] LOCATION '<path>'

create EXTERNAL table lineitem (
  L_ORDERKEY bigint, 
  L_PARTKEY bigint, 
  ...
  L_COMMENT text) 

USING csv WITH ('csvfile.delimiter'='|','compression.codec'='org.apache.hadoop.io.compress.DeflateCodec')
LOCATION 'hdfs://localhost:9010/tajo/warehouse/lineitem_100_snappy';

DROP TABLE

DROP TABLE <table_name>

SQL Expressions

CAST ( expression AS type )
expression::type

function_name ([expression [, expression ... ]] )

SELECT

Synopsis

SELECT [distinct [all]] * | <expression> [[AS] <alias>] [, ...]
  [FROM <table name> [[AS] <table alias name>] [, ...]]
  [WHERE <condition>]
  [GROUP BY <expression> [, ...]]
  [HAVING <condition>]
  [ORDER BY <expression> [ASC|DESC] [NULL FIRST|NULL LAST] [, ...]]

WHERE

column_reference IN (val1, val2, ..., valN)
column_reference NOT IN (val1, val2, ..., valN)

-- this statement filters lists down all the records where col1 value is 1, 2 or 3:
SELECT col1, col2 FROM table1 WHERE col1 IN (1, 2, 3);

-- this statement filters lists down all the records where col1 value is neither 1, 2 nor 3:
SELECT col1, col2 FROM table1 WHERE col1 NOT IN (1, 2, 3);

SELECT col1, col2 FROM table1 WHERE col2 IN ('tajo', 'hadoop');

SELECT col1, col2 FROM table1 WHERE col2 NOT IN ('tajo', 'hadoop');

string LIKE pattern
string NOT LIKE pattern

string ILIKE pattern
string NOT ILIKE pattern

string SIMILAR TO pattern
string NOT SIMILAR TO pattern

string ~ pattern
string !~ pattern

string ~* pattern
string !~* pattern

'abc'   ~   '.*c'               true
'abc'   ~   'c'                 false
'aaabc' ~   '([a-z]){3}bc       true
'abc'   ~*  '.*C'               true
'abc'   !~* 'B.*'               true

string REGEXP pattern
string NOT REGEXP pattern

string RLIKE pattern
string NOT RLIKE pattern

INSERT (OVERWRITE) INTO

INSERT OVERWRITE statement overwrites a table data of an existing table or a data in a given directory. Tajo’s INSERT OVERWRITE statement follows ‘INSERT INTO SELECT’ statement of SQL. The examples are as follows:

create table t1 (col1 int8, col2 int4, col3 float8);

-- when a target table schema and output schema are equivalent to each other
INSERT OVERWRITE INTO t1 SELECT l_orderkey, l_partkey, l_quantity FROM lineitem;
-- or
INSERT OVERWRITE INTO t1 SELECT * FROM lineitem;

-- when the output schema are smaller than the target table schema
INSERT OVERWRITE INTO t1 SELECT l_orderkey FROM lineitem;

-- when you want to specify certain target columns
INSERT OVERWRITE INTO t1 (col1, col3) SELECT l_orderkey, l_quantity FROM lineitem;

In addition, INSERT OVERWRITE statement overwrites table data as well as a specific directory.

INSERT OVERWRITE INTO LOCATION '/dir/subdir' SELECT l_orderkey, l_quantity FROM lineitem;

Standard Functions

String Operator and Functions

Administration

SQL dump

SQL dump is an easy and strong way. If you use this approach, you don’t need to concern database-level compatibilities. If you want to backup your catalog, just use bin/tajo_dump command. The basic usage of this command is:

$ tajo_dump table_name > outfile

For example, if you want to backup a table customer, you should type a command as follows:

$ bin/tajo_dump customer > table_backup.sql
$
$ cat table_backup.sql
-- Tajo database dump
-- Dump date: 10/04/2013 16:28:03
--

--
-- Name: customer; Type: TABLE; Storage: CSV
-- Path: file:/home/hyunsik/tpch/customer
--
CREATE EXTERNAL TABLE customer (c_custkey INT8, c_name TEXT, c_address TEXT, c_nationkey INT8, c_phone TEXT, c_acctbal FLOAT8, c_mktsegment TEXT, c_comment TEXT) USING CSV LOCATION 'file:/home/hyunsik/tpch/customer';

If you want to restore the catalog from the SQL dump file, please type the below command:

$ bin/tsql -f table_backup.sql

If you use an option ‘-a’, tajo_dump will dump all table DDLs.

$ bin/tajo_dump -a > all_backup.sql

Database-level backup

(TODO)