<Impala><Overview><UDF>

Overview

• Apache Impala (incubating) is the open source, native analytic database for apache Hadoop.

Features

• Do BI-style Queries on Hadoop:
  • low latency and high concurrency for BI/analytic queries on Hadoop(not delivered by batch frameworks such as Apache Hive).
  • scales linearly, even in multitenant environments.
• Unify ur Infrasturecture: Utilize the same file and data formats and metadata, security, and resource management frameworks as your Hadoop deployment—no redundant infrastructure or data conversion/duplication.
• Implement Quickly: supports SQL
• Count on Enterprise-class Security
• Retain Freedom from Lock-in: open-source
• Expand the Hadoop User-verse

Architecuture

• Circumvents MapReduce to avoid latency, directly access the data through a specialized distributed query engine that is very similar to those found in commercial parallel RDBMSs.
• Some advantages:
  • Thx to local processing on data nodes, network bottlenecks are avoided.
  • A signle, open, and unified metadata store can be utilized.
  • Costly data format conversion is unnecessary and thus no overhead is incurred.
  • All data is immediately query-able, with no delays for ETL.
  • All hardware is utilized for Impala queries as well as for MR.
  • Only a single machine pool is needed to scale.

Documentation

... skip

Impala User-Defined Functions(UDFs)

• UDF let you code ur own application logic for processing column values during an Impala query.

UDFS Concepts

• U can code either scalar functions for producing results one row at a time.
• Or more complex aggregate functions for doing analysis across.

UDFs and UDAFs

• The most general kind of udf takes single input value and produces a single output value. When used in a query, it is called once for each row in the result set. eg:

  select customer_name, is_frequent_customer(customer_id) from customers;
  select obfuscate(sensitive_column) from sensitive_data;

• A user-defined aggergate function(UDAF) accepts a group of values and returns a single value. U can use UDAFs to summarize and condense sets of rows, in the same style as the built-in COUNT, MAX(), SUM(), and AVG() functions. When called in a query that uses the GROUP BY clause, the function is called once for each combination of GROUP BY values. eg:

  -- Evaluates multiple rows but returns a single value
  select closest_restaurant(latitude, longitude) from places;
  
  -- Evaluates batches of rows and returns a separate value for each batch.
  select most_profitable_locartion(store_id, sales, expenses, tax_rate, depreciation) from franchise_data group by year;

• Currently, Impala does not support other categories of udf, such as user-defined table functions(UDTFs) or window functions.

Native Impala UDFs

• Impala supports UDFs written in C++, in addition to supporting existing Hive UDFs written in Java.
• Where practical, use C++ UDFs because the compiled native code can yield higher performance, with UDF execution time often 10x faster for a C++ UDF than the equivalent Java UDF.

Using Hive UDFs with Impala

• Impala can run Java-based user-defined functions (UDFs), originally written for Hive, with no changes, subject to the following conditions:
  • The parameter and return value must all use scalar data types supported by Impala. That's to say, complex or nested types are not supported.
  • Currently, Hive UDFs that accept or return the TIMESTAMP type are not supported.
  • Hive UDAFs and UDTFs are not supported.
  • Typically, a Java UDF will execute several times slower in Impala than the equivalent native UDF written in C++.
• What to do next?
  • write ur udf
  • upload the jar to a hdfs path(where impala can read)
  • for each Java-based UDF that u want to call through Impala, issue a CREATE FUNCTION statement, with a LOCATION clause containing the full HDFS path or the JAR file, and a SYMBOL clause with the fully qualified name of the class, using dots as separators and without the .class extension. eg:

    create function my_neg(bigint)
    returns bigint location '/user/hive/udfs/hive.jar'
    symbol = 'org.apache.hadoop.hive.ql.udf.UDFOPNegative';

  • call the function from ur queries, passing arguments of the correct type to match the function signature.

FYI

• Impala documentation
• Impala string functions
• Impala Date and Time functions