Hadoop 权威指南学习2 (Sqoop)

6. Sqoop

Apache sqoop is an open source tool that allow users to extract data from structured data store into Hadoop or back.

1) run

self :        $SQOOP_HOME/bin/sqoop

standard: sqoop (默认目录是／usr/bin/sqoop)

2) common commands

sqoop help           ##  the list of available tools

sqoop help import #provided with a tool, will get its usage

sqoop import         # run the tool

sqoop-toolname     #alternatate way  to run the tool

3) Sqoop Connector

Sqoop has an entension framework that makes it possible to import data from and export to ,any external storage system that has bulk

data transfer capabilities.

Sqoop Connector is a modular component to use it to enable extracting work.

4) An import example:

% sqoop import --connect jdbc:mysql://localhost/hadoopDB --table myTest -m 1

 

• Sqoop tool will run a MapReduce job that connects to the MySQL db and reads the table.
• By default, it will call four map tasks. But we could specify just one map task as (-m 1).
• By default, it will generate comma-delimited text files.

5） generated code

Besides importing to HDFS successfully, Sqoop also provides you with a generated Java source file in current local directory.

• Could be used for some special needs.
• Coegen tool could generate the source code without performing an import.

%sqoop codegen  --connect jdbc:mysql://localhost/hadoopDB  --table myTest  --class-name myNeed

6) import process

A better importing process should use a splitting column to divide table querying aross multiple nodes.

6)

7)