基于【CentOS-7+ Ambari 2.7.0 + HDP 3.0】搭建HAWQ数据仓库04 —— 安装HAWQ插件PXF3.3.0.0

一、 安装PXF3.3.0.0，这里所安装的pxf的包文件都包含在apache-hawq-rpm-2.3.0.0-incubating.tar.gz里面
下面步骤都是以root身份执行
这里注意，pxf插件要用到tomcat服务，必须使用安装包里面的7.0.62， 不能安装或升级为 tomcat8，这会造成依赖的catalina.jar包 版本不匹配，以致pxf启动！

安装时由于pxf的包都是el6版本的，但是我用的centos7，所以rpm带参数“--nodeps”以忽略RPM命令对依赖包的检测。

cd /opt/gpadmin/hawq_rpm_packages

rpm -ivh  apache-tomcat-7.0.-el6.noarch.rpm
rpm -ivh --nodeps pxf-service-3.3.0.0-.el6.noarch.rpm
rpm -ivh --nodeps pxf-hdfs-3.3.0.0-.el6.noarch.rpm
rpm -ivh --nodeps pxf-hive-3.3.0.0-.el6.noarch.rpm
rpm -ivh --nodeps pxf-hbase-3.3.0.0-.el6.noarch.rpm
rpm -ivh --nodeps pxf-jdbc-3.3.0.0-.el6.noarch.rpm
rpm -ivh --nodeps pxf-json-3.3.0.0-.el6.noarch.rpm
rpm -ivh --nodeps pxf-3.3.0.0-.el6.noarch.rpm

二、 配置PXF

1，由于hadoop为HDP版本，所以使用hdp相关的jar包配置

[root@ep-bd01 ~]cp  /etc/pxf/conf/pxf-privatehdp.classpath  /etc/pxf/conf/pxf-private.classpath
[root@ep-bd01 ~]cp  /etc/pxf/conf/pxf-profiles.xml   /etc/pxf/conf/pxf-profiles.default.xml

2，修改pxf目录所有者：

[root@ep-bd01 ~] chown -R pxf:pxf /opt/pxf-3.3.0.0
[root@ep-bd01 ~] chown -R pxf:pxf /tmp/logs

3，建立软连接目录：

[root@ep-bd01 ~] ln -s /etc/pxf-3.3.0.0/conf /opt/pxf-3.3.0.0/conf
[root@ep-bd01 ~] ln -s /usr/lib/pxf  /opt/pxf-3.3.0.0/lib

4，建立一个init过程需要的目录和template文件

[root@ep-bd01 ~] mkdir /opt/pxf-3.3.0.0/conf-templates
[root@ep-bd01 ~] cp /opt/pxf/conf/pxf-privatehdp.classpath /opt/pxf/conf-templates/pxf-private-hdp.classpath.template

5，修改pxf-service文件为pxf，因为init过程需要建立同名的目录，同时需要修改init.d目录中的链接：

[root@ep-bd01 ~] mv /opt/pxf/pxf-service  /opt/pxf/pxf
[root@ep-bd01 ~] unlink /etc/init.d/pxf-service
[root@ep-bd01 ~] ln -s /opt/pxf/pxf   /etc/init.d/pxf-service

6，修改tomcat/conf目录的权限，同时在pxf目录中建立链接

[root@ep-bd01 ~] chmod  -R /opt/apache-tomcat/conf/
[root@ep-bd01 ~] ln -s /opt/apache-tomcat /opt/pxf-3.3.0.0/apache-tomcat

7，编辑/etc/pxf/conf/pxf-env.sh，修改 PARENT_SCRIPT_DIR和LD_LIBRARY_PATH的值

[root@ep-bd01 ~] vim /etc/pxf/conf/pxf-env.sh
export PARENT_SCRIPT_DIR=/opt/pxf-3.3..0
export PXF_HOME=/opt/pxf-3.3.0.0
export LD_LIBRARY_PATH=/usr/hdp/current/hadoop-client/lib/native:${LD_LIBRARY_PATH}

8，修改pxf脚本文件，设置PXF_HOME

[root@ep-bd01 ~] vim /opt/pxf-3.3.0.0/pxf

export PARENT_SCRIPT_DIR=/opt/pxf-3.3.0.0

export PXF_HOME=/opt/pxf-3.3.0.0

9，编辑/etc/pxf/conf/pxf-public.classpath，添加一系列的jar包

[root@ep-bd01 pxf-3.3.0.0]# vim /etc/pxf-3.3.0.0/conf/pxf-public.classpath
/usr/hdp/current/hadoop-client/lib/commons-beanutils-1.9.3.jar
/usr/hdp/current/hadoop-client/lib/commons-cli-1.2.jar
/usr/hdp/current/hadoop-client/lib/commons-codec-1.11.jar
/usr/hdp/current/hadoop-client/lib/commons-collections-3.2.2.jar
/usr/hdp/current/hadoop-client/lib/commons-compress-1.4.1.jar
/usr/hdp/current/hadoop-client/lib/commons-configuration2-2.1.1.jar
/usr/hdp/current/hadoop-client/lib/commons-io-2.5.jar
/usr/hdp/current/hadoop-client/lib/commons-lang-2.6.jar
/usr/hdp/current/hadoop-client/lib/commons-lang3-3.4.jar
/usr/hdp/current/hadoop-client/lib/commons-logging-1.1.3.jar
/usr/hdp/current/hadoop-client/lib/commons-math3-3.1.1.jar
/usr/hdp/current/hadoop-client/lib/commons-net-3.6.jar
/usr/hdp/current/hadoop-client/lib/jersey-core-1.19.jar
/usr/hdp/current/hadoop-client/lib/jersey-json-1.19.jar
/usr/hdp/current/hadoop-client/lib/jersey-server-1.19.jar
/usr/hdp/current/hadoop-client/lib/jersey-servlet-1.19.jar
/usr/hdp/current/hadoop-client/lib/jsr311-api-1.1.1.jar
/usr/hdp/current/hadoop-client/lib/woodstox-core-5.0.3.jar
/usr/hdp/current/hadoop-client/lib/stax2-api-3.1.4.jar
/usr/hdp/current/hadoop-client/lib/htrace-core4-4.1.0-incubating.jar
/usr/hdp/current/hadoop-client/lib/re2j-1.1.jar
/usr/hdp/3.0.0.0-1634/hbase/lib/atlas-hbase-plugin-impl/commons-configuration-1.10.jar
/usr/hdp/current/hadoop-hdfs-datanode/hadoop-hdfs-rbf.jar
/usr/hdp/current/hadoop-hdfs-datanode/hadoop-hdfs-nfs.jar
/usr/hdp/current/hadoop-hdfs-datanode/hadoop-hdfs-native-client.jar
/usr/hdp/current/hadoop-hdfs-datanode/hadoop-hdfs-httpfs.jar
/usr/hdp/current/hadoop-hdfs-datanode/hadoop-hdfs-client-3.1.0.3.0.0.0-1634.jar
/opt/pxf-3.3.0.0/lib/pxf-service-3.3.0.0.jar
/opt/pxf-3.3.0.0/lib/pxf-api-3.3.0.0.jar
:wq

10，复制pxf-profiles.xml 为pxf-profiles-default.xml并编辑，添加profile配置

[root@ep-bd05 pxf-3.3.0.0]# cp /etc/pxf/conf/pxf-profiles.xml   /etc/pxf/conf/pxf-profiles-default.xml
[root@ep-bd05 pxf-3.3.0.0]# vim /etc/pxf/conf/pxf-profiles-default.xml
<profiles>
  <profile>
        <name>HdfsTextSimple</name>
        <description>This profile is suitable for using when reading delimited single line records from plain text files
            on HDFS
        </description>
        <plugins>
            <fragmenter>org.apache.hawq.pxf.plugins.hdfs.HdfsDataFragmenter</fragmenter>
            <accessor>org.apache.hawq.pxf.plugins.hdfs.LineBreakAccessor</accessor>
            <resolver>org.apache.hawq.pxf.plugins.hdfs.StringPassResolver</resolver>
        </plugins>
    </profile>
    <profile>
        <name>HdfsTextMulti</name>
        <description>This profile is suitable for using when reading delimited single or multi line records (with quoted
            linefeeds) from plain text files on HDFS. It is not splittable (non parallel) and slower than HdfsTextSimple.
        </description>
        <plugins>
            <fragmenter>org.apache.hawq.pxf.plugins.hdfs.HdfsDataFragmenter</fragmenter>
            <accessor>org.apache.hawq.pxf.plugins.hdfs.QuotedLineBreakAccessor</accessor>
            <resolver>org.apache.hawq.pxf.plugins.hdfs.StringPassResolver</resolver>
        </plugins>
    </profile>
</profiles>

三、初始化pxf，必须使用pxf用户

1,设置pxf的密码

passwd pxf

2，初始化，需要使用用户pxf

[root@ep-bd03 pxf]# source /etc/pxf/conf/pxf-env.sh
[root@ep-bd03 pxf]# sudo -u pxf service pxf-service init

Generating /opt/pxf-3.3.0.0/conf/pxf-private.classpath file from /opt/pxf-3.3.0.0/conf-templates/pxf-private-hdp.classpath.template ...

cp /opt/pxf/pxf-service/webapps/pxf/WEB-INF/lib/*.jar  /opt/pxf/lib/

四、启动PXF service

1，启动：

sudo -u pxf service pxf-service start

Checking if tomcat is up and running...
tomcat not responding, re-trying after 1 second (attempt number 1)
Checking if PXF webapp is up and running...
PXF webapp is listening on port 51200

2，测试：

使用pxf插件访问已经事先从oracle导入到HDFS上的数据（使用了sqoop的--compress选项，是gz压缩格式，但是HdfsTextSimple可以直接访问），下面是建立hawq外部表的命令，注意路径中的星号。

drop external table ext.yx_bw;
create external table ext.yx_bw (occur_time date, ......    ) 
location ('pxf://192.168.58.15:51200/var/data/ext/yx_bw/*?profile=hdfstextsimple') format 'text'(delimiter ',' null '');    

**注意** ，此处的主机地址，我直接使用的是主机的地址，如果使用主机名称则hawq访问失败，据我观察应该是没有正确转换，一直没能解决此问题，如果哪位大侠知道请一定不吝赐教，先谢过了！如果地址使用location ('pxf://bd05:51200/var/data/ext/yx_bw/*?profile=hdfstextsimple') ，外部表可以建立，但是访问数据时显示如下错误，且没有详细信息，pxf服务的log也找不到访问失败的记录！

epbd=> select * from ext.yx_bw;
ERROR:  remote component error (): (libchurl.c:)

下面是系统中的/etc/hosts文件和/etc/host.conf文件，由于本集群可以访问外网，可以看到nslookup返回了错误地址，但是ping和curl访问都是正确的。

root@ep-bd05 pg_log]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.58.11  ep-bd01 bd01
192.168.58.12  ep-bd02 bd02
192.168.58.13  ep-bd03 bd03
192.168.58.14  ep-bd04 bd04
192.168.58.15  ep-bd05 bd05

[root@ep-bd05 pg_log]# cat /etc/host.conf
multi off

[root@ep-bd05 pg_log]# nslookup bd01
Server:         211.137.160.5
Address:        211.137.160.5#

Non-authoritative answer:
Name:   bd01
Address: 211.137.170.246

[root@ep-bd05 pg_log]# ping bd01
PING ep-bd01 (192.168.58.11) () bytes of data.
 bytes from ep-bd01 (192.168.58.11): icmp_seq= ttl= time=0.156 ms
 bytes from ep-bd01 (192.168.58.11): icmp_seq= ttl= time=0.160 ms
^C
--- ep-bd01 ping statistics ---
 packets transmitted,  received, % packet loss, time 999ms
rtt min/avg/max/mdev = 0.156/0.158/0.160/0.002 ms
[root@ep-bd05 pg_log]# curl http://bd01:51200/pxf/v0
Wrong version v0, supported version is v15
[root@ep-bd05 pg_log]# 

五、可选操作：

1，修改/opt/pxf/pxf-service/conf/catalina.properties，修改 base.shutdown.port

#base.shutdown.port=-1
base.shutdown.port=8005

2，修改/opt/pxf/pxf-service/conf/tomcat-users.xml,给用户tomcat添加角色manager-gui ，以便可以在浏览器中管理webapps

[root@ep-bd01 ~] vim /opt/pxf/pxf-service/conf/tomcat-users.xml
<role rolename="tomcat"/>
<role rolename="manager-gui"/>
<user username="tomcat" password="tomcat" roles="tomcat,manager-gui"/>
<user username="both" password="tomcat" roles="tomcat,role1"/>
<user username="role1" password="tomcat" roles="role1"/>