Apache Hive 执行HQL语句报错 ( 10G )

# 故障描述：

hive > select substring(request_body["uuid"], -, ) as uuid, count(distinct(request_body["uuid"])) as count
from log_bftv_api
where year= and month= and day= and request_body["method"] = "bv.lau.urecommend" and length(request_body["uuid"]) =
group by
order by uuid;

# hive 执行该HQL语句时报错信息如下：( 数据量小的时候没有问题 )

# 报错信息：

MapReduce Total cumulative CPU time:  minutes  seconds  msec
Ended Job = job_1510050683827_0137 with errors
Error during job, obtaining debugging information...
Examining task ID: task_1510050683827_0137_m_000002 (and more) from job job_1510050683827_0137

Task with the most failures():
-----
Task ID:
  task_1510050683827_0137_m_000000

URL:
  http://namenode:8088/taskdetails.jsp?jobid=job_1510050683827_0137&tipid=task_1510050683827_0137_m_000000
-----
Diagnostic Messages for this Task:
Error: Java heap space

FAILED: Execution Error, return code  from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
MapReduce Jobs Launched:
Stage-Stage-: Map:   Reduce:    Cumulative CPU: 106.07 sec   HDFS Read:  HDFS Write:  FAIL
Total MapReduce CPU Time Spent:  minutes  seconds  msec

# 原因分析：

报错显示 Error: Java heap space、return code  from org.apache.hadoop.hive.ql.exec.mr.MapRedTask

查资料说是因为内存的原因，由于HQL实际上是被转换成mapreduce的java任务，所以做了以下操作。

解决方法：

hadoop shell > vim etc/hadoop/hadoop-env.sh

# 默认
export HADOOP_HEAPSIZE=

hadoop shell > vim etc/hadoop/yarn-env.sh

# 默认
YARN_HEAPSIZE=

# 跟据实际情况，按需调整！

hadoop shell > vim etc/hadoop/mapred-site.xml

    <property>
        <name>mapreduce.map.memory.mb</name>
        <value></value>
    </property>

    <property>
        <name>mapreduce.map.java.opts</name>
        <value>-Xmx1024M</value>
    </property>

    <property>
        <name>mapreduce.reduce.memory.mb</name>
        <value></value>
    </property>

    <property>
        <name>mapreduce.reduce.java.opts</name>
        <value>-Xmx2560M</value>
    </property>

    <property>
        <name>mapreduce.task.io.sort.mb</name>
        <value></value>
    </property>

    <property>
        <name>mapreduce.task.io.sort.factor</name>
        <value></value>
    </property>

    <property>
        <name>mapreduce.reduce.shuffle.parallelcopies</name>
        <value></value>
    </property>

# 新增这些参数 ( 跟据机器实际情况，按需成倍调整 )
# 我的这个测试环境是4台8核8G的KVM虚拟机，一个NameNode，三个DataNode！

# 经过这次参数调整，目前600G的数据集上没出过问题，HDFS 上还在不断的写入历史数据、新数据。