Python测试进阶——（6）Bash脚本启动Python监控程序并传递PID

用HiBench执行Hadoop——Sort测试用例，进入 /HiBench-master/bin/workloads/micro/sort/hadoop 目录下，执行命令：

[root@node1 hadoop]# ./run.sh

执行后返回如下信息：

[root@node1 hadoop]# ./run.sh
patching args=　　#enter_bench()
Parsing conf: /home/cf/app/HiBench-master/conf/hadoop.conf
Parsing conf: /home/cf/app/HiBench-master/conf/hibench.conf
Parsing conf: /home/cf/app/HiBench-master/conf/spark.conf
Parsing conf: /home/cf/app/HiBench-master/conf/workloads/micro/sort.conf
probe -.cdh5./lib/hadoop/../../jars/hadoop-mapreduce-client-jobclient--cdh5.14.2-tests.jar
start HadoopSort bench
hdfs -.cdh5./bin/hadoop --config /etc/hadoop/conf.cloudera.yarn fs -rm -r -skipTrash hdfs://node1:8020/HiBench/Sort/Output
Deleted hdfs://node1:8020/HiBench/Sort/Output　　#rmr_hdfs()
hdfs -.cdh5./bin/hadoop --config /etc/hadoop/conf.cloudera.yarn fs -du -s hdfs://node1:8020/HiBench/Sort/Input
Submit MapReduce Job: /opt/cloudera/parcels/CDH--.cdh5./bin/hadoop --config /etc/hadoop/conf.cloudera.yarn jar /opt/cloudera/parcels/CDH--.cdh5./lib/hadoop/../../jars/hadoop-mapreduce-examples--cdh5. hdfs://node1:8020/HiBench/Sort/Input hdfs://node1:8020/HiBench/Sort/Output #run_hadoop_job()
The job took  seconds.
finish HadoopSort bench

发现HiBench执行Python监控程序脚本的命令为：

UID        PID  PPID  C STIME TTY          TIME CMD

root             : pts/    :: python2 /home/cf/app/HiBench-master/bin/functions/monitor.py HadoopSort  /home/cf/app/HiBench-master/report/sort/hadoop/conf/../monitor.log /home/cf/app/H

root         : pts/    :: python2 /home/cf/app/HiBench-master/bin/functions/execute_with_log.py /home/cf/app/HiBench-master/report/sort/hadoop/conf/../bench.log /opt/cloudera/parcels/CD

查看文件run.sh的内容：

  current_dir=`dirname "$0"`
  current_dir=`cd "$current_dir"; pwd`
  root_dir=${current_dir}/../../../../../
  workload_config=${root_dir}/conf/workloads/micro/sort.conf
  . "${root_dir}/bin/functions/load_bench_config.sh"

  enter_bench HadoopSort ${workload_config} ${current_dir}
  show_bannar start

  rmr_hdfs $OUTPUT_HDFS || true

  SIZE=`dir_size $INPUT_HDFS`
  START_TIME=`timestamp`
  run_hadoop_job ${HADOOP_EXAMPLES_JAR} sort -outKey org.apache.hadoop.io.Text -outValue org.apache.hadoop.io.Text -r ${NUM_REDS} ${INPUT_HDFS} ${OUTPUT_HDFS}

  END_TIME=`timestamp`
  gen_report ${START_TIME} ${END_TIME} ${SIZE}
  show_bannar finish
  leave_bench

在文件run.sh中，发现 run_hadoop_job() 调用了 start_monitor 方法：

function run_hadoop_job(){
    ENABLE_MONITOR=
    if [ "$1" = "--without-monitor" ]; then
        ENABLE_MONITOR=

    fi
    local job_jar=$
    shift
    local job_name=$
    shift
    local tail_arguments=$@
    local CMD="${HADOOP_EXECUTABLE} --config ${HADOOP_CONF_DIR} jar $job_jar $job_name $tail_arguments"
    echo -e "${BGreen}Submit MapReduce Job: ${Green}$CMD${Color_Off}"
     ]; then
        MONITOR_PID=`start_monitor`
    fi
    execute_withlog ${CMD}
    result=$?
     ]; then
        stop_monitor ${MONITOR_PID}
    fi
     ]; then
        echo -e "${BRed}ERROR${Color_Off}: Hadoop job ${BYellow}${job_jar} ${job_name}${Color_Off} failed to run successfully."
        echo -e "${BBlue}Hint${Color_Off}: You can goto ${BYellow}${WORKLOAD_RESULT_FOLDER}/bench.log${Color_Off} to check for detailed log.\nOpening log tail for you:\n"
        tail ${WORKLOAD_RESULT_FOLDER}/bench.log
        exit $result
    fi
}

查看 start_monitor 方法的定义：

function start_monitor(){
    MONITOR_PID=`${workload_func_bin}/monitor.py ${HIBENCH_CUR_WORKLOAD_NAME} $$ ${WORKLOAD_RESULT_FOLDER}/monitor.log ${WORKLOAD_RESULT_FOLDER}/bench.log ${WORKLOAD_RESULT_FOLDER}/monitor.html ${SLAVES} &`
#    echo "start monitor, got child pid:${MONITOR_PID}" > /dev/stderr
    echo ${MONITOR_PID}
}

还有 stop_monitor  方法的定义：

function stop_monitor(){
    MONITOR_PID=$
    assert $ "monitor pid missing"
#    echo "stop monitor, kill ${MONITOR_PID}" > /dev/stderr
    kill ${MONITOR_PID}
}

以及 execute_withlog 方法的定义：

function execute_withlog () {
    CMD="$@"
     ] ; then          # Terminal, beautify the output.
        ${workload_func_bin}/execute_with_log.py ${WORKLOAD_RESULT_FOLDER}/bench.log $CMD
    else                        # pipe, do nothing.
        $CMD
    fi
}

在 run.sh 中加入以下三行：

  echo "PID of this script: $$"
  echo "PPID of this script: $PPID"
  echo "UID of this script: $UID"

文件的部分内容如下：

  SIZE=`dir_size $INPUT_HDFS`
  START_TIME=`timestamp`
  run_hadoop_job ${HADOOP_EXAMPLES_JAR} sort -outKey org.apache.hadoop.io.Text -outValue org.apache.hadoop.io.Text -r ${NUM_REDS} ${INPUT_HDFS} ${OUTPUT_HDFS}

  echo "PID of this script: $$"
  echo "PPID of this script: $PPID"
  echo "UID of this script: $UID"

  END_TIME=`timestamp`
  gen_report ${START_TIME} ${END_TIME} ${SIZE}
  show_bannar finish
  leave_bench

执行后返回如下信息：

PID of this script:
PPID of this script:
UID of this script: 

用 ps -ef 查看发现 18804 和 32331 分别对应如下进程：

UID        PID  PPID  C STIME TTY          TIME CMD
root         : pts/    :: -bash
root         : pts/    :: /bin/bash ./run_bak.sh

经查找，发现：

在bash中，子shell进程的PID存储在一个特殊的变量\$\$中。这个变量只读，你不可以在脚本中修改它。除了\$\$, bash shell还会导出其他的只读变量。比如，\$PPID存储子shell父进程的ID（也就是主shell）。\$UID存储了执行这个脚本的当前用户ID。上面输出中，PID每次执行都会变化。这个因为每次运行都会创建一个新的shell。另一方面，PPID每次都会一样只要你在同一个shell中运行。

分析 monitor.py 发现如下方法调用：

pid=os.fork()

分析 execute_with_log.py 发现如下方法调用：

proc = subprocess.Popen(" ".join(command_lines), shell=True, bufsize=1, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)

经过思考，利用 subprocess.Popen( ) 方法启动子进程并获取子进程返回值，子进程利用 \$\$ 变量获取PID并作为返回值传递给父进程，父进程可以启动监控程序记录子进程的运行数据。

参考：

https://www.jb51.net/article/62370.htm

https://www.cnblogs.com/ratels/p/11039813.html

https://www.cnblogs.com/ratels/p/11070615.html

https://www.cnblogs.com/zhoug2020/p/5079407.html