EasyPlayer Android基于ffmpeg实现播放（RTSP/RTMP/HTTP/HLS）同步录像功能

之前有博客专门介绍了EasyPlayer的本地录像的功能,简单来说,EasyPlayer是一款RTSP播放器,它将RTSP流里的音视频媒体帧解析出来,并用安卓系统提供的MediaMuxer类进行录像.那EasyPlayerPro可以这样实现吗?答案是不太现实,因为Pro支持绝大多数的流媒体协议,并不单单是RTSP协议,包括hTTP\RTSP\RTMP\HLS\FILE等格式都支持.要将这些数据分别解析出来并抛给上层,并不合适.EasyPlayerPro最终是基于FFMPEG来做的媒体流的解析(demux),既然用FFMPEG来demux,那同样也可以基于FFMPEG来实现录像.录像在FFMPEG里面,就是mux的过程.

作者参考了ffmpeg mux的相关代码,在Pro上成功实现了播放同时的录像功能.现在尚处于测试阶段,此文亦起到一个记录与总结的目的.

EasyPlayerPro是参考ffplay的实现,开启专门的接收线程来接收和解析音视频媒体帧.我们可以在收到媒体帧后进行MUX操作.

Created with Raphaël 2.1.0av_read_frameav_read_frameav_interleaved_write_frameav_interleaved_write_frameMUX

在接收线程中,我们可以设置一个录像标志位:recording.我们设定该值为0的话表示未在录像;1表示要启动录像;2表示录像正在启动,等待关键帧;大于2表示正在录像正在启动中.这里不再多说了,结合代码来做叙述.

请看相关代码以及注释:

接收线程读取媒体帧.

ret = av_read_frame(ic, pkt);
if (ret < 0) {
    .. // error handle
}

录像开始

外部线程更改read thread的recording状态量来控制录像的状态.录像开始时,将状态位设置为1,这时read thread会进行录像的一些初始化操作:

START_RECORDING:
    if (is->recording == 1){    // 录像即将启动
        do
        {
            // 首先我们判断一下,当前如果是视频的话,需要为关键帧.
            av_log(NULL, AV_LOG_INFO, "try start record:%s", is->record_filename);
            AVStream *ins = ic->streams[pkt->stream_index];
            if (ins->codec->codec_type == AVMEDIA_TYPE_VIDEO ){
                av_log(NULL, AV_LOG_DEBUG, "check video  key frame.");
                if (!(pkt->flags & AV_PKT_FLAG_KEY)){   // 不是关键帧,跳出录像块.下次继续尝试.
                    av_log(NULL,AV_LOG_WARNING,"waiting for key frame of video stream:%d.", pkt->stream_index);
                    break;
                }
                is->recording++;
            }
            // 至此,已经找到了首个关键帧,录像可以启动了.
            av_log(NULL, AV_LOG_INFO, "start record:%s", is->record_filename);
            // 创建AVFormatContext,作为录像的Context
            avformat_alloc_output_context2(&o_fmt_ctx, NULL, NULL, is->record_filename);
            if (!o_fmt_ctx){    // 创建失败.
                is->recording = 0;
                av_log(NULL, AV_LOG_WARNING, "avformat_alloc_output_context2 error");
                o_fmt_ctx = is->oc = NULL;
                goto START_RECORDING;
            }
            ofmt = o_fmt_ctx->oformat;
            // 在这里遍历所有的媒体流
            for (i = 0; i < ic->nb_streams; i++) {
                // 目前MP4Muxer支持的AV_CODEC类型,我们在这里判断下,加一些打印日志.这些代码摘自FFMPEG的muxer部分
                AVStream *in_stream = ic->streams[i];
                AVCodecParameters *par = in_stream->codecpar;
                unsigned int tag = 0;
                if      (par->codec_id == AV_CODEC_ID_H264)      tag = MKTAG('a','v','c','1');
                else if (par->codec_id == AV_CODEC_ID_HEVC)      tag = MKTAG('h','e','v','1');
                else if (par->codec_id == AV_CODEC_ID_VP9)       tag = MKTAG('v','p','0','9');
                else if (par->codec_id == AV_CODEC_ID_AC3)       tag = MKTAG('a','c','-','3');
                else if (par->codec_id == AV_CODEC_ID_EAC3)      tag = MKTAG('e','c','-','3');
                else if (par->codec_id == AV_CODEC_ID_DIRAC)     tag = MKTAG('d','r','a','c');
                else if (par->codec_id == AV_CODEC_ID_MOV_TEXT)  tag = MKTAG('t','x','3','g');
                else if (par->codec_id == AV_CODEC_ID_VC1)       tag = MKTAG('v','c','-','1');
                else if (par->codec_id == AV_CODEC_ID_DVD_SUBTITLE)  tag = MKTAG('m','p','4','s');
                av_log(NULL, AV_LOG_INFO, "par->codec_id:%d, tag:%d\n", par->codec_id, tag);
                if (tag == 0) {
                    // 这个CODEC不支持,打印下.
                    av_log(NULL, AV_LOG_WARNING, "unsupported codec codec_id:%d\n", par->codec_id);
                    // continue;
                }
            // av_log(NULL, AV_LOG_INFO, "-ffplay : %d", __LINE__);
                //Create output AVStream according to input AVStream
                if(ic->streams[i]->codec->codec_type ==AVMEDIA_TYPE_VIDEO){     // 这个是视频帧
                    // 我们在这里检查一下宽\高是否合法.
                    if ((par->width <= 0 || par->height <= 0) &&
                        !(ofmt->flags & AVFMT_NODIMENSIONS)) {
                        av_log(NULL, AV_LOG_ERROR, "dimensions not set\n");
                        continue;
                    }
                    // 加入视频流至Muxer.
                    AVStream *out_stream = avformat_new_stream(o_fmt_ctx, in_stream->codec->codec);
                    // 将视频流的一些参数拷贝到muxer.
                    if (avcodec_copy_context(out_stream->codec, in_stream->codec) < 0) {
                        // 失败了,做一些错误处理\释放.
                        // printf( "Failed to copy context from input to output stream codec context\n");
                        av_log(NULL, AV_LOG_WARNING,
                            "Failed to copy context from input to output stream codec context\n");
                        is->recording = 0;
                        avformat_free_context(o_fmt_ctx);
                        o_fmt_ctx = is->oc = NULL;
                        goto START_RECORDING;
                    }
                    // av_log(NULL, AV_LOG_INFO, "-ffplay:%d out_stream:%p, in_stream:%p", __LINE__, out_stream->codec, in_stream->codec);
                    out_stream->codec->codec_tag = 0;
                    if (o_fmt_ctx->oformat->flags & AVFMT_GLOBALHEADER)
                        out_stream->codec->flags |= CODEC_FLAG_GLOBAL_HEADER;
                    av_log(NULL, AV_LOG_INFO, "-ffplay : %d video added", __LINE__);
                }else if(ic->streams[i]->codec->codec_type==AVMEDIA_TYPE_AUDIO){  //    这个是音频帧
                    av_log(NULL, AV_LOG_INFO, "-ffplay : %d", __LINE__);
                    // 我们检查下采样率是否合法?
                    if (par->sample_rate <= 0) {
                        av_log(NULL, AV_LOG_ERROR, "sample rate not set\n");
                        continue;
                    }
                    // 加入音频流至Muxer.
                    AVStream *out_stream = avformat_new_stream(o_fmt_ctx, in_stream->codec->codec);
                    // 将音频流的一些参数拷贝到muxer.
                    if (avcodec_copy_context(out_stream->codec, in_stream->codec) < 0) {
                        // 失败了,做一些错误处理\释放.
                        av_log(NULL, AV_LOG_WARNING,
                            "Failed to copy context from input to output stream codec context 2\n");
                        is->recording = 0;
                        avformat_free_context(o_fmt_ctx);
                        o_fmt_ctx = is->oc = NULL;
                        goto START_RECORDING;
                    }
                    out_stream->codec->codec_tag = 0;
                    if (o_fmt_ctx->oformat->flags & AVFMT_GLOBALHEADER)
                        out_stream->codec->flags |= CODEC_FLAG_GLOBAL_HEADER;
                    av_log(NULL, AV_LOG_INFO, "-ffplay : %d audio added", __LINE__);
                }
            }
            // 至此,AVFormatContext里面应该至少含有一条流了.否则录像也算没有启动.
            if (o_fmt_ctx->nb_streams < 1){
                av_log(NULL, AV_LOG_WARNING,
                    "NO available stream found in muxer \n");
                is->recording = 0;
                avformat_free_context(o_fmt_ctx);
                o_fmt_ctx = is->oc = NULL;
                goto START_RECORDING;
            }
            // 下面开始创建文件
            if (!(ofmt->flags & AVFMT_NOFILE)){
                av_log(NULL, AV_LOG_INFO, "-ffplay : %d AVFMT_NOFILE", __LINE__);
                if (avio_open(&o_fmt_ctx->pb, is->record_filename, AVIO_FLAG_WRITE) < 0) {
                    // 出错了,错误处理.
                    av_log(NULL,AV_LOG_WARNING, "Could not open output file '%s'", is->record_filename);
                    is->recording = 0;
                    avformat_free_context(o_fmt_ctx);
                    o_fmt_ctx = is->oc = NULL;
                    goto START_RECORDING;
                }
            }
            // 下面写入header. Allocate the stream private data and write the stream header to an output media file.
            int r = avformat_write_header(o_fmt_ctx, NULL);
            if (r < 0) {    // error handle
                av_log(NULL,AV_LOG_WARNING, "Error occurred when opening output file:%d\n",r);
                is->recording = 0;
                avformat_free_context(o_fmt_ctx);
                o_fmt_ctx = is->oc = NULL;
                goto START_RECORDING;
            }
            // 输出一下OUTPUT格式.
            av_dump_format(o_fmt_ctx, 0, is->record_filename, 1);
            // 将标志位改为2,表示录像已经启动.开始等待关键帧.
            is->recording = 2;
        }
        while(0);
    }

录像中

当录像初始化完成后,状态量会变为2,状态改为录像中:

    do{
        if (is->recording >= 2){
            // 忽略未被加入的stream
            if (pkt->stream_index >= o_fmt_ctx->nb_streams)
            {
                av_log(NULL,AV_LOG_WARNING,"stream_index large than nb_streams %d:%d\n", pkt->stream_index,  o_fmt_ctx->nb_streams);
                break;
            }

            AVStream *ins = ic->streams[pkt->stream_index];
            av_log(NULL,AV_LOG_DEBUG,"before write frame.stream index:%d, codec:%d,type:%d\n", ins->index, ins->codec->codec_id,  ins->codec->codec_type);
            if(is->recording == 2)  // 等待关键帧..
            if (ins->codec->codec_type == AVMEDIA_TYPE_VIDEO ){
                av_log(NULL, AV_LOG_DEBUG, "check video  key frame.");
                if (!(pkt->flags & AV_PKT_FLAG_KEY)){
                    av_log(NULL,AV_LOG_WARNING,"waiting for key frame of video stream:%d.", pkt->stream_index);
                    break;
                }
                // 表示关键帧得到了.
                is->recording++;
            }
            // 将接收到的AVPacket拷贝一份
            AVPacket *newPkt = av_packet_clone(pkt);
            AVStream *in_stream, *out_stream;
            in_stream  = ic->streams[newPkt->stream_index];
            out_stream = o_fmt_ctx->streams[newPkt->stream_index];
            // 下面转换一下PTS和DTS.这几句的意思大概是,将原先以输入的time_base为基准的时间戳转换为以输出的time_base为基准的时间戳,
            // 要把这几句加上,最终录像时间戳才正常..
            //Convert PTS/DTS
            newPkt->pts = av_rescale_q_rnd(newPkt->pts, in_stream->time_base, out_stream->time_base, AV_ROUND_NEAR_INF|AV_ROUND_PASS_MINMAX);
            newPkt->dts = av_rescale_q_rnd(newPkt->dts, in_stream->time_base, out_stream->time_base, AV_ROUND_NEAR_INF|AV_ROUND_PASS_MINMAX);
            newPkt->duration = av_rescale_q(newPkt->duration, in_stream->time_base, out_stream->time_base);
            // 下面开始写入AVPacket.
            int r;
            if (o_fmt_ctx->nb_streams < 2){ // 如果只有视频或只有音频,那直接写入
                r = av_write_frame(o_fmt_ctx, newPkt);
            }else {                         // 多条流的情况下,调用av_interleaved_write_frame,内部会做一些排序,再写入.
                r = av_interleaved_write_frame(o_fmt_ctx, newPkt);
            }
            if (r < 0) {
                printf( "Error muxing packet\n");
                break;
            }
        }
    }while(0);

停止录像

外部控制recording状态量,置为0时,表示要停止录像了.这时候做一些反初始化的工作:

    if (is->recording == 0){    // 要停止录像了.
        if (o_fmt_ctx != NULL){
            av_log(NULL, AV_LOG_INFO, "stop record~~~~~~~");
            // 一定要先写入trailer
            av_write_trailer(o_fmt_ctx);
            // 释放context
            avformat_free_context(o_fmt_ctx);
            o_fmt_ctx = is->oc = NULL;
        }
    }

EasyPlayerPro下载地址：https://fir.im/EasyPlayerPro

相关介绍见：http://www.easydarwin.org/article/news/117.html

获取更多信息

邮件：support@easydarwin.org

WEB：www.EasyDarwin.org

QQ交流群：587254841

Copyright © EasyDarwin.org 2012-2017