记录一款Unity VR视频播放器插件的开发

效果图##

先上一个效果图：

背景

公司最近在做VR直播平台，VR开发我们用到了Unity，而在Unity中播放视频就需要一款视频插件，我们调研了几个视频插件，记录两个，如下：

Unity视频插件调研

网上搜了搜，最流行的有以下两款Unity插件：

• AVPro 这个在Unity商店售价150$，最新release版本为1.6.15,功能包括：

Powerful cross-platform video playback solution for Unity.

Native video playback on Android, iOS, macOS and tvOS (Apple TV), WebGL, Windows, Windows Phone and UWP.

Features include:

• New Unity 2017 supported
• New New iOS video playback path that uses less memory
• One API for video playback on all supported platforms
• Unity 4.6 - 5.x supported
• 8K video (on supported hardware)
• VR Support (mono, stereo, equirectangular and cubemap)
• Transparency support (native and packed)
• Subtitles support (external SRT)
• Fast flexible video playback
• In-editor playback support for Windows and macOS
• Free watermarked trial version available
• Components for IMGUI, uGUI and NGUI
• Over 64 PlayMaker actions included
• Easy to use drag and drop components
• Linear and Gamma colour spaces supported
• Fast native Direct3D, OpenGL and Metal texture updates
• Desktop support for Hap, Hap Alpha, Hap Q and Hap Q Alpha
• Streaming video from URL (when supported by platform)

此插件支持HLS视频播放,使用文档很详细，但是此插件没有源码，不适合做以后的个性化开发。

• 鼎鼎大名的EasyMovieTexture.售价65$，支持功能如下：

Supported resolutions:

• Android: General devices support up to 1920 * 1080.
  
  The latest device supports up to 4k.
• iOS: General devices support up to 1920 * 1080.
  
  The latest device is support up to 2560 * 1440.
  
  iPhone 6s Plus supports up to 4k.
• It also supports StreamingAssets, external storage, and streaming services.
• Android streaming support list: http, HLS (http live streaming),rtsp
• iOS streaming support list: http,HLS (http live streaming)
• EasyMovieTexture requires Android 4.0 or above.
• EasyMovieTexture requires iOS 6.0 or Above.
• Unity 4.X requires an iOS Pro.
• In Unity 5.X it does not require a Pro.
• Supports multithreaded rendering options. (Only supports Unity 5.X.)

这个插件貌似是个人开发的，没有说明文档，有部分java源码，native code并没有给出。我们需要有源码的插件方便以后的个性化开发。

自己动手，风衣足食##

综合以上调研结果，我们决定自己动手实现一个简单能满足我们要求的Unity播放器插件，有两个难点要突破：

• 一个是找一个合适的开源播放器。
• 另一个就是如何把播放视频画面映射到Unity中的物体表面，这个是最关键的。

寻找素材

从下面这个帖子中，找到了一些可以参考的资料。

unity 3d 中如何实现以物体的表面作为播放视频的位置，比如在墙面播放视频？

寻找开源播放器

本来打算使用VLC播放器的，但是同事发现有一个商用的开源播放器，并且使用的人数也不少，B站的ijkplayer。正好在上面的帖子中回复人也提到了这个播放器，我们决定使用这个播放器。

如何做视频画面映射

没有一点Unity开发经验，只能从头一点点学起，知乎的帖子里面，有个人回复可以参考OVR里面的例子。阅读了里面的代码，同时也参考了easyMovieTexture中的源码（easyMovie中只有java代码，关键的native code并没有给）。看的有些似懂非懂，尝试了之后，居然成功了。

最关键的一点我描述成下面的话：

将Ijkplayer的AndroidSurfaceTexture纹理ID和Unity中Texture2D的纹理ID分别同时绑定到不同的目标上。AndroidSurfaceTexture绑定到GL_TEXTURE_EXTERNAL_OES，Unity的纹理ID绑定到GL_TEXTURE_2D

从头到尾梳理一遍流程

初始化####

• Unity

Unity端初始化一个Texture2D纹理ID用于显示视频帧。

m_VideoTexture = new Texture2D (Call_GetVideoWidth (), Call_GetVideoHeight (), TextureFormat.RGB565, false);

• OVR

这里使用了OVR里面的native code，OVR中初始化AndroidSurfaceTexture和相关的函数：

static const char * className = "android/graphics/SurfaceTexture";
	const jclass surfaceTextureClass = jni->FindClass(className);
	if ( surfaceTextureClass == 0 ) {
		FAIL( "FindClass( %s ) failed", className );
	}

	// find the constructor that takes an int
	const jmethodID constructor = jni->GetMethodID( surfaceTextureClass, "<init>", "(I)V" );
	if ( constructor == 0 ) {
		FAIL( "GetMethodID( <init> ) failed" );
	}

	jobject obj = jni->NewObject( surfaceTextureClass, constructor, textureId );
	if ( obj == 0 ) {
		FAIL( "NewObject() failed" );
	}

	javaObject = jni->NewGlobalRef( obj );
	if ( javaObject == 0 ) {
		FAIL( "NewGlobalRef() failed" );
	}

	// Now that we have a globalRef, we can free the localRef
	jni->DeleteLocalRef( obj );

    updateTexImageMethodId = jni->GetMethodID( surfaceTextureClass, "updateTexImage", "()V" );
    if ( !updateTexImageMethodId ) {
    	FAIL( "couldn't get updateTexImageMethodId" );
    }

    getTimestampMethodId = jni->GetMethodID( surfaceTextureClass, "getTimestamp", "()J" );
    if ( !getTimestampMethodId ) {
    	FAIL( "couldn't get getTimestampMethodId" );
    }

	setDefaultBufferSizeMethodId = jni->GetMethodID( surfaceTextureClass, "setDefaultBufferSize", "(II)V" );
    if ( !setDefaultBufferSizeMethodId ) {
		FAIL( "couldn't get setDefaultBufferSize" );
    }

	// jclass objects are localRefs that need to be freed
	jni->DeleteLocalRef( surfaceTextureClass );

初始化纹理ID，并将其绑定到目标GL_TEXTURE_2D上:

glGenTextures( 1, &textureId );
	glBindTexture( GL_TEXTURE_EXTERNAL_OES, textureId );
	glTexParameterf( GL_TEXTURE_EXTERNAL_OES, GL_TEXTURE_MIN_FILTER, GL_LINEAR );
	glTexParameterf( GL_TEXTURE_EXTERNAL_OES, GL_TEXTURE_MAG_FILTER, GL_LINEAR );
	glTexParameterf( GL_TEXTURE_EXTERNAL_OES, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE );
	glTexParameterf( GL_TEXTURE_EXTERNAL_OES, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE );
	glBindTexture( GL_TEXTURE_EXTERNAL_OES, 0 );

将Unity的纹理ID传递到OVR中，用于绑定到目标GL_TEXTURE_EXTERNAL_OES上：

jobject OVR_Media_Surface( void * texPtr, int const width, int const height )
{
	GLuint texId = (GLuint)(size_t)(texPtr);
	LOG( "OVR_Media_Surface(%i, %i, %i)", texId, width, height );
	return _msp.VideoSurface.Bind( texId, width, height );
}

• Ijkplayer

创建一个播放器，注意这里我们使用OVR中已经实例化的AndroidMovieTexture来初始化播放器。

 m_IjkMediaPlayer.setSurface(m_Surface);

刷新####

刷新操作由Unity中的Update函数触发，最终在OVR中执行，首先调用AndroidMovieTexture中的Update函数，接下来就是绑定纹理操作，Ijkplayer的纹理ID每刷新一次绑定一次。而Unity的纹理ID只有在视频图像长度或者宽度发生变化才会绑定。

void MediaSurface::Update()
{
	if ( !AndroidSurfaceTexture )
	{
		LOG( "!AndroidSurfaceTexture" );
		return;
	}
	if ( TexId <= 0 )
	{
		//LOG( "TexId <= 0" );
		return;
	}
	AndroidSurfaceTexture->Update();
	if ( AndroidSurfaceTexture->GetNanoTimeStamp() == LastSurfaceTexNanoTimeStamp )
	{
		//LOG( "No new surface!" );
		return;
	}
	LastSurfaceTexNanoTimeStamp = AndroidSurfaceTexture->GetNanoTimeStamp()

   // If the SurfaceTexture has changed dimensions, we need to
	// reallocate the texture and FBO.
	glActiveTexture( GL_TEXTURE0 );
	glBindTexture( GL_TEXTURE_EXTERNAL_OES, AndroidSurfaceTexture->GetTextureId() );
	if ( TexIdWidth != BoundWidth || TexIdHeight != BoundHeight )
	{
		LOG( "New surface size: %ix%i", BoundWidth, BoundHeight );

		TexIdWidth = BoundWidth;
		TexIdHeight = BoundHeight;

		if ( Fbo )
		{
			glDeleteFramebuffers( 1, &Fbo );
		}

		glActiveTexture( GL_TEXTURE1 );
		glBindTexture( GL_TEXTURE_2D, TexId );
		glTexImage2D( GL_TEXTURE_2D, 0, GL_RGBA,
				TexIdWidth, TexIdHeight, 0, GL_RGBA, GL_UNSIGNED_BYTE, NULL );
		glTexParameteri( GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE );
		glTexParameteri( GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE );
		glTexParameteri( GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR_MIPMAP_LINEAR );
		glTexParameteri( GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR );

		glBindTexture( GL_TEXTURE_2D, 0 );
		glActiveTexture( GL_TEXTURE0 );

		glGenFramebuffers( 1, &Fbo );
		glBindFramebuffer( GL_FRAMEBUFFER, Fbo );
		glFramebufferTexture2D( GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_2D,
				TexId, 0 );
		glBindFramebuffer( GL_FRAMEBUFFER, 0 );
	}
}

最后的结果可能是这个样子的：Ijkplayer负责推动视频不停向前播放，播放器的纹理也会不停刷新，这会带动Unity纹理跟着刷新，最终显示在Unity的Material上。