YUV display in OpenGL

http://stackoverflow.com/questions/1106741/need-help-with-yuv-display-in-opengl 

I am having trouble displaying a raw YUV file that it is in NV12 format.

I can display a selected frame, however, it is still mainly in black and white with certain shades of pink and green.

Here is how my output looks like 

Anyways, here is how my program works. (This is done in cocoa/objective-c, but I need your expert advice on program algorithm, not on syntax.)

Prior to program execution, the YUV file is stored in a binary file named "test.yuv". The file is in NV12 format, meaning the Y plan is stored first, then the UV plan is interlaced. My file extraction has no problem because I did a lot testings on it.

During initiation, I create a lookup table that will convert binary/8 bites/a byte/chars into respected Y, U, V float values

For the Y plane this is my code

-(void)createLookupTableY //creates a lookup table for converting a single byte into a float between 0 and 1{ NSLog(@"YUVFrame: createLookupTableY");     lookupTableY = new float [256]; for (int i = 0; i < 256; i++) {     	lookupTableY[i] = (float) i /255; //NSLog(@"lookupTableY[%d]: %f",i,lookupTableY[i]);//prints out the value of each float }}

The U Plane lookup table

-(void)createLookupTableU //creates a lookup table for converting a single byte into a float between 0 and 1{ NSLog(@"YUVFrame: createLookupTableU");     lookupTableU = new float [256]; for (int i = 0; i < 256; i++) {     	lookupTableU[i] = -0.436 + (float) i / 255* (0.436*2); NSLog(@"lookupTableU[%d]: %f",i,lookupTableU[i]);//prints out the value of each float }}

And the V look-up table

-(void)createLookupTableV //creates a lookup table for converting a single byte into a float between 0 and 1{ NSLog(@"YUVFrame: createLookupTableV");     lookupTableV = new float [256]; for (int i = 0; i < 256; i++) {     	lookupTableV[i] = -0.615 + (float) i /255 * (0.615*2); NSLog(@"lookupTableV[%d]: %f",i,lookupTableV[i]);//prints out the value of each float }}

after this point, I extract the Y & UV plan and store them into two buffers, yBuffer & uvBuffer

at this point, I attempt to convert the YUV data and stored it into a RGB buffer array called "frameImage"

-(void)sortAndConvert//sort the extracted frame data into an array of float{ NSLog(@"YUVFrame: sortAndConvert"); int frameImageCounter = 0; int pixelCounter = 0; for (int y = 0; y < YUV_HEIGHT; y++)//traverse through the frame's height { for ( int x = 0; x < YUV_WIDTH; x++)//traverse through the frame's width { float Y = lookupTableY [yBuffer [y*YUV_WIDTH + x] ]; float U = lookupTableU [uvBuffer [ ((y / 2) * (YUV_WIDTH / 2) + (x/2)) * 2 ] ]; float V = lookupTableV [uvBuffer [ ((y / 2) * (YUV_WIDTH / 2) + (x/2)) * 2 + 1] ]; float RFormula = Y + 1.13983f * V; float GFormula = Y - 0.39465f * U - 0.58060f * V; float BFormula = Y + 2.03211f * U;      		 frameImage [frameImageCounter++] = [self clampValue:RFormula];     		 frameImage [frameImageCounter++] = [self clampValue:GFormula];     		 frameImage [frameImageCounter++] = [self clampValue:BFormula]; } }}

then I tried to draw the Image in OpenGl

-(void)drawFrame:(int )x { GLuint texture;      glGenTextures(1, &texture);     glBindTexture(GL_TEXTURE_2D, texture);     glTexImage2D(GL_TEXTURE_2D, 0, GL_RGB, YUV_WIDTH, YUV_HEIGHT, 0, GL_RGB, GL_FLOAT, frameImage);      glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);      glEnable(GL_TEXTURE_2D);     glBindTexture(GL_TEXTURE_2D, texture);     glRotatef( 180.0f, 1.0f, 0.0f, 0.0f );      glBegin( GL_QUADS );     glTexCoord2d(0.0,0.0); glVertex2d(-1.0,-1.0);     glTexCoord2d(1.0,0.0); glVertex2d(+1.0,-1.0);     glTexCoord2d(1.0,1.0); glVertex2d(+1.0,+1.0);     glTexCoord2d(0.0,1.0); glVertex2d(-1.0,+1.0);     glEnd();      glFlush();}

so basically this my program in a nut shell. essentially i read the binary YUV files, stores all the data in a char array buffer. i then translate these values into their respected YUV float values.

This is where I think the error might be: in the Y lookup table I standardize the Y plane to [0,1], in the U plane I standardized the values between [-0.435,0.436], and in the V plane I standardized it bewteen [-0.615,0.615]. I did this because those are the YUV value ranges according to wikipedia.

And the YUV to RGB formula is the same formula from Wikipedia. I have also tried various other formulas, and this is the only formula that gives the rough outlook of the frame. Anyone might venture to guess to why my program is not correctly displaying the YUV frame data. I think it is something to do with my standardization technique, but it seems alright to me.

I have done a lot of testings, and I am 100% certain that the error is caused by by look up table. I don't think my setting formulas are correct.

A note to everyone who is reading this. For the longest time, my frame was not displaying properly because I was not able to extract the frame data correctly.

When I first started to program, I was under the impression that in a clip of say 30 frames, all 30 Y planar datas are first written in the data file, followed then by UV plane datas.

What I found out through trial and error was that for a YUV data file, specifically NV12, the data file is stored in the following fashion

Y(1) UV(1) Y(2) UV(2) ... ...

@nschmidt

I changed my code to what you suggested

float U = lookupTableU [uvBuffer [ (y * (YUV_WIDTH / 4) + (x/4)) * 2 ] ]float V = lookupTableU [uvBuffer [ (y * (YUV_WIDTH / 4) + (x/4)) * 2 + 1] ]

and this is the result that i get 

here is the print line from the console. i am print out the values for Y, U, V and RGB value that are being translated and stored on in the frameImage array

YUV:[0.658824,-0.022227,-0.045824] RGBFinal:[0.606593,0.694201,0.613655]

YUV:[0.643137,-0.022227,-0.045824] RGBFinal:[0.590906,0.678514,0.597969]

YUV:[0.607843,-0.022227,-0.045824] RGBFinal:[0.555612,0.643220,0.562675]

YUV:[0.592157,-0.022227,-0.045824] RGBFinal:[0.539926,0.627534,0.546988]

YUV:[0.643137,0.025647,0.151941] RGBFinal:[0.816324,0.544799,0.695255]

YUV:[0.662745,0.025647,0.151941] RGBFinal:[0.835932,0.564406,0.714863]

YUV:[0.690196,0.025647,0.151941] RGBFinal:[0.863383,0.591857,0.742314]

Update July 13, 2009

The problem was finally solved thanks to the recommendation from nschmidt . It turns out that my YUV file was actually in YUV 4:1:1 format. I was originally told that it was in YUV NV12 format. Anyways, I would like to share my results with you.

Here is output

and my code for decode is as follows

float Y = (float) yBuffer [y*YUV_WIDTH + x] ;float U = (float) uvBuffer [ ((y / 2) * (YUV_WIDTH  / 2) + (x/2)) ] ; float V = (float) uvBuffer [ ((y / 2) * (YUV_WIDTH  / 2) + (x/2)) + UOffset] ;float RFormula = (1.164*(Y-16) + (1.596* (V - 128) ));float GFormula = ((1.164 * (Y - 16)) - (0.813 * ((V) - 128)) - (0.391 * ((U) - 128)));float BFormula = ((1.164 * (Y - 16)) + (2.018 * ((U) - 128)));   		frameImage [frameImageCounter] = (unsigned char)( (int)[self clampValue:RFormula]); 		frameImageCounter ++; 		frameImage [frameImageCounter] = (unsigned char)((int)[self clampValue:GFormula]); 		frameImageCounter++; 		frameImage [frameImageCounter] = (unsigned char)((int) [self clampValue:BFormula]); 		frameImageCounter++;GLuint texture;  glGenTextures(1, &texture); glEnable(GL_TEXTURE_2D);  glBindTexture(GL_TEXTURE_2D, texture); glTexParameteri(GL_TEXTURE_2D, GL_GENERATE_MIPMAP_SGIS, GL_TRUE);  glTexImage2D(GL_TEXTURE_2D, 0, GL_RGB, YUV_WIDTH, YUV_HEIGHT, 0, GL_RGB, GL_UNSIGNED_BYTE, frameImage);//glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR); glTexParameteri(GL_TEXTURE_2D,GL_TEXTURE_WRAP_S,GL_CLAMP_TO_EDGE_SGIS); glTexParameteri(GL_TEXTURE_2D,GL_TEXTURE_WRAP_T,GL_CLAMP_TO_EDGE_SGIS); glTexParameteri(GL_TEXTURE_2D,GL_TEXTURE_MAG_FILTER,GL_LINEAR); glTexParameteri(GL_TEXTURE_2D,GL_TEXTURE_MIN_FILTER,GL_LINEAR_MIPMAP_LINEAR);  glRotatef( 180.0f, 1.0f, 0.0f, 0.0f );  glBegin( GL_QUADS ); glTexCoord2d(0.0,0.0); glVertex2d(-1.0,-1.0); glTexCoord2d(1.0,0.0); glVertex2d(+1.0,-1.0); glTexCoord2d(1.0,1.0); glVertex2d(+1.0,+1.0); glTexCoord2d(0.0,1.0); glVertex2d(-1.0,+1.0); glEnd();NSLog(@"YUVFrameView: drawRect complete"); glFlush();

essentially, I used NSData for the raw file extraction. stored in a char array buffer. For YUV to RGB conversion, I used the above formula, afterwards, I clamped the values to [0:255]. then I used a 2DTexture in OpenGL for display.

So if you are converting YUV to RGB in the future, use the formula from above. If are using the YUV to RGB conversion formula from the earlier post, then you need to display the texture in GL_Float from the values for RGB are clamped between [0:1]

objective-c opengl rgb yuv

share|improve this question

<div "="" style="margin: 0px; padding: 0px; border: 0px; vertical-align: baseline; height: 35px; width: 185px; background: transparent;"> edited Jul 13 '09 at 15:56

community wiki

10 revs, 2 users 100%ReachConnection

add a comment

3 Answers

activeoldestvotes

up vote7down voteaccepted

Next try :) I think you're uv buffer is not interleaved. It looks like the U values come first and then the array of V values. Changing the lines to

unsigned int voffset = YUV_HEIGHT * YUV_WIDTH / 2;float U = lookupTableU [uvBuffer [ y * (YUV_WIDTH / 2) + x/2] ]; float V = lookupTableV [uvBuffer [ voffset + y * (YUV_WIDTH / 2) + x/2] ];

might indicate if this is really the case.

share|improve this answer

<div "="" style="margin: 0px; padding: 0px; border: 0px; vertical-align: baseline; height: 35px; width: 185px; background: transparent;"> answered Jul 11 '09 at 12:12 nschmidt 1,816919

wow, I tested out my program with your adjustment and it finally worked. thanks a lot man.it turns out that you were right, the U & V plane are not interlaced as I was originally told. thank you so much nschmidt. I'm rating you up –  ReachConnection Jul 13 '09 at 15:07

no problem :). for the record the first guess about the format being 4:1:1 was actually incorrect. You have 4:2:2 format. The real problem was only your data not being interleaved. –  nschmidt Jul 14 '09 at 17:54

add a comment

up vote3down vote

I think you're addressing U and V values incorrectly. Rather than:

float U = lookupTableU [uvBuffer [ ((y / 2) * (x / 2) + (x/2)) * 2 ] ]; float V = lookupTableV [uvBuffer [ ((y / 2) * (x / 2) + (x/2)) * 2 + 1] ];

It should be something along the lines of

float U = lookupTableU [uvBuffer [ ((y / 2) * (YUV_WIDTH / 2) + (x/2)) * 2 ] ]; float V = lookupTableV [uvBuffer [ ((y / 2) * (YUV_WIDTH / 2) + (x/2)) * 2 + 1] ];

share|improve this answer

<div "="" style="margin: 0px; padding: 0px; border: 0px; vertical-align: baseline; height: 35px; width: 185px; background: transparent;"> answered Jul 10 '09 at 11:47 nschmidt 1,816919

thanks, I missed this mistake. however, I made the changes and the displayed frame is still mainly in black and white, with pink and green shades here and there. –  ReachConnection Jul 10 '09 at 15:08

I just added a image to show how my output looks like –  ReachConnection Jul 10 '09 at 15:33

add a comment

up vote0down vote

The picture looks like you have 4:1:1 format. You should change your lines to float U = lookupTableU [uvBuffer [ (y * (YUV_WIDTH / 4) + (x/4)) * 2 ] ]float V = lookupTableU [uvBuffer [ (y * (YUV_WIDTH / 4) + (x/4)) * 2 + 1] ] Maybe you can post the result to see, what else is wrong. I find it always hard to think about it. It's much easier to approach this iteratively. share|improve this answer