Overview to “Toon/Cel shading”

转自：https://blog.felixkate.net/2017/01/19/toon-shading/

For the last couple of weeks I often had discussions about toon / anime esque shading.
In this post I want to list a few approaches I tried out.

---

Shading Methods

---

Flat shading / Unlit

This is not directly shading but it should still be mentioned since this is obviously the cheapest way and is often used for weak performance platforms like mobile.
If you go for this approach you should also paint in the shadows.

MatCap

You might have heared of this before if you have looked into ZBrush or Mudbox.
By sampling a round spherical texture we can fake light distribution from a locked perspective (turning the object will move the “lightsource” / view too.
It is useful if you have a locked perspective and lightsource as it is pretty cheap but that is the limit on what it can do.
For this approach we only want a light distribution grayscale map instead of the full material so we can later use it together with our main textures:

The calculation for getting the coordinates for sampling the map simply require us to calculate the view normal in the vertex shader:

float2 matCapCoord = ((mul((float3x3) Matrix_Inverse_WorldView, normal) + ) * 0.5).xy; // "normal" is the unmodified input normal
matCapCoord.y =  - matCapCoord.y; // Depending on our environment we need to invert the y coordinate  

Step shading

A really simple approach.
We start of by doing a simple nDotL calculation to get the basic light information. Casted shadows can also be multiplied into it.
From here on we use a simple calculation to remap the light/shadow gradient into multiple steps.

float nDotL = dot(wNormal, lightVector);
float light = floor(nDotL * _StepCount) / stepcount; // _Stepcount is an input value that depends on how many color tones we want  

Gradient shading

The last approach I list here uses a gradient texture to define our steps.
We simply use our NdotL calculation as x coordinate input for sampling the gradient texture.

float nDotL = dot(wNormal, lightVector);
float light = tex2D(_GradientSampler, float2(NdotL, 0.5));  

Fresnel
This is not used for the light calculation itself but can add a nice touch depending on what kind of style we want to go for.
It can either be used together with the basic shading or be added ontop afterwards.
The example below is just for getting the basic outside glow. Fresnels mostly use a pow() node to manipulate the falloff but a smoothstep could also be used.

float nDotL = dot(wNormal, lightVector);
float nDotV = dot(wNormal, viewVector); // The basic view dot normal calculation
float fresnel =  - nDotV; // We want to invert our fresnel here so that the glow goes to the outside 

　　　　　　Applying color

---

Multiply

The probably most common approach for coloring.
By multiplying our color / texture with one of the light calculations of the previous section we can get a basic cel shading effect.
There are a few options here as to multiply the texture by itself to colorize darker areas a bit more.

Shift hue

An alteration to the multiply. By calculating a hue shift we try bringing more variation into our shadows.
It might look more interesting but is not really a practical approach as we need to make a lot of calcluations to get the color to hsv space and then back to rgb after the hue shift which is a bit expensive.

Gradient

This one is a bit special and can’t be used in all kind of situations.
It is similar to the gradient shading process I described above but can use a palette of multiple gradients. (Not in this example since it would require some work to make the index texture)

At the time we are sampling our gradient texture the y coordinate will be used to decide which palette we want to use.
This gives us the ultimate freedom over how our shadows should look but at the same time limits us in many ways.
Blending multiple gradients is a bit problematic so we can either limit it to hard edges only or use another “outline” mask to hide borders between colors.
Ways to assign the palette index could be either by directly vertex painting it into the red channel of our mesh or having a grayscale lookup texture. (No bilinear filtering  for that texture since it would blend borders)

float index = tex2D(_MainTex, texcoord); // Or vertexcolor.r depending on our setup
float color = tex2D(_GradientSampler, float2(light, _Index));  

Shadow textures

A really simple but effective way to solve our problem and the approach I used for this character.
By providing a second texture that holds a fully shadowed version of our main texture we can gain full control over our shadows.
This obviously only works when we have a clear shadow/light situation and comes at the cost of that second texture.
A clear advantage of it is that we can already pre-paint shadow areas into our main texture and blend them together with the calculated ones without getting any visible borders.

float mainTex = tex2D(_MainTex, texcoord);
float shadowTex = tex2D(_ShadowTex, texcoord);
float color = lerp(shadowTex, mainTex, light);  

　　　　　　Outlines

---

The sobel part is a bit blurry since I had to use an AntiAliasing image effect to get rid of jagged borders.

Inverted mesh
An often used method for creating outlines. By using an extra pass we calculate a copy of our geometry that has it’s cull mode inverted so that the faces look inside.
Then we need to push our vertices out inside the vertex shader.
We can either return 0 for a black outline in our fragment/pixel shader or sample our texture again and make it darker to have outlines similar to the “inside” colors.
This method requires us to have smooth normals or holes will appear when we push the vertices out. Another problem is that certain details like mouth or eyes can get outlines we don’t actually want.

float4 pos = mul( MVP_Matrix, vertexPos ); // Calculate our basic rendering position like usual
pos = pos + lineThickness * mul( MVP_Matrix, normalize(normal)); // Make the outline by pushing it into the normals direction; We can also multiply the distance to the camera here to render a thicker line the farther we zoom out  

Sobel
A post process way to do outlines.
This works by sampling a scene texture (either colors, normals, depth) multiple times with an offset into different directions.
Then it compares the overlying pixels and if the difference is bigger then a threshold it is considered to be an outline.
If we want only certain objects to have an outline we have to provide some kind of mask beforehand.
I won’t include a code example in here but if you are using Unity you can look into the standard ImageEffects package to see their edge detection effect implementation for it.
For an Unreal implementation (that also takes colors into account) you can take a look at the node network in the last part of the Unreal shading model blog post.

Other useful hints

---

Prepaint shadow areas
If we provide a grayscale texture or vertexpaint beforehand we can manipulate the light calculation a bit.
This texture would hold informations about how much the NdotL calculation would affect certain areas and can guide the shadows a bit.

Check your geometry
The hard borders between light and shadow in most toon shading approaches make hard angle switches in the geometry really apparent.
If you have a character try to model it so that the mesh flow is really smootlh and try to avoid abrupt changes in the angle.
Usually more vertices here mean you get a better looking lighting.

Normal maps
Another really nice addition. Using normal maps for toon shading works fine but is often less apparent.
It can’t completly replace geometry but can still be used to add a bit of detail.

Multiple lightsources (multi pass)
This might be a bit tricky because we can’t rely on additive blending. (It would break our “handpicked” colors)
One way to do it is by using a hard zero/one alpha blending to only add light areas ontop of the previous passes.
Another a bit difficult way is to calculate a grayscale light situation first and apply our colors in a post process.
Using a deferred pipeline here could greatly help us with the second one.

Hatching

Sometimes you want to use hatching textures instead of shadows. This is pretty easy.
We need to input a hatching texture (grayscale) into our shader and scale it not with the texture coordinates of the object but the screen coordinates.
We also need to repeat it depending on the resolution. The last step is simply to use our shadow as a mask for the hatching texture.

Smoothstep
A really handy function to remap a gradient. You can use this to bring the gradient down to a really thin line to avoid too hard edges at the light / shadow borders.

That’s it for now. I will try adding more information to it at a later date if I find something interesting.