Rendering

// Image Processing

Parallax Occlusion Mapping

This technique uses a displacement (height) map to fake complex geometry on flat surfaces. The idea is that this might be used instead of having a really high-poly mesh, for example, the mesh in the gif only consists of 6 vertices. 

The shader uses ray-marching for finding intersections between the rendered pixel and a blocking pixel. This intersection is found by comparing heights. Creating a ray and stepping along it is easy. The hard part about this shader was that all calculations had to be done in tangent-space. With other words, relative to the vertex-normal of the surface. 

Tangent information in our pipeline is only available when generating the information about the visible scene, when looping through all meshes. This means that these calculations has to be done for every mesh that is using this technique.  

Screen Space Reflections

Reflections is a hard problem to solve within real time rendering, since accurate reflections require ray-tracing, which goes against the standard rasterized model and is quite expensive. The most common solution is to use cubemaps for faking environmental lighting. Cubemaps are cheap, but provide no accurate reflections from the scene. Even with baked cubemaps, it’s nowhere close to being realistic.

But we’ve got a bunch of information about the visible scene so we might as well calculate whatever reflections we can with what we got! This effect, combined with accurate cubemaps, provides a good result while keeping the cost extremely low.

Screen Space Reflections works by taking the direction from the camera to each pixel, reflecting it against the pixel normal, and then ray marching until something is hit, breaking if something is blocking the way.

Screen Space Ambient Occlusion

Tight corners allow for less light to enter, which creates a soft shadow, often barely visible, but crucial for the realism of a scene. This shader tries to approximate this effect by comparing the normal of each pixel to a few random surrounding pixels. The larger the difference between the normals, the more occlusion.

This effect is applied before the final light calculations, and is then used in addition to the baked ambient occlusion of each model.

Motion Blur

There are two common methods for producing motion blur in video games. Per object or based on camera transformation. Both of these methods require the current and previous transform of whatever you are blurring. They are also often combined.

My implementation only applies camera based blur. The effect requires four steps:

1. Calculate world position of current pixel.
2. Translate world position into screen space using the transformation from previous frame. This screen space position also represents a UV coordinate.
3. Calculate the UV-coordinate difference between current and previous frame.
4. Sample some pixels between these coordinates and blend.

Fast Approximate Anti-Aliasing

This is an effect that you’ve probably heard of before. It is the most common anti-aliasing effect in games. This is because it is fast and easy to implement.

FXAA works in two steps. The first step tries to figure out if each pixel is an edge or not, as well as the direction of the edge. If the pixel is an edge, then the second step blends the pixels along the edge, making the edge a little blurry, but a lot smoother.

LUT (Color Correction)

LUT is a fast color correction technique which allows for a unique workflow. With some screenshots, artists change the colors however they see fit in a photo editing software, and then export their settings as a LUT-texture. When this texture is used in the LUT shader, their corrections are directly applied to the final image.

LUT stands for Look Up Table. It is the texture that makes up this table. This table has three axes where each cell has a color value. The shader uses the rgb-values of each pixel as coordinates in this table to figure how the color should be translated.

Godrays

Something I vividly remember from my Fallout 4 experience is the overwhelming godrays. Could I create something similar myself?

Godrays are rays of light visible in fog and dust. There are different ways of achieveing this effect. I settled for a simple and fast screen-space effect. A pixel is inside a ray if nothing is blocking the pixel from the light source. Step towards the light source, and add to the effect for every step that nothing is blocking! I almost couldn’t believe how easy this one was.

Checkerboard Rendering

I was playing Resident Evil when I saw an interesting graphics option called “interlaced rendering”. I discovered that this game had the ability to only render every other row of pixels each frame, almost doubling the performance with minimal graphical artifacts. The idea stuck with me, but I didn’t quite like the interlaced part of it.

I decided to try every other pixel instead. It’s easier to visualize this if you imagine the screen as a checkerboard, each pixel being either white or black. I alternate between rendering the black ones and white ones every frame. For the pixels that I don’t render, I simply reuse the pixel from the frame before.

The artifacts are most of the time very similar to a poorly implemented motion blur, and almost aren't noticable in a game with slower pace. Unfortunately, sometimes my checkerboarding-function in the shader fails and the checker-pattern becomes irregular. This might be because of rounding errors.

// Shadow Baking

Shadows are expensive. How can I make them cheaper? My goal is to avoid calculating shadows for as many objects as possible each frame. 

There are two types of shadows in our engine, directional and point shadows.

Point Shadows

Point light shadows are a bit more complicated because of their conditions. What shadows can be baked and what can’t?

This is what I ended up with:

Static shadow + only static objects

Bake

Static shadow + dynamic objects and static objects

Bake static objects, use dynamic objects culled by camera frustum for dynamic shadows

Dynamic shadow + static objects

Use static objects culled by camera frustum

Dynamic shadow + dynamic objects

Use all objects culled by camera frustum

This means that our static point shadows needed to support a combination of baked and dynamic shadows. This was solved by simply sending two shadow cubemaps to the shader, and per pixel selecting the smallest shadow value.

This workflow wasn’t easy to figure out, but it allows every static point light in a scene to cast shadows against every static object, without affecting performance, while also allowing the closest and most important light sources to cast normal dynamic shadows.

// Render Pipeline Optimizations

Producing a beautiful image is important for the overall game experience, but so is making the game feel responsive. Rendering is often the main contributor to poor game performance and responsiveness. I like fast games, poor responsiveness is among the worst thing I know in video games. Fortunately, there are increadibly many techniques for improving specifically rendering performance.

I was responsible for all 3D rendering in our engine during the second year at TGA, and if the games didn't turn out responsive, it would probably be because of my rendering pipeline. The challenge facing me gave me just the motivation I needed when researching and implementing these techniques.

Threaded Rendering

One of the most common techniques used for speeding up any kind of computer process is multithreading. My goal was to perform all rendering logic on a separate thread, thus spreading out the workload and increasing overall performance. 

Buffers

Since the logic was going to communicate with the rendering a lot, I felt that there was a need for a common interface which could manage these mutex-locks automatically. Another early realization was that most communication between these parts of the engine was going to be one-sided. Almost no information had to be returned from the rendering thread. 

The StaticBuffer

But most objects in our scenes are marked as static. This means that their transformations do not change. It feels like a real waste to send the exact same information through a buffer every frame, just for it to be discarded.

I created a modified version of the Buffer class called StaticBuffer. There only exists one of this buffer, meaning that the mutex has to be manually locked and unlocked whenever information is accessed. This might sound troublesome because of the risk of one thread having to wait for the other, but since static objects are only added or removed when a scene loads, this isn't really a problem. In addition, to avoid having to lock and unlock this buffer thousands of times every render-loop, It is locked at the beginnig of the loop and unlocked again at the end. 

The ICullable Interface

I quickly realized that there were more things than deferred meshes that could be culled using the same methods. I decided to use a combination of templates and inheritance to automatically cull many kinds of objects. The ICullable class provides a common interface for everything required in the culling process. All an object has to do in order to be culled is to inherit from this class and overwrite its functions. 

Culling Volumes

During the second project using this engine, we aimed to create a small, heavily detailed indoor world. I was able to cull objects against the camera frustum, but no checks were done against the geometry. Thousands of objects were sent to the graphics card every frame even if they were hidden behind walls. 

This environment was perfect for something that I call Culling Volumes. These volumes are hand-placed by our level designers, who then specify which other volumes should be visible from the current one. When the camera is inside a volume, only the specified volumes are rendered. This puts some extra work on our LDs, but it is something that only has to be done once per room and significantly decreases the total number of objects that has to be rendered at any time. Objects outside of volumes are still rendered and culled normally, and the system supports for objects to be inside multiple volumes.

Separating the objects and culltree cells into volumes is also performed while generating the CullTree.

In our engine, this culling process is used for forward meshes, deferred meshes, decals and lights. 

// Skeletal Animations

Baked Skeletal Animations

TGA provided some code for skeletal animations loaded from an FBX. This code provided accurate animations, stepping through and animating the entire bone hierarchy each frame. This is an expensive way of doing it, since recursively stepping through the hierarchy requires a large number of matrix multiplications, which grow exponentially for every step down in the hierarchy. In some early benchmarks, this process took up to 20% of the frame time. All this just to produce a simple array of matrices, which are then sent to the graphics card. 

My idea was that this process might be baked, storing one array of pre-calculated matrices for every frame of the animation. This meant that the most expensive part of having an animated object on screen would be to actually send those arrays to the graphics card. With other words, animations at practically no performance cost.

The baking process

This means that I need to run through the entire animation once using the original method and a fixed framerate, storing the result. With a couple of long, detailed player animations, this process actually takes quite a bit of time. Of course, we don’t want to keep the player waiting for the baking process to finish, so let's put it on another thread, letting it load in the background. Hopefully, the player won’t mind that all animations aren’t loaded the first second after starting the game.

There were two major drawbacks to this method, both relating to blending:

Bone Socketing

is the process of attaching in-game objects to the bones of an animation. It is a simple but very useful tool. In principle, all you have to do is to transform the final bone transformation into local-space compared to the parent object. Our implementation was simple, just put an entity as a child to an animated object, and give the entity the same name as the bone you want it to follow. The offset between original transformation of the entity and the bone is stored and applied each frame, allowing for precise positioning.

Additive Animations

is the process of combining animations by adding the result of their bone transformations. In order to play multiple animations at the same time using the same animator, I had to add support for animation layers. Each layer has it's own state-machine with custom transitions and states, while properties are shared across all layers. 

This is a case where baking the animations actually made everything a lot easier. Because the transformations already were relative to their origin, all I had to do was to multiply each layer in the correct order. Sometimes an additive animation should only be applied to some bones. This was solved by specifying a root bone in the animation layer. Only bones below the root in the hierarchy are affected by that layer. 

The main character in the police game uses a combination of all of these features. Bone socketing is used to attach the flashlight to the hand and additive animations are used to keep his hand up while the flashlight is lit.

// Decals

Decals are textures that are projected onto other models. It's perfect for effects like dust, dirt and blood. Our decals follow the same pipline as deferred rendered objects. They are culled and statically optimized the same way, the main difference being that they use cubes for projection instead of uv-mapped models.

Decals do not affect the depth buffer and only overwrite existing information about the visible scene. This means that our decals use exactly the same lighting calculations as normal meshes, while not affecting the performance of these calculations at all.  

Projections

This is an area of computer graphics that has always been a little extra magical to me. The calculations that this effect required was something that I had a hard time visualizing before actually sitting down and spending time thinking about. Reading about decals online was a good starting point, and got me thinking about our home cinema. How does that image project onto our wall? What was the uv-translation that I was trying to achieve?