Game Engine
For the TGA game projects
I started working on a game engine called Diablo2D as preparation for the first 2D C++ project. This engine was used for 4 TGA projects: Kero Kero, Swarmed, Temple of Chakram and a project made by another group.
Second year of TGA meant 3D games which required a faster, more advanced and more structured engine. I felt that Diablo2D would not meet these requirements and it was therefore scrapped. Those first few months was an intense race to make the new engine run before the deadline of our first game using it. This improved engine was built with a new underlying structure and design principles. At the core lay a simple entity component system, a 3D rendering pipeline, PhysX and better resource management. We decided to use Unity as our level editor.
// Entity Component System
Me and Marc Nilsson co-developed the ECS. Our ECS is not threaded and uses a sparse component array initialized on startup. The ECS is at the core of the engine, used for almost everything. This required stability and ease of use, which we achieved through careful planning of the structure and interface of the classes exposed to the user.
Components
Components in an ECS are just collections of data. This means that the memory becomes extremely easy to manage. This component contains one serializable property and one normal variable. Most variables need a default value. This is what the Reset() function is for. No functions or executable code should exist in a component, with Reset() being the only exception.
Systems
Five lines of code. This is an entire system. The system contains a collection of entities that are automatically managed. All the user has to do is to loop through these entities and update the neccessary components.
Of course, there is additional functionality for more advanced uses, like collision callbacks and manually changing the priority of this system. These functions are optional.
Properties
Components aren't really that useful in an engine if variables can't be loaded from a scene or prefab. Properties fill this need. Properties are automatically loaded and reset. They will show up in the inspect-window and can there be manipulated in run-time. There are variations of this macro which let the user define a default value and custom edit functions. More about the export / loading process can be found under "Unity Export Pipeline".
// PhysX
I was responsible for the PhysX implementation in our engine. Unity uses PhysX and already has a lot of physics-related components and ways for the user to customize how everything should be simulated. My idea was that neccessary Unity functionality could be replicated in our engine, making the workflow easier for our level-designers.
Besides trying to replicate the neccessary features of the Unity interface, my goals were to keep performance cost and compile times low.
Performance & Compile times
I created an extra layer between PhysX and the user. This allows me to control exactly what functionality should be accessible.
This layer lets me move most PhysX includes to cpp files, making most of the includes necessary for the user extremely lightweight. By having the physics source code in a separate project, it only has to be compiled once.
The extra control made it easier for me to correct for the differences between Unity and PhysX while keeping everything optimized.
// Resource Management
Background
The first version of our resource management system was written by Albin Engström and was specifically made for threaded file loading, file watching and recompiling shaders in runtime.
Albin's idea was to use an identifier for resources, and use templates to support different types. The identifier was used whenever a resource was accessed and acted like a smart pointer. The idea was really good but the interface was hard to understand and very specific to the classes that the system was originally written for.
My Contributions
My goal was to improve the interface and make the resource management generalized for all system resources. The interface was simplified by using inheritance and by separating between the base resource class and the reference class. I also decided to let each resource deal with the raw data array however they wanted, moving the creation process from the file loader class to each individual resource.
Inheritance
Inheritance let me hide most internal functionality which did not have to be accessible to the user in the ResourceBase class. Additional template arguments lets the user specify different resource options which are optimized in compile-time.
Resource References
The Resource::Ref class improved upon the existing identifier system while providing support for default resources. Use a path and identifier in the constructor and the reference will automatically be linked with the correct resource which, if not already loaded, starts to load on a separate thread. If the resource failed to load or in case of a bad reference, the default resource will be returned instead.
// Unity Export Pipeline
Introduction
We decided to use Unity as our level editor. This meant that the world built by our level designers needed to be saved to a file that could then be loaded in our engine, preferably in as few clicks as possible. This proved to be a great challenge. Currently, exporting all scenes, prefabs and assets requires 1 click and takes ca 2-5 minutes. Exporting all prefabs and the current scene takes less than 2 seconds, and that includes the time to start our engine.
Objects
The first step was to figure out an effective way to export objects. Unity uses something called gameobjects. These objects consists of a few basic attributes and a collection of components. Was there a way to access all neccessary data from these gameobjects? Could this information be written and read from a file? We decided to use json. It provided a readable file format, helpful when debugging, and very good integration with existing C# reflection. After an afternoon, we had a readable json file telling us all we had to know about an object, exported directly from Unity into our engine folder.
Components
Exporting components seems simple. Loop through every component on an object and serialize relevant properties. C# provides reflection, which means that classes know about their own properties. This means that the entire process could be completely automated for any component. It was truly simple yet effective:
But it was all too good to be true. It turns out that most built-in Unity components are not written with reflection in mind. Unity uses C++ behind the scenes, and most built-in Unity components accessible through C# are nothing more than interfaces. This means that there is no way to automatically access any properties from these components. We had to manually write the export function for each one:
The decision to use a component based engine and replicating most Unity components 1:1 made the export process much easier. The PROPERTY() functionality mentioned above automated the process of loading the variables of components.
Scenes and Hierarchies
In Unity, a scene is a collection of objects. After exporting one object, exporting an array of objects was no challenge. But Unity also supports object hierarchy. An array of objects would not be enough, objects need to be able to contain other objects.
With some experience in Unity, you will know that this hierarchy only really affects the transformations. After the entities are created and the components are loaded, the hierarchy only has to exist in the transform-component.
Prefabs
What if we wanted to create an object in run-time? In Unity, this is solved using prefabs. I felt like this was a good approach for our engine as well. Prefabs in Unity can be used in exactly the same way as any other gameobject. All I had to do was to reuse the code for exporting individual objects, and load exported prefabs on engine startup.
Assets
Assets are files like models, textures and sounds and have to be shared between the Unity project and the engine. They could theoretically share folders, but that would get messy quite fast. An asset export process was required. All world, prefab and asset changes are first added to the Unity project before being exported to the engine. The solution was simple. Loop through all Unity assets recursively and copy to the engine folder! This means that the file path and structure are always guaranteed to match. All files aren't meant to be copied, only specific ones like models, textures and sounds. By specifying file extensions, this wasn't a problem. I also added support for folders to be ignored, which became helpful in specific cases.
One problem remained: files removed from Unity would not be removed from the engine folder. Bloat adds up fast. This was solved by cleaning the engine folders before copying, using those same filters as when copying.
Other Improvements
- Iteration speed
A level designer has to test their work, a lot. The process of going from editor to game has to be fast. I wanted to emulate the play button in Unity, a one click experience. I added an option that, in addition to exporting prefabs and the current scene, starts the latest version of the engine.
- Scene loading optimization & automatic prefab generation
After getting this far, I realized that the json scene-files were starting to become quite big, at >50k lines for a relatively small scene. I figured that the scene could be optimized by letting objects linked to a prefab reference that prefab, and only export properties that had changed. I felt like this optimization would make a great difference, but then I realized that most objects in our Unity-scenes were not linked to any prefab. This wasn't only an optimization problem, it also meant that if a property of an object was changed, it would only be applied locally, not to the other copies of that same object. I asked our level designers. It turns out that making prefabs can be a hustle, especially when there are tens of models that first has to be unpacked. Less than 20 lines of code automated that entire process.