UE4 #includes, Precompiled Headers and IWYU (Include What You Use)

Ever had Unreal's build tool suddenly spring some incomprehensible error on you, pointing to a line of code that you didn't touch? If so, read on.

A couple of engine versions back, Epic introduced the 'Include What You Use' model to UE4 in an attempt to help clean up the situation with include dependencies. This was definitely a big improvement, but there was initially very little information given on the changes made and how they applied to game projects. The above docs page is now much better, but still a little unclear on a few things. I'm no authority, but I'm going to try to at least clear up a couple of misunderstandings that I've since seen doing the rounds.

Switching to IWYU

First off, switching to IWYU essentially means doing the following:

1. Add PCHUsage = PCHUsageMode.UseExplicitOrSharedPCHs; to the .Build.cs file of all your modules.
2. Delete any existing module precompiled header files you have* (the first include in every cpp file in the old system) if they were nothing more than a list of includes. If you had declarations inside a precompiled header, move them into a new header.
3. Remove all includes of the old precompiled header.
4. Fix everything.**

* See final section for why you may not necessarily want to do this step.

** You'll get heaps of errors because you no longer have required headers included. You need to add explicit includes for the types and declarations you need; you can find the header a type is defined in at the bottom of its page in the API reference (Epic in their wisdom decided to discontinue the offline reference as well as butcher the search functionality for the online one, which doesn't help with this). This might seem like a pain, but it's essentially the entire point of the change - being more explicit about includes. It will make your code more robust going forwards.

All neat and sorted?

Now, the idea is that at this point, your code is self-contained and no longer dependent upon includes within precompiled headers to compile. As stated in the documentation: "With IWYU, every file includes only what it needs, and any PCH file we choose to use, purely acts as a layer of optimization on top of the underlying source files. They can be modified to minimize build times, independently of changing the source files themselves, and without affecting whether the code compiles successfully or not."

This is a bit misleading though. Certain modules in the engine (in particular Core, CoreUObject, Engine, Slate and UnrealEd) create shared precompiled headers, which by default Unreal Build Tool will use when compiling any module dependent on these modules. In practice, that's every module; furthermore, these are some of the biggest engine modules and the combined includes inside them account for a significant portion of the engine's headers. These precompiled headers are included on the command line which means the headers they reference are essentially included before all your explicit includes, for each of your cpp files. You can see this in action by adding a cpp file that doesn't include any engine headers. Add a reference to a common engine type - for example a global variable of type UStaticMesh* - and you'll see that so long as you have a dependency for the type's module (in this case Engine), the code will compile without need for an explicit include. This is precisely the kind of thing that IWYU was supposed to avoid.

So, how to check that your code really is well formed and self-contained? This can be done, but you need to adjust the build configuration to completely disable use of precompiled headers. You also need to turn off unity builds, since these cause cpp files to be merged together, meaning one cpp might compile only because another one that is merged above it includes something it relies on. On the command line, this amounts to adding: -NoPCH -NoSharedPCH -DisableUnity. To make the change inside BuildConfiguration.xml, add the following to the BuildConfiguration section:

<bUsePCHFiles>false</bUsePCHFiles>
<bUseSharedPCHs>false</bUseSharedPCHs>
<bUseUnityBuild>false</bUseUnityBuild>

If you do this for your project, you will likely find that even after IWYU updates, your code is still not truly self-contained and you have to go through step 4 again. I think it's worth doing, and if you have automated builds set up then running them with the above changes at least some of the time is a good idea. It's especially important for any code which you're distributing for reuse, such as plugins.

Note that these adjustments do mean much longer build times, so I don't recommend using them as default for regular iterative builds.

Why bother at all?

For me, just not liking the idea of having code that is technically malformed despite compiling is reason enough to put the time in to maintaining self-contained includes in my projects.

If that's not enough though, there's another reason: those annoying errors alluded to at the top of the article. They generally come about because of a missing include, and the nature of C++ is such that the description in the error is often misleading if you don't have many years of experience behind you to help decrypt it.

In UE4 it's even worse - they show up at apparently arbitrary times, on code that was compiling fine before and hasn't been changed. The reason for this is PCHs and unity builds (and sometimes just UBT being weird). If your code isn't fully consistent in its includes, it will often still compile due to the way these build optimizations work. But the precise contents of engine shared PCHs, and the composition and order of cpp files in a unity build, is dynamic and not well defined. A change in some seemingly unrelated source file, an engine update, an added or removed plugin, or some other adjustment to the build environment - these can all result in changes to the above, which in turn can cause these compilation errors. The thing to remember is, when you get these errors, it means your code was malformed all along and you just got away with it. If you make the effort to keep it consistent in the first place, then you (or someone else using your code) won't have to deal with these errors.

A note on performance and precompiled headers

The procedure given by Epic for switching to IWYU implies removing project-level precompiled headers (PCHs) in favour of using only the engine shared PCHs. In most cases, this will be fine, but something bugged me about this from the start and I only recently dug into it. PCHs speed up compilation by preprocessing code in header files, thereby preventing the same work being done over and over by the compiler in each translation unit (cpp file, or in the case of unity builds, each batch of cpp files) that includes the given header. Now it's true, the engine is huge and dwarfs all projects, so it's most important to have PCHs covering the core engine headers. For larger projects though, do we really want to give up project-level PCHs?

It's more maintenance, and it will depend heavily on the size, organization and level of flux of your project's codebase, but it's worth being aware that the default IWYU transition could actually lower compile times relative to having a carefully crafted dedicated PCH in your module(s). I recently did a reorganization of the codebase for my primary project. It's about 150,000 lines, now split into roughly 20 modules (a number of them very small). In the process I changed a few of the larger, more stable modules back to having their own private PCH file, and saw a very significant improvement in compile times.

Note that using a private PCH for a project module isn't really going against IWYU, it's just that the documentation gives the impression you no longer need them. While that is generally going to be true, you can use a PCH in your module while still conforming to IWYU practices.

If you want to try it out, do the following:

1. Add a header file to your module's Private folder. Name is unimportant.
2. In your module's .Build.cs, after the PCHUsage = PCHUsageMode.UseExplicitOrSharedPCHs; line, add: PrivatePCHHeaderFile = "Private/YourPCHFile.h";
3. Add a bunch of includes to your precompiled header include file.

Step 3 needs to be done right. PCHs are a tradeoff - generally, the more you add, the more you reduce compile time of translation units. However, you also increase the time it takes to generate the PCH, and more importantly, you increase the frequency with which it will need to be regenerated, and each regeneration means a recompile of every translation unit in the module. That's why it's super important to include only those headers that are stable and will very rarely be modified. There's also little point including headers that will only ever be included by one or two cpp files.

One other point - when you specify a PCH like this, Unreal will no longer use the engine shared PCHs, so you should explicitly add includes for engine header files that your module uses into your PCH. Having to manage this is a pain, so it's only worth considering on large projects where compile time is problematic.

UE4 Plugins

Following on from the previous post on UE4 modules, I wanted to briefly cover plugins as another form of encapsulation available in UE4.

Plugins vs modules

In UE4, plugins are simply a higher level unit of encapsulation than modules, rather than an alternative approach. Like projects, plugins can contain any number of code modules within them. They can also optionally contain content.

UE4 plugins are not strictly plugins

UE4's concept of a plugin deviates a little from the norm. A true plugin conforms to and implements a pre-existing interface, thereby extending an application in a predefined way. While UE4 plugins can take this approach, they can also export new types and be used as libraries, with project code having a direct dependency on the plugin. See this excellent AnswerHub post for an overview of this distinction.

Another way in which the UE4 approach differs is that, by default at least, its plugins are linked and their content processed when packaging a release build of a project. The underlying architecture may support delaying this process and loading them dynamically as true plugins (I'd be interested to know if anyone has tried), however it certainly doesn't seem to be the intended usage.

Engine level vs project level plugins

Plugins can exist at either the engine level ([UE4 Installation]/Engine/Plugins), or the project level ([Project Folder]/Plugins). In a launcher installation, plugins at the engine level must be prebuilt. Project level plugins containing source code will always be compiled as part of the project build process, regardless of engine installation type.

Creating a plugin

While the editor has a plugin wizard to automate the process, I'm going to go over the manual approach to detail all the elements of a UE4 plugin. In the following example, the plugin is named 'MyPlugin'.

Here is the basic structure of a plugin. The following would be placed in either /Engine/Plugins or [Project Folder]/Plugins.

- MyPlugin
    - MyPlugin.uplugin
    - Source [Optional]
        [C++ modules here]
    - Content [Optional]
    - Config [Optional]
        - FilterPlugin.ini

The plugin descriptor

The plugin descriptor file lives in the root folder of the plugin, and is named [PluginName].uplugin. In this example, 'MyPlugin.uplugin'. Here is a basic example.

{
    "FileVersion": 3,
    "Version": 1,
    "VersionName": "1.0",
    "FriendlyName": "My Plugin",
    "Description": "An example plugin.",
    "Category": "Kantan Dev",
    "Modules": [
        {
            "Name": "MyPluginMainModule",
            "Type": "Runtime",
            "WhitelistPlatforms": [ "Win64", "Win32", "Android", "Mac", "IOS", "Linux" ]
        },
        {
            "Name": "MyPluginEditorModule",
            "Type": "Editor",
            "WhitelistPlatforms": [ "Win64", "Mac", "Linux" ]
        }
    ],
    "EnabledByDefault": false,
    "CanContainContent": true
}

See the official documentation for more details.

Plugin source

The Source folder contains C++ module code, in exactly the same structure as for a project's Source folder. See my earlier post for a rundown of modules.

In the above example, there would be subfolders for two modules, MyPluginMainModule and MyPluginEditorModule.

Plugin content

If you want to include content in your plugin, the easiest way is to create it in the editor in the context of a project. Make sure you have "CanContainContent" in the plugin descriptor set to true. I generally create a dedicated development project for each plugin I make and do all coding and development in the context of that project. In the editor, ensure you have Show Plugin Content enabled in the content browser view options, and you should see a content folder visible for your plugin. You can then create whatever assets you need there.

Note that the editor's content migration does not support moving assets into plugin content folders, and that copying assets between plugin and project content folders in the content browser is also unreliable when it comes to asset references.

Building a plugin

As noted above, at project level, plugin code will be compiled along with project code. However if you intend to distribute a plugin, the recommended way to do so is with the engine's automation tool. On Windows, run the RunUAT batch file located at [UE4 Installation]/Engine/Build/BatchFiles as follows:

RunUAT.bat BuildPlugin -plugin=[path/to/pluginname.uplugin] -package=[path/to/output/directory]

This will compile the plugin code for its whitelisted platforms (for Development, DevelopmentEditor and Shipping configurations), and output the full plugin ready for distribution, including source, intermediate, binaries and content, along with anything specified by the plugin filter (see below).

FilterPlugin.ini

This optional config file specifies any additional files that should be included when building a plugin for distribution using the automation tool as described above. Note that this is not the same as packaging a project which uses a plugin - that is a later step, and any additional files that should also be copied at that stage must be specified separately via the RuntimeDependencies array inside the Build.cs file of one of your plugin's modules.

For example syntax of the FilterPlugin.ini, just run the BuildPlugin command above. An example FilterPlugin.ini file will be generated in /Config directory of the source plugin.

Conclusion

The vast majority of my UE4 programming is done in plugin form. Just like with modules, the encapsulation encourages good design and less dependencies. Plugins can have content bundled with them when necessary, and can be easily shared and reused. They also offer a way of reducing project recompilation time, by putting infrequently changing code into a binary-only or engine level plugin.

UE4 Code Modules

What are UE4 code modules?

In UE4, a module is a distinct unit of C++ code, with an accompanying C# build file. A game project can be made up of one or more modules, as can a plugin. Under the hood, a module corresponds to a dynamic library, although by default in shipping builds all code is linked together into a single executable.

Why use multiple modules?

It's possible to create a game (or plugin) that is composed of a single module, and for smaller projects that's fine. There are a number of reasons to consider splitting up your code however.

• Encapsulation and organisation. It's always a good idea to encapsulate code as much as possible. Building a component or system in its own module encourages you to keep dependencies down.
• Code reuse. A module is a natural unit of code for reusing across multiple projects. Separating logically distinct systems at the code level makes it easier to reuse something, even if initially you didn't envisage it being useful outside the project for which you originally wrote it. One effective approach, if you use git, is to put your reusable module(s) into their own git repository, and then incorporate that as a git submodule in any project repository.
• Configuration-specific code. If you want to write editor extension code for your custom classes and game systems, you should put it into a dedicated Editor module. While preprocessor definitions (#if WITH_EDITOR) can be used in some cases, any non-trivial amount of editor-specific code should go into an Editor module. It's also possible to create Development only modules, so you can have, for example, debugging code which gets automatically compiled out of shipping builds. The same goes for server/client-only code.
• Platform-specific code. Again, preprocessor macros for platform-specific code should be kept to a minimum. It's possible to provide platform-specific implementations of project components, each in their own module, and selectively build and package based on the target.

Adding a module

For the remainder of this article, wherever you see 'YourModuleName', regardless of case, be it as part of a filename or in code, replace it with whatever you want to name your module.

Adding an extra module is essentially the same whether you're adding it to a project or a plugin. Inside the source directory, create a new folder named YourModuleName. Within that, you first need the module build file, named YourModuleName.Build.cs. A basic one will look something like this:

// YourModuleName.Build.cs

using UnrealBuildTool;

public class YourModuleName : ModuleRules
{
    public YourModuleName(ReadOnlyTargetRules Target) : base(Target)
    {
        PCHUsage = PCHUsageMode.UseExplicitOrSharedPCHs;

        PublicDependencyModuleNames.AddRange(new string[] {
            "Core",
            "CoreUObject",
            "Engine",
            });
    }
}

Then you will want to add two folders, named Public and Private. The source (.cpp) files always go in the Private folder. Header files can go in either. If they define types or functions that you want to use from code inside other modules, put them in Public. Otherwise, put them in Private for maximum encapsulation.

A bare minimum compilable module requires a source file (YourModuleNameModule.cpp is a good standard) in the Private folder containing the following:

#include "ModuleManager.h"
IMPLEMENT_MODULE(FDefaultModuleImpl, YourModuleName);

To ensure your module gets built with your project, you can add a reference to it in your target file (YourProjectName.Target.cs/YourProjectNameEditor.Target.cs) as follows:

ExtraModuleNames.Add("YourModule");

However, in practice you will generally have a dependency chain connecting your module to the main project module (through one or more PublicDependencyModuleNames additions in .Build.cs files), in which case this step is not actually necessary.

Finally, you need to add a module reference to the .uproject (or .uplugin) descriptor, in the "Modules" array:

...
"Modules": [
    {
        ...,
        {
            "Name": YourModuleName,
            "Type": "Runtime",
            "LoadingPhase": "Default"
        }
    }
],
...

See the engine documentation for information on options for module type and loading phase. The linked page is written for plugins, but is the most up-to-date and also applies for adding modules at the project level.

Exposing code to other modules

In some cases, your module might not need to expose anything at all. It could just define some AActor/UObject types that will be picked up by the engine for use in the editor only, or it might just be registering some editor extensions. Often, however, you'll write a module that acts as a library, providing types and functionality to be used by other module code. In that case, you need to explicitly provide access to those elements.

I'll just detail the standard approach here. There is an alternative that has some benefits as well as restrictions, but I'll leave that for a later article.

In your public headers, add the YOURMODULENAME_API macro to declarations of types or functions that should be exposed.

// Exposed code from module 'YourModuleName', for use in other modules.

UCLASS()
class YOURMODULENAME_API AMyActor: public AActor
{...};

struct YOURMODULENAME_API FMyStruct
{...};

YOURMODULENAME_API void MyFunction();

Then these specific types/functions will be accessible from within the code of other modules. Your other module will need to add a static dependency in its build file. For example, if SomeOtherModule needs to use types from YourModuleName, add the following to SomeOtherModule.Build.cs:

PublicDependencyModuleNames.Add("YourModuleName");

Examples

The repository containing source code examples for other articles on this site has been written in modular form, so is a good reference. Kantan Charts, also on Github, is another example of using multiple modules, this time in the context of a plugin.

Conclusion

Reducing dependencies and keeping code organized can save you some major headaches down the line. Modules are great, use them!

I'll go into some more details, and also touch on plugins, in another article soon.