UE4 #includes, Precompiled Headers and IWYU (Include What You Use)

/

Ever had Unreal's build tool suddenly spring some incomprehensible error on you, pointing to a line of code that you didn't touch? If so, read on.

A couple of engine versions back, Epic introduced the 'Include What You Use' model to UE4 in an attempt to help clean up the situation with include dependencies. This was definitely a big improvement, but there was initially very little information given on the changes made and how they applied to game projects. The above docs page is now much better, but still a little unclear on a few things. I'm no authority, but I'm going to try to at least clear up a couple of misunderstandings that I've since seen doing the rounds.

Switching to IWYU

First off, switching to IWYU essentially means doing the following:

  1. Add PCHUsage = PCHUsageMode.UseExplicitOrSharedPCHs; to the .Build.cs file of all your modules.
  2. Delete any existing module precompiled header files you have* (the first include in every cpp file in the old system) if they were nothing more than a list of includes. If you had declarations inside a precompiled header, move them into a new header.
  3. Remove all includes of the old precompiled header.
  4. Fix everything.**

* See final section for why you may not necessarily want to do this step.

** You'll get heaps of errors because you no longer have required headers included. You need to add explicit includes for the types and declarations you need; you can find the header a type is defined in at the bottom of its page in the API reference (Epic in their wisdom decided to discontinue the offline reference as well as butcher the search functionality for the online one, which doesn't help with this). This might seem like a pain, but it's essentially the entire point of the change - being more explicit about includes. It will make your code more robust going forwards.

All neat and sorted?

Now, the idea is that at this point, your code is self-contained and no longer dependent upon includes within precompiled headers to compile. As stated in the documentation: "With IWYU, every file includes only what it needs, and any PCH file we choose to use, purely acts as a layer of optimization on top of the underlying source files. They can be modified to minimize build times, independently of changing the source files themselves, and without affecting whether the code compiles successfully or not."

This is a bit misleading though. Certain modules in the engine (in particular Core, CoreUObject, Engine, Slate and UnrealEd) create shared precompiled headers, which by default Unreal Build Tool will use when compiling any module dependent on these modules. In practice, that's every module; furthermore, these are some of the biggest engine modules and the combined includes inside them account for a significant portion of the engine's headers. These precompiled headers are included on the command line which means the headers they reference are essentially included before all your explicit includes, for each of your cpp files. You can see this in action by adding a cpp file that doesn't include any engine headers. Add a reference to a common engine type - for example a global variable of type UStaticMesh* - and you'll see that so long as you have a dependency for the type's module (in this case Engine), the code will compile without need for an explicit include. This is precisely the kind of thing that IWYU was supposed to avoid.

So, how to check that your code really is well formed and self-contained? This can be done, but you need to adjust the build configuration to completely disable use of precompiled headers. You also need to turn off unity builds, since these cause cpp files to be merged together, meaning one cpp might compile only because another one that is merged above it includes something it relies on. On the command line, this amounts to adding: -NoPCH -NoSharedPCH -DisableUnity. To make the change inside BuildConfiguration.xml, add the following to the BuildConfiguration section:

<bUsePCHFiles>false</bUsePCHFiles>
<bUseSharedPCHs>false</bUseSharedPCHs>
<bUseUnityBuild>false</bUseUnityBuild>

If you do this for your project, you will likely find that even after IWYU updates, your code is still not truly self-contained and you have to go through step 4 again. I think it's worth doing, and if you have automated builds set up then running them with the above changes at least some of the time is a good idea. It's especially important for any code which you're distributing for reuse, such as plugins.

Note that these adjustments do mean much longer build times, so I don't recommend using them as default for regular iterative builds.

Why bother at all?

For me, just not liking the idea of having code that is technically malformed despite compiling is reason enough to put the time in to maintaining self-contained includes in my projects.

If that's not enough though, there's another reason: those annoying errors alluded to at the top of the article. They generally come about because of a missing include, and the nature of C++ is such that the description in the error is often misleading if you don't have many years of experience behind you to help decrypt it.

In UE4 it's even worse - they show up at apparently arbitrary times, on code that was compiling fine before and hasn't been changed. The reason for this is PCHs and unity builds (and sometimes just UBT being weird). If your code isn't fully consistent in its includes, it will often still compile due to the way these build optimizations work. But the precise contents of engine shared PCHs, and the composition and order of cpp files in a unity build, is dynamic and not well defined. A change in some seemingly unrelated source file, an engine update, an added or removed plugin, or some other adjustment to the build environment - these can all result in changes to the above, which in turn can cause these compilation errors. The thing to remember is, when you get these errors, it means your code was malformed all along and you just got away with it. If you make the effort to keep it consistent in the first place, then you (or someone else using your code) won't have to deal with these errors.

A note on performance and precompiled headers

The procedure given by Epic for switching to IWYU implies removing project-level precompiled headers (PCHs) in favour of using only the engine shared PCHs. In most cases, this will be fine, but something bugged me about this from the start and I only recently dug into it. PCHs speed up compilation by preprocessing code in header files, thereby preventing the same work being done over and over by the compiler in each translation unit (cpp file, or in the case of unity builds, each batch of cpp files) that includes the given header. Now it's true, the engine is huge and dwarfs all projects, so it's most important to have PCHs covering the core engine headers. For larger projects though, do we really want to give up project-level PCHs?

It's more maintenance, and it will depend heavily on the size, organization and level of flux of your project's codebase, but it's worth being aware that the default IWYU transition could actually lower compile times relative to having a carefully crafted dedicated PCH in your module(s). I recently did a reorganization of the codebase for my primary project. It's about 150,000 lines, now split into roughly 20 modules (a number of them very small). In the process I changed a few of the larger, more stable modules back to having their own private PCH file, and saw a very significant improvement in compile times.

Note that using a private PCH for a project module isn't really going against IWYU, it's just that the documentation gives the impression you no longer need them. While that is generally going to be true, you can use a PCH in your module while still conforming to IWYU practices.

If you want to try it out, do the following:

  1. Add a header file to your module's Private folder. Name is unimportant.
  2. In your module's .Build.cs, after the PCHUsage = PCHUsageMode.UseExplicitOrSharedPCHs; line, add: PrivatePCHHeaderFile = "Private/YourPCHFile.h";
  3. Add a bunch of includes to your precompiled header include file.

Step 3 needs to be done right. PCHs are a tradeoff - generally, the more you add, the more you reduce compile time of translation units. However, you also increase the time it takes to generate the PCH, and more importantly, you increase the frequency with which it will need to be regenerated, and each regeneration means a recompile of every translation unit in the module. That's why it's super important to include only those headers that are stable and will very rarely be modified. There's also little point including headers that will only ever be included by one or two cpp files.

One other point - when you specify a PCH like this, Unreal will no longer use the engine shared PCHs, so you should explicitly add includes for engine header files that your module uses into your PCH. Having to manage this is a pain, so it's only worth considering on large projects where compile time is problematic.