SetDllDirectory inheritance issue

[Created 09/Sep/2011: First public version of this page.]

Contents:

Executive summary:

In Windows, the SetDllDirectory API does not only affect the calling process; it also affects how child processes load implicitly-linked DLLs.

This isn't the end of the world but it is a pain in the arse.

Background:

(You can skip this section if you already know about the binary planting issue and the way SetDllDirectory can be used to make Windows programs more secure.)

There is a 'binary planting' issue in Windows where some programs can be tricked into executing code inside DLLs placed in the current working directory. The issue can be prevented entirely by setting CWDIllegalInDllSearch = 0xFFFFFFFF in the registry, but this is not done by default because it will break some software. Instead, the default registry setting only fixes the binary planting issue when the current directory is on a network share, leaving software vulnerable to it on local drives.

For example, if you download a zip archive containing lots of images and extract it to your local drive, when you double-click one of the images your picture viewer may be tricked into executing code within a DLL that was hiding among the files.

Microsoft had a difficult choice between making Windows more secure and breaking software that people use. The compromise they decided on is completely understandable.

(Having said that, I wish they had done a bit more. New software, including a major graphics-card driver and a major game-distribution platform, is still being released which breaks on systems configured to be more secure. Loading a DLL via the CWD could trigger a non-modal alert in Action Center (or a taskbar bubble or whatever). Something like, "Hey, just to let you know, some terrible code is running on your system, probably written by people who hate kittens!" Developers would be shamed into fixing their code and users would start to distrust anything left unfixed. I'm not advocating UAC-style modal dialogs that require confirmation every time the OS detects crappy code -- that would drive people crazy -- but silently ignoring the issue means it will never go away. MS are expecting developers to be far more proactive about security than they ever without coercion. There is no hope of progressing to a secure-by-default world when there are no consequences for writing new software that depends on less secure behaviour.)

Regardless of the CWDIllegalInDllSearch registry setting, individual applications can always explicitly opt-in to the more secure mode by calling SetDllDirectory(""). In our example, if the picture viewer had done that then it could have avoided the risk of loading the unwanted DLL, even on system with the default registry setting.

(At least, assuming the picture viewer doesn't load any plugins (etc.) which might undo the SetDllDirectory call, and assuming that SetDllDirectory is called early enough, which may actually be impossible if the program is implicitly-linked to a missing/optional DLL or if the program calls a framework, like earlier versions of MFC, which triggers dangerous DLL-lookups before control has even passed to the main program code. Those are all real issues, and in many ways more significant than the one discussed on this page. I won't go into them further here as there's already information about them on the web.)

If you want to read more about Binary Planting in general, here are some pages by other people (no affiliation) which I found good/useful:

The problem with SetDllDirectory:

So,

  • Some programs require the old, less secure behaviour and will never be fixed.
  • Most programs can, and should, opt into the new, more secure behaviour.
  • Therefore, good developers can simply call SetDllDirectory("") and know they've done their bit.

...Unfortunately, it's not that simple.

The current documentation for SetDllDirectory only talks about the process that calls it, and what happens when that process calls LoadLibrary or LoadLibraryEx. From reading it, and other advice, you could conclude that you should call SetDllDirectory("") provided you're sure none of the code in your process depends on the current directory being in the DLL search path. That conclusion is incorrect.

SetDllDirectory("") also affects how child processes loads their DLLs.

Good news: It does not affect dynamic linking. When a child process calls LoadLibrary after starting, the parent's settings do not matter. It's up to the child process how it finds those DLLs.

Bad news: It affects implicit linking. When a child process is starting up and Windows itself is finding the DLL dependencies, whether or not the current directory is searched depends on whether or not the parent process has called SetDllDirectory("").

(At least, that is what happens on Windows 7 x64 SP1. A sample project with source-code is provided for anyone who wants to try it themselves.)

As a consequence, you cannot opt-in to the more secure mode with your own code without potentially breaking other programs that your code may be asked to launch. Programs which are completely outside your knowledge and control. Also, since the DLL search path is (understandably) process-wide, you cannot safely change it temporarily while you launch other processes.

Okay, you can go ahead and call SetDllDirectory("") in programs which have no facility for running third-party tools and which are completely isolated from any shell extensions or other plugins that may do similar. If your program is dedicated to a very fixed, non-interactive task, and never shows a File Open dialog or anything like that, then you're probably in the clear.

For everything else, we're left with a choice between being breaking a few things (in ways which will be hard to diagnose when it happens) or reduced security. Basically, the same choice Microsoft had when addressing the problem.

Could it be done better?

There may be factors I'm unaware of, so I don't know how realistic this idea is, but it seems to me that Windows could/should allow applications to specify the DLL-lookup behaviour as part of their manifests and entirely remove the influence the parent process has on its children during startup.

A manifest setting would fix the implicit-linked DLL issue both ways. (I've mainly talked about children that need CWD in search path while their parents want it excluded, but the problem exists the other way as well. It the parent has not called SetDllDirectory, leaving CWD included, but the child wants CWD excluded, the child currently has no say in the matter until it is too late.)

A manifest setting would also fix problems where SetDllDirectory is called too late, which is sometimes impossible to solve as things stand. (e.g. Frameworks which call LoadLibrary before passing control to the main program.)

A manifest setting could also, perhaps as an option, specify that the CWD cannot be put back into the DLL search path for the process. That would prevent plugin code (etc.) from accidentally resetting the process to the more dangerous defaults.

The existing SetDllDirectory behaviour could still exist, of course; otherwise existing programs that call SetDllDirectory for security would become less secure until updated to use the manifest. The manifest setting would just override any changes to the process made via SetDllDirectory.

That seems like several upsides and no downsides, apart from the development effort. Is there any reason for Microsoft not to do this?

Download the sample code:

Here is the simple C++ code for VS2008 which I used to confirm the behaviour, after a bug report about it in a much more complex application.

Building the solution will output three binaries below the Debug or Release dirs:

  • DLL\TestLinkLib.dll -- Wizard-generated DLL that just exports a function which returns "42".
  • EXE\TestLink.exe -- EXE which is implicitly-linked to the DLL, calls the function and verifies that it returns "42".
  • TestRun.exe -- EXE which will run TestLink.exe in three different ways.

If you run TestRun.exe you should see three message boxes, one for each time TestLink.exe is run:

  1. The first message should indicate success. This is TestLink.exe being run under default conditions.
  2. The second message should indicate failure and come from Windows itself. This is TestLink.exe being run after SetDllDirectory(""), and being unable to find TestLinkLib.dll as a result.
  3. The third message should indicate success again. This is TestLink.exe being run after a call to SetDllDirectory(NULL) has reset the DLL search path back to the default.

Note that all three runs will work okay if you change TestLink.exe to load the DLL dynamically, via LoadLibrary, instead of using implicit linking.

On the other hand, all three runs will fail if you have CWDIllegalInDllSearch = 0xFFFFFFFF in the registry. That forces the most secure mode on for all applications. It's good for increased security, and normally how I run my own system, but it won't let you see the thing I'm talking about.