How do I securely configure a CI server to digitally sign binaries?

Question

There are many sites that explain how to run signtool.exe on a .pfx certificate file, which boil down to:

signtool.exe sign /f mycert.pfx /p mypassword /t http://timestamp.server.com \
  /d "My description"   file1.exe file2.exe

I have a continuous integration CI process setup (using TeamCity) which like most CI processes, does everything: checks out source, compiles, signs all .exes, packages into an installer, and signs the installer .exe. There are currently 3 build agents, running identical VMs, and any of them can run this process.

Insecure implementation

To accomplish this today, I do a couple Bad Things(TM) as far as security is concerned: the .pfx file is in source control, and the password for it is in the build script (also in source control). This means that any developers with access to source code repository can take the pfx file and do whatever nefarious things they'd like with. (We're a relatively small dev shop and trust everyone with access, but clearly this still isn't good).

The ultimate secure implementation

All I can find about doing this "correctly", is that you:

• Keep the pfx and password on some secure medium (like an encrypted USB drive with finger-based unlock), and probably not together
• Designate only a couple of people to have access to sign files
• Only sign final builds on a non-connected, dedicated machine that's kept in a locked-up vault until you need to bring it out for this code-signing ceremony.

While I can see merit in the security of this.. it is a very heavy process, and expensive in terms of time (running through this process, securely keeping backups of certificates, ensuring the code-signing machine is in a working state, etc).

I'm sure some people skip steps and just manually sign files with the certificate stored on their personal system, but that's still not great.

It also isn't compatible with signing files that are then used within the installer (which is also built by the build server) -- and this is important when you have an installed .exe that has a UAC prompt to get admin access.

Middle ground?

I am far more concerned with not presenting a scary "untrusted application" UAC prompt to users than proving it is my company. At the same time, storing the private key AND password in the source code repository that every developer (plus QA and high-tier tech support) have access to is clearly not a good security practice.

What I'd like is for the CI server to still sign during the build process like it does today, but without the password (or private key portion of the certificate) to be accessible to everyone with access to the source code repository.

Is there a way to keep the password out of the build or secure somehow? Should I be telling signtool to use a certificate store (and how do I do that, with 3 build agents and the build running as a non-interactive user account)? Something else?

Answer 1

I ended up doing a very similar approach to what @GiulioVlan suggested, but with a few changes.

MSBuild Task

I created a new MSBuild task that executes signtool.exe. This task serves a couple main purposes:

• It hides the password from ever being displayed
• It can retry against the timestamp server(s) upon failures
• It makes it easy to call

Source: https://gist.github.com/gregmac/4cfacea5aaf702365724

This specifically takes all output and runs it through a sanitizer function, replacing the password with all *'s.

I'm not aware of a way to censor regular MSBuild commands, so if you pass the password on commandline directly to signtool.exe using it will display the password -- hence the need for this task (aside from other benefits).

Password in registry

I debated about a few ways to store the password "out-of-band", and ended up settling on using the registry. It's easy to access from MSBuild, it's fairly easy to manage manually, and if users don't have RDP and remote registry access to the machine, it's actually reasonably secure (can anyone say otherwise?). Presumably there are ways to secure it using fancy GPO stuff as well, but that's beyond the length I care to go.

This can be easily read by msbuild:

$(Registry:HKEY_LOCAL_MACHINE\SOFTWARE\1 Company Dev@CodeSigningCertPassword)

And is easy to manage via regedit:

Why not elsewhere?

• In the build script: it's visible by anyone with source code
• Encrypted/obfuscated/hidden in source control: if someone gets a copy of the source, they can still figure this out
• Environment variables: In the Teamcity web UI, there is a detail page for each build agent that actually displays all environment variables and their values. Access to this page can be restricted but it means some other functionality is also restricted
• A file on the build server: Possible, but seems a bit more likely it's inadvertently made accessible via file sharing or something

Calling From MSBuild

In the tag:

<Import Project="signtool.msbuild.tasks"/>

(You could also put this in a common file with other tasks, or even embed directly)

Then, in whichever target you want to use for signing:

<SignTool  SignFiles="file1.exe;file2.exe" 
   PfxFile="cert.pfx" 
   PfxPassword="$(Registry:HKEY_LOCAL_MACHINE\SOFTWARE\1 Company Dev@CodeSigningCertPassword)"
   TimestampServer="http://timestamp.comodoca.com/authenticode;http://timestamp.verisign.com/scripts/timstamp.dll" />

So far this works well.

Answer 2

One common technique is to leave keys and certificates in Version Control, but protect them with a password or passphrase. The password is saved in environment variables local to the machine, which can be easily accessed from scripts (e.g. %PASSWORD_FOR_CERTIFICATES%).

One must be careful not to log these values in plain text.