Environmental Variables in a Deployment Pipeline: A Workflow / by Alexander Hadik

Part of my responsibility on the Bluemix Design team at IBM is to create small web apps on various cloud providers to learn how our products work - so we can design the best possible experience for them. After standing up a handful of Node apps with local, staging and production deployments, I got pretty tired of dealing with different sets of environmental variables for each environment - especially since you can't put API keys and passwords (a common type of data stored this way) in public repositories for obvious security reasons.

So how to reconcile open-source code bases that are part of a deployment pipeline that relies on these environmental variables? I'll explore that in this post.

After a lot of experimentation, the basic steps I've put together are:

  • Encryption
  • Selective, automated decryption

Let's set up some context. Consider this simple Node app with the following file hierarchy:

Notice the private directory that contains some basic Bash scripts where we store environmental variables for each deployment. So for example, this app might have a different hosted database for staging and production. The credentials needed to access these two databases are stored in env-staging and env as environmental variables like so:

export DB_CRED=abc123

On staging servers, during deployment, env-staging is sourced and on production servers, env is sourced. Then the Node app uses these values with process.env.DB_CRED. The problem is, if a developer has to modify the variables for some reason, they need to be included in the deployment pipeline so the files are sent to staging and production servers to be sourced. This means the files with sensitive information are either sucked up into version control, or have to be manually sent to the deployment environments through some other means.

If part of the deployment pipeline is a testing platform like Travis, options are even more limited because now the environmental variables are required to be included in the version control so they can be sourced for building in a Travis environment.

Encryption

So what to do? Encrypt! Instead of committing the plain-text environmental variables, instead commit an encrypted version of them. Then, ensure that each deployment environment has the password to decrypt the file for that environment.

There are a number of tools to encrypt and decrypt plain-text files. I'll be using a simple Node module called config-leaf which uses the crypto module. config-leaf exposes two commands inside the app's root directory, encrypt and decrypt. The full executions of this module can be aliased as scripts inside the package.json file:

"encrypt": "encrypt private/env private/env.cast5 && encrypt private/staging-env private/staging-env.cast5",
"decrypt": "decrypt private/env.cast5 private/env && decrypt private/staging-env.cast5 private/staging-env"

The argument order of the config-leaf commands is <PLAINTEXT INPUT> <ENCRYPTED OUTPUT> for encryption and <ENCRYPTED INPUT> <PLAINTEXT OUTPUT> for decryption.

So this module solves the encryption part of the problem. Before committing changes, encrypt the environmental variables with npm run encrypt, and make sure that the plain-text files are in a .gitignore file. Then add, commit and push the encrypted versions as part of the changes. Now the sensitive info is part of public source control with no concerns!

config-leaf is a great tool, however it relies on user interactivity to type in a password during encryption and decryption.

npm run encrypt
Enter the config password (env):

This becomes rather tiresome. On a local system, a password, perhaps stored as an environmental variables, can be piped in like so:

echo $PW | encrypt private/env private/env.cast5

On some cloud environments though, when executing this command automatically during a build process, this piping doesn't work (any explanation as to why would be very helpful). To remedy this, I created a modified version of config-leaf to which you can pass the name of an environmental variable that stores a password for you like this:

encrypt private/env private/env.cast5 --PW=ENV_PW

The only difference here is the --PW flag which indicates that the env file should be encrypted using the value found at $ENV_PW or whatever environmental variable you fancy.

Decryption

Now it's time for decryption. Again, let's get some context. Since we left off, new code along with sensitive, encrypted environmental variables were committed to the staging branch in GitHub. This triggered a deployment to the Staging environment which consists of several virtual servers. Another post is forthcoming on deploying to multiple servers with PM2.

The Staging deployment process tunnels into the staging servers, pulls the new code from the staging branch, and builds it using a build command, such as npm run build. Part of the build process is decrypting the environmental variables included in the new code. When the staging servers were originally stood up, the master password used for encryption was set up as an environmental variable. This is a single, static configuration that is done once and never again needs to be touched so long as the password used for encryption doesn't change.

The build process now simply uses the decrypt command like so:

decrypt private/env.cast5 private/env --PW=ENV_PW && decrypt private/env-staging.cast5 private/env-staging --PW=ENV_PW

Again, this is included in the package.json file as a script, or equivalent:

"decrypt": "decrypt private/env.cast5 private/env --PW=ENV_PW && decrypt private/env-staging.cast5 private/env-staging --PW=ENV_PW"

So a simple call of npm run decrypt will decrypt the environmental variable files and write them to disk.

In Review  

To recap the basic process described here, the steps are:

-Add the modifed version of config-leaf that uses environmental variables to package.json using the root: http://www.github.com/ahadik/config-leaf
-Add a password for encryption as an environmental variable to the local dev environment and each deployment environment.
-Add the same password under the same variable name to each testing, staging or production server.
-Ensure your build process includes decrypting the environmental variables into their original file names and sources the right config file for the environment its in (dev, testing, staging, production, etc.)
-Add plaintext files containing environmental variables to .gitignore.
-Encrypt the plaintext files using the encrypt <PLAINTEXT INPUT> <ENCRYPTED OUTPUT> --PW=<ENV VAR PW> or alias the command in package.json.
-Add, commit and push encrypted environmental variables to a repository.
-Let the automated deployment and build pipeline take care of the rest! If configured correctly, the build process will decrypt and source the appropriate environmental variable file.

This is a process I've worked out that works for small scale apps. I'm sure there are better, more robust processes for larger apps. More importantly, if don't have to worry about exposing API keys inside your repository, perhaps because it's not a public repo, then this process is a bit overkill. However, for the many apps I stand up and want to expose the repositories for, this workflow has become crucial.