NPM private git module on Heroku

Question

I am trying to deploy my app to Heroku however I rely on using some private git repos as modules. I do this for code reuse between projects, e.g. I have a custom logger I us

Answer 1

Basic auth

GitHub has support for basic auth:

"dependencies" : {
    "my-module" : "git+https://my_username:my_password@github.com/my_github_account/my_repo.git"
}

As does BitBucket:

"dependencies" : {
    "my-module": "git+https://my_username:my_password@bitbucket.org/my_bitbucket_account/my_repo.git"
}

But having plain passwords in your package.json is probably not desired.

Personal access tokens (GitHub)

To make this answer more up-to-date, I would now suggest using a personal access token on GitHub instead of username/password combo.

You should now use:

"dependencies" : {
    "my-module" : "git+https://<username>:<token>@github.com/my_github_account/my_repo.git"
}

For Github you can generate a new token here:

https://github.com/settings/tokens

App passwords (Bitbucket)

App passwords are primarily intended as a way to provide compatibility with apps that don't support two-factor authentication, and you can use them for this purpose as well. First, create an app password, then specify your dependency like this:

"dependencies" : {
    "my-module": "git+https://<username>:<app-password>@bitbucket.org/my_bitbucket_account/my_repo.git"
}

[Deprecated] API key for teams (Bitbucket)

For BitBucket you can generate an API Key on the Manage Team page and then use this URL:

"dependencies" : {
    "my-module" : "git+https://<teamname>:<api-key>@bitbucket.org/team_name/repo_name.git"
}

Answer 2

You can use in package.json private repository with authentication example below:

https://usernamegit:passwordgit@github.com/reponame/web/tarball/branchname

Answer 3

Update 2016-03-26

The method described no longer works if you are using npm3, since npm3 fetches all modules described in package.json before running the preinstall script. This has been confirmed as a bug.

The official node.js Heroku buildpack now includes heroku-prebuild and heroku-postbuild, which will be run before and after npm install respectively. You should use these scripts instead of preinstall and postinstall in all cases, to support both npm2 and npm3.

In other words, your package.json should resemble:

 "scripts": {
      "heroku-prebuild": "bash preinstall.sh",
      "heroku-postbuild": "bash postinstall.sh"
    }

---

I've come up with an alternative to Michael's answer, retaining the (IMO) favourable requirement of keeping your credentials out of source control, whilst not requiring a custom buildpack. This was borne out of frustration that the buildpack linked by Michael is rather out of date.

The solution is to setup and tear down the SSH environment in npm's preinstall and postinstall scripts, instead of in the buildpack.

Follow these instructions:

• Create two scripts in your repo, let's call them preinstall.sh and postinstall.sh.
• Make them executable (chmod +x *.sh).
• Add the following to preinstall.sh:

    #!/bin/bash
    # Generates an SSH config file for connections if a config var exists.

    if [ "$GIT_SSH_KEY" != "" ]; then
      echo "Detected SSH key for git. Adding SSH config" >&1
      echo "" >&1

      # Ensure we have an ssh folder
      if [ ! -d ~/.ssh ]; then
        mkdir -p ~/.ssh
        chmod 700 ~/.ssh
      fi

      # Load the private key into a file.
      echo $GIT_SSH_KEY | base64 --decode > ~/.ssh/deploy_key

      # Change the permissions on the file to
      # be read-only for this user.
      chmod 400 ~/.ssh/deploy_key

      # Setup the ssh config file.
      echo -e "Host github.com\n"\
              " IdentityFile ~/.ssh/deploy_key\n"\
              " IdentitiesOnly yes\n"\
              " UserKnownHostsFile=/dev/null\n"\
              " StrictHostKeyChecking no"\
              > ~/.ssh/config
    fi

• Add the following to postinstall.sh:

    #!/bin/bash

    if [ "$GIT_SSH_KEY" != "" ]; then
      echo "Cleaning up SSH config" >&1
      echo "" >&1

      # Now that npm has finished running, we shouldn't need the ssh key/config anymore.
      # Remove the files that we created.
      rm -f ~/.ssh/config
      rm -f ~/.ssh/deploy_key

      # Clear that sensitive key data from the environment
      export GIT_SSH_KEY=0
    fi

• Add the following to your package.json:

  "scripts": {
    "preinstall": "bash preinstall.sh",
    "postinstall": "bash postinstall.sh"
  }
  

• Generate a private/public key pair using ssh-agent.

• Add the public key as a deploy key on Github.
• Create a base64 encoded version of your private key, and set it as the Heroku config var GIT_SSH_KEY.
• Commit and push your app to Github.

When Heroku builds your app, before npm installs your dependencies, the preinstall.sh script is run. This creates a private key file from the decoded contents of the GIT_SSH_KEY environment variable, and creates an SSH config file to tell SSH to use this file when connecting to github.com. (If you are connecting to Bitbucket instead, then update the Host entry in preinstall.sh to bitbucket.org). npm then installs the modules using this SSH config. After installation, the private key is removed and the config is wiped.

This allows Heroku to pull down your private modules via SSH, while keeping the private key out of the codebase. If your private key becomes compromised, since it is just one half of a deploy key, you can revoke the public key in GitHub and regenerate the keypair.

As an aside, since GitHub deploy keys have read/write permissions, if you are hosting the module in a GitHub organization, you can instead create a read-only team and assign a 'deploy' user to it. The deploy user can then be configured with the public half of the keypair. This adds an extra layer of security to your module.

Answer 4

Based on the answer from @fiznool I created a buildpack to solve this problem using a custom ssh key stored as an environment variable. As the buildpack is technology agnostic, it can be used to download dependencies using any tool like composer for php, bundler for ruby, npm for javascript, etc: https://github.com/simon0191/custom-ssh-key-buildpack

1. Add the buildpack to your app:

   $ heroku buildpacks:add --index 1 https://github.com/simon0191/custom-ssh-key-buildpack
   

2. Generate a new SSH key without passphrase (lets say you named it deploy_key)

3. Add the public key to your private repository account. For example:

   • Github: https://help.github.com/articles/adding-a-new-ssh-key-to-your-github-account/

   • Bitbucket: https://confluence.atlassian.com/bitbucket/add-an-ssh-key-to-an-account-302811853.html

4. Encode the private key as a base64 string and add it as the CUSTOM_SSH_KEY environment variable of the heroku app.

5. Make a comma separated list of the hosts for which the ssh key should be used and add it as the CUSTOM_SSH_KEY_HOSTS environment variable of the heroku app.

   # MacOS
   $ heroku config:set CUSTOM_SSH_KEY=$(base64 --input ~/.ssh/deploy_key) CUSTOM_SSH_KEY_HOSTS=bitbucket.org,github.com
   # Ubuntu
   $ heroku config:set CUSTOM_SSH_KEY=$(base64 ~/.ssh/deploy_key) CUSTOM_SSH_KEY_HOSTS=bitbucket.org,github.com
   

6. Deploy your app and enjoy :)