How to handle secrets in Google App Engine?

Question

My application needs a bunch of secrets to run: database credentials, API credentials, etc. It\'s running in Google App Engine Standard Java 11. I need these secrets

Answer 1

[Update] (as of Feb 2020) GCP's Secret Manager is in beta, see:

https://cloud.google.com/secret-manager/docs/overview

For Java-specific implementation, see: https://cloud.google.com/secret-manager/docs/creating-and-accessing-secrets#secretmanager-access-secret-version-java

Your specific solution would depend how your app is set up, but you should be able to access the secret(s) and create environment variables with the values or otherwise pass them to your app.

You can use GCP IAM to create a service accounts to manage access or add a role like Secret Manager Secret Accessor to an existing member/service (e.g., in this case, I added that permision to the App Engine default service account).

I tried it out with Node.js on GAE standard, and it seems to work well; I didn't do any performance tests but it should be fine, particularly if you primarily need the secrets on app start or as part of a build process.

For local (non-GCP) development/testing, you can create a service account with appropriate secret manager permissions and get the json service key. You then set an environment variable named GOOGLE_APPLICATION_CREDENTIALS to the path of the file, e.g.:

export GOOGLE_APPLICATION_CREDENTIALS=/path/to/local_service_key.json

and the app running in that shell session should pick up the permissions without any additional auth code. See: https://cloud.google.com/docs/authentication/getting-started (You would want to exclude the key file from version control.)

Answer 2

Berglas looks interesting.

Another option is to put the secrets in app.yaml file(s) (you can have more than one) and encrypt it before committing it to version control.

There are many tools to encrypt secrets before putting them in version control, like https://github.com/StackExchange/blackbox

Pros:

• Very versatile
• I find it simple to understand compared to other options
• Easy to get started

Cons:

• You can't really remove access for a person (since the file could always be copied) so you have rotate secrets sometimes
• Can be hard to keep the unencrypted files out of the repo. Ones you get used to it, and have ignore files and/or scripts, it's usually OK.

Answer 3

At this date, App Engine Standard Standard does not have a Google provided solution for storing application secrets.

[UPDATE]

I noticed your comment on another answer that you require environment variables to be valid before you have application control. In that case, you have no options for App Engine today. I would deploy to a different service (Kubernetes) better suited for your system goals that can provided managed secrets.

[END UPDATE]

You have two choices for secrets for App Engine Standard:

1. Store the secrets as environment variables in app.yaml
2. Store the secrets someplace else.

For both options, you can add a layer of security by encrypting them. However, adding encryption adds another secret (decryption key) that you must somehow provide to your app. The chicken-or-egg situation.

App Engine Standard uses a Service Account. This service account can be used as an identity to control access to other resources. Examples of other resources are KMS and Cloud Storage. This means that you can securely access KMS or Cloud Storage without adding another secret to App Engine.

Let's assume that your company wants all application secrets encrypted. We can use the App Engine Service Account as the identity authorized to access KMS for a single key.

Note: The following examples use Windows syntax. Replace the line continuation ^ with \ for Linux/macOS.

Create the KMS Keyring. Keyrings cannot be deleted, so this is a one-time operation.

set GCP_KMS_KEYRING=app-keyring
set GCP_KMS_KEYNAME=app-keyname

gcloud kms keyrings create %GCP_KMS_KEYRING% --location global

Create the KMS Key.

gcloud kms keys create %GCP_KMS_KEYNAME% ^
--location global ^
--keyring %GCP_KMS_KEYRING% ^
--purpose encryption

Add the service account to the KMS policy for the keyring and key that we created.

This will allow App Engine to decrypt data without requiring secrets for KMS. The service account identity provides access control. No roles are required for KMS. You will need to provide the KMS Keyring and Keyname which can be included in app.yaml.

set GCP_SA=<replace with the app engine service acccount email affffdress>
set GCP_KMS_ROLE=roles/cloudkms.cryptoKeyDecrypter

gcloud kms keys add-iam-policy-binding %GCP_KMS_KEYNAME% ^
--location global ^
--keyring %GCP_KMS_KEYRING% ^
--member serviceAccount:%GCP_SA% ^
--role %GCP_KMS_ROLE%

For this example, let's assume that you need to access a MySQL database. We will store the credentials in a JSON file and encrypt it. The file is named config.json.

{
        "DB_HOST": "127.0.0.1",
        "DB_PORT": "3306",
        "DB_USER": "Roberts",
        "DB_PASS": "Keep-This-Secret"
}

Encrypt config.json using Cloud KMS and store the encrypted results in config.enc:

call gcloud kms encrypt ^
--location=global ^
--keyring %GCP_KMS_KEYRING% ^
--key=%GCP_KMS_KEYNAME% ^
--plaintext-file=config.json ^
--ciphertext-file=config.enc

The encrypted file can be stored in Cloud Storage. Since it is encrypted, you could store the file with your build files, but I do not recommend that.

The final piece is to write the code in Java that is part of your program that uses KMS to decrypt the file config.enc using KMS. Google has a number of examples of KMS decryption:

Java KMS Decrypt

Java Samples

Answer 4

The options are more or less what you have mentioned. Google Cloud Key Management is a great way of dealing with credentials and secret information in general. It has an REST API with methods which has to be used in order to achieve your goals with it. For example, you have the "get" method for getting cryptokeys, keyRings, cryptoKeysVersions etc..

For example here there is a guide on how to set up authentication for your app. Here there is also a nice code example for this on github.

If you think extra security is required you can also use the secret rotation technique.

Hard coding the values in the app.yaml is an option, too, although it is not recommended as the best practice. The app.yaml will not live only on your machine. It will be deployed, also and you can even see it from the App engine, "Versions" -> "Config" panel in the GCP Console. So, you do not need to worry about it.

Answer 5

For secret management, I'm personally fan of Berglas project. It's based on KMS and, in addition, manage DEK and KEK

It's today write in Go and it's not compliant with Java. I wrote a python library for some colleagues. I can write a Java package if you plan to use it. It's not very hard.

Let me know