Python 使用 app.yaml 在 GAE 中安全地存储环境变量

Securely storing environment variables in GAE with app.yaml

提问by Ben

I need to store API keys and other sensitive information in app.yamlas environment variables for deployment on GAE. The issue with this is that if I push app.yamlto GitHub, this information becomes public (not good). I don't want to store the info in a datastore as it does not suit the project. Rather, I'd like to swap out the values from a file that is listed in .gitignoreon each deployment of the app.

我需要将 API 密钥和其他敏感信息app.yaml作为环境变量存储在 GAE 上进行部署。这样做的问题是，如果我推app.yaml送到 GitHub，这些信息就会公开（不好）。我不想将信息存储在数据存储中，因为它不适合项目。相反，我想从.gitignore应用程序的每个部署中列出的文件中换出值。

Here is my app.yaml file:

这是我的 app.yaml 文件：

application: myapp
version: 3 
runtime: python27
api_version: 1
threadsafe: true

libraries:
- name: webapp2
  version: latest
- name: jinja2
  version: latest

handlers:
- url: /static
  static_dir: static

- url: /.*
  script: main.application  
  login: required
  secure: always
# auth_fail_action: unauthorized

env_variables:
  CLIENT_ID: ${CLIENT_ID}
  CLIENT_SECRET: ${CLIENT_SECRET}
  ORG: ${ORG}
  ACCESS_TOKEN: ${ACCESS_TOKEN}
  SESSION_SECRET: ${SESSION_SECRET}

Any ideas?

有任何想法吗？

采纳答案by Martin Omander

If it's sensitive data, you should not store it in source code as it will be checked into source control. The wrong people (inside or outside your organization) may find it there. Also, your development environment probably uses different config values from your production environment. If these values are stored in code, you will have to run different code in development and production, which is messy and bad practice.

如果是敏感数据，则不应将其存储在源代码中，因为它将被检入源代码管理。错误的人（组织内部或外部）可能会在那里找到它。此外，您的开发环境可能使用与生产环境不同的配置值。如果这些值存储在代码中，您将不得不在开发和生产中运行不同的代码，这是一种混乱和不好的做法。

In my projects, I put config data in the datastore using this class:

在我的项目中，我使用这个类将配置数据放在数据存储中：

from google.appengine.ext import ndb

class Settings(ndb.Model):
  name = ndb.StringProperty()
  value = ndb.StringProperty()

  @staticmethod
  def get(name):
    NOT_SET_VALUE = "NOT SET"
    retval = Settings.query(Settings.name == name).get()
    if not retval:
      retval = Settings()
      retval.name = name
      retval.value = NOT_SET_VALUE
      retval.put()
    if retval.value == NOT_SET_VALUE:
      raise Exception(('Setting %s not found in the database. A placeholder ' +
        'record has been created. Go to the Developers Console for your app ' +
        'in App Engine, look up the Settings record with name=%s and enter ' +
        'its value in that record\'s value field.') % (name, name))
    return retval.value

Your application would do this to get a value:

您的应用程序会这样做以获得一个值：

API_KEY = Settings.get('API_KEY')

If there is a value for that key in the datastore, you will get it. If there isn't, a placeholder record will be created and an exception will be thrown. The exception will remind you to go to the Developers Console and update the placeholder record.

如果数据存储区中有该键的值，您将获得它。如果没有，将创建占位符记录并抛出异常。异常会提醒您转到开发人员控制台并更新占位符记录。

I find this takes the guessing out of setting config values. If you are unsure of what config values to set, just run the code and it will tell you!

我发现这消除了设置配置值的猜测。如果您不确定要设置什么配置值，只需运行代码，它就会告诉您！

The code above uses the ndb library which uses memcache and the datastore under the hood, so it's fast.

上面的代码使用了 ndb 库，它使用了内存缓存和引擎盖下的数据存储，因此速度很快。

---

---

Update:

更新：

jelderasked for how to find the Datastore values in the App Engine console and set them. Here is how:

jelder询问如何在 App Engine 控制台中查找数据存储区值并设置它们。方法如下：

1. Go to https://console.cloud.google.com/datastore/

2. Select your project at the top of the page if it's not already selected.

3. In the Kinddropdown box, select Settings.

4. If you ran the code above, your keys will show up. They will all have the value NOT SET. Click each one and set its value.

1. 转至https://console.cloud.google.com/datastore/

2. 如果尚未选择，请在页面顶部选择您的项目。

3. 在种类下拉框中，选择设置。

4. 如果你运行上面的代码，你的密钥就会显示出来。它们都将具有值NOT SET。单击每个并设置其值。

Hope this helps!

希望这可以帮助！

回答by Gwyn Howell

Best way to do it, is store the keys in a client_secrets.json file, and exclude that from being uploaded to git by listing it in your .gitignore file. If you have different keys for different environments, you can use app_identity api to determine what the app id is, and load appropriately.

最好的方法是将密钥存储在 client_secrets.json 文件中，并通过在 .gitignore 文件中列出它来将其从上传到 git 中排除。如果不同环境有不同的key，可以使用app_identity api来判断app id是什么，并适当加载。

There is a fairly comprehensive example here -> https://developers.google.com/api-client-library/python/guide/aaa_client_secrets.

这里有一个相当全面的例子 -> https://developers.google.com/api-client-library/python/guide/aaa_client_secrets。

Here's some example code:

下面是一些示例代码：

# declare your app ids as globals ...
APPID_LIVE = 'awesomeapp'
APPID_DEV = 'awesomeapp-dev'
APPID_PILOT = 'awesomeapp-pilot'

# create a dictionary mapping the app_ids to the filepaths ...
client_secrets_map = {APPID_LIVE:'client_secrets_live.json',
                      APPID_DEV:'client_secrets_dev.json',
                      APPID_PILOT:'client_secrets_pilot.json'}

# get the filename based on the current app_id ...
client_secrets_filename = client_secrets_map.get(
    app_identity.get_application_id(),
    APPID_DEV # fall back to dev
    )

# use the filename to construct the flow ...
flow = flow_from_clientsecrets(filename=client_secrets_filename,
                               scope=scope,
                               redirect_uri=redirect_uri)

# or, you could load up the json file manually if you need more control ...
f = open(client_secrets_filename, 'r')
client_secrets = json.loads(f.read())
f.close()

回答by therewillbesnacks

It sounds like you can do a few approaches. We have a similar issue and do the following (adapted to your use-case):

听起来你可以做一些方法。我们有一个类似的问题，并执行以下操作（适合您的用例）：

• Create a file that stores any dynamic app.yaml values and place it on a secure server in your build environment. If you are really paranoid, you can asymmetrically encrypt the values. You can even keep this in a private repo if you need version control/dynamic pulling, or just use a shells script to copy it/pull it from the appropriate place.
• Pull from git during the deployment script
• After the git pull, modify the app.yaml by reading and writing it in pure python using a yaml library

• 创建一个存储任何动态 app.yaml 值的文件，并将其放置在构建环境中的安全服务器上。如果您真的很偏执，您可以对这些值进行非对称加密。如果您需要版本控制/动态拉取，您甚至可以将其保存在私有存储库中，或者仅使用 shell 脚本从适当的位置复制/拉取它。
• 在部署脚本期间从 git 中拉取
• git pull 之后，修改 app.yaml，通过使用 yaml 库在纯 python 中读写

The easiest way to do this is to use a continuous integration server such as Hudson, Bamboo, or Jenkins. Simply add some plug-in, script step, or workflow that does all the above items I mentioned. You can pass in environment variables that are configured in Bamboo itself for example.

最简单的方法是使用持续集成服务器，例如Hudson、Bamboo或Jenkins。只需添加一些插件、脚本步骤或工作流来完成我提到的所有上述项目。例如，您可以传入 Bamboo 本身配置的环境变量。

In summary, just push in the values during your build process in an environment you only have access to. If you aren't already automating your builds, you should be.

总之，只需在您只能访问的环境中的构建过程中推送值。如果你还没有自动化你的构建，你应该是。

Another option option is what you said, put it in the database. If your reason for not doing that is that things are too slow, simply push the values into memcache as a 2nd layer cache, and pin the values to the instances as a first-layer cache. If the values can change and you need to update the instances without rebooting them, just keep a hash you can check to know when they change or trigger it somehow when something you do changes the values. That should be it.

另一个选项选项是你说的，把它放在数据库中。如果您不这样做的原因是事情太慢，只需将值作为第二层缓存推送到 memcache 中，并将这些值作为第一层缓存固定到实例中。如果值可以更改并且您需要在不重新启动它们的情况下更新实例，只需保留一个哈希值，您可以检查它们何时更改或在您更改值时以某种方式触发它。应该是这样。

回答by Bernd Verst

My approach is to store client secrets onlywithin the App Engine app itself. The client secrets are neither in source control nor on any local computers. This has the benefit that anyApp Engine collaborator can deploy code changes without having to worry about the client secrets.

我的方法是仅将客户端机密存储在 App Engine 应用程序本身中。客户端机密既不在源代码管理中，也不在任何本地计算机上。这样做的好处是任何App Engine 协作者都可以部署代码更改，而不必担心客户端机密。

I store client secrets directly in Datastore and use Memcache for improved latency accessing the secrets. The Datastore entities only need to be created once and will persist across future deploys. of course the App Engine console can be used to update these entities at any time.

我将客户端机密直接存储在 Datastore 中，并使用 Memcache 来改善访问机密的延迟。数据存储实体只需创建一次，并将在未来的部署中持续存在。当然，App Engine 控制台可用于随时更新这些实体。

There are two options to perform the one-time entity creation:

有两个选项可以执行一次性实体创建：

• Use the App Engine Remote APIinteractive shell to create the entities.
• Create an Admin only handler that will initialize the entities with dummy values. Manually invoke this admin handler, then use the App Engine console to update the entities with the production client secrets.

• 使用 App Engine Remote API交互式 shell 创建实体。
• 创建一个仅限管理员的处理程序，它将使用虚拟值初始化实体。手动调用此管理处理程序，然后使用 App Engine 控制台使用生产客户端机密更新实体。

回答by jla

You can use the -E command line option of appcfg.py to setup the environment variables when you deploy your app to GAE (appcfg.py update)

当您将应用程序部署到 GAE 时，您可以使用 appcfg.py 的 -E 命令行选项来设置环境变量（appcfg.py 更新）

$ appcfg.py
...
-E NAME:VALUE, --env_variable=NAME:VALUE
                    Set an environment variable, potentially overriding an
                    env_variable value from app.yaml file (flag may be
                    repeated to set multiple variables).
...

回答by gbruins

Just wanted to note how I solved this problem in javascript/nodejs. For local development I used the 'dotenv' npm package which loads environment variables from a .env file into process.env. When I started using GAE I learned that environment variables need to be set in a 'app.yaml' file. Well, I didn't want to use 'dotenv' for local development and 'app.yaml' for GAE (and duplicate my environment variables between the two files), so I wrote a little script that loads app.yaml environment variables into process.env, for local development. Hope this helps someone:

只是想注意我是如何在 javascript/nodejs 中解决这个问题的。对于本地开发，我使用了“dotenv”npm 包，该包将 .env 文件中的环境变量加载到 process.env 中。当我开始使用 GAE 时，我了解到需要在“app.yaml”文件中设置环境变量。好吧，我不想对本地开发使用“dotenv”，对 GAE 使用“app.yaml”（并在两个文件之间复制我的环境变量），所以我写了一个小脚本，将 app.yaml 环境变量加载到进程中.env，用于本地开发。希望这有助于某人：

yaml_env.js:

yaml_env.js：

(function () {
    const yaml = require('js-yaml');
    const fs = require('fs');
    const isObject = require('lodash.isobject')

    var doc = yaml.safeLoad(
        fs.readFileSync('app.yaml', 'utf8'), 
        { json: true }
    );

    // The .env file will take precedence over the settings the app.yaml file
    // which allows me to override stuff in app.yaml (the database connection string (DATABASE_URL), for example)
    // This is optional of course. If you don't use dotenv then remove this line:
    require('dotenv/config');

    if(isObject(doc) && isObject(doc.env_variables)) {
        Object.keys(doc.env_variables).forEach(function (key) {
            // Dont set environment with the yaml file value if it's already set
            process.env[key] = process.env[key] || doc.env_variables[key]
        })
    }
})()

Now include this file as early as possible in your code, and you're done:

现在尽可能早地在代码中包含这个文件，你就完成了：

require('../yaml_env')

回答by JSBach

Extending Martin's answer

扩展马丁的答案

from google.appengine.ext import ndb

class Settings(ndb.Model):
    """
    Get sensitive data setting from DataStore.

    key:String -> value:String
    key:String -> Exception

    Thanks to: Martin Omander @ Stackoverflow
    https://stackoverflow.com/a/35261091/1463812
    """
    name = ndb.StringProperty()
    value = ndb.StringProperty()

    @staticmethod
    def get(name):
        retval = Settings.query(Settings.name == name).get()
        if not retval:
            raise Exception(('Setting %s not found in the database. A placeholder ' +
                             'record has been created. Go to the Developers Console for your app ' +
                             'in App Engine, look up the Settings record with name=%s and enter ' +
                             'its value in that record\'s value field.') % (name, name))
        return retval.value

    @staticmethod
    def set(name, value):
        exists = Settings.query(Settings.name == name).get()
        if not exists:
            s = Settings(name=name, value=value)
            s.put()
        else:
            exists.value = value
            exists.put()

        return True

回答by Prince Odame

There is a pypi package called gae_envthat allows you to save appengine environment variables in Cloud Datastore. Under the hood, it also uses Memcache so its fast

有一个名为gae_env的 pypi 包，它允许您在 Cloud Datastore 中保存 appengine 环境变量。在引擎盖下，它还使用 Memcache，因此速度很快

Usage:

用法：

import gae_env

API_KEY = gae_env.get('API_KEY')

If there is a value for that key in the datastore, it will be returned. If there isn't, a placeholder record __NOT_SET__will be created and a ValueNotSetErrorwill be thrown. The exception will remind you to go to the Developers Consoleand update the placeholder record.

如果数据存储中存在该键的值，则将返回该值。如果没有，__NOT_SET__将创建一个占位符记录并ValueNotSetError抛出 a 。异常会提醒您转到开发人员控制台并更新占位符记录。

---

---

Similar to Martin's answer, here is how to update the value for the key in Datastore:

与 Martin 的回答类似，这里是如何更新 Datastore 中键的值：

1. Go to Datastore Sectionin the developers console

2. Select your project at the top of the page if it's not already selected.

3. In the Kinddropdown box, select GaeEnvSettings.

4. Keys for which an exception was raised will have value __NOT_SET__.

1. 转到开发人员控制台中的数据存储部分

2. 如果尚未选择，请在页面顶部选择您的项目。

3. 在种类下拉框中，选择GaeEnvSettings。

4. 引发异常的键将具有 value __NOT_SET__。

---

---

Go to the package's GitHub pagefor more info on usage/configuration

有关使用/配置的更多信息，请访问包的 GitHub 页面

回答by Shih-Wen Su

This solution is simple but may not suit all different teams.

这个解决方案很简单，但可能并不适合所有不同的团队。

First, put the environment variables in an env_variables.yaml, e.g.,

首先，将环境变量放在env_variables.yaml 中，例如，

env_variables:
  SECRET: 'my_secret'

Then, include this env_variables.yamlin the app.yaml

然后，将其包含env_variables.yaml在app.yaml

includes:
  - env_variables.yaml

Finally, add the env_variables.yamlto .gitignore, so that the secret variables won't exist in the repository.

最后，添加env_variables.yamlto .gitignore，以便存储库中不存在秘密变量。

In this case, the env_variables.yamlneeds to be shared among the deployment managers.

在这种情况下，env_variables.yaml需要在部署经理之间共享。

回答by Jason F

Most answers are outdated. Using google cloud datastore is actually a bit different right now. https://cloud.google.com/python/getting-started/using-cloud-datastore

大多数答案已经过时。现在使用谷歌云数据存储实际上有点不同。https://cloud.google.com/python/getting-started/using-cloud-datastore

Here's an example:

下面是一个例子：

from google.cloud import datastore
client = datastore.Client()
datastore_entity = client.get(client.key('settings', 'TWITTER_APP_KEY'))
connection_string_prod = datastore_entity.get('value')

This assumes the entity name is 'TWITTER_APP_KEY', the kind is 'settings', and 'value' is a property of the TWITTER_APP_KEY entity.

这假设实体名称是“TWITTER_APP_KEY”，种类是“设置”，而“值”是 TWITTER_APP_KEY 实体的属性。