如何使用 Google App Engine 管理第三方 Python 库？（virtualenv？pip？）

How do I manage third-party Python libraries with Google App Engine? (virtualenv? pip?)

提问by a paid nerd

What's the best strategy for managing third-party Python libraries with Google App Engine?

使用 Google App Engine 管理第三方 Python 库的最佳策略是什么？

Say I want to use Flask, a webapp framework. A blog entrysays to do this, which doesn't seem right:

假设我想使用 Flask，一个 webapp 框架。一个博客条目说要这样做，这似乎不对：

$ cd /tmp/
$ wget http://pypi.python.org/packages/source/F/Flask/Flask-0.6.1.tar.gz
$ tar zxf Flask-0.6.1.tar.gz
$ cp -r Flask-0.6.1/flask ~/path/to/project/
(... repeat for other packages ...)

There must be a better way to manage third-party code, especially if I want to track versions, test upgrades or if two libraries share a subdirectory. I know that Python can import modules from zipfiles and that pipcan work with a wonderful REQUIREMENTS file, and I've seen that piphas a zipcommand for use with GAE.

必须有更好的方法来管理第三方代码，尤其是如果我想跟踪版本、测试升级或两个库共享一个子目录时。我知道 Python 可以从 zipfiles 导入模块，并且pip可以处理一个很棒的 REQUIREMENTS 文件，而且我已经看到pip有一个zip用于 GAE的命令。

(Note: There's a handful of similar questions — 1, 2, 3, 4, 5— but they're case-specific and don't really answer my question.)

（注意：有一些类似的问题—— 1、2、3、4、5—— 但它们是针对特定案例的，并不能真正回答我的问题。）

采纳答案by Wernight

What about simply:

简单地说：

$ pip install -r requirements.txt -t <your_app_directory/lib>

Create/edit <your_app_directory>/appengine_config.py:

创建/编辑<your_app_directory>/appengine_config.py：

"""This file is loaded when starting a new application instance."""
import sys
import os.path

# add `lib` subdirectory to `sys.path`, so our `main` module can load
# third-party libraries.
sys.path.insert(0, os.path.join(os.path.dirname(__file__), 'lib'))

UPDATE:

更新：

Google updated their sample to appengine_config.py, like:

Google 将他们的样本更新为appengine_config.py，例如：

    from google.appengine.ext import vendor
    vendor.add('lib')

Note: Even though their example has .gitignoreignoring lib/directory you still need to keep that directory under source control if you use git-pushdeployment method.

注意：即使他们的示例.gitignore忽略了lib/目录，如果您使用git-push部署方法，您仍然需要将该目录保持在源代码控制之下。

回答by Y.H Wong

Here's how I do it:

这是我的方法：

• project
  • .Python
  • bin
  • lib
    • python2.5
      • site-packages
        • < pip install packages here >
  • include
  • src
    • app.yaml
    • index.yaml
    • main.yaml
    • < symlink the pip installed packages in ../lib/python2.5/site-packages

• 项目
  • 。Python
  • 垃圾桶
  • 库
    • 蟒蛇2.5
      • 站点包
        • < pip 安装包在这里>
  • 包括
  • 源文件
    • 应用程序.yaml
    • 索引.yaml
    • 主文件
    • < 符号链接 ../lib/python2.5/site-packages 中的 pip 安装包

The projectdirectory is the top level directory where the virtualenv sits. I get the virtualenv using the following commands:

该project目录是 virtualenv 所在的顶级目录。我使用以下命令获取 virtualenv：

cd project
virtualenv -p /usr/bin/python2.5 --no-site-packages --distribute .

The srcdirectory is where all your code goes. When you deploy your code to GAE, *only*deploy those in the src directory and nothing else. The appcfg.pywill resolve the symlinks and copy the library files to GAE for you.

该src目录是您所有代码所在的位置。当您将代码部署到 GAE 时，*仅*将它们部署在 src 目录中，而没有其他任何内容。在appcfg.py将解决符号链接和复制库文件到GAE你。

I don't install my libraries as zip files mainly for convenience in case I need to read the source code, which I happen to do a lot just out of curiosity. However, if you really want to zip the libraries, put the following code snippet into your main.py

我不会将我的库安装为 zip 文件，主要是为了方便以防我需要阅读源代码，我只是出于好奇而做了很多。但是，如果您真的想压缩库，请将以下代码片段放入您的 main.py

import sys
for p in ['librarie.zip', 'package.egg'...]:
    sys.path.insert(0, p)

After this you can import your zipped up packages as usual.

在此之后，您可以像往常一样导入压缩包。

One thing to watch out for is setuptools' pkg_resources.py. I copied that directly into my srcdirectory so my other symlinked packages can use it. Watch out for anything that uses entry_points. In my case I'm using Toscawidgets2 and I had to dig into the source code to manually wire up the pieces. It can become annoying if you had a lot of libraries that rely on entry_point.

需要注意的一件事是 setuptools' pkg_resources.py。我将它直接复制到我的src目录中，以便我的其他符号链接包可以使用它。注意任何使用entry_points 的东西。就我而言，我使用的是 Toscawidgets2，我不得不深入研究源代码以手动连接各个部分。如果您有很多依赖于entry_point.

回答by kamalgill

Note: this answer is specific for Flask on Google App Engine.

注意：此答案特定于 Google App Engine 上的 Flask。

See the flask-appengine-template project for an example of how to get Flask extensions to work on App Engine. https://github.com/kamalgill/flask-appengine-template

有关如何让 Flask 扩展在 App Engine 上工作的示例，请参阅 flask-appengine-template 项目。 https://github.com/kamalgill/flask-appengine-template

Drop the extension into the namespace package folder at src/packages/flaskext and you're all set. https://github.com/kamalgill/flask-appengine-template/tree/master/src/lib/flaskext

将扩展放到 src/packages/flaskext 的命名空间包文件夹中，一切就绪。 https://github.com/kamalgill/flask-appengine-template/tree/master/src/lib/flaskext

Non-Flask packages can be dropped into the src/packages folder as zip files, eggs, or unzipped packages, as the project template includes the sys.path.insert() snippet posted above.

非 Flask 包可以作为 zip 文件、eggs 或解压缩包放入 src/packages 文件夹中，因为项目模板包含上面发布的 sys.path.insert() 片段。

回答by tesdal

I prefer buildout.

我更喜欢buildout。

You set up dependencies in setup.py in your project or buildout.cfg, pin the versions in buildout.cfg, and specify which packages are not available on GAE and should be included in packages.zip. rod.recipe.appengine will copy required packages into packages.zip, and as long as you insert packages.zip into the sys.path, they can be imported anywhere.

您在项目或 buildout.cfg 中的 setup.py 中设置依赖项，在 buildout.cfg 中固定版本，并指定哪些包在 GAE 上不可用而应包含在 packages.zip 中。rod.recipe.appengine 会将需要的包复制到packages.zip 中，只要在sys.path 中插入packages.zip 就可以导入到任何地方。

You can also use forks from github if the package you need is not on pypi

如果你需要的包不在 pypi 上，你也可以使用 github 的 fork

find-links =
    https://github.com/tesdal/pusher_client_python/tarball/rewrite#egg=pusher-2.0dev2

[versions]
pusher = 2.0dev2

and all of these settings and dependencies are versioned in git.

并且所有这些设置和依赖项都在 git 中进行了版本控制。

Instead of wondering which copy of Flask is currently included in your source tree and perhaps copied into your version control (or requiring new developers to manually unpack and upgrade), you simply check the version in buildout.cfg. If you want a new version, change buildout.cfg and rerun buildout.

您不必想知道当前源代码树中包含 Flask 的哪个副本，并且可能复制到您的版本控制中（或需要新开发人员手动解包和升级），您只需在 buildout.cfg 中检查版本即可。如果您想要新版本，请更改 buildout.cfg 并重新运行 buildout。

You can also use it to insert variables into config file templates, like setting the appspot id and version in app.yaml if you have staging server with staging.cfg and so on.

您还可以使用它来将变量插入到配置文件模板中，例如在 app.yaml 中设置 appspot id 和版本，如果您有带有 staging.cfg 的登台服务器等等。

回答by Faisal

I recently created a tool for this called gaenv. It follows a requirements.txt format, but doesn't install it, you can install with pip install -r requirements.txt then run the command line tool gaenv.

我最近为此创建了一个名为 gaenv 的工具。它遵循requirements.txt格式，但不安装它，您可以使用pip install -r requirements.txt安装然后运行命令行工具gaenv。

$ pip install -r requirements.txt
$ gaenv

This creates symlinks automatically, you could install gaenv in your virtualenv too and run the binary from there. Here is a blog post about it:

这会自动创建符号链接，您也可以在您的 virtualenv 中安装 gaenv 并从那里运行二进制文件。这是一篇关于它的博客文章：

http://blog.altlimit.com/2013/06/google-app-engine-virtualenv-tool-that.html

http://blog.altlimit.com/2013/06/google-app-engine-virtualenv-tool-that.html

also on github

也在github上

https://github.com/faisalraja/gaenv

https://github.com/faisalraja/gaenv

回答by webmaven

Wernight's solutionis the closest to current practice in the official Flask example app, which I've already improved by changing the sys.path.insert()call to site.addsitedir()in order to allow for namespace packagesby processing their attendant .pthfiles (which are important for frameworks like Pyramid).

Wernight 的解决方案与官方 Flask 示例应用程序中的当前实践最接近，我已经通过更改对它的sys.path.insert()调用进行了改进site.addsitedir()，以便通过处理它们的随附文件（这对于 Pyramid 等框架很重要）来允许命名空间包.pth。

So far so good, but that appendsthe directory to the path, and so loses the opportunity to override the included libraries (like WebOb and requests) with newer versions.

到目前为止一切顺利，但这会将目录附加到路径中，因此失去了用较新版本覆盖包含的库（如 WebOb 和请求）的机会。

What is needed then in appengine_config.py(and I am trying to get this change accepted into the official repos as well) is the following:

然后在appengine_config.py（并且我试图让这个更改也被官方存储库接受）中需要的是以下内容：

"""This file is loaded when starting a new application instance."""
import os.path
import site.addsitedir
import sys.path

dirname = 'lib'
dirpath = os.path.join(os.path.dirname(__file__), dirname)

# split path after 1st element ('.') so local modules are always found first
sys.path, remainder = sys.path[:1], sys.path[1:]

# add `lib` subdirectory as a site directory, so our `main` module can load
# third-party libraries.
site.addsitedir(dirpath)

# append the rest of the path
sys.path.extend(remainder)

The final version of this code may end up hidden away in a vendor.pymodule and called like insertsitedir(index, path)or some other variation, as you can see in the discussion attending this pull request, but the logic is more or less how it will work regardless, to allow a simple pip install -r requirements.txt -t lib/to work for all packages including namespace ones, and to still allow overriding the included libraries with new versions, as I have so far been unable to find a simpler alternative.

这段代码的最终版本可能最终隐藏在一个vendor.py模块中并被称为 likeinsertsitedir(index, path)或其他一些变体，正如您在参与此 pull request 的讨论中看到的那样，但逻辑或多或少是无论如何它将如何工作，以允许简单pip install -r requirements.txt -t lib/适用于所有包，包括命名空间包，并且仍然允许使用新版本覆盖包含的库，因为我迄今为止无法找到更简单的替代.