我应该如何构建包含 Cython 代码的 Python 包

Question

提问by Craig McQueen

I'd like to make a Python package containing some Cythoncode. I've got the the Cython code working nicely. However, now I want to know how best to package it.

我想制作一个包含一些Cython代码的 Python 包。我的 Cython 代码运行良好。但是，现在我想知道如何最好地打包它。

For most people who just want to install the package, I'd like to include the .cfile that Cython creates, and arrange for setup.pyto compile that to produce the module. Then the user doesn't need Cython installed in order to install the package.

对于大多数只想安装包的人，我想包含.cCython 创建的文件，并安排setup.py编译它以生成模块。然后用户不需要安装 Cython 来安装包。

But for people who may want to modify the package, I'd also like to provide the Cython .pyxfiles, and somehow also allow for setup.pyto build them using Cython (so those users wouldneed Cython installed).

但对于谁可能要修改包的人，我也想提供用Cython.pyx文件，并以某种方式也允许setup.py使用用Cython（所以这些用户建立他们将需要安装用Cython）。

How should I structure the files in the package to cater for both these scenarios?

我应该如何构建包中的文件以满足这两种情况？

The Cython documentation gives a little guidance. But it doesn't say how to make a single setup.pythat handles both the with/without Cython cases.

在用Cython文档提供了一些指导。但它没有说明如何制作一个setup.py可以同时处理有/没有 Cython 情况的单曲。

Answer 1

采纳答案by Craig McQueen

I've done this myself now, in a Python package simplerandom(BitBucket repo- EDIT: now github) (I don't expect this to be a popular package, but it was a good chance to learn Cython).

我现在已经在 Python 包simplerandom（BitBucket repo- 编辑：现在github）中自己完成了这个（我不希望这是一个流行的包，但这是学习 Cython 的好机会）。

This method relies on the fact that building a .pyxfile with Cython.Distutils.build_ext(at least with Cython version 0.14) always seems to create a .cfile in the same directory as the source .pyxfile.

这种方法依赖于这样一个事实，即.pyx使用Cython.Distutils.build_ext（至少使用 Cython 0.14 版）构建文件似乎总是.c在与源.pyx文件相同的目录中创建文件。

Here is a cut-down version of setup.pywhich I hope shows the essentials:

这是一个精简版setup.py，我希望它显示出要点：

from distutils.core import setup
from distutils.extension import Extension

try:
    from Cython.Distutils import build_ext
except ImportError:
    use_cython = False
else:
    use_cython = True

cmdclass = {}
ext_modules = []

if use_cython:
    ext_modules += [
        Extension("mypackage.mycythonmodule", ["cython/mycythonmodule.pyx"]),
    ]
    cmdclass.update({'build_ext': build_ext})
else:
    ext_modules += [
        Extension("mypackage.mycythonmodule", ["cython/mycythonmodule.c"]),
    ]

setup(
    name='mypackage',
    ...
    cmdclass=cmdclass,
    ext_modules=ext_modules,
    ...
)

I also edited MANIFEST.into ensure that mycythonmodule.cis included in a source distribution (a source distribution that is created with python setup.py sdist):

我还进行了编辑MANIFEST.in以确保mycythonmodule.c包含在源分发中（使用创建的源分发python setup.py sdist）：

...
recursive-include cython *
...

I don't commit mycythonmodule.cto version control 'trunk' (or 'default' for Mercurial). When I make a release, I need to remember to do a python setup.py build_extfirst, to ensure that mycythonmodule.cis present and up-to-date for the source code distribution. I also make a release branch, and commit the C file into the branch. That way I have a historical record of the C file that was distributed with that release.

我不承诺mycythonmodule.c版本控制“主干”（或 Mercurial 的“默认”）。当我发布一个版本时，我需要记住先做一个python setup.py build_ext，以确保mycythonmodule.c源代码分发是存在的和最新的。我还创建了一个发布分支，并将 C 文件提交到该分支中。这样我就有了随该版本分发的 C 文件的历史记录。

Answer 2

回答by Lennart Regebro

The easiest is to include both but just use the c-file? Including the .pyx file is nice, but it's not needed once you have the .c file anyway. People who want to recompile the .pyx can install Pyrex and do it manually.

最简单的方法是包含两者但只使用 c 文件？包含 .pyx 文件很好，但是一旦有了 .c 文件就不需要了。想要重新编译 .pyx 的人可以安装 Pyrex 并手动完成。

Otherwise you need to have a custom build_ext command for distutils that builds the C file first. Cython already includes one. http://docs.cython.org/src/userguide/source_files_and_compilation.html

否则，您需要为首先构建 C 文件的 distutils 定制 build_ext 命令。Cython 已经包含了一个。http://docs.cython.org/src/userguide/source_files_and_compilation.html

What that documentation doesn't do is say how to make this conditional, but

该文档没有做的是说明如何使此条件成为条件，但是

try:
     from Cython.distutils import build_ext
except ImportError:
     from distutils.command import build_ext

Should handle it.

应该处理。

Answer 3

回答by kynan

Adding to Craig McQueen's answer: see below for how to override the sdistcommand to have Cython automatically compile your source files before creating a source distribution.

添加到 Craig McQueen 的回答中：请参阅下文，了解如何sdist在创建源分发之前覆盖命令以让 Cython 自动编译您的源文件。

That way your run no risk of accidentally distributing outdated Csources. It also helps in the case where you have limited control over the distribution process e.g. when automatically creating distributions from continuous integration etc.

这样您就没有意外分发过时C资源的风险。在您对分发过程的控制有限的情况下，例如从持续集成自动创建分发等时，它也有帮助。

from distutils.command.sdist import sdist as _sdist

...

class sdist(_sdist):
    def run(self):
        # Make sure the compiled Cython files in the distribution are up-to-date
        from Cython.Build import cythonize
        cythonize(['cython/mycythonmodule.pyx'])
        _sdist.run(self)
cmdclass['sdist'] = sdist

Answer 4

回答by Colonel Panic

http://docs.cython.org/en/latest/src/userguide/source_files_and_compilation.html#distributing-cython-modules

It is strongly recommended that you distribute the generated .c files as well as your Cython sources, so that users can install your module without needing to have Cython available.
It is also recommended that Cython compilation not be enabled by default in the version you distribute. Even if the user has Cython installed, he probably doesn't want to use it just to install your module. Also, the version he has may not be the same one you used, and may not compile your sources correctly.
This simply means that the setup.py file that you ship with will just be a normal distutils file on the generated .c files, for the basic example we would have instead:
from distutils.core import setup
from distutils.extension import Extension

setup(
    ext_modules = [Extension("example", ["example.c"])]
)

强烈建议您分发生成的 .c 文件以及 Cython 源，以便用户可以安装您的模块，而无需使用 Cython。
还建议在您分发的版本中默认不要启用 Cython 编译。即使用户安装了 Cython，他也可能不想仅仅为了安装你的模块而使用它。此外，他拥有的版本可能与您使用的版本不同，并且可能无法正确编译您的源代码。
这只是意味着您随附的 setup.py 文件将只是生成的 .c 文件中的普通 distutils 文件，对于基本示例，我们将改为：
from distutils.core import setup
from distutils.extension import Extension

setup(
    ext_modules = [Extension("example", ["example.c"])]
)

Answer 5

回答by zzart

This is a setup script I wrote which makes it easier to include nested directories inside the build. One needs to run it from folder within a package.

这是我编写的一个设置脚本，它可以更轻松地在构建中包含嵌套目录。需要从包中的文件夹运行它。

Givig structure like this:

Givig 结构如下：

__init__.py
setup.py
test.py
subdir/
      __init__.py
      anothertest.py

setup.py

设置文件

from setuptools import setup, Extension
from Cython.Distutils import build_ext
# from os import path
ext_names = (
    'test',
    'subdir.anothertest',       
) 

cmdclass = {'build_ext': build_ext}
# for modules in main dir      
ext_modules = [
    Extension(
        ext,
        [ext + ".py"],            
    ) 
    for ext in ext_names if ext.find('.') < 0] 
# for modules in subdir ONLY ONE LEVEL DOWN!! 
# modify it if you need more !!!
ext_modules += [
    Extension(
        ext,
        ["/".join(ext.split('.')) + ".py"],     
    )
    for ext in ext_names if ext.find('.') > 0]

setup(
    name='name',
    ext_modules=ext_modules,
    cmdclass=cmdclass,
    packages=["base", "base.subdir"],
)
#  Build --------------------------
#  python setup.py build_ext --inplace

Happy compiling ;)

快乐编译;)

Answer 6

回答by McKelvin

Including (Cython) generated .c files are pretty weird. Especially when we include that in git. I'd prefer to use setuptools_cython. When Cython is not available, it will build an egg which has built-in Cython environment, and then build your code using the egg.

包括（Cython）生成的 .c 文件非常奇怪。尤其是当我们将其包含在 git 中时。我更喜欢使用setuptools_cython。当 Cython 不可用时，它会构建一个内置 Cython 环境的 Egg，然后使用该 Egg 构建您的代码。

A possible example: https://github.com/douban/greenify/blob/master/setup.py

一个可能的例子：https: //github.com/douban/greenify/blob/master/setup.py

Update(2017-01-05):

更新（2017-01-05）：

Since setuptools 18.0, there's no need to use setuptools_cython. Hereis an example to build Cython project from scratch without setuptools_cython.

因为setuptools 18.0，没有必要使用setuptools_cython. 这是一个从头开始构建 Cython 项目的示例，无需setuptools_cython.

Answer 7

回答by kay - SE is evil

The simple hack I came up with:

我想出的简单黑客：

from distutils.core import setup

try:
    from Cython.Build import cythonize
except ImportError:
    from pip import pip

    pip.main(['install', 'cython'])

    from Cython.Build import cythonize


setup(…)

Just install Cython if it could not be imported. One should probably not share this code, but for my own dependencies it's good enough.

如果无法导入，只需安装 Cython。人们可能不应该共享此代码，但对于我自己的依赖项，它已经足够了。

Answer 8

回答by MosteM

The easiest way I found using only setuptools instead of the feature limited distutils is

我发现仅使用 setuptools 而不是功能受限的 distutils 的最简单方法是

from setuptools import setup
from setuptools.extension import Extension
try:
    from Cython.Build import cythonize
except ImportError:
    use_cython = False
else:
    use_cython = True

ext_modules = []
if use_cython:
    ext_modules += cythonize('package/cython_module.pyx')
else:
    ext_modules += [Extension('package.cython_module',
                              ['package/cython_modules.c'])]

setup(name='package_name', ext_modules=ext_modules)

Answer 9

回答by bluenote10

All other answers either rely on

所有其他答案要么依赖于

distutils
importing from Cython.Build, which creates a chicken-and-egg problem between requiring cython via setup_requiresand importing it.

分配器
导入 from Cython.Build，这会在需要 cython viasetup_requires和导入它之间产生先有鸡还是先有蛋的问题。

A modern solution is to use setuptools instead, see this answer(automatic handling of Cython extensions requires setuptools 18.0, i.e., it's available for many years already). A modern standard setup.pywith requirements handling, an entry point, and a cython module could look like this:

现代解决方案是改用 setuptools，请参阅此答案（自动处理 Cython 扩展需要 setuptools 18.0，即它已经可用多年了）。setup.py具有需求处理、入口点和 cython 模块的现代标准可能如下所示：

from setuptools import setup, Extension

with open('requirements.txt') as f:
    requirements = f.read().splitlines()

setup(
    name='MyPackage',
    install_requires=requirements,
    setup_requires=[
        'setuptools>=18.0',  # automatically handles Cython extensions
        'cython>=0.28.4',
    ],
    entry_points={
        'console_scripts': [
            'mymain = mypackage.main:main',
        ],
    },
    ext_modules=[
        Extension(
            'mypackage.my_cython_module',
            sources=['mypackage/my_cython_module.pyx'],
        ),
    ],
)

Answer 10

回答by summentier

I think I found a pretty good way of doing this by providing a custom build_extcommand. The idea is the following:

我想我通过提供自定义build_ext命令找到了一个很好的方法。想法如下：

I add the numpy headers by overriding finalize_options()and doing import numpyin the body of the function, which nicely avoids the problem of numpy not being available before setup()installs it.
If cython is available on the system, it hooks into the command's check_extensions_list()method and by cythonizes all out-of-date cython modules, replacing them with C extensions that can later handled by the build_extension()method. We just provide the latter part of the functionality in our module too: this means that if cython is not available but we have a C extension present, it still works, which allows you to do source distributions.

我通过在函数体中覆盖finalize_options()和执行import numpy来添加 numpy 头文件，这很好地避免了 numpy 在setup()安装之前不可用的问题。
如果 cython 在系统上可用，它会挂接到命令的check_extensions_list()方法中，并通过 cythonizes 所有过时的 cython 模块，将它们替换为以后可由该build_extension()方法处理的 C 扩展。我们也只是在我们的模块中提供了后一部分功能：这意味着如果 cython 不可用但我们有一个 C 扩展，它仍然可以工作，这允许您进行源代码分发。

Here's the code:

这是代码：

import re, sys, os.path
from distutils import dep_util, log
from setuptools.command.build_ext import build_ext

try:
    import Cython.Build
    HAVE_CYTHON = True
except ImportError:
    HAVE_CYTHON = False

class BuildExtWithNumpy(build_ext):
    def check_cython(self, ext):
        c_sources = []
        for fname in ext.sources:
            cname, matches = re.subn(r"(?i)\.pyx$", ".c", fname, 1)
            c_sources.append(cname)
            if matches and dep_util.newer(fname, cname):
                if HAVE_CYTHON:
                    return ext
                raise RuntimeError("Cython and C module unavailable")
        ext.sources = c_sources
        return ext

    def check_extensions_list(self, extensions):
        extensions = [self.check_cython(ext) for ext in extensions]
        return build_ext.check_extensions_list(self, extensions)

    def finalize_options(self):
        import numpy as np
        build_ext.finalize_options(self)
        self.include_dirs.append(np.get_include())

This allows one to just write the setup()arguments without worrying about imports and whether one has cython available:

这允许人们只编写setup()参数而不必担心导入以及是否有可用的 cython：

setup(
    # ...
    ext_modules=[Extension("_my_fast_thing", ["src/_my_fast_thing.pyx"])],
    setup_requires=['numpy'],
    cmdclass={'build_ext': BuildExtWithNumpy}
    )

我应该如何构建包含 Cython 代码的 Python 包

提问by Craig McQueen

采纳答案by Craig McQueen

回答by Lennart Regebro

回答by kynan

回答by Colonel Panic

回答by zzart

回答by McKelvin

回答by kay - SE is evil

回答by MosteM

回答by bluenote10

回答by summentier

相关推荐

最近更新

标签

我应该如何构建包含 Cython 代码的 Python 包

提问by Craig McQueen

采纳答案by Craig McQueen

回答by Lennart Regebro

回答by kynan

回答by Colonel Panic

回答by zzart

回答by McKelvin

回答by kay - SE is evil

回答by MosteM

回答by bluenote10

回答by summentier

相关推荐

Python Argparse 可选的位置参数？

Python 格式字符列表

如何跳入 Python 中的行？

Python 将字符列表转换为字符串

相关推荐

最近更新

标签