python 加速python“导入”加载器

Speeding up the python "import" loader

提问by YGA

I'm getting seriously frustrated at how slow python startup is. Just importing more or less basic modules takes a second, since python runs down the sys.path looking for matching files (and generating 4 stat()calls - ["foo", "foo.py", "foo.pyc", "foo.so"] - for each check). For a complicated project environment, with tons of different directories, this can take around 5 seconds -- all to run a script that might fail instantly.

我对 python 启动的缓慢感到非常沮丧。仅仅导入或多或少的基本模块需要一秒钟，因为 python 沿着 sys.path 运行寻找匹配的文件（并生成 4 个stat()调用 - ["foo", "foo.py", "foo.pyc", "foo.so"） "] - 每次检查）。对于具有大量不同目录的复杂项目环境，这可能需要大约 5 秒的时间——所有这些都是为了运行一个可能会立即失败的脚本。

Do folks have suggestions for how to speed up this process? For instance, one hack I've seen is to set the LD_PRELOAD_32environment variable to a library that caches the result of ENOENTcalls (e.g. failed stat()calls) between runs. Of course, this has all sorts of problems (potentially confusing non-python programs, negative caching, etc.).

人们对如何加快这个过程有什么建议吗？例如，我见过的一个 hack 是将LD_PRELOAD_32环境变量设置为一个库，该库在运行之间缓存ENOENT调用（例如失败的stat()调用）的结果。当然，这有各种各样的问题（可能会混淆非 Python 程序、负缓存等）。

回答by Alex Martelli

zipping up as many pycfiles as feasible (with proper directory structure for packages), and putting that zipfile as the very first entry in sys.path (on the best available local disk, ideally) can speed up startup times a lot.

尽可能多地压缩pyc文件（具有适当的包目录结构），并将该 zip 文件作为 sys.path 中的第一个条目（最好在可用的本地磁盘上，理想情况下）可以大大加快启动时间。

回答by Seth

The first things that come to mind are:

首先想到的是：

• Try a smaller path
• Make sure your modules are pyc's so they'll load faster
• Make sure you don't double import, or import too much

• 尝试更小的路径
• 确保您的模块是 pyc 的，以便它们加载得更快
• 确保不要重复导入，或导入太多

Other than that, are you sure that the disk operations are what's bogging you down? Is your disk/operating system really busy or old and slow?

除此之外，您确定是磁盘操作使您陷入困境吗？您的磁盘/操作系统真的很忙还是又旧又慢？

Maybe a defrag is in order?

也许碎片整理是为了？

回答by BrainCore

If you run out of options, you can create a ramdiskto store your python packages. A ramdisk appears as a directory in your file system, but will actually be mapped directly to your computer's RAM. Here are some instructionsfor Linux/Redhat.

如果你的选项用完了，你可以创建一个ramdisk来存储你的 python 包。ramdisk 在您的文件系统中显示为一个目录，但实际上会直接映射到您计算机的 RAM。下面是一些指令为Linux /红帽。

Beware: A ramdisk is volatile, so you'll also need to keep a backup of your files on your regular hard drive, otherwise you'll lose your data when your computer shuts down.

请注意：虚拟磁盘是易失性的，因此您还需要在常规硬盘驱动器上保留文件备份，否则在计算机关闭时您将丢失数据。

回答by Chris B.

Something's missing from your premise--I've never seen some "more-or-less" basic modules take over a second to import, and I'm not running Python on what I would call cutting-edge hardware. Either you're running on some seriously old hardware, or you're running on an overloaded machine, or either your OS or Python installation is broken in some way. Or you're not really importing "basic" modules.

你的前提中缺少一些东西——我从来没有见过一些“或多或少”的基本模块需要花一秒钟的时间来导入，而且我没有在我称之为尖端硬件的东西上运行 Python。要么您在一些非常旧的硬件上运行，要么在超载的机器上运行，要么您的操作系统或 Python 安装以某种方式损坏。或者你并没有真正导入“基本”模块。

If it's any of the first three issues, you need to look at the root problem for a solution. If it's the last, we really need to know what the specific packages are to be of any help.

如果是前三个问题中的任何一个，则需要查看根本问题以找到解决方案。如果是最后一个，我们真的需要知道具体的包有什么帮助。

回答by Nico Schl?mer

When trying to speed things up, profiling is key. Otherwise, how will you know which parts of your code are really the slow ones?

在尝试加快速度时，分析是关键。否则，你怎么知道你的代码的哪些部分真的很慢？

A while ago, I've created the runtime and import profile visualizer tuna, and I think it may be useful here. Simply create an import profile (with Python 3.7+) and run tuna on it:

不久前，我创建了运行时和导入配置文件可视化工具tuna，我认为它在这里可能很有用。只需创建一个导入配置文件（使用 Python 3.7+）并在其上运行 tuna：

python3.7 -X importtime -c "import scipy" 2> scipy.log
tuna scipy.log