Apache 使用过多的 CPU

Apache uses excessive CPU

提问by Vegard Larsen

We run a medium-size site that gets a few hundred thousand pageviews a day. Up until last weekend we ran with a load usually below 0.2 on a virtual machine. The OS is Ubuntu.

我们运营着一个中等规模的网站，每天的浏览量只有几十万。直到上周末，我们在虚拟机上的负载通常低于 0.2。操作系统是 Ubuntu。

When deploying the latest version of our application, we also did an apt-get dist-upgrade before deploying. After we had deployed we noticed that the load on the CPU had spiked dramatically (sometimes reaching 10 and stopping to respond to page requests).

在部署最新版本的应用程序时，我们还在部署之前执行了 apt-get dist-upgrade。部署后，我们注意到 CPU 上的负载急剧增加（有时达到 10 并停止响应页面请求）。

We tried dumping a full minute of Xdebug profiling data from PHP, but looking through it revealed only a few somewhat slow parts, but nothing to explain the huge jump.

我们尝试从 PHP 转储整整一分钟的 Xdebug 分析数据，但查看它只发现了一些有点慢的部分，但无法解释巨大的跳跃。

We are now pretty sure that nothing in the new version of our website is triggering the problem, but we have no way to be sure. We have rolled back a lot of the changes, but the problem still persists.

我们现在非常确定我们网站的新版本中没有任何内容会触发该问题，但我们无法确定。我们已经回滚了很多更改，但问题仍然存在。

When look at processes, we see that single Apache processes use quite a bit of CPU over a longer period of time than strictly necessary. However, when using strace on the affected process, we never see anything but

在查看进程时，我们看到单个 Apache 进程在比严格必要的更长的时间内使用了相当多的 CPU。但是，当在受影响的进程上使用 strace 时，我们什么也看不到

accept(3,

and it hangs for a while before receiving a new connection, so we can't actually see what is causing the problem.

并且在接收新连接之前它会挂起一段时间，因此我们实际上无法看到导致问题的原因。

The stack is PHP 5, Apache 2 (prefork), MySQL 5.1. Most things run through Memcached. We've tried APC and eAccelerator.

堆栈是 PHP 5、Apache 2（prefork）、MySQL 5.1。大多数事情都通过 Memcached 运行。我们已经尝试了 APC 和 eAccelerator。

So, what should be our next step? Are there any profiling methods we overlooked/don't know about?

那么，我们下一步应该做什么？是否有任何我们忽略/不知道的分析方法？

回答by Vegard Larsen

The answer ended up being not-Apache related. As mentioned, we were on a virtual machine. Our user sessions are pretty big (think 500kB per active user), so we had a lot of disk IO. The disk was nearly full, meaning that Ubuntu spent a lot of time moving things around (or so we think). There was no easy way to extend the disk (because it was not set up properly for VMWare). This completely killed performance, and Apache and MySQL would occasionally use 100% CPU (for a very short time), and the system would be so slow to update the CPU usage meters that it seemed to be stuck there.

答案最终与 Apache 无关。如前所述，我们在虚拟机上。我们的用户会话非常大（想想每个活跃用户 500kB），所以我们有很多磁盘 IO。磁盘快满了，这意味着 Ubuntu 花了很多时间来移动东西（或者我们认为是这样）。没有简单的方法来扩展磁盘（因为它没有为 VMWare 正确设置）。这完全扼杀了性能，Apache 和 MySQL 偶尔会使用 100% 的 CPU（在很短的时间内），并且系统更新 CPU 使用率表会很慢，以至于它似乎卡在那里。

We ended up setting up a new VM (which also gave us the opportunity to thoroughly document everything on the server). On the new VM we allocated plenty of disk space, and moved sessions into memory (using memcached). Our load dropped to 0.2 on off-peak use and around 1 near peak use (on a 2-CPU VM). Moving the sessions into memcached took a lot of disk IO away (we were constantly using about 2MB/s of disk IO, which is very bad).

我们最终设置了一个新的 VM（这也让我们有机会彻底记录服务器上的所有内容）。在新 VM 上，我们分配了大量磁盘空间，并将会话移到内存中（使用 memcached）。我们的负载在非高峰使用时降至 0.2，在接近高峰使用时降至 1（在 2-CPU 虚拟机上）。将会话移至 memcached 会占用大量磁盘 IO（我们一直使用大约 2MB/s 的磁盘 IO，这非常糟糕）。

Conclusion; sometimes you just have to start over... :)

结论; 有时你只需要重新开始...... :)

回答by Jon Topper

Seeing an accept() call from your Apache process isn't at all unusual - that's the webserver waiting for a new request.

从 Apache 进程中看到 accept() 调用并不罕见 - 那是网络服务器在等待新请求。

First of all, you want to establish what the parameters of the load are. Something like

首先，您要确定负载的参数是什么。就像是

vmstat 1

will show you what your system is up to. Look in the 'swap' and 'io' columns. If you see anything other than '0' in the 'si' and 'so' columns, your system is swapping because of a low memory condition. Consider reducing the number of running Apache children, or throwing more RAM in your server.

会告诉你你的系统在做什么。查看“交换”和“io”列。如果您在 'si' 和 'so' 列中看到除 '0' 以外的任何内容，则您的系统正在交换，因为内存不足。考虑减少正在运行的 Apache 子进程的数量，或者在您的服务器中投入更多 RAM。

If RAM isn't an issue, look at the 'cpu' columns. You're interested in the 'us' and 'sy' columns. These show you the percentage of CPU time spent in either user processes or system. A high 'us' number points the finger at Apache or your scripts - or potentially something else on the server.

如果 RAM 不是问题，请查看“cpu”列。您对“us”和“sy”列感兴趣。这些显示您在用户进程或系统中花费的 CPU 时间百分比。较高的“我们”数字将矛头指向 Apache 或您的脚本 - 或者可能是服务器上的其他内容。

Running

跑步

top

will show you which processes are the most active.

将显示哪些进程最活跃。

Have you ruled out your database? The most common cause of unexpectedly high load I've seen on production LAMP stacks come down to database queries. You may have deployed new code with an expensive query in it; or got to the point where there are enough rows in your dataset to cause previously cheap queries to become expensive.

你有没有排除你的数据库？我在生产 LAMP 堆栈上看到的意外高负载的最常见原因归结为数据库查询。您可能已经部署了包含昂贵查询的新代码；或者到了数据集中有足够多的行导致以前廉价的查询变得昂贵的地步。

During periods of high load, do

在高负载期间，请执行

echo "show full processlist" | mysql | grep -v Sleep

to see if there are either long-running queries, or huge numbers of the same query operating at once. Other mysql tools will help you optimise these.

查看是否有长时间运行的查询，或者是否有大量相同的查询同时运行。其他 mysql 工具将帮助您优化这些。

You may find it useful to configure and use mod_status for Apache, which will allow you to see what request each Apache child is serving and for how long it has been doing so.

您可能会发现为 Apache 配置和使用 mod_status 很有用，这将允许您查看每个 Apache 子进程正在服务的请求以及它已经这样做了多长时间。

Finally, get some long-term statistical monitoring set up. Something like zabbix is straightforward to configure, and will let you monitor resource usage over time, such that if things get slow, you have historical baselines to compare against, and a better ieda of when problems started.

最后，设置一些长期的统计监控。像 zabbix 这样的东西很容易配置，并且可以让你随着时间的推移监控资源使用情况，这样如果事情变慢，你就有历史基线来比较，以及更好的问题开始时间。

回答by Paul Whelan

Perhaps you where using worker MPM before and now you are not?

也许你以前使用过工人 MPM 而现在你不是？

I know PHP5 does not work with the Worker MPM. On my Ubuntu server, PHP5 can only be installed with the Prefork MPM. It seems that PHP5 module is not compatible with multithreading version of Apache.

我知道 PHP5 不适用于 Worker MPM。在我的 Ubuntu 服务器上，PHP5 只能与 Prefork MPM 一起安装。PHP5 模块似乎与 Apache 的多线程版本不兼容。

I found a link here that will show you how to get better performance with mod_fcgid

我在这里找到了一个链接，它将向您展示如何使用mod_fcgid获得更好的性能

To see what worker MPM is see here.

要查看工作 MPM 是什么，请参见此处。

回答by Robert Gould

I'd use dTrace to solve this mystery... if it was running on Solaris or Mac... but since Linux doesn't have it you might want to try their Systemtap, however I can't say anything about its usability since I haven't used it.

我会使用 dTrace 来解决这个谜团……如果它在 Solaris 或 Mac 上运行……但由于 Linux 没有它，你可能想尝试他们的Systemtap，但是我不能说它的可用性，因为我没用过。

With dTrace you could easily sniff out the culprits within a day, and would hope with Systemtap it would be similiar

使用 dTrace，您可以在一天内轻松找出罪魁祸首，并希望 Systemtap 也能做到这一点

回答by Robert Gould

Another option that I can't assure you will do any good, but it's more than worth the effort. Is to read the detailed changelog for the new version, and review what might have changed that could remotely affect you.

我不能保证你会有任何好处的另一个选择，但它值得付出努力。是阅读新版本的详细更改日志，并查看可能会远程影响您的更改。

Going through the changelogs has saved me more than once. Especially when some config options have changed and when something got deprecated. Worst case is it'll give you some clues as to where to look next

浏览变更日志不止一次救了我。特别是当某些配置选项发生变化并且某些内容被弃用时。最坏的情况是它会给你一些关于下一步去哪里的线索