增加 Linux 中 TCP/IP 连接的最大数量

Increasing the maximum number of TCP/IP connections in Linux

提问by

I am programming a server and it seems like my number of connections is being limited since my bandwidth isn't being saturated even when I've set the number of connections to "unlimited".

我正在对服务器进行编程，似乎我的连接数量受到限制，因为即使我将连接数量设置为“无限”，我的带宽也没有饱和。

How can I increase or eliminate a maximum number of connections that my Ubuntu Linux box can open at a time? Does the OS limit this, or is it the router or the ISP? Or is it something else?

如何增加或消除我的 Ubuntu Linux 机器一次可以打开的最大连接数？操作系统是否限制了这一点，还是路由器或 ISP？或者是别的什么？

回答by derobert

There are a couple of variables to set the max number of connections. Most likely, you're running out of file numbers first. Check ulimit -n. After that, there are settings in /proc, but those default to the tens of thousands.

有几个变量可以设置最大连接数。最有可能的是，您首先用完了文件号。检查 ulimit -n。之后，在/proc中有设置，但默认为数万。

More importantly, it sounds like you're doing something wrong. A single TCP connection ought to be able to use all of the bandwidth between two parties; if it isn't:

更重要的是，听起来你做错了什么。单个 TCP 连接应该能够使用两方之间的所有带宽；如果不是：

• Check if your TCP window setting is large enough. Linux defaults are good for everything except really fast inet link (hundreds of mbps) or fast satellite links. What is your bandwidth*delay product?
• Check for packet loss using ping with large packets (ping -s 1472...)
• Check for rate limiting. On Linux, this is configured with tc
• Confirm that the bandwidth you think exists actually exists using e.g., iperf
• Confirm that your protocol is sane. Remember latency.
• If this is a gigabit+ LAN, can you use jumbo packets? Are you?

• 检查您的 TCP 窗口设置是否足够大。Linux 默认值适用于除真正快速的 inet 链接（数百 mbps）或快速卫星链接之外的所有内容。你的带宽*延迟乘积是多少？
• 使用 ping 大数据包检查数据包丢失 ( ping -s 1472...)
• 检查速率限制。在 Linux 上，这是配置为tc
• 使用例如确认您认为存在的带宽实际存在， iperf
• 确认您的协议是健全的。记住延迟。
• 如果这是千兆+ LAN，您可以使用巨型数据包吗？你是？

Possibly I have misunderstood. Maybe you're doing something like Bittorrent, where you need lots of connections. If so, you need to figure out how many connections you're actually using (try netstator lsof). If that number is substantial, you might:

可能我理解错了。也许你正在做类似 Bittorrent 的事情，你需要大量的连接。如果是这样，您需要弄清楚您实际使用了多少个连接（尝试netstat或lsof）。如果这个数字很大，你可能会：

• Have a lot of bandwidth, e.g., 100mbps+. In this case, you may actually need to up the ulimit -n. Still, ~1000 connections (default on my system) is quite a few.
• Have network problems which are slowing down your connections (e.g., packet loss)
• Have something else slowing you down, e.g., IO bandwidth, especially if you're seeking. Have you checked iostat -x?

• 有很多带宽，例如，100mbps+。在这种情况下，您实际上可能需要启动ulimit -n. 尽管如此，大约 1000 个连接（我系统上的默认设置）还是不少的。
• 有网络问题会减慢您的连接速度（例如，数据包丢失）
• 有其他东西会减慢您的速度，例如 IO 带宽，尤其是在您正在寻找时。你检查了iostat -x吗？

Also, if you are using a consumer-grade NAT router (Linksys, Netgear, DLink, etc.), beware that you may exceed its abilities with thousands of connections.

此外，如果您使用的是消费级 NAT 路由器（Linksys、Netgear、DLink 等），请注意数千个连接可能会超出其能力。

I hope this provides some help. You're really asking a networking question.

我希望这能提供一些帮助。你真的在问一个网络问题。

回答by mdk

Maximum number of connections are impacted by certain limits on both client & server sides, albeit a little differently.

最大连接数受客户端和服务器端的某些限制影响，尽管略有不同。

On the client side:Increase the ephermal port range, and decrease the tcp_fin_timeout

在客户端：增加临时端口范围，并减少tcp_fin_timeout

To find out the default values:

要找出默认值：

sysctl net.ipv4.ip_local_port_range
sysctl net.ipv4.tcp_fin_timeout

The ephermal port range defines the maximum number of outbound sockets a host can create from a particular I.P. address. The fin_timeoutdefines the minimum time these sockets will stay in TIME_WAITstate (unusable after being used once). Usual system defaults are:

临时端口范围定义了主机可以从特定 IP 地址创建的最大出站套接字数。所述fin_timeout定义的最小时间这些插座会留在TIME_WAIT状态（在使用一次后无法使用）。通常的系统默认值是：

• net.ipv4.ip_local_port_range = 32768 61000
• net.ipv4.tcp_fin_timeout = 60

• net.ipv4.ip_local_port_range = 32768 61000
• net.ipv4.tcp_fin_timeout = 60

This basically means your system cannot consistently guarantee more than (61000 - 32768) / 60 = 470sockets per second. If you are not happy with that, you could begin with increasing the port_range. Setting the range to 15000 61000is pretty common these days. You could further increase the availability by decreasing the fin_timeout. Suppose you do both, you should see over 1500 outbound connections per second, more readily.

这基本上意味着您的系统不能始终如一地保证超过(61000 - 32768) / 60 = 470每秒的套接字数。如果您对此不满意，可以从增加port_range. 如今，将范围设置15000 61000为非常普遍。您可以通过减少fin_timeout. 假设您同时执行这两项操作，您应该更容易看到每秒 1500 个以上的出站连接。

To change the values:

要更改值：

sysctl net.ipv4.ip_local_port_range="15000 61000"
sysctl net.ipv4.tcp_fin_timeout=30

The above should not be interpreted as the factors impacting system capability for making outbound connections per second. But rather these factors affect system's ability to handle concurrent connections in a sustainable manner for large periods of "activity."

以上不应被解释为影响系统每秒建立出站连接能力的因素。而是这些因素影响系统以可持续的方式处理大量“活动”期间并发连接的能力。

Default Sysctl values on a typical Linux box for tcp_tw_recycle& tcp_tw_reusewould be

典型 Linux 机器上tcp_tw_recycle& 的默认 Sysctl 值tcp_tw_reuse是

net.ipv4.tcp_tw_recycle=0
net.ipv4.tcp_tw_reuse=0

These do not allow a connection from a "used" socket (in wait state) and force the sockets to last the complete time_waitcycle. I recommend setting:

这些不允许来自“已使用”套接字（处于等待状态）的连接并强制套接字持续整个time_wait周期。我建议设置：

sysctl net.ipv4.tcp_tw_recycle=1
sysctl net.ipv4.tcp_tw_reuse=1 

This allows fast cycling of sockets in time_waitstate and re-using them. But before you do this change make sure that this does not conflict with the protocols that you would use for the application that needs these sockets. Make sure to read post "Coping with the TCP TIME-WAIT" from Vincent Bernatto understand the implications. The net.ipv4.tcp_tw_recycleoption is quite problematic for public-facing servers as it won't handle connections from two different computers behind the same NAT device, which is a problem hard to detect and waiting to bite you. Note that net.ipv4.tcp_tw_recyclehas been removedfrom Linux?4.12.

这允许在time_wait状态下快速循环套接字并重新使用它们。但在您进行此更改之前，请确保这与您将用于需要这些套接字的应用程序的协议不冲突。请务必阅读Vincent Bernat 的文章“Coping with the TCP TIME-WAIT”以了解其含义。该net.ipv4.tcp_tw_recycle选项对于面向公众的服务器来说是相当有问题的，因为它不会处理来自同一 NAT 设备后面的两台不同计算机的连接，这是一个难以检测并等待咬你的问题。需要注意的是net.ipv4.tcp_tw_recycle已被删除从Linux呢？4.12。

On the Server Side:The net.core.somaxconnvalue has an important role. It limits the maximum number of requests queued to a listen socket. If you are sure of your server application's capability, bump it up from default 128 to something like 128 to 1024. Now you can take advantage of this increase by modifying the listen backlog variable in your application's listen call, to an equal or higher integer.

在服务器端：该net.core.somaxconn值具有重要作用。它限制排队到侦听套接字的最大请求数。如果您确定您的服务器应用程序的能力，请将其从默认值 128 提高到 128 到 1024 之类的值。现在您可以通过将应用程序的侦听调用中的侦听积压变量修改为相等或更高的整数来利用这种增加。

sysctl net.core.somaxconn=1024

txqueuelenparameter of your ethernet cards also have a role to play. Default values are 1000, so bump them up to 5000 or even more if your system can handle it.

txqueuelen您的以太网卡的参数也有作用。默认值是 1000，所以如果你的系统可以处理的话，可以将它们提高到 5000 甚至更多。

ifconfig eth0 txqueuelen 5000
echo "/sbin/ifconfig eth0 txqueuelen 5000" >> /etc/rc.local

Similarly bump up the values for net.core.netdev_max_backlogand net.ipv4.tcp_max_syn_backlog. Their default values are 1000 and 1024 respectively.

类似地增加极值net.core.netdev_max_backlog和net.ipv4.tcp_max_syn_backlog。它们的默认值分别是 1000 和 1024。

sysctl net.core.netdev_max_backlog=2000
sysctl net.ipv4.tcp_max_syn_backlog=2048

Now remember to start both your client and server side applications by increasing the FD ulimts, in the shell.

现在请记住通过在 shell 中增加 FD ulimts 来启动客户端和服务器端应用程序。

Besides the above one more popular technique used by programmers is to reduce the number of tcp writecalls. My own preference is to use a buffer wherein I push the data I wish to send to the client, and then at appropriate points I write out the buffered data into the actual socket. This technique allows me to use large data packets, reduce fragmentation, reduces my CPU utilization both in the user land and at kernel-level.

除上述之外，程序员使用的一种更流行的技术是减少tcp 写入调用的数量。我自己的偏好是使用一个缓冲区，将我希望发送到客户端的数据推送到其中，然后在适当的时候将缓冲的数据写出到实际的套接字中。这种技术允许我使用大数据包，减少碎片，降低我在用户领域和内核级别的 CPU 使用率。

回答by whitehat237

To improve upon the answer given by derobert,

为了改进 derobert 给出的答案，

You can determine what your OS connection limit is by catting nf_conntrack_max.

您可以通过 catting nf_conntrack_max 来确定您的操作系统连接限制是多少。

For example: cat /proc/sys/net/netfilter/nf_conntrack_max

例如： cat /proc/sys/net/netfilter/nf_conntrack_max

You can use the following script to count the number of tcp connections to a given range of tcp ports. By default 1-65535.

您可以使用以下脚本来计算到给定范围的 tcp 端口的 tcp 连接数。默认为 1-65535。

This will confirm whether or not you are maxing out your OS connection limit.

这将确认您是否正在最大化您的操作系统连接限制。

Here's the script.

这是脚本。

#!/bin/bash
OS=$(uname)

case "$OS" in
    'SunOS')
            AWK=/usr/bin/nawk
            ;;
    'Linux')
            AWK=/bin/awk
            ;;
    'AIX')
            AWK=/usr/bin/awk
            ;;
esac

netstat -an | $AWK -v start=1 -v end=65535 ' $NF ~ /TIME_WAIT|ESTABLISHED/ &&  !~ /127\.0\.0\.1/ {
    if ( ~ /\./)
            {sip=}
    else {sip=}

    if ( sip ~ /:/ )
            {d=2}
    else {d=5}

    split( sip, a, /:|\./ )

    if ( a[d] >= start && a[d] <= end ) {
            ++connections;
            }
    }
    END {print connections}'

回答by Kev

In an application level, here are something a developer can do:

在应用程序级别，开发人员可以执行以下操作：

From server side:

从服务器端：

1. Check if load balancer(if you have),works correctly.

2. Turn slow TCP timeouts into 503 Fast Immediate response, if you load balancer work correctly, it should pick the working resource to serve, and it's better than hanging there with unexpected error massages.

1. 检查负载平衡器（如果有）是否正常工作。

2. 将慢速 TCP 超时转换为 503 快速立即响应，如果负载均衡器正常工作，它应该选择工作资源来提供服务，这比用意外错误消息挂在那里要好。

Eg: If you are using node server, u can use toobusy from npm. Implementation something like:

例如：如果您使用的是节点服务器，您可以使用 npm 中的 toobusy。实现类似于：

var toobusy = require('toobusy');
app.use(function(req, res, next) {
  if (toobusy()) res.send(503, "I'm busy right now, sorry.");
  else next();
});

Why 503? Here are some good insights for overload: http://ferd.ca/queues-don-t-fix-overload.html

为什么是503？这里有一些关于过载的很好的见解：http: //ferd.ca/queues-don-t-fix-overload.html

We can do some work in client side too:

我们也可以在客户端做一些工作：

1. Try to group calls in batch, reduce the traffic and total requests number b/w client and server.

2. Try to build a cache mid-layer to handle unnecessary duplicates requests.

1. 尝试对调用进行批量分组，减少客户端和服务器的流量和总请求数。

2. 尝试构建一个缓存中间层来处理不必要的重复请求。