Python IOError：设备上没有剩余空间 - 哪个设备？

IOError: No space left on device - which device?

提问by ABM

I'm uploading a small file (8.5 Mb) to a flask test server.

我正在将一个小文件（8.5 Mb）上传到烧瓶测试服务器。

When the file finishes uploading, the server reports:

文件上传完成后，服务器报告：

    File "/home/ubuntu/.virtualenvs/eco_app/lib/python2.7/site-packages/wtforms/form.py", 
        line 212, in __call__
    return type.__call__(cls, *args, **kwargs)
    File "/home/ubuntu/.virtualenvs/eco_app/lib/python2.7/site-packages/flask_wtf/form.py", line 49, in __init__
        formdata = request.form
    File "/home/ubuntu/.virtualenvs/eco_app/lib/python2.7/site-packages/werkzeug/local.py", 
        line 338, in __getattr__
    return getattr(self._get_current_object(), name)
    File "/home/ubuntu/.virtualenvs/eco_app/lib/python2.7/site-packages/werkzeug/utils.py", 
         line 71, in __get__
    value = self.func(obj)
    File "/home/ubuntu/.virtualenvs/eco_app/lib/python2.7/site-packages/werkzeug/wrappers.py", 
         line 484, in form
    self._load_form_data()
    File "/home/ubuntu/.virtualenvs/eco_app/lib/python2.7/site-packages/flask/wrappers.py", 
        line 165, in _load_form_data
    RequestBase._load_form_data(self)
    File "/home/ubuntu/.virtualenvs/eco_app/lib/python2.7/site-packages/werkzeug/wrappers.py", 
        line 356, in _load_form_data
    mimetype, content_length, options)
    File "/home/ubuntu/.virtualenvs/eco_app/lib/python2.7/site-packages/werkzeug/formparser.py", 
        line 193, in parse
    content_length, options)
    File "/home/ubuntu/.virtualenvs/eco_app/lib/python2.7/site-packages/werkzeug/formparser.py",
        line 99, in wrapper
    return f(self, stream, *args, **kwargs)
    File "/home/ubuntu/.virtualenvs/eco_app/lib/python2.7/site-packages/werkzeug/formparser.py",
        line 210, in _parse_multipart
    form, files = parser.parse(stream, boundary, content_length)
    File "/home/ubuntu/.virtualenvs/eco_app/lib/python2.7/site-packages/werkzeug/formparser.py", 
        line 520, in parse
    return self.cls(form), self.cls(files)
    File "/home/ubuntu/.virtualenvs/eco_app/lib/python2.7/site-packages/werkzeug/datastructures.py", 
        line 373, in __init__
    for key, value in mapping or ():
    File "/home/ubuntu/.virtualenvs/eco_app/lib/python2.7/site-packages/werkzeug/formparser.py", 
        line 518, in <genexpr>
    form = (p[1] for p in formstream if p[0] == 'form')
    File "/home/ubuntu/.virtualenvs/eco_app/lib/python2.7/site-packages/werkzeug/formparser.py", 
        line 494, in parse_parts
        _write(ell)
IOError: [Errno 28] No space left on device

Now, the server has plenty of free space - over 3Gb.

现在，服务器有足够的可用空间 - 超过 3Gb。

I've looked at the Werkzeug github repoto try to find the location that Werkzeug is trying to write to, but can't track it down.

我查看了Werkzeug github 存储库，试图找到 Werkzeug 试图写入的位置，但无法追踪到它。

I've also checked tempfile.gettempdir()which gives /var/tmp as the temporary file directory, but this folder is practically empty, so I don't think that's the location that's creating the issue.

我还检查了tempfile.gettempdir()，它提供了 /var/tmp 作为临时文件目录，但这个文件夹实际上是空的，所以我认为这不是造成问题的位置。

How do I find the device that has no space?

如何找到没有空间的设备？

采纳答案by ABM

@Tom Hunt's comment was on the right track.

@Tom Hunt 的评论是正确的。

This unix SE answer explains what happened.

这个 unix SE 答案解释了发生了什么。

As a protection against low disc space, some daemons automatically "shadows" the current /tmp/ dir with a ram disc if the the root partition runs out of disc space. Sadly there's no automatic reversion of that process once enough disc space is free again.

为了防止磁盘空间不足，如果根分区磁盘空间不足，一些守护程序会自动使用 ram 磁盘“遮蔽”当前的 /tmp/ 目录。遗憾的是，一旦有足够的磁盘空间再次可用，该过程就不会自动恢复。

I unmounted /tmp directory and followed Nitesh's suggestion:

我卸载了 /tmp 目录并遵循了 Nitesh 的建议：

sudo umount /tmp
sudo echo 'MINTMPKB=0' > sudo /etc/default/mountoverflowtmp

and now uploads are working properly.

现在上传工作正常。

回答by Tom Hunt

If you can get a shell on the server, try typing df -hand looking for any entries which show Use%of 100%, or Availof less than your file size.

如果您可以在服务器上获得 shell，请尝试键入df -h并查找显示Use%为 100% 或Avail小于您的文件大小的任何条目。

回答by Martijn Pieters

Werkzeug stores files over a certain size in your temp directory using tempfile.TemporaryFile(), but take into account that those files are unlinkedfor security. You won't see them listed in the directory. tempfile.gettempdir()is the correct method of determining the directory used for this.

Werkzeug 使用 将超过特定大小的文件存储在您的临时目录中tempfile.TemporaryFile()，但考虑到这些文件未链接以确保安全。您不会在目录中看到它们。tempfile.gettempdir()是确定用于此的目录的正确方法。

Your /var/tmpdirectory is probably configured for a smaller partition. Check with df -hto see if that partition still has enough space left. You also need to check for free inodes, with df -i.

您的/var/tmp目录可能配置为较小的分区。检查df -h该分区是否还有足够的空间。你还需要检查免费的inode，用df -i。

It could also be that a process (possibly yours) is hanging on to such unlinked files too long and the space hasn't been returned to the OS yet. You can check for processes holding on to deleted files with:

也可能是某个进程（可能是您的）挂在此类未链接的文件上的时间太长，而空间尚未返回给操作系统。您可以通过以下方式检查保留已删除文件的进程：

lsof -nP | grep '/var/tmp' | grep '(deleted)'

or

或者

find /proc/*/fd -ls | grep '/var/tmp' | grep  '(deleted)'

回答by Dima Medvedev

Try df -i, maybe there are no free inodes.

试试吧df -i，也许没有空闲的 inode。

EDIT:

编辑：

another option, find werkzeug pid, let it be 777,

另一种选择，找到werkzeug pid，让它是777，

• run strace -p 777 &> /tmp/strace_log
• try to upload a file
• stop strace
• find that No space left on devicemessage, it will be like write(1, "blah blah ..."..., 57) = -1 ENOSPC (No space left on device)first argument to write is a file descriptor
• go up and try to locate specific open(... = Xsyscall , that X is a file descriptor that will fail in "write" syscall step

• 跑 strace -p 777 &> /tmp/strace_log
• 尝试上传文件
• 停止跟踪
• 找到那个No space left on device消息，它会像 write(1, "blah blah ..."..., 57) = -1 ENOSPC (No space left on device)write 的第一个参数是一个文件描述符
• 上去尝试定位特定的open(... = X系统调用，X 是一个文件描述符，它将在“写入”系统调用步骤中失败

Pretty cumbersome, I know, but that's debugging.

我知道这很麻烦，但那是调试。