IOError: No space left on device - which device?

Question

I'm uploading a small file (8.5 Mb) to a flask test server.

When the file finishes uploading, the server reports:

    File "/home/ubuntu/.virtualenvs/eco_app/lib/python2.7/site-packages/wtforms/form.py", 
        line 212, in __call__
    return type.__call__(cls, *args, **kwargs)
    File "/home/ubuntu/.virtualenvs/eco_app/lib/python2.7/site-packages/flask_wtf/form.py", line 49, in __init__
        formdata = request.form
    File "/home/ubuntu/.virtualenvs/eco_app/lib/python2.7/site-packages/werkzeug/local.py", 
        line 338, in __getattr__
    return getattr(self._get_current_object(), name)
    File "/home/ubuntu/.virtualenvs/eco_app/lib/python2.7/site-packages/werkzeug/utils.py", 
         line 71, in __get__
    value = self.func(obj)
    File "/home/ubuntu/.virtualenvs/eco_app/lib/python2.7/site-packages/werkzeug/wrappers.py", 
         line 484, in form
    self._load_form_data()
    File "/home/ubuntu/.virtualenvs/eco_app/lib/python2.7/site-packages/flask/wrappers.py", 
        line 165, in _load_form_data
    RequestBase._load_form_data(self)
    File "/home/ubuntu/.virtualenvs/eco_app/lib/python2.7/site-packages/werkzeug/wrappers.py", 
        line 356, in _load_form_data
    mimetype, content_length, options)
    File "/home/ubuntu/.virtualenvs/eco_app/lib/python2.7/site-packages/werkzeug/formparser.py", 
        line 193, in parse
    content_length, options)
    File "/home/ubuntu/.virtualenvs/eco_app/lib/python2.7/site-packages/werkzeug/formparser.py",
        line 99, in wrapper
    return f(self, stream, *args, **kwargs)
    File "/home/ubuntu/.virtualenvs/eco_app/lib/python2.7/site-packages/werkzeug/formparser.py",
        line 210, in _parse_multipart
    form, files = parser.parse(stream, boundary, content_length)
    File "/home/ubuntu/.virtualenvs/eco_app/lib/python2.7/site-packages/werkzeug/formparser.py", 
        line 520, in parse
    return self.cls(form), self.cls(files)
    File "/home/ubuntu/.virtualenvs/eco_app/lib/python2.7/site-packages/werkzeug/datastructures.py", 
        line 373, in __init__
    for key, value in mapping or ():
    File "/home/ubuntu/.virtualenvs/eco_app/lib/python2.7/site-packages/werkzeug/formparser.py", 
        line 518, in <genexpr>
    form = (p[1] for p in formstream if p[0] == 'form')
    File "/home/ubuntu/.virtualenvs/eco_app/lib/python2.7/site-packages/werkzeug/formparser.py", 
        line 494, in parse_parts
        _write(ell)
IOError: [Errno 28] No space left on device

Now, the server has plenty of free space - over 3Gb.

I've looked at the Werkzeug github repo to try to find the location that Werkzeug is trying to write to, but can't track it down.

I've also checked tempfile.gettempdir() which gives /var/tmp as the temporary file directory, but this folder is practically empty, so I don't think that's the location that's creating the issue.

How do I find the device that has no space?

Answer 1

@Tom Hunt's comment was on the right track.

This unix SE answer explains what happened.

As a protection against low disc space, some daemons automatically "shadows" the current /tmp/ dir with a ram disc if the the root partition runs out of disc space. Sadly there's no automatic reversion of that process once enough disc space is free again.

I unmounted /tmp directory and followed Nitesh's suggestion:

sudo umount /tmp
sudo echo 'MINTMPKB=0' > sudo /etc/default/mountoverflowtmp

and now uploads are working properly.

Answer 2

If you can get a shell on the server, try typing df -h and looking for any entries which show Use% of 100%, or Avail of less than your file size.

Answer 3

Werkzeug stores files over a certain size in your temp directory using tempfile.TemporaryFile(), but take into account that those files are unlinked for security. You won't see them listed in the directory. tempfile.gettempdir() is the correct method of determining the directory used for this.

Your /var/tmp directory is probably configured for a smaller partition. Check with df -h to see if that partition still has enough space left. You also need to check for free inodes, with df -i.

It could also be that a process (possibly yours) is hanging on to such unlinked files too long and the space hasn't been returned to the OS yet. You can check for processes holding on to deleted files with:

lsof -nP | grep '/var/tmp' | grep '(deleted)'

or

find /proc/*/fd -ls | grep '/var/tmp' | grep  '(deleted)'

Answer 4

Try df -i , maybe there are no free inodes.

EDIT:

another option, find werkzeug pid, let it be 777,

• run strace -p 777 &> /tmp/strace_log
• try to upload a file
• stop strace
• find that No space left on device message, it will be like write(1, "blah blah ..."..., 57) = -1 ENOSPC (No space left on device) first argument to write is a file descriptor
• go up and try to locate specific open(... = X syscall , that X is a file descriptor that will fail in "write" syscall step

Pretty cumbersome, I know, but that's debugging.