MySQL: Can't create/write to file '/tmp/#sql_3c6_0.MYI' (Errcode: 2) - What does it even mean?

Question

For some reason my production DB decided to spew out this message. All application calls fail to the DB with the error:

PreparedStatementCallback; SQL [ /*lo         


        

Answer 1

Tremendous thanks to ArturZ for pointing me in the right direction on this. I don't have tmpwatch installed on my system so that isn't the cause of the problem in my case. But the end result is the same: The private /tmp that systemd creates is getting removed. Here's what happens:

1. systemd creates a new process via clone() with the CLONE_NEWNS flag to obtain a private namespace. Or maybe it calls unshare() with CLONE_NEWNS. Same thing.

2. systemd creates a subdirectory in /tmp (e.g. /tmp/systemd-namespace-XRiWad/private) and mounts it on /tmp. Because CLONE_NEWNS was set in #1, this mountpoint is invisible to all other processes.

3. systemd then invokes mysqld in this private namespace.

4. Some specific database operations (e.g. "describe ;") create & remove temporary files, which has the side effect of updating the timestamp on /tmp/systemd-namespace-XRiWad/private. Other database operations execute without using /tmp at all.

5. Eventually 10 days go by where even though the database itself remains active, no operations occur that update the timestamp on /tmp/systemd-namespace-XRiWad/private.

6. /bin/systemd-tmpfiles comes along and removes the "old" /tmp/systemd-namespace-XRiWad/private directory, effectively rendering the private /tmp unusable for mysqld while the public /tmp remains available for everything else on the system.

Restarting mysqld works because this starts everything over again at step #1, with a brand new private /tmp directory. However, the problem eventually comes back again. And again.

The simple solution is to configure /bin/systemd-tmpfiles so that it preserves anything in /tmp with the name /tmp/systemd-namespace-*. I did this by creating /etc/tmpfiles.d/privatetmp.conf with the following contents:

x   /tmp/systemd-namespace-*
x   /tmp/systemd-namespace-*/private

Problem solved.