I would like to install a package with a security profile that does not have access to /tmp , but has its own temporary directory, for example /tmp/jeroen
The correct answer was given by @hadley in the comments: in order to make sure install.packages uses the custom temporary directory, one needs to do
Sys.setenv(TMPDIR="/tmp/jeroen")
in addition to:
configure.vars="TMPDIR=/tmp/jeroen"
This way both the extraction and the installation of the package avoids the system or user default temporary directory.
The documentation in help(tempdir) pretty clearly states that TMP, TMPDIR, ... are used:
By default, ‘tmpdir’ will be the directory given by ‘tempdir()’. This will be a subdirectory of the per-session temporary directory found by the following rule when the R session is started. The environment variables ‘TMPDIR’, ‘TMP’ and ‘TEMP’ are checked in turn and the first found which points to a writable directory is used: if none succeeds ‘/tmp’ is used.
So if setting one alone does not help, maybe you want to set several, and make sure the permissions on your 'replacement directory' are permissive enough etc pp.