Whenever I so much as load ggplot2, X11 jumps into action (hogging resources).
library(ggplot2)
Is what I do, and boo
As mentioned in the comments and the github issue in the ggplot2 page, this is cause by the package colorspace version "1.2-5". You can wait to a new version of the package with a fix. Or, a temporary solution is to install a previous version of colorspace. If you have the devtools package installed you can easily do:
library(devtools)
install_version("colorspace", "1.2-4")