According to the documentation, if DEBUG is set to False and something is provided under the ADMINS setting, Django will send an email
DEBUG
False
ADMINS
My web hosting provider - Webfaction - only allows emails to be sent From an email that has been explicitly created in the administrator panel. Creating one fixed the problem.