im running a fabric script that, amongst other things, is supposed to restart gunicorn on an ubuntu server, the command is below:
supervisorctl status projectname:gunicorn | sed "s/.*[pid ]\([0-9]\+\)\,.*/\1/" | xargs kill -HUP the problem is, is that gunicorn doesnt appear to be running in the first place so the process cannot be killed, ive ssh'd into the amazon ec2 instance and ran
sudo supervisorctl restart projectname:gunicorn' and I get an error response that says:
projectname:gunicorn: ERROR (not running) projectname:gunicorn ERROR (abnormal termination)
so i attempted to start gunicorn by running
sudo supervisorctl start projectname:gunicorn and the error says
'projectname:gunicorn: Error (abnormal termination)'
So I need gunicorn to run, and im having trouble acheiving this
Ive also checked the gunicorn log and the text below, below is the relevant output
2014-01-17 14:58:14 [12260] [INFO] Starting gunicorn 0.14.3 2014-01-17 14:58:14 [12260] [INFO] Listening at: http://127.0.0.1:9000 (12260) 2014-01-17 14:58:14 [12260] [INFO] Using worker: sync 2014-01-17 14:58:14 [12263] [INFO] Booting worker with pid: 12263 2014-01-17 14:58:14 [12264] [INFO] Booting worker with pid: 12264 2014-01-17 14:58:14 [12265] [INFO] Booting worker with pid: 12265 2014-01-17 14:58:14 [12266] [INFO] Booting worker with pid: 12266 2014-01-17 14:58:14 [12263] [INFO] Worker exiting (pid: 12263) 2014-01-17 14:58:14 [12264] [INFO] Worker exiting (pid: 12264) 2014-01-17 14:58:14 [12265] [INFO] Worker exiting (pid: 12265) 2014-01-17 14:58:14 [12266] [INFO] Worker exiting (pid: 12266) Traceback (most recent call last): File "/opt/screening/env/bin/gunicorn_django", line 9, in <module> load_entry_point('gunicorn==0.14.3', 'console_scripts', 'gunicorn_django')() File "/opt/compliance_engine/env/local/lib/python2.7/site-packages/gunicorn/app/djangoapp.py", line 129, in run DjangoApplication("%prog [OPTIONS] [SETTINGS_PATH]").run() File "/opt/compliance_engine/env/local/lib/python2.7/site-packages/gunicorn/app/base.py", line 129, in run Arbiter(self).run() File "/opt/compliance_engine/env/local/lib/python2.7/site-packages/gunicorn/arbiter.py", line 184, in run self.halt(reason=inst.reason, exit_status=inst.exit_status) File "/opt/compliance_engine/env/local/lib/python2.7/site-packages/gunicorn/arbiter.py", line 279, in halt self.stop() File "/opt/compliance_engine/env/local/lib/python2.7/site-packages/gunicorn/arbiter.py", line 327, in stop self.reap_workers() File "/opt/compliance_engine/env/local/lib/python2.7/site-packages/gunicorn/arbiter.py", line 413, in reap_workers raise HaltServer(reason, self.WORKER_BOOT_ERROR) gunicorn.errors.HaltServer: <HaltServer 'Worker failed to boot.' 3> also, here is the conf file
[program:gunicorn] command=/opt/screening/env/bin/gunicorn_django --pythonpath . ce.settings -w 4 --bind 127.0.0.1:9000 directory=/opt/screening/repository user=www-data autostart=true autorestart=true stdout_logfile=/opt/screening/logs/gunicorn.log redirect_stderr=true [program:celeryd] command=/opt/screening/env/bin/python manage.py celeryd --autoscale=16,2 -E -l INFO --pidfile=/opt/screening/tmp/pids/celeryd.pid directory=/opt/screening/repository user=www-data autostart=true autorestart=true stdout_logfile=/opt/screening/logs/celeryd.log redirect_stderr=true [program:celerybeat] command=/opt/screening/env/bin/python manage.py celerybeat -l INFO -- schedule=/opt/screening/tmp/celerybeat-schedule -- pidfile=/opt/screening/tmp/pids/celerybeat.pid directory=/opt/screening/repository user=www-data autostart=true autorestart=true stdout_logfile=/opt/screening/logs/celerybeat.log redirect_stderr=true [program:celerycam] command=/opt/screening/env/bin/python manage.py celerycam -- pidfile=/opt/screening/tmp/pids/celerycam.pid directory=/opt/screening/repository user=www-data autostart=true autorestart=true stdout_logfile=/opt/screening/logs/celerycam.log redirect_stderr=true [group:screening] programs=gunicorn,celeryd,celerybeat,celerycam any ideas? I understand that this is a lot of text, any hints or pointers would be much appreciated
Thanks for reading,
edit:
ran unicorn on its own, activated the virtual env and ran
python manage.py run_gunicorn the terminal printed the below output
2014-01-19 22:02:35 [14735] [INFO] Starting gunicorn 0.14.3 2014-01-19 22:02:35 [14735] [INFO] Listening at: http://127.0.0.1:8000 (14735) 2014-01-19 22:02:35 [14735] [INFO] Using worker: sync 2014-01-19 22:02:35 [14742] [INFO] Booting worker with pid: 14742 also ran the run server in the virtualenv:
python manage.py runserver 7000 Validating models... 0 errors found Django version 1.3, using settings 'ce.settings' Development server is running at http://127.0.0.1:7000/ Quit the server with CONTROL-C. so no apparent errors there
edit 2:
have spoken to a couple other people about this, and was advised to look at the permissions for the gunicorn logs, here they are:
-rw-rw-r-- 1 www-data ubuntu 3270504 2014-01-19 23:23 gunicorn.log the www-data user matches the one set in the supervisor config
edit 3: I ran the gunicorn command again, but this time added logging info:
gunicorn_django --pythonpath . ce.settings -w 4 --bind 127.0.0.1:9000 --debug --log-level debug and received the following error message:
Traceback (most recent call last): File "/opt/screening/env/local/lib/python2.7/site-packages/gunicorn/arbiter.py", line 453, in spawn_worker worker.init_process() File "/opt/screening/env/local/lib/python2.7/site-packages/gunicorn/workers/base.py", line 99, in init_process self.wsgi = self.app.wsgi() File "/opt/screening/env/local/lib/python2.7/site-packages/gunicorn/app/base.py", line 101, in wsgi self.callable = self.load() File "/opt/screening/env/local/lib/python2.7/site-packages/gunicorn/app/djangoapp.py", line 87, in load mod = util.import_module("gunicorn.app.django_wsgi") File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module __import__(name) File "/opt/screening/env/local/lib/python2.7/site-packages/gunicorn/app/django_wsgi.py", line 18, in <module> from django.core.management.validation import get_validation_errors File "/opt/screening/env/local/lib/python2.7/site-packages/django/core/management/validation.py", line 3, in <module> from django.contrib.contenttypes.generic import GenericForeignKey, GenericRelation File "/opt/screening/env/local/lib/python2.7/site-packages/django/contrib/contenttypes/generic.py", line 6, in <module> from django.db import connection File "/opt/screening/env/local/lib/python2.7/site-packages/django/db/__init__.py", line 14, in <module> if not settings.DATABASES: File "/opt/screening/env/local/lib/python2.7/site-packages/django/utils/functional.py", line 276, in __getattr__ self._setup() File "/opt/screening/env/local/lib/python2.7/site-packages/django/conf/__init__.py", line 42, in _setup self._wrapped = Settings(settings_module) File "/opt/screening/env/local/lib/python2.7/site-packages/django/conf/__init__.py", line 89, in __init__ raise ImportError("Could not import settings '%s' (Is it on sys.path?): %s" % (self.SETTINGS_MODULE, e)) ImportError: Could not import settings 'ce.settings' (Is it on sys.path?): No module named ce.settings 2014-01-20 09:14:22 [31830] [INFO] Worker exiting (pid: 31830) Traceback (most recent call last): File "/opt/screening/env/bin/gunicorn_django", line 9, in <module> load_entry_point('gunicorn==0.14.3', 'console_scripts', 'gunicorn_django')() File "/opt/screening/env/local/lib/python2.7/site-packages/gunicorn/app/djangoapp.py", line 129, in run DjangoApplication("%prog [OPTIONS] [SETTINGS_PATH]").run() File "/opt/screening/env/local/lib/python2.7/site-packages/gunicorn/app/base.py", line 129, in run Arbiter(self).run() File "/opt/screening/env/local/lib/python2.7/site-packages/gunicorn/arbiter.py", line 184, in run self.halt(reason=inst.reason, exit_status=inst.exit_status) File "/opt/screening/env/local/lib/python2.7/site-packages/gunicorn/arbiter.py", line 279, in halt self.stop() File "/opt/screening/env/local/lib/python2.7/site-packages/gunicorn/arbiter.py", line 327, in stop self.reap_workers() File "/opt/screening/env/local/lib/python2.7/site-packages/gunicorn/arbiter.py", line 413, in reap_workers raise HaltServer(reason, self.WORKER_BOOT_ERROR) gunicorn.errors.HaltServer: <HaltServer 'Worker failed to boot.' 3> so it appears that the salient info is this:
ImportError: Could not import settings 'ce.settings' (Is it on sys.path?): No module named ce.settings My settings are in a settings directory, and the init file is present, so the issue isnt that. Also the application starts on the runserver so the settings file must be importable