How can I make nginx/gunicorn use structured logging?

问题

I currently see

172.19.0.1 - - [09/Feb/2018:07:00:32 +0000] "GET /ping HTTP/1.1" 200 1 "-" "curl/7.47.0"

within my logs, but I use structured logging like this.

I even tried

ch = logging.StreamHandler()
ch.setFormatter(pythonjsonlogger.jsonlogger.JsonFormatter())
logging.getLogger("urllib3").addHandler(ch)

but I still see those messages. I have nginx/gunicorn like this (source):

nginx = subprocess.Popen(['nginx', '-c', '/opt/program/nginx.conf'])
gunicorn = subprocess.Popen(['gunicorn',
                             '--timeout', str(model_server_timeout),
                             '-k', 'gevent',
                             '-b', 'unix:/tmp/gunicorn.sock',
                             '-w', str(model_server_workers),
                             'server.wsgi:app'])

I guess this is where the log messages come from. But I have no idea how to get structured logging there.

回答1:

The logging format set in Python process has nothing to do with the one in Nginx and Gunicorn.

If you want to set Nginx logging format, see

https://www.nginx.com/resources/admin-guide/logging-and-monitoring/#access_log

For gunicorn, see http://docs.gunicorn.org/en/stable/settings.html#access-log-format

回答2:

Found it. I had to edit the nginx.conf:

Within http:

log_format structured '{"remote_addr": "$remote_addr", "remote_user": "$remote_user", "request": "$request", "status": $status, "body_bytes_sent": $body_bytes_sent, "http_referer": "$http_referer", "http_user_agent": "$http_user_agent"}';

and within http > server:

access_log /dev/stdout structured;

See http://nginx.org/en/docs/http/ngx_http_log_module.html

Now the messages look like this:

{
    "remote_addr": "10.32.0.2",
    "remote_user": "-",
    "request": "GET /ping HTTP/1.1",
    "status": 200,
    "body_bytes_sent": 1,
    "http_referer": "-",
    "http_user_agent": "AHC/2.0"
}

Still TODO

It would be nice if the request was

{"method": "GET",
 "endpoint": "/ping",
 "protocol": "HTTP/1.1"}

来源：https://stackoverflow.com/questions/48700404/how-can-i-make-nginx-gunicorn-use-structured-logging

标签

python

nginx

gunicorn