JSON Logging With NGINX

As we are rolling out Elastic Search at work it has become necessary to adjust the logging of each of our applications – normally this involves reviewing each possible line of output and then groking out the information you need. However, some applications are more flexible than others, with NGINX you can write your logs exactly as you want them and in JSON format too. With our NGINX setup we are also able to insert HTTP headers at both the Varnish and Tomcat layers and print all the information we need to know about our stack in a single log file – ready to be pushed by rsyslog on each server into our logstash parser and then on to our ES cluster.

Here’s a sample NGINX log format called logstash_json:

log_format access_json '{"timestamp_date": "$time_iso8601", '
                         '"@tenant": "some-tennant", '
                         '"@type": "nginx-access-logs", '
                         '"@level": "global_daily", '
                         '"remote_addr_ip": "$remote_addr", '
                         '"remote_user": "$remote_user", '
                         '"body_bytes_sent_l": "$body_bytes_sent", '
                         '"request_time_d": "$request_time", '
                         '"status": "$status", '
                         '"request": "$request_uri", '
                         '"request_method": "$request_method", '
                         '"http_referrer": "$http_referer", '
                         '"request_body": "$request_body", '
                         '"cache_status": "$upstream_http_x_cache_status", '
                         '"request_valid": "$upstream_http_x_request_valid", '
                         '"http_user_agent": "$http_user_agent", '
                         '"message": "$time_iso8601 $remote_addr $remote_user $body_bytes_sent $request_time $status $request $request_method $http_referer $request_body $upstream_http_x_cache_status $upstream_http_x_request_valid $http_user_agent" }';

The upstream HTTP header x_cache_status is set by Varnish depending on whether the request was a hit/miss/pass, x_request_valid is set based on whether the client requested a valid ISIN format.

Next we use rsyslogd to push the log file into our logstash agent, use something like this in rsyslog.d:

module(load="imfile" PollingInterval="10
template(name="nginxAccessTemplate" type="string" string="%msg%\n")
input(type="imfile"
File="/some/log/dir/nginx_json.log"
Tag="nginx.access"
StateFile="stat-file-api-nginx-access"
Severity="info"
Facility="user"
ruleset="nginx-access")

ruleset(name="nginx-access")
{
action(type="omfwd"
template="nginxAccessTemplate"
Target="some-server"
Protocol="udp")
}

As we have already written our log in JSON and with all the fields we need, it means we only need minimal config in logstash:

input
{
udp
  {
  port => 5544
  codec => json
  }
}

That’s it! Instead of reading in the standard logs and then writing grok filters for each field, you can do the formatting with NGINX, its easier, better performing and gives you more flexibility.

Published by

Guy Tabrar

*NIX Admin.