How to get rid of unwanted access log entries

The NGINX access logs contains quite a lot of valuable information that helps provide insights into what content visitors are trying to access, how NGINX responds to the requests, and provides metadata about the visitor requesting the information.

This information is super valuable when it comes to managing your websites, keeping your users happy, and providing insights on how your setup can be fortified.

However, the logs can also easily get out of hand and get filled up with useless and undesirable information that makes it more difficult to scan logs for things that you're actually interested in. Some users even report Fail2Ban locking some of their site visitors out for misinterpreting harmless activity...

To filter out undesirable logs, we can leverage NGINX variables and apply an if statement to our logging directive.

We'll provide a couple of examples. One at the user agent level and another at the directory / file extension level.

In Cleavr, go to your Site > Site NGINX Config.

# Map user agent and filter out logs from known clients, such as Doneo.io which is an uptime monitoring tool

map $http_user_agent $loggable {
    "doneo.io" 0;
    default     1;
}

# Map a request URI, check for extensions to ignore

map $request_uri $loggable {
    ~*\.(ico)$  0;
    default     1;
}

# Then, add the if statement to the logging directive
server {
    access_log /var/log/nginx/example.com-access.log    combined if=$loggable;
    access_log /var/log/nginx/access.log                combined if=$loggable;
}

In the above, we are defining a variable loggable. You can use whatever variable name you'd prefer. We're also assigning a 0 to criteria in which we do not wish to log, such as requests from 'WP Rocket' as well as requests being made for .ico extensions.

We hope this helps you get your access logs nice and tidy so that you can get the most use out of them without being distracted.