Fail2ban with nftables and Crowd-Sourced Blacklists

2026/05/13 in Security

Fail2ban watches your logs for suspicious patterns and temporarily bans offending IPs at the firewall level. Combined with crowd-sourced blacklists from AbuseIPDB, Bitwire, and Spamhaus, you get a layered defense that catches both repeat offenders hitting your server right now and known bad actors flagged across the internet. This guide covers the full setup — SSH hardening, Nginx forbidden-request jails, AbuseIPDB reporting, incremental banning, and daily blacklist population with nftables.

How the pieces fit together

There are two independent systems working in parallel, both feeding into nftables:

  • fail2ban (reactive) — watches your Nginx error log and SSH auth log in real time. When an IP triggers a jail rule enough times, fail2ban adds it to an nftables set and optionally reports it to AbuseIPDB. Bans expire on a timer.
  • Daily blacklist cron (proactive) — fetches known-bad IP ranges from AbuseIPDB, Bitwire, and Spamhaus every night, then loads them into separate nftables sets. These IPs are dropped before they ever reach Nginx or SSH. The sets are flushed and repopulated each run, so entries follow each source's refresh cycle.

Both systems insert drop rules into the same inet f2b-table. fail2ban handles the targeted attackers probing your site right now; the blacklist handles the global background radiation. Together they cover both the directed attacks and the drive-by noise.

Prerequisites

  • A Linux server with nftables (the default firewall backend on modern Debian/Ubuntu/RHEL)
  • Nginx with deny all rules already in place (the jail triggers on 403 responses)
  • An AbuseIPDB API key (free tier allows 1,000 reports per day, 3,000 for verified domains)

Step 1: Install fail2ban and nftables

# Debian/Ubuntu
sudo apt install fail2ban nftables

# RHEL/Fedora/Rocky
sudo dnf install fail2ban nftables

# Enable nftables and fail2ban at boot
sudo systemctl enable nftables fail2ban

Verify nftables is the active backend — modern systems use it by default, but it's worth confirming:

sudo nft list ruleset

If you see an empty ruleset (just a table definition or nothing at all), nftables is running and ready.

Step 2: Set up nftables for the blacklist sets

The crowd-sourced blacklist script (Step 7) populates two nftables sets — abuse-blacklist (IPv4) and abuse-blacklist6 (IPv6). Create the table and sets now so they exist before anything references them. Add this to /etc/nftables.conf or apply it directly:

sudo nft add table inet f2b-table
sudo nft add set inet f2b-table abuse-blacklist  { type ipv4_addr; flags interval; }
sudo nft add set inet f2b-table abuse-blacklist6 { type ipv6_addr; flags interval; }

# Drop traffic from blacklisted IPs — must come before your accept rules
sudo nft insert rule inet f2b-table input ip saddr @abuse-blacklist drop
sudo nft insert rule inet f2b-table input ip6 saddr @abuse-blacklist6 drop

If you don't already have an inet f2b-table (fail2ban creates one automatically on first ban), create it manually first. The flags interval; is critical — it allows the set to store CIDR ranges, not just individual IPs.

Step 3: Design your Nginx blocks to work with fail2ban

The nginx-forbidden jail triggers on 403 responses. This means your Nginx deny all; rules are now part of your intrusion detection system — every 403 is a signal. But for URLs where you don't want to risk accidental bans (legitimate users hitting a valid but temporarily unavailable endpoint), use return 444 instead:

# Deny — produces a 403, triggers fail2ban
location ~* ^/(wp-config\.php|readme\.html|license\.txt) {
    deny all;
}

# Silent drop — produces no response, does NOT trigger fail2ban
location = /some-endpoint {
    return 444;
}

A 444 closes the connection immediately with no HTTP response. It's not logged as a 403, so fail2ban ignores it. This gives you a two-tier system:

  • 403: "You're looking for something you shouldn't be — and I'm telling the whole neighborhood about it" (logged, reported to AbuseIPDB, and banned on repeat)
  • 444: "I don't want to talk to you, but I'm not making a federal case out of it" (silent, no ban risk, used for rate-limited or temporarily disabled endpoints)

.well-known paths that should never 403: Google Chrome sends automated probes to /.well-known/traffic-advice for performance optimization hints. AI crawlers check /.well-known/tdmrep.json for EU AI Act training-data opt-out. Both expect a silent acknowledgment, not a 403. Add these exact-match locations before your dot-file catch-all:

location = /.well-known/traffic-advice { return 204; }
location = /.well-known/tdmrep.json    { return 204; }

204 No Content tells the client "acknowledged, nothing to return." It generates no error log entry, costs zero bytes of body, and won't trigger your nginx-forbidden jail. Without these, each probe produces a 403 that fills your error log with noise and risks banning Googlebot.

Step 4: Create the AbuseIPDB action

Fail2ban ships with a default AbuseIPDB action, but you may want a custom one with finer control over categories. Create /etc/fail2ban/action.d/abuseipdb.local:

[Definition]
actionban = curl -s -o /dev/null --fail --tlsv1.2 \
    -H "Key: <abuseipdb_apikey>" \
    -H "Accept: application/json" \
    -d "ip=<ip>" \
    -d "categories=<abuseipdb_category>" \
    -d "comment=<abuseipdb_comment>" \
    "https://api.abuseipdb.com/api/v2/report"

[Init]
abuseipdb_apikey = YOUR_API_KEY_HERE
abuseipdb_category = 18,22
abuseipdb_comment = Fail2Ban banned this IP after repeated failed attempts

Adjust abuseipdb_category to match what you're blocking:

Category Use for
18 Brute-force (SSH, login pages)
19 HTTP POST/scanning attacks
21 Web application attacks
22 SSH abuse

Replace YOUR_API_KEY_HERE with your actual AbuseIPDB key. Restrict permissions on the file — it contains a secret:

sudo chmod 600 /etc/fail2ban/action.d/abuseipdb.local

Step 5: Create the fail2ban filters

You need three filters. Create each file as shown below.

5a. nginx-forbidden — catches 403 responses from your deny rules

Create /etc/fail2ban/filter.d/nginx-forbidden.conf:

[Definition]
failregex = ^\s*\[error\] \d+#\d+: \*\d+ access forbidden by rule, client: <HOST>
            ^<HOST> - .* "(GET|POST|HEAD|PUT|DELETE|OPTIONS|PATCH).*" 403

ignoreregex =

The first line matches the error log — fail2ban strips the timestamp, leaving a line that starts with [error] (possibly preceded by whitespace, which \s* absorbs). No datepattern override is needed; fail2ban's default date detectors handle Nginx's YYYY/MM/DD HH:MM:SS format correctly. The key phrase access forbidden by rule catches every deny all block in your configuration: literal /.env, encoded /%2eenv, double-encoded /%252eenv, and any future encoding trick — all of which Nginx has already decoded and rejected before writing the log entry.

The second line is a fallback that matches the access log — catches 403 responses from other sources like limit_req rejections or application-level forbidden responses.

Why \s* matters: when fail2ban strips the timestamp, it can leave a single space at the start of the remaining text. Without \s*, the regex expects [error] at the very first character and misses. With it, the pattern is tolerant of whatever whitespace the date detector leaves behind.

Real-world example: Why the default filter fails on encoded requests

On the night of May 12, 2026, a scanner hit this server with a wave of URL-encoded requests designed to slip past naive path-matching filters:

GET /core/sql/database%2eenv HTTP/1.1
GET /src/config/%2eenv HTTP/1.1
GET /%2eenv%2eyml HTTP/1.1
GET /users/%2eenv%2ebak HTTP/1.1
GET /jenkins/%2eenv HTTP/1.1
GET /dev/%2eenv%2ejpg HTTP/1.1
GET /%2eenv%2elocal%2ejpg HTTP/1.1
GET /files/%2eenv%2ejpg HTTP/1.1
GET /customer/config%2eyaml HTTP/1.1

%2e is a URL-encoded dot — Nginx decodes it, matches it against your deny all rules, and returns a 403. The problem isn't Nginx blocking the request. The problem is that fail2ban never saw it happen.

Many distributions ship a default Nginx error filter that looks like this:

[INCLUDES]
before = nginx-error-common.conf

[Definition]
failregex = ^%(__prefix_line)saccess forbidden by rule, client: <HOST>

datepattern = {^LN-BEG}

There are three reasons this fails on encoded requests:

1. The Normalization Gap — Nginx normalizes the URI (decoding %2e into .) before checking security rules, which is why your deny rules correctly trigger a 403. But Nginx records the raw, unparsed request in your logs. Your log file shows /%2eenv, not /.env. The regex never sees what Nginx saw.

2. Literal String Matching — Fail2ban is a text parser with no URL-decoding logic. To a regex engine, %2e is a four-character string. It will never match a literal dot \. in your pattern.

3. Structural Brittleness — The default regex access forbidden by rule, client: <HOST> is rigid. When an encoded request produces a longer log line with additional quoted fields, the <HOST> token can be pushed into a position the default regex doesn't account for. Fail2ban skips the line.

The explicit regex above solves all three: it matches the outcome (the forbidden-by-rule error or the 403 status code) rather than the shape of the request, so it works regardless of encoding.

If you prefer to keep the __prefix_line approach — some distributions patch it to handle timestamps. In that case, you can add a second failregex that specifically targets encoded attacks:

failregex = ^%(__prefix_line)saccess forbidden by rule, client: <HOST>
            ^%(__prefix_line)saccess forbidden by rule, client: <HOST>.*(%%25?2[eE]|%%25?2[fF]|%%00)

%%25?2[eE] catches %2e, %2E, and double-encoded %252e. %%25?2[fF] catches encoded slashes. %%00 catches null-byte terminators. But this still inherits whatever prefix-matching quirks __prefix_line has on your system. The explicit regex above is the more reliable approach.

5b. nginx-ratelimit — catches bots that exceed your PHP rate limit

Create /etc/fail2ban/filter.d/nginx-ratelimit.conf:

[Definition]
failregex = ^<HOST> - - .* 429
ignoreregex =

This matches HTTP 429 (Too Many Requests) responses in your access log — the signal your limit_req zone emits when a client exceeds the rate limit you'll configure in Step 6.

5c. nginx-404 — catches scanners probing non-existent paths at high speed

Create /etc/fail2ban/filter.d/nginx-404.conf:

[Definition]
failregex = ^<HOST> - - .* "(GET|POST|HEAD|PUT|DELETE|OPTIONS|PATCH) [^"]*" 404
ignoreregex =

This catches any 404 response. The threshold is set high enough in the jail config (Step 6) that normal visitors with a few broken links won't trigger it — only scanners firing off dozens of unique non-existent paths in a short window.

Once all three filters are in place, reload fail2ban to pick them up:

sudo systemctl reload fail2ban
sudo fail2ban-client status