Introduction

When deploying a public MQTT backend, brute-force login attempts and unauthorized scans are inevitable. Off-the-shelf solutions like Fail2Ban are helpful, but they often fall short when dealing with sophisticated attackers who rotate IP addresses or use shared networks. This post documents how I built an intelligent firewall system for my Mosquitto MQTT server — combining journalctl, mosquitto.log, and Fail2Ban — and how a I refined the automation logic to be both accurate and safe and more effective.

Background

I run an MQTT backend (Mosquitto) on a Linux server, exposed to the public internet. Over time, I noticed patterns of unauthorized access attempts. Using fail2ban worked to an extent, but I needed something that could:

  • Detect rotating IPs.
  • Avoid banning good clients.
  • Scale without human intervention.
  • simplify complex iptables rules.

My initial implementation was based on two cron jobs:

# Parse Mosquitto logs and ban unauthorized IPs every minute
* * * * * /usr/local/bin/mqtt-ban-ips.sh

# Add rotating subnets (/24) to hosts.deny based on fail2ban logs every hour
@hourly deny-net.sh

Initial Implementation

The first script (mqtt-ban-ips.sh) parsed /var/log/mosquitto/mosquitto.log and extracted unauthorized IPs based on CONNACK codes 2 and 5, which indicate failed authentication or protocol errors. These IPs were then injected into Fail2Ban dynamically.

The second script (deny-net.sh) reviewed fail2ban.log and, for repeated offenders, added them to /etc/hosts.deny.

This setup works well but an improvement can be made. The script is relying on mosquitto logs that track IPs who try to connect and fail. If the IP happens to drop the connection after hitting the kernel iptables and before it gets to MQTT connect stage, it will not be detected.

The Refinement: Smarter Subnet Banning

Upon closer inspection, I realized banning subnets based on IP rotation was more effective as long as good IPs in the subnet were not blocked. So I needed to:

  1. Extract all IPs from journalctl entries (not from Mosquitto logs.) This will take care of those IPs that drop the connection before they hit mosquitto.
  2. Cross-reference with mosquitto.log:
    • If the IP had a successful connection: add to an allowlist.
    • If the IP had multiple CONNACK 2|5 failures: add to the banlist.
    • If multiple IPs from the same /24 subnet failed within a short window: only then ban the subnet.
    • If the IP re-appeared aggressively, and/or went undetected by mosquitto, then permanently drop it by using an iptables rule.

This logic minimizes false positives while still catching rotating IP attackers.

Final Script Overview

A combined script is being developed to:

  • Parse journalctl for PreNAT connection attempts.
  • Match each IP against mosquitto.log.
  • Maintain allow/deny IP sets.
  • For frequent subnet-wide attackers, insert /24 subnet bans into iptables.

Note: Due to brevity, the first script is included in the appendix section and works fine. It uses awk, grep, and iptables commands. You can easily integrate it with a systemd timer or cron job. The second Final script is still being developed and is not tested yet.

Deployment Steps (on a new server)

  1. Install Dependencies:

    sudo apt update
sudo apt install fail2ban iptables mosquitto

  2. Create Script Directory:

    sudo mkdir -p /usr/local/bin
sudo nano /usr/local/bin/mqtt-intelligent-firewall.sh

    (Paste final script here — see Appendix)

  3. Make It Executable:

    sudo chmod +x /usr/local/bin/mqtt-intelligent-firewall.sh

  4. Setup Cron Job:

    */5 * * * * /usr/local/bin/mqtt-intelligent-firewall.sh

Results and Impact

With this script and crons running for a few weeks:

  • Hundreds of brute-force attempts were blocked.
  • Zero legitimate clients were mistakenly banned.
  • Subnet bans only occurred when at least 3–5 unique hosts misbehaved.
  • After starting to use this method on Apr 25 the frequency of attacks was dropped significantly, see the number of mqtt log rotations below:
'linaro@dragon:/var/log/mosquitto$ ls -all'
total 352
drwxr-xr-x  2 mosquitto mosquitto   4096 Apr 25 00:11 .
drwxr-xr-x 10 root      root        4096 May 11 00:13 ..
-rw-------  1 mosquitto mosquitto 105513 May 12 13:26 mosquitto.log
-rw-------  1 mosquitto mosquitto 163988 Apr 25 00:11 mosquitto.log.1
-rw-------  1 mosquitto mosquitto  12899 Apr 24 14:56 mosquitto.log.2.gz
-rw-------  1 mosquitto mosquitto  15699 Apr 23 21:10 mosquitto.log.3.gz
-rw-------  1 mosquitto mosquitto  20277 Apr 23 00:32 mosquitto.log.4.gz
-rw-------  1 mosquitto mosquitto  15920 Apr 22 00:28 mosquitto.log.5.gz

This level of granularity turned my passive firewall into an intelligent guardian.

Conclusion

Automated firewalls are great, but they need fine-tuning. Blindly banning IPs or subnets can do more harm than good. The method presented here is simple to understand and fine tune and is free. Cross-referencing multiple logs, detecting ip-rotations and only baning /24 when statistically justified made this solution both powerful and precise.

You’re welcome to reuse and improve this setup — and if you’re a developer working on secure IoT backends, I hope this gives you a head start.

Appendix: Sample Script

You can find the full script here