Editing Tornedo.tetrain.com Runbook (section)

== Phase 9 — Security incident found + remediated during this work (fail2ban) ==

Not part of the migration itself, but found while investigating why <code><nowiki>admin</nowiki></code>'s Kerberos account kept hitting lockout thresholds (<code><nowiki>krbPwdMaxFailure</nowiki></code>) during cleanup work — real, active exposure.

'''Diagnosis''': <code><nowiki>krb5kdc.log</nowiki></code> showed repeated auth failures against <code><nowiki>kadmin/changepw</nowiki></code> for a long-disabled account (<code><nowiki>nsAccountLock: TRUE</nowiki></code>, expired since 2022) from <code><nowiki>relay.isgec.com</nowiki></code>, plus <code><nowiki>admin</nowiki></code> itself getting hit from two other boxes. On <code><nowiki>relay.isgec.com</nowiki></code>: <code><nowiki>ss -tnp</nowiki></code> showed multiple simultaneous SSH connections from wildly different, geographically scattered public IPs — signature of a distributed credential-stuffing botnet. That box had '''no fail2ban''' and '''<code><nowiki>PasswordAuthentication yes</nowiki></code>''' — fully exposed.

<pre>
<nowiki>
# CentOS 7, post-EOL mirrors pruned — fail2ban-server needs systemd-python, which needs a
# systemd bump unavailable on the regular (dead) mirrors:
sed -i 's|mirrorlist=|#mirrorlist=|; s|#baseurl=http://mirror.centos.org|baseurl=http://vault.centos.org/7.9.2009|' \
  /etc/yum.repos.d/CentOS-Base.repo
yum install -y fail2ban
systemctl is-system-running   # confirm "running" after the systemd bump, no reboot needed on EL7
                               # (live daemon-reexec handles it)
</nowiki>
</pre>

<code><nowiki>/etc/fail2ban/jail.local</nowiki></code>:

<pre>
<nowiki>
[sshd]
enabled = true
maxretry = 5
bantime = 3600
findtime = 600
ignoreip = 127.0.0.1/8 <your-own-VPN-egress-IP>
</nowiki>
</pre>

<pre>
<nowiki>
systemctl enable --now fail2ban
fail2ban-client status sshd            # confirm jail active
fail2ban-client get sshd ignoreip      # confirm your own IP is whitelisted BEFORE relying on this
</nowiki>
</pre>

This picked up the already-in-progress attack immediately (214 failed attempts recorded, 5 attacking IPs banned on the spot).

'''This is very likely fleet-wide, not just this one host''' — was not swept beyond this box. Before considering this closed elsewhere: check <code><nowiki>fail2ban</nowiki></code> presence + <code><nowiki>PasswordAuthentication</nowiki></code> setting on every reachable box in the fleet, apply the same fix pattern, and consider disabling SSH password auth fleet-wide (key-only) as the actual fix rather than just rate-limiting.

----