Editing Tornedo.tetrain.com Upgrade Plan (section)

== 10. Security incident found and remediated: distributed SSH brute-force → cascading KDC lockouts ==

While waiting for <code><nowiki>admin</nowiki></code>'s temporary lockout to clear (see below), investigated *why* it got locked in the first place — turned out to be a real, active security issue, not a fluke.

=== 10.1 What happened ===

<code><nowiki>admin</nowiki></code>'s Kerberos account hit <code><nowiki>krbPwdMaxFailure</nowiki></code> (6 failures within <code><nowiki>krbPwdFailureCountInterval</nowiki></code> = 60s, resulting in <code><nowiki>krbPwdLockoutDuration</nowiki></code> = 600s lockout). Checked <code><nowiki>krb5kdc.log</nowiki></code> for the source of the failures and found '''two separate accounts being hit''':

* <code><nowiki>arun@TORNEDO.TETRAIN.COM</nowiki></code> — repeatedly, every 10-15 seconds, from <code><nowiki>relay.isgec.com</nowiki></code> (<code><nowiki>176.58.97.66</nowiki></code>), targeting <code><nowiki>kadmin/changepw</nowiki></code>. Checked the account: <code><nowiki>nsAccountLock: TRUE</nowiki></code> (explicitly disabled) and <code><nowiki>krbPasswordExpiration: 2022-05-16</nowiki></code> — a long-dead account, not a current employee.
* <code><nowiki>admin@TORNEDO.TETRAIN.COM</nowiki></code> — hit from <code><nowiki>upyog.tetrain.com</nowiki></code> and <code><nowiki>mail.sceh.net</nowiki></code>.

Checked <code><nowiki>relay.isgec.com</nowiki></code> directly: no cron job, no local script — <code><nowiki>ss -tnp</nowiki></code> showed multiple simultaneous established SSH connections from wildly different, geographically scattered public IPs (<code><nowiki>195.9.34.73</nowiki></code>, <code><nowiki>201.98.60.254</nowiki></code>, <code><nowiki>94.23.34.95</nowiki></code>, <code><nowiki>210.61.148.174</nowiki></code>). This is the signature of a '''distributed SSH brute-force/credential-stuffing botnet''' hitting port 22, cycling through known/guessed usernames (<code><nowiki>arun</nowiki></code> from old breach data or reconnaissance, <code><nowiki>admin</nowiki></code> as an obvious guess). Because all these servers now share the same centralized IPA/Kerberos backend, failed SSH attempts on *any* of them cascade into real Kerberos account lockouts for legitimate users on the shared KDC — this is what was blocking our own <code><nowiki>ipa host-del</nowiki></code> cleanup work.

Checked <code><nowiki>relay.isgec.com</nowiki></code>'s exposure: '''no <code><nowiki>fail2ban</nowiki></code> installed at all''', and '''<code><nowiki>PasswordAuthentication yes</nowiki></code>''' — fully open to this kind of attack with zero rate-limiting.

=== 10.2 Remediation on <code><nowiki>relay.isgec.com</nowiki></code> — DONE ===

# <code><nowiki>yum install -y fail2ban</nowiki></code> failed initially — <code><nowiki>fail2ban-server</nowiki></code> hard-requires <code><nowiki>systemd-python</nowiki></code>, which wasn't installed, and installing it wanted to bump core <code><nowiki>systemd</nowiki></code> packages to a version unavailable on the regular (post-EOL, pruned) CentOS 7 mirrors — same mirror-pruning issue seen on <code><nowiki>tornedo.tetrain.com</nowiki></code> earlier.
# Repointed <code><nowiki>/etc/yum.repos.d/CentOS-Base.repo</nowiki></code> at CentOS Vault (<code><nowiki>vault.centos.org/7.9.2009/</nowiki></code>), same fix pattern as before. This box's existing <code><nowiki>CentOS-Vault.repo</nowiki></code> file only covered old point releases (7.0–7.5), not 7.9, so edited <code><nowiki>CentOS-Base.repo</nowiki></code> directly instead of using it.
# Retried — succeeded, including the <code><nowiki>systemd</nowiki></code> bump (<code><nowiki>219-62.el7</nowiki></code> → <code><nowiki>219-78.el7_9.9</nowiki></code>). '''Verified system health after''': <code><nowiki>systemctl is-system-running</nowiki></code> → <code><nowiki>running</nowiki></code>, <code><nowiki>sssd</nowiki></code>/<code><nowiki>sshd</nowiki></code> both active, uptime unchanged (no reboot triggered, live daemon-reexec handled it as expected on EL7).
# Configured <code><nowiki>/etc/fail2ban/jail.local</nowiki></code>: <code><nowiki>sshd</nowiki></code> jail enabled, <code><nowiki>maxretry=5</nowiki></code>, <code><nowiki>bantime=3600</nowiki></code>/<code><nowiki>findtime=600</nowiki></code>, '''<code><nowiki>ignoreip</nowiki></code> includes the client's VPN egress IP (<code><nowiki>45.79.120.55</nowiki></code>, also <code><nowiki>access.tetrain.com</nowiki></code>'s IP) so it's never banned'''.
# Enabled and started. '''Immediately effective''': picked up the already-ongoing attack (214 total failed attempts recorded) and banned 5 attacking IPs on the spot, including the exact ones seen live in <code><nowiki>ss -tnp</nowiki></code> earlier. Confirmed whitelist via <code><nowiki>fail2ban-client get sshd ignoreip</nowiki></code>.

=== 10.3 Still open — this is very likely fleet-wide, not just <code><nowiki>relay.isgec.com</nowiki></code> ===

The <code><nowiki>admin</nowiki></code> lockout was also triggered from <code><nowiki>upyog.tetrain.com</nowiki></code> and <code><nowiki>mail.sceh.net</nowiki></code> — strongly suggesting the same exposure (no fail2ban, password auth enabled) exists on other boxes too. '''Not yet swept.''' Recommended next steps whenever picked back up:

# Check <code><nowiki>fail2ban</nowiki></code> presence + <code><nowiki>PasswordAuthentication</nowiki></code> setting across the rest of the reachable fleet, same two checks used on <code><nowiki>relay.isgec.com</nowiki></code>.
# Install/configure fail2ban (with the same VPN whitelist) on every exposed box found.
# Longer-term: consider disabling SSH password auth fleet-wide in favor of key-only, which removes this attack surface entirely rather than just rate-limiting it.
# The disabled <code><nowiki>arun</nowiki></code> account being targeted specifically suggests old/leaked credentials are circulating — worth a broader check for any other long-disabled accounts that might be similarly targeted, and confirming they're all properly locked (not just disabled at the OS level but also <code><nowiki>nsAccountLock: TRUE</nowiki></code> in IPA, as <code><nowiki>arun</nowiki></code> correctly was).