Passwords fail because they are guessable, reusable and phishable, and a public SSH port turns every one of those weaknesses into a standing invitation. Time-based one-time passwords close most of that gap with nothing more exotic than a phone app and a PAM module: the server issues a secret at enrollment, the app generates a fresh six-digit code every thirty seconds, and the login prompt demands both the password and the current code. This article builds the full chain on a real server: installing the module, enrolling users with google-authenticator, wiring PAM and sshd together without locking the door behind you, and building the recovery story around scratch codes so that a lost phone never becomes a lost server.
How TOTP slots into the PAM stack
The time-based one-time password algorithm, standardized in RFC 6238, derives each code from a shared secret and the current time step, which is why codes expire on a fixed thirty-second rhythm and why a correct clock is part of the security guarantee. Nothing about the algorithm is tied to any vendor: the same standard powers the popular authenticator apps across every phone platform, and the secret itself is just a base32 string that the enrollment process renders as a QR code for convenience.
On Linux the glue between login prompts and the algorithm is PAM, the Pluggable Authentication Modules framework that sshd already uses for password verification. Adding TOTP means adding one module line to the sshd PAM stack, pam_google_authenticator.so, and telling sshd to actually present the extra prompt. The module reads a small per-user file, ~/.google_authenticator, which holds the secret, options and the emergency scratch codes generated at enrollment. Because the file lives in the home directory, filesystem permissions and SELinux contexts on home paths both become part of the setup, a detail that bites RHEL administrators in particular.
The mental model worth keeping is that PAM runs modules in order and each module can be required, requisite, sufficient or optional. The TOTP module goes in as required, meaning login succeeds only when the one-time code also checks out, and it composes with the existing password check rather than replacing it. Two factors, by definition, means both are actually required.
Installing the module and enrolling the first user
On Debian-family systems the package is libpam-google-authenticator, and on RHEL-family systems it ships from EPEL under the name google-authenticator. One install command covers it everywhere:
apt install libpam-google-authenticator
The enrollment is per-user and interactive, run as the user who will log in, not as root:
$ google-authenticator
Do you want authentication tokens to be time-based? (y/n) y
The tool prints five lines that matter: the base32 secret, a QR code rendered in terminal text, one verification code and a scratch code shown as confirmation, and then it asks five configuration questions. The answers worth recommending on a server are yes to time-based tokens, yes to updating the file, yes to disallowing multiple uses of the same token, and yes to rate limiting, which makes brute-force guessing of six-digit codes pointless by throttling after a few failures. The window question, allowing codes from adjacent time steps, deserves a yes with a small tolerance: a value of 3 smooths over clock drift between the phone and the server without meaningfully weakening the scheme, and it prevents the classic support ticket of the user whose phone clock is ninety seconds ahead.
The result lands in ~/.google_authenticator: the secret on the first line, the options as keywords on subsequent lines, and the five emergency scratch codes at the end. The user scans the QR code into any TOTP-capable authenticator app, and from that moment the phone and the file share the seed. A useful operational habit is enrolling two phones per on-call role, one primary and one stored in a sealed envelope, because the alternative story is covered in the backup section below.
Wiring PAM and sshd together without locking the door
The PAM stack for sshd lives in /etc/pam.d/sshd, and the module line goes at the top of the auth section, before the common auth include, so the one-time prompt precedes the password prompt in the conversation:
auth required pam_google_authenticator.so
#%PAM-1.0
auth substack password-auth
The exact include line varies by distribution, password-auth on RHEL-family, common-auth on Debian-family, but the position of the module line stays the same: first among auth entries. The module refuses a user whose home directory has no enrollment file, which is exactly as secure and as inconvenient as it sounds. The standard compromise for a gradual rollout is the nullok option:
auth required pam_google_authenticator.so nullok
With nullok, users without an enrollment file pass through the module and log in with just their password, which converts the TOTP layer from a breaking change into an opt-in per user. The rollout then proceeds one user at a time, and once every account has enrolled, the option is removed, closing the gate behind the migration.
The sshd side needs two settings in /etc/ssh/sshd_config, and the second one is where documentation age causes real incidents. The server must enable keyboard-interactive authentication and PAM:
KbdInteractiveAuthentication yes
UsePAM yes
Older guides reference ChallengeResponseAuthentication, the name that OpenSSH used before release 8.7 renamed the directive, and on a modern sshd the old name still parses but the new one is what config generators and validators recognize. The restart that applies the change is systemctl restart sshd, and it is the exact moment the door swings shut, so the next section is best read before typing it.
The live-test protocol that prevents self-lockout
Every setup guide promises that the change is safe, and the only proof that matters is a live session that stays open while the service restarts. The protocol fits on one line: keep a root shell open, open a second connection with the new method, and only close the first when the second works. Concretely, the sequence is: enroll your own user, add the PAM line with nullok, set the sshd options, and restart sshd from a session that stays connected, because restarting the service never kills established sessions.
The verification from a separate terminal:
$ ssh admin@server
Verification code:
Password:
The Verification code prompt appearing at all is the first checkpoint, and a code from the authenticator app accepting is the second. After confirming the full two-step, the rollback story is the reverse of the rollout: removing the PAM line and restarting sshd returns the server to single-factor while the investigation happens. Testing a deliberately wrong code once, expecting rejection, completes the loop, because a TOTP prompt that accepts anything is worse than no TOTP at all.
For teams that also use public-key authentication, one architectural decision shapes everything else. By default SSH keys skip the keyboard-interactive path entirely, so a key-based login never sees the code prompt. Requiring both factors, key and code, is done in sshd_config with:
AuthenticationMethods publickey,keyboard-interactive:pam
This single line turns key logins into a two-step ceremony, and it is the setting that makes stolen keys worthless without the phone. The tradeoff is that every automated system using keys must be exempted or moved to a different path, so the setting belongs to mature setups with a real key inventory, not to the first evening of experimentation.
Scratch codes and the recovery story
A phone dies, falls into a lake or gets left at an airport, and the user's access should survive the event. The five scratch codes printed at enrollment are exactly this mechanism: single-use one-time codes that the module accepts instead of a TOTP value. Each code works once, and the honest recovery procedure is written in the runbook rather than improvised: log in with a scratch code, immediately re-enroll with google-authenticator on the server, scan the new QR code with the replacement phone, and note that the old secret is dead the moment the new enrollment overwrites the file.
Scratch codes deserve physical treatment. A code saved in the same password manager as the server password is one incident away from useless, because the attacker who phishes one phishes both. The standard practice is printing them, sealing them in an envelope, and storing them where the team stores office spare keys, with the same quarterly audit that asks whether the envelope is still sealed and whether the person who sealed it still works here.
The second half of the recovery story is administrative. Root on the console, or physical access, or the management interface of the hosting platform can always rescue a server from a bad PAM line, which is why the console path is tested once during setup instead of being assumed. A console that nobody has logged into for two years is a locked door with a handle that might not turn.
SELinux, file permissions and rate limiting details
On RHEL-family systems the module reads a file inside each user's home directory, and SELinux has opinions about services reading user files. The stock policy allows sshd to read the home directory contents in the targeted policy, but setups with encrypted homes, unusual mount points or custom contexts produce denials that look like the module failing mysteriously. The diagnosis is the standard ausearch -m AVC -ts recent from the SELinux article earlier in this series, and the fix follows the same pattern: the correct context on the file, not a permissive mode on the whole server.
Filesystem discipline is simpler but just as real. The module rejects a ~/.google_authenticator that is writable by group or others, and the enforcement is silent failure at login time, so the enrollment tool's own permission setting, mode 600, is left exactly as it lands. Equally, the file belongs in a filesystem that is not world-readable through some other path, which sounds obvious until the first setup with a shared backup user that can read all home directories.
Rate limiting deserves its explicit confirmation. Enrollment answers yes to the rate-limit question, and the module then allows three failed attempts per thirty-second window per user, which makes guessing a six-digit code statistically hopeless, one in a million per attempt, three attempts per window, and the window expiring with the code. Disabling the limit is an option that exists in the file's keywords and should be read about exactly once, to recognize it in someone else's configuration and nowhere else.
Rollout order and the production checklist
The rollout that reaches production without incidents follows an order. Install the module on the server. Enroll the administrator accounts first, test two-step login on a non-critical host or a staging box, then enroll the remaining users with a deadline in calendar form, a week is realistic, because self-enrollment takes two minutes but adoption takes reminders. Keep nullok during the enrollment window, remove it after the last holdout enrolls, and restart sshd one final time, from an open session, as always.
The checklist that closes the article is the one worth printing. PAM line in /etc/pam.d/sshd, positioned before the common auth include. KbdInteractiveAuthentication and UsePAM both yes in sshd_config. An enrollment file for every human user, mode 600, owner correct. Scratch codes printed and stored physically. Clock synchronization confirmed on the server, because TOTP with a drifting clock is a lottery. A console or management path tested once. A rollback command rehearsed once, meaning the sed that comments the PAM line, and a colleague who knows it. Done in this order, the second factor stops being a project and becomes a property of the server, quietly doing its thirty-second work on every login for years.