Every administrator who has distributed raw public keys across a fleet knows the arithmetic that ruins the model: keys multiply, machines forget which key belonged to whom, offboarding becomes an archaeological dig through authorized_keys files, and the one key that unlocks everything has a lifetime measured in years because nobody dares rotate it. SSH certificates turn that arithmetic inside out. The servers trust exactly one key, the authority's, and every user holds a signed credential with an identity, a validity window and permissions baked into the signature. This article builds the whole system: creating the authority, configuring servers to trust it, issuing user certificates with sane lifetimes, closing the host-key prompt problem with host certificates, and wiring revocation so that a leaked key dies at the authority rather than at a hundred machines.
Why an authority beats a thousand authorized_keys files
The raw-key model distributes trust to every host: each machine holds a list of acceptable public keys, and the security of the fleet is the union of the correctness of all those lists. The certificate model concentrates trust in one place: each server holds exactly one public key, the authority's, and accepts any credential signed by it whose identity, principals, validity window and options satisfy the policy. The server never learns individual user keys, so joining the team means getting a certificate, not getting a key onto fifty machines, and leaving the team means the certificate expiring or being revoked, not a key hunt across the fleet.
A certificate is a signed statement, and its fields are worth naming up front because the rest of the article configures each one. The key id, set with -I, names the credential for logs and humans. The principals, set with -n, state which accounts the holder may access. The validity, set with -V, bounds the credential in time. The serial, set with -z, gives the authority a numbering scheme that revocation refers to. And the extensions, set with -O, carry permissions like pty allocation or port forwarding, permissions that travel with the credential instead of living in server-side per-key stanzas. The signature binds it all, and the server verifies the whole bundle with one public key it never needs to rotate.
The authority's own key is the crown jewel of the design, and its handling deserves the discipline: generated on a dedicated, offline or heavily restricted host, never present on any server it signs for, backed up to two media that are not stored next to each other, and rotated on a calendar measured in years with a documented ceremony. Losing the authority key is a fleet-wide event, and so is its theft, which is why it lives where routine automation cannot reach it.
Creating the authority and pointing servers at it
The authority is nothing more than a key pair, generated with the same tool that makes user keys:
$ ssh-keygen -t ed25519 -f ca_user -C "user-ca-2026"
The public half, ca_user.pub, travels to every server, and the private half never does. On the server side, sshd learns to trust the authority with one line in /etc/ssh/sshd_config:
TrustedUserCAKeys /etc/ssh/user_ca_key.pub
and after a service reload, any certificate signed by that authority authenticates, subject to its own fields. The file holds one line, the authority's public key, and it is the only key material the server needs for user access, which means the authorized_keys cleanup on the fleet is not a project but a deletion.
Principals connect the certificate to real accounts, and there are two ways to wire them. The direct way relies on the certificate naming the target accounts: a certificate with principals set to alice,deploy admits its holder to the alice and deploy accounts on every trusting server, simple and rigid. The flexible way adds a per-user mapping on the server:
AuthorizedPrincipalsFile /etc/ssh/principals/%u
where the file /etc/ssh/principals/deploy contains role names, one per line, such as dev-team or ops-team, and a certificate listing the principal dev-team reaches the deploy account on exactly the servers whose mapping includes it. The mapping turns principals into roles, and roles into a permission model: the certificate says who the holder is, the server says which roles may reach this account, and access control becomes a pair of small, reviewable lists instead of a key fingerprint registry.
Issuing user certificates with sane lifetimes
Signing a user key into a certificate is one command, and its arguments are the whole policy:
$ ssh-keygen -s ca_user -I "alice-laptop-2026q4" -n dev-team -V +52w -z 1042 alice_key.pub
Signed user key alice_key-cert.pub: id "alice-laptop-2026q4" serial 1042 for dev-team valid from 2026-10-14 to 2027-10-13
The identity string names the credential in every log line, so the convention worth enforcing is user-device-period, which turns the sshd log from a fingerprint dump into a readable access record. The validity window is the policy decision that defines the culture of the system: a year, as in the example, suits a small stable team with strong handling of the authority, while shorter windows shift the burden toward automation. The serial number belongs in the authority's ledger, a flat file listing serial, identity, dates and requester, because revocation and audit both refer to it.
The client side needs nothing exotic: the certificate sits next to the private key, named alice_key-cert.pub, and ssh presents it automatically. The ~/.ssh/config gains the identity line, IdentityFile ~/.ssh/alice_key, and the companion certificate is found by name, no directives required. From the first connection onward, the server log shows the certificate id, and the access question of who connected stops being a fingerprint lookup and starts being a readable string with a date.
Extensions deserve their own habit, because the defaults are generous: a certificate without explicit options permits terminal allocation, port forwarding and agent forwarding, none of which a database access credential needs. Restricting a purpose-built certificate is one option flag:
$ ssh-keygen -s ca_user -I "backup-job" -n backup -V +1w -O clear -O permit-pty=false -O permit-port-forwarding=false backup_key.pub
The clear option wipes the default extensions, and the explicit permits re-enable only what the job uses, a pattern that makes the least-privilege principle a property of the signature rather than of server-side discipline that erodes over time.
Host certificates and the end of the trust prompt
The same mechanism, pointed the other way, solves the oldest social problem in SSH: the changed host key prompt that trains every user to type yes without reading. A second authority, the host authority, signs each server's host key, and the clients trust signatures rather than fingerprints.
On the authority host, the signing command differs by one flag:
$ ssh-keygen -s ca_host -h -I "app-01-2026" -n app-01.internal,app-01.example.com -V +104w /etc/ssh/ssh_host_ed25519_key.pub
The -h marks the operation as a host certificate, and the principals list every name by which the host is reached. The server publishes its certificate with two lines in sshd_config, HostKey staying as it was and HostCertificate pointing at the generated -cert.pub file, and after a reload, every client that trusts the host authority accepts the server without ever seeing a raw fingerprint.
The client side declares trust in the authority with one special line in known_hosts, distributed the same way the config aliases were in the bastion article:
@cert-authority .example.com,10.0.0. ssh-ed25519 AAAAC3... host-ca-2026
From that moment, a reinstalled server presents its fresh certificate and the client verifies it against the authority, silently, because the fingerprint question has been replaced by a signature question whose answer was decided once, centrally. The quarterly drill from the bastion article survives intact: an unexpected host authority change is still a signal event, but it is now one key to compare against one published value, instead of a fingerprint for every machine in the fleet.
Revocation, automation and the issuance loop
Expiration is the built-in revocation, and a validity policy means most dead credentials simply time out, but the cases that cannot wait, a lost laptop, a departure under bad terms, need active revocation that reaches the fleet instantly. SSH key revocation lists, KRLs, are the native answer: a compact binary file that names revoked certificates by serial or by authority plus key hash, checked by sshd on every authentication. Building and updating one is a two-command ceremony:
$ ssh-keygen -k -f revoked_keys -s ca_user.pub
$ ssh-keygen -k -f revoked_keys -u -s ca_user.pub -z 1042
where the second command, with -u and the serial, adds one certificate to the existing list. The server holds one line, RevokedKeys /etc/ssh/revoked_keys, and a reload picks up updates; distribution through configuration management turns the update into a fleet-wide action with a timestamp. A KRL revokes a certificate regardless of its validity window, which is exactly the property that makes the short-lifetime policy safe to relax slightly: rotation handles hygiene, revocation handles incidents.
The issuance loop is where certificates meet the working week, and the design that survives is the one where getting a credential is easier than circumventing the process. The minimal loop: a small script on the authority host takes a user's public key, checks the requester against the team roster, signs with the current quarter's identity and the standard validity, appends to the ledger, and returns the certificate. The public key arrives through any channel, it is public, and the script's real job is the bookkeeping that keeps the ledger answerable. From there the loop grows toward the organization's shape, signed requests, an approval step, an identity provider integration, but every later refinement builds on the same two-command core, and teams that start with the script have the ledger and the conventions that the fancier tools assume already exist.
Rollout order and the field checklist
The migration that ends well follows the same order as the design. Create the authorities on their dedicated host, with the ceremony and the backups the key deserves. Point one pilot server at the user authority, issue certificates to two volunteers, and work through the edge cases on a machine whose failure costs nothing: which principals reach which accounts, what the log lines look like, what breaks when the certificate expires mid-session. Widen to the fleet with configuration management, keeping raw authorized_keys in place during the transition, then remove them server by server as the logs confirm certificate-only access. Add host certificates last, because they are invisible until the known_hosts line lands, and their first visible effect, the missing prompt, is the reward the team feels rather than debugs.
The checklist fits on a card. Authority keys: dedicated host, restricted access, two backups, rotation documented. Servers: TrustedUserCAKeys, an AuthorizedPrincipalsFile mapping per role, RevokedKeys wired, raw keys removed after the transition window. Issuance: identity convention, validity policy, serial ledger, extension discipline for purpose-built credentials. Hosts: host authority, certificates on every server, @cert-authority line distributed to every client. Incidents: KRL update rehearsed once, expiry monitoring on the ledger, a quarterly review that reads the access log by certificate id. The fleet that passes that review has converted its largest unsolved security question, who has access to what, into a signature, a date and a readable line in a ledger.