Production secrets in Vault or cloud stores not in environment files

The production secret's natural history is a tragedy in three acts. Act one is convenience, the database password in the .env file, the API key in the config yaml, committed to the repository because the gitignore was written a week late. Act two is the quiet compounding, the secret in the git history forever, the copies on every laptop that cloned the repo, the backup archives, the CI logs, and the rotation that everyone knows is overdue and nobody schedules because nothing is broken yet. Act three is the audit or the incident, whichever arrives first, and both ask the same unanswerable question, who has seen this secret, and the honest answer is everyone who ever had the repository, plus anyone who will find the leak. This article builds the alternative properly, the secret store as the single source of truth, the injection patterns that keep secrets out of files, the rotation that becomes automated rather than overdue, and the governance that answers the auditor's question with a query rather than a shrug.

Why the environment file fails, the audit nobody can run

The .env file's failure modes deserve the honest list because the file feels so innocent. The git history is the first, the secret committed before the gitignore existed, and the removal of the file from the current tree removes nothing from the history, the secret remains in every clone forever. The sprawl is the second, the same secret copied into the CI variables, the deployment manifests, the docker compose files, the runbooks, and the rotation that must find every copy is the rotation that misses one. The access scope is the third, the file on the laptop of the engineer who left the team last quarter, the secret's blast radius identical to the repository's blast radius, which is the widest surface the organization has. The audit impossibility is the fourth and the structural one, the question who accessed this secret and when has no answer, because a file has no access log, and the compliance frameworks that demand the answer are the compliance frameworks the organization eventually meets.

The secret store's value proposition inverts every item, the secret stored once, accessed by authenticated clients, logged per access, rotated on schedule, and the store's two mature families are the practice's menu. HashiCorp Vault, the self hosted store with the rich engine catalog, the dynamic database credentials, the transit encryption, the PKI issuance, and the operational surface of a self hosted stateful service. The cloud native stores, AWS Secrets Manager and Parameter Store, the managed services with the rotation integrations and the IAM-based access, and the choice prices the same way every self hosted versus managed choice in this series prices, the operational hours against the provider's bill, with the compliance boundary as the tiebreaker.

The dynamic credential, the rotation that finally happens

The dynamic database credential is the pattern that converts rotation from the overdue meeting to an automated property, and it is the single highest value capability in the secret store catalog. The store holds the database's administrative credential, and the application's credential is generated per request, the TTL scoped lease, the username and password that exist for hours rather than quarters:

vault read database/creds/app-readonly
# Key            Value
# lease_id       database/creds/app-readonly/3xQ...
# lease_duration 1h
# username       v-root-app-readonly-7xK
# password       random-generated-32-chars

The rotation question disappears because there is nothing long lived to rotate, the credential that leaked yesterday expired by design, and the blast radius of any single credential is its lease duration. The application's side is the client library's credential renewal, the lease checked, the new credential fetched before the old expires, the database connection pool refreshed, and the pattern's operational requirement is the honest one, the database authentication that accepts the generated users, the role grants that the generated credentials inherit, and the monitoring that alerts on the lease renewal failures, which are the pattern's failure mode, the credential that the application failed to renew and the connection that died mid lease.

The static secrets that remain, the third party API keys that the vendor issues for quarters, live in the store with the rotation policy the store automates, the scheduled rotation that updates the store and notifies the consumers, and the rotation that was overdue for two years becomes a calendar entry the service runs.

The injection patterns, secrets that never touch the filesystem

The consumption side is where the practice's discipline concentrates, because the store is only as good as the path that connects it to the application, and the patterns in production converge on a few shapes. The sidecar or the init container in the Kubernetes deployment, the container that authenticates to the store, fetches the secrets, and writes them to the shared volume or the environment, the application reading the file or the variables that the sidecar populated:

spec:
  containers:
    - name: app
      env:
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef: { name: db-creds, key: password }
  initContainers:
    - name: vault-init
      # authenticates, fetches, writes the projected volume

The external secrets operator is the Kubernetes native pattern, the controller that synchronizes the store's secrets into the cluster's secret objects, the store as the source of truth and the cluster as the managed projection, and the pattern that fits the platform the earlier orchestration article priced. The agent pattern is the long running sidecar, the Vault Agent template that renders the credential file and renews it, the application reading the local file that the agent keeps fresh, and the pattern's discipline is the same everywhere, the secret transits from the store to the authenticated process without a file in the repository, a value in the pipeline log, or a variable in the developer's terminal.

The development experience is the pattern that makes the practice livable, the local developer authenticating to the store with their own scoped role, the local run fetching the development secrets the same way production does, and the .env file that the practice retired stays retired, because the local experience did not degrade to make the production posture improve.

The access model and the audit trail

The store's access model is the governance's mechanism, and the two vocabularies are the policy and the identity. The policy is the least privilege statement, the path scoped grants, the read on database/creds/app-readonly and nothing else, and the policy's review is the same code review the terraform and pipeline policies get, the grant in the repository, the change in a pull request. The identity is the authentication scope, the application's service identity, the Kubernetes service account or the cloud role, the engineer's personal identity with the time bound credential, and the access that the policy grants is the access that the identity carries, no shared passwords anywhere in the model.

The audit trail completes the governance, the store logging every access, every lease issuance, every denied request, and the auditor's question, who has seen this secret, is answered by the query, the access log filtered by the secret's path, the names, the times, the outcomes:

vault audit list
vault list sys/audit
# the audit device writing the access log to the sink

The break glass procedure is the access model's emergency valve, the procedure that grants the incident's broad access quickly and expires it automatically, the elevated role that the on call assumes with the approval trail, and the procedure that is documented, rehearsed and time bound rather than the shared admin credential that never expires, which is the alternative's quiet permanent leak.

Migrating the existing estate without the big bang

The adoption question is always the estate, the dozens of secrets already living in files and variables, and the migration that works is the strangler pattern the legacy article taught, applied to the credentials. The inventory is the first sprint, the repository scans for the known patterns, the entropy detectors that flag the high entropy strings, the team's own memory of where the secrets live, and the inventory is the map that the migration walks, one service per sprint. The per service sequence is mechanical, the secret created in the store, the application's consumption moved to the injection pattern, the file's secret removed, and the git history's residue acknowledged and rotated, because the migration that removes the file without rotating the secret it contained is the migration that moves the tragedy's act two into the store's history.

The rotation is the migration's proof, the credential that was moved and rotated is the credential that leaked to zero people, and the credential that was moved and never rotated is the credential that still lives in every clone. The estate's long tail is the last mile the migration plans for honestly, the forgotten integration's key, the service nobody dares restart, and the long tail's discipline is the deadline calendar, the date after which the file's secret stops working, because the infinite deadline is the deadline nobody meets.

One auditor's question answered from the access log

The governance story is concrete in the audit that finally runs, and the walkthrough is worth printing because it is the practice's acceptance test. The auditor's first question is the inventory, what secrets exist, and the answer is the store's catalog, the paths and the policies, the query that lists the estate in seconds. The second question is the access history, who has seen the payment provider's key, and the answer is the audit log's query, the path scoped, the identities named, the timestamps and the outcomes, the answer that the .env estate could never produce. The third question is the rotation evidence, and the answer is the lease history and the rotation log, the credential that expires hourly rather than the promise that someone changed it last summer.

The audit that runs against the store is an afternoon of queries, and the audit that runs against the file estate is the archaeology of laptops, history and memory, the same comparison the structured logging article drew between the query and the grep, and the same conclusion, the discipline that records is the discipline that survives its audits.

The working rules for the secret estate

The practice's rules compress into the list that fits the wall, and the list is short:

1. Store every production secret in the store, and the file in the repository is the bug the review rejects;
2. Prefer dynamic credentials everywhere a store engine exists, the lease is the rotation;
3. Inject through the authenticated patterns, the sidecar, the operator, the agent, and never the environment file;
4. Scope the policies to the paths per service, and review the grants as code;
5. Keep the local development on the same store with scoped roles, the degraded experience is the practice's death;
6. Rehearse the break glass quarterly, the emergency access that is not rehearsed is the credential that becomes permanent.

The secret's life without the store is the tragedy in three acts, convenience, compounding, the audit that cannot answer, and the secret's life in the store is the boring administrative record, issued, accessed, logged, expired, rotated, with the blast radius of a lease duration and the audit trail of a query. The boring version is the practice's whole promise, the secret that protects the product without becoming the product's most dangerous file, and the store, the injection pattern and the rotation policy are the machinery that makes the boring version the only version that ships.

The store is not a convenience, it is the difference between the credential as a liability and the credential as an audited resource, and the team that completes the migration owns the boring version, the estate that the auditor queries and the incident rotates without the drama that the file era made routine.