sdlc.dev

Security basics every engineer needs path · Protecting data · Topic 12 of 16

Secrets management, and what to do when one leaks

Not started

A secret is anything that grants access: API keys, database passwords, signing keys, OAuth client secrets, webhook signing secrets, private keys. Managing them well means keeping them out of code, giving each workload only the ones it needs, preferring short-lived credentials over stored ones, being able to rotate any of them quickly, and knowing exactly what to do when one escapes.

Why it matters: Leaked credentials are one of the most common ways attackers get in, and keys pushed to public repositories are found by scanners within minutes.

  1. 1Not in code, not in git

    Secrets come from a secret store or the platform at runtime. Committed secrets live on in history, forks and clones even after you delete them.

  2. 2Prefer no secret at all

    Workload identity (cloud IAM roles, OIDC federation) gives short-lived credentials without a stored key.

  3. 3Rotate on a schedule and on demand

    If rotating a secret is scary, it will not happen in an emergency. Make it routine and automated.

  4. 4Leak response starts with revoke

    Revoke or rotate first, then check logs for use, then clean up history. Deleting the commit alone fixes nothing.

1

How it works

The explanation, with examples you can run.

Where secrets should live

  • Not in source code or git. Not in .env files that get committed, not in Dockerfiles, not in client-side JavaScript or mobile apps (anything shipped to users is public).
  • In a secret store: a cloud secret manager, HashiCorp Vault, or your platform's secrets (Kubernetes Secrets with encryption at rest and tight RBAC, or synced from an external store). The app reads them at start-up or on demand.
  • Better still, nowhere: with workload identity the platform vouches for the running service (a cloud IAM role, a Kubernetes service account federated to the cloud, GitHub Actions OIDC), and the service exchanges that for short-lived credentials. Vaults can also mint dynamic secrets: a unique database user per instance that expires on its own.

Give each service its own secrets, per environment. A staging key that also works in production turns a minor leak into a major one.

Rotation

Every secret should have an owner, a way to rotate it without downtime (support two valid versions during the switch), and ideally automation that does it on a schedule. Teams that never rotate discover in the middle of an incident that nobody knows which systems use the key.

Handling secrets in code

Compare secrets, MACs and tokens in constant time. A naive comparison stops at the first differing character, so the time it takes leaks how much of a guess was right. This toy counts the steps instead of timing them:

⌘/Ctrl + Enter
OutputNot run yet

Press Run to see what your code prints, in order.

Runs in this browser, in an isolated frame with no access to this page. console.log, info, warn and error are shown; code runs as a classic script, so use an async function for await.

The naive version takes longer the more leading characters are right, which is a signal; the constant-time version always does the same work. This is only an illustration: JavaScript engines don't promise constant timing, so in real code use the platform function (crypto.timingSafeEqual in Node, hmac.compare_digest in Python, MessageDigest.isEqual in Java) rather than writing your own.

Also keep secrets out of logs, error messages, URLs (they end up in access logs and browser history) and crash reports.

Catching leaks early

Turn on secret scanning with push protection (GitHub, GitLab and others), which blocks a push containing a recognised secret, and run a scanner such as gitleaks or trufflehog in pre-commit hooks and CI.

When a secret leaks

  1. Revoke or rotate it at the source, immediately. This is the only step that removes the risk.
  2. Check for use: the provider's audit logs, unusual API calls, new resources, data access, since the earliest possible exposure.
  3. Find every place it was used and deploy the new value.
  4. Clean up: remove it from code, and rewrite history if needed (git filter-repo), though by now that is housekeeping, not protection.
  5. Learn: why was a long-lived secret there at all? Could it have been workload identity?
2

Check yourself

3 questions. Get them all right to finish this topic.

  1. 1.You pushed an API key to a public repository, noticed a minute later and force-pushed it away. Is it safe now?
  2. 2.Why compare webhook signatures with a constant-time function?
  3. 3.Which is the best way for a service on a cloud VM or Kubernetes pod to get database credentials?

0 of 3 right so far. Get all three to finish this topic.

Go deeper