The problem is annoying and difficult. Also secrets can be easy to rotate, can not rotate or ones that attackers use.

You could be doing so many good data security practices, like secure-by-design frameworks, database and field-level encryption, zero-touch production, access control… but logging bypasses all of that… and ultimately degrades trust, in your systems and in your company.

It happens to companies of all sizes: X, Google Cloud, Facebook

Causes:

1. Direct logging
2. Kitchen sinks: objects that contain or hold secrets, often in opaque or unexpected ways. Errors of requests are examples.
3. Configuration changes: turning logging level to debug.
4. Embedded secrets: a token shared by URL
5. Telemetry: error monitoring and analytics are logs. They often provide the local variable context.
6. User Input: the user provides wrong but PII data in a wrong field for example.

Fixes:

1. Data architecture:part of the solution is reducing the number of data flows and shrinking the problem space so you simply have less things to worry about and protect. One logging utility!
2. Data transformation: minimization, redaction, tokenization (and the trolls: hashing, encryption, masking)
3. Domain primitives: “combines secure constructs and value objects to define the smallest building block of a domain”. new Secret("..."). They provide security invariants and guarantees that basic string primitives simply cannot.
4. Compile-time: a logging function that never accepts secrets (TS branded types helps!)
5. Run-time classes (extends String): it identifies the secrets. Overwrite the toString() method in JS to return [redacted] but an explicit unwrap() method for example.
6. Read-once objects: they throw an error or [redacted] in case of second read.
7. Taint checking: the general idea here is that you add taint to various sources (like database objects), and yell loudly if the data flows into certain sinks (like logs). Demo: https://semgrep.dev/playground/s/4bq5L It's awesome and not awesome as the same time.
8. Log formatters: redact known dangerous property names
9. Unit tests
10. Sensitive data scanner
11. Sampling (every cases instead of proportions)
12. Log pre-processors such as Vector
13. People

Strategy:

1. Lay the foundation: Developing expectations, culture, and support is a must-have. Define what a secret is. Use structured logs to allow operations on them.
2. Understand the data flow: with the foundation laid, the next best thing to do is to understand and chart out how secrets flow through your system.
3. Protect at chokepoints: CI/CD and App code first, before relying on the loggging library and other operation services.
4. Apply defense-in-depth: data transformation, read-once objects, log formatters in the library, log pre-processors, sensitive scanners, people
5. Plan for response and recovery

(via https://sebsauvage.net/links/?LscYqg)