Observability and Security

Part 1: Observability

Observability is how well you can understand what a system is doing from the outside, using the data it emits. When something breaks at 3 a.m., you need answers without logging into servers. Three kinds of data work together.

Use metrics to notice, traces to locate, logs to explain.

Logs

• Write structured logs (JSON with fields) instead of free text, so they can be searched: {"level":"error","order_id":"o-17","msg":"payment timeout"}.
• Include a request id or trace id in every line, so one request can be followed across services.
• Never log passwords, tokens or card numbers.
• Logs are expensive at volume. Sample or lower the level for noisy paths.

Metrics

Watch the RED signals for each service:

• Rate: requests per second.
• Errors: share of failed requests.
• Duration: how long requests take, as percentiles.

A percentile says what share of requests were faster than a value. p99 latency of 800 ms means 99% of requests took 800 ms or less and 1% took longer. Averages hide the slow tail, so alert on p95 or p99.

Traces

A trace is the whole journey of one request. It is made of spans: one per step, with a start time and duration. A trace id is passed in an HTTP header (the W3C traceparent standard) so each service attaches its span to the same trace. OpenTelemetry is the vendor-neutral standard for producing logs, metrics and traces.

Alerts should fire on user pain (error rate, latency against the SLO) and link to a runbook. Alerts on every CPU spike train people to ignore them.