SYSTEM DESIGN PRACTICE 2 min read Library →

Design a Shared Observability Platform

Principal-level prompt about collecting logs, metrics, and traces across teams while controlling cost and protecting sensitive data.

Functional requirements (FR)

  • Add functional requirements here.

Non-functional requirements (NFR)

  • Add non‑functional requirements here.

The prompt

Design a shared observability platform for a large engineering organization. Applications emit logs, metrics, and traces; teams need to investigate incidents quickly without each team operating an isolated stack. Telemetry volume can spike during incidents, and access must follow organizational boundaries.

Functional requirements

  • Collect logs, metrics, and traces from supported runtimes and infrastructure.
  • Normalize, enrich, filter, and route telemetry to storage and query systems.
  • Let teams search, chart, correlate, and alert on authorized telemetry.
  • Support service ownership, environment, region, and deployment metadata.
  • Provide retention tiers, sampling, redaction, and tenant usage views.
  • Enable incident investigation across service boundaries with auditable access.
  • Offer onboarding and service health views for teams adopting the platform.

Non-functional requirements

  • Ingestion remains available during traffic spikes and incident bursts.
  • Query performance and freshness have explicit objectives by data type and tier.
  • Ingestion failures are detectable; important telemetry is not silently discarded.
  • Tenant access, secrets, and sensitive fields are protected and auditable.
  • Cost is visible and controllable without hiding critical reliability signals.
  • The system isolates noisy tenants and supports regional failure recovery.
  • Instrumentation agents do not materially degrade application performance.

Discussion prompts

  • Which data should be sampled, aggregated, dropped, or retained during overload?
  • How do you provide a shared platform while preserving team ownership and autonomy?
  • How do you correlate traces, logs, and metrics across services and schemas?
  • How do retention tiers and query guarantees shape the cost model?
  • How can the platform reduce alert fatigue while keeping service-level objectives actionable?

results matching ""

    No results matching ""