Observability & Logging
Running a production system means nothing without visibility into what it's actually doing. I built a full observability stack using Grafana, Prometheus, Grafana Alloy, and Loki — commonly known as the PAL stack — to give me real-time metrics, centralized log aggregation, and alerting across every service in the cluster.
Why Self-Hosted Observability?
Third-party logging SaaS products charge by ingestion volume and retention, which makes them expensive and unpredictable for a self-hosted platform. More importantly, shipping application logs off-cluster means sending potentially sensitive user data to external providers. By keeping the entire observability pipeline in-cluster, I retain full control over data residency, retention policies, and query performance.
The Stack
Grafana Alloy (Collector & Shipper)
Alloy is Grafana's unified telemetry collector. It runs as a DaemonSet on every node, tailing container logs from the kubelet and forwarding them to Loki. It replaces the older Promtail agent and also handles metric scraping for services that expose Prometheus-format endpoints.
Prometheus (Metrics)
Prometheus scrapes time-series metrics from the Node Exporter (host-level CPU, memory, disk I/O), Kube-State-Metrics (pod status, deployment health, restart counts), and application-level endpoints. This gives me full visibility into both infrastructure health and application behavior.
Loki (Log Aggregation)
Loki stores and indexes container logs using a label-based approach rather than full-text indexing. This keeps storage costs dramatically lower than Elasticsearch-based stacks while still allowing fast queries by namespace, pod, container, and log level. Loki runs with a chunks cache and results cache for query performance.
Grafana (Visualization & Alerting)
Grafana ties everything together. I use it to build dashboards that correlate metrics and logs side by side — for example, viewing a spike in API response times alongside the Ktor application logs from that same time window. Grafana authenticates via OIDC against the dslan Keycloak realm, so I get SSO access without managing separate credentials.
The real power of the PAL stack is correlation. When Prometheus shows an anomaly — a pod restart loop, a memory spike, or elevated error rates — I can jump directly from the metric graph into Loki logs for that exact pod and time range, all within the same Grafana dashboard.