Pinnacle’s Shift to Unified, Open-Source Observability

The InfraShift Strategy: The LGTM Stack on Azure

InfraShift architected a modern, open-source observability pipeline often referred to as the LGTM stack (Loki, Grafana, Tempo, Mimir/Prometheus). By decoupling the "intelligence" of the monitoring from the "storage" of the cloud provider, we gave Pinnacle total control over their data and their costs.

The Mission: Replace expensive, fragmented monitoring with a centralized, high-performance platform for logs, metrics, and traces.

The "Data Tax" of Proprietary Monitoring Pinnacle, a rapidly scaling enterprise, was facing a dual crisis with their Azure-native monitoring stack. First, the cost of log ingestion and metric storage was ballooning, becoming a significant line item in their cloud budget. Second, their data was fragmented. Engineers had to jump between multiple tools and write complex KQL (Kusto Query Language) queries just to correlate a single spike in latency with a specific log entry.

They needed a centralized, "single pane of glass" that was easy for the whole team to use - without the high premium of proprietary analytics.

The Execution: Building the Unified Observability Hub

We bypassed the high-cost ingestion points by rerouting all Azure diagnostic data through a custom-built processing engine.

1. Centralized Log Ingestion with Azure Event Hub & Loki

Instead of sending logs directly to high-cost analytics workspaces, we streamed all Azure Diagnostic Logs and metrics into Azure Event Hub.

• Loki as the Time-Series DB: We used Loki to consume these events. Because Loki only indexes metadata (labels) rather than the entire log line, it is significantly cheaper and faster than traditional indexing engines.
• No More KQL: We simplified the user experience. Engineers no longer need to be KQL experts; they simply use the Loki Explorer in Grafana to perform high-speed string searches across the entire infrastructure.

2. Full-Stack Metrics & Traces (Prometheus & Tempo)

To provide a 360-degree view, we integrated the industry standards for metrics and distributed tracing:

• Prometheus: Deployed to extract real-time performance metrics directly from the Kubernetes (AKS) clusters.
• Tempo: Integrated to track distributed traces, allowing Pinnacle to follow a single user request across multiple microservices to pinpoint exactly where a bottleneck occurs.

3. The "Red-to-Green" Dashboard Strategy

We consolidated everything into a series of highly intuitive Grafana dashboards.

• The Drill-Down Engine: We built a high-level "Executive View" where the health of every service is represented by a simple Red/Green status.
• Single-Click Debugging: When a service turns Red, an engineer can click the metric to instantly see the correlated logs in Loki and the specific traces in Tempo—all within the same window.
• Unified Alerting: We replaced fragmented alerts with a centralized Alertmanager system configured directly within Grafana.