Impact Analysis

Impact analysis compares metrics before and after an event to help you understand whether a deployment, rollback, or other change made things better or worse.

How It Works

1. Event occurs — A deployment, rollback, or other change is recorded on the timeline
2. Metrics are collected — Connected analytics providers continuously report metrics (error rates, response times, page views, etc.)
3. Before/after comparison — OpsTrails compares metric values in a configurable window before and after the event
4. Impact assessment — Significant changes are flagged, helping you correlate deployments with metric movements

Comparison Windows

When querying impact, you can specify the comparison window size:

• 1h — Tight comparison for quick-impact changes
• 2h — Default window, good for most deployments
• 4h — Wider view for gradual impacts
• 6h, 12h, 24h — Broader windows for slow-rolling changes

Worked Example: Before/After a Deployment

Here's what impact analysis looks like in practice. Suppose api-service v2.5.1 was deployed to production at 2:45 PM. OpsTrails compares metrics in a 2-hour window before and after the event:

Interpretation: The error rate and latency both jumped significantly after the deployment, while throughput remained stable. This pattern suggests the deployment introduced a bug that causes errors and slower responses, but hasn't affected overall traffic. This deployment is a strong candidate for rollback.

Which Metrics Matter Most

The most useful metrics depend on what you're investigating:

• Correctness — error_rate and crash_free_rate tell you if the change introduced bugs. Most useful metric for assessing deployment impact. Sentry provides crash_free_rate; Datadog and New Relic provide error_rate.
• Performance — p50_latency, p95_latency, and p99_latency reveal if the change slowed things down. A spike in p99 with stable p50 often indicates an edge case affecting a subset of requests.
• Availability — page_views and throughput show whether traffic is being served. A sudden drop in page views or throughput can indicate a service outage. Google Analytics provides page_views; Datadog and New Relic provide throughput.

Using Impact Analysis via MCP

When an AI assistant is connected to OpsTrails via MCP, it uses the get_metrics_around_event tool to perform impact analysis automatically. This is the same tool that powers the before/after comparisons shown above. See the MCP Tools Reference for full tool documentation.

Example AI queries that trigger impact analysis:

• “Did the error rate spike after the last deployment?”
• “How did page load times change after the v2.1.0 release?”
• “Compare metrics before and after the rollback”
• “What was the impact of the 2pm deployment on latency?”
• “Show me error rates around the last 3 deployments”

The AI will call query_events to find the relevant event, then get_metrics_around_event with the event's timestamp and subject to retrieve the before/after metric comparison.

Using with AI

When connected via MCP, AI assistants automatically use the get_metrics_around_event tool to assess impact. You can ask:

• “Did the error rate spike after the last deployment?”
• “How did page load times change after the v2.1.0 release?”
• “Compare metrics before and after the rollback”

Example Metrics

Common metrics used for impact analysis:

• error_rate — Percentage of requests resulting in errors
• p50_latency — Median response time
• p99_latency — 99th percentile response time
• page_views — Total page views from analytics
• crash_free_rate — Percentage of sessions without crashes
• throughput — Requests per second