Benchmarks & Testing

Sentinel includes a comprehensive testing framework validating performance, stability, and resilience. All results below are from actual test runs.

Test Coverage Overview

Performance Benchmarks

Native Performance (2025-12-31)

Tested on macOS ARM64, native binary (not containerized):

Sentinel is 2.5% faster than Envoy in native benchmarks.

Load Test Results

Latency Distribution

The latency distribution shows excellent tail latency characteristics:

• p50 (median): 1.1ms — half of all requests complete in just over 1 millisecond
• p95: 32.3ms — 95% of requests complete within 32ms
• p99: 61.5ms — only 1% of requests exceed 62ms

Component Latency

Core operations like rate limiting and geo filtering are sub-100μs, meaning they add negligible overhead to request processing. The agent IPC overhead (100-500μs) is the primary cost of extensibility, which is why latency-critical features like rate limiting were moved into the core.

WAF Engine Benchmarks (sentinel-modsec)

Pure Rust ModSecurity implementation with full OWASP CRS compatibility. Benchmarks run on macOS ARM64 using Criterion.

Throughput

Detection Operators

Rule Parsing

Optimized with lazy regex compilation for fast rule loading.

Rule parsing is now 3.6x faster than libmodsecurity for complex rules.

Transformations

Body Processing Throughput

Key Findings:

• Sub-microsecond transaction processing enables >900K requests/second
• XSS detection is faster than SQLi (57ns vs 123ns) due to optimized Aho-Corasick + RegexSet
• Pattern matching (@pm) uses Aho-Corasick for O(n) multi-pattern search
• Body processing scales linearly at ~40 MB/s throughput
• Pure Rust - zero C/C++ dependencies, full OWASP CRS compatibility

Rust vs C++: sentinel-modsec vs libmodsecurity

Rule Parsing Comparison

Soak Test Results

Extended duration tests validate stability and detect memory leaks or resource exhaustion.

1-Hour Soak Test (2026-01-01)

Memory Analysis

Key Findings:

• Memory decreased over time, demonstrating efficient Rust memory management
• Throughput remained stable throughout the test
• 99% of requests completed in under 62ms
• Connection errors (0.05%) occurred only during startup/shutdown phases

Leak Detection Methodology

Our analysis uses linear regression to detect memory growth patterns:

Analysis Method:

• Linear regression on memory samples over time
• R-squared correlation to detect consistent growth
• Monotonic increase ratio (>80% samples increasing = suspicious)

Test Configuration

From tests/soak/soak-config.kdl:

system {
    worker-threads 2
    max-connections 10000
    graceful-shutdown-timeout-secs 30
}

Chaos Testing

Fault injection validates resilience under failure conditions. All scenarios are in tests/chaos/.

Test Scenarios (10 Total)

Agent Failure Tests

Upstream Failure Tests

Resilience Tests

Circuit Breaker Behavior

The circuit breaker protects the system from cascading failures when agents become unhealthy. It operates as a state machine with three states:

┌─────────────────────────────────────────────────────────────────┐
│                                                                 │
│   ┌──────────┐   5 failures   ┌──────────┐   10s timeout       │
│   │  CLOSED  │ ─────────────> │   OPEN   │ ─────────────┐      │
│   │ (normal) │                │(fast-fail│               │      │
│   └────▲─────┘                └──────────┘               │      │
│        │                                                 ▼      │
│        │  2 successes         ┌───────────┐                     │
│        └───────────────────── │ HALF-OPEN │ ◄───────────┘      │
│                               │  (probe)  │                     │
│                               └───────────┘                     │
│                                     │                           │
│                                     │ failure                   │
│                                     └──────────> back to OPEN   │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘

State Descriptions:

• CLOSED (green): Normal operation. All requests pass through to the agent. Failures are counted.
• OPEN (red): Circuit is tripped. Requests are immediately failed without contacting the agent. This prevents hammering a broken service.
• HALF-OPEN (yellow): After a timeout, the circuit allows a limited number of probe requests. If they succeed, the circuit closes. If they fail, it reopens.

Configuration from chaos-config.kdl:

circuit-breaker {
    failure-threshold 5      // Opens after 5 consecutive failures
    success-threshold 2      // Closes after 2 successes in half-open
    timeout-seconds 10       // Time before transitioning to half-open
    half-open-max-requests 2 // Probe requests allowed in half-open
}

Recovery Timeline

Recovery Times

Security Validation

WAF Testing (OWASP CRS)

Validated against OWASP attack patterns with the WAF agent running OWASP Core Rule Set:

All OWASP Top 10 attack patterns are blocked with the CRS rule set.

Comparison with Envoy

Native vs Docker Performance

Root Cause Analysis:

The performance difference in Docker for Mac is due to how Docker Desktop virtualizes the Linux kernel. Pingora’s async I/O model (tokio/epoll) interacts poorly with the virtualized network stack, while Envoy’s libevent+threads model is more resilient to this overhead.

Key Finding: This is an environmental issue specific to Docker Desktop on macOS, not a code problem. On native Linux or bare metal, Sentinel matches or exceeds Envoy performance.

Integration Test Environment

Full integration tests run in Docker Compose with:

Running the Tests

Unit Tests

cargo test --workspace

Integration Tests

cd tests
./integration_test.sh           # Full suite
./integration_test.sh --quick   # Smoke tests only

Chaos Tests

cd tests/chaos
make quick                      # 4 core scenarios (~5 min)
make test                       # All 10 scenarios (~20 min)
make test-circuit-breaker       # Single scenario

Soak Tests

cd tests/soak
./run-soak-test.sh --duration 1    # 1 hour quick test
./run-soak-test.sh --duration 24   # Standard 24h test
./run-soak-test.sh --duration 72   # Extended 72h test

# Analyze results
python3 analyze-results.py results/<timestamp>/

Prometheus Metrics

Milestone Achievements

Known Gaps

We’re actively working on:

Last updated: January 2026 | Sentinel v0.2.3