Varnish Technology Intelligence

What is Varnish?

Varnish Cache is a high-performance HTTP accelerator designed for content-heavy, high-traffic websites. Created by Poul-Henning Kamp in 2006, Varnish sits in front of web servers, caching responses and serving them directly from memory at speeds approaching RAM bandwidth.

Varnish can handle hundreds of thousands of requests per second on modest hardware. It's used by major websites including Wikipedia, The New York Times, Twitter, and Vimeo. The cache can reduce origin server load by 90% or more for cacheable content.

Unlike other caching solutions, Varnish uses its own configuration language (VCL) that compiles to C, enabling complex caching logic with minimal overhead. Varnish is purely a cache—it doesn't handle SSL/TLS, relying on front-end proxies like nginx or Hitch.

Varnish Architecture

Memory-Based: Stores cached content in RAM. Optional disk storage for overflow. Memory-mapped files. Blazing fast retrieval.

VCL Language: Varnish Configuration Language. Compiles to native C code. Custom cache logic. Request/response manipulation.

Grace Mode: Serve stale content during backend fetch. Seamless user experience. Backend failure tolerance. Configurable staleness.

Health Checks: Backend health monitoring. Automatic failover. Director load balancing. Probe configuration.

ESI Support: Edge Side Includes for partial caching. Cache page fragments. Different TTLs per section. Personalization with caching.

Purging: Invalidate specific URLs. Ban patterns with regex. Soft purge for grace. PURGE HTTP method.

Varnish Use Cases

News & Media: High-traffic news sites. Breaking news traffic spikes. Article page caching. Homepage edge caching.

E-commerce: Product catalog caching. Category page performance. API response caching. Flash sale handling.

API Acceleration: REST API response caching. GraphQL query caching. Rate limiting. Response normalization.

Streaming: Video manifest caching. CDN origin shielding. Segment caching. Playlist acceleration.

WordPress: Full-page caching. Plugin compatibility. Logged-in user handling. Comment bypass.

High Availability: Backend failure protection. Grace mode serving. Load distribution. Traffic shaping.

Varnish Configuration

VCL Configuration Example

vcl 4.1;

backend default {
    .host = "127.0.0.1";
    .port = "8080";
    .probe = {
        .url = "/health";
        .interval = 5s;
        .timeout = 2s;
    }
}

sub vcl_recv {
    # Strip cookies for static files
    if (req.url ~ "\.(css|js|jpg|png|gif|ico|woff2)$") {
        unset req.http.Cookie;
        return (hash);
    }

    # Bypass cache for logged-in users
    if (req.http.Cookie ~ "session_id") {
        return (pass);
    }

    # Normalize Accept-Encoding
    if (req.http.Accept-Encoding) {
        if (req.http.Accept-Encoding ~ "gzip") {
            set req.http.Accept-Encoding = "gzip";
        } else {
            unset req.http.Accept-Encoding;
        }
    }
}

sub vcl_backend_response {
    # Cache static files for 1 day
    if (bereq.url ~ "\.(css|js|jpg|png)$") {
        set beresp.ttl = 1d;
        unset beresp.http.Set-Cookie;
    }

    # Enable grace mode
    set beresp.grace = 1h;
}

sub vcl_deliver {
    # Add cache hit/miss header
    if (obj.hits > 0) {
        set resp.http.X-Cache = "HIT";
    } else {
        set resp.http.X-Cache = "MISS";
    }
}

Varnish Benefits

Extreme Speed: Serves from RAM at memory speed. Microsecond response times. Handles 100K+ requests/second. Near-zero latency.

Origin Offload: 90%+ cache hit rates achievable. Reduce backend servers. Lower infrastructure costs. Handle traffic spikes.

Flexible Logic: VCL enables custom caching rules. Request manipulation. A/B testing at edge. Complex routing.

Grace Mode: Serve stale during backend issues. User never sees errors. Backend recovery time. Resilient architecture.

ESI Composition: Cache page fragments separately. Personalization with caching. Different TTLs per component. Efficient updates.

Open Source: Free core version. Active community. Enterprise version available. No vendor lock-in.

Proven Scale: Wikipedia, Twitter, NYT use Varnish. Massive traffic handling. Battle-tested reliability.