Observability

acton-service includes an observability stack built on OpenTelemetry standards. The three pillars of observability—distributed tracing, metrics collection, and structured logging—are configured by default. Every request is automatically tracked, traced, and logged with correlation IDs for end-to-end visibility across distributed services.

Overview

The observability stack provides comprehensive visibility into your service's behavior with automatic instrumentation:

• Distributed Tracing - OpenTelemetry integration with OTLP exporter for distributed request tracing across services
• Metrics Collection - Automatic HTTP request metrics including count, duration histograms, active requests, and request/response sizes
• Structured Logging - JSON-formatted logs with correlation IDs, automatic sensitive data masking, and log level control

All observability features are enabled by default when you include the observability feature flag:

[dependencies]
acton-service = { version = "0.39.0", features = ["http", "observability"] }

Distributed Tracing

acton-service automatically instruments all HTTP requests with OpenTelemetry distributed tracing. Every request creates a trace span that propagates through your service mesh, allowing you to track requests across multiple services and understand latency bottlenecks.

Automatic OpenTelemetry Integration

The framework initializes OpenTelemetry tracing with an OTLP exporter by default. No manual instrumentation is required:

use acton_service::prelude::*;

#[tokio::main]
async fn main() -> Result<()> {
    let routes = VersionedApiBuilder::new()
        .with_base_path("/api")
        .add_version(ApiVersion::V1, |router| {
            router.route("/users", get(list_users))
        })
        .build_routes();

    // OpenTelemetry tracing automatically enabled
    ServiceBuilder::new()
        .with_routes(routes)
        .build()
        .serve()
        .await
}

Request ID Propagation

The request tracking middleware automatically generates and propagates correlation IDs across service boundaries using standard distributed tracing headers:

• x-request-id - Unique identifier for each incoming request
• x-trace-id - Trace identifier for the entire request flow across services
• x-span-id - Span identifier for the current service operation
• x-correlation-id - Correlation identifier for related requests

These headers are automatically:

• Generated if not present in incoming requests
• Propagated to downstream services
• Included in logs for correlation
• Returned in HTTP responses

// Request tracking is automatically enabled by ServiceBuilder
ServiceBuilder::new()
    .with_routes(routes)
    .build()
    .serve()
    .await?;

Configuration

Configure the OpenTelemetry OTLP endpoint via environment variables:

# OTLP endpoint (default: http://localhost:4317)
export OTEL_EXPORTER_OTLP_ENDPOINT="http://jaeger:4317"

# Service name for traces
export OTEL_SERVICE_NAME="my-service"

# Optional: OTLP protocol (grpc or http)
export OTEL_EXPORTER_OTLP_PROTOCOL="grpc"

Or in config.toml, via the [otlp] section:

[otlp]
endpoint = "http://jaeger:4317"
service_name = "my-service"
enabled = true

Metrics

acton-service automatically collects comprehensive HTTP request metrics using OpenTelemetry. Two export models are available, selected by feature flag:

• otel-metrics — push: metrics are exported over OTLP to an OpenTelemetry collector every 15 seconds
• prometheus-metrics — pull: metrics are exposed at GET /metrics in Prometheus text-exposition format for direct scraping, no collector required

The features are independent: enable either or both. With both enabled, a single meter provider feeds the OTLP exporter and the Prometheus registry simultaneously, so transport metrics and any application metrics you record through get_meter() appear in both.

Automatic HTTP Metrics

The following metrics are automatically collected for every HTTP request:

Request Count

• Total number of requests
• Labeled by HTTP method, route path, status code

Request Duration Histograms

• Request latency distribution
• Percentiles (p50, p95, p99) for latency analysis
• Labeled by HTTP method and route

100,000 requests with this distribution:
- p50 = 20ms   (median user experience - pretty good!)
- p95 = 45ms   (worst case for 95% of users - still good)
- p99 = 2000ms (worst case for 1% of users - bad!)
- Average = 40ms (hides the tail latency problem!)

Active Requests

• Current number of in-flight requests
• Useful for understanding service load

Request and Response Sizes

• Total bytes received in request bodies
• Total bytes sent in response bodies
• Helps identify bandwidth usage patterns

Metrics Middleware

Metrics collection is automatic when using the observability middleware:

// Request tracking is automatically enabled by ServiceBuilder
ServiceBuilder::new()
    .with_routes(routes)
    .build()
    .serve()
    .await?;
// OpenTelemetry metrics are automatically collected

The metrics layer is automatically applied to all routes and includes:

• Request start/end timestamps
• HTTP method and path
• Response status codes
• Request duration calculation

Prometheus Scrape Endpoint

With the prometheus-metrics feature, ServiceBuilder mounts GET /metrics alongside /health and /ready — no wiring required:

curl http://localhost:8080/metrics
# HELP http_server_request_duration_seconds Duration of HTTP server requests.
# TYPE http_server_request_duration_seconds histogram
http_server_request_duration_seconds_bucket{http_request_method="GET",http_route="/api/v1/hello",...} 2

The response uses Content-Type: text/plain; version=0.0.4 as Prometheus expects. Point a scrape job directly at the service:

# prometheus.yml
scrape_configs:
  - job_name: 'my-service'
    static_configs:
      - targets: ['my-service:8080']

Exporter listener

The route above inherits whatever your main listener is: TLS, authentication, CORS, body limits. That is correct for an application surface and wrong for a scrape target, because the collectors that matter in practice speak plain HTTP to a declared port and offer no TLS knobs at all — Fly.io's [[metrics]] block, GKE managed collection, and the defaults of most PodMonitor/ServiceMonitor resources. Terminate TLS on your main listener and none of them can scrape you.

[middleware.metrics.exporter] is the other answer: an opt-in second socket carrying the same bytes, from the same registry, through the same handler.

[middleware.metrics]
enabled = true

[middleware.metrics.exporter]
bind = "::"
port = 9090

# Plain HTTP, even when the main listener is HTTPS.
curl http://localhost:9090/metrics

Both keys are required — a plaintext socket is a security-relevant act, so there is no default address to open unasked. Every path other than GET /metrics returns 404, and the listener carries no middleware of any kind: no CORS, no compression, no tracing, no timeout, no body limit.

Absent table, absent listener. Nothing changes for a deployment that does not write it.

Fly.io — the collector scrapes your instance over the private 6PN network, so bind :::

# fly.toml
[[metrics]]
port = 9090
path = "/metrics"

Kubernetes — the container port, the Service port and the ServiceMonitor all name the exporter, not the application listener:

ports:
  - name: metrics
    containerPort: 9090
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
spec:
  endpoints:
    - port: metrics
      path: /metrics
      interval: 30s

These are refused at startup, before anything binds, rather than surfacing later as a scrape that silently never arrives:

• port = 0, which asks the OS for an ephemeral port that no scrape job can name and that changes on every restart.
• A port shared with the HTTP listener or the separate-port gRPC listener. The check is on the port alone, deliberately: :: and 0.0.0.0 overlap on the same port under Linux's dual-stack default, so a differing bind address is not a reliable escape.
• The table appearing in a build without the prometheus-metrics feature, where there is no registry to serve and every scrape would answer 503.

The exporter binds before the service's own listeners, so a port clash refuses to start rather than leaving a half-serving process; it drains after them, so the final scrape still observes the drain.

Histogram Bucket Boundaries

Declare a duration histogram with the semantic-convention unit and it is bucketed in seconds automatically:

use acton_service::observability::get_meter;

if let Some(meter) = get_meter() {
    let histogram = meter
        .f64_histogram("db.client.operation.duration")
        .with_unit("s")      // <- this is what selects the seconds boundaries
        .build();
    histogram.record(elapsed.as_secs_f64(), &[]);
}

The framework registers a metric view that selects on the instrument's unit, not its name, so any histogram following the OpenTelemetry semantic conventions gets the right boundaries without wiring anything up:

0.001, 0.0025, 0.005, 0.01, 0.025, 0.05, 0.075, 0.1,
0.25, 0.5, 0.75, 1.0, 2.5, 5.0, 7.5, 10.0

This is the semconv set for http.server.request.duration, extended downward with two sub-5ms boundaries rather than replaced, so dashboards and alerts written against the standard values still line up.

Boundaries for the HTTP request-duration histogram

The view above applies to every seconds-valued histogram except one: http.server.request.duration, which takes its boundaries from configuration instead.

[middleware.metrics]
enabled = true
# Milliseconds; the instrument is in seconds and these are converted on the way in.
latency_buckets_ms = [50.0, 150.0, 250.0, 400.0, 750.0]

The exclusion is what makes the key work at all. A view that matches an instrument does not supply a default for it — it overrules whatever that instrument asked for, because the SDK reads an instrument's own boundaries only on the branch it takes when no view matched. Omit the key and you get the same boundaries the view applies, so nothing moves until you ask it to.

Boundaries must be finite, positive and strictly increasing. Anything else is logged and ignored in favour of the instrumentation library's defaults, on the same principle as everything else here: a mis-bucketed metric must not take a service down at boot.

Structured Logging

All logs are emitted in structured JSON format with automatic field injection for correlation and debugging.

JSON Format Logging

Logs are automatically formatted as JSON with consistent fields:

{
  "timestamp": "2025-11-16T10:30:45.123Z",
  "level": "INFO",
  "target": "my_service::handlers",
  "message": "User created successfully",
  "request_id": "req_01h455vb4pex5vsknk084sn02q",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "span_id": "00f067aa0ba902b7",
  "correlation_id": "user-signup-flow"
}

Request IDs use TypeID format: req_ prefix + base32-encoded UUIDv7 for human-readable, time-sortable identifiers.

Correlation ID Propagation

Correlation IDs from the request tracking middleware are automatically included in all log entries within the request context. This allows you to filter logs by request ID and trace the entire request flow:

use tracing::{info, error};

async fn create_user(Json(user): Json<User>) -> Result<Json<User>> {
    // Request ID automatically included in logs
    info!("Creating user: {}", user.email);

    // All logs in this request context include the same request_id
    let result = db.create_user(&user).await?;

    info!("User created successfully");
    Ok(Json(result))
}

Sensitive Header Masking

The observability middleware automatically masks sensitive headers in logs to prevent credential leakage:

Automatically Masked Headers:

• Authorization
• Cookie
• Set-Cookie
• X-API-Key
• X-Auth-Token
• Any header containing "token", "secret", "password", "key" (case-insensitive)

Masked headers appear in logs as:

Authorization: [REDACTED]
X-API-Key: [REDACTED]

// These will be automatically masked:
X-API-Secret: [REDACTED]
X-Auth-Token: [REDACTED]
Custom-Password-Header: [REDACTED]

// These will appear in logs:
X-Tenant-ID: tenant-123
X-Request-Priority: high
Content-Type: application/json

Log Level Control

Control logging verbosity via environment variables:

# Set global log level
export RUST_LOG=info

# Set per-module log levels
export RUST_LOG=my_service=debug,acton_service=info,sqlx=warn

# Disable all logs except errors
export RUST_LOG=error

Or in code:

use acton_service::prelude::*;

#[tokio::main]
async fn main() -> Result<()> {
    let config = Config::load()?;

    // Initialize tracing with config
    init_tracing(&config)?;

    // Your service code...
}

Journald Integration

On Linux systems with systemd, acton-service can write tracing events directly to the journal with native structured fields instead of embedding JSON strings.

Why Native Journal Fields?

Without journald integration, logs appear as opaque JSON strings in the journal:

Mar 09 10:30:45 myhost my-service[1234]: {"timestamp":"...","level":"INFO","message":"User created"}

With native journal fields, each field is independently queryable:

journalctl -t my-service LEVEL=INFO
journalctl -t my-service F_USER_ID=123

Setup

Enable the journald feature:

[dependencies]
acton-service = { version = "0.39.0", features = ["journald", "http", "observability"] }

Configure in config.toml:

[journald]
enabled = true
syslog_identifier = "my-service"  # for journalctl -t filtering
# field_prefix = "F"              # prefix for custom fields (default)
# disable_fmt_layer = true        # suppress JSON stdout if journal captures stdout

Querying with journalctl

# Filter by service
journalctl -t my-service

# Filter by priority
journalctl -t my-service -p warning

# Filter by custom fields (prefixed with F_ by default)
journalctl -t my-service F_REQUEST_ID=req_01h455vb4pex5vsknk084sn02q

# Show all fields
journalctl -t my-service -o verbose

Suppressing Double Output

On systemd systems, stdout is typically captured by the journal. When journald integration is enabled, you may see duplicate log entries — once from the JSON fmt layer (via stdout) and once from the journald layer (via the journal socket). To prevent this, set disable_fmt_layer = true:

[journald]
enabled = true
syslog_identifier = "my-service"
disable_fmt_layer = true  # suppress JSON stdout, journal-only

Platform Compatibility

The journald feature compiles on all platforms but only activates on Linux systems with systemd. On other platforms (macOS, containers without journald), it falls back gracefully — a warning is printed and the service continues without journald output.

Configuration

Environment Variables

Configure observability behavior using environment variables:

# OpenTelemetry OTLP endpoint
export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4317"

# Service name for traces and metrics
export OTEL_SERVICE_NAME="my-service"

# Log level (trace, debug, info, warn, error)
export RUST_LOG=info

# OTLP protocol (grpc or http)
export OTEL_EXPORTER_OTLP_PROTOCOL="grpc"

Configuration File

Or configure via ~/.config/acton-service/my-service/config.toml. There is no [observability] section — OpenTelemetry export is configured under [otlp], and the log level lives under [service]:

[service]
name = "my-service"
# Log level (trace, debug, info, warn, error)
log_level = "info"

[otlp]
# OpenTelemetry OTLP endpoint (required when the section is present)
endpoint = "http://jaeger:4317"

# Service name reported on traces and metrics
# (optional — defaults to the service name when omitted)
service_name = "my-service"

# Enable OTLP export (default: true)
enabled = true

[otlp] accepts exactly these three keys: endpoint, service_name, and enabled. Omit the whole section to run without OTLP export.

Integration Examples

Jaeger (Distributed Tracing)

Run Jaeger locally for trace visualization:

# Start Jaeger with OTLP support
docker run -d --name jaeger \
  -e COLLECTOR_OTLP_ENABLED=true \
  -p 16686:16686 \
  -p 4317:4317 \
  -p 4318:4318 \
  jaegertracing/all-in-one:latest

# Configure service to export to Jaeger
export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4317"
export OTEL_SERVICE_NAME="my-service"

# Run your service
cargo run

# View traces at http://localhost:16686

Prometheus (Metrics)

The simplest integration is the prometheus-metrics feature: Prometheus scrapes the service's /metrics endpoint directly (see Prometheus Scrape Endpoint above), and no collector is involved.

Alternatively, with the otel-metrics feature, export metrics through an OpenTelemetry collector:

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

exporters:
  prometheus:
    endpoint: "0.0.0.0:8889"
    namespace: "acton_service"

service:
  pipelines:
    metrics:
      receivers: [otlp]
      exporters: [prometheus]

# Start OpenTelemetry Collector
docker run -d --name otel-collector \
  -p 4317:4317 \
  -p 8889:8889 \
  -v $(pwd)/otel-collector-config.yaml:/etc/otel-collector-config.yaml \
  otel/opentelemetry-collector-contrib:latest \
  --config=/etc/otel-collector-config.yaml

# Configure Prometheus to scrape metrics
# prometheus.yml
scrape_configs:
  - job_name: 'acton-service'
    static_configs:
      - targets: ['localhost:8889']

Grafana (Visualization)

Combine Jaeger and Prometheus in Grafana:

# Start Grafana
docker run -d --name grafana \
  -p 3000:3000 \
  grafana/grafana:latest

# Add Prometheus data source at http://localhost:3000
# Add Jaeger data source at http://localhost:16686

Complete Docker Compose Stack

# docker-compose.yml
version: '3.8'

services:
  jaeger:
    image: jaegertracing/all-in-one:latest
    environment:
      - COLLECTOR_OTLP_ENABLED=true
    ports:
      - "16686:16686"  # Jaeger UI
      - "4317:4317"    # OTLP gRPC
      - "4318:4318"    # OTLP HTTP

  otel-collector:
    image: otel/opentelemetry-collector-contrib:latest
    command: ["--config=/etc/otel-collector-config.yaml"]
    volumes:
      - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
    ports:
      - "4317:4317"    # OTLP gRPC
      - "8889:8889"    # Prometheus metrics

  prometheus:
    image: prom/prometheus:latest
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    ports:
      - "9090:9090"

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    depends_on:
      - prometheus
      - jaeger

# Start the observability stack
docker-compose up -d

# Run your service with observability
export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4317"
export OTEL_SERVICE_NAME="my-service"
cargo run

# Access dashboards:
# - Jaeger: http://localhost:16686
# - Prometheus: http://localhost:9090
# - Grafana: http://localhost:3000

Next Steps

• See Examples for complete observability setup with Jaeger and Prometheus
• Learn about Middleware customization and the request tracking middleware
• Read about Configuration for environment-based observability settings
• Explore Health Checks for monitoring service dependencies