Deployment

Production Checklist

New to acton-service?

Start with the homepage to understand what acton-service is, then explore Core Concepts for foundational explanations. See the Glossary for technical term definitions.

This checklist covers security, observability, performance, and operational considerations for acton-service deployments.

Configuration Review

Service Configuration

  • [ ] Set explicit service name in configuration
  • [ ] Configure appropriate port binding (default: 8080)
  • [ ] Enable TLS/HTTPS for production traffic (see TLS / HTTPS and Crypto Provider)
  • [ ] Configure certificate rotation (reload_interval_secs or reload_on_sighup) so renewed certificates don't require a restart (see TLS / HTTPS)
  • [ ] Set connection timeout values appropriate for workload
  • [ ] Configure graceful shutdown timeout
  • [ ] Review and set request body size limits
  • [ ] Enable compression for responses (gzip, br, deflate, zstd)

Environment Variables

  • [ ] Use environment-specific configuration files
  • [ ] Validate all required environment variables are set
  • [ ] Store sensitive values in secure secret management
  • [ ] Never commit secrets to version control
  • [ ] Use XDG-compliant config directories (~/.config/acton-service/)
  • [ ] Document all required configuration values

Feature Flags

  • [ ] Enable only required features to minimize dependencies
  • [ ] Review enabled middleware and their performance impact
  • [ ] Disable development-only features (verbose logging, debug endpoints)
  • [ ] Configure observability features (observability flag)

Security Hardening

Authentication & Authorization

  • [ ] Implement JWT authentication for protected endpoints
  • [ ] Use strong signing algorithms (RS256, ES256 preferred over HS256)
  • [ ] Rotate JWT signing keys regularly
  • [ ] Set appropriate token expiration times
  • [ ] Implement token revocation with Redis backend
  • [ ] Configure Cedar policy-based authorization for fine-grained access
  • [ ] Review and test all Cedar policies before deployment
  • [ ] Validate JWT claims structure (roles, permissions, user/client ID)

Network Security

  • [ ] Enable CORS with restrictive origins (not * in production)
  • [ ] Configure TLS with modern cipher suites
  • [ ] Use HTTPS-only in production
  • [ ] For FedRAMP/FISMA/DoD workloads, confirm the aws-lc-rs build is FIPS-enabled (see Crypto Provider — FIPS 140-3)
  • [ ] Implement rate limiting (per-user and per-client)
  • [ ] Set up distributed rate limiting with Redis for multi-instance deployments
  • [ ] Configure firewall rules to restrict service access

Input Validation

  • [ ] Set maximum request body sizes
  • [ ] Validate all user inputs
  • [ ] Implement request timeouts to prevent slow clients
  • [ ] Enable panic recovery middleware
  • [ ] Configure sensitive header masking in logs

Secrets Management

  • [ ] Never hardcode credentials in code or config files
  • [ ] Use environment variables or secret managers (Vault, AWS Secrets Manager)
  • [ ] Rotate database credentials regularly
  • [ ] Secure Redis connection strings
  • [ ] Protect NATS authentication tokens
  • [ ] Review file permissions on configuration files

Health Check Validation

Kubernetes Probes

  • [ ] Test /health endpoint returns 200 when service is healthy
  • [ ] Test /ready endpoint validates all dependencies
  • [ ] Configure appropriate initialDelaySeconds for startup time
  • [ ] Set periodSeconds to balance responsiveness and overhead
  • [ ] Configure failureThreshold to prevent flapping
  • [ ] Test liveness probe triggers pod restart on failure
  • [ ] Verify readiness probe removes unhealthy pods from load balancer

Dependency Health

  • [ ] Verify database connectivity in readiness check
  • [ ] Verify Redis connectivity in readiness check
  • [ ] Verify NATS connectivity in readiness check
  • [ ] Configure appropriate health check timeouts
  • [ ] Test service behavior when dependencies are unavailable

Startup and Shutdown

serve() binds every listener and then accepts until shutdown. To publish the bound address first (a supervisor, a readiness file, a test harness using port = 0), split it:

let bound = service.bind().await?;       // binds, accepts nothing yet
publish(bound.local_addr());
bound.serve().await?;

Shutdown starts on SIGINT, SIGTERM, or the future passed to ServiceBuilder::with_shutdown, and every listener drains before serve() returns. serve() also returns an error, after draining the rest, when any listener, TLS handshake pump or the metrics exporter stops on its own. Treat a non-zero exit from serve() as a crash your process manager should restart, not as a clean stop.

  • [ ] Process manager restarts the service when serve() returns an error
  • [ ] Shutdown path tested: a signal or with_shutdown drains in-flight requests
  • [ ] tls_handshake_failures_total scraped and alerted per kind

Observability Setup

Logging

  • [ ] Enable structured JSON logging for production
  • [ ] Set appropriate log level (info or warn for production)
  • [ ] Configure log aggregation (ELK, Loki, CloudWatch)
  • [ ] Verify sensitive data is masked in logs
  • [ ] Enable request ID generation and propagation
  • [ ] Configure correlation IDs for distributed tracing

Metrics

  • [ ] Enable OpenTelemetry metrics collection
  • [ ] Configure Prometheus scraping endpoint (/metrics)
  • [ ] Set up ServiceMonitor for Kubernetes deployments
  • [ ] Monitor HTTP request count and duration
  • [ ] Monitor active request count
  • [ ] Monitor request/response sizes
  • [ ] Set up alerting on key metrics (error rate, latency, throughput)

Distributed Tracing

  • [ ] Enable OpenTelemetry tracing
  • [ ] Configure trace exporter (Jaeger, Zipkin, OTLP)
  • [ ] Propagate trace headers (x-request-id, x-trace-id, x-span-id)
  • [ ] Sample traces appropriately for production load
  • [ ] Verify trace context propagation across services

Alerting

  • [ ] Set up alerts for high error rates (>1%)
  • [ ] Set up alerts for high latency (p99 > threshold)
  • [ ] Set up alerts for pod restart loops
  • [ ] Set up alerts for resource exhaustion
  • [ ] Set up alerts for circuit breaker trips
  • [ ] Configure PagerDuty/Opsgenie for critical alerts

Performance Optimization

Resource Configuration

  • [ ] Set appropriate memory requests and limits
  • [ ] Set appropriate CPU requests and limits
  • [ ] Configure connection pool sizes for database
  • [ ] Configure connection pool sizes for Redis
  • [ ] Configure NATS connection pool settings
  • [ ] Tune request timeout values

Resilience Patterns

  • [ ] Enable circuit breaker middleware
  • [ ] Configure failure rate threshold (recommended: 0.5 = 50%)
  • [ ] Enable retry logic with exponential backoff
  • [ ] Set maximum retry attempts (recommended: 3)
  • [ ] Enable bulkhead pattern for concurrency limiting
  • [ ] Set appropriate concurrent request limits
  • [ ] Test circuit breaker behavior under failure

Caching

  • [ ] Configure Redis caching for Cedar policy decisions
  • [ ] Set appropriate cache TTLs
  • [ ] Implement cache warming for critical data
  • [ ] Monitor cache hit rates
  • [ ] Configure cache eviction policies

Database Optimization

  • [ ] Enable connection pooling with appropriate size
  • [ ] Set connection timeout and idle timeout
  • [ ] Create database indexes for frequent queries
  • [ ] Test query performance under load
  • [ ] Enable prepared statement caching

Deployment Strategy

Container Configuration

  • [ ] Use multi-stage Docker builds
  • [ ] Run containers as non-root user
  • [ ] Use minimal base images (debian:bookworm-slim)
  • [ ] Install only required dependencies
  • [ ] Add Docker health checks
  • [ ] Scan images for vulnerabilities (Docker Scout, Trivy)
  • [ ] Use specific version tags (not latest)

Kubernetes Configuration

  • [ ] Deploy with 3+ replicas for high availability
  • [ ] Configure rolling update strategy
  • [ ] Set maxSurge: 1 and maxUnavailable: 0 for zero-downtime
  • [ ] Enable horizontal pod autoscaling (HPA)
  • [ ] Configure pod disruption budget (PDB)
  • [ ] Set resource requests and limits
  • [ ] Use dedicated namespaces per environment

API Versioning

  • [ ] All routes use VersionedApiBuilder
  • [ ] Version deprecated APIs with deprecation warnings
  • [ ] Document API version lifecycle
  • [ ] Plan for version sunset timeline
  • [ ] Test backward compatibility
  • [ ] Provide migration guides for version changes

Pre-Deployment Testing

Functional Testing

  • [ ] All unit tests passing
  • [ ] All integration tests passing
  • [ ] End-to-end tests covering critical paths
  • [ ] Test all API versions
  • [ ] Test health and readiness endpoints
  • [ ] Test graceful shutdown behavior

Load Testing

  • [ ] Perform load testing at expected production traffic levels
  • [ ] Test at 2x expected peak traffic
  • [ ] Verify latency under load (p50, p95, p99)
  • [ ] Test circuit breaker activation under failure
  • [ ] Test rate limiting behavior
  • [ ] Monitor resource usage under load

Security Testing

  • [ ] Run security scanner on dependencies (cargo audit)
  • [ ] Test JWT authentication with invalid tokens
  • [ ] Test authorization with unauthorized users
  • [ ] Test CORS configuration
  • [ ] Test rate limiting enforcement
  • [ ] Validate input sanitization

Monitoring & Maintenance

Operational Procedures

  • [ ] Document runbook for common issues
  • [ ] Document rollback procedures
  • [ ] Set up log aggregation and search
  • [ ] Configure automated backups for databases
  • [ ] Test disaster recovery procedures
  • [ ] Document on-call escalation process

Continuous Monitoring

  • [ ] Monitor error rates and latency trends
  • [ ] Monitor resource utilization (CPU, memory, disk)
  • [ ] Monitor database connection pool usage
  • [ ] Monitor cache hit rates
  • [ ] Track API version usage
  • [ ] Monitor circuit breaker metrics

Regular Maintenance

  • [ ] Update dependencies regularly (cargo update)
  • [ ] Scan for security vulnerabilities regularly
  • [ ] Review and rotate credentials quarterly
  • [ ] Review and update Cedar policies as needed
  • [ ] Monitor and archive old logs
  • [ ] Review and optimize database queries

Go-Live Checklist

Final checks before deploying to production:

  • [ ] All above sections reviewed and completed
  • [ ] Configuration reviewed and approved
  • [ ] Security audit completed
  • [ ] Load testing passed
  • [ ] Monitoring and alerting configured
  • [ ] Runbook documented
  • [ ] Rollback plan prepared
  • [ ] Team trained on operational procedures
  • [ ] Deploy to staging environment first
  • [ ] Verify staging deployment before production
  • [ ] Schedule deployment during low-traffic window
  • [ ] Monitor deployment closely for first 24 hours

Post-Deployment

After deploying to production:

  • [ ] Verify health endpoints return 200
  • [ ] Verify metrics are being collected
  • [ ] Verify logs are being aggregated
  • [ ] Verify traces are being exported
  • [ ] Test critical API endpoints
  • [ ] Monitor error rates for 1 hour
  • [ ] Verify autoscaling works as expected
  • [ ] Update documentation with production URLs
  • [ ] Notify stakeholders of successful deployment

Next Steps

Previous
Service Discovery
Next
Examples