diff --git a/docs/CONFIGURATION.md b/docs/CONFIGURATION.md new file mode 100644 index 000000000..a9ca2c118 --- /dev/null +++ b/docs/CONFIGURATION.md @@ -0,0 +1,623 @@ +# Probod Configuration Reference + +This document provides a comprehensive reference for configuring the Probo compliance management platform daemon (`probod`). + +For installation instructions, please refer to the [Installation Guide](./INSTALLATION.md). + +## Configuration File Format + +Probod uses YAML format for configuration files. The configuration is structured hierarchically with the root key `probod` containing all service-specific settings. + +### Basic Configuration Structure + +```yaml +unit: + metrics: + addr: "localhost:8081" + + tracing: + addr: "localhost:4317" + max-batch-size: 1000 + batch-timeout: 10 + export-timeout: 10 + max-queue-size: 10000 + +probod: + hostname: "localhost:8080" + encryption-key: "base64-encoded-encryption-key" + chrome-dp-addr: "localhost:9222" + + api: + addr: "localhost:8080" + cors: + allowed-origins: ["http://localhost:3000"] + extra-header-fields: + "Custom-Header": "value" + + pg: + addr: "localhost:5432" + username: "probod" + password: "probod" + database: "probod" + pool-size: 100 + ca-cert-bundle: | + -----BEGIN CERTIFICATE----- + ...certificate content... + -----END CERTIFICATE----- + + auth: + disable-signup: false + invitation-confirmation-token-validity: 3600 + cookie: + name: "SSID" + domain: "localhost" + secret: "cookie-signing-secret" + duration: 24 + password: + pepper: "password-hashing-pepper" + iterations: 1000000 + + trust-auth: + cookie-name: "TCT" + cookie-domain: "localhost" + cookie-duration: 24 + token-duration: 168 + report-url-duration: 15 + token-secret: "trust-token-signing-secret" + scope: "trust_center_readonly" + token-type: "trust_center_access" + + aws: + region: "us-east-1" + bucket: "probod" + access-key-id: "access-key" + secret-access-key: "secret-key" + endpoint: "http://127.0.0.1:9000" + + mailer: + sender-name: "Probo" + sender-email: "no-reply@notification.getprobo.com" + smtp: + addr: "localhost:1025" + user: "smtp-username" + password: "smtp-password" + tls-required: true + + openai: + api-key: "openai-api-key" + temperature: 0.1 + model-name: "gpt-4o" + + connectors: + - name: "github" + type: "oauth2" + config: + client-id: "github-client-id" + client-secret: "github-client-secret" + redirect-uri: "https://localhost:8080/api/console/v1/connectors/complete" + auth-url: "https://github.com/login/oauth/authorize" + token-url: "https://github.com/login/oauth/access_token" +``` + +## Telemetry and Observability + +Probod includes built-in support for Prometheus metrics and OpenTelemetry tracing. The telemetry configuration is part of the main configuration file structure. + +### Telemetry Configuration + +The telemetry configuration is specified at the top level of the configuration file under the `unit` section: + +```yaml +unit: + metrics: + addr: "localhost:8081" + + tracing: + addr: "localhost:8082" + max-batch-size: 1000 + batch-timeout: 10 + export-timeout: 10 + max-queue-size: 10000 + +probod: + # ... rest of probod configuration +``` + +### Prometheus Metrics + +#### `unit.metrics.addr` (string) + +**Default**: Same as API address + +Network address where the Prometheus metrics endpoint will be exposed. The metrics are available at `/metrics` on this address. + +#### Example Configuration + +```yaml +unit: + metrics: + addr: "0.0.0.0:8081" # Expose metrics on port 8081 +``` + +### OpenTelemetry Tracing + +#### `unit.tracing.addr` (string) + +**Default**: Not configured (tracing disabled) + +Network address for the OpenTelemetry trace exporter endpoint. + +#### `unit.tracing.max-batch-size` (integer) + +**Default**: `512` + +Maximum number of spans to batch before exporting. + +#### `unit.tracing.batch-timeout` (integer) + +**Default**: `5` (seconds) + +Maximum time to wait before exporting a batch of spans. + +#### `unit.tracing.export-timeout` (integer) + +**Default**: `30` (seconds) + +Maximum time to wait for span export to complete. + +#### `unit.tracing.max-queue-size` (integer) + +**Default**: `2048` + +Maximum queue size for spans waiting to be exported. + +#### Example Configuration + +```yaml +unit: + tracing: + addr: "tempo:4317" # OTLP gRPC endpoint + max-batch-size: 1000 + batch-timeout: 10 + export-timeout: 10 + max-queue-size: 10000 +``` + +#### Built-in Instrumentation + +Probod automatically instruments: + +- Database operations (PostgreSQL queries) +- HTTP requests and responses +- GraphQL operations and resolvers +- S3 storage operations +- PDF generation processes + +#### Prometheus Scrape Configuration + +Configure Prometheus to scrape metrics from probod: + +```yaml +# prometheus.yml +global: + scrape_interval: 15s + +scrape_configs: + - job_name: "probod" + scrape_interval: 15s + static_configs: + - targets: ["probod:8081"] # Use metrics addr from config + metrics_path: "/metrics" +``` + +### Logging + +Probod provides automatic structured JSON logging with: + +- Request correlation IDs +- Integration with OpenTelemetry trace and span IDs +- Component-specific loggers +- Consistent formatting across all services + +## Configuration Sections + +### General Settings + +#### `hostname` (string) + +**Default**: `"localhost:8080"` + +The hostname and port where the Probod service will be accessible externally. This setting affects URL generation for redirects and API responses. + +#### `encryption-key` (string) + +**Required** + +Base64-encoded encryption key used for encrypting sensitive data at rest. Must be provided for production deployments. + +#### `chrome-dp-addr` (string) + +**Default**: `"localhost:9222"` + +Address of the Chrome DevTools Protocol endpoint used for PDF generation and document processing. + +### API Configuration + +#### `api.addr` (string) + +**Default**: `"localhost:8080"` + +The network address and port where the Probod API server will listen for incoming connections. + +#### `api.cors.allowed-origins` (array of strings) + +**Default**: `[]` + +List of origins allowed for Cross-Origin Resource Sharing (CORS) requests. Required for web applications accessing the API from different domains. + +#### `api.extra-header-fields` (map of string to string) + +**Default**: `{}` + +Additional HTTP headers to include in API responses. Useful for custom security headers or integration requirements. + +### Database Configuration + +#### `pg.addr` (string) + +**Default**: `"localhost:5432"` + +PostgreSQL database server address and port. + +#### `pg.username` (string) + +**Default**: `"probod"` + +Database username for authentication. + +#### `pg.password` (string) + +**Default**: `"probod"` + +Database password for authentication. + +#### `pg.database` (string) + +**Default**: `"probod"` + +Name of the PostgreSQL database to connect to. + +#### `pg.pool-size` (integer) + +**Default**: `100` + +Maximum number of database connections in the connection pool. + +#### `pg.ca-cert-bundle` (string) + +**Optional** + +PEM-encoded CA certificate bundle for TLS database connections. Required when connecting to databases with custom or self-signed certificates. + +### Authentication Configuration + +#### `auth.disable-signup` (boolean) + +**Default**: `false` + +When set to `true`, disables new user registration through the web interface. + +#### `auth.invitation-confirmation-token-validity` (integer) + +**Default**: `3600` + +Validity period (in seconds) for email invitation confirmation tokens. + +#### `auth.cookie.name` (string) + +**Default**: `"SSID"` + +Name of the HTTP cookie used for session management. + +#### `auth.cookie.domain` (string) + +**Default**: `"localhost"` + +Domain scope for session cookies. + +#### `auth.cookie.secret` (string) + +**Default**: Auto-generated + +Secret key used for signing session cookies. Should be at least 32 bytes for security. + +#### `auth.cookie.duration` (integer) + +**Default**: `24` + +Session cookie lifetime in hours. + +#### `auth.password.pepper` (string) + +**Default**: Auto-generated + +Additional secret value used in password hashing. Should be at least 32 bytes and kept confidential. + +#### `auth.password.iterations` (integer) + +**Default**: `1000000` + +Number of iterations for password hashing algorithm (PBKDF2). Higher values increase security but require more computational resources. + +### Trust Center Authentication + +#### `trust-auth.cookie-name` (string) + +**Default**: `"TCT"` + +Name of the HTTP cookie used for trust center access tokens. + +#### `trust-auth.cookie-domain` (string) + +**Default**: `"localhost"` + +Domain scope for trust center cookies. + +#### `trust-auth.cookie-duration` (integer) + +**Default**: `24` + +Trust center cookie lifetime in hours. + +#### `trust-auth.token-duration` (integer) + +**Default**: `168` + +Trust center access token lifetime in hours. + +#### `trust-auth.report-url-duration` (integer) + +**Default**: `15` + +Validity period for generated report URLs in minutes. + +#### `trust-auth.token-secret` (string) + +**Default**: Auto-generated + +Secret key used for signing trust center tokens. Should be at least 32 bytes. + +#### `trust-auth.scope` (string) + +**Default**: `"trust_center_readonly"` + +OAuth2 scope for trust center access. + +#### `trust-auth.token-type` (string) + +**Default**: `"trust_center_access"` + +Type identifier for trust center access tokens. + +### AWS Configuration + +#### `aws.region` (string) + +**Default**: `"us-east-1"` + +AWS region for S3 bucket operations. + +#### `aws.bucket` (string) + +**Default**: `"probod"` + +S3 bucket name for file storage. + +#### `aws.access-key-id` (string) + +**Required** + +AWS access key ID for authentication. + +#### `aws.secret-access-key` (string) + +**Required** + +AWS secret access key for authentication. + +#### `aws.endpoint` (string) + +**Optional** + +Custom S3-compatible endpoint URL. Useful for local development with MinIO or other S3-compatible services. + +### Email Configuration + +#### `mailer.sender-name` (string) + +**Default**: `"Probo"` + +Display name for outgoing emails. + +#### `mailer.sender-email` (string) + +**Default**: `"no-reply@notification.getprobo.com"` + +Email address used as the sender for outgoing emails. + +#### `mailer.smtp.addr` (string) + +**Default**: `"localhost:1025"` + +SMTP server address and port. + +#### `mailer.smtp.user` (string) + +**Optional** + +Username for SMTP authentication. + +#### `mailer.smtp.password` (string) + +**Optional** + +Password for SMTP authentication. + +#### `mailer.smtp.tls-required` (boolean) + +**Default**: `false` + +Whether TLS encryption is required for SMTP connections. + +### OpenAI Integration + +#### `openai.api-key` (string) + +**Required for AI features** + +API key for OpenAI services integration. + +#### `openai.temperature` (float) + +**Default**: `0.1` + +Temperature parameter for AI model responses (0.0 to 1.0). Lower values produce more deterministic outputs. + +#### `openai.model-name` (string) + +**Default**: `"gpt-4o"` + +OpenAI model identifier to use for AI-powered features. + +### External Connectors + +The `connectors` section defines external service integrations for data import and synchronization. + +#### OAuth2 Connector Configuration + +```yaml +connectors: + - name: "service-name" + type: "oauth2" + config: + client-id: "oauth2-client-id" + client-secret: "oauth2-client-secret" + redirect-uri: "https://your-domain/api/console/v1/connectors/complete" + auth-url: "https://service.com/oauth/authorize" + token-url: "https://service.com/oauth/token" + scopes: + - "scope1" + - "scope2" +``` + +##### `connectors[].name` (string) + +**Required** + +Unique identifier for the connector instance. + +##### `connectors[].type` (string) + +**Required** + +Connector protocol type. Currently supported: `"oauth2"`. + +##### `connectors[].config.client-id` (string) + +**Required** + +OAuth2 client identifier provided by the external service. + +##### `connectors[].config.client-secret` (string) + +**Required** + +OAuth2 client secret provided by the external service. + +##### `connectors[].config.redirect-uri` (string) + +**Required** + +OAuth2 redirect URI. Must match the URI registered with the external service. + +##### `connectors[].config.auth-url` (string) + +**Required** + +OAuth2 authorization endpoint URL. + +##### `connectors[].config.token-url` (string) + +**Required** + +OAuth2 token exchange endpoint URL. + +##### `connectors[].config.scopes` (array of strings) + +**Optional** + +List of OAuth2 scopes to request during authorization. + +## Security Considerations + +### Secrets Management + +- **Encryption Keys**: The `encryption-key` should be generated using a cryptographically secure random number generator and stored securely. +- **Database Credentials**: Use strong passwords and consider database connection encryption for production deployments. +- **Cookie Secrets**: Authentication and trust auth secrets should be unique, random, and at least 32 bytes long. +- **API Keys**: Store external service API keys securely and rotate them regularly. + +### Network Security + +- Configure appropriate firewall rules to restrict access to the Probod service. +- Use TLS/SSL termination at the load balancer or reverse proxy level. +- Implement proper CORS configuration to prevent unauthorized cross-origin requests. + +### Database Security + +- Use TLS encryption for database connections in production. +- Implement database user permissions following the principle of least privilege. +- Regular database backups and security updates are recommended. + +## Environment-Specific Configuration + +### Development + +For development environments, you can use the provided example configurations in the `cfg/` directory: + +- `cfg/dev.yaml` - Basic development configuration +- `cfg/gearnode.yaml` - Advanced development configuration with connectors + +### Production + +Production deployments should: + +1. Use strong, unique secrets for all authentication components +2. Enable TLS for all external communications +3. Use managed database services with encryption at rest +4. Implement proper monitoring and logging +5. Regular security updates and vulnerability assessments + +## Troubleshooting + +### Common Configuration Issues + +1. **Database Connection Failures**: Verify database credentials, network connectivity, and certificate configuration. +2. **Authentication Problems**: Check cookie domain settings and secret key configuration. +3. **External Connector Issues**: Verify OAuth2 client credentials and redirect URI configuration. +4. **File Upload Problems**: Ensure AWS credentials and S3 bucket configuration are correct. + +### Logging + +Probod provides structured logging that can help diagnose configuration issues. Enable debug logging by setting appropriate log levels in your deployment environment. + +## Configuration Validation + +Probod validates configuration on startup and will report specific errors for: + +- Missing required fields +- Invalid data formats +- Unreachable external services +- Invalid secrets or keys + +Review startup logs carefully to identify and resolve configuration issues. diff --git a/docs/INSTALLATION.md b/docs/INSTALLATION.md new file mode 100644 index 000000000..d143f0af1 --- /dev/null +++ b/docs/INSTALLATION.md @@ -0,0 +1,311 @@ +# Probo Installation Guide + +This document provides comprehensive installation instructions for the Probo compliance management platform daemon (`probod`). + +## Installation Methods + +Probo can be deployed using either the official Docker image or pre-compiled binaries available through GitHub releases. + +## Docker Installation + +The official Docker images are available on GitHub Container Registry and support multiple architectures: + +- **Multi-architecture image**: `ghcr.io/getprobo/probo:latest` +- **AMD64 (x86_64)**: `ghcr.io/getprobo/probo:latest-amd64` +- **ARM64**: `ghcr.io/getprobo/probo:latest-arm64` + +### Basic Docker Setup + +To run Probo using Docker: + +```bash +docker run -d \ + --name probod \ + -p 8080:8080 \ + -v /path/to/config.yaml:/etc/probod/config.yaml \ + ghcr.io/getprobo/probo:latest +``` + +### Docker Compose Setup + +For a complete setup with dependencies, you can use Docker Compose: + +```yaml +version: "3.8" + +services: + probod: + image: ghcr.io/getprobo/probo:latest + ports: + - "8080:8080" + volumes: + - ./config.yaml:/etc/probod/config.yaml + - ./data:/data + environment: + - PROBOD_CONFIG=/etc/probod/config.yaml + depends_on: + - postgres + - minio + + postgres: + image: postgres:15 + environment: + POSTGRES_DB: probod + POSTGRES_USER: probod + POSTGRES_PASSWORD: probod + volumes: + - postgres_data:/var/lib/postgresql/data + ports: + - "5432:5432" + + minio: + image: minio/minio:latest + command: server /data --console-address ":9001" + environment: + MINIO_ROOT_USER: probod + MINIO_ROOT_PASSWORD: thisisnotasecret + volumes: + - minio_data:/data + ports: + - "9000:9000" + - "9001:9001" + +volumes: + postgres_data: + minio_data: +``` + +### Docker Architecture Support + +The Docker images support the following architectures: + +- **linux/amd64** (x86_64) - Standard 64-bit Intel/AMD processors +- **linux/arm64** - ARM 64-bit processors (Apple Silicon, AWS Graviton, etc.) + +Multi-architecture images automatically select the appropriate variant for your platform. + +## Binary Installation + +Pre-compiled binaries are available for download from the [GitHub releases page](https://github.com/getprobo/probo/releases). + +### Supported Platforms + +The following platforms are officially supported: + +- **Windows**: AMD64 (x86_64) +- **macOS**: AMD64 (x86_64) and ARM64 (Apple Silicon) +- **Linux**: AMD64 (x86_64) and ARM64 (via Docker) + +If your specific platform is not available, please contact us, and we may be able to add support for additional architectures. + +### Installation Steps + +1. **Download the Binary** + + Visit the [GitHub releases page](https://github.com/getprobo/probo/releases) and download the appropriate archive for your platform: + - **Windows**: `probod_Windows_x86_64.zip` + - **macOS (Intel)**: `probod_Darwin_x86_64.tar.gz` + - **macOS (Apple Silicon)**: `probod_Darwin_arm64.tar.gz` + +2. **Extract the Archive** + + ```bash + # For tar.gz files (macOS/Linux) + tar -xzf probod_Darwin_x86_64.tar.gz + + # For zip files (Windows) + # Use your preferred extraction tool + ``` + +3. **Install the Binary** + + **macOS/Linux:** + + ```bash + # Move to a directory in your PATH + sudo mv probod /usr/local/bin/ + + # Make executable (if not already) + sudo chmod +x /usr/local/bin/probod + ``` + + **Windows:** + + ```cmd + # Move probod.exe to a directory in your PATH + # Or add the current directory to your PATH environment variable + ``` + +4. **Verify Installation** + + ```bash + probod --version + ``` + +### Running the Binary + +Once installed, you can run Probo with a configuration file: + +```bash +# Using the default configuration location +probod --config /etc/probod/config.yaml + +# Or specify a custom configuration file +probod --config ./my-config.yaml +``` + +## System Requirements + +### Minimum Requirements + +- **CPU**: 1 core, 2 GHz +- **Memory**: 1 GB RAM +- **Storage**: 10 GB available space +- **Network**: Internet connectivity for external integrations + +### Recommended Requirements + +- **CPU**: 2+ cores, 2.4 GHz +- **Memory**: 4 GB RAM +- **Storage**: 50 GB available space (SSD preferred) +- **Network**: Stable internet connection + +### Dependencies + +Probo requires the following external services: + +1. **PostgreSQL Database** (version 12 or higher) +2. **S3-Compatible Storage** (AWS S3, MinIO, etc.) +3. **Chrome/Chromium** (for PDF generation via Chrome DevTools Protocol) + +Optional dependencies: + +- **SMTP Server** (for email notifications) +- **OpenAI API** (for AI-powered features) + +## Quick Start + +### 1. Database Setup + +Create a PostgreSQL database for Probo: + +```sql +CREATE DATABASE probod; +CREATE USER probod WITH PASSWORD 'your_secure_password'; +GRANT ALL PRIVILEGES ON DATABASE probod TO probod; +``` + +### 2. Configuration + +Create a basic configuration file (`config.yaml`): + +```yaml +probod: + hostname: "localhost:8080" + encryption-key: "your-base64-encoded-encryption-key" + + pg: + addr: "localhost:5432" + username: "probod" + password: "your_secure_password" + database: "probod" + + aws: + region: "us-east-1" + bucket: "probod" + access-key-id: "your-access-key" + secret-access-key: "your-secret-key" + endpoint: "http://localhost:9000" # For MinIO +``` + +### 3. Start the Service + +```bash +# Using Docker +docker run -d \ + --name probod \ + -p 8080:8080 \ + -v ./config.yaml:/etc/probod/config.yaml \ + ghcr.io/getprobo/probo:latest + +# Using Binary +probod --config config.yaml +``` + +### 4. Access the Application + +Open your web browser and navigate to `http://localhost:8080` to access the Probo web interface. + +## Production Deployment + +### Security Considerations + +1. **Use strong, unique secrets** for all authentication components +2. **Enable TLS** for all external communications +3. **Use managed database services** with encryption at rest +4. **Implement proper monitoring** and logging +5. **Regular security updates** and vulnerability assessments + +### Load Balancing + +For high-availability deployments, consider using: + +- **Reverse Proxy**: Nginx, HAProxy, or cloud load balancers +- **Database Clustering**: PostgreSQL with read replicas +- **File Storage**: Distributed S3-compatible storage + +### Monitoring + +Probo provides metrics and health checks: + +- **Health Check**: `GET /health` +- **Metrics**: Prometheus-compatible metrics endpoint +- **Logging**: Structured JSON logging with configurable levels + +## Troubleshooting + +### Common Installation Issues + +1. **Permission Denied (Binary)** + + ```bash + chmod +x probod + ``` + +2. **Database Connection Failed** + - Verify database credentials and network connectivity + - Check PostgreSQL is running and accepting connections + +3. **Docker Image Pull Failed** + + ```bash + docker login ghcr.io + docker pull ghcr.io/getprobo/probo:latest + ``` + +4. **Port Already in Use** + + ```bash + # Find process using port 8080 + lsof -i :8080 + + # Or use a different port in configuration + ``` + +### Getting Help + +- **Documentation**: Check the [configuration reference](./CONFIGURATION.md) +- **GitHub Issues**: Report bugs and request features +- **Community**: Join our community discussions + +### Log Analysis + +Enable debug logging to troubleshoot issues: + +```yaml +probod: + # ... other configuration + log-level: debug +``` + +Check startup logs for configuration validation errors and service initialization issues. diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 000000000..d971a0cb9 --- /dev/null +++ b/docs/README.md @@ -0,0 +1,137 @@ +# Probod Operations Documentation + +This documentation provides operational guidance for deploying, configuring, and managing the Probod compliance management daemon. + +## Service Components + +Probod requires the following components for operation: + +- **probod**: Main service daemon +- **PostgreSQL**: Database backend (version 17+) +- **S3-Compatible Storage**: Object storage (AWS S3, MinIO, etc.) +- **Chrome DevTools Protocol**: PDF generation service + +## Deployment Guide + +- **[Installation Guide](./INSTALLATION.md)** - Docker and binary deployment procedures +- **[Configuration Reference](./CONFIGURATION.md)** - Complete configuration options and examples + +## Quick Deployment + +### Docker Deployment + +```bash +docker run -d \ + --name probod \ + -p 8080:8080 \ + -v /path/to/config.yaml:/etc/probod/config.yaml \ + ghcr.io/getprobo/probo:latest +``` + +### Service Verification + +```bash +# Health check +curl http://localhost:8080/health + +# Service logs +docker logs probod +``` + +## System Requirements + +### Minimum Requirements + +- **CPU**: 1 core, 2 GHz +- **Memory**: 1 GB RAM +- **Storage**: 10 GB available space +- **Database**: PostgreSQL 12+ +- **Storage Backend**: S3-compatible object storage + +### Recommended for Production + +- **CPU**: 2+ cores, 2.4 GHz +- **Memory**: 4 GB RAM +- **Storage**: 50 GB SSD +- **Database**: Managed PostgreSQL with backups +- **Load Balancer**: For high availability deployments + +## External Dependencies + +### Required Services + +- **PostgreSQL Database**: Primary data storage +- **S3-Compatible Storage**: File and document storage (AWS S3, MinIO, etc.) +- **Chrome/Chromium**: PDF generation via DevTools Protocol + +### Optional Integrations + +- **SMTP Server**: Email notifications +- **OpenAI API**: AI-powered features +- **OAuth2 Providers**: External service integrations (GitHub, Slack, etc.) + +## Production Deployment + +### High Availability + +- Deploy multiple probod instances behind a load balancer +- Use managed PostgreSQL with read replicas +- Configure distributed object storage +- Implement monitoring and alerting + +### Security Configuration + +- Enable TLS for all external communications +- Use strong, unique secrets for authentication components +- Configure network security groups and firewalls +- Regular security updates and vulnerability scanning + +## Operations + +### Monitoring + +- **Health Check**: `GET /health` +- **Metrics**: Prometheus-compatible metrics at `/metrics` endpoint +- **Tracing**: OpenTelemetry distributed tracing support +- **Logging**: Structured JSON logging with configurable levels + +#### Configuration-Based Observability + +Probod provides built-in observability configured through the main configuration file: + +- **Prometheus Metrics**: Configure `unit.metrics.addr` for metrics endpoint +- **OpenTelemetry Tracing**: Configure `unit.tracing` section for trace collection +- **Structured Logging**: Automatic JSON logging with correlation IDs + +#### Configuration Example + +```yaml +unit: + metrics: + addr: "0.0.0.0:8081" + tracing: + addr: "tempo:4317" +``` + +See [Configuration Reference](./CONFIGURATION.md#telemetry-and-observability) for complete details. + +### Backup and Recovery + +- **Database**: Regular PostgreSQL backups with point-in-time recovery +- **Object Storage**: S3 versioning and cross-region replication +- **Configuration**: Version control all configuration files + +### Troubleshooting + +Common operational issues: + +- Verify database connectivity and credentials +- Check S3 storage access permissions +- Validate configuration file syntax +- Review service logs for startup errors + +## Support + +- **Repository**: [getprobo/probo](https://github.com/getprobo/probo) +- **Issues**: Report operational problems on GitHub +- **Releases**: [GitHub releases](https://github.com/getprobo/probo/releases)