> ## Documentation Index
> Fetch the complete documentation index at: https://docs.usedatabrain.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Observability Setup

> Configure logging and OpenTelemetry for monitoring your self-hosted Databrain instance

# Observability Setup for Self-Hosted Deployments

This guide explains how to configure logging levels and enable OpenTelemetry for comprehensive observability in your self-hosted Databrain deployment.

## Log Level Configuration

Databrain supports configurable log levels to help with troubleshooting. By default, the log level is set to `info`.

### Available Log Levels

| Level   | Description                                            |
| ------- | ------------------------------------------------------ |
| `error` | Only error messages                                    |
| `warn`  | Warnings and errors                                    |
| `info`  | Informational messages, warnings, and errors (default) |
| `debug` | All messages including detailed debug information      |

### Configuration

Add to your `.env` file:

```bash theme={"dark"}
LOG_LEVEL=info
```

To enable debug logging for troubleshooting:

```bash theme={"dark"}
LOG_LEVEL=debug
```

## OpenTelemetry Integration

Databrain supports OpenTelemetry (OTel) for distributed tracing and metrics. This enables you to:

* Track API latency for all endpoints
* Monitor error rates
* See processing time breakdowns (database queries, external API calls, etc.)
* Correlate logs with traces for easier debugging

### Prerequisites

To use OpenTelemetry, you need an OTLP-compatible collector or backend. Popular options include:

* **Self-hosted**: [SigNoz](https://signoz.io/), [Jaeger](https://www.jaegertracing.io/), [Grafana Tempo](https://grafana.com/oss/tempo/)
* **Cloud**: Datadog, New Relic, Honeycomb, or any OTLP-compatible backend

### Configuration

Add these environment variables to your `.env` file:

```bash theme={"dark"}
# Enable OpenTelemetry (disabled by default)
OTEL_ENABLED=true

# Your OTLP collector endpoint
OTEL_EXPORTER_OTLP_ENDPOINT=http://your-collector:4318

# Service name for traces (optional, defaults to databrain-api)
OTEL_SERVICE_NAME=databrain-api
```

### Docker Compose Setup

If you want to run an OpenTelemetry Collector alongside Databrain, uncomment the collector service in your `docker-compose.yml`:

```yaml theme={"dark"}
otel-collector:
  image: otel/opentelemetry-collector-contrib:latest
  restart: always
  ports:
    - "4317:4317"   # OTLP gRPC receiver
    - "4318:4318"   # OTLP HTTP receiver
  volumes:
    - ./otel-config.yaml:/etc/otelcol/config.yaml
  networks:
    - datab
```

Then update your backend environment:

```bash theme={"dark"}
OTEL_ENABLED=true
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318
```

### Example: Collector Configuration for Jaeger

Create an `otel-config.yaml` file:

```yaml theme={"dark"}
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 1s
    send_batch_size: 1024

exporters:
  jaeger:
    endpoint: jaeger:14250
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [jaeger]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [jaeger]
```

## What Gets Instrumented

When OpenTelemetry is enabled, Databrain automatically instruments:

| Component      | What's Tracked                                   |
| -------------- | ------------------------------------------------ |
| HTTP Requests  | Incoming API requests with latency, status codes |
| Express Routes | Route-level timing and errors                    |
| PostgreSQL     | Database query duration and errors               |
| Redis          | Cache operations timing                          |
| Outgoing HTTP  | External API calls (Hasura, Keycloak, etc.)      |

### Example Trace

A typical API request trace shows:

```
GET /api/v2/metric/execute (1,247ms)
├── middleware.auth (12ms)
│   └── HTTP POST keycloak/auth/verify (10ms)
├── getMetricConfig (45ms)
│   └── HTTP POST hasura/v1/graphql (43ms)
├── buildQuery (8ms)
├── executeQuery (1,150ms)
│   └── PostgreSQL Query (1,148ms)
└── formatResponse (32ms)
```

## Troubleshooting

### Logs Not Appearing at Expected Level

1. Verify `LOG_LEVEL` is set correctly in your `.env`
2. Restart the backend service after changing the value
3. Log levels are case-insensitive (`DEBUG`, `debug`, `Debug` all work)

### OpenTelemetry Not Working

1. Verify `OTEL_ENABLED=true` (must be exactly `true`)
2. Check that `OTEL_EXPORTER_OTLP_ENDPOINT` is reachable from the backend container
3. Check backend logs for `[Telemetry] OpenTelemetry initialized` message
4. If you see `OTEL_ENABLED=true but OTEL_EXPORTER_OTLP_ENDPOINT not set`, configure the endpoint

### High Overhead Concerns

OpenTelemetry adds minimal overhead when enabled. If you experience performance issues:

1. Ensure your collector can handle the volume
2. Consider sampling in the collector configuration
3. Disable OTel temporarily by setting `OTEL_ENABLED=false`

## Structured Logging

Databrain uses Winston for structured, JSON-formatted logging. All logs include:

* **Timestamp**: ISO format with milliseconds
* **Level**: error, warn, info, or debug
* **Message**: Human-readable description
* **Metadata**: Structured data as key-value pairs
* **Trace Context**: When OpenTelemetry is enabled, includes `trace_id` and `span_id`

### Example Log Output

```json theme={"dark"}
{
  "level": "info",
  "message": "API request completed",
  "timestamp": "2026-02-03 10:30:45.123",
  "method": "POST",
  "url": "/api/v2/metric/execute",
  "statusCode": 200,
  "duration": 1247,
  "userId": "123",
  "trace_id": "1a2b3c4d5e6f7g8h",
  "span_id": "9i0j1k2l3m4n"
}
```

### Usage in Code

```typescript theme={"dark"}
import logger from 'utils/logger';

// Good - structured logging
logger.info('User logged in', { userId: '123', method: 'oauth' });
logger.error('Database error', { error: err.message, query: sqlQuery });

// Avoid - unstructured logging
console.log('User 123 logged in'); // ❌ Don't use console.log
```

## Default Behavior (No Configuration Required)

If you don't configure any observability settings:

* Log level defaults to `info`
* OpenTelemetry is disabled (zero overhead)
* Logs output to stdout in JSON format
* Structured logging with Winston is always enabled

This maintains backward compatibility with existing deployments.