Observability Setup for Self-Hosted Deployments

This guide explains how to configure logging levels and enable OpenTelemetry for comprehensive observability in your self-hosted Databrain deployment.

Log Level Configuration

Databrain supports configurable log levels to help with troubleshooting. By default, the log level is set to info.

Available Log Levels

Level	Description
`error`	Only error messages
`warn`	Warnings and errors
`info`	Informational messages, warnings, and errors (default)
`debug`	All messages including detailed debug information

Configuration

Add to your .env file:

LOG_LEVEL=info

To enable debug logging for troubleshooting:

LOG_LEVEL=debug

OpenTelemetry Integration

Databrain supports OpenTelemetry (OTel) for distributed tracing and metrics. This enables you to:

Track API latency for all endpoints
Monitor error rates
See processing time breakdowns (database queries, external API calls, etc.)
Correlate logs with traces for easier debugging

Prerequisites

To use OpenTelemetry, you need an OTLP-compatible collector or backend. Popular options include:

Self-hosted: SigNoz, Jaeger, Grafana Tempo
Cloud: Datadog, New Relic, Honeycomb, or any OTLP-compatible backend

Configuration

Add these environment variables to your .env file:

# Enable OpenTelemetry (disabled by default)
OTEL_ENABLED=true

# Your OTLP collector endpoint
OTEL_EXPORTER_OTLP_ENDPOINT=http://your-collector:4318

# Service name for traces (optional, defaults to databrain-api)
OTEL_SERVICE_NAME=databrain-api

Docker Compose Setup

If you want to run an OpenTelemetry Collector alongside Databrain, uncomment the collector service in your docker-compose.yml:

otel-collector:
  image: otel/opentelemetry-collector-contrib:latest
  restart: always
  ports:
    - "4317:4317"   # OTLP gRPC receiver
    - "4318:4318"   # OTLP HTTP receiver
  volumes:
    - ./otel-config.yaml:/etc/otelcol/config.yaml
  networks:
    - datab

Then update your backend environment:

OTEL_ENABLED=true
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318

Example: Collector Configuration for Jaeger

Create an otel-config.yaml file:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 1s
    send_batch_size: 1024

exporters:
  jaeger:
    endpoint: jaeger:14250
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [jaeger]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [jaeger]

What Gets Instrumented

When OpenTelemetry is enabled, Databrain automatically instruments:

Component	What’s Tracked
HTTP Requests	Incoming API requests with latency, status codes
Express Routes	Route-level timing and errors
PostgreSQL	Database query duration and errors
Redis	Cache operations timing
Outgoing HTTP	External API calls (Hasura, Keycloak, etc.)

Example Trace

A typical API request trace shows:

GET /api/v2/metric/execute (1,247ms)
├── middleware.auth (12ms)
│   └── HTTP POST keycloak/auth/verify (10ms)
├── getMetricConfig (45ms)
│   └── HTTP POST hasura/v1/graphql (43ms)
├── buildQuery (8ms)
├── executeQuery (1,150ms)
│   └── PostgreSQL Query (1,148ms)
└── formatResponse (32ms)

Troubleshooting

Logs Not Appearing at Expected Level

Verify LOG_LEVEL is set correctly in your .env
Restart the backend service after changing the value
Log levels are case-insensitive (DEBUG, debug, Debug all work)

OpenTelemetry Not Working

Verify OTEL_ENABLED=true (must be exactly true)
Check that OTEL_EXPORTER_OTLP_ENDPOINT is reachable from the backend container
Check backend logs for [Telemetry] OpenTelemetry initialized message
If you see OTEL_ENABLED=true but OTEL_EXPORTER_OTLP_ENDPOINT not set, configure the endpoint

High Overhead Concerns

OpenTelemetry adds minimal overhead when enabled. If you experience performance issues:

Ensure your collector can handle the volume
Consider sampling in the collector configuration
Disable OTel temporarily by setting OTEL_ENABLED=false

Structured Logging

Databrain uses Winston for structured, JSON-formatted logging. All logs include:

Timestamp: ISO format with milliseconds
Level: error, warn, info, or debug
Message: Human-readable description
Metadata: Structured data as key-value pairs
Trace Context: When OpenTelemetry is enabled, includes trace_id and span_id

Example Log Output

{
  "level": "info",
  "message": "API request completed",
  "timestamp": "2026-02-03 10:30:45.123",
  "method": "POST",
  "url": "/api/v2/metric/execute",
  "statusCode": 200,
  "duration": 1247,
  "userId": "123",
  "trace_id": "1a2b3c4d5e6f7g8h",
  "span_id": "9i0j1k2l3m4n"
}

Usage in Code

import logger from 'utils/logger';

// Good - structured logging
logger.info('User logged in', { userId: '123', method: 'oauth' });
logger.error('Database error', { error: err.message, query: sqlQuery });

// Avoid - unstructured logging
console.log('User 123 logged in'); // ❌ Don't use console.log

Default Behavior (No Configuration Required)

If you don’t configure any observability settings:

Log level defaults to info
OpenTelemetry is disabled (zero overhead)
Logs output to stdout in JSON format
Structured logging with Winston is always enabled

This maintains backward compatibility with existing deployments.

Embedding Setup

MCP Server

Framework Integration

Component Options

Authentication & Security

Filtering & Personalization

Customization

Events & Interactions

End-User Features

AI Chat Mode

Self-Hosted Deployment

Solution Patterns

Testing & Debugging

Migration & Legacy

Additional Resources

Observability Setup

Observability Setup for Self-Hosted Deployments

Log Level Configuration

Available Log Levels

Configuration

OpenTelemetry Integration

Prerequisites

Configuration

Docker Compose Setup

Example: Collector Configuration for Jaeger

What Gets Instrumented

Example Trace

Troubleshooting

Logs Not Appearing at Expected Level

OpenTelemetry Not Working

High Overhead Concerns

Structured Logging

Example Log Output

Usage in Code

Default Behavior (No Configuration Required)

​Observability Setup for Self-Hosted Deployments

​Log Level Configuration

​Available Log Levels

​Configuration

​OpenTelemetry Integration

​Prerequisites

​Configuration

​Docker Compose Setup

​Example: Collector Configuration for Jaeger

​What Gets Instrumented

​Example Trace

​Troubleshooting

​Logs Not Appearing at Expected Level

​OpenTelemetry Not Working

​High Overhead Concerns

​Structured Logging

​Example Log Output

​Usage in Code

​Default Behavior (No Configuration Required)

Observability Setup for Self-Hosted Deployments

Log Level Configuration

Available Log Levels

Configuration

OpenTelemetry Integration

Prerequisites

Configuration

Docker Compose Setup

Example: Collector Configuration for Jaeger

What Gets Instrumented

Example Trace

Troubleshooting

Logs Not Appearing at Expected Level

OpenTelemetry Not Working

High Overhead Concerns

Structured Logging

Example Log Output

Usage in Code

Default Behavior (No Configuration Required)