Skip to main content

​
Integrating Databrain with Datadog

This guide explains how to send OpenTelemetry traces, metrics, and logs from your self-hosted Databrain instance to Datadog.

​
Prerequisites

  • Databrain self-hosted version with OpenTelemetry support
  • Datadog account with APM enabled
  • Datadog API key

​
Configuration

​
1. Get Your Datadog API Key

  1. Log into your Datadog account
  2. Navigate to Organization Settings → API Keys
  3. Copy an existing API key or create a new one

​
2. Determine Your Datadog Site

Identify your Datadog site (region):
SiteOTLP Endpoint
US1 (default)https://api.datadoghq.com
US3https://api.us3.datadoghq.com
US5https://api.us5.datadoghq.com
EUhttps://api.datadoghq.eu
US1-FEDhttps://api.ddog-gov.com
AP1https://api.ap1.datadoghq.com

​
3. Configure Databrain Environment Variables

Add these environment variables to your Databrain backend: 
# Enable OpenTelemetry
OTEL_ENABLED=true

# Datadog OTLP endpoint (adjust for your site)
OTEL_EXPORTER_OTLP_ENDPOINT=https://api.datadoghq.com

# Service name (appears in Datadog APM)
OTEL_SERVICE_NAME=databrain-api

# Datadog API key (required for authentication)
DD_API_KEY=your_datadog_api_key_here

# Optional: Set environment tag
DD_ENV=production

# Optional: Set version tag
DD_VERSION=1.0.0

# Optional: Enable debug logging
LOG_LEVEL=info

​
4. Docker Compose Configuration

Update your docker-compose.yml: 
services:
  databrainbackend:
    environment:
      OTEL_ENABLED: "true"
      OTEL_EXPORTER_OTLP_ENDPOINT: "https://api.datadoghq.com"
      OTEL_SERVICE_NAME: "databrain-api"
      DD_API_KEY: "${DD_API_KEY}"
      DD_ENV: "production"
      DD_VERSION: "1.0.0"
      LOG_LEVEL: "info"
Security Note: Store your DD_API_KEY in a .env file, not directly in docker-compose.yml.

​
5. Kubernetes Configuration

For Kubernetes deployments: 
apiVersion: v1
kind: Secret
metadata:
  name: datadog-secret
type: Opaque
stringData:
  api-key: your_datadog_api_key_here
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: databrain-backend
spec:
  template:
    spec:
      containers:
      - name: backend
        env:
          - name: OTEL_ENABLED
            value: "true"
          - name: OTEL_EXPORTER_OTLP_ENDPOINT
            value: "https://api.datadoghq.com"
          - name: OTEL_SERVICE_NAME
            value: "databrain-api"
          - name: DD_API_KEY
            valueFrom:
              secretKeyRef:
                name: datadog-secret
                key: api-key
          - name: DD_ENV
            value: "production"

​
Alternative: Using Datadog Agent

For better performance and additional features, use the Datadog Agent as an OTLP collector:

​
1. Deploy Datadog Agent

Docker Compose: 
services:
  datadog-agent:
    image: gcr.io/datadoghq/agent:latest
    environment:
      DD_API_KEY: "${DD_API_KEY}"
      DD_SITE: "datadoghq.com"  # Adjust for your site
      DD_APM_ENABLED: "true"
      DD_APM_NON_LOCAL_TRAFFIC: "true"
      DD_OTLP_CONFIG_RECEIVER_PROTOCOLS_HTTP_ENDPOINT: "0.0.0.0:4318"
      DD_OTLP_CONFIG_RECEIVER_PROTOCOLS_GRPC_ENDPOINT: "0.0.0.0:4317"
    ports:
      - "4317:4317"  # OTLP gRPC
      - "4318:4318"  # OTLP HTTP
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /proc/:/host/proc/:ro
      - /sys/fs/cgroup/:/host/sys/fs/cgroup:ro
    networks:
      - databrain

  databrainbackend:
    environment:
      OTEL_ENABLED: "true"
      OTEL_EXPORTER_OTLP_ENDPOINT: "http://datadog-agent:4318"
      OTEL_SERVICE_NAME: "databrain-api"
Kubernetes (using Datadog Operator): 
apiVersion: datadoghq.com/v2alpha1
kind: DatadogAgent
metadata:
  name: datadog
spec:
  global:
    credentials:
      apiKey: <YOUR_API_KEY>
    site: datadoghq.com
  features:
    apm:
      enabled: true
    otlp:
      receiver:
        protocols:
          http:
            enabled: true
            endpoint: "0.0.0.0:4318"
          grpc:
            enabled: true
            endpoint: "0.0.0.0:4317"
Then configure Databrain: 
env:
  - name: OTEL_ENABLED
    value: "true"
  - name: OTEL_EXPORTER_OTLP_ENDPOINT
    value: "http://datadog-agent.datadog.svc.cluster.local:4318"
  - name: OTEL_SERVICE_NAME
    value: "databrain-api"

​
What Gets Sent to Datadog

Once configured, Databrain automatically sends:
Telemetry TypeDatadog ProductDescription
TracesAPMAPI request spans with timing, status codes, and errors
MetricsMetricsRequest latency histograms, error rates, throughput
LogsLog ManagementCorrelated logs with trace context (trace_id, span_id)

​
Verification

​
1. Restart Databrain

docker compose restart databrainbackend
# or
kubectl rollout restart deployment/databrain-backend

​
2. Generate Test Traffic

Make a few API requests to your Databrain instance: 
curl -X GET "https://your-databrain-instance.com/api/health"
curl -X POST "https://your-databrain-instance.com/api/v2/metric/execute" \
  -H "Content-Type: application/json" \
  -d '{"metricId": "test-123"}'

​
3. Check Datadog UI

  1. APM Traces:
    • Navigate to APM → Traces
    • Filter by service:databrain-api
    • You should see traces within 1-2 minutes
  2. Service Map:
    • Go to APM → Service Map
    • Look for databrain-api and its dependencies (PostgreSQL, Redis, etc.)
  3. Metrics:
    • Navigate to Metrics → Explorer
    • Search for trace.http.request.duration or http.server.duration
  4. Logs:
    • Go to Logs → Search
    • Filter by service:databrain-api
    • Click on a log entry to see correlated traces

​
Custom Tags and Attributes

Add custom tags to all telemetry: 
# Environment-specific tags
DD_TAGS="env:production,team:backend,region:us-east-1"

# In docker-compose.yml
environment:
  DD_TAGS: "env:production,team:backend,region:us-east-1"
These tags will appear in Datadog and can be used for filtering and grouping.

​
Advanced Configuration

​
Sampling Configuration

To reduce costs, configure trace sampling in the Datadog Agent: 
# datadog.yaml for Agent
apm_config:
  max_traces_per_second: 10
  trace_sample_rate: 0.5  # Sample 50% of traces

​
Resource Attributes

Databrain automatically includes these resource attributes:
  • service.name - Your service name (databrain-api)
  • service.version - Databrain version
  • deployment.environment - From DD_ENV
  • host.name - Container/pod hostname

​
Troubleshooting

IssueSolution
No data in DatadogVerify OTEL_ENABLED=true and DD_API_KEY is correct
403 ForbiddenCheck API key has APM permissions in Datadog
Connection timeoutVerify endpoint URL matches your Datadog site
Missing tracesWait 2-3 minutes; Datadog has ingestion delay
High costsImplement sampling in Datadog Agent configuration

​
Debug Mode

Enable debug logging to troubleshoot: 
LOG_LEVEL=debug
Check Databrain logs for: 
[Telemetry] OpenTelemetry initialized - service: databrain-api

​
Verify Agent Connectivity (if using Agent)

Check Datadog Agent logs: 
# Docker
docker logs datadog-agent | grep -i otlp

# Kubernetes
kubectl logs -n datadog daemonset/datadog-agent | grep -i otlp
You should see: 
OTLP HTTP receiver listening on 0.0.0.0:4318

​
Datadog Features

​
APM Dashboard

Datadog APM provides:
  • Service Overview: Latency percentiles (p50, p75, p95, p99), throughput, error rates
  • Flame Graphs: Visual representation of trace spans
  • Deployment Tracking: Compare performance before/after deployments
  • Error Tracking: Automatic error detection and grouping

​
Log Correlation

Click on any trace in Datadog APM to see:
  • All logs generated during that request
  • Database queries executed
  • External API calls made
  • Full request/response context

​
Alerts

Set up monitors in Datadog: 
Alert when:
  avg(last_5m):avg:trace.http.request.duration{service:databrain-api} > 2000

​
Pricing Considerations

Datadog pricing is based on:
  • Indexed Spans: Number of traces retained for search
  • Ingested Spans: Total spans sent (used for metrics)
  • Custom Metrics: Number of unique metric timeseries
Cost Optimization Tips:
  1. Use sampling to reduce indexed spans (keeps metrics accurate)
  2. Filter low-value traces (health checks, static assets)
  3. Use the Datadog Agent for local aggregation
  4. Set retention periods appropriately

​
Support

For Databrain configuration issues, contact your Databrain support team.