Integrating Databrain with Datadog

This guide explains how to send OpenTelemetry traces, metrics, and logs from your self-hosted Databrain instance to Datadog.

Prerequisites

Databrain self-hosted version with OpenTelemetry support
Datadog account with APM enabled
Datadog API key

Configuration

1. Get Your Datadog API Key

Log into your Datadog account
Navigate to Organization Settings → API Keys
Copy an existing API key or create a new one

2. Determine Your Datadog Site

Identify your Datadog site (region):

Site	OTLP Endpoint
US1 (default)	`https://api.datadoghq.com`
US3	`https://api.us3.datadoghq.com`
US5	`https://api.us5.datadoghq.com`
EU	`https://api.datadoghq.eu`
US1-FED	`https://api.ddog-gov.com`
AP1	`https://api.ap1.datadoghq.com`

3. Configure Databrain Environment Variables

Add these environment variables to your Databrain backend:

# Enable OpenTelemetry
OTEL_ENABLED=true

# Datadog OTLP endpoint (adjust for your site)
OTEL_EXPORTER_OTLP_ENDPOINT=https://api.datadoghq.com

# Service name (appears in Datadog APM)
OTEL_SERVICE_NAME=databrain-api

# Datadog API key (required for authentication)
DD_API_KEY=your_datadog_api_key_here

# Optional: Set environment tag
DD_ENV=production

# Optional: Set version tag
DD_VERSION=1.0.0

# Optional: Enable debug logging
LOG_LEVEL=info

4. Docker Compose Configuration

Update your docker-compose.yml:

services:
  databrainbackend:
    environment:
      OTEL_ENABLED: "true"
      OTEL_EXPORTER_OTLP_ENDPOINT: "https://api.datadoghq.com"
      OTEL_SERVICE_NAME: "databrain-api"
      DD_API_KEY: "${DD_API_KEY}"
      DD_ENV: "production"
      DD_VERSION: "1.0.0"
      LOG_LEVEL: "info"

Security Note: Store your DD_API_KEY in a .env file, not directly in docker-compose.yml.

5. Kubernetes Configuration

For Kubernetes deployments:

apiVersion: v1
kind: Secret
metadata:
  name: datadog-secret
type: Opaque
stringData:
  api-key: your_datadog_api_key_here
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: databrain-backend
spec:
  template:
    spec:
      containers:
      - name: backend
        env:
          - name: OTEL_ENABLED
            value: "true"
          - name: OTEL_EXPORTER_OTLP_ENDPOINT
            value: "https://api.datadoghq.com"
          - name: OTEL_SERVICE_NAME
            value: "databrain-api"
          - name: DD_API_KEY
            valueFrom:
              secretKeyRef:
                name: datadog-secret
                key: api-key
          - name: DD_ENV
            value: "production"

Alternative: Using Datadog Agent

For better performance and additional features, use the Datadog Agent as an OTLP collector:

1. Deploy Datadog Agent

Docker Compose:

services:
  datadog-agent:
    image: gcr.io/datadoghq/agent:latest
    environment:
      DD_API_KEY: "${DD_API_KEY}"
      DD_SITE: "datadoghq.com"  # Adjust for your site
      DD_APM_ENABLED: "true"
      DD_APM_NON_LOCAL_TRAFFIC: "true"
      DD_OTLP_CONFIG_RECEIVER_PROTOCOLS_HTTP_ENDPOINT: "0.0.0.0:4318"
      DD_OTLP_CONFIG_RECEIVER_PROTOCOLS_GRPC_ENDPOINT: "0.0.0.0:4317"
    ports:
      - "4317:4317"  # OTLP gRPC
      - "4318:4318"  # OTLP HTTP
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /proc/:/host/proc/:ro
      - /sys/fs/cgroup/:/host/sys/fs/cgroup:ro
    networks:
      - databrain

  databrainbackend:
    environment:
      OTEL_ENABLED: "true"
      OTEL_EXPORTER_OTLP_ENDPOINT: "http://datadog-agent:4318"
      OTEL_SERVICE_NAME: "databrain-api"

Kubernetes (using Datadog Operator):

apiVersion: datadoghq.com/v2alpha1
kind: DatadogAgent
metadata:
  name: datadog
spec:
  global:
    credentials:
      apiKey: <YOUR_API_KEY>
    site: datadoghq.com
  features:
    apm:
      enabled: true
    otlp:
      receiver:
        protocols:
          http:
            enabled: true
            endpoint: "0.0.0.0:4318"
          grpc:
            enabled: true
            endpoint: "0.0.0.0:4317"

Then configure Databrain:

env:
  - name: OTEL_ENABLED
    value: "true"
  - name: OTEL_EXPORTER_OTLP_ENDPOINT
    value: "http://datadog-agent.datadog.svc.cluster.local:4318"
  - name: OTEL_SERVICE_NAME
    value: "databrain-api"

What Gets Sent to Datadog

Once configured, Databrain automatically sends:

Telemetry Type	Datadog Product	Description
Traces	APM	API request spans with timing, status codes, and errors
Metrics	Metrics	Request latency histograms, error rates, throughput
Logs	Log Management	Correlated logs with trace context (trace_id, span_id)

Verification

1. Restart Databrain

docker compose restart databrainbackend
# or
kubectl rollout restart deployment/databrain-backend

2. Generate Test Traffic

Make a few API requests to your Databrain instance:

curl -X GET "https://your-databrain-instance.com/api/health"
curl -X POST "https://your-databrain-instance.com/api/v2/metric/execute" \
  -H "Content-Type: application/json" \
  -d '{"metricId": "test-123"}'

3. Check Datadog UI

APM Traces:
- Navigate to APM → Traces
- Filter by service:databrain-api
- You should see traces within 1-2 minutes
Service Map:
- Go to APM → Service Map
- Look for databrain-api and its dependencies (PostgreSQL, Redis, etc.)
Metrics:
- Navigate to Metrics → Explorer
- Search for trace.http.request.duration or http.server.duration
Logs:
- Go to Logs → Search
- Filter by service:databrain-api
- Click on a log entry to see correlated traces

Custom Tags and Attributes

Add custom tags to all telemetry:

# Environment-specific tags
DD_TAGS="env:production,team:backend,region:us-east-1"

# In docker-compose.yml
environment:
  DD_TAGS: "env:production,team:backend,region:us-east-1"

These tags will appear in Datadog and can be used for filtering and grouping.

Advanced Configuration

Sampling Configuration

To reduce costs, configure trace sampling in the Datadog Agent:

# datadog.yaml for Agent
apm_config:
  max_traces_per_second: 10
  trace_sample_rate: 0.5  # Sample 50% of traces

Resource Attributes

Databrain automatically includes these resource attributes:

service.name - Your service name (databrain-api)
service.version - Databrain version
deployment.environment - From DD_ENV
host.name - Container/pod hostname

Troubleshooting

Issue	Solution
No data in Datadog	Verify `OTEL_ENABLED=true` and DD_API_KEY is correct
403 Forbidden	Check API key has APM permissions in Datadog
Connection timeout	Verify endpoint URL matches your Datadog site
Missing traces	Wait 2-3 minutes; Datadog has ingestion delay
High costs	Implement sampling in Datadog Agent configuration

Debug Mode

Enable debug logging to troubleshoot:

LOG_LEVEL=debug

Check Databrain logs for:

[Telemetry] OpenTelemetry initialized - service: databrain-api

Verify Agent Connectivity (if using Agent)

Check Datadog Agent logs:

# Docker
docker logs datadog-agent | grep -i otlp

# Kubernetes
kubectl logs -n datadog daemonset/datadog-agent | grep -i otlp

You should see:

OTLP HTTP receiver listening on 0.0.0.0:4318

Datadog Features

APM Dashboard

Datadog APM provides:

Service Overview: Latency percentiles (p50, p75, p95, p99), throughput, error rates
Flame Graphs: Visual representation of trace spans
Deployment Tracking: Compare performance before/after deployments
Error Tracking: Automatic error detection and grouping

Log Correlation

Click on any trace in Datadog APM to see:

All logs generated during that request
Database queries executed
External API calls made
Full request/response context

Alerts

Set up monitors in Datadog:

Alert when:
  avg(last_5m):avg:trace.http.request.duration{service:databrain-api} > 2000

Pricing Considerations

Datadog pricing is based on:

Indexed Spans: Number of traces retained for search
Ingested Spans: Total spans sent (used for metrics)
Custom Metrics: Number of unique metric timeseries

Cost Optimization Tips:

Use sampling to reduce indexed spans (keeps metrics accurate)
Filter low-value traces (health checks, static assets)
Use the Datadog Agent for local aggregation
Set retention periods appropriately

Support

Datadog Documentation: https://docs.datadoghq.com/tracing/
Datadog Support: Available via in-app chat or email
OpenTelemetry Compatibility: https://docs.datadoghq.com/tracing/trace_collection/otel_instrumentation/

For Databrain configuration issues, contact your Databrain support team.

​Integrating Databrain with Datadog

​Prerequisites

​Configuration

​1. Get Your Datadog API Key

​2. Determine Your Datadog Site

​3. Configure Databrain Environment Variables

​4. Docker Compose Configuration

​5. Kubernetes Configuration

​Alternative: Using Datadog Agent

​1. Deploy Datadog Agent

​What Gets Sent to Datadog

​Verification

​1. Restart Databrain

​2. Generate Test Traffic

​3. Check Datadog UI

​Custom Tags and Attributes

​Advanced Configuration

​Sampling Configuration

​Resource Attributes

​Troubleshooting

​Debug Mode

​Verify Agent Connectivity (if using Agent)

​Datadog Features

​APM Dashboard

​Log Correlation

​Alerts

​Pricing Considerations

​Support