Documentation Index
Fetch the complete documentation index at: https://docs.usedatabrain.com/llms.txt
Use this file to discover all available pages before exploring further.
Integrating Databrain with Datadog
This guide explains how to send OpenTelemetry traces, metrics, and logs from your self-hosted Databrain instance to Datadog.
Prerequisites
- Databrain self-hosted version with OpenTelemetry support
- Datadog account with APM enabled
- Datadog API key
Configuration
1. Get Your Datadog API Key
- Log into your Datadog account
- Navigate to Organization Settings → API Keys
- Copy an existing API key or create a new one
2. Determine Your Datadog Site
Identify your Datadog site (region):
| Site | OTLP Endpoint |
|---|
| US1 (default) | https://api.datadoghq.com |
| US3 | https://api.us3.datadoghq.com |
| US5 | https://api.us5.datadoghq.com |
| EU | https://api.datadoghq.eu |
| US1-FED | https://api.ddog-gov.com |
| AP1 | https://api.ap1.datadoghq.com |
# Enable OpenTelemetry
OTEL_ENABLED=true
# Datadog OTLP endpoint (adjust for your site)
OTEL_EXPORTER_OTLP_ENDPOINT=https://api.datadoghq.com
# Service name (appears in Datadog APM)
OTEL_SERVICE_NAME=databrain-api
# Datadog API key (required for authentication)
DD_API_KEY=your_datadog_api_key_here
# Optional: Set environment tag
DD_ENV=production
# Optional: Set version tag
DD_VERSION=1.0.0
# Optional: Enable debug logging
LOG_LEVEL=info
4. Docker Compose Configuration
Update your docker-compose.yml:
services:
databrainbackend:
environment:
OTEL_ENABLED: "true"
OTEL_EXPORTER_OTLP_ENDPOINT: "https://api.datadoghq.com"
OTEL_SERVICE_NAME: "databrain-api"
DD_API_KEY: "${DD_API_KEY}"
DD_ENV: "production"
DD_VERSION: "1.0.0"
LOG_LEVEL: "info"
DD_API_KEY in a .env file, not directly in docker-compose.yml.
5. Kubernetes Configuration
For Kubernetes deployments:
apiVersion: v1
kind: Secret
metadata:
name: datadog-secret
type: Opaque
stringData:
api-key: your_datadog_api_key_here
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: databrain-backend
spec:
template:
spec:
containers:
- name: backend
env:
- name: OTEL_ENABLED
value: "true"
- name: OTEL_EXPORTER_OTLP_ENDPOINT
value: "https://api.datadoghq.com"
- name: OTEL_SERVICE_NAME
value: "databrain-api"
- name: DD_API_KEY
valueFrom:
secretKeyRef:
name: datadog-secret
key: api-key
- name: DD_ENV
value: "production"
Alternative: Using Datadog Agent
For better performance and additional features, use the Datadog Agent as an OTLP collector:
1. Deploy Datadog Agent
Docker Compose:
services:
datadog-agent:
image: gcr.io/datadoghq/agent:latest
environment:
DD_API_KEY: "${DD_API_KEY}"
DD_SITE: "datadoghq.com" # Adjust for your site
DD_APM_ENABLED: "true"
DD_APM_NON_LOCAL_TRAFFIC: "true"
DD_OTLP_CONFIG_RECEIVER_PROTOCOLS_HTTP_ENDPOINT: "0.0.0.0:4318"
DD_OTLP_CONFIG_RECEIVER_PROTOCOLS_GRPC_ENDPOINT: "0.0.0.0:4317"
ports:
- "4317:4317" # OTLP gRPC
- "4318:4318" # OTLP HTTP
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
- /proc/:/host/proc/:ro
- /sys/fs/cgroup/:/host/sys/fs/cgroup:ro
networks:
- databrain
databrainbackend:
environment:
OTEL_ENABLED: "true"
OTEL_EXPORTER_OTLP_ENDPOINT: "http://datadog-agent:4318"
OTEL_SERVICE_NAME: "databrain-api"
apiVersion: datadoghq.com/v2alpha1
kind: DatadogAgent
metadata:
name: datadog
spec:
global:
credentials:
apiKey: <YOUR_API_KEY>
site: datadoghq.com
features:
apm:
enabled: true
otlp:
receiver:
protocols:
http:
enabled: true
endpoint: "0.0.0.0:4318"
grpc:
enabled: true
endpoint: "0.0.0.0:4317"
env:
- name: OTEL_ENABLED
value: "true"
- name: OTEL_EXPORTER_OTLP_ENDPOINT
value: "http://datadog-agent.datadog.svc.cluster.local:4318"
- name: OTEL_SERVICE_NAME
value: "databrain-api"
What Gets Sent to Datadog
Once configured, Databrain automatically sends:
| Telemetry Type | Datadog Product | Description |
|---|
| Traces | APM | API request spans with timing, status codes, and errors |
| Metrics | Metrics | Request latency histograms, error rates, throughput |
| Logs | Log Management | Correlated logs with trace context (trace_id, span_id) |
Verification
1. Restart Databrain
docker compose restart databrainbackend
# or
kubectl rollout restart deployment/databrain-backend
2. Generate Test Traffic
Make a few API requests to your Databrain instance:
curl -X GET "https://your-databrain-instance.com/api/health"
curl -X POST "https://your-databrain-instance.com/api/v2/metric/execute" \
-H "Content-Type: application/json" \
-d '{"metricId": "test-123"}'
3. Check Datadog UI
-
APM Traces:
- Navigate to APM → Traces
- Filter by
service:databrain-api
- You should see traces within 1-2 minutes
-
Service Map:
- Go to APM → Service Map
- Look for
databrain-api and its dependencies (PostgreSQL, Redis, etc.)
-
Metrics:
- Navigate to Metrics → Explorer
- Search for
trace.http.request.duration or http.server.duration
-
Logs:
- Go to Logs → Search
- Filter by
service:databrain-api
- Click on a log entry to see correlated traces
Add custom tags to all telemetry:
# Environment-specific tags
DD_TAGS="env:production,team:backend,region:us-east-1"
# In docker-compose.yml
environment:
DD_TAGS: "env:production,team:backend,region:us-east-1"
Advanced Configuration
Sampling Configuration
To reduce costs, configure trace sampling in the Datadog Agent:
# datadog.yaml for Agent
apm_config:
max_traces_per_second: 10
trace_sample_rate: 0.5 # Sample 50% of traces
Resource Attributes
Databrain automatically includes these resource attributes:
service.name - Your service name (databrain-api)service.version - Databrain versiondeployment.environment - From DD_ENVhost.name - Container/pod hostname
Troubleshooting
| Issue | Solution |
|---|
| No data in Datadog | Verify OTEL_ENABLED=true and DD_API_KEY is correct |
| 403 Forbidden | Check API key has APM permissions in Datadog |
| Connection timeout | Verify endpoint URL matches your Datadog site |
| Missing traces | Wait 2-3 minutes; Datadog has ingestion delay |
| High costs | Implement sampling in Datadog Agent configuration |
Debug Mode
Enable debug logging to troubleshoot:
Check Databrain logs for:
[Telemetry] OpenTelemetry initialized - service: databrain-api
Verify Agent Connectivity (if using Agent)
Check Datadog Agent logs:
# Docker
docker logs datadog-agent | grep -i otlp
# Kubernetes
kubectl logs -n datadog daemonset/datadog-agent | grep -i otlp
OTLP HTTP receiver listening on 0.0.0.0:4318
Datadog Features
APM Dashboard
Datadog APM provides:
- Service Overview: Latency percentiles (p50, p75, p95, p99), throughput, error rates
- Flame Graphs: Visual representation of trace spans
- Deployment Tracking: Compare performance before/after deployments
- Error Tracking: Automatic error detection and grouping
Log Correlation
Click on any trace in Datadog APM to see:
- All logs generated during that request
- Database queries executed
- External API calls made
- Full request/response context
Alerts
Set up monitors in Datadog:
Alert when:
avg(last_5m):avg:trace.http.request.duration{service:databrain-api} > 2000
Pricing Considerations
Datadog pricing is based on:
- Indexed Spans: Number of traces retained for search
- Ingested Spans: Total spans sent (used for metrics)
- Custom Metrics: Number of unique metric timeseries
Cost Optimization Tips:
- Use sampling to reduce indexed spans (keeps metrics accurate)
- Filter low-value traces (health checks, static assets)
- Use the Datadog Agent for local aggregation
- Set retention periods appropriately
Support
For Databrain configuration issues, contact your Databrain support team.
