> ## Documentation Index
> Fetch the complete documentation index at: https://docs.usedatabrain.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Datadog Integration

> Send OpenTelemetry traces, metrics, and logs from Databrain to Datadog

# Integrating Databrain with Datadog

This guide explains how to send OpenTelemetry traces, metrics, and logs from your self-hosted Databrain instance to Datadog.

## Prerequisites

* Databrain self-hosted version with OpenTelemetry support
* Datadog account with APM enabled
* Datadog API key

## Configuration

### 1. Get Your Datadog API Key

1. Log into your [Datadog account](https://app.datadoghq.com/)
2. Navigate to **Organization Settings** → **API Keys**
3. Copy an existing API key or create a new one

### 2. Determine Your Datadog Site

Identify your Datadog site (region):

| Site          | OTLP Endpoint                   |
| ------------- | ------------------------------- |
| US1 (default) | `https://api.datadoghq.com`     |
| US3           | `https://api.us3.datadoghq.com` |
| US5           | `https://api.us5.datadoghq.com` |
| EU            | `https://api.datadoghq.eu`      |
| US1-FED       | `https://api.ddog-gov.com`      |
| AP1           | `https://api.ap1.datadoghq.com` |

### 3. Configure Databrain Environment Variables

Add these environment variables to your Databrain backend:

```bash theme={"dark"}
# Enable OpenTelemetry
OTEL_ENABLED=true

# Datadog OTLP endpoint (adjust for your site)
OTEL_EXPORTER_OTLP_ENDPOINT=https://api.datadoghq.com

# Service name (appears in Datadog APM)
OTEL_SERVICE_NAME=databrain-api

# Datadog API key (required for authentication)
DD_API_KEY=your_datadog_api_key_here

# Optional: Set environment tag
DD_ENV=production

# Optional: Set version tag
DD_VERSION=1.0.0

# Optional: Enable debug logging
LOG_LEVEL=info
```

### 4. Docker Compose Configuration

Update your `docker-compose.yml`:

```yaml theme={"dark"}
services:
  databrainbackend:
    environment:
      OTEL_ENABLED: "true"
      OTEL_EXPORTER_OTLP_ENDPOINT: "https://api.datadoghq.com"
      OTEL_SERVICE_NAME: "databrain-api"
      DD_API_KEY: "${DD_API_KEY}"
      DD_ENV: "production"
      DD_VERSION: "1.0.0"
      LOG_LEVEL: "info"
```

**Security Note**: Store your `DD_API_KEY` in a `.env` file, not directly in `docker-compose.yml`.

### 5. Kubernetes Configuration

For Kubernetes deployments:

```yaml theme={"dark"}
apiVersion: v1
kind: Secret
metadata:
  name: datadog-secret
type: Opaque
stringData:
  api-key: your_datadog_api_key_here
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: databrain-backend
spec:
  template:
    spec:
      containers:
      - name: backend
        env:
          - name: OTEL_ENABLED
            value: "true"
          - name: OTEL_EXPORTER_OTLP_ENDPOINT
            value: "https://api.datadoghq.com"
          - name: OTEL_SERVICE_NAME
            value: "databrain-api"
          - name: DD_API_KEY
            valueFrom:
              secretKeyRef:
                name: datadog-secret
                key: api-key
          - name: DD_ENV
            value: "production"
```

## Alternative: Using Datadog Agent

For better performance and additional features, use the Datadog Agent as an OTLP collector:

### 1. Deploy Datadog Agent

**Docker Compose:**

```yaml theme={"dark"}
services:
  datadog-agent:
    image: gcr.io/datadoghq/agent:latest
    environment:
      DD_API_KEY: "${DD_API_KEY}"
      DD_SITE: "datadoghq.com"  # Adjust for your site
      DD_APM_ENABLED: "true"
      DD_APM_NON_LOCAL_TRAFFIC: "true"
      DD_OTLP_CONFIG_RECEIVER_PROTOCOLS_HTTP_ENDPOINT: "0.0.0.0:4318"
      DD_OTLP_CONFIG_RECEIVER_PROTOCOLS_GRPC_ENDPOINT: "0.0.0.0:4317"
    ports:
      - "4317:4317"  # OTLP gRPC
      - "4318:4318"  # OTLP HTTP
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /proc/:/host/proc/:ro
      - /sys/fs/cgroup/:/host/sys/fs/cgroup:ro
    networks:
      - databrain

  databrainbackend:
    environment:
      OTEL_ENABLED: "true"
      OTEL_EXPORTER_OTLP_ENDPOINT: "http://datadog-agent:4318"
      OTEL_SERVICE_NAME: "databrain-api"
```

**Kubernetes (using Datadog Operator):**

```yaml theme={"dark"}
apiVersion: datadoghq.com/v2alpha1
kind: DatadogAgent
metadata:
  name: datadog
spec:
  global:
    credentials:
      apiKey: <YOUR_API_KEY>
    site: datadoghq.com
  features:
    apm:
      enabled: true
    otlp:
      receiver:
        protocols:
          http:
            enabled: true
            endpoint: "0.0.0.0:4318"
          grpc:
            enabled: true
            endpoint: "0.0.0.0:4317"
```

Then configure Databrain:

```yaml theme={"dark"}
env:
  - name: OTEL_ENABLED
    value: "true"
  - name: OTEL_EXPORTER_OTLP_ENDPOINT
    value: "http://datadog-agent.datadog.svc.cluster.local:4318"
  - name: OTEL_SERVICE_NAME
    value: "databrain-api"
```

## What Gets Sent to Datadog

Once configured, Databrain automatically sends:

| Telemetry Type | Datadog Product | Description                                              |
| -------------- | --------------- | -------------------------------------------------------- |
| **Traces**     | APM             | API request spans with timing, status codes, and errors  |
| **Metrics**    | Metrics         | Request latency histograms, error rates, throughput      |
| **Logs**       | Log Management  | Correlated logs with trace context (trace\_id, span\_id) |

## Verification

### 1. Restart Databrain

```bash theme={"dark"}
docker compose restart databrainbackend
# or
kubectl rollout restart deployment/databrain-backend
```

### 2. Generate Test Traffic

Make a few API requests to your Databrain instance:

```bash theme={"dark"}
curl -X GET "https://your-databrain-instance.com/api/health"
curl -X POST "https://your-databrain-instance.com/api/v2/metric/execute" \
  -H "Content-Type: application/json" \
  -d '{"metricId": "test-123"}'
```

### 3. Check Datadog UI

1. **APM Traces**:
   * Navigate to **APM** → **Traces**
   * Filter by `service:databrain-api`
   * You should see traces within 1-2 minutes

2. **Service Map**:
   * Go to **APM** → **Service Map**
   * Look for `databrain-api` and its dependencies (PostgreSQL, Redis, etc.)

3. **Metrics**:
   * Navigate to **Metrics** → **Explorer**
   * Search for `trace.http.request.duration` or `http.server.duration`

4. **Logs**:
   * Go to **Logs** → **Search**
   * Filter by `service:databrain-api`
   * Click on a log entry to see correlated traces

## Custom Tags and Attributes

Add custom tags to all telemetry:

```bash theme={"dark"}
# Environment-specific tags
DD_TAGS="env:production,team:backend,region:us-east-1"

# In docker-compose.yml
environment:
  DD_TAGS: "env:production,team:backend,region:us-east-1"
```

These tags will appear in Datadog and can be used for filtering and grouping.

## Advanced Configuration

### Sampling Configuration

To reduce costs, configure trace sampling in the Datadog Agent:

```yaml theme={"dark"}
# datadog.yaml for Agent
apm_config:
  max_traces_per_second: 10
  trace_sample_rate: 0.5  # Sample 50% of traces
```

### Resource Attributes

Databrain automatically includes these resource attributes:

* `service.name` - Your service name (databrain-api)
* `service.version` - Databrain version
* `deployment.environment` - From DD\_ENV
* `host.name` - Container/pod hostname

## Troubleshooting

| Issue              | Solution                                               |
| ------------------ | ------------------------------------------------------ |
| No data in Datadog | Verify `OTEL_ENABLED=true` and DD\_API\_KEY is correct |
| 403 Forbidden      | Check API key has APM permissions in Datadog           |
| Connection timeout | Verify endpoint URL matches your Datadog site          |
| Missing traces     | Wait 2-3 minutes; Datadog has ingestion delay          |
| High costs         | Implement sampling in Datadog Agent configuration      |

### Debug Mode

Enable debug logging to troubleshoot:

```bash theme={"dark"}
LOG_LEVEL=debug
```

Check Databrain logs for:

```
[Telemetry] OpenTelemetry initialized - service: databrain-api
```

### Verify Agent Connectivity (if using Agent)

Check Datadog Agent logs:

```bash theme={"dark"}
# Docker
docker logs datadog-agent | grep -i otlp

# Kubernetes
kubectl logs -n datadog daemonset/datadog-agent | grep -i otlp
```

You should see:

```
OTLP HTTP receiver listening on 0.0.0.0:4318
```

## Datadog Features

### APM Dashboard

Datadog APM provides:

* **Service Overview**: Latency percentiles (p50, p75, p95, p99), throughput, error rates
* **Flame Graphs**: Visual representation of trace spans
* **Deployment Tracking**: Compare performance before/after deployments
* **Error Tracking**: Automatic error detection and grouping

### Log Correlation

Click on any trace in Datadog APM to see:

* All logs generated during that request
* Database queries executed
* External API calls made
* Full request/response context

### Alerts

Set up monitors in Datadog:

```
Alert when:
  avg(last_5m):avg:trace.http.request.duration{service:databrain-api} > 2000
```

## Pricing Considerations

Datadog pricing is based on:

* **Indexed Spans**: Number of traces retained for search
* **Ingested Spans**: Total spans sent (used for metrics)
* **Custom Metrics**: Number of unique metric timeseries

**Cost Optimization Tips**:

1. Use sampling to reduce indexed spans (keeps metrics accurate)
2. Filter low-value traces (health checks, static assets)
3. Use the Datadog Agent for local aggregation
4. Set retention periods appropriately

## Support

* **Datadog Documentation**: [https://docs.datadoghq.com/tracing/](https://docs.datadoghq.com/tracing/)
* **Datadog Support**: Available via in-app chat or email
* **OpenTelemetry Compatibility**: [https://docs.datadoghq.com/tracing/trace\_collection/otel\_instrumentation/](https://docs.datadoghq.com/tracing/trace_collection/otel_instrumentation/)

For Databrain configuration issues, contact your Databrain support team.