> ## Documentation Index
> Fetch the complete documentation index at: https://docs.usedatabrain.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Elastic APM Integration

> Send OpenTelemetry traces, metrics, and logs from Databrain to Elastic APM

# Integrating Databrain with Elastic APM

This guide explains how to send OpenTelemetry traces, metrics, and logs from your self-hosted Databrain instance to Elastic APM (Application Performance Monitoring).

## Why Elastic APM?

Elastic APM is ideal if you're already using the Elastic Stack (ELK):

* **Unified Platform**: APM data alongside logs and metrics in Kibana
* **OpenTelemetry Support**: Native OTLP ingestion
* **Powerful Querying**: Kibana's full search capabilities
* **Flexible Deployment**: Cloud or self-hosted options

## Prerequisites

* Databrain self-hosted version with OpenTelemetry support
* Elastic Stack 8.0+ (Elasticsearch, Kibana, APM Server)
* APM Server configured for OpenTelemetry

## Option 1: Elastic Cloud (Easiest)

### 1. Set Up Elastic Cloud

1. Sign up for [Elastic Cloud](https://cloud.elastic.co/)
2. Create a deployment (includes Elasticsearch, Kibana, and APM)
3. Note your:
   * **Cloud ID**
   * **APM endpoint**: `https://<deployment-id>.apm.<region>.cloud.es.io`
   * **Secret token** or **API key**

### 2. Configure Databrain

```bash theme={"dark"}
# Enable OpenTelemetry
OTEL_ENABLED=true

# Elastic Cloud APM endpoint
OTEL_EXPORTER_OTLP_ENDPOINT=https://<deployment-id>.apm.<region>.cloud.es.io

# Elastic APM authentication (use secret token OR API key)
OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer YOUR_SECRET_TOKEN

# Service name (appears in Kibana)
OTEL_SERVICE_NAME=databrain-api

# Optional: Set environment
OTEL_RESOURCE_ATTRIBUTES=deployment.environment=production,service.version=1.0.0

# Optional: Enable debug logging
LOG_LEVEL=info
```

### 3. Docker Compose Configuration

```yaml theme={"dark"}
services:
  databrainbackend:
    environment:
      OTEL_ENABLED: "true"
      OTEL_EXPORTER_OTLP_ENDPOINT: "https://${ELASTIC_DEPLOYMENT_ID}.apm.${ELASTIC_REGION}.cloud.es.io"
      OTEL_SERVICE_NAME: "databrain-api"
      OTEL_EXPORTER_OTLP_HEADERS: "Authorization=Bearer ${ELASTIC_APM_SECRET_TOKEN}"
      OTEL_RESOURCE_ATTRIBUTES: "deployment.environment=production,service.version=1.0.0"
      LOG_LEVEL: "info"
```

Store credentials in `.env`:

```bash theme={"dark"}
# .env
ELASTIC_DEPLOYMENT_ID=abc123def456
ELASTIC_REGION=us-east-1
ELASTIC_APM_SECRET_TOKEN=your_secret_token_here
```

## Option 2: Self-Hosted Elastic Stack

### 1. Deploy Elastic Stack with Docker Compose

```yaml theme={"dark"}
version: '3.8'

services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.11.0
    container_name: elasticsearch
    environment:
      - discovery.type=single-node
      - xpack.security.enabled=true
      - ELASTIC_PASSWORD=changeme
      - "ES_JAVA_OPTS=-Xms2g -Xmx2g"
    volumes:
      - elasticsearch-data:/usr/share/elasticsearch/data
    ports:
      - "9200:9200"
    networks:
      - elastic
    restart: unless-stopped

  kibana:
    image: docker.elastic.co/kibana/kibana:8.11.0
    container_name: kibana
    environment:
      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
      - ELASTICSEARCH_USERNAME=elastic
      - ELASTICSEARCH_PASSWORD=changeme
    ports:
      - "5601:5601"
    networks:
      - elastic
    depends_on:
      - elasticsearch
    restart: unless-stopped

  apm-server:
    image: docker.elastic.co/apm/apm-server:8.11.0
    container_name: apm-server
    command: >
      apm-server -e
        -E apm-server.rum.enabled=true
        -E setup.kibana.host=kibana:5601
        -E setup.template.settings.index.number_of_replicas=0
        -E apm-server.kibana.enabled=true
        -E apm-server.kibana.host=kibana:5601
        -E output.elasticsearch.hosts=["elasticsearch:9200"]
        -E output.elasticsearch.username=elastic
        -E output.elasticsearch.password=changeme
        -E apm-server.secret_token=my_secret_token
        -E apm-server.auth.secret_token=my_secret_token
    ports:
      - "8200:8200"
    networks:
      - elastic
      - databrain
    depends_on:
      - elasticsearch
      - kibana
    restart: unless-stopped

  filebeat:
    image: docker.elastic.co/beats/filebeat:8.11.0
    container_name: filebeat
    user: root
    volumes:
      - ./filebeat.yml:/usr/share/filebeat/filebeat.yml:ro
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
    networks:
      - elastic
    depends_on:
      - elasticsearch
    restart: unless-stopped

networks:
  elastic:
    driver: bridge
  databrain:
    external: true

volumes:
  elasticsearch-data:
```

### 2. Configure APM Server for OpenTelemetry

Create `apm-server.yml`:

```yaml theme={"dark"}
apm-server:
  host: "0.0.0.0:8200"
  
  # OpenTelemetry configuration
  rum:
    enabled: true
  
  # Authentication
  auth:
    secret_token: "my_secret_token"
  
  # Kibana integration
  kibana:
    enabled: true
    host: "http://kibana:5601"

# Elasticsearch output
output.elasticsearch:
  hosts: ["http://elasticsearch:9200"]
  username: "elastic"
  password: "changeme"
  
# Index lifecycle management
setup.ilm:
  enabled: true
  
# Kibana dashboards
setup.dashboards:
  enabled: true

# Logging
logging.level: info
logging.to_files: true
```

### 3. Configure Filebeat for Log Ingestion

Create `filebeat.yml`:

```yaml theme={"dark"}
filebeat.inputs:
  - type: container
    paths:
      - '/var/lib/docker/containers/*/*.log'
    processors:
      - add_docker_metadata:
          host: "unix:///var/run/docker.sock"
      - decode_json_fields:
          fields: ["message"]
          target: ""
          overwrite_keys: true

output.elasticsearch:
  hosts: ["http://elasticsearch:9200"]
  username: "elastic"
  password: "changeme"

setup.kibana:
  host: "http://kibana:5601"
```

### 4. Start Elastic Stack

```bash theme={"dark"}
docker compose up -d

# Wait for Elasticsearch to be ready
docker logs elasticsearch -f

# Initialize Kibana (first time only)
docker exec -it kibana curl -X POST \
  "http://localhost:5601/api/saved_objects/_bulk_get" \
  -H "kbn-xsrf: true" \
  -H "Content-Type: application/json"
```

### 5. Configure Databrain for Self-Hosted APM

```yaml theme={"dark"}
services:
  databrainbackend:
    environment:
      OTEL_ENABLED: "true"
      OTEL_EXPORTER_OTLP_ENDPOINT: "http://apm-server:8200"
      OTEL_SERVICE_NAME: "databrain-api"
      OTEL_EXPORTER_OTLP_HEADERS: "Authorization=Bearer my_secret_token"
      LOG_LEVEL: "info"
    networks:
      - databrain
      - elastic
```

## Option 3: Using Elastic APM Agent (Alternative)

If you prefer the native Elastic APM agent over OpenTelemetry:

```bash theme={"dark"}
npm install elastic-apm-node --save
```

Then configure in your application:

```typescript theme={"dark"}
// src/index.ts (at the very top, before other imports)
import apm from 'elastic-apm-node';

if (process.env.ELASTIC_APM_ACTIVE === 'true') {
  apm.start({
    serviceName: process.env.ELASTIC_APM_SERVICE_NAME || 'databrain-api',
    serverUrl: process.env.ELASTIC_APM_SERVER_URL,
    secretToken: process.env.ELASTIC_APM_SECRET_TOKEN,
    environment: process.env.ELASTIC_APM_ENVIRONMENT || 'production',
    logLevel: 'info',
  });
}

// ... rest of your application imports and code
```

Environment variables:

```bash theme={"dark"}
ELASTIC_APM_ACTIVE=true
ELASTIC_APM_SERVER_URL=http://apm-server:8200
ELASTIC_APM_SECRET_TOKEN=my_secret_token
ELASTIC_APM_SERVICE_NAME=databrain-api
ELASTIC_APM_ENVIRONMENT=production
```

## Kubernetes Deployment

### Deploy Elastic Cloud on Kubernetes (ECK)

```bash theme={"dark"}
# Install ECK operator
kubectl create -f https://download.elastic.co/downloads/eck/2.10.0/crds.yaml
kubectl apply -f https://download.elastic.co/downloads/eck/2.10.0/operator.yaml

# Deploy Elasticsearch
cat <<EOF | kubectl apply -f -
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: databrain-es
  namespace: elastic-system
spec:
  version: 8.11.0
  nodeSets:
  - name: default
    count: 3
    config:
      node.store.allow_mmap: false
    volumeClaimTemplates:
    - metadata:
        name: elasticsearch-data
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 50Gi
EOF

# Deploy Kibana
cat <<EOF | kubectl apply -f -
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: databrain-kibana
  namespace: elastic-system
spec:
  version: 8.11.0
  count: 1
  elasticsearchRef:
    name: databrain-es
EOF

# Deploy APM Server
cat <<EOF | kubectl apply -f -
apiVersion: apm.k8s.elastic.co/v1
kind: ApmServer
metadata:
  name: databrain-apm
  namespace: elastic-system
spec:
  version: 8.11.0
  count: 1
  elasticsearchRef:
    name: databrain-es
  kibanaRef:
    name: databrain-kibana
EOF
```

### Configure Databrain on Kubernetes

```yaml theme={"dark"}
apiVersion: v1
kind: Secret
metadata:
  name: elastic-apm-secret
  namespace: databrain
type: Opaque
stringData:
  secret-token: "my_secret_token"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: databrain-backend
  namespace: databrain
spec:
  template:
    spec:
      containers:
      - name: backend
        env:
          - name: OTEL_ENABLED
            value: "true"
          - name: OTEL_EXPORTER_OTLP_ENDPOINT
            value: "http://databrain-apm-apm-http.elastic-system.svc.cluster.local:8200"
          - name: OTEL_SERVICE_NAME
            value: "databrain-api"
          - name: ELASTIC_APM_SECRET_TOKEN
            valueFrom:
              secretKeyRef:
                name: elastic-apm-secret
                key: secret-token
          - name: OTEL_EXPORTER_OTLP_HEADERS
            value: "Authorization=Bearer $(ELASTIC_APM_SECRET_TOKEN)"
```

## What Gets Sent to Elastic APM

| Telemetry Type | Elastic Product | Description                                          |
| -------------- | --------------- | ---------------------------------------------------- |
| **Traces**     | APM             | Distributed traces with transaction and span details |
| **Metrics**    | APM             | Service metrics, system metrics, custom metrics      |
| **Logs**       | Elasticsearch   | Structured logs with APM correlation                 |
| **Errors**     | APM             | Error tracking with stack traces                     |

## Verification

### 1. Check APM Server Status

```bash theme={"dark"}
# Check if APM Server is running
curl http://localhost:8200/

# Should return APM Server info
```

### 2. Generate Test Traffic

```bash theme={"dark"}
curl https://your-databrain-instance.com/api/health
```

### 3. View in Kibana

1. Open Kibana: [http://localhost:5601](http://localhost:5601)
2. Navigate to **Observability** → **APM** → **Services**
3. You should see **databrain-api** in the services list
4. Click on the service to see:
   * **Transaction duration**: Latency distribution
   * **Throughput**: Requests per minute
   * **Error rate**: Percentage of failed requests
   * **Transactions**: Individual endpoints
   * **Dependencies**: External services (DB, Redis, etc.)

### 4. View Traces

1. Click on **Transactions** tab
2. Select an endpoint (e.g., `/api/v2/metric/execute`)
3. Click on a specific transaction to see the trace waterfall

### 5. View Logs with APM Correlation

1. Go to **Observability** → **Logs** → **Stream**
2. Filter: `service.name: databrain-api`
3. Click on any log entry
4. Click **View in APM** to see the related trace

## Creating Kibana Dashboards

### Service Overview Dashboard

1. Go to **Dashboard** → **Create dashboard**
2. Add visualizations:

**Request Rate (Lens)**:

* Index pattern: `apm-*`
* Metric: Count of `processor.event: transaction`
* Time range: Last 15 minutes
* Breakdown: service.name

**Average Latency**:

* Metric: Average of `transaction.duration.us`
* Filter: `service.name: databrain-api`

**Error Rate**:

* Formula: `(count where transaction.result = error) / count * 100`
* Filter: `service.name: databrain-api`

**Top Transactions by Duration**:

* Metric: P95 of `transaction.duration.us`
* Group by: `transaction.name`
* Top 10

### Database Performance Dashboard

**Slow Queries**:

* Filter: `span.type: db AND span.subtype: postgresql`
* Metric: P95 of `span.duration.us`
* Group by: `span.db.statement`

## Setting Up Alerts

### Create Alert Rules in Kibana

1. Go to **Observability** → **Alerts** → **Create rule**

#### High Error Rate Alert

* **Rule type**: APM
* **Service**: databrain-api
* **Alert when**: Transaction error rate is above threshold
* **Threshold**: > 5% for 5 minutes
* **Actions**: Send to Slack/Email/PagerDuty

#### High Latency Alert

* **Rule type**: APM
* **Service**: databrain-api
* **Transaction type**: request
* **Alert when**: Latency threshold is exceeded
* **Threshold**: P95 > 2000ms for 5 minutes

#### Service Down Alert

* **Rule type**: Uptime
* **Monitor**: databrain-api
* **Alert when**: Service is down
* **Check frequency**: Every 1 minute

## Machine Learning (X-Pack)

If you have an Elastic license, enable ML anomaly detection:

1. Go to **Machine Learning** → **Anomaly Detection**
2. Create job for APM:
   * **Job type**: APM
   * **Service**: databrain-api
   * **Analyze**: Response times and throughput
3. ML will automatically detect:
   * Unusual latency spikes
   * Traffic anomalies
   * Error rate changes

## Troubleshooting

| Issue              | Solution                                                     |
| ------------------ | ------------------------------------------------------------ |
| No data in Kibana  | Check APM Server logs: `docker logs apm-server`              |
| 401 Unauthorized   | Verify secret token matches between Databrain and APM Server |
| Connection refused | Check network connectivity and firewall rules                |
| Missing logs       | Ensure Filebeat is running and configured correctly          |
| High disk usage    | Configure ILM (Index Lifecycle Management) policies          |

### Debug APM Server

```bash theme={"dark"}
# Check APM Server logs
docker logs apm-server -f

# Check Elasticsearch connectivity
docker exec apm-server curl -u elastic:changeme http://elasticsearch:9200/_cluster/health

# Verify APM Server is receiving data
curl -H "Authorization: Bearer my_secret_token" \
  http://localhost:8200/intake/v2/rum/events
```

### Check Elasticsearch Indices

```bash theme={"dark"}
# List APM indices
curl -u elastic:changeme http://localhost:9200/_cat/indices/apm-*

# Check document count
curl -u elastic:changeme http://localhost:9200/apm-*/_count
```

## Index Lifecycle Management (ILM)

Configure ILM to manage disk usage:

1. Go to **Stack Management** → **Index Lifecycle Policies**
2. Edit APM policies:

**APM Traces** (apm-*-transaction, apm-*-span):

* Hot phase: 1 day
* Warm phase: 7 days
* Delete phase: 30 days

**APM Metrics**:

* Hot phase: 3 days
* Warm phase: 14 days
* Delete phase: 90 days

**APM Errors**:

* Hot phase: 7 days
* Warm phase: 30 days
* Delete phase: 90 days

## Best Practices

### 1. Use Index Templates

Customize APM index templates for better performance:

```json theme={"dark"}
PUT _index_template/apm-custom
{
  "index_patterns": ["apm-*"],
  "priority": 500,
  "template": {
    "settings": {
      "number_of_shards": 3,
      "number_of_replicas": 1,
      "refresh_interval": "5s"
    }
  }
}
```

### 2. Enable Cross-Cluster Search

For multi-region deployments, use cross-cluster search to aggregate data.

### 3. Use APM Correlations

Let Elastic automatically find correlations between slow transactions and attributes:

1. Go to **APM** → **databrain-api** → **Latency correlations**
2. View attributes that correlate with slow requests

## Cost Optimization

For Elastic Cloud:

* **Optimize data retention**: Use ILM to delete old data
* **Reduce sampling**: Not all traces need to be kept
* **Use smaller instance sizes**: Start small, scale as needed

For self-hosted:

* **Use hot-warm architecture**: Move old data to cheaper storage
* **Enable compression**: Reduce storage by 50-70%
* **Optimize shard size**: 20-50GB per shard is ideal

## Support

* **Elastic Documentation**: [https://www.elastic.co/guide/en/apm/](https://www.elastic.co/guide/en/apm/)
* **Elastic Forum**: [https://discuss.elastic.co/](https://discuss.elastic.co/)
* **Elastic Support**: Available for paid subscriptions

For Databrain configuration issues, contact your Databrain support team.