Integrating Databrain with Elastic APM

This guide explains how to send OpenTelemetry traces, metrics, and logs from your self-hosted Databrain instance to Elastic APM (Application Performance Monitoring).

Why Elastic APM?

Elastic APM is ideal if you’re already using the Elastic Stack (ELK):

Unified Platform: APM data alongside logs and metrics in Kibana
OpenTelemetry Support: Native OTLP ingestion
Powerful Querying: Kibana’s full search capabilities
Flexible Deployment: Cloud or self-hosted options

Prerequisites

Databrain self-hosted version with OpenTelemetry support
Elastic Stack 8.0+ (Elasticsearch, Kibana, APM Server)
APM Server configured for OpenTelemetry

Option 1: Elastic Cloud (Easiest)

1. Set Up Elastic Cloud

Sign up for Elastic Cloud
Create a deployment (includes Elasticsearch, Kibana, and APM)
Note your:
- Cloud ID
- APM endpoint: https://<deployment-id>.apm.<region>.cloud.es.io
- Secret token or API key

2. Configure Databrain

# Enable OpenTelemetry
OTEL_ENABLED=true

# Elastic Cloud APM endpoint
OTEL_EXPORTER_OTLP_ENDPOINT=https://<deployment-id>.apm.<region>.cloud.es.io

# Elastic APM authentication (use secret token OR API key)
OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer YOUR_SECRET_TOKEN

# Service name (appears in Kibana)
OTEL_SERVICE_NAME=databrain-api

# Optional: Set environment
OTEL_RESOURCE_ATTRIBUTES=deployment.environment=production,service.version=1.0.0

# Optional: Enable debug logging
LOG_LEVEL=info

3. Docker Compose Configuration

services:
  databrainbackend:
    environment:
      OTEL_ENABLED: "true"
      OTEL_EXPORTER_OTLP_ENDPOINT: "https://${ELASTIC_DEPLOYMENT_ID}.apm.${ELASTIC_REGION}.cloud.es.io"
      OTEL_SERVICE_NAME: "databrain-api"
      OTEL_EXPORTER_OTLP_HEADERS: "Authorization=Bearer ${ELASTIC_APM_SECRET_TOKEN}"
      OTEL_RESOURCE_ATTRIBUTES: "deployment.environment=production,service.version=1.0.0"
      LOG_LEVEL: "info"

Store credentials in .env:

# .env
ELASTIC_DEPLOYMENT_ID=abc123def456
ELASTIC_REGION=us-east-1
ELASTIC_APM_SECRET_TOKEN=your_secret_token_here

Option 2: Self-Hosted Elastic Stack

1. Deploy Elastic Stack with Docker Compose

version: '3.8'

services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.11.0
    container_name: elasticsearch
    environment:
      - discovery.type=single-node
      - xpack.security.enabled=true
      - ELASTIC_PASSWORD=changeme
      - "ES_JAVA_OPTS=-Xms2g -Xmx2g"
    volumes:
      - elasticsearch-data:/usr/share/elasticsearch/data
    ports:
      - "9200:9200"
    networks:
      - elastic
    restart: unless-stopped

  kibana:
    image: docker.elastic.co/kibana/kibana:8.11.0
    container_name: kibana
    environment:
      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
      - ELASTICSEARCH_USERNAME=elastic
      - ELASTICSEARCH_PASSWORD=changeme
    ports:
      - "5601:5601"
    networks:
      - elastic
    depends_on:
      - elasticsearch
    restart: unless-stopped

  apm-server:
    image: docker.elastic.co/apm/apm-server:8.11.0
    container_name: apm-server
    command: >
      apm-server -e
        -E apm-server.rum.enabled=true
        -E setup.kibana.host=kibana:5601
        -E setup.template.settings.index.number_of_replicas=0
        -E apm-server.kibana.enabled=true
        -E apm-server.kibana.host=kibana:5601
        -E output.elasticsearch.hosts=["elasticsearch:9200"]
        -E output.elasticsearch.username=elastic
        -E output.elasticsearch.password=changeme
        -E apm-server.secret_token=my_secret_token
        -E apm-server.auth.secret_token=my_secret_token
    ports:
      - "8200:8200"
    networks:
      - elastic
      - databrain
    depends_on:
      - elasticsearch
      - kibana
    restart: unless-stopped

  filebeat:
    image: docker.elastic.co/beats/filebeat:8.11.0
    container_name: filebeat
    user: root
    volumes:
      - ./filebeat.yml:/usr/share/filebeat/filebeat.yml:ro
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
    networks:
      - elastic
    depends_on:
      - elasticsearch
    restart: unless-stopped

networks:
  elastic:
    driver: bridge
  databrain:
    external: true

volumes:
  elasticsearch-data:

2. Configure APM Server for OpenTelemetry

Create apm-server.yml:

apm-server:
  host: "0.0.0.0:8200"
  
  # OpenTelemetry configuration
  rum:
    enabled: true
  
  # Authentication
  auth:
    secret_token: "my_secret_token"
  
  # Kibana integration
  kibana:
    enabled: true
    host: "http://kibana:5601"

# Elasticsearch output
output.elasticsearch:
  hosts: ["http://elasticsearch:9200"]
  username: "elastic"
  password: "changeme"
  
# Index lifecycle management
setup.ilm:
  enabled: true
  
# Kibana dashboards
setup.dashboards:
  enabled: true

# Logging
logging.level: info
logging.to_files: true

3. Configure Filebeat for Log Ingestion

Create filebeat.yml:

filebeat.inputs:
  - type: container
    paths:
      - '/var/lib/docker/containers/*/*.log'
    processors:
      - add_docker_metadata:
          host: "unix:///var/run/docker.sock"
      - decode_json_fields:
          fields: ["message"]
          target: ""
          overwrite_keys: true

output.elasticsearch:
  hosts: ["http://elasticsearch:9200"]
  username: "elastic"
  password: "changeme"

setup.kibana:
  host: "http://kibana:5601"

4. Start Elastic Stack

docker compose up -d

# Wait for Elasticsearch to be ready
docker logs elasticsearch -f

# Initialize Kibana (first time only)
docker exec -it kibana curl -X POST \
  "http://localhost:5601/api/saved_objects/_bulk_get" \
  -H "kbn-xsrf: true" \
  -H "Content-Type: application/json"

5. Configure Databrain for Self-Hosted APM

services:
  databrainbackend:
    environment:
      OTEL_ENABLED: "true"
      OTEL_EXPORTER_OTLP_ENDPOINT: "http://apm-server:8200"
      OTEL_SERVICE_NAME: "databrain-api"
      OTEL_EXPORTER_OTLP_HEADERS: "Authorization=Bearer my_secret_token"
      LOG_LEVEL: "info"
    networks:
      - databrain
      - elastic

Option 3: Using Elastic APM Agent (Alternative)

If you prefer the native Elastic APM agent over OpenTelemetry:

npm install elastic-apm-node --save

Then configure in your application:

// src/index.ts (at the very top, before other imports)
import apm from 'elastic-apm-node';

if (process.env.ELASTIC_APM_ACTIVE === 'true') {
  apm.start({
    serviceName: process.env.ELASTIC_APM_SERVICE_NAME || 'databrain-api',
    serverUrl: process.env.ELASTIC_APM_SERVER_URL,
    secretToken: process.env.ELASTIC_APM_SECRET_TOKEN,
    environment: process.env.ELASTIC_APM_ENVIRONMENT || 'production',
    logLevel: 'info',
  });
}

// ... rest of your application imports and code

Environment variables:

ELASTIC_APM_ACTIVE=true
ELASTIC_APM_SERVER_URL=http://apm-server:8200
ELASTIC_APM_SECRET_TOKEN=my_secret_token
ELASTIC_APM_SERVICE_NAME=databrain-api
ELASTIC_APM_ENVIRONMENT=production

Kubernetes Deployment

Deploy Elastic Cloud on Kubernetes (ECK)

# Install ECK operator
kubectl create -f https://download.elastic.co/downloads/eck/2.10.0/crds.yaml
kubectl apply -f https://download.elastic.co/downloads/eck/2.10.0/operator.yaml

# Deploy Elasticsearch
cat <<EOF | kubectl apply -f -
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: databrain-es
  namespace: elastic-system
spec:
  version: 8.11.0
  nodeSets:
  - name: default
    count: 3
    config:
      node.store.allow_mmap: false
    volumeClaimTemplates:
    - metadata:
        name: elasticsearch-data
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 50Gi
EOF

# Deploy Kibana
cat <<EOF | kubectl apply -f -
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: databrain-kibana
  namespace: elastic-system
spec:
  version: 8.11.0
  count: 1
  elasticsearchRef:
    name: databrain-es
EOF

# Deploy APM Server
cat <<EOF | kubectl apply -f -
apiVersion: apm.k8s.elastic.co/v1
kind: ApmServer
metadata:
  name: databrain-apm
  namespace: elastic-system
spec:
  version: 8.11.0
  count: 1
  elasticsearchRef:
    name: databrain-es
  kibanaRef:
    name: databrain-kibana
EOF

Configure Databrain on Kubernetes

apiVersion: v1
kind: Secret
metadata:
  name: elastic-apm-secret
  namespace: databrain
type: Opaque
stringData:
  secret-token: "my_secret_token"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: databrain-backend
  namespace: databrain
spec:
  template:
    spec:
      containers:
      - name: backend
        env:
          - name: OTEL_ENABLED
            value: "true"
          - name: OTEL_EXPORTER_OTLP_ENDPOINT
            value: "http://databrain-apm-apm-http.elastic-system.svc.cluster.local:8200"
          - name: OTEL_SERVICE_NAME
            value: "databrain-api"
          - name: ELASTIC_APM_SECRET_TOKEN
            valueFrom:
              secretKeyRef:
                name: elastic-apm-secret
                key: secret-token
          - name: OTEL_EXPORTER_OTLP_HEADERS
            value: "Authorization=Bearer $(ELASTIC_APM_SECRET_TOKEN)"

What Gets Sent to Elastic APM

Telemetry Type	Elastic Product	Description
Traces	APM	Distributed traces with transaction and span details
Metrics	APM	Service metrics, system metrics, custom metrics
Logs	Elasticsearch	Structured logs with APM correlation
Errors	APM	Error tracking with stack traces

Verification

1. Check APM Server Status

# Check if APM Server is running
curl http://localhost:8200/

# Should return APM Server info

2. Generate Test Traffic

curl https://your-databrain-instance.com/api/health

3. View in Kibana

Open Kibana: http://localhost:5601
Navigate to Observability → APM → Services
You should see databrain-api in the services list
Click on the service to see:
- Transaction duration: Latency distribution
- Throughput: Requests per minute
- Error rate: Percentage of failed requests
- Transactions: Individual endpoints
- Dependencies: External services (DB, Redis, etc.)

4. View Traces

Click on Transactions tab
Select an endpoint (e.g., /api/v2/metric/execute)
Click on a specific transaction to see the trace waterfall

5. View Logs with APM Correlation

Go to Observability → Logs → Stream
Filter: service.name: databrain-api
Click on any log entry
Click View in APM to see the related trace

Creating Kibana Dashboards

Service Overview Dashboard

Go to Dashboard → Create dashboard
Add visualizations:

Request Rate (Lens):

Index pattern: apm-*
Metric: Count of processor.event: transaction
Time range: Last 15 minutes
Breakdown: service.name

Average Latency:

Metric: Average of transaction.duration.us
Filter: service.name: databrain-api

Error Rate:

Formula: (count where transaction.result = error) / count * 100
Filter: service.name: databrain-api

Top Transactions by Duration:

Metric: P95 of transaction.duration.us
Group by: transaction.name
Top 10

Database Performance Dashboard

Slow Queries:

Filter: span.type: db AND span.subtype: postgresql
Metric: P95 of span.duration.us
Group by: span.db.statement

Setting Up Alerts

Create Alert Rules in Kibana

Go to Observability → Alerts → Create rule

High Error Rate Alert

Rule type: APM
Service: databrain-api
Alert when: Transaction error rate is above threshold
Threshold: > 5% for 5 minutes
Actions: Send to Slack/Email/PagerDuty

High Latency Alert

Rule type: APM
Service: databrain-api
Transaction type: request
Alert when: Latency threshold is exceeded
Threshold: P95 > 2000ms for 5 minutes

Service Down Alert

Rule type: Uptime
Monitor: databrain-api
Alert when: Service is down
Check frequency: Every 1 minute

Machine Learning (X-Pack)

If you have an Elastic license, enable ML anomaly detection:

Go to Machine Learning → Anomaly Detection
Create job for APM:
- Job type: APM
- Service: databrain-api
- Analyze: Response times and throughput
ML will automatically detect:
- Unusual latency spikes
- Traffic anomalies
- Error rate changes

Troubleshooting

Issue	Solution
No data in Kibana	Check APM Server logs: `docker logs apm-server`
401 Unauthorized	Verify secret token matches between Databrain and APM Server
Connection refused	Check network connectivity and firewall rules
Missing logs	Ensure Filebeat is running and configured correctly
High disk usage	Configure ILM (Index Lifecycle Management) policies

Debug APM Server

# Check APM Server logs
docker logs apm-server -f

# Check Elasticsearch connectivity
docker exec apm-server curl -u elastic:changeme http://elasticsearch:9200/_cluster/health

# Verify APM Server is receiving data
curl -H "Authorization: Bearer my_secret_token" \
  http://localhost:8200/intake/v2/rum/events

Check Elasticsearch Indices

# List APM indices
curl -u elastic:changeme http://localhost:9200/_cat/indices/apm-*

# Check document count
curl -u elastic:changeme http://localhost:9200/apm-*/_count

Index Lifecycle Management (ILM)

Configure ILM to manage disk usage:

Go to Stack Management → Index Lifecycle Policies
Edit APM policies:

APM Traces (apm--transaction, apm--span):

Hot phase: 1 day
Warm phase: 7 days
Delete phase: 30 days

APM Metrics:

Hot phase: 3 days
Warm phase: 14 days
Delete phase: 90 days

APM Errors:

Hot phase: 7 days
Warm phase: 30 days
Delete phase: 90 days

Best Practices

1. Use Index Templates

Customize APM index templates for better performance:

PUT _index_template/apm-custom
{
  "index_patterns": ["apm-*"],
  "priority": 500,
  "template": {
    "settings": {
      "number_of_shards": 3,
      "number_of_replicas": 1,
      "refresh_interval": "5s"
    }
  }
}

2. Enable Cross-Cluster Search

For multi-region deployments, use cross-cluster search to aggregate data.

3. Use APM Correlations

Let Elastic automatically find correlations between slow transactions and attributes:

Go to APM → databrain-api → Latency correlations
View attributes that correlate with slow requests

Cost Optimization

For Elastic Cloud:

Optimize data retention: Use ILM to delete old data
Reduce sampling: Not all traces need to be kept
Use smaller instance sizes: Start small, scale as needed

For self-hosted:

Use hot-warm architecture: Move old data to cheaper storage
Enable compression: Reduce storage by 50-70%
Optimize shard size: 20-50GB per shard is ideal

Support

Elastic Documentation: https://www.elastic.co/guide/en/apm/
Elastic Forum: https://discuss.elastic.co/
Elastic Support: Available for paid subscriptions

For Databrain configuration issues, contact your Databrain support team.

​Integrating Databrain with Elastic APM

​Why Elastic APM?

​Prerequisites

​Option 1: Elastic Cloud (Easiest)

​1. Set Up Elastic Cloud

​2. Configure Databrain

​3. Docker Compose Configuration

​Option 2: Self-Hosted Elastic Stack

​1. Deploy Elastic Stack with Docker Compose

​2. Configure APM Server for OpenTelemetry

​3. Configure Filebeat for Log Ingestion

​4. Start Elastic Stack

​5. Configure Databrain for Self-Hosted APM

​Option 3: Using Elastic APM Agent (Alternative)

​Kubernetes Deployment

​Deploy Elastic Cloud on Kubernetes (ECK)

​Configure Databrain on Kubernetes

​What Gets Sent to Elastic APM

​Verification

​1. Check APM Server Status

​2. Generate Test Traffic

​3. View in Kibana

​4. View Traces

​5. View Logs with APM Correlation

​Creating Kibana Dashboards

​Service Overview Dashboard

​Database Performance Dashboard

​Setting Up Alerts

​Create Alert Rules in Kibana

​High Error Rate Alert

​High Latency Alert

​Service Down Alert

​Machine Learning (X-Pack)

​Troubleshooting

​Debug APM Server

​Check Elasticsearch Indices

​Index Lifecycle Management (ILM)

​Best Practices

​1. Use Index Templates

​2. Enable Cross-Cluster Search

​3. Use APM Correlations

​Cost Optimization

​Support