Skip to main content

​
Integrating Databrain with Elastic APM

This guide explains how to send OpenTelemetry traces, metrics, and logs from your self-hosted Databrain instance to Elastic APM (Application Performance Monitoring).

​
Why Elastic APM?

Elastic APM is ideal if you’re already using the Elastic Stack (ELK):
  • Unified Platform: APM data alongside logs and metrics in Kibana
  • OpenTelemetry Support: Native OTLP ingestion
  • Powerful Querying: Kibana’s full search capabilities
  • Flexible Deployment: Cloud or self-hosted options

​
Prerequisites

  • Databrain self-hosted version with OpenTelemetry support
  • Elastic Stack 8.0+ (Elasticsearch, Kibana, APM Server)
  • APM Server configured for OpenTelemetry

​
Option 1: Elastic Cloud (Easiest)

​
1. Set Up Elastic Cloud

  1. Sign up for Elastic Cloud
  2. Create a deployment (includes Elasticsearch, Kibana, and APM)
  3. Note your:
    • Cloud ID
    • APM endpoint: https://<deployment-id>.apm.<region>.cloud.es.io
    • Secret token or API key

​
2. Configure Databrain

# Enable OpenTelemetry
OTEL_ENABLED=true

# Elastic Cloud APM endpoint
OTEL_EXPORTER_OTLP_ENDPOINT=https://<deployment-id>.apm.<region>.cloud.es.io

# Elastic APM authentication (use secret token OR API key)
OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer YOUR_SECRET_TOKEN

# Service name (appears in Kibana)
OTEL_SERVICE_NAME=databrain-api

# Optional: Set environment
OTEL_RESOURCE_ATTRIBUTES=deployment.environment=production,service.version=1.0.0

# Optional: Enable debug logging
LOG_LEVEL=info

​
3. Docker Compose Configuration

services:
  databrainbackend:
    environment:
      OTEL_ENABLED: "true"
      OTEL_EXPORTER_OTLP_ENDPOINT: "https://${ELASTIC_DEPLOYMENT_ID}.apm.${ELASTIC_REGION}.cloud.es.io"
      OTEL_SERVICE_NAME: "databrain-api"
      OTEL_EXPORTER_OTLP_HEADERS: "Authorization=Bearer ${ELASTIC_APM_SECRET_TOKEN}"
      OTEL_RESOURCE_ATTRIBUTES: "deployment.environment=production,service.version=1.0.0"
      LOG_LEVEL: "info"
Store credentials in .env: 
# .env
ELASTIC_DEPLOYMENT_ID=abc123def456
ELASTIC_REGION=us-east-1
ELASTIC_APM_SECRET_TOKEN=your_secret_token_here

​
Option 2: Self-Hosted Elastic Stack

​
1. Deploy Elastic Stack with Docker Compose

version: '3.8'

services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.11.0
    container_name: elasticsearch
    environment:
      - discovery.type=single-node
      - xpack.security.enabled=true
      - ELASTIC_PASSWORD=changeme
      - "ES_JAVA_OPTS=-Xms2g -Xmx2g"
    volumes:
      - elasticsearch-data:/usr/share/elasticsearch/data
    ports:
      - "9200:9200"
    networks:
      - elastic
    restart: unless-stopped

  kibana:
    image: docker.elastic.co/kibana/kibana:8.11.0
    container_name: kibana
    environment:
      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
      - ELASTICSEARCH_USERNAME=elastic
      - ELASTICSEARCH_PASSWORD=changeme
    ports:
      - "5601:5601"
    networks:
      - elastic
    depends_on:
      - elasticsearch
    restart: unless-stopped

  apm-server:
    image: docker.elastic.co/apm/apm-server:8.11.0
    container_name: apm-server
    command: >
      apm-server -e
        -E apm-server.rum.enabled=true
        -E setup.kibana.host=kibana:5601
        -E setup.template.settings.index.number_of_replicas=0
        -E apm-server.kibana.enabled=true
        -E apm-server.kibana.host=kibana:5601
        -E output.elasticsearch.hosts=["elasticsearch:9200"]
        -E output.elasticsearch.username=elastic
        -E output.elasticsearch.password=changeme
        -E apm-server.secret_token=my_secret_token
        -E apm-server.auth.secret_token=my_secret_token
    ports:
      - "8200:8200"
    networks:
      - elastic
      - databrain
    depends_on:
      - elasticsearch
      - kibana
    restart: unless-stopped

  filebeat:
    image: docker.elastic.co/beats/filebeat:8.11.0
    container_name: filebeat
    user: root
    volumes:
      - ./filebeat.yml:/usr/share/filebeat/filebeat.yml:ro
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
    networks:
      - elastic
    depends_on:
      - elasticsearch
    restart: unless-stopped

networks:
  elastic:
    driver: bridge
  databrain:
    external: true

volumes:
  elasticsearch-data:

​
2. Configure APM Server for OpenTelemetry

Create apm-server.yml: 
apm-server:
  host: "0.0.0.0:8200"
  
  # OpenTelemetry configuration
  rum:
    enabled: true
  
  # Authentication
  auth:
    secret_token: "my_secret_token"
  
  # Kibana integration
  kibana:
    enabled: true
    host: "http://kibana:5601"

# Elasticsearch output
output.elasticsearch:
  hosts: ["http://elasticsearch:9200"]
  username: "elastic"
  password: "changeme"
  
# Index lifecycle management
setup.ilm:
  enabled: true
  
# Kibana dashboards
setup.dashboards:
  enabled: true

# Logging
logging.level: info
logging.to_files: true

​
3. Configure Filebeat for Log Ingestion

Create filebeat.yml: 
filebeat.inputs:
  - type: container
    paths:
      - '/var/lib/docker/containers/*/*.log'
    processors:
      - add_docker_metadata:
          host: "unix:///var/run/docker.sock"
      - decode_json_fields:
          fields: ["message"]
          target: ""
          overwrite_keys: true

output.elasticsearch:
  hosts: ["http://elasticsearch:9200"]
  username: "elastic"
  password: "changeme"

setup.kibana:
  host: "http://kibana:5601"

​
4. Start Elastic Stack

docker compose up -d

# Wait for Elasticsearch to be ready
docker logs elasticsearch -f

# Initialize Kibana (first time only)
docker exec -it kibana curl -X POST \
  "http://localhost:5601/api/saved_objects/_bulk_get" \
  -H "kbn-xsrf: true" \
  -H "Content-Type: application/json"

​
5. Configure Databrain for Self-Hosted APM

services:
  databrainbackend:
    environment:
      OTEL_ENABLED: "true"
      OTEL_EXPORTER_OTLP_ENDPOINT: "http://apm-server:8200"
      OTEL_SERVICE_NAME: "databrain-api"
      OTEL_EXPORTER_OTLP_HEADERS: "Authorization=Bearer my_secret_token"
      LOG_LEVEL: "info"
    networks:
      - databrain
      - elastic

​
Option 3: Using Elastic APM Agent (Alternative)

If you prefer the native Elastic APM agent over OpenTelemetry: 
npm install elastic-apm-node --save
Then configure in your application: 
// src/index.ts (at the very top, before other imports)
import apm from 'elastic-apm-node';

if (process.env.ELASTIC_APM_ACTIVE === 'true') {
  apm.start({
    serviceName: process.env.ELASTIC_APM_SERVICE_NAME || 'databrain-api',
    serverUrl: process.env.ELASTIC_APM_SERVER_URL,
    secretToken: process.env.ELASTIC_APM_SECRET_TOKEN,
    environment: process.env.ELASTIC_APM_ENVIRONMENT || 'production',
    logLevel: 'info',
  });
}

// ... rest of your application imports and code
Environment variables: 
ELASTIC_APM_ACTIVE=true
ELASTIC_APM_SERVER_URL=http://apm-server:8200
ELASTIC_APM_SECRET_TOKEN=my_secret_token
ELASTIC_APM_SERVICE_NAME=databrain-api
ELASTIC_APM_ENVIRONMENT=production

​
Kubernetes Deployment

​
Deploy Elastic Cloud on Kubernetes (ECK)

# Install ECK operator
kubectl create -f https://download.elastic.co/downloads/eck/2.10.0/crds.yaml
kubectl apply -f https://download.elastic.co/downloads/eck/2.10.0/operator.yaml

# Deploy Elasticsearch
cat <<EOF | kubectl apply -f -
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: databrain-es
  namespace: elastic-system
spec:
  version: 8.11.0
  nodeSets:
  - name: default
    count: 3
    config:
      node.store.allow_mmap: false
    volumeClaimTemplates:
    - metadata:
        name: elasticsearch-data
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 50Gi
EOF

# Deploy Kibana
cat <<EOF | kubectl apply -f -
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: databrain-kibana
  namespace: elastic-system
spec:
  version: 8.11.0
  count: 1
  elasticsearchRef:
    name: databrain-es
EOF

# Deploy APM Server
cat <<EOF | kubectl apply -f -
apiVersion: apm.k8s.elastic.co/v1
kind: ApmServer
metadata:
  name: databrain-apm
  namespace: elastic-system
spec:
  version: 8.11.0
  count: 1
  elasticsearchRef:
    name: databrain-es
  kibanaRef:
    name: databrain-kibana
EOF

​
Configure Databrain on Kubernetes

apiVersion: v1
kind: Secret
metadata:
  name: elastic-apm-secret
  namespace: databrain
type: Opaque
stringData:
  secret-token: "my_secret_token"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: databrain-backend
  namespace: databrain
spec:
  template:
    spec:
      containers:
      - name: backend
        env:
          - name: OTEL_ENABLED
            value: "true"
          - name: OTEL_EXPORTER_OTLP_ENDPOINT
            value: "http://databrain-apm-apm-http.elastic-system.svc.cluster.local:8200"
          - name: OTEL_SERVICE_NAME
            value: "databrain-api"
          - name: ELASTIC_APM_SECRET_TOKEN
            valueFrom:
              secretKeyRef:
                name: elastic-apm-secret
                key: secret-token
          - name: OTEL_EXPORTER_OTLP_HEADERS
            value: "Authorization=Bearer $(ELASTIC_APM_SECRET_TOKEN)"

​
What Gets Sent to Elastic APM

Telemetry TypeElastic ProductDescription
TracesAPMDistributed traces with transaction and span details
MetricsAPMService metrics, system metrics, custom metrics
LogsElasticsearchStructured logs with APM correlation
ErrorsAPMError tracking with stack traces

​
Verification

​
1. Check APM Server Status

# Check if APM Server is running
curl http://localhost:8200/

# Should return APM Server info

​
2. Generate Test Traffic

curl https://your-databrain-instance.com/api/health

​
3. View in Kibana

  1. Open Kibana: http://localhost:5601
  2. Navigate to Observability → APM → Services
  3. You should see databrain-api in the services list
  4. Click on the service to see:
    • Transaction duration: Latency distribution
    • Throughput: Requests per minute
    • Error rate: Percentage of failed requests
    • Transactions: Individual endpoints
    • Dependencies: External services (DB, Redis, etc.)

​
4. View Traces

  1. Click on Transactions tab
  2. Select an endpoint (e.g., /api/v2/metric/execute)
  3. Click on a specific transaction to see the trace waterfall

​
5. View Logs with APM Correlation

  1. Go to Observability → Logs → Stream
  2. Filter: service.name: databrain-api
  3. Click on any log entry
  4. Click View in APM to see the related trace

​
Creating Kibana Dashboards

​
Service Overview Dashboard

  1. Go to Dashboard → Create dashboard
  2. Add visualizations:
Request Rate (Lens):
  • Index pattern: apm-*
  • Metric: Count of processor.event: transaction
  • Time range: Last 15 minutes
  • Breakdown: service.name
Average Latency:
  • Metric: Average of transaction.duration.us
  • Filter: service.name: databrain-api
Error Rate:
  • Formula: (count where transaction.result = error) / count * 100
  • Filter: service.name: databrain-api
Top Transactions by Duration:
  • Metric: P95 of transaction.duration.us
  • Group by: transaction.name
  • Top 10

​
Database Performance Dashboard

Slow Queries:
  • Filter: span.type: db AND span.subtype: postgresql
  • Metric: P95 of span.duration.us
  • Group by: span.db.statement

​
Setting Up Alerts

​
Create Alert Rules in Kibana

  1. Go to Observability → Alerts → Create rule

​
High Error Rate Alert

  • Rule type: APM
  • Service: databrain-api
  • Alert when: Transaction error rate is above threshold
  • Threshold: > 5% for 5 minutes
  • Actions: Send to Slack/Email/PagerDuty

​
High Latency Alert

  • Rule type: APM
  • Service: databrain-api
  • Transaction type: request
  • Alert when: Latency threshold is exceeded
  • Threshold: P95 > 2000ms for 5 minutes

​
Service Down Alert

  • Rule type: Uptime
  • Monitor: databrain-api
  • Alert when: Service is down
  • Check frequency: Every 1 minute

​
Machine Learning (X-Pack)

If you have an Elastic license, enable ML anomaly detection:
  1. Go to Machine Learning → Anomaly Detection
  2. Create job for APM:
    • Job type: APM
    • Service: databrain-api
    • Analyze: Response times and throughput
  3. ML will automatically detect:
    • Unusual latency spikes
    • Traffic anomalies
    • Error rate changes

​
Troubleshooting

IssueSolution
No data in KibanaCheck APM Server logs: docker logs apm-server
401 UnauthorizedVerify secret token matches between Databrain and APM Server
Connection refusedCheck network connectivity and firewall rules
Missing logsEnsure Filebeat is running and configured correctly
High disk usageConfigure ILM (Index Lifecycle Management) policies

​
Debug APM Server

# Check APM Server logs
docker logs apm-server -f

# Check Elasticsearch connectivity
docker exec apm-server curl -u elastic:changeme http://elasticsearch:9200/_cluster/health

# Verify APM Server is receiving data
curl -H "Authorization: Bearer my_secret_token" \
  http://localhost:8200/intake/v2/rum/events

​
Check Elasticsearch Indices

# List APM indices
curl -u elastic:changeme http://localhost:9200/_cat/indices/apm-*

# Check document count
curl -u elastic:changeme http://localhost:9200/apm-*/_count

​
Index Lifecycle Management (ILM)

Configure ILM to manage disk usage:
  1. Go to Stack Management → Index Lifecycle Policies
  2. Edit APM policies:
APM Traces (apm--transaction, apm--span):
  • Hot phase: 1 day
  • Warm phase: 7 days
  • Delete phase: 30 days
APM Metrics:
  • Hot phase: 3 days
  • Warm phase: 14 days
  • Delete phase: 90 days
APM Errors:
  • Hot phase: 7 days
  • Warm phase: 30 days
  • Delete phase: 90 days

​
Best Practices

​
1. Use Index Templates

Customize APM index templates for better performance: 
PUT _index_template/apm-custom
{
  "index_patterns": ["apm-*"],
  "priority": 500,
  "template": {
    "settings": {
      "number_of_shards": 3,
      "number_of_replicas": 1,
      "refresh_interval": "5s"
    }
  }
}
For multi-region deployments, use cross-cluster search to aggregate data.

​
3. Use APM Correlations

Let Elastic automatically find correlations between slow transactions and attributes:
  1. Go to APM → databrain-api → Latency correlations
  2. View attributes that correlate with slow requests

​
Cost Optimization

For Elastic Cloud:
  • Optimize data retention: Use ILM to delete old data
  • Reduce sampling: Not all traces need to be kept
  • Use smaller instance sizes: Start small, scale as needed
For self-hosted:
  • Use hot-warm architecture: Move old data to cheaper storage
  • Enable compression: Reduce storage by 50-70%
  • Optimize shard size: 20-50GB per shard is ideal

​
Support

For Databrain configuration issues, contact your Databrain support team.