Integrating Databrain with Honeycomb

This guide explains how to send OpenTelemetry traces and events from your self-hosted Databrain instance to Honeycomb.

Why Honeycomb?

Honeycomb is purpose-built for OpenTelemetry and excels at:

High-cardinality data: Query on any attribute combination
BubbleUp: Automatically surface anomalies and outliers
Distributed tracing: Full request lifecycle visibility
Real-time analysis: Sub-second query performance

Prerequisites

Databrain self-hosted version with OpenTelemetry support
Honeycomb account (free tier available)
Honeycomb API key

Configuration

1. Get Your Honeycomb API Key

Log into Honeycomb
Go to Account → Team Settings → API Keys
Create a new API key or copy an existing one
Note your dataset name (or create a new one)

2. Configure Databrain Environment Variables

Honeycomb supports OpenTelemetry natively, making configuration simple:

# Enable OpenTelemetry
OTEL_ENABLED=true

# Honeycomb OTLP endpoint
OTEL_EXPORTER_OTLP_ENDPOINT=https://api.honeycomb.io:443

# Honeycomb API key
OTEL_EXPORTER_OTLP_HEADERS=x-honeycomb-team=YOUR_API_KEY_HERE

# Service name (appears in Honeycomb)
OTEL_SERVICE_NAME=databrain-api

# Optional: Honeycomb dataset (defaults to service name)
HONEYCOMB_DATASET=databrain-production

# Optional: Enable debug logging
LOG_LEVEL=info

3. Docker Compose Configuration

Update your docker-compose.yml:

services:
  databrainbackend:
    environment:
      OTEL_ENABLED: "true"
      OTEL_EXPORTER_OTLP_ENDPOINT: "https://api.honeycomb.io:443"
      OTEL_SERVICE_NAME: "databrain-api"
      OTEL_EXPORTER_OTLP_HEADERS: "x-honeycomb-team=${HONEYCOMB_API_KEY}"
      HONEYCOMB_DATASET: "databrain-production"
      LOG_LEVEL: "info"

Security: Store your API key in .env:

# .env
HONEYCOMB_API_KEY=your_honeycomb_api_key_here

4. Kubernetes Configuration

apiVersion: v1
kind: Secret
metadata:
  name: honeycomb-secret
type: Opaque
stringData:
  api-key: your_honeycomb_api_key_here
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: databrain-backend
spec:
  template:
    spec:
      containers:
      - name: backend
        env:
          - name: OTEL_ENABLED
            value: "true"
          - name: OTEL_EXPORTER_OTLP_ENDPOINT
            value: "https://api.honeycomb.io:443"
          - name: OTEL_SERVICE_NAME
            value: "databrain-api"
          - name: HONEYCOMB_API_KEY
            valueFrom:
              secretKeyRef:
                name: honeycomb-secret
                key: api-key
          - name: OTEL_EXPORTER_OTLP_HEADERS
            value: "x-honeycomb-team=$(HONEYCOMB_API_KEY)"
          - name: HONEYCOMB_DATASET
            value: "databrain-production"

Alternative: Using OpenTelemetry Collector

For advanced features like sampling and enrichment:

Docker Compose with Collector

services:
  otel-collector:
    image: otel/opentelemetry-collector-contrib:latest
    command: ["--config=/etc/otel-collector-config.yaml"]
    volumes:
      - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
    ports:
      - "4317:4317"  # OTLP gRPC
      - "4318:4318"  # OTLP HTTP
    environment:
      HONEYCOMB_API_KEY: "${HONEYCOMB_API_KEY}"
      HONEYCOMB_DATASET: "databrain-production"
    networks:
      - databrain

  databrainbackend:
    environment:
      OTEL_ENABLED: "true"
      OTEL_EXPORTER_OTLP_ENDPOINT: "http://otel-collector:4318"
      OTEL_SERVICE_NAME: "databrain-api"
    depends_on:
      - otel-collector

Collector Configuration for Honeycomb

Create otel-collector-config.yaml:

receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  batch:
    timeout: 1s
    send_batch_size: 1024
  
  # Add resource attributes
  resource:
    attributes:
      - key: deployment.environment
        value: production
        action: upsert
      - key: service.version
        value: 1.0.0
        action: upsert
  
  # Tail-based sampling (only send interesting traces)
  tail_sampling:
    decision_wait: 10s
    num_traces: 100
    policies:
      # Always sample errors
      - name: errors
        type: status_code
        status_code:
          status_codes: [ERROR]
      
      # Sample slow requests
      - name: slow-requests
        type: latency
        latency:
          threshold_ms: 1000
      
      # Probabilistic sampling for everything else
      - name: baseline
        type: probabilistic
        probabilistic:
          sampling_percentage: 10

exporters:
  otlp:
    endpoint: "api.honeycomb.io:443"
    headers:
      "x-honeycomb-team": "${HONEYCOMB_API_KEY}"
      "x-honeycomb-dataset": "${HONEYCOMB_DATASET}"

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch, resource, tail_sampling]
      exporters: [otlp]
    
    metrics:
      receivers: [otlp]
      processors: [batch, resource]
      exporters: [otlp]

What Gets Sent to Honeycomb

Data Type	Description
Traces	Full request traces with all spans and timing
Spans	Individual operations (DB queries, API calls, etc.)
Events	Structured logs that appear as trace events
Metrics	Request rates, latencies, error counts

Verification

1. Restart Databrain

docker compose restart databrainbackend
# or
kubectl rollout restart deployment/databrain-backend

2. Generate Test Traffic

# Simple health check
curl https://your-databrain-instance.com/api/health

# More complex request
curl -X POST "https://your-databrain-instance.com/api/v2/metric/execute" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -d '{"metricId": "test-123"}'

3. View in Honeycomb

Go to Honeycomb UI
Select your dataset (e.g., databrain-production)
You should see traces within seconds (Honeycomb is real-time!)
Click on Recent Traces to explore

4. Check Backend Logs

Look for:

{
  "level": "info",
  "message": "[Telemetry] OpenTelemetry initialized - service: databrain-api, endpoint: https://api.honeycomb.io:443"
}

Using Honeycomb Features

1. BubbleUp - Find Anomalies

BubbleUp automatically identifies what makes slow or failing requests different:

In Honeycomb, go to your dataset
Click New Query
Filter: WHERE service.name = databrain-api
Visualize: HEATMAP(duration_ms)
Click on slow requests cluster
Click BubbleUp → Honeycomb shows which attributes correlate with slowness

2. Tracing Waterfall

View full request lifecycle:

Click on any trace
See the waterfall view with all spans
Identify bottlenecks (database queries, external APIs)
Click on spans to see attributes

3. High-Cardinality Queries

Query on any attribute combination:

WHERE service.name = databrain-api
  AND http.status_code >= 500
  AND userId EXISTS
GROUP BY userId, http.route
CALCULATE COUNT, P95(duration_ms)

4. Time-based Comparison

Compare performance across time periods:

Set time range (e.g., last hour)
Click Compare to → Previous period
See what changed (requests, latency, errors)

5. Service Map

Visualize service dependencies:

Go to Service Map (if enabled)
See how databrain-api connects to PostgreSQL, Redis, Hasura, etc.
Click on connections to see request rates and error rates

Example Queries

Slowest Endpoints

WHERE service.name = databrain-api
  AND span.kind = server
GROUP BY http.route
CALCULATE P95(duration_ms), COUNT
ORDER BY P95(duration_ms) DESC

Error Analysis

WHERE service.name = databrain-api
  AND status.code = ERROR
GROUP BY error.type, http.route
CALCULATE COUNT

User Experience Analysis

WHERE service.name = databrain-api
  AND userId EXISTS
GROUP BY userId
CALCULATE P95(duration_ms), COUNT
ORDER BY COUNT DESC

Database Query Performance

WHERE service.name = databrain-api
  AND db.system = postgresql
GROUP BY db.statement
CALCULATE P95(duration_ms), COUNT
ORDER BY P95(duration_ms) DESC

Compare Error Rates Over Time

WHERE service.name = databrain-api
GROUP BY status.code
CALCULATE COUNT
COMPARE TO 1 hour ago

Creating Triggers (Alerts)

Set up alerts in Honeycomb:

High Error Rate

Go to Triggers → New Trigger
Query:

WHERE service.name = databrain-api
  AND status.code = ERROR
CALCULATE COUNT

Threshold: Alert when COUNT > 10 per minute
Add recipient (email, Slack, PagerDuty)

High Latency

WHERE service.name = databrain-api
  AND span.kind = server
CALCULATE P95(duration_ms)

Threshold: Alert when P95(duration_ms) > 2000

Anomaly Detection

Use Honeycomb’s Anomaly Detection trigger:

Automatically learns normal behavior
Alerts on statistical anomalies
No manual threshold setting required

Boards (Dashboards)

Create custom boards in Honeycomb:

Service Health Board

Request Rate

WHERE service.name = databrain-api
CALCULATE COUNT

Error Rate

WHERE service.name = databrain-api
CALCULATE COUNT(status.code = ERROR) / COUNT

Latency Heatmap

WHERE service.name = databrain-api
HEATMAP(duration_ms)

Top Endpoints by Traffic

WHERE service.name = databrain-api
GROUP BY http.route
CALCULATE COUNT

Database Performance Board

Query Latency

WHERE db.system = postgresql
HEATMAP(duration_ms)

Slowest Queries

WHERE db.system = postgresql
GROUP BY db.statement
CALCULATE P95(duration_ms)
ORDER BY P95(duration_ms) DESC
LIMIT 10

Markers (Deploy Events)

Track deployments with markers:

# After deploying
curl https://api.honeycomb.io/1/markers/databrain-production \
  -X POST \
  -H "X-Honeycomb-Team: ${HONEYCOMB_API_KEY}" \
  -d '{
    "message": "Deployed v1.2.3",
    "type": "deploy",
    "url": "https://github.com/yourorg/databrain/releases/v1.2.3"
  }'

Markers appear as vertical lines in time-series graphs, making it easy to correlate performance changes with deployments.

SLOs (Service Level Objectives)

Create SLOs in Honeycomb:

Go to SLOs → New SLO
Define SLI (Service Level Indicator):

Example: Availability SLO (99.9%)

Denominator: COUNT (all requests)
WHERE service.name = databrain-api

Numerator: COUNT (successful requests)
WHERE service.name = databrain-api
  AND http.status_code < 500

Set target: 99.9% over 30 days
Track SLO compliance and error budgets

Troubleshooting

Issue	Solution
No data in Honeycomb	Verify API key is correct and has write permissions
401 Unauthorized	Check `x-honeycomb-team` header contains valid API key
Data in wrong dataset	Set `HONEYCOMB_DATASET` explicitly
Missing attributes	Ensure Winston logger includes metadata in logs
High ingestion costs	Implement tail-based sampling in collector

Debug Mode

Enable verbose logging:

LOG_LEVEL=debug
OTEL_LOG_LEVEL=debug

Check logs for export confirmations.

Test API Key

curl https://api.honeycomb.io/1/auth \
  -H "X-Honeycomb-Team: ${HONEYCOMB_API_KEY}"

Should return team information if valid.

Cost Optimization

Honeycomb pricing is based on event volume (spans/events per month). Free tier: 20M events/month Optimization strategies:

Tail-based sampling: Only send interesting traces (errors, slow requests)
Filter health checks: Exclude /health endpoint
Sample high-throughput endpoints: Sample 10% of common endpoints
Set retention: Adjust in Honeycomb settings (default 60 days)

Sampling Configuration

In collector config:

processors:
  tail_sampling:
    policies:
      - name: errors
        type: status_code
        status_code: {status_codes: [ERROR]}
      
      - name: slow
        type: latency
        latency: {threshold_ms: 1000}
      
      - name: sample-baseline
        type: probabilistic
        probabilistic: {sampling_percentage: 5}

This keeps:

100% of errors
100% of slow requests (>1s)
5% of everything else

Best Practices

1. Use Wide Events

Include rich context in your logs:

logger.info('Order processed', {
  orderId: '12345',
  userId: 'user-789',
  amount: 99.99,
  currency: 'USD',
  paymentMethod: 'credit_card',
  processingTime: 245,
  region: 'us-east-1'
});

All these attributes become queryable in Honeycomb.

2. Add Custom Instrumentation

For critical business logic, add custom spans:

import { trace } from '@opentelemetry/api';

const tracer = trace.getTracer('databrain-business-logic');

async function processOrder(orderId: string) {
  const span = tracer.startSpan('processOrder');
  span.setAttribute('orderId', orderId);
  
  try {
    // Your business logic
    span.addEvent('Order validated');
    
    // ...
    
    span.setStatus({ code: SpanStatusCode.OK });
  } catch (error) {
    span.recordException(error);
    span.setStatus({ code: SpanStatusCode.ERROR });
    throw error;
  } finally {
    span.end();
  }
}

3. Use Derived Columns

Create computed columns in Honeycomb:

Error rate: COUNT_DISTINCT(status.code = ERROR) / COUNT_DISTINCT(trace.id)
Apdex score: Custom formula for user satisfaction
Business KPIs: Revenue per request, conversion rate, etc.

Support

Honeycomb Documentation: https://docs.honeycomb.io/
Honeycomb Community: https://www.honeycomb.io/community
Honeycomb Support: Available via in-app chat or email

For Databrain configuration issues, contact your Databrain support team.

​Integrating Databrain with Honeycomb

​Why Honeycomb?

​Prerequisites

​Configuration

​1. Get Your Honeycomb API Key

​2. Configure Databrain Environment Variables

​3. Docker Compose Configuration

​4. Kubernetes Configuration

​Alternative: Using OpenTelemetry Collector

​Docker Compose with Collector

​Collector Configuration for Honeycomb

​What Gets Sent to Honeycomb

​Verification

​1. Restart Databrain

​2. Generate Test Traffic

​3. View in Honeycomb

​4. Check Backend Logs

​Using Honeycomb Features

​1. BubbleUp - Find Anomalies

​2. Tracing Waterfall

​3. High-Cardinality Queries

​4. Time-based Comparison

​5. Service Map

​Example Queries

​Slowest Endpoints

​Error Analysis

​User Experience Analysis

​Database Query Performance

​Compare Error Rates Over Time

​Creating Triggers (Alerts)

​High Error Rate

​High Latency

​Anomaly Detection

​Boards (Dashboards)

​Service Health Board

​Database Performance Board

​Markers (Deploy Events)

​SLOs (Service Level Objectives)

​Troubleshooting

​Debug Mode

​Test API Key

​Cost Optimization

​Sampling Configuration

​Best Practices

​1. Use Wide Events

​2. Add Custom Instrumentation

​3. Use Derived Columns

​Support