> ## Documentation Index
> Fetch the complete documentation index at: https://docs.usedatabrain.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Honeycomb Integration

> Send OpenTelemetry traces, metrics, and logs from Databrain to Honeycomb

# Integrating Databrain with Honeycomb

This guide explains how to send OpenTelemetry traces and events from your self-hosted Databrain instance to Honeycomb.

## Why Honeycomb?

Honeycomb is purpose-built for OpenTelemetry and excels at:

* **High-cardinality data**: Query on any attribute combination
* **BubbleUp**: Automatically surface anomalies and outliers
* **Distributed tracing**: Full request lifecycle visibility
* **Real-time analysis**: Sub-second query performance

## Prerequisites

* Databrain self-hosted version with OpenTelemetry support
* Honeycomb account (free tier available)
* Honeycomb API key

## Configuration

### 1. Get Your Honeycomb API Key

1. Log into [Honeycomb](https://ui.honeycomb.io/)
2. Go to **Account** → **Team Settings** → **API Keys**
3. Create a new API key or copy an existing one
4. Note your **dataset name** (or create a new one)

### 2. Configure Databrain Environment Variables

Honeycomb supports OpenTelemetry natively, making configuration simple:

```bash theme={"dark"}
# Enable OpenTelemetry
OTEL_ENABLED=true

# Honeycomb OTLP endpoint
OTEL_EXPORTER_OTLP_ENDPOINT=https://api.honeycomb.io:443

# Honeycomb API key
OTEL_EXPORTER_OTLP_HEADERS=x-honeycomb-team=YOUR_API_KEY_HERE

# Service name (appears in Honeycomb)
OTEL_SERVICE_NAME=databrain-api

# Optional: Honeycomb dataset (defaults to service name)
HONEYCOMB_DATASET=databrain-production

# Optional: Enable debug logging
LOG_LEVEL=info
```

### 3. Docker Compose Configuration

Update your `docker-compose.yml`:

```yaml theme={"dark"}
services:
  databrainbackend:
    environment:
      OTEL_ENABLED: "true"
      OTEL_EXPORTER_OTLP_ENDPOINT: "https://api.honeycomb.io:443"
      OTEL_SERVICE_NAME: "databrain-api"
      OTEL_EXPORTER_OTLP_HEADERS: "x-honeycomb-team=${HONEYCOMB_API_KEY}"
      HONEYCOMB_DATASET: "databrain-production"
      LOG_LEVEL: "info"
```

**Security**: Store your API key in `.env`:

```bash theme={"dark"}
# .env
HONEYCOMB_API_KEY=your_honeycomb_api_key_here
```

### 4. Kubernetes Configuration

```yaml theme={"dark"}
apiVersion: v1
kind: Secret
metadata:
  name: honeycomb-secret
type: Opaque
stringData:
  api-key: your_honeycomb_api_key_here
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: databrain-backend
spec:
  template:
    spec:
      containers:
      - name: backend
        env:
          - name: OTEL_ENABLED
            value: "true"
          - name: OTEL_EXPORTER_OTLP_ENDPOINT
            value: "https://api.honeycomb.io:443"
          - name: OTEL_SERVICE_NAME
            value: "databrain-api"
          - name: HONEYCOMB_API_KEY
            valueFrom:
              secretKeyRef:
                name: honeycomb-secret
                key: api-key
          - name: OTEL_EXPORTER_OTLP_HEADERS
            value: "x-honeycomb-team=$(HONEYCOMB_API_KEY)"
          - name: HONEYCOMB_DATASET
            value: "databrain-production"
```

## Alternative: Using OpenTelemetry Collector

For advanced features like sampling and enrichment:

### Docker Compose with Collector

```yaml theme={"dark"}
services:
  otel-collector:
    image: otel/opentelemetry-collector-contrib:latest
    command: ["--config=/etc/otel-collector-config.yaml"]
    volumes:
      - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
    ports:
      - "4317:4317"  # OTLP gRPC
      - "4318:4318"  # OTLP HTTP
    environment:
      HONEYCOMB_API_KEY: "${HONEYCOMB_API_KEY}"
      HONEYCOMB_DATASET: "databrain-production"
    networks:
      - databrain

  databrainbackend:
    environment:
      OTEL_ENABLED: "true"
      OTEL_EXPORTER_OTLP_ENDPOINT: "http://otel-collector:4318"
      OTEL_SERVICE_NAME: "databrain-api"
    depends_on:
      - otel-collector
```

### Collector Configuration for Honeycomb

Create `otel-collector-config.yaml`:

```yaml theme={"dark"}
receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  batch:
    timeout: 1s
    send_batch_size: 1024
  
  # Add resource attributes
  resource:
    attributes:
      - key: deployment.environment
        value: production
        action: upsert
      - key: service.version
        value: 1.0.0
        action: upsert
  
  # Tail-based sampling (only send interesting traces)
  tail_sampling:
    decision_wait: 10s
    num_traces: 100
    policies:
      # Always sample errors
      - name: errors
        type: status_code
        status_code:
          status_codes: [ERROR]
      
      # Sample slow requests
      - name: slow-requests
        type: latency
        latency:
          threshold_ms: 1000
      
      # Probabilistic sampling for everything else
      - name: baseline
        type: probabilistic
        probabilistic:
          sampling_percentage: 10

exporters:
  otlp:
    endpoint: "api.honeycomb.io:443"
    headers:
      "x-honeycomb-team": "${HONEYCOMB_API_KEY}"
      "x-honeycomb-dataset": "${HONEYCOMB_DATASET}"

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch, resource, tail_sampling]
      exporters: [otlp]
    
    metrics:
      receivers: [otlp]
      processors: [batch, resource]
      exporters: [otlp]
```

## What Gets Sent to Honeycomb

| Data Type   | Description                                         |
| ----------- | --------------------------------------------------- |
| **Traces**  | Full request traces with all spans and timing       |
| **Spans**   | Individual operations (DB queries, API calls, etc.) |
| **Events**  | Structured logs that appear as trace events         |
| **Metrics** | Request rates, latencies, error counts              |

## Verification

### 1. Restart Databrain

```bash theme={"dark"}
docker compose restart databrainbackend
# or
kubectl rollout restart deployment/databrain-backend
```

### 2. Generate Test Traffic

```bash theme={"dark"}
# Simple health check
curl https://your-databrain-instance.com/api/health

# More complex request
curl -X POST "https://your-databrain-instance.com/api/v2/metric/execute" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -d '{"metricId": "test-123"}'
```

### 3. View in Honeycomb

1. Go to [Honeycomb UI](https://ui.honeycomb.io/)
2. Select your dataset (e.g., `databrain-production`)
3. You should see traces within seconds (Honeycomb is real-time!)
4. Click on **Recent Traces** to explore

### 4. Check Backend Logs

Look for:

```json theme={"dark"}
{
  "level": "info",
  "message": "[Telemetry] OpenTelemetry initialized - service: databrain-api, endpoint: https://api.honeycomb.io:443"
}
```

## Using Honeycomb Features

### 1. BubbleUp - Find Anomalies

BubbleUp automatically identifies what makes slow or failing requests different:

1. In Honeycomb, go to your dataset
2. Click **New Query**
3. Filter: `WHERE service.name = databrain-api`
4. Visualize: `HEATMAP(duration_ms)`
5. Click on slow requests cluster
6. Click **BubbleUp** → Honeycomb shows which attributes correlate with slowness

### 2. Tracing Waterfall

View full request lifecycle:

1. Click on any trace
2. See the waterfall view with all spans
3. Identify bottlenecks (database queries, external APIs)
4. Click on spans to see attributes

### 3. High-Cardinality Queries

Query on any attribute combination:

```
WHERE service.name = databrain-api
  AND http.status_code >= 500
  AND userId EXISTS
GROUP BY userId, http.route
CALCULATE COUNT, P95(duration_ms)
```

### 4. Time-based Comparison

Compare performance across time periods:

1. Set time range (e.g., last hour)
2. Click **Compare to** → **Previous period**
3. See what changed (requests, latency, errors)

### 5. Service Map

Visualize service dependencies:

1. Go to **Service Map** (if enabled)
2. See how databrain-api connects to PostgreSQL, Redis, Hasura, etc.
3. Click on connections to see request rates and error rates

## Example Queries

### Slowest Endpoints

```
WHERE service.name = databrain-api
  AND span.kind = server
GROUP BY http.route
CALCULATE P95(duration_ms), COUNT
ORDER BY P95(duration_ms) DESC
```

### Error Analysis

```
WHERE service.name = databrain-api
  AND status.code = ERROR
GROUP BY error.type, http.route
CALCULATE COUNT
```

### User Experience Analysis

```
WHERE service.name = databrain-api
  AND userId EXISTS
GROUP BY userId
CALCULATE P95(duration_ms), COUNT
ORDER BY COUNT DESC
```

### Database Query Performance

```
WHERE service.name = databrain-api
  AND db.system = postgresql
GROUP BY db.statement
CALCULATE P95(duration_ms), COUNT
ORDER BY P95(duration_ms) DESC
```

### Compare Error Rates Over Time

```
WHERE service.name = databrain-api
GROUP BY status.code
CALCULATE COUNT
COMPARE TO 1 hour ago
```

## Creating Triggers (Alerts)

Set up alerts in Honeycomb:

### High Error Rate

1. Go to **Triggers** → **New Trigger**
2. Query:

```
WHERE service.name = databrain-api
  AND status.code = ERROR
CALCULATE COUNT
```

3. Threshold: Alert when `COUNT > 10` per minute
4. Add recipient (email, Slack, PagerDuty)

### High Latency

```
WHERE service.name = databrain-api
  AND span.kind = server
CALCULATE P95(duration_ms)
```

Threshold: Alert when `P95(duration_ms) > 2000`

### Anomaly Detection

Use Honeycomb's **Anomaly Detection** trigger:

* Automatically learns normal behavior
* Alerts on statistical anomalies
* No manual threshold setting required

## Boards (Dashboards)

Create custom boards in Honeycomb:

### Service Health Board

1. **Request Rate**

```
WHERE service.name = databrain-api
CALCULATE COUNT
```

2. **Error Rate**

```
WHERE service.name = databrain-api
CALCULATE COUNT(status.code = ERROR) / COUNT
```

3. **Latency Heatmap**

```
WHERE service.name = databrain-api
HEATMAP(duration_ms)
```

4. **Top Endpoints by Traffic**

```
WHERE service.name = databrain-api
GROUP BY http.route
CALCULATE COUNT
```

### Database Performance Board

1. **Query Latency**

```
WHERE db.system = postgresql
HEATMAP(duration_ms)
```

2. **Slowest Queries**

```
WHERE db.system = postgresql
GROUP BY db.statement
CALCULATE P95(duration_ms)
ORDER BY P95(duration_ms) DESC
LIMIT 10
```

## Markers (Deploy Events)

Track deployments with markers:

```bash theme={"dark"}
# After deploying
curl https://api.honeycomb.io/1/markers/databrain-production \
  -X POST \
  -H "X-Honeycomb-Team: ${HONEYCOMB_API_KEY}" \
  -d '{
    "message": "Deployed v1.2.3",
    "type": "deploy",
    "url": "https://github.com/yourorg/databrain/releases/v1.2.3"
  }'
```

Markers appear as vertical lines in time-series graphs, making it easy to correlate performance changes with deployments.

## SLOs (Service Level Objectives)

Create SLOs in Honeycomb:

1. Go to **SLOs** → **New SLO**
2. Define SLI (Service Level Indicator):

**Example: Availability SLO (99.9%)**

```
Denominator: COUNT (all requests)
WHERE service.name = databrain-api

Numerator: COUNT (successful requests)
WHERE service.name = databrain-api
  AND http.status_code < 500
```

3. Set target: 99.9% over 30 days
4. Track SLO compliance and error budgets

## Troubleshooting

| Issue                 | Solution                                               |
| --------------------- | ------------------------------------------------------ |
| No data in Honeycomb  | Verify API key is correct and has write permissions    |
| 401 Unauthorized      | Check `x-honeycomb-team` header contains valid API key |
| Data in wrong dataset | Set `HONEYCOMB_DATASET` explicitly                     |
| Missing attributes    | Ensure Winston logger includes metadata in logs        |
| High ingestion costs  | Implement tail-based sampling in collector             |

### Debug Mode

Enable verbose logging:

```bash theme={"dark"}
LOG_LEVEL=debug
OTEL_LOG_LEVEL=debug
```

Check logs for export confirmations.

### Test API Key

```bash theme={"dark"}
curl https://api.honeycomb.io/1/auth \
  -H "X-Honeycomb-Team: ${HONEYCOMB_API_KEY}"
```

Should return team information if valid.

## Cost Optimization

Honeycomb pricing is based on **event volume** (spans/events per month).

**Free tier**: 20M events/month

**Optimization strategies**:

1. **Tail-based sampling**: Only send interesting traces (errors, slow requests)
2. **Filter health checks**: Exclude `/health` endpoint
3. **Sample high-throughput endpoints**: Sample 10% of common endpoints
4. **Set retention**: Adjust in Honeycomb settings (default 60 days)

### Sampling Configuration

In collector config:

```yaml theme={"dark"}
processors:
  tail_sampling:
    policies:
      - name: errors
        type: status_code
        status_code: {status_codes: [ERROR]}
      
      - name: slow
        type: latency
        latency: {threshold_ms: 1000}
      
      - name: sample-baseline
        type: probabilistic
        probabilistic: {sampling_percentage: 5}
```

This keeps:

* 100% of errors
* 100% of slow requests (>1s)
* 5% of everything else

## Best Practices

### 1. Use Wide Events

Include rich context in your logs:

```typescript theme={"dark"}
logger.info('Order processed', {
  orderId: '12345',
  userId: 'user-789',
  amount: 99.99,
  currency: 'USD',
  paymentMethod: 'credit_card',
  processingTime: 245,
  region: 'us-east-1'
});
```

All these attributes become queryable in Honeycomb.

### 2. Add Custom Instrumentation

For critical business logic, add custom spans:

```typescript theme={"dark"}
import { trace } from '@opentelemetry/api';

const tracer = trace.getTracer('databrain-business-logic');

async function processOrder(orderId: string) {
  const span = tracer.startSpan('processOrder');
  span.setAttribute('orderId', orderId);
  
  try {
    // Your business logic
    span.addEvent('Order validated');
    
    // ...
    
    span.setStatus({ code: SpanStatusCode.OK });
  } catch (error) {
    span.recordException(error);
    span.setStatus({ code: SpanStatusCode.ERROR });
    throw error;
  } finally {
    span.end();
  }
}
```

### 3. Use Derived Columns

Create computed columns in Honeycomb:

* **Error rate**: `COUNT_DISTINCT(status.code = ERROR) / COUNT_DISTINCT(trace.id)`
* **Apdex score**: Custom formula for user satisfaction
* **Business KPIs**: Revenue per request, conversion rate, etc.

## Support

* **Honeycomb Documentation**: [https://docs.honeycomb.io/](https://docs.honeycomb.io/)
* **Honeycomb Community**: [https://www.honeycomb.io/community](https://www.honeycomb.io/community)
* **Honeycomb Support**: Available via in-app chat or email

For Databrain configuration issues, contact your Databrain support team.