> ## Documentation Index
> Fetch the complete documentation index at: https://docs.usedatabrain.com/llms.txt
> Use this file to discover all available pages before exploring further.

# SigNoz Integration

> Send OpenTelemetry traces, metrics, and logs from Databrain to SigNoz (open-source observability)

# Integrating Databrain with SigNoz

This guide explains how to send OpenTelemetry traces, metrics, and logs from your self-hosted Databrain instance to SigNoz, an open-source observability platform.

## Why SigNoz?

SigNoz is a great choice for self-hosted observability:

* **100% Open Source**: No vendor lock-in
* **OpenTelemetry Native**: Built specifically for OTel
* **All-in-One**: Traces, metrics, and logs in a single platform
* **Cost-Effective**: Self-hosted means no per-event pricing
* **Easy to Deploy**: Docker Compose or Kubernetes deployment

## Prerequisites

* Databrain self-hosted version with OpenTelemetry support
* Docker and Docker Compose (for SigNoz installation)
* 4GB+ RAM for SigNoz (8GB recommended for production)

## Option 1: SigNoz Cloud (Easiest)

If you prefer a managed solution, use SigNoz Cloud:

### 1. Sign Up for SigNoz Cloud

1. Go to [https://signoz.io/teams/](https://signoz.io/teams/)
2. Create a free account
3. Note your **ingestion endpoint** and **ingestion key**

### 2. Configure Databrain

```bash theme={"dark"}
# Enable OpenTelemetry
OTEL_ENABLED=true

# SigNoz Cloud endpoint (adjust for your region)
OTEL_EXPORTER_OTLP_ENDPOINT=https://ingest.{region}.signoz.cloud:443

# SigNoz ingestion key
OTEL_EXPORTER_OTLP_HEADERS=signoz-access-token=YOUR_INGESTION_KEY

# Service name
OTEL_SERVICE_NAME=databrain-api

# Optional: Enable debug logging
LOG_LEVEL=info
```

### 3. Docker Compose Configuration

```yaml theme={"dark"}
services:
  databrainbackend:
    environment:
      OTEL_ENABLED: "true"
      OTEL_EXPORTER_OTLP_ENDPOINT: "https://ingest.{region}.signoz.cloud:443"
      OTEL_SERVICE_NAME: "databrain-api"
      OTEL_EXPORTER_OTLP_HEADERS: "signoz-access-token=${SIGNOZ_INGESTION_KEY}"
      LOG_LEVEL: "info"
```

## Option 2: Self-Hosted SigNoz (Recommended for Production)

### 1. Install SigNoz

#### Using Docker Compose (Quick Start)

```bash theme={"dark"}
# Clone SigNoz repository
git clone -b main https://github.com/SigNoz/signoz.git
cd signoz/deploy/

# Start SigNoz
docker compose -f docker/clickhouse-setup/docker-compose.yaml up -d
```

This starts:

* **SigNoz UI**: [http://localhost:3301](http://localhost:3301)
* **OTLP Receiver (gRPC)**: localhost:4317
* **OTLP Receiver (HTTP)**: localhost:4318
* **ClickHouse**: For data storage
* **Query Service**: For querying data

#### Using Docker Compose (Custom Network)

If Databrain is on the same Docker network:

```yaml theme={"dark"}
# signoz-docker-compose.yaml
version: "3.8"

networks:
  databrain:
    external: true
  signoz:
    name: signoz

services:
  clickhouse:
    image: clickhouse/clickhouse-server:23.7.3-alpine
    container_name: signoz-clickhouse
    volumes:
      - ./clickhouse-config.xml:/etc/clickhouse-server/config.d/config.xml
      - ./clickhouse-users.xml:/etc/clickhouse-server/users.d/users.xml
      - signoz-data:/var/lib/clickhouse/
    networks:
      - signoz
    restart: on-failure

  otel-collector:
    image: signoz/signoz-otel-collector:0.88.11
    container_name: signoz-otel-collector
    command: ["--config=/etc/otel-collector-config.yaml"]
    volumes:
      - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
    ports:
      - "4317:4317"  # OTLP gRPC receiver
      - "4318:4318"  # OTLP HTTP receiver
    networks:
      - signoz
      - databrain
    depends_on:
      - clickhouse
    restart: on-failure

  query-service:
    image: signoz/query-service:0.38.0
    container_name: signoz-query-service
    command: ["-config=/root/config/prometheus.yml"]
    volumes:
      - ./prometheus.yml:/root/config/prometheus.yml
    environment:
      - ClickHouseUrl=tcp://clickhouse:9000
    networks:
      - signoz
    depends_on:
      - clickhouse
    restart: on-failure

  frontend:
    image: signoz/frontend:0.38.0
    container_name: signoz-frontend
    ports:
      - "3301:3301"
    environment:
      - FRONTEND_API_ENDPOINT=http://query-service:8080
    networks:
      - signoz
    depends_on:
      - query-service
    restart: on-failure

volumes:
  signoz-data:
```

### 2. Configure Databrain for Self-Hosted SigNoz

#### Same Docker Network

```yaml theme={"dark"}
# docker-compose.yml
services:
  databrainbackend:
    environment:
      OTEL_ENABLED: "true"
      OTEL_EXPORTER_OTLP_ENDPOINT: "http://signoz-otel-collector:4318"
      OTEL_SERVICE_NAME: "databrain-api"
      LOG_LEVEL: "info"
    networks:
      - databrain
      - signoz

networks:
  databrain:
  signoz:
    external: true
```

#### Separate Hosts

If SigNoz is on a different host:

```bash theme={"dark"}
# Enable OpenTelemetry
OTEL_ENABLED=true

# SigNoz OTLP endpoint (adjust host)
OTEL_EXPORTER_OTLP_ENDPOINT=http://signoz-host:4318

# Service name
OTEL_SERVICE_NAME=databrain-api
```

### 3. Kubernetes Deployment

#### Deploy SigNoz on Kubernetes

```bash theme={"dark"}
# Add SigNoz Helm repo
helm repo add signoz https://charts.signoz.io
helm repo update

# Create namespace
kubectl create namespace signoz

# Install SigNoz
helm install signoz signoz/signoz -n signoz \
  --set clickhouse.persistence.size=50Gi \
  --set frontend.ingress.enabled=true \
  --set frontend.ingress.hosts[0].host=signoz.your-domain.com
```

#### Configure Databrain

```yaml theme={"dark"}
apiVersion: apps/v1
kind: Deployment
metadata:
  name: databrain-backend
spec:
  template:
    spec:
      containers:
      - name: backend
        env:
          - name: OTEL_ENABLED
            value: "true"
          - name: OTEL_EXPORTER_OTLP_ENDPOINT
            value: "http://signoz-otel-collector.signoz.svc.cluster.local:4318"
          - name: OTEL_SERVICE_NAME
            value: "databrain-api"
          - name: LOG_LEVEL
            value: "info"
```

## What Gets Sent to SigNoz

| Telemetry Type | Description                                    |
| -------------- | ---------------------------------------------- |
| **Traces**     | Full distributed traces with all spans         |
| **Metrics**    | Request rates, latency histograms, error rates |
| **Logs**       | Structured logs with trace correlation         |

All three are integrated in SigNoz's unified interface.

## Verification

### 1. Check SigNoz is Running

```bash theme={"dark"}
# Check containers
docker ps | grep signoz

# Check OTLP receiver is listening
curl http://localhost:4318/v1/traces
# Should return 405 Method Not Allowed (endpoint exists but expects POST)
```

### 2. Restart Databrain

```bash theme={"dark"}
docker compose restart databrainbackend
```

### 3. Generate Test Traffic

```bash theme={"dark"}
# Health check
curl https://your-databrain-instance.com/api/health

# Sample request
curl -X POST "https://your-databrain-instance.com/api/v2/metric/execute" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -d '{"metricId": "test-123"}'
```

### 4. View in SigNoz UI

1. Open SigNoz: [http://localhost:3301](http://localhost:3301) (or your configured URL)
2. Go to **Services** → You should see `databrain-api`
3. Click on service to see metrics:
   * Request rate (requests per second)
   * Error rate (%)
   * Latency (p50, p90, p95, p99)
   * Apdex score
4. Go to **Traces** → Filter by `serviceName=databrain-api`
5. Click on any trace to see the full waterfall view

### 5. Check Logs

1. Go to **Logs** tab
2. Filter: `service_name = databrain-api`
3. Click on any log entry
4. See **Related Traces** section for correlated traces

## SigNoz Features

### 1. Service Dashboard

The service dashboard shows:

* **RED metrics**: Rate, Errors, Duration
* **Apdex score**: User satisfaction metric
* **Operations**: Individual endpoint performance
* **Database calls**: Query performance breakdown
* **External calls**: Downstream service latency

### 2. Traces Explorer

Advanced trace filtering:

```
serviceName = databrain-api
AND duration > 1000ms
AND statusCode >= 500
```

**Available filters**:

* Service name
* Operation name
* HTTP status code
* Duration
* Error status
* Custom tags (userId, metricId, etc.)

### 3. Service Map

Visualize service dependencies:

1. Go to **Service Map**
2. See databrain-api and its dependencies:
   * PostgreSQL
   * Redis
   * Hasura
   * Keycloak
   * S3
3. Click on connections to see request rates and error rates

### 4. Logs Management

SigNoz provides powerful log querying:

**Query examples**:

```
service_name = databrain-api AND level = error
service_name = databrain-api AND userId = 123
service_name = databrain-api AND duration > 1000
```

**Log to trace correlation**:

* Click on any log → see **Trace** button
* Jump directly to the trace that generated this log

### 5. Alerts

Create alerts in SigNoz:

#### High Error Rate Alert

1. Go to **Alerts** → **New Alert**
2. Alert Type: **Metrics Alert**
3. Metric Query:

```
A: error_count = count(signoz_latency_bucket{service_name="databrain-api", status_code=~"5.."})
B: total_count = count(signoz_latency_bucket{service_name="databrain-api"})
Formula: (A / B) * 100
```

4. Condition: `> 5` (5% error rate)
5. For: `5 minutes`
6. Add notification channel (Slack, email, webhook)

#### High Latency Alert

1. Metric Query:

```
histogram_quantile(0.95, 
  sum by (le)(rate(signoz_latency_bucket{service_name="databrain-api"}[5m]))
)
```

2. Condition: `> 2000` (2 seconds)
3. For: `5 minutes`

### 6. Dashboards

Create custom dashboards:

1. Go to **Dashboards** → **New Dashboard**
2. Add panels:

**Request Rate Panel**:

```
rate(signoz_calls_total{service_name="databrain-api"}[5m])
```

**Error Rate Panel**:

```
rate(signoz_calls_total{service_name="databrain-api", status_code=~"5.."}[5m])
/
rate(signoz_calls_total{service_name="databrain-api"}[5m])
```

**Latency Distribution Panel**:

```
histogram_quantile(0.50, sum(rate(signoz_latency_bucket[5m])) by (le))
histogram_quantile(0.95, sum(rate(signoz_latency_bucket[5m])) by (le))
histogram_quantile(0.99, sum(rate(signoz_latency_bucket[5m])) by (le))
```

**Database Query Latency**:

```
histogram_quantile(0.95,
  sum(rate(signoz_latency_bucket{
    service_name="databrain-api",
    db_system="postgresql"
  }[5m])) by (le)
)
```

## Advanced Configuration

### Sampling

Add sampling to the OTel collector config:

```yaml theme={"dark"}
# otel-collector-config.yaml
processors:
  # Tail-based sampling
  tail_sampling:
    decision_wait: 10s
    num_traces: 100
    policies:
      # Always keep errors
      - name: error-policy
        type: status_code
        status_code:
          status_codes: [ERROR]
      
      # Keep slow requests
      - name: latency-policy
        type: latency
        latency:
          threshold_ms: 1000
      
      # Sample 10% of normal traffic
      - name: probabilistic-policy
        type: probabilistic
        probabilistic:
          sampling_percentage: 10

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [tail_sampling, batch]
      exporters: [clickhouse]
```

### Resource Attributes

Add custom resource attributes:

```yaml theme={"dark"}
processors:
  resource:
    attributes:
      - key: deployment.environment
        value: production
        action: upsert
      - key: cloud.provider
        value: aws
        action: upsert
      - key: cloud.region
        value: us-east-1
        action: upsert
```

## Troubleshooting

| Issue                    | Solution                                                       |
| ------------------------ | -------------------------------------------------------------- |
| No data in SigNoz        | Check OTel collector logs: `docker logs signoz-otel-collector` |
| Cannot connect to SigNoz | Verify networks and firewall rules                             |
| Missing logs             | Ensure Winston is configured with JSON format                  |
| High disk usage          | Adjust ClickHouse retention settings                           |
| Slow queries             | Increase ClickHouse resources or add indexes                   |

### Debug SigNoz Collector

```bash theme={"dark"}
# Check collector logs
docker logs signoz-otel-collector -f

# Check for OTLP receiver
docker logs signoz-otel-collector | grep "OTLP receiver"

# Should see:
# OTLP receiver started on 0.0.0.0:4317
# OTLP receiver started on 0.0.0.0:4318
```

### Debug Databrain Connection

```bash theme={"dark"}
# Test connectivity from Databrain container
docker exec databrain-backend curl -v http://signoz-otel-collector:4318/v1/traces
```

### Check ClickHouse

```bash theme={"dark"}
# Connect to ClickHouse
docker exec -it signoz-clickhouse clickhouse-client

# Check traces table
SELECT count() FROM signoz_traces.signoz_index_v2;

# Check for recent traces
SELECT timestamp, serviceName, name FROM signoz_traces.signoz_index_v2 
ORDER BY timestamp DESC LIMIT 10;
```

## Data Retention

Configure retention in ClickHouse:

```sql theme={"dark"}
-- Connect to ClickHouse
docker exec -it signoz-clickhouse clickhouse-client

-- Set 30-day retention for traces
ALTER TABLE signoz_traces.signoz_index_v2 
MODIFY TTL toDateTime(timestamp) + INTERVAL 30 DAY;

-- Set 90-day retention for metrics
ALTER TABLE signoz_metrics.distributed_samples_v2 
MODIFY TTL toDateTime(timestamp) + INTERVAL 90 DAY;
```

Or configure in `clickhouse-config.xml`:

```xml theme={"dark"}
<clickhouse>
  <merge_tree>
    <ttl_only_drop_parts>1</ttl_only_drop_parts>
  </merge_tree>
</clickhouse>
```

## Performance Tuning

### For High Traffic (>10K requests/minute)

1. **Increase ClickHouse resources**:

```yaml theme={"dark"}
clickhouse:
  resources:
    limits:
      memory: 8Gi
      cpu: 4
```

2. **Batch processing**:

```yaml theme={"dark"}
processors:
  batch:
    timeout: 10s
    send_batch_size: 10000
```

3. **Use buffer**:

```yaml theme={"dark"}
processors:
  memory_limiter:
    limit_mib: 512
    spike_limit_mib: 128
    check_interval: 5s
```

## Backup and Restore

### Backup ClickHouse Data

```bash theme={"dark"}
# Stop SigNoz
docker compose down

# Backup data directory
tar -czf signoz-backup-$(date +%Y%m%d).tar.gz \
  ./deploy/docker/clickhouse-setup/data/

# Restart SigNoz
docker compose up -d
```

### Restore from Backup

```bash theme={"dark"}
# Stop SigNoz
docker compose down

# Restore data
tar -xzf signoz-backup-20260203.tar.gz -C ./deploy/docker/clickhouse-setup/

# Restart
docker compose up -d
```

## Upgrading SigNoz

```bash theme={"dark"}
cd signoz/deploy/

# Pull latest images
docker compose -f docker/clickhouse-setup/docker-compose.yaml pull

# Restart with new images
docker compose -f docker/clickhouse-setup/docker-compose.yaml up -d
```

## Community & Support

* **Documentation**: [https://signoz.io/docs/](https://signoz.io/docs/)
* **Slack Community**: [https://signoz.io/slack](https://signoz.io/slack)
* **GitHub**: [https://github.com/SigNoz/signoz](https://github.com/SigNoz/signoz)
* **GitHub Discussions**: [https://github.com/SigNoz/signoz/discussions](https://github.com/SigNoz/signoz/discussions)

For Databrain configuration issues, contact your Databrain support team.