Grafana Tempo is a high-scale distributed tracing backend that's fully compatible with OpenTelemetry. Combined with Grafana, it provides powerful trace visualization and querying capabilities.

1Create Docker Compose Configuration

Create a docker-compose.yml file with Tempo and Grafana:

docker-compose.yml

yaml

version: '3'

services:
  tempo:
    image: grafana/tempo:latest
    command: [ "-config.file=/etc/tempo.yaml" ]
    volumes:
      - ./tempo.yaml:/etc/tempo.yaml
      - ./tempo-data:/tmp/tempo
    ports:
      - "4318:4318"  # OTLP HTTP
      - "3200:3200"  # Tempo

  grafana:
    image: grafana/grafana:latest
    volumes:
      - ./grafana-datasources.yaml:/etc/grafana/provisioning/datasources/datasources.yaml
    environment:
      - GF_AUTH_ANONYMOUS_ENABLED=true
      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
      - GF_AUTH_DISABLE_LOGIN_FORM=true
    ports:
      - "3000:3000"
    depends_on:
      - tempo

2Create Tempo Configuration

Create tempo.yaml to configure the OTLP receiver:

tempo.yaml

yaml

server:
  http_listen_port: 3200

distributor:
  receivers:
    otlp:
      protocols:
        http:
          endpoint: 0.0.0.0:4318
        grpc:
          endpoint: 0.0.0.0:4317

ingester:
  max_block_duration: 5m

compactor:
  compaction:
    block_retention: 1h

storage:
  trace:
    backend: local
    local:
      path: /tmp/tempo/blocks
    wal:
      path: /tmp/tempo/wal

querier:
  frontend_worker:
    frontend_address: tempo:9095

3Create Grafana Datasource Configuration

Create grafana-datasources.yaml to connect Grafana to Tempo:

grafana-datasources.yaml

yaml

apiVersion: 1

datasources:
  - name: Tempo
    type: tempo
    access: proxy
    orgId: 1
    url: http://tempo:3200
    basicAuth: false
    isDefault: true
    version: 1
    editable: false
    jsonData:
      httpMethod: GET
      tracesToLogs:
        datasourceUid: 'loki'

4Start the Services

bash

# Start Tempo and Grafana
docker-compose up -d

# Verify services are running
docker-compose ps

# View logs
docker-compose logs -f tempo grafana

First-time setup

It may take 30-60 seconds for Grafana to fully start and configure the Tempo datasource.

5Configure Traccia

Configure your Traccia SDK to send traces to Tempo:

main.py

python

from traccia import init

# Point to local Tempo OTLP HTTP endpoint
init(
    endpoint="http://localhost:4318/v1/traces",
    service_name="my-agent"
)

Or via configuration file:

traccia.toml

toml

[tracing]
endpoint = "http://localhost:4318/v1/traces"
service_name = "my-agent"
use_otlp = true

6Run Your Agent and View Traces

agent.py

python

from traccia import observe
from openai import OpenAI

client = OpenAI()

@observe()
def my_agent(query: str):
    response = client.chat.completions.create(
        model="gpt-4",
        messages=[{"role": "user", "content": query}]
    )
    return response.choices[0].message.content

# Run the agent
result = my_agent("What is quantum computing?")

Open Grafana at http://localhost:3000:

Click "Explore" in the left sidebar
Select "Tempo" as the data source
Use "Search" tab to filter by service name, duration, etc.
Click any trace to see the detailed waterfall view
Expand spans to see attributes like token counts and costs

Querying Traces in Grafana

Grafana provides powerful query capabilities using TraceQL:

text

# Find traces with errors
{ status = error }

# Find expensive traces (cost > $1)
{ resource.llm.cost > 1.0 }

# Find slow traces
{ duration > 5s }

# Find traces for specific agent
{ resource.agent.id = "support-bot" }

# Complex query
{ resource.service.name = "my-agent" && duration > 1s && status = ok }

Cleanup

When you're done, stop and remove the containers:

bash

# Stop services
docker-compose down

# Remove volumes (deletes all trace data)
docker-compose down -v

Grafana Tempo Setup

1Create Docker Compose Configuration

2Create Tempo Configuration

3Create Grafana Datasource Configuration

4Start the Services

5Configure Traccia

6Run Your Agent and View Traces

Querying Traces in Grafana

Cleanup

Next Steps

Try Jaeger Instead

Grafana Tempo Documentation