Troubleshooting

╔════════════════════════════════════════════════════════════════════════════╗ ║ OpenTelemetry + Loki + Grafana Tutorial for Rust Developers ║ ║ Project Summary ║ ╚════════════════════════════════════════════════════════════════════════════╝

PROJECT LOCATION: /Users/shion/workspace/otel-tutorial-rust

WHAT YOU'VE BUILT: A complete, production-ready observability system demonstrating: ✅ Distributed tracing with OpenTelemetry & Jaeger ✅ Structured logging with Loki ✅ Metrics collection with Prometheus ✅ Unified visualization with Grafana ✅ Best practices for instrumenting Rust applications

PROJECT STRUCTURE: 📁 src/ ├── main.rs - Application entry point ├── observability.rs - OTEL & tracing configuration ├── handlers.rs - API endpoints with instrumentation └── custom_middleware.rs - Request ID middleware

⚙️ config/ ├── loki-config.yml - Loki settings ├── promtail-config.yml - Log collector configuration ├── prometheus.yml - Metrics scraper rules └── grafana/ - Grafana provisioning

📚 Documentation/ ├── ../guides/ONBOARDING.md - Your learning path (START HERE) ├── ../getting-started/QUICKSTART.md - 5-minute setup guide ├── ../guides/TECH-STACK.md - Deep dive into architecture └── ../guides/ARCHITECTURE.md - Complete reference

🐳 Infrastructure/ ├── docker-compose.yml - Full observability stack ├── Cargo.toml - Rust dependencies └── .env.example - Environment configuration

GETTING STARTED (3 STEPS):

  1. Start the full stack: $ docker-compose up -d

  2. Build and run the application: $ cargo build --release $ RUST_LOG=info ./target/release/otel-tutorial

  3. Open the dashboards:

    • Grafana: http://localhost:3000 (admin/admin)
    • Jaeger: http://localhost:16686
    • Prometheus: http://localhost:9090
    • Loki: http://localhost:3100 (via Grafana)

TECHNOLOGY STACK:

Backend: • Rust 1.70+ (edition 2024) • Actix-web (HTTP framework) • OpenTelemetry (tracing instrumentation) • Tracing (structured logging)

Infrastructure: • Jaeger (distributed tracing backend) • Loki (log aggregation) • Prometheus (metrics collection) • Grafana (visualization) • Promtail (log collector) • Docker & Docker Compose (orchestration)

KEY CONCEPTS YOU'LL LEARN:

Traces: Complete journey of a request through the system Spans: Individual operations within a trace Logs: Structured events from the application Metrics: Quantitative measurements over time

LEARNING PATH:

  1. Read Onboarding Guide (10 min) - Understand your journey
  2. Follow Quick Start (5 min) - Get everything running
  3. Study Tech Stack (30 min) - Learn how it all works
  4. Explore the source code (30 min) - See the implementation
  5. Read Architecture Guide (30 min) - Deep dives and troubleshooting

WHAT THE APPLICATION DOES:

Endpoints: GET /api/health - Simple health check GET /api/users - List all users POST /api/users - Create a new user GET /api/users/{id} - Get user by ID POST /api/compute - Compute fibonacci (CPU-intensive)

Instrumentation: • Every endpoint is traced with detailed timing • Request IDs track requests through the system • Structured logs record all operations • Errors are automatically captured with context

FEATURES INCLUDED:

✅ Automatic span creation with #[tracing::instrument] ✅ Request ID middleware for correlation ✅ Structured JSON logging ✅ Jaeger trace visualization ✅ Loki log aggregation and search ✅ Prometheus metrics collection ✅ Grafana unified dashboard ✅ Production-ready Docker setup ✅ Comprehensive documentation ✅ Learning exercises and examples

DEPENDENCIES OVERVIEW:

Tracing: • tracing = "0.1" • tracing-subscriber = "0.3" (with json, fmt features)

OpenTelemetry: • opentelemetry = "0.20" • opentelemetry-jaeger = "0.19" • tracing-opentelemetry = "0.21"

Web: • actix-web = "4.4" • actix-rt = "2.9"

Data: • serde = "1.0" • uuid = "1.6" • chrono = "0.4"

PRODUCTION CONSIDERATIONS:

For moving to production: • Use persistent storage for Jaeger and Prometheus • Add authentication to Grafana • Enable TLS/HTTPS for data in transit • Set up proper log retention policies • Configure alerting rules • Use a distributed database instead of local storage • Scale components based on traffic volume • Add service discovery for multiple instances

CUSTOMIZATION:

To add to your own project: 1. Copy src/observability.rs 2. Add observability dependencies to Cargo.toml 3. Call setup_telemetry() in main() 4. Add #[tracing::instrument] to functions 5. Adjust docker-compose.yml for your services 6. Point your app's traces to Jaeger

TESTING THE SETUP:

Generate observability data: # Terminal 1: Start the app ./target/release/otel-tutorial

# Terminal 2: Make requests
curl http://localhost:8080/api/health
curl http://localhost:8080/api/users
curl -X POST http://localhost:8080/api/users \
  -H "Content-Type: application/json" \
  -d '{"name": "John", "email": "john@example.com"}'

# View in Grafana (logs), Jaeger (traces), Prometheus (metrics)

TROUBLESHOOTING:

Containers won't start? $ docker-compose logs

Can't see logs in Loki? 1. Ensure logs/app.log is being written (tail the file). 2. Check Promtail logs for errors: docker-compose logs promtail. 3. Confirm Grafana query uses {job="rust-app"} | json.

Jaeger shows no traces? Check JAEGER_AGENT_HOST=localhost in your environment

Grafana datasources failing? Verify the internal Docker network: docker network inspect observability

NEXT STEPS:

  1. ✅ Complete Quick Start and get everything running
  2. ✅ Understand Tech Stack concepts
  3. ✅ Study the source code examples
  4. ✅ Create your own instrumented Rust service
  5. ✅ Add custom metrics and dashboards
  6. ✅ Deploy to production with proper storage

RESOURCES:

OpenTelemetry: https://opentelemetry.io/docs/ Jaeger: https://www.jaegertracing.io/docs/ Loki: https://grafana.com/docs/loki/ Tracing Crate: https://docs.rs/tracing/ Grafana: https://grafana.com/docs/grafana/ Rust Docs: https://doc.rust-lang.org/

PROJECT STATUS:

✅ Core implementation complete ✅ Docker infrastructure ready ✅ All services configured ✅ Application compiled and tested ✅ Comprehensive documentation written ✅ Learning path designed ✅ Production-ready patterns demonstrated

SUPPORT:

For issues or questions: 1. Check the relevant documentation file (Quick Start, Tech Stack, Architecture) 2. Review the source code (src/*.rs) 3. Check docker-compose logs 4. Consult official documentation for specific tools

═══════════════════════════════════════════════════════════════════════════════

You now have everything you need to understand and use observability with Rust!

Start with: Quick Start → Tech Stack → Architecture → Source Code

Happy learning! 🚀

═══════════════════════════════════════════════════════════════════════════════