Started with OpenTelemetry in Go

Below is a step-by-step tutorial to help you get started with OpenTelemetry in Go for collecting metrics, plus best practices for deploying in Kubernetes.

---

1. What is OpenTelemetry?

OpenTelemetry is an open-source observability framework that provides:

1. Traces: Collecting distributed traces from your services.
2. Metrics: Recording and exporting metrics from your applications.
3. Logs: Capturing logs consistently (still evolving in OpenTelemetry).

For Go, you can use the official go.opentelemetry.io/otel libraries to instrument your services. You then export the collected data to your choice of backend (e.g., Prometheus, Jaeger, Grafana, etc.)—often via the OpenTelemetry Collector.

---

2. Basic Concepts

2.1 Instrumentation

Instrumentation is the process of adding code to your application to generate telemetry data (metrics, traces, logs).

2.2 OpenTelemetry Collector

The OpenTelemetry Collector is a vendor-agnostic service that can receive, process, and export telemetry data. It is often deployed as a sidecar or standalone agent in Kubernetes.

2.3 Metrics

Metrics are points or sets of data that measure your application. Common metric types include:

• Counter: Always goes up (e.g., requests handled).
• UpDownCounter: Can go up or down (e.g., active connections).
• Gauge: Captures a snapshot of a value at a point in time.
• Histogram: Tracks distribution of a value (e.g., request latency).

---

3. Instrumenting a Go Application

Below is a minimal Go example showing how to set up metrics collection with OpenTelemetry.

Note: The examples below use the new OpenTelemetry Metrics API (stable as of OpenTelemetry-Go v1.0).

3.1 Dependencies

In your go.mod:

module my-awesome-service

go 1.20

require (
    go.opentelemetry.io/otel v1.11.2 // example version
    go.opentelemetry.io/otel/exporters/prometheus v0.41.0 // example version
    go.opentelemetry.io/otel/sdk/metric v0.41.0 // example version
)

Adjust versions as needed.

3.2 Simple Setup with Prometheus

Here’s a simple example for exposing Prometheus metrics:

package main

import (
    "context"
    "fmt"
    "log"
    "net/http"

    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/metric/global"
    "go.opentelemetry.io/otel/exporters/prometheus"
    "go.opentelemetry.io/otel/sdk/metric"
    "go.opentelemetry.io/otel/sdk/metric/controller/push"
    "go.opentelemetry.io/otel/sdk/metric/processor/basic"
    "go.opentelemetry.io/otel/sdk/metric/selector/simple"
)

func main() {
    ctx := context.Background()

    // 1. Create the Prometheus exporter
    exporter, err := prometheus.New(prometheus.Config{})
    if err != nil {
        log.Fatalf("failed to initialize prometheus exporter %v", err)
    }

    // 2. Create the Metric Provider (Controller in older code) – uses the exporter
    controller := push.New(
        basic.NewFactory(
            simple.NewWithHistogramDistribution(),
            exporter,
        ),
        exporter,
    )
    // Start the controller
    err = controller.Start(ctx)
    if err != nil {
        log.Fatalf("failed to start controller %v", err)
    }
    defer func() {
        _ = controller.Stop(ctx)
    }()

    // 3. Set global metric provider
    global.SetMeterProvider(controller.MeterProvider())

    // 4. Create a meter and instruments
    meter := global.Meter("my-awesome-service")

    requestCounter, err := meter.Int64Counter("requests_total")
    if err != nil {
        log.Fatalf("failed to create counter: %v", err)
    }

    // 5. Example HTTP server with instrumentation
    mux := http.NewServeMux()
    mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
        requestCounter.Add(ctx, 1) // increment the counter
        fmt.Fprintf(w, "Hello, OpenTelemetry!\n")
    })

    // 6. Expose metrics endpoint at /metrics
    mux.Handle("/metrics", exporter)

    srv := &http.Server{
        Addr:    ":8080",
        Handler: mux,
    }

    log.Println("Starting server at :8080...")
    if err := srv.ListenAndServe(); err != nil {
        log.Fatal(err)
    }
}

Key Points in the code above:

1. Prometheus Exporter: We create and configure the exporter to allow Prometheus to scrape metrics directly from the /metrics endpoint.
2. Metric Provider: This sets up how the library collects and processes metrics.
3. Instrument: We create a requestCounter instrument to track how many requests our server receives.

If you want to send metrics to another backend (e.g. an OpenTelemetry Collector), you’d configure a different exporter.

---

4. Using the OpenTelemetry Collector

4.1 Basic Architecture

A common pattern on Kubernetes is:

[ Go App ] -> [ OpenTelemetry Collector ] -> [ Metrics Backend (e.g., Prometheus, Grafana, etc.) ]

4.2 Collector Configuration

A typical collector config (otel-collector-config.yaml) might look like this:

receivers:
  otlp:
    protocols:
      grpc:
      http:
exporters:
  prometheus:
    endpoint: "0.0.0.0:9464"
    # or specify a custom metrics namespace, if you want
    namespace: "my_otel_app"
processors:
  batch: {}

service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [prometheus]

In this config:

• Receiver: Listens for OTLP signals on gRPC & HTTP.
• Processor: Here, batch is used to batch metrics before exporting to Prometheus.
• Exporter: Exports metrics in Prometheus format on port 9464.

You can then set your Go application to export metrics via OTLP to the collector’s endpoint (otel-collector:4317 if you use the default gRPC port in your K8s environment, for instance).

4.3 Configure Go App to Send Metrics to Collector

For OTLP, you’d replace the Prometheus exporter snippet with an OTLP exporter. Here’s a simplified example:

import (
    "context"
    "log"

    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
    "go.opentelemetry.io/otel/sdk/metric"
    "go.opentelemetry.io/otel/sdk/metric/controller/push"
    "go.opentelemetry.io/otel/sdk/metric/processor/basic"
    "go.opentelemetry.io/otel/sdk/metric/selector/simple"
    "google.golang.org/grpc/credentials/insecure"
)

func initMetrics() (func(context.Context) error, error) {
    ctx := context.Background()

    // Create OTLP exporter (for metrics). Depending on your version, you might need the metric-specific OTLP exporter.
    // Not all versions of OTLP have a stable metric exporter yet, so check the docs for the latest approach.
    // This is an example. The actual usage might differ for your version:
    metricExporter, err := otlptracegrpc.New(ctx,
        otlptracegrpc.WithInsecure(),
        otlptracegrpc.WithEndpoint("otel-collector:4317"),
        otlptracegrpc.WithDialOption(insecure.NewCredentials()),
    )
    if err != nil {
        return nil, err
    }

    // Build a push controller for the metrics pipeline
    controller := push.New(
        basic.NewFactory(
            simple.NewWithHistogramDistribution(),
            metricExporter,
        ),
        metricExporter,
    )

    // Start the controller
    if err := controller.Start(ctx); err != nil {
        return nil, err
    }

    // Return a shutdown function
    shutdown := func(ctx context.Context) error {
        return controller.Stop(ctx)
    }

    return shutdown, nil
}

You’d then have the Collector export those OTLP metrics to Prometheus, as shown in the collector configuration above.

---

5. Deploying on Kubernetes

Below is an outline of how you might deploy everything on Kubernetes.

5.1 Kubernetes Manifests

5.1.1 OpenTelemetry Collector Deployment

A simple Deployment for the collector (using the official otel/opentelemetry-collector image or a custom image):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: otel-collector
spec:
  replicas: 1
  selector:
    matchLabels:
      app: otel-collector
  template:
    metadata:
      labels:
        app: otel-collector
    spec:
      containers:
      - name: otel-collector
        image: otel/opentelemetry-collector:latest
        args: ["--config=/etc/otel-collector-config.yaml"]
        ports:
        - containerPort: 4317 # OTLP gRPC
        - containerPort: 4318 # OTLP HTTP
        - containerPort: 9464 # Prometheus exporter
        volumeMounts:
        - name: collector-config
          mountPath: /etc/otel-collector-config.yaml
          subPath: otel-collector-config.yaml
      volumes:
      - name: collector-config
        configMap:
          name: otel-collector-config
---
apiVersion: v1
kind: Service
metadata:
  name: otel-collector
spec:
  type: ClusterIP
  selector:
    app: otel-collector
  ports:
  - name: otlp-grpc
    port: 4317
    targetPort: 4317
  - name: otlp-http
    port: 4318
    targetPort: 4318
  - name: prometheus
    port: 9464
    targetPort: 9464

5.1.2 ConfigMap for Collector

apiVersion: v1
kind: ConfigMap
metadata:
  name: otel-collector-config
data:
  otel-collector-config.yaml: |
    receivers:
      otlp:
        protocols:
          grpc:
          http:
    processors:
      batch: {}
    exporters:
      prometheus:
        endpoint: "0.0.0.0:9464"
        namespace: "my_otel_app"
    service:
      pipelines:
        metrics:
          receivers: [otlp]
          processors: [batch]
          exporters: [prometheus]

5.2 Instrumented Go App Deployment

In your application Deployment, ensure:

1. The environment variable(s) or config specifying the OTLP endpoint points to the otel-collector Service (e.g., otel-collector:4317).
2. The Go code is built to export metrics via OTLP.

Example snippet for your app:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-awesome-service
spec:
  replicas: 2
  selector:
    matchLabels:
      app: my-awesome-service
  template:
    metadata:
      labels:
        app: my-awesome-service
    spec:
      containers:
      - name: my-awesome-service
        image: my-awesome-service:latest
        ports:
        - containerPort: 8080
        env:
        - name: OTEL_EXPORTER_OTLP_ENDPOINT
          value: "otel-collector:4317"
        # ... other environment variables as needed ...

Your Go service in the container references OTEL_EXPORTER_OTLP_ENDPOINT to know where to send OTLP data.

---

6. Best Practices

1. Use the Collector for Flexibility
   Instead of exporting directly to one vendor (e.g., Prometheus or Jaeger) from your app, always prefer using the OpenTelemetry Collector to decouple your instrumentation from the backend. It allows you to change or add new exporters without re-deploying your app.
2. Batch Processor
   Always use the batch processor in the Collector to avoid sending telemetry data for every single measurement. Batching improves performance and network usage.
3. Resource Attribution
   Add resource attributes to your instrumentation, e.g., service name, version, environment. This helps you distinguish metrics from different services or environments (prod/staging). import "go.opentelemetry.io/otel/sdk/resource" rsc, err := resource.New(ctx, resource.WithAttributes( semconv.ServiceNameKey.String("my-awesome-service"), semconv.ServiceVersionKey.String("v1.0.0"), // Additional attributes... ), ) // Attach 'rsc' to your metric/tracer provider
4. Secure OTLP Communication
   If possible, use TLS (instead of insecure) for OTLP data. (In production, you’d configure your Collector with TLS certificates.)
5. Limit Instrumentation Overhead
   • Avoid creating new meters or instruments in high-frequency code paths. Create them once and reuse them.
   • Use asynchronous instruments (e.g., UpDownCounter, ObservableGauge) if your metrics are best measured by polling a current state rather than incrementing counters.
6. Label Cardinality
   • Keep an eye on the number of unique labels you attach to metrics. High-cardinality labels (e.g., user ID, session ID) can blow up the number of time series.
7. Automate
   • Use Helm Charts or Operators (like OpenTelemetry Operator for Kubernetes) to manage your OpenTelemetry Collector deployment.

---

7. Putting It All Together

1. Instrument your Go services with the OpenTelemetry Go libraries.
2. Configure your Go service to export metrics via OTLP to the OpenTelemetry Collector.
3. Use the Collector to process, batch, and export to your desired backend (Prometheus for metrics scraping, or any other supported exporter).
4. In Kubernetes, deploy both your instrumented service and the Collector. Link them via a service (e.g., otel-collector:4317).
5. Confirm metrics are accessible from your final backend (e.g., Prometheus, or a Prometheus-based monitoring solution like Grafana Cloud).

Following this guide, you’ll have a robust, standards-based observability pipeline that’s flexible and easy to evolve as your observability needs change.

---

Additional References

• OpenTelemetry for Go documentation
• OpenTelemetry Collector documentation
• OpenTelemetry Operator for K8s (automates deployment and management of the Collector)
• Prometheus Operator (if you want an operator-based approach for Prometheus resources)

---

That’s it! By combining the OpenTelemetry Go SDK, an OpenTelemetry Collector, and a metrics backend like Prometheus, you’ll have a powerful and scalable approach for collecting metrics in your Kubernetes environment.