OpenTelemetry

Create and Export Custom Metrics

Collecting and analyzing metrics from your infrastructure is a great start but sometimes you may need data that your platform doesn’t provide. Using the OpenTelemetry libraries, you can create and export custom metrics to meet these needs. Let’s do this for our web service.

  1. Add the following packages to the dependencies section of the package.json file in both services (`/users/package.json’ and ‘/web/package.json’). You need to do this for both services because the instrumentation file you will create below is shared by both services and depends on these packages.

    "@opentelemetry/exporter-metrics-otlp-proto": "^0.46.0",
"@opentelemetry/sdk-metrics": "^1.19.0",

  2. Edit /opentelemetry/src/instrumentation.js

  3. Add the following statements at the top of the file with the other require statements

    const {
  OTLPMetricExporter,
} = require("@opentelemetry/exporter-metrics-otlp-proto");
const {
  PeriodicExportingMetricReader,
} = require("@opentelemetry/sdk-metrics");

  4. Configure the metricReader in the NodeSDK options by adding the following code right after the traceExporter configuration

    metricReader: new PeriodicExportingMetricReader({
  exporter: new OTLPMetricExporter({
    url: "http://otel-collector:4318/v1/metrics",
  }),
  exportIntervalMillis: 3000,
}),

    This configures the SDK to export metrics to the metrics endpoint of your OpenTelemetry collector every 3 seconds.

    The full SDK configuration should look like this:

    const sdk = new opentelemetry.NodeSDK({
  traceExporter: new OTLPTraceExporter({
    url: "http://otel-collector:4317",
  }),
  metricReader: new PeriodicExportingMetricReader({
    exporter: new OTLPMetricExporter({
      url: "http://otel-collector:4318/v1/metrics",
    }),
    exportIntervalMillis: 3000,
  }),
  instrumentations: [getNodeAutoInstrumentations()],
});

  5. Edit the web service (/web/src/index.js)

  6. Add the following code right after the const tracer = api.trace.getTracer... code:

    const meter = api.metrics.getMeter("default");

    This acquires a meter that is used to create metrics from.

  7. We will create two custom metrics to track the API response codes of the checkWeather function. Add the following lines of code above the async function checkWeather(...) definition

    // create the metric counters for our response code metrics
const counter_2xx = meter.createCounter("workshop.web.2xx_count");
const counter_5xx = meter.createCounter("workshop.web.5xx_count");

  8. Add the following code just above the break statement in the case: "rain": block

    counter_2xx.add(1);

    This increases our counter for the 2xx response codes by one since the method returns a 202 when the condition is “rain”. 9. Add the same line of code at the bottom of the default: statement since the return is a 200 when the default condition is reached

    counter_2xx.add(1);

  9. Add the following line of code before the break statement in the case: "snow": block

    counter_5xx.add(1);

    This increases our 5xx counter since the method returns a 500 when the condition is snow.

  10. Edit the collector configuration (/opentelemetry/conf/config.yaml)

  11. Add otlp to the list of receivers in the metrics pipeline. The metrics pipeline configuration should look like this

    metrics:
  receivers: [otlp, prometheus/docker]
  processors: [batch]
  exporters: [logging, otlp/lightstep]

  12. Save your changes

Deploy and Test

Now let’s deploy your changes and test these custom metrics.

  1. If your containers are still running, press Ctrl+C in the terminal and wait for them to gracefully stop

  2. Run the following command to restart the containers

    docker-compose up --build

  3. Once your services are up and running, make some requests to http://localhost:4000, http://localhost:4000/?weather=snow and http://localhost:4000/?weather=rain

View the Metrics in Cloud Observability

  1. If you are not still logged into Cloud Observability, login again

  2. Click on Notebooks in the navigation bar

  3. In the query editor, type workshop.web.2xx_count in the textbox and hit Enter workshop.web.2xx_count query

    You should see a line chart with some data on it. This is showing the rate of requests that returned a 2xx response based on the custom metric emitted from the service.

  4. Add another chart by clicking the + button in the bottom left below the current chart

  5. In the query editor, type workshop.web.5xx_count in the textbox and hit Enter workshop.web.5xx_count query

    You should again see a line chart with some data on it. This one is showing the rate of requests that returned a 5xx response based on the custom metric.

In this workshop we covered metrics, specifically how to capture infrastructure metrics from Docker and how to generate and export custom metrics for our service code. We then used Cloud Observability to query and visual these metrics. Before wrapping up, go ahead and shut down your services by pressing Ctrl+C in the terminal.

next: Conclusion