Exporting to Cloud Observability

Now that we have the collector configured to receive, process and export telemetry data we just need to update the configuration to export the data to ServiceNow Cloud Observability (formerly Lightstep). To do this, we need to configure an exporter for Cloud Observability in the collector.

1. Add the following otlp/lightstep configuration to the exporters in the /opentelemetry/conf/config.yaml configuration file:

   exporters:
     # Exports data to the console
     debug:
       verbosity: detailed
   
     # Exports data to Cloud Observability
     otlp/lightstep:
       endpoint: ingest.lightstep.com:443
       headers:
         "lightstep-access-token": "{LIGHTSTEP_ACCESS_TOKEN}"
       tls:
         insecure: false
         insecure_skip_verify: true
   

   This configures the exporter to use the secure Cloud Observability ingest endpoint. We will replace the access token value in an upcoming step.

2. Edit the traces pipeline configuration to include the otlp/lightstep exporter in the list of exporters

   pipelines:
     traces:
       receivers: [otlp]
       processors: [batch]
       exporters: [debug, otlp/lightstep]
   

The exporter configuration is almost complete and has been added to the traces pipeline. The last thing we need to do is provide a Cloud Observability access token.

Get Your Cloud Observability Access Token

Cloud Observability uses access tokens to determine which project to associate your telemetry data with when the data is ingested. When you create a project in Cloud Observability, an access token is created automatically. To get your access token, follow these steps:

1. Log in to your Cloud Observability account

2. Click on the Settings icon in the navigation bar

3. Click on Access tokens under TOKENS AND KEYS

4. Click the icon button next to your access token to copy the token to your clipboard

   

5. In your config.yaml file, replace {LIGHTSTEP_ACCESS_TOKEN} with the access token you just copied and save the file

Test

Now were ready to test our configuration and see our telemetry data in Cloud Observability.

1. If your containers are still running, press Ctrl+C in the terminal and wait for them to gracefully stop

2. Run the following command to restart the containers

   docker-compose up
   

Now make some requests to http://localhost:4000 and http://localhost:4000/api/data in your browser. Also try some of the previous variations such as http://localhost:4000?loop=3 and http://localhost:4000?weather=rain.

You should see the trace data outputting to your console. This is because our pipeline is defined to use both the logging and otlp/lightstep exporters. Let’s login in to Cloud Observability and look at the data there.

View Data in Cloud Observability

1. If you are not still logged into Cloud Observability, login again

2. Click on Notebooks in the navigation bar

3. In the query editor, enter web-service in the textbox and select Use "service" as service value from the suggestions. This tells the query editor that this will be a span query and we want to get span data from the service web-service. Back in the Manual Instrumentation step we added this value to the environment variables for our web service OTEL_RESOURCE_ATTRIBUTES=service.name=web-service,service.version=2.0.0. The service.name=web-service is what defines the name of our service in OpenTelemetry.

   

4. In the chart below the query editor you should see some dots. These dots are span exemplars which represent a given span that matches your query. When you mouseover over the exemplars you can see some details like the operation name, service name and in the case of this query, the latency value.

   

5. You can click on these exemplars to view the trace details. Let’s open this in a new tab. Hold Ctrl or Cmd and click on an exemplar.

6. This opens the full trace of which this exemplar is part of. In this view you can see all the parent/child relationships, how each operation contributes to latency and and attributes and log events associated with each span. In the example below, you can see the generateWork span that we created in the Manual Instrumentation step. Notice the 3 operations with the same name and how they relate to each other as well as the custom attributes and log event that we added.

   

Continue exploring within Cloud Observability to see how the work you did in prior steps is viewed and interacted with in the Cloud Observability platform. Try adding another query in Notebooks for the users-service and look at the spans for requests with a ?weather=rain versus without and notice the differences in attribute values. To learn more more about using Cloud Observability, check out our product documentation. Before wrapping up, let’s go ahead and shut down your services by pressing Ctrl+C in the terminal.