Sites

Menu
These are the docs for 12.2, an old version of SpatialOS. The docs for this version are frozen: we do not correct, update or republish them. 14.1 is the newest →

Accessing metrics and logs on your own analytics platform

This page explains how to view SpatialOS metrics and logs within your own in-house or third-party analytics platform, using the monitoring proxy that lets you access your deployment’s logs and metrics.

You can also use the monitoring proxy to access metrics through code.

In your analytics platform, you’ll need to:

  1. Add Prometheus and Elasticsearch as data sources.
  2. Add the monitoring proxy URL.
  3. Enter the HTTP basic authentication username and password that we’ve given you.

We’ve provided a worked example below using Grafana.

Authentication

Make sure you use HTTPS when making requests against the monitoring proxy.

The monitoring proxy uses HTTP basic authentication. You will need to supply longshot_user as the username, and a valid refresh token as the password.

For access from a dashboard or other automated service, we strongly recommend you use the refresh token of a service account. To set up a service account, please reach out to Improbable customer support via the helpdesk (for customers with a service agreement) or the forums.

Worked example: Grafana

Before you start

  • If you don’t already have a Grafana server, you’ll need to set one up. You can install one yourself or use Grafana cloud.

  • This service is in alpha, so is subject to change. In particular, all URLs (especially https://monitoring.service.improbable.io/@proxyhost/), metric names, and log messages are likely to change in the future.

Set up SpatialOS metrics in your own Grafana instance

  1. Ask Improbable customer support to set up a metrics service account for you.

    They’ll send you a username and password.

  2. Log in to your Grafana server as an admin.

  3. Under Data Sources, add a new Prometheus Data Source:

    Data source fields Details
    Data source name SpatialOS
    Data source type Prometheus
    URL https://monitoring.service.improbable.io/@proxyhost/metrics.monitoring.spatial.improbable.io
    HTTP access Proxy
    HTTP auth Basic auth
    Basic auth details The username and password you got from Improbable customer support
  4. Now you can add graphs of these metrics to your dashboard.

    For details of how to do this, see the Grafana docs on Creating a Prometheus graph.

    You can see which metrics are available, and get example queries to run, on the Metrics reference page.

  • Only create alerts using documented metrics.

  • Be careful with the queries you write: if they’re very expensive, you may experience problems with the metrics or logging servers. If you’re not sure, ask customer support to review the queries.

  • Don’t set a refresh rate more frequent than 60 seconds: if the refresh rate is too frequent, requests from your dashboard may fail due to the restrictions on metric querying.

Set up SpatialOS logs in your own Grafana instance

  1. Ask Improbable customer support to set up a logs service account for you.

    They’ll send you a username and password.

  2. Log in to your Grafana server as an admin.

  3. Under Data Sources, add a new Elasticsearch Data Source.

    For example, for the project test_project:

    Data source fields Details
    Data source name SpatialOS
    Data source type Elasticsearch
    URL https://monitoring.service.improbable.io/@proxyhost/logs.monitoring.spatial.improbable.io
    HTTP access Proxy
    HTTP auth Basic auth
    Basic auth details The username and password you got from Improbable customer support
    Elasticsearch-specific fields Details
    Index name [fablog-test_project-]YYYY.MM.DD
    Pattern Daily
    Time field name @timestamp
    Version 5.6.x
  4. Click Save & Test. You’ll see the following:

    Success: Index OK. Field name OK.

Don’t create alerts from the exact wording of logs. The wording can change without notice.

Restrictions on metric querying

  • Queries must specify the project being queried.

    For example, spatialos_worker_connected::sum{project="test_project"}

  • Each project has a limit of 200 queries per minute.

    For example, if one user issues 200 of the above queries, no users will be able to query metrics for that project for up to a minute.

  • Each project has a limit of 60s of query processing time, reset every minute.

  • Queries will time out after 30s of processing.

Search results

Was this page helpful?

Thanks for letting us know!

Thanks for your feedback

Need more help? Ask on the forums