Get SpatialOS

Sites

Menu

Build a metrics dashboard

The monitoring proxy is currently only available to select customers for testing. Contact support if you’d like access.

This page explains how to view SpatialOS logs and metrics in your own Grafana instance, so you can create your own dashboards without asking Improbable support.

This uses the public metrics documented on the Metrics reference page, and the monitoring proxy that lets you access your deployment’s logs metrics.

You can also use the monitoring proxy to access metrics through code.

Before you start

  • You need to set up your own Grafana server (if you don’t have one already). You can install one yourself or use Grafana cloud.

  • This service is in alpha, so is subject to change. In particular, all URLs (especially https://lame.service.improbable.io/@proxyhost/), metric names, and log messages are likely to change in the future.

  • Be careful with the queries you write: if they’re very expensive, they can cause problems with the metrics or logging servers. If you’re not sure, ask customer support to review the queries.

Set up SpatialOS metrics in your own Grafana instance

  1. Ask Improbable customer support to set up a metrics service account for you.

    They’ll send you a username and password.

  2. Log in to your Grafana server as an admin.

  3. Under Data Sources, add a new Prometheus Data Source for each cluster you use (eg eu1-prod, us2-prod).

    For example, for a deployment running on eu2-prod, you’d add the following details:

    • Name: eu2-prod

      The name here must be the same as the name of the cluster.

    • Type: Prometheus

    • HTTP URL: https://lame.service.improbable.io/@proxyhost/eu2-prod.monitoring.spatial.improbable.io

    • HTTP access: proxy 

    • HTTP auth: Basic auth

    • Basic auth details: the username and password you got from Improbable customer support

  4. Click Save & Test. You’ll see the following error:

    HTTP Error: Forbidden

    This is fine. The connection is successful, but the metric used on Prometheus to test the connection by Grafana is “forbidden” to your account.

  5. Now you can add graphs of these metrics to your dashboard.

    For details of how to do this, see the Grafana docs on Creating a Prometheus graph.

    You can see which metrics are available, and get example queries to run, on the Metrics reference page.

Only create alerts using documented metrics.

Set up SpatialOS logs in your own Grafana instance

  1. Ask Improbable customer support to set up a logs service account for you.

    They’ll send you a username and password.

  2. Log in to your Grafana server as an admin.

  3. Under Data Sources, add a new Elasticsearch Data Source.

    For example, for the project test_project on the eu2-prod cluster:

    • Name: es-eu2-prod

      The name here must be the same as the name of the cluster.

    • Type: Elasticsearch

    • HTTP URL: https://lame.service.improbable.io/@proxyhost/es-eu2-prod.logs.spatial.improbable.io

    • HTTP access: Proxy

    • HTTP auth: Basic auth

    • Basic auth details: the username and password you got from Improbable customer support

    • Elasticsearch details:

      • Index name: [fablog-test_project-]YYYY.MM.DD
      • Pattern: Daily
      • Time field name: @timestamp
      • Version: 2.x
  4. Click Save & Test. You’ll see the following:

    Success: Index OK. Field name OK.

Don’t create alerts from the exact wording of logs. The wording can change without notice.

Things you should bear in mind

The gross query rate is limited to 50 requests per minute. To avoid hitting this limit, in the Grafana refresh time settings, set the auto-refresh interval to at least 1 minute.

Was this page helpful?

Thanks for letting us know!

Thanks for your feedback

Need more help? Ask on the forums