Sites

Menu
These are the docs for 14.1, an old version of SpatialOS. The docs for this version are frozen: we do not correct, update or republish them. 14.3 is the newest →

Debugging cloud worker instances

This page covers features for debugging cloud worker instances using the SpatialOS CLI.

When troubleshooting deployment failures, you can often use logs, metrics, and the Inspector to narrow down the cause, reproduce the issue locally, and debug using familiar tools and workflows.

However, you might sometimes run into issues that manifest only on cloud deployments, or even only on large-scale cloud deployments. In addition to logs, metrics, and the Inspector, you can use the SpatialOS CLI to create secure tunnels into a cloud worker instance. It allows for shell access and TCP port forwarding, which enables remote profiling and debugging.

Shell access

Use the following command to establish shell access to a worker container:

spatial project deployment worker shell --deployment_name=<deployment_name> --worker_id=<worker_id>

You can then use the interactive shell to run debug commands or to inspect the environment or filesystem (for example, to view procedurally generated files or raw worker logs).

Note: On Windows, the only supported terminals are Command Prompt and mintty (with winpty). Unsupported terminals might produce unexpected results when using special keys such as arrow keys.

TCP port forwarding

Use the following command to establish a tunnel from a local port to a port on the container that the worker instance is running on:

spatial project deployment worker port-forward -d=<deployment_name> -w=<worker_id> -p=<port>

In the command above, <port> is both the local port and the worker port. If you want to use a different local port to the one that is being used on the worker instance, use the option -l=<local_port>.

Example

  1. To use a remote debugger such as GDB Server, first install it in the worker container (apt-get install gdbserver). Then start your worker instance with it:

    gdbserver localhost:2000 my_worker
    

    GDB Server suspends execution of the worker instance and waits for a debugger to connect on port 2000.

  2. On your local machine, set up the tunnel on port 2000:

    spatial project deployment worker port-forward -d=my_deployment -w=my_worker -p=2000
    
  3. Start GDB on a local copy of the program:

    gdb my_worker
    
  4. Connect GDB to the cloud worker instance via the tunnel:

    (gdb) target remote localhost:2000
    
  5. Debug as normal.

  6. Use Ctrl + C to stop port forwarding and close the tunnel.

Common issues

Issue: You’re using GDB Server and the remote connection immediately closes with the following message on the worker side:

A problem internal to GDBserver has been detected.
Unknown register ymm0h requested

Solution: Check the version of GDB Server, as this is a known issue with older versions. We’ve verified that version 8.2 works. You can download and install it onto the worker container by adding the following to your launch configuration file:

wget http://archive.ubuntu.com/ubuntu/pool/main/g/gdb/gdbserver_8.2-0ubuntu1_amd64.deb
apt-get install ./gdbserver_8.2-0ubuntu1_amd64.deb

Search results

Was this page helpful?

Thanks for letting us know!

Thanks for your feedback

Need more help? Ask on the forums