Debugging cloud worker instances
However, you might sometimes run into issues that manifest only on cloud deployments, or even only on large-scale cloud deployments. In addition to logs, metrics, and the Inspector, you can use the SpatialOS CLI to create secure tunnels into a cloud worker instance. It allows for shell access and TCP port forwarding, which enables remote profiling and debugging.
Use the following command to establish shell access to a worker container:
spatial project deployment worker shell --deployment_name=<deployment_name> --worker_id=<worker_id>
You can then use the interactive shell to run debug commands or to inspect the environment or filesystem (for example, to view procedurally generated files or raw worker logs).
Note: On Windows, the only supported terminals are Command Prompt and mintty (with winpty). Unsupported terminals might produce unexpected results when using special keys such as arrow keys.
TCP port forwarding
Use the following command to establish a tunnel from a local port to a port on the container that the worker instance is running on:
spatial project deployment worker port-forward -d=<deployment_name> -w=<worker_id> -p=<port>
In the command above,
<port> is both the local port and the worker port. If you want to use a different local port to the one that is being used on the worker instance, use the option
To use a remote debugger such as GDB Server, first install it in the worker container (
apt-get install gdbserver). Then start your worker instance with it:
gdbserver localhost:2000 my_worker
GDB Server suspends execution of the worker instance and waits for a debugger to connect on port 2000.
On your local machine, set up the tunnel on port 2000:
spatial project deployment worker port-forward -d=my_deployment -w=my_worker -p=2000
Start GDB on a local copy of the program:
Connect GDB to the cloud worker instance via the tunnel:
(gdb) target remote localhost:2000
Debug as normal.
Ctrl + Cto stop port forwarding and close the tunnel.
Issue: You’re using GDB Server and the remote connection immediately closes with the following message on the worker side:
A problem internal to GDBserver has been detected. Unknown register ymm0h requested
Solution: Check the version of GDB Server, as this is a known issue with older versions. We’ve verified that version 8.2 works. You can download and install it onto the worker container by adding the following to your launch configuration file:
wget http://archive.ubuntu.com/ubuntu/pool/main/g/gdb/gdbserver_8.2-0ubuntu1_amd64.deb apt-get install ./gdbserver_8.2-0ubuntu1_amd64.deb