Sending data to SpatialOS

A worker can send data such as logging, metrics, and component updates to SpatialOS using the improbable.worker.Connection.

Sending and receiving metrics

You can optionally send metrics by calling Connection.sendMetrics. You can view metrics by going to the Inspector World view and clicking on a worker.

There are two typical use cases for sending metrics:

1. Reporting your own custom metrics. Both time series and histogram metrics are supported.

2. Updating a worker’s load.

   The load of a worker is a floating-point value, with 0 indicating an unloaded worker and values above 1 corresponding to an overloaded worker. The reported values direct SpatialOS’s load balancing strategy.

The following example demonstrates both use cases:

// A queue of tasks the worker has to complete.
private final static java.util.Queue TaskQueue = new java.util.ArrayDeque<>();
private final static double MAXIMUM_QUEUE_SIZE = 200.0; // An arbitrary maximum value.

// Collect and send user-defined metrics to SpatialOS.
// Note that you should generally call sendMetrics on the same user thread as you call
// other Connection methods.
private static void sendUserMetrics(Connection connection) {
    Metrics metrics = new Metrics();
    // Update the current load of the worker.
    double load = TaskQueue.size() / MAXIMUM_QUEUE_SIZE;
    metrics.load = improbable.collections.Option.of(load);
    // Add custom metrics.
    metrics.gaugeMetrics.put("MyCustomMetric", 1.0);
    HistogramMetric histogram = new HistogramMetric();
    histogram.recordObservation(123);
    metrics.histogramMetrics.put("MyCustomHistogram", histogram);
    connection.sendMetrics(metrics);
}

Workers automatically send several built-in, internal gauge metrics at a period defined by the builtInMetricsReportPeriodMillis field of ConnectionParameters. You can register a callback to receive these metrics inside a Ops.MetricsOp using Dispatcher.OnMetrics:

private static void registerMetricsCallback(Dispatcher dispatcher) {
    dispatcher.onMetrics(op -> {
        double shortCircuitRate = op.metrics.gaugeMetrics.get("connection_command_request_short_circuit_rate");
        // Do something with the metric, or store it...
        System.out.println("Command requests short-circuited per second: " + shortCircuitRate);
    });
}

The full list of built-in gauge metrics is as follows. All rate metrics are per-second.

Metric name (String)	Metric value (double)
connection_send_queue_size	The current size of the send queue (the messages waiting to be sent to SpatialOS).
connection_send_queue_fill_rate	The rate at which messages are being added to the send queue.
connection_receive_queue_size	The current size of the send queue.
connection_receive_queue_fill_rate	The rate at which messages are being received from SpatialOS and added to the receive queue.
connection_oplist_queue_size	The current size of the op list.
connection_oplist_queue_fill_rate	The rate at which ops are being added to the internal OpList (the queue of processed messages that workers operate on).
connection_log_message_send_rate	The rate at which log messages are being sent.
connection_component_update_send_rate	The rate at which component updates are being sent.
connection_command_request_send_rate	The rate at which command requests are being sent.
connection_command_response_send_rate	The rate at which successful command responses are being sent.
connection_command_failure_send_rate	The rate at which command failure responses are being sent.
connection_local_command_timeouts	The total local commands that timed out when waiting for a response.
connection_local_command_timeouts_rate	The rate at which local commands time out when waiting for a response.
connection_unexpected_command_response_receives	The total unexpected command responses recieved.
connection_unexpected_command_response_receives_rate	The rate at which unexpected command responses are recieved.
connection_reserve_entity_id_request_send_rate	The rate at which requests to reserve an entity ID are being sent.
connection_reserve_entity_ids_request_send_rate	The rate at which requests to reserve multiple entity IDs are being sent.
connection_create_entity_request_send_rate	The rate at which entity creation requests are being sent.
connection_delete_entity_request_send_rate	The rate at which entity deletion requests are being sent.
connection_entity_query_request_send_rate	The rate at which entity query requests are being sent.
connection_component_interest_send_rate	The rate at which component interest updates are being sent.
connection_authority_loss_imminent_acknowledgement_send_rate	The rate at which imminent authority loss acknowledgements are being sent.
connection_command_request_short_circuit_rate	The rate at which command requests are being short-circuited.
connection_command_response_short_circuit_rate	The rate at which successful command responses are being short-circuited.
connection_command_failure_short_circuit_rate	The rate at which command failure responses are being sent.
connection_flag_update_op_receive_rate	The rate at which FlagUpdate Ops are being received.
connection_critical_section_op_receive_rate	The rate at which CriticalSection Ops are being received.
connection_add_entity_op_receive_rate	The rate at which AddEntity Ops are being received.
connection_remove_entity_op_receive_rate	The rate at which RemoveEntity Ops are being received.
connection_reserve_entity_id_response_op_receive_rate	The rate at which ReserveEntityIdResponse Ops are being received.
connection_reserve_entity_ids_response_op_receive_rate	The rate at which ReserveEntityIdsResponse Ops are being received.
connection_create_entity_response_op_receive_rate	The rate at which CreateEntityResponse Ops are being received.
connection_delete_entity_response_op_receive_rate	The rate at which DeleteEntityResponse Ops are being received.
connection_entity_query_response_op_receive_rate	The rate at which EntityQueryResponse Ops are being received.
connection_add_component_op_receive_rate	The rate at which AddComponent Ops are being received.
connection_remove_component_op_receive_rate	The rate at which RemoveComponent Ops are being received.
connection_authority_change_op_receive_rate	The rate at which AuthorityChange Ops are being received.
connection_component_update_op_receive_rate	The rate at which ComponentUpdate Ops are being received.
connection_command_request_op_receive_rate	The rate at which CommandRequest Ops are being received.
connection_command_response_op_receive_rate	The rate at which CommandResponse Ops are being received.
connection_egress_bytes	The number of bytes that have been sent. This refers to bytes encoded by the application layer - the actual number of bytes transmitted on the transport may be slightly higher.
connection_egress_bytes_rate	The rate at which data is being sent, in bytes per second.
connection_ingress_bytes	The number of bytes that have been received. This refers to bytes decoded by the application layer - the actual number of bytes received on the transport may be slightly higher.
connection_ingress_bytes_rate	The rate at which data is being received, in bytes per second.
connection_delta_compression_egress_bandwidth_saved_bytes	The number of network egress bytes saved through delta compressing component updates.
connection_delta_compression_egress_bandwidth_saved_bytes_rate	The rate at which network egress bandwidth is saved through delta compressing component updates, in bytes per second.
connection_delta_compression_egress_total_diffs_sent	The number of delta compressed component updates sent.
connection_delta_compression_egress_total_diffs_sent_rate	The rate at which delta compressed component updates are sent, in updates per second.
connection_delta_compression_egress_diffs_abandoned	The number of delta compressed component updates abandoned (due to taking too long to compute or being too large).
connection_delta_compression_egress_diffs_abandoned_rate	The rate at which delta compressed component updates are abandoned, in updates per second.
connection_delta_compression_ingress_bandwidth_saved_bytes	The number of network ingress bytes saved through delta compressing component updates.
connection_delta_compression_ingress_bandwidth_saved_bytes_rate	The rate at which network ingress bandwidth is saved through delta compressing component updates, in bytes per second.
raknet_receive_buffer_size	The current size of the RakNet receive buffer.
raknet_send_buffer_size	The current size of the RakNet rend buffer.
raknet_send_buffer_size_bytes	The number of bytes in the RakNet send buffer.
raknet_resend_buffer_size	The number of messages waiting in the RakNet resend buffer.
raknet_resend_buffer_size_bytes	The number of bytes in the RakNet resend buffer.
raknet_packet_loss_last_second	The packet loss over the last second. This number will range from 0.0 to 1.0.
raknet_packet_loss_lifetime	The packet loss average over the lifetime of the connection. This number will range from 0.0 to 1.0.
raknet_last_ping_seconds	The response time of the last ping emitted by the RakNet client.

Sending and receiving component updates

When the worker has authority over a particular component for some entity, it can send component updates to SpatialOS. This is done using the improbable.worker.Connection method sendComponentUpdate, which takes an EntityId instance and an instance of appropriate update class defined in the schema-generated code. The update classes are always called Update and are defined as a static nested classes of each implementation of ComponentMetaclass. For example, if we want to update a component MyComponent, the appropriate update class is nested in it and can by accessed as MyComponent.Update.

A component update can modify the value of a property or trigger an event. You can modify multiple properties and trigger multiple events in the same component update.

Component updates sent by the worker will appear in the operation list returned by a subsequent call to improbable.worker.Connection.getOpList. This means that Ops.ComponentUpdate callbacks will also be invoked for component updates triggered by the worker itself, but not immediately.

There are a couple of (related) reasons that callbacks for sent component updates should not be invoked immediately: this can lead to extremely unintuitive control flow when components are recursively updated inside a callback; it violates the guarantee that callbacks are only invoked as a result of a call to the process method on the improbable.worker.Dispatcher.

To be notified when a worker receives a component update on an entity in the worker’s local view of the simulation, use the improbable.worker.Dispatcher method onComponentUpdate with the same ComponentMetaclass as for sending updates. Note that the component updates can be partial, i.e. only update some properties of the component, do not necessarily have to contain data that is different from the workers current view of the component, and could have been sent by SpatialOS rather than a worker for synchronization purposes.

Sending and receiving component events

Sending and receiving events works much in the same way as component updates. For a schema like the following,

package example;

type SwitchToggled {
  int64 time = 1;
}

component Switch {
  id = 1234;
  bool is_enabled = 1;
  event SwitchToggled toggled;
}

triggering an event can be done as follows:

private static void triggerEvent(Connection connection, EntityId entityId) {
    Switch.Update update = new Switch.Update();
    update.addToggled(new SwitchToggled(1));
    connection.sendComponentUpdate(Switch.COMPONENT, entityId, update);
}

If you are not authoritative on the component, your event will be silently ignored.

Receiving an event works just like receiving a component update, by registering a callback on the dispatcher:

callbackKey = dispatcher.onComponentUpdate(Switch.COMPONENT, op -> {
    Switch.Update update = op.update;
    for (SwitchToggled toggleEvent : update.getToggled()) {
        System.out.println("Switch has been toggled at " + toggleEvent.getTime());
    }
});

Sending component interests

For each entity in its view, a worker receives updates for a set of the entity’s components. This set is the union of the set of components the worker has authority over, and a set of components it is explicitly interested in.

The initial set of explicitly interested components can be configured in the bridge settings. This is the default set of explicit interest for every entity. At runtime, the worker can override this set on a per-entity basis, by using the Connection.sendComponentInterest method.

Whenever the set of interested components for an entity changes, the worker will receive the appropriate onAddComponent and onRemoveComponent callbacks to reflect this change.

For example, you might be interested in one specific switch, but not every switch in the world:

Note that a worker is always considered to be interested in the components of the entities it is authoritative on, in addition to any others specificed in the bridge settings or using the method described above.

Sending and receiving component commands

To send a command to be executed by the worker currently authoritative over a particular component for a particular entity, use the improbable.worker.Connection method sendCommandRequest. This function takes a Class<C> instance, an entity ID, a request object, an optional timeout and an optional CommandParameters object. The C type needs to be a subclass of CommandMetaclass defined in the schema-generated code, and the request object needs has to be of the same type as defined in schema for given command C. The CommandParameters object contains a field called AllowShortCircuiting, which if set to true will try to “short-circuit” the command and avoid a round trip to SpatialOS in some cases. See documentation on commands for more information.

Before sending the command, a callback to handle the response should be registered with the improbable.worker.Dispatcher with onCommandResponse(Class<C>). The request ID (of type improbable.worker.RequestId<OutgoingCommandRequest>) returned by sendCommandRequest can be matched up with the one in the improbable.worker.Ops.CommandResponse to identify the request that is being responded to.

Note that commands may fail, so the improbable.worker.StatusCode field in the improbable.worker.Ops.CommandResponse should be checked, and the command can be retried as necessary. The caller will always get a response callback, but it can be one of several failure cases, including:

• ApplicationError (rejected by the target worker or by SpatialOS)
• AuthorityLost (target worker lost authority, or no worker had authority)
• NotFound (target entity, or target component on the entity, didn’t exist)
• PermissionDenied (sending worker didn’t have permission to send request)
• Timeout
• InternalError (most likely indicates a bug in SpatialOS, should be reported)

To handle commands issued by another worker, the opposite flow is used. Register a callback with the improbable.worker.Dispatcher with onCommandRequest(Class<C>); when the callback is executed, the worker should make sure to call the improbable.worker.Connection method sendCommandResponse, supplying the request ID (of type improbable.worker.RequestId<IncomingCommandRequest>) provided by the improbable.worker.Ops.CommandRequest and an appropriate response object, matching the response type defined in the schema. Alternatively, the worker can call sendCommandFailure to fail the command instead.

Entity queries

Note: In order to send an entity query, a worker must have permission to do so. For more information, see the Worker permissions page.

A worker can run remote entity queries against the simulation by using the improbable.worker.Connection method sendEntityQueryRequest. This takes an improbable.worker.Query.EntityQuery object and an optional timeout.

The query object is made of an improbable.worker.Query.Constraint and an improbable.worker.Query.ResultType. The constraint determines which entities are matched by the query, and the result type determines what data is returned for matched entities. Available constraints and result types are described below.

Constraint	Description
EntityIdConstraint	Matches a specific entity ID.
ComponentConstraint	Matches entities with a particular component.
SphereConstraint	Matches entities contained in the given sphere.
AndConstraint	Matches entities that match all of the given subconstraints.
OrConstraint	Matches entities that match any of the given subconstraints.
NotConstraint	Matches entities that do not match the given subconstraint.

Result type	Description
CountResultType	Returns the number of entities that matched the query.
SnapshotResultType	Returns a snapshot of component data for entities that matched the query. To select all components, use new SnapshotResultType(). To select every component whose ID is contained in the given set, use new SnapshotResultType(componentIdSet) (thus, pass an empty set to get no components but entity IDs only).

Important: You should keep entity queries as limited as possible. All queries hit the network and cause a runtime lookup, which is expensive even in the best cases. This means you should:

• always limit queries to a specific sphere of the world
• only return the information you need from queries (eg the specific components you care about)
• if you’re looking for entities that are within your worker’s checkout radius, search internally on the worker instead of using a query

Like other request methods, this returns an improbable.worker.RequestId request ID, which can be used to match a request with its response. The response is received via a callback registered with the improbable.worker.Dispatcher using the onEntityQueryResponse method.

The Ops.EntityQueryResponse contains an int resultCount field (for CountResultType requests) and a Map<EntityId, Entity> (for SnapshotResultType) requests. Again, success or failure of the request is indicated by the statusCode field of the response object, but in the failure case the result may still contain some data: the count or snapshot map might still contain the data for some entities that matched the query, but won’t necessarily contain all matching entities. This is because the worker might still be able to do something useful with a partial result.

Search results