Serialization reference

Unlike the C++, C# and Java SDKs, which provide generated classes that you can make use of to interact with schema in your workers, the C API leaves code generation completely up to you. To let you do this, the C API exposes a serialization library (contained within improbable/c_schema.h) with the Schema_ prefix. This page explains the two different methods of providing serialization code, and provides a reference of how to serialize various schema types.

Handle types

For each component known to a particular worker, the C API make use of 4 different data handle types:

• ComponentData - Represents a component at a single point in time. Should contain all fields.
• ComponentUpdate - Represents a component update. Can contain any number of fields from the component, as it represents a diff of some component data between two points of time.
• CommandRequest - Represents some command request data. Should contain all fields of the command request type.
• CommandResponse - Represents some command response data. Should contain all fields of the command response type.

In the other SDKs, these handle types are hidden (because they’re dealt with by the schema compiler); in the C API, they’re fully exposed. It’s up to you to serialize the above data (either directly when triggering an operation or by providing vtable functions, as described in the next section).

Specifying serialization code

All 4 handle types have the following structure (using Worker_ComponentUpdate as an example):

typedef struct Worker_ComponentUpdate {
  Worker_ComponentId component_id;
  Schema_ComponentUpdate* schema_type;
  Worker_ComponentUpdateHandle* user_handle;
} Worker_ComponentUpdate;

When using any handle type for any reason (for example, sending a component update), you are given the choice between either directly creating a serialized object (schema_type), or passing in your own custom handle (user_handle), which will later get serialized by a vtable function.

Serialize directly

Using this method, you perform the serialization directly when performing an operation (for example, sending a component update). This is the simplest way to serialize and deserialize data. To enable this, you need to specify a default vtable (so it applies to all components) which is empty (so it uses manual serialization):

Worker_ComponentVtable default_vtable;
memset(&default_vtable, 0, sizeof(Worker_ComponentVtable));
Worker_ConnectionParameters params = Worker_DefaultConnectionParameters();
// ...
params.default_component_vtable = &default_vtable;

To serialize a component update directly, you create the Schema_ComponentUpdate object (more on this below), serialize your data into it, then set the schema_type field to the Schema_ComponentUpdate object, leaving user_handle as NULL. When you do this, the C API will use this serialized data (and take ownership of it, so you don’t need to free anything) without doing any more work. For example:

Worker_ComponentUpdate update;
update.component_id = 54;
update.schema_type = Schema_CreateComponentUpdate(54);
// serialize into update.schema_type.
Worker_Connection_SendComponentUpdate(connection, 1, &update);

Deserialization is similar. For example, when processing a component update op, a Worker_ComponentUpdate will be provided with schema_type assigned to some serialized data received from the network (and user_handle set to NULL). You can then read this component update using the schema functions, as described below. For example:

Worker_ComponentUpdate* update;
if (op->op_type == WORKER_OP_TYPE_COMPONENT_UPDATE) {
  update = &op->component_update.update;
  // deserialize update from "update->schema_type", depending on the component ID.
}

Vtables

One drawback to the above approach is that serialization / deserialization needs to be done correctly in all the places that a particular component is used, so it’s easy to miss a case somewhere and end up with a bug which is difficult to track. Another drawback is that serialization is done in the game thread, rather than being done in the internal C API connection threads (assuming the language being used with the C API supports this).

The other option for serializing data is using “vtables” (named after the concept used to implement polymorphism). A vtable is a struct containing a component ID, and a set of 16 different function pointers (4 for each handle type). Instead of providing a serialized object in the schema_object field, you specify a custom handle in the user_handle field, leaving schema_object as NULL. This custom handle can be any object you like, as it’s a typedef to void*. For each handle type, you must provide the following functions:

• Serialize - converts a custom handle into a serialized Schema_* object.
• Deserialize - converts a serialized Schema_* object into a custom handle.
• Copy- creates a copy of a custom handle object (managed then deleted by the C API using your Free method).
• Free - frees a custom handle allocated using Copy.

These functions have the following signature (using component update as an example):

typedef void Worker_ComponentUpdateFree(Worker_ComponentId component_id, void* user_data, Worker_ComponentUpdateHandle* handle);
typedef Worker_ComponentUpdateHandle* Worker_ComponentUpdateCopy(Worker_ComponentId component_id, void* user_data, Worker_ComponentUpdateHandle* handle);
typedef uint8_t Worker_ComponentUpdateDeserialize(Worker_ComponentId component_id, void* user_data, Schema_ComponentUpdate* source, Worker_ComponentUpdateHandle** handle_out);
typedef void Worker_ComponentUpdateSerialize(Worker_ComponentId component_id, void* user_data, Worker_ComponentUpdateHandle* handle, Schema_ComponentUpdate** target_out);

Note that each function signature contains a user_data parameter. This is unrelated to the user_handle specified in the handle type. Instead, this is an arbitrary pointer that is set inside the Worker_ComponentVtable object and also controlled by you. You can leave this as NULL if it’s not needed, but one use case for this parameter could be to pass in a dynamic language VM context so you can invoke some dynamic language code from your C callback.

The simplest implementation of these methods would use malloc and free in C to allocate a struct you manage yourself, and using the pointer returned by malloc as your Worker_ComponentUpdateHandle. If the C API is used in a language which doesn’t support pointers, a more complicated example could use a hash map or dictionary. You’d use an index as a key to a blob of data, and use that index as your Worker_ComponentUpdateHandle. This is similar to the approach used in the C# and Java SDKs.

A simplified example in pseudo-C# (assuming that C bindings have been set up) could look like:

// An example of a generated struct for "Position".
struct Position
{
  public double x;
  public double y;
  public double z;
};

// A class which manages a dictionary from an integer to a blob of managed memory.
class Serialization
{
  internal static Dictionary<Integer, Object> ObjectHandles;
  
  internal static Integer AddObject(object o) { ... }
  internal static void RemoveObject(Integer handle) { ... }
}

// An example of a generated class which implements the vtable functions for "Position".
class PositionSerialization
{
  static unsafe void ComponentUpdateFree(
    ComponentId componentId,
    void* userData,
    ComponentUpdateHandle* handle)
  {
    // Removes the object with index 'handle' from the dictionary. The object will get garbage
    // collected later.
    Serialization.RemoveObject(handle);
  }

  static unsafe ComponentUpdateHandle* ComponentUpdateCopy(
    ComponentId componentId,
    void* userData,
    ComponentUpdateHandle* handle)
  {
    // This function will not copy the data, but create a second handle which points to the same
    // data.
    object o;
    Serialization.ObjectHandles.TryGetValue(handle, out o);
    if (o != null)
    {
      // Allocate a second handle to the same data.
      Integer newHandle = Serialization.AddObject(o); 
      return newHandle;
    }
    return null;
  }

  static unsafe byte ComponentUpdateDeserialize(
    ComponentId componentId,
    void* userData,
    SchemaComponentUpdate* source,
    ComponentUpdateHandle** targetOut)
  {
    var position = new Position();
    // read `source` using schema APIs and write data to position struct.
    // return 0 (false) if serialization fails.
    Integer newHandle = Serialization.AddObject(position);
    *targetOut = newHandle;
    return 1; // (true)
  }

  static unsafe void ComponentUpdateSerialize(
    Worker_ComponentId componentId,
    void* userData,
    ComponentUpdateHandle* handle,
    SchemaComponentUpdate** targetOut)
  {
    object o;
    Serialization.ObjectHandles.TryGetValue(handle, o)
    if (o == null)
    {
      return;
    }

    var position = (Position) o;
    // Read position struct and write data using schema APIs to *targetOut.
  }
}

Serialization reference

We provide a C example project which serializes components with both approaches described above. The remainder of this page is a reference which describes in detail how to use the schema C API to serialize different schema types, such as commands, lists, options, etc.

In the code examples, we assume that Schema_Object* fields_object; has been defined already and is set to the schema object which contains the fields of the Example schema type at the beginning of each section. More about schema objects can be found below.

Primitive types

enum MyEnum {
  FOO = 1;
  BAR = 2;
  BAZ = 3;
}

type Example {
  int32 value = 3;
  EntityId entity_id = 4;
  MyEnum my_enum = 5;
}

To write a primitive type to a schema object, you can use any of the Schema_Add* functions, depending on the primitive type. For example, to write an integer 1234 to the field above, you can write the following code:

Schema_AddInt32(fields_object, 3, 1234);

To read a primitive type from a schema object, you can use any of the Schema_Get* functions, depending on the primitive type. For example, to read the integer in the field above into a variable, you can write the following code:

int my_value = Schema_GetInt32(fields_object, 3);

The mapping from schema type to function family is given below:

.schema type	function family
int32	Int32
int64	Int64
uint32	Uint32
uint64	Uint64
sint32	Sint32
sint64	Sint64
fixed32	Fixed32
fixed64	Fixed64
sfixed32	Sfixed32
sfixed64	Sfixed64
bool	Bool
float	Float
double	Double
string	Bytes
EntityId	EntityId
bytes	Bytes
user-defined enum	Enum
user-defined type	Object

An EntityId is stored as an int64 under the hood, but for clarity, we expose separate functions for serializing and deserializing entity IDs. For example:

Worker_EntityId my_value = Schema_GetEntityId(fields_object, 4);

An enum is stored as an uint32 under the hood, but similarly to EntityId, we expose separate functions for serializing and deserializing enums. For example:

typedef enum MyEnum { FOO = 1, BAR = 2, BAZ = 3 } MyEnum;
MyEnum my_value = (MyEnum)Schema_GetEnum(fields_object, 5);

Lists

type Example {
  list<float> value = 6;
}

A list field is a field whose ID occurs 0 or more times. In generated code, you add values to a list field any number of times to populate your list (whilst for singular primitive types mentioned above, you must add a value exactly once). An example of writing a list in this way can be the following (using the above list field as an example):

float my_list[4] = {1, 2, 3, 4};
Schema_AddFloat(fields_object, 6, my_list[0]);
Schema_AddFloat(fields_object, 6, my_list[1]);
Schema_AddFloat(fields_object, 6, my_list[2]);
Schema_AddFloat(fields_object, 6, my_list[3]);

However, this is relatively slow in practice due to having to iterate over the list and copy the data. The array can be provided to the C API without copying using the Schema_Add*List family of functions. As a result, the above code can be simplified to:

float my_list[4] = {1, 2, 3, 4};
Schema_AddFloatList(fields_object, 6, my_list, 4);

When reading a list field from a schema object, you can:

• obtain the count with the Schema_Get*Count family of functions.
• retrieve individual elements with the Schema_Index* family of functions.
• copy the entire list into a buffer managed by yourself using the Schema_Get*List family of functions.

For example:

/* access the 3rd element (index 2). */
float single_element = Schema_IndexFloat(fields_object, 6, 2);

/* obtaining the complete list. */
uint32_t count = Schema_GetFloatCount(fields_object, 6);
float* array_data = (float*)malloc(sizeof(float) * count);
Schema_GetFloatList(fields_object, 6, array_data);

Strings and bytes

type Example {
  string value_string = 1;
  bytes value_bytes = 2;
}

String and bytes fields are both treated as bytes in the C API, because they are both treated in the same way over the wire. In SpatialOS, the only difference is how code is generated for them in the C++, C#, and Java SDK, and how they’re visualized in the inspector. To write a byte buffer to a schema object (without copying), you can do this with the following code:

/* write string. ensure to exclude the null terminator. */
const char* text = "Hello World.";
Schema_AddBytes(fields_object, 1, (const uint8_t*)text, sizeof(char) * strlen(text));

/* write bytes */
uint32_t bytes_length;
unsigned char* bytes = create_byte_buffer(&bytes_length);
Schema_AddBytes(fields_object, 2, bytes, bytes_length);

Sometimes, it’s difficult to ensure that the byte buffers live long enough. In this case, you can reserve a buffer to a specified length using Schema_AllocateBuffer, which is managed by the C API and is guaranteed to have the same lifetime as the schema object. You can modify the above example to use this function in the following way:

/* write string. */
const char* text = "Hello World.";
uint32_t text_length = sizeof(char) * strlen(text); /* ensure to exclude null-terminator. */
uint8_t* text_buffer = Schema_AllocateBuffer(fields_object, text_length);
memcpy(text_buffer, text, text_length);
Schema_AddBytes(fields_object, 1, text_buffer, text_length);

/* write bytes */
uint32_t bytes_length;
unsigned char* bytes = create_byte_buffer(&bytes_length);
uint8_t* byte_buffer = Schema_AllocateBuffer(fields_object, bytes_length);
memcpy(byte_buffer, bytes, bytes_length);
free_byte_buffer(bytes);
Schema_AddBytes(fields_object, 2, byte_buffer, bytes_length);

To read bytes from a schema object, you can make use of Schema_GetBytesLength to obtain the length of the byte buffer, and Schema_GetBytes to get a pointer to the byte buffer itself. For example:

/* read string. */
uint32_t text_length = Schema_GetBytesLength(fields_object, 1);
const uint8_t* text_buffer = Schema_GetBytes(fields_object, 1);
/* ensure to include space for null terminator. */
char* text = (char*)malloc(text_length + 1);
memcpy(text, text_buffer, text_length);
text[text_length] = '\0';

/* read bytes. */
uint32_t bytes_length = Schema_GetBytesLength(fields_object, 2);
const uint8_t* bytes = Schema_GetBytes(fields_object, 2);
/* do something with bytes and bytes_length. */

Objects

type MyType {
  int32 some_value = 1;
  float another_value = 2;
}

type Example {
  MyType data = 4;
}

Objects in schema are containers that contain fields, including potentially more object fields. You can manipulate object fields in a similar way to how you manipulate primitive types, but can also manipulate fields within those object fields.

To write an object following the above schema, you could use the following code:

Schema_Object* type = Schema_AddObject(fields_object, 4);
Schema_AddInt32(type, 1, 1234);
Schema_AddFloat(type, 2, 1234.0f);

Similarly, we can read the object using the following code:

typedef struct MyType {
  int32_t some_value;
  float another_value;
} MyType;

Schema_Object* type = Schema_GetObject(fields_object, 4);

MyType data;
data.some_value = Schema_GetInt32(type, 1);
data.another_value = Schema_GetFloat(type, 2);

Options

type Example {
  option<uint32> value = 2;
}

An option field is a field whose ID occurs either 0 or 1 times. It can be thought of as a list which can have either 0 or 1 elements. Therefore, options are used in a similar way to lists. To write an option value, you can call Schema_Add* to set the option to a value, or you can keep the value empty by omitting the call to Schema_Add*. For example:

/* places 1234 in a uint32 option field. */
Schema_AddUint32(fields_object, 2, 1234);

When reading an option field from a schema object, you can use the Schema_Get*Count family of functions to determine whether the option field is set. For example:

uint32_t* value = NULL;
if (Schema_GetUint32Count(fields_object, 2) == 1) {
  /* obtain the value. */
  value = (uint32_t*)malloc(sizeof(uint32_t));
  *value = Schema_GetUint32(fields_object, 2);
}

Maps

type Example {
  map<uint32, float> value = 1;
}

A map field is a field which stores key-value pairs (with unique keys). In terms of serialization, it is functionally equivalent to reading and writing a list of objects, where each object is a key-value pair. The field IDs of the key and value are 1 and 2 respectively, which are also defined with the SCHEMA_MAP_KEY_FIELD_ID and SCHEMA_MAP_VALUE_FIELD_ID constants respectively. To write a map, you can use the following code to add a single key value pair [100 -> 25.0f]:

/* for each key value pair.. */
Schema_Object* kvpair = Schema_AddObject(fields_object, 1);
Schema_AddUint32(kvpair, SCHEMA_MAP_KEY_FIELD_ID, 100);
Schema_AddFloat(kvpair, SCHEMA_MAP_VALUE_FIELD_ID, 25.0f);

Similarly, to read a map field, you would make use of Schema_GetObjectCount to obtain the number of key-value pairs, then use Schema_IndexObject to read them. For example:

uint32_t map_size = Schema_GetObjectCount(fields_object, 1);
for (uint32_t i = 0; i < map_size; ++i) {
  Schema_Object* kvpair = Schema_IndexObject(fields_object, 1, i);
  uint32_t key = Schema_GetUint32(kvpair, SCHEMA_MAP_KEY_FIELD_ID);
  float value = Schema_GetFloat(kvpair, SCHEMA_MAP_VALUE_FIELD_ID);
  // use 'key' and 'value'
  (void)key;
  (void)value;
}

Component data

component TestComponent {
  id = 10000;
  int32 foo = 1;
  float bar = 2;
}

Component data is represented as a single object, known as the “fields” object. If you create a Schema_ComponentData object to be saved to a snapshot, or you receive some component data when handling an “AddComponentOp”, you can access the “fields” object using the Schema_GetComponentDataFields function. If the fields object does not exist yet, then this function will automatically create it for you. This “fields” object will contain the fields of your component (defined either inline in the component, or using the data statement in .schema). For example, to create a component data object that corresponds to TestComponent:

Schema_ComponentData* schema_data = Schema_CreateComponentData(10000);
Schema_Object* fields_object = Schema_GetComponentDataFields(schema_data);
Schema_AddInt32(fields_object, 1, 1234);
Schema_AddFloat(fields_object, 2, 55.0f);
/* use schema_data somewhere. */

Similarly, you can read a component data object with the following code:

/* assume we have a "Schema_ComponentData* schema_data", possibly contained in a
 * Worker_SnapshotData or in a vtable callback. */
Schema_Object* fields_object = Schema_GetComponentDataFields(schema_data);
int32_t foo = Schema_GetInt32(fields_object, 1);
float bar = Schema_GetFloat(fields_object, 2);

Component updates

type TestEvent {
  uint32 counter = 1;
  float size = 2;
}

component TestComponent {
  id = 10000;
  int32 foo = 1;
  float bar = 2;

  event TestEvent my_event;
}

A component update can be thought of as a diff between two component snapshots. They contain two objects, a “fields” object and an “events” object.

Fields

In terms of serialization, the “fields” object is equivalent to the “fields” object in a component data. However, all primitive fields are treated as option<...>’s instead (options, lists and maps are treated the same). That means that you can choose to not call Schema_Add* for a particular field ID, and that field will not be included in the component update. For example:

Schema_ComponentUpdate* schema_update = Schema_CreateComponentUpdate(10000);
Schema_Object* fields_object = Schema_GetComponentUpdateFields(schema_update);
Schema_AddInt32(fields_object, 1, 1234);
/* use schema_update somewhere. */

When receiving a component update, there is no guarantee that a field will contain a value. Similar to options, you should use the Schema_Get*Count family of functions to check whether a field is contained within an update. To read an update to TestComponent:

/* assume we have a "Schema_ComponentUpdate* schema_update", possibly contained in a
 * Worker_SnapshotUpdate or in a vtable callback. */
Schema_Object* fields_object = Schema_GetComponentUpdateFields(schema_update);
int32_t foo;
float bar;
if (Schema_GetInt32Count(fields_object, 1) > 0) {
  foo = Schema_GetInt32(fields_object, 1);
}
if (Schema_GetFloatCount(fields_object, 2) > 0) {
  bar = Schema_GetFloat(fields_object, 2);
}

Clearing fields

So far, it’s possible to write an option, list or map which as at least one element in it, otherwise the update is considered to not have a change to that field ID. However, in some cases, you wish to assign an option, list or map field to the empty state rather than leave it unchanged. The Schema_*ComponentUpdateClearedField family of functions can be used for this purpose. For example, to clear a list field (by setting the field to an empty list), you can write:

/* assume we have a "Schema_ComponentUpdate* schema_update", possibly contained in a
 * Worker_SnapshotUpdate or in a vtable callback. */
/* specify that this update sets field 3 to the empty option/list/map. */
Schema_AddComponentUpdateClearedField(schema_update, 3);

When receiving a component update, you can obtain the list of fields to assign to the empty option/list/map with the following code:

/* assume we have a "Schema_ComponentUpdate* schema_update", possibly contained in a
 * Worker_SnapshotUpdate or in a vtable callback. */
uint32_t cleared_field_count = Schema_GetComponentUpdateClearedFieldCount(schema_update);
Schema_FieldId cleared_field_id;
for (uint32_t i = 0; i < cleared_field_count; ++i) {
  cleared_field_id = Schema_IndexComponentUpdateClearedField(schema_update, i);
  /* assigned `cleared_field_id` to the empty option/list/map. */
}

Events

Events are stored as lists of objects which contain the event data, as part of the “events” object. Each field ID in the “events” object corresponds to the “event ID”, which is a 1-based index depending on the order of the events in the component.

To trigger a single ‘my_event’ event as part of a component update, you can use the following code:

Schema_ComponentUpdate* schema_update = Schema_CreateComponentUpdate(10000);
Schema_Object* events_object = Schema_GetComponentUpdateEvents(schema_update);
Schema_Object* event_data = Schema_AddObject(events_object, 1); /* event ID 1 */
Schema_AddUint32(event_data, 1, 122);                           /* counter */
Schema_AddFloat(event_data, 2, 25.0f);                          /* size */
/* use schema_update somewhere. */

When receiving a component update, you need to ensure that all instances of each event type are processed, as a single component update can contain many copies of the same event being triggered. For example:

/* assume we have a "Schema_ComponentUpdate* schema_update", possibly contained in a
 * Worker_SnapshotUpdate or in a vtable callback. */
Schema_Object* events_object = Schema_GetComponentUpdateEvents(schema_update);
/* iterate over all 'my_event' events in this update */
Schema_Object* event_data;
for (uint32_t i = 0; i < Schema_GetObjectCount(events_object, 1); i++) {
  event_data = Schema_IndexObject(events_object, 1, i); /* event ID 1 */
  uint32_t counter = Schema_GetUint32(event_data, 1);
  float size = Schema_GetFloat(event_data, 2);
  // use 'counter' and 'size'.
  (void)counter;
  (void)size;
}

Command requests and responses

type CmdRequest {
  bytes data = 1;
  float value = 2;
}

type CmdResponse {
  int32 value = 1;
}

component TestComponent2 {
  id = 10001;

  command CmdResponse my_command(CmdRequest);
}

Command requests and response objects are serialized in almost exactly the same way as component data. The only difference is that you need to call Schema_GetCommandRequestObject to access the command request object from an instance of Schema_CommandRequest*, or Schema_GetCommandResponseObject to access the command response object from an instance of Schema_CommandResponse*. In addition, you need to specify a “command ID” when creating a command request or response object. This ID is similar to an “event ID”, because it is also a 1-based index depending on the order of the commands in the component. To write a command request or response, you can use the following code (using request as an example):

Schema_CommandRequest* schema_request = Schema_CreateCommandRequest(10001, 1);
Schema_Object* request_object = Schema_GetCommandRequestObject(schema_request);
uint32_t bytes_length;
const uint8_t* bytes_buffer = create_byte_buffer(&bytes_length);
Schema_AddBytes(request_object, 1, bytes_buffer, bytes_length);
Schema_AddFloat(request_object, 2, 55.0f);
/* use schema_request somewhere. */

Similarly, you can read a command request or response object with the following code (using request as an example):

/* assume we have a "Schema_CommandRequest* schema_request", possibly contained in a
 * Worker_SnapshotData or in a vtable callback. */
Schema_Object* request_object = Schema_GetCommandRequestObject(schema_request);
const uint8_t* bytes_buffer = Schema_GetBytes(request_object, 1);
uint32_t bytes_length = Schema_GetBytesLength(request_object, 1);
float value = Schema_GetFloat(request_object, 2);

Search results