Technical Infrastructure

Descriptions of connection methods and encoding.

Connecting with ZeroMQ

ZeroMQ is the transport mechanism used to connect to the System from a Client. Connections are implmented using the Broker/Dealer model within the ZeroMQ paradigm with the Client being the Dealer to the Server's Broker. A separate connection is required for transactions, requests and broadcasts as a discrete thread on its own port services each one of these kinds of messages.

Connecting for Transactions

To connect for transaction entry and responses, the Client uses a ZMQ_DEALER socket to connect to the address:

tcp://IPADDR:11100

IPADDR is the IP address of the server. 11100 is the default transaction port (an alternative port could be provided, depending on the specific implementation). For a transaction connection, it is necessary to log into the Server with the Login message before any other messages can be used.

Connecting for Requests

To connect for requests and responses, the Client uses a ZMQ_DEALER socket to connect to the address:

tcp://IPADDR:11500

IPADDR is the IP address of the server. 11500 is the default request port (an alternative port could be provided, depending on the specific implementation). For a request connection, it is necessary to log into the Server with the Login message before any other messages can be used.

Encoding in the API

Each message is a fixed size C struct.

The maximum size of an input message is the size of the union type te_queue_in_t and the maximum size of an output message is the size of the union type te_queue_out_t. The actual size of the message on the wire may be less than the size of the union. To determine the real message size, use the function TE_sizeof_message.

All the lengths described in the following descriptions are given in bits.

Numerical Encoding

Numbers are encoded in little endian binary format.

String Encoding

Strings are fixed-width UTF-8 strings (to support international characters) that are null-padded to the right up to the length described in the message specifications. The length is given in bytes. There is no guarantee of a terminating null byte — in other words, a 20-byte string may contain up to 20 character bytes.

Timestamp Encoding

Timestamp fields are encoded as a 64-bit number that is the number of nanoseconds since 00:00:00 1st January 2000 UTC.

Key-Value Pairs

Most messages in the API contain one or more key-value pairs.

A key-value pair consists either of a key field and a fixed-length data field, or a key field, a length field and a variable-length data field. The high bit of the key field determines which of these two formats applies. The meaning of each key is described in a separate document since they vary between instances and releases.

If the high bit of the key field is 0, the format for a key-value pair is:

<KEY><DATA>

where:

<KEY> is a short (16 bit) code that defines the data field. Note that since the high bit is reserved, the code is encoded in the low 15 bits of the field.
<DATA> is an int (64 bit) data value.

If the high bit of the key field is 1, the format for a key-value pair is:

<KEY><LENGTH><DATA>

where:

<KEY> is a short (16 bit) code that defines the data field. Note that since the high bit is reserved, the code is encoded in the low 15 bits of the field.
<LENGTH> a short (16 bit) number that gives the number of bytes in the <DATA> field that follows.
<DATA> is the data defined the by the <KEY> field, of length <LENGTH> bytes.

Id Encoding

Every static table has a field called "Id".

The value is always a 64-bit integer. Effectively, this field is the key field for the table and it is used to uniquely reference the rows of each table in other messages for each running of the system.

The Id is an internal key for records in the system. They are designed with a specific format for internal reasons related performance and operational integration.

The 64-bit Id value is encoded into two parts as follows:

8 most significant bits encode a group, such as an exchange number.
24 least significant bits encode the actual Id within the group. This encoding scheme means that the whole Id-bit value can be used to filter a specific row in any table, or the 8 most significant bits can be used (with the 24 least significant bits set to 0) to filter all Ids within a particular group.