Introduction

MarketGrid uses a bespoke shared memory database for design and performance reasons. This memory database is made resilient by the application of three types of persistence Files;

• Text Files
• Transaction Logs
• Cache Files

MarketGrid Shared Memory ABI

When MarketGrid is built, part of the build process assembles all the "elements" that define the ABI of the current version of the system into a structure that is then hashed to produce the signature of this version of the platform.

The ABI consists of:

• all of the unloadable static tables, their fields, types and indexes. For example:

"table.Industry": {
	"Id": "signed int(64)",
	"Status": "set(unsigned int(32), Status)",
  "ShortName": "required string(24)",
  "Name": "required string(48)",
  "counterparty_work": "signed int(64)",
  "UpdateNumber": "unsigned int(64)",
  "OwnerUser": "foreign_key(User)",
  "OwnerGroup": "foreign_key(Group)",
  "OwnerFirm": "foreign_key(Firm)",
  "CreateUser": "foreign_key(User)",
  "CreateTimestamp": "unsigned int(64) as TIMESTAMP",
  "UpdateUser": "foreign_key(User)",
  "UpdateTimestamp": "unsigned int(64) as TIMESTAMP"
},
"table.Industry.indexes": [
  "Unique(Name)",
  "Unique(ShortName)"
],

• All of the messages, their fields and types. For example:

"message.StatusSet": {
  "Table": "required enum(unsigned int(8), Table)",
  "Id": "foreign_key(All)",
  "Type": "required enum(unsigned int(8), StatusType)",
  "Status": "required set(unsigned int(32), Status)",
  "ClearFlag": "boolean"
},

• Some of the internal C structs such as messaging headers and request structs. For example:

"struct.ts_request_data_t": {
  "size": "signed int(32)",
  "data": "required string(200)",
  "messages": "required enum(unsigned int(16), MessageType)"
},

• All of the enums and their values. For example:

	"enum.Aggressor": {
		"0": "None",
		"1": "Buyer",
		"-1": "Seller"
	},

Persistence Files

Everything that changes data in MarketGrid is a Transaction. That is, everything is accomplished by the submission of messages to the Matching Engine which sequences, persists and processes said message producing a response to the sender and computing and then committing the resulting changes to the shared memory database.

This includes, the initial bootstrapping of the engine with static data. Each static data record is loaded as a message, either directly from a cache file or after transformation from within a tab separated record in a text file (tsv).

Text Files

MarketGrid tsv files are the human readable and editable files from which the system can be bootstrapped.

These files are used to provide initial datasets from which to start MarketGrid instances and also provide the representation on which transformations can be performed. Such transformations are a key part of the migration process between versions of MarketGrid.

Transaction Logs

When a transaction is received by the Matching Engine. It is received in the ZeroMQ infrastructure and the logging thread within the Matching Engine process takes one transaction at a time, effectively sequencing the input from the multitude of connected clients (including the Matching Engine itself) into a single stream of messages that are then persisted in this sequence into the transaction log.

The transaction log consists of two files:

• Index File
• Transaction File

Together these files allow for the replay and inspection of a given sequence of inbound messages.

The structures used within the transaction log are described in the Transaction Log Structure documentation. Note that the Transaction Log Record in the transaction log index file is a duplicate of the record located in transaction log file. The Matching Engine first writes the transaction log file copy and then duplicates this into the sepearate index file. This allows the index to be reconstructed from the just the transaction log file iteself.

There is a convention for the naming and storage location of these files. Whilst this convention can be overriden at invocation time it is rarely if ever necessary. By convention, the transaction log is created in a directory called logs/transactions/YYYYMMDD-hhmmss, where the directory is named after the timestamp at the moment that the Matching Engine was started and then the two files are named;

YYYYMMDD-hhmmss.idx
YYYYMMDD-hhmmss

for the Index File and Transaction File respectively.

The MarketGrid Inspector (mg inspect command) provides a web interface for the examination of these transaction logs.

Index File

The index file is a series of fixed size records. The MarketGrid Inspector is the best way to examine these structures since they may change from time to time. The key point is that in addition to all the meta data contained in each index record, there are two fields;

• offset - the byte offset from the start of the Transaction File to find the specific entry for this message
• size - the number of bytes in the message itself.

Transaction File

The format of the transaction file is a header record followed by a sequence of records containing:

• The same record as stored in the Index File
• A header record for the message received - meta data about the actual message
• The body of the message received

Cache Files

Cache files are the native persistence format between invocations of MarketGrid. A set of cache files consist of one Transaction File and one Index File file per table and each pair of Transaction and Index files is effectively a "mini" Transaction Log containing the static data messages that can be "replayed" in order to load the data in each file.

Cache files are written in a number of different circumstances;

• Graceful shutdown of the Matching Engine when writecachefilesonsignal option is in force
• During a system cycle (mg cycle command)

Transformations

It is possible to convert between .tsv files and Transaction Logs (and vice versa) using some utilities provided with MarketGrid.

Extracting Static Data

Extracting the static data messages from a complete MarketGrid transaction log can be done using the program log_to_tsv to write them to individual .tsv files. The tsv_to_log program is used to convert any given set of .tsv files back into individual cache files.

Converting Whole Transaction Logs

A complete MarketGrid transaction log can be converted to JSON format using the log_to_json program. Any correctly formatted JSON file can be converted back into a MarketGrid transaction log using the json_to_log program.

Subject to the dangerous implications of changing the existing sequence of transactions, excising a transaction from the JSON format before converting back to transaction log format would have the effect of removing that transaction from the original MarketGrid transaction log.

Persisting Data

Using writecacheonsigterm Option

The writecacheonsigterm option determines the behaviour of the Matching Engine when it shuts down. When this option is set, the engine will write the current state of the Shared Memory database to a set of cache files.

Using mg cycle

A "system cycle" is initiated by the CycleSystem message. See the (Shared Memory Management)[operation/shared-memory-management/Shared-Memory-Management] documentation.

Starting from Persisted Data

When MarketGrid is started the settings provided to the command at run time, either by the values in the scenario being started or by the override parameters given on the command line, determine which source of data is used to start the system.

In a MarketGrid scenario file, the load option is used to specify the persisted data from which the system should start.

Starting from Text Files

In a MarketGrid scenario file, the load option is used to specify the persistend

Using autoload Option

When the Matching Engine is started with the autoload option set, it well look for some persisted data from which to start. This search will override the base data defined in the scenario file until the last step in the process in which case it will select the source defined in the load option/configuration setting.

The sequence of persistence data through which the Matching engine will search is described in the (Shared Memory Management)[operation/shared-memory-management/Shared-Memory-Management] documentation,