Migration

How trading system data is persisted and loaded between invocations of MarketGrid with different versions of the system.

Introduction

MarketGrid is designed to detect changes between versions and manage the process to allow the new version to bootstrap from the persisted state of the previous version.

MarketGrid Internal Version

Each version of MarketGrid has an "Shared Memory ABI" (SMABI) that is stamped into the binaries and various persistance files created by the system.

When changes are made in a new version of MarketGrid that affect the SMABI a migration job will be needed to allow the new version of MarketGrid to start from data that is persisted from the previous version.

There are two types of data that may need to be transformed in a data migration:

  • Engine Persistence Data
  • Market Database

Engine Persistence Data

The data that the matching engine loads into shared memory during standard restarts is stored in binary encoded cache files. There is more information on these files in the Shared Memory Management. section of the documentation. The engine checks if an engine data migration is necessary by comparing the SMABI checksum which is in each cache file with the engine's own SMABI checksum.

Market Database

The historical data that is generated by the engine is stored in the market database (MDB). If there is a change in the schema to this database, the data here will also need to be migrated to the new schema.

In a similar fashion to the engine data, the market database determines if the database needs migration by comparing a checksum of the new version's lib/TorQ/sym.q file and the existing /data/hdb/checksum file. This sym.q file contains the full database schema.

Note: The migration of Engine and MDB data are independent and either, neither or both may need to be migrated during a version change.

Migration Process

This guide assumes that the engine is configured to persist data and the writecacheonsigterm and autoload flags are enabled in the scenario. More details on these flags are can be found in the documentation for the te_engine process.

Prevalidation

It is possible to test the migration process in a deployed environment. This should always be done before running the migration on a production system.

Testing a Migration

The following needs to be prepared and tested

  1. Acquire both the Existing Version image and the New Version image
  2. Determine if engine data and/or MDB data needs to be migrated
  3. Create the migration script(s) (if not provided by Carta in the release)
  4. Test the migration scripts using production data (or similar)
  5. Test the system after the migration

More detailed documents:

describe the processes for creating the two types of migration job that may be required.

Overview

The following sequence diagram gives an overview of the migration process.

%%{
    init: {
        'theme': 'base',
        'themeVariables': {
            'primaryColor': '#5098FC',
            'secondaryColor': '#AACCCC',
            'fontFamily': 'trebuchet ms',
            'actorFontFamily': 'trebuchet ms'
        }
    }
}%%
sequenceDiagram
    Operation->>Matching Engine: mg start
    alt engine_cache_dir link exists
        alt SMABI unchanged
			Matching Engine->>+Logging: Load Cache files
            loop for each file in cache file list
                Logging ->>- Matching Engine: Send "Load Cache File" Message
            end
        else SMABI changed
			Matching Engine->>+Logging: Load Cache files
            loop for each file in cache file list
                Logging ->>+ Migrate Job: Convert cache file to tsv file
            end
            loop for each file in tsv file list (converted)
                Migrate Job ->>+ Engine Migration: Migrate tsv
                Engine Migration ->>- Migrate Job: report location
            end
            Migrate Job ->>- Logging: report location
            loop for each file in tsv file list (migrated)
                loop for row in tsv file (migrated)
                    Logging ->> Logging: Convert tsv row into "Static Data" Message
                    Logging ->>- Matching Engine: Send "Static Data" Message
                end
            end
        end
    else transaction_log link exists
        alt SMABI unchanged
			Matching Engine ->>+ Logging: Load Transaction Log
            loop for each transaction in transaction log
                Logging ->>- Matching Engine: Send logged Transaction
            end
        else SMABI changed
			Matching Engine ->>+ Logging: Load Transaction Log
            Logging ->>+ Migrate Job: Extract tsv files from transaction log
            loop for each file in tsv file list (converted)
                Migrate Job ->>+ Engine Migration: Migrate tsv
                Engine Migration ->>- Migrate Job: report location
            end
            Migrate Job ->>- Logging: report location
            loop for each file in tsv file list (migrated)
                loop for row in tsv file (migrated)
                    Logging ->> Logging: Convert tsv row into "Static Data" Message
                    Logging ->> Matching Engine: Send "Static Data" Message
                end
            end
            loop for each transaction in transaction log (migrated)
                Logging ->>- Matching Engine: Replay stored Transaction
            end
        end
    else fallback
        Matching Engine ->>+ Logging: Load Text files
        loop for each file in tsv file list
            loop for row in tsv file
                Logging ->> Logging: Convert tsv row into "Static Data" Message
                Logging ->>- Matching Engine: Send "Static Data" Message
            end
        end
    end

The system may be deployed or orchestrated via a number of different means, the following sections are just examples using a Docker based containerised deployment to demonstrate the output of the processes involved in the migration process.

The logging samples shown below are created from a docker-compose file as shown below where the <image> might be any two arbitrary versions of MarketGrid.

version: "3.5"

services:
   mg:
    container_name: mg
    image: <image>
    ports:
      - 10080:10060 # UI port
      - 10443:10443 # UI port
      - 8888:8888 # inspect
    shm_size: '16gb' # this will set the size when RUNNING
    command: "mg start mg_migrate"
    volumes:
      - ./data/datasets/:/opt/MarketGrid/datasets
      - ./data/scenarios/:/opt/MarketGrid/scenarios
      - ./data/home/mg/:/home/mg
      - ./data/logs:/opt/MarketGrid/logs
      - ./data/migrations/:/opt/MarketGrid/migrations

Startup Version Current

Actions

  • mg start

Logs

Recreating mg ... done
Attaching to mg
mg    | Starting scenario: mg_migrate
mg    | Starting services: matching_engine, transaction_server, ui_server, nginx
mg    | Running task: ExpandTemplate(nginx/nginx.conf) [nginx]
mg    | transaction_server: [ INFO ] ZMQ version: 4.3.4 (src/te/te_tserver.cc:49)
mg    | transaction_server: [ INFO ] Listening on ipc:///opt/MarketGrid/var/sockets/mg_11000_transaction_raw_socket_11200 (src/te/te_tserver.cc:460)
mg    | transaction_server: [ INFO ] Listening on tcp://0.0.0.0:11200 (src/te/te_tserver.cc:460)
mg    | transaction_server: [ INFO ] Listening on ipc:///opt/MarketGrid/var/sockets/mg_11000_transaction_blob_socket_11600 (src/te/te_tserver.cc:460)
mg    | transaction_server: [ INFO ] Listening on tcp://0.0.0.0:11600 (src/te/te_tserver.cc:460)
mg    | transaction_server: [ INFO ] Connecting to ipc:///opt/MarketGrid/var/sockets/mg_11000_broadcast_queue_12004 (src/te/te_tserver.cc:419)
mg    | matching_engine: [ INFO ] MarketGrid 3.11.5-c2e11c86 (ABI: 7255cac751ffd2284b382851e5087d40b312271e) (src/te/te_engine.cc:47)
mg    | matching_engine: [ INFO ] Cleaning up shared memory instance with key mg_11000 (src/te/te_memory.cc:87)
mg    | matching_engine: [ INFO ] Creating shared memory instance key mg_11000 and size 44797397344 bytes (src/te/te_create.cc:122)
mg    | matching_engine: [ INFO ] Matching Engine (/opt/MarketGrid/bin/te_engine) Checksum: 7a5d3018cf69ae567cc9788df28f31 (src/te/te_engine.cc:97)
mg    | matching_engine: [ INFO ] ZMQ Version: 4.3.4 (src/te/te_queue.cc:197)
mg    | matching_engine: [ INFO ] Writing transaction log: /opt/MarketGrid/logs/transactions/20220602-044222/MGTransLog-20220602-044222 (src/te/te_queue.cc:290)
mg    | matching_engine: [ INFO ] Started transaction logger thread as Stand Alone machine (using 10000 buffers) (src/te/te_queue.cc:1666)
mg    | matching_engine: [ INFO ] Loading file: /opt/MarketGrid/datasets/xchg/cbl_baseline/Account.tsv (src/te/te_load_textfiles.cc:58)
mg    | matching_engine: [ INFO ] Loading file: /opt/MarketGrid/datasets/xchg/cbl_baseline/AccountType.tsv (src/te/te_load_textfiles.cc:58)
...
mg    | matching_engine: [ INFO ] Loading file: /opt/MarketGrid/datasets/xchg/cbl_baseline/ZoneRuleBatchZone.tsv (src/te/te_load_textfiles.cc:58)
mg    | matching_engine: [ INFO ] Loading file: /opt/MarketGrid/datasets/xchg/cbl_baseline/ZoneRules.tsv (src/te/te_load_textfiles.cc:58)
mg    | matching_engine: [ INFO ] Loaded 10185 records from /opt/MarketGrid/datasets/xchg/cbl_baseline (src/te/te_load_textfiles.cc:174)
mg    | matching_engine: [ INFO ] Set system key for JWT authentication with length 799 (src/te/te_usermanager.cc:221)
mg    | matching_engine: [ INFO ] Checking foreign paths in shared memory (pass 1) (build/generated/te_shm_load.cc:58719)
mg    | matching_engine: [ INFO ] Setting system mode to Initialising (src/te/te_status.cc:379)
mg    | matching_engine: [ INFO ] System time zone is: SYS (src/te/te_statics.cc:775)
mg    | matching_engine: [ INFO ] Setting system mode to Ready (src/te/te_status.cc:379)
mg    | transaction_server: [ INFO ] Connected to upstream at ipc:///opt/MarketGrid/var/sockets/mg_11000_broadcast_queue_12004 (src/te/te_tserver.cc:316)
mg    | transaction_server: [ INFO ] Connected to upstream at ipc:///opt/MarketGrid/var/sockets/mg_11000_transaction_queue_12001 (src/te/te_tserver.cc:317)
mg    | transaction_server: [ INFO ] Connected to upstream at ipc:///opt/MarketGrid/var/sockets/mg_11000_blob_queue_12009 (src/te/te_tserver.cc:318)
...
mg    | ui_server: [ INFO ] Started Websocket server listening at /opt/MarketGrid/var/sockets/ui_server_websocket
  1. Note the MarketGrid version and ABI checksum ( in this case 3.11.5-c2e11c86 and 7255cac751ffd2284b382851e5087d40b312271e respectively)
  2. Elided from this output are the Finished reading engine cache file lines that are printed for each file loaded

This is just a standard start of MarketGrid. There is nothing specific about the migration process that is affected by the startup prior to the migration.

There are a number of operational models under which MarketGrid can run from this point onwards. This document is not concerned with the differences in those models. For example, the system may be cycled on a daily basis, left running and cycled weekly or run continuously and use Shared Memory recycling from time to time.

Regardless, the next step as far as rolling out a new version of the MarketGrid software begins with a shutdown of the system. It is not possible to "hot load" a new version of the MarketGrid software, a system cycle is required.

Shutdown Version Current

Methods

  • mg stop
  • docker - terminate the container (gracefully)
  • kubernetes - terminate the pod

Given the importance of preserved state prior to a software update, when using an orchestration service such as kubernetes, it makes sense to engage the shutdown as an ad hoc process rather than just relying on the container shutdown. It's not an unrecoverable problem if there is an issue on the shutdown but given that the event ought to be a scheduled downtime it is a variable that can be removed from that process by using an ad hoc mg stop rather than just shutting down the container.

Log

mg    | matching_engine: [ WARN ] SIGTERM received, stopping process (te_engine) (src/te/te_signals.cc:70)
mg    | matching_engine: [ INFO ] Trading Engine shutdown starting (src/te/te_queue.cc:148)
mg    | transaction_server: [ INFO ] Finished (Terminated) (src/te/te_tserver.cc:54)
mg    | matching_engine: [ INFO ] Finished transaction logger thread (src/te/te_queue.cc:2135)
mg    | matching_engine: [ INFO ] Created cache directory: /opt/MarketGrid/logs/engine_cache/20220602-031017 (src/te/te_cache.cc:103)
mg    | matching_engine: [ INFO ] Setting system mode to Unload (src/te/te_status.cc:380)
mg    | matching_engine: [ INFO ] Doing unload pre-processing (src/te/te_statics.cc:1928)
mg    | matching_engine: [ INFO ] Orders - Retiring: 0, Writing out: 2863 (src/te/te_statics.cc:1758)
mg    | matching_engine: [ INFO ] Trades - Retiring: 0, Writing out: 841 (src/te/te_statics.cc:1804)
mg    | matching_engine: [ INFO ] Writing cache file (table ContactDetail[1]): /opt/MarketGrid/logs/engine_cache/20220602-031017/ContactDetail (src/te/te_cache.cc:310)
mg    | matching_engine: [ INFO ] Wrote 103 records to cache file: /opt/MarketGrid/logs/engine_cache/20220602-031017/ContactDetail (Max Id - 3715) (src/te/te_cache.cc:403)
mg    | matching_engine: [ INFO ] Writing cache file (table Enterprise[2]): /opt/MarketGrid/logs/engine_cache/20220602-031017/Enterprise (src/te/te_cache.cc:310)
mg    | matching_engine: [ INFO ] Wrote 14 records to cache file: /opt/MarketGrid/logs/engine_cache/20220602-031017/Enterprise (Max Id - 35061) (src/te/te_cache.cc:403)
mg    | matching_engine: [ INFO ] Writing cache file (table Firm[3]): /opt/MarketGrid/logs/engine_cache/20220602-031017/Firm (src/te/te_cache.cc:310)
...
mg    | matching_engine: [ INFO ] Wrote 630 records to cache file: /opt/MarketGrid/logs/engine_cache/20220602-031017/ZoneRules (Max Id - 630) (src/te/te_cache.cc:403)
mg    | matching_engine: [ INFO ] Writing cache file (table ZoneRuleBatch[109]): /opt/MarketGrid/logs/engine_cache/20220602-031017/ZoneRuleBatch (src/te/te_cache.cc:310)
mg    | matching_engine: [ INFO ] Wrote 6 records to cache file: /opt/MarketGrid/logs/engine_cache/20220602-031017/ZoneRuleBatch (Max Id - 6) (src/te/te_cache.cc:403)
mg    | matching_engine: [ INFO ] Writing cache file (table ZoneRuleBatchZone[110]): /opt/MarketGrid/logs/engine_cache/20220602-031017/ZoneRuleBatchZone (src/te/te_cache.cc:310)
mg    | matching_engine: [ INFO ] Wrote 31 records to cache file: /opt/MarketGrid/logs/engine_cache/20220602-031017/ZoneRuleBatchZone (Max Id - 31) (src/te/te_cache.cc:403)
mg    | matching_engine: [ INFO ] Setting system mode to Closed (src/te/te_status.cc:380)
mg    | matching_engine: [ INFO ] Engine cache write done (src/te/te_cache.cc:436)
mg    | matching_engine: [ INFO ] Cleaning up shared memory instance with key mg_11000 (src/te/te_memory.cc:87)
mg    | matching_engine: [ INFO ] Trading Engine shutdown complete (src/te/te_queue.cc:180)
  1. Elided from this output are the Writing cache file and Wrote NNN records that are produced for each table in the system.
  2. Note the presence of the Retiring: NNN, Writing out: MMM messages for some tables.
    1. This is the normal process of removing deleted records from the shared memory database.
      1. A deleted record is one for which the Status.Active bit has been turned off in the records Status field.
    2. All the data is still persisted in the Market Database for posterity.

Startup Version Next

Methods

Launching this process is the same as a normal startup. There are no additional parameters or activities required. As such there is no need for a variation to the standard method of the deployment model selected.

Log

Attaching to mg
mg    | Starting scenario: mg_migrate
mg    | Starting services: matching_engine, transaction_server, ui_server, nginx
mg    | matching_engine: [ INFO ] MarketGrid 3.13.3-cbd23f57 (ABI: 11c395d0adf9d9e96faf37ebf98469672fa93e95) (src/te/te_engine.cc:47)
mg    | matching_engine: [ INFO ] Cleaning up shared memory instance with key mg_11000 (src/te/te_memory.cc:87)
mg    | transaction_server: [ INFO ] ZMQ version: 4.3.4 (src/te/te_tserver.cc:49)
mg    | matching_engine: [ INFO ] Creating shared memory instance key mg_11000 and size 44798197360 bytes (src/te/te_create.cc:122)
mg    | transaction_server: [ INFO ] Listening on ipc:///opt/MarketGrid/var/sockets/mg_11000_transaction_raw_socket_11200 (src/te/te_tserver.cc:460)
mg    | transaction_server: [ INFO ] Listening on tcp://0.0.0.0:11200 (src/te/te_tserver.cc:460)
mg    | transaction_server: [ INFO ] Listening on ipc:///opt/MarketGrid/var/sockets/mg_11000_transaction_blob_socket_11600 (src/te/te_tserver.cc:460)
mg    | transaction_server: [ INFO ] Listening on tcp://0.0.0.0:11600 (src/te/te_tserver.cc:460)
mg    | transaction_server: [ INFO ] Connecting to ipc:///opt/MarketGrid/var/sockets/mg_11000_broadcast_queue_12004 (src/te/te_tserver.cc:419)
mg    | Running task: ExpandTemplate(nginx/nginx.conf) [nginx]
mg    | matching_engine: [ INFO ] Matching Engine (/opt/MarketGrid/bin/te_engine) Checksum: b1bbc0686b448d37d803e2ca9f18e7 (src/te/te_engine.cc:99)
mg    | matching_engine: [ INFO ] ZMQ Version: 4.3.4 (src/te/te_queue.cc:201)
mg    | matching_engine: [ INFO ] Writing transaction log: /opt/MarketGrid/logs/transactions/20220602-044635/MGTransLog-20220602-044635 (src/te/te_queue.cc:289)
mg    | matching_engine: [ INFO ] Started transaction logger thread as Stand Alone machine (using 10000 buffers) (src/te/te_queue.cc:1667)
mg    | matching_engine: [ INFO ] Loading from cache directory: /opt/MarketGrid/logs/engine_cache/20220602-044222 (src/te/te_cache.cc:168)
mg    | matching_engine: [ WARN ] Migrating engine cache logs with ABI: 7255cac751ffd2284b382851e5087d40b312271e, required ABI: 11c395d0adf9d9e96faf37ebf98469672fa93e95 (src/te/te_cache.cc:205)
mg    | matching_engine: [ INFO ] Converting cache files to TSVs using temporary directory: /tmp/migration-1KfCsO (src/te/te_cache.cc:135)
mg    | log_to_tsv: [ INFO ] Loading log file: /opt/MarketGrid/logs/engine_cache/20220602-044222/Account (src/te/log_to_tsv.cc:226)
mg    | log_to_tsv: [ INFO ] Writing to file: /tmp/migration-1KfCsO/Account.tsv (src/te/log_to_tsv.cc:48)
mg    | log_to_tsv: [ INFO ] Wrote 21 records (src/te/log_to_tsv.cc:238)
mg    | log_to_tsv: [ INFO ] Loading log file: /opt/MarketGrid/logs/engine_cache/20220602-044222/AccountType (src/te/log_to_tsv.cc:226)
mg    | log_to_tsv: [ INFO ] Writing to file: /tmp/migration-1KfCsO/AccountType.tsv (src/te/log_to_tsv.cc:48)
mg    | log_to_tsv: [ INFO ] Wrote 3 records (src/te/log_to_tsv.cc:238)
...
mg    | log_to_tsv: [ INFO ] Loading log file: /opt/MarketGrid/logs/engine_cache/20220602-044222/ZoneRuleBatchZone (src/te/log_to_tsv.cc:226)
mg    | log_to_tsv: [ INFO ] Writing to file: /tmp/migration-1KfCsO/ZoneRuleBatchZone.tsv (src/te/log_to_tsv.cc:48)
mg    | log_to_tsv: [ INFO ] Wrote 31 records (src/te/log_to_tsv.cc:238)
mg    | log_to_tsv: [ INFO ] Loading log file: /opt/MarketGrid/logs/engine_cache/20220602-044222/ZoneRules (src/te/log_to_tsv.cc:226)
mg    | log_to_tsv: [ INFO ] Writing to file: /tmp/migration-1KfCsO/ZoneRules.tsv (src/te/log_to_tsv.cc:48)
mg    | log_to_tsv: [ INFO ] Wrote 630 records (src/te/log_to_tsv.cc:238)
mg    | matching_engine: [ INFO ] Running cache file migration: /opt/MarketGrid/bin/../migrations/7255cac751ffd2284b382851e5087d40b312271e /tmp/migration-1KfCsO /opt/MarketGrid/bin (src/te/te_cache.cc:216)
mg    | matching_engine: [ INFO ] Loading file: /tmp/migration-1KfCsO/Account.tsv (src/te/te_load_textfiles.cc:58)
...
mg    | matching_engine: [ INFO ] Loading file: /tmp/migration-1KfCsO/Venue.tsv (src/te/te_load_textfiles.cc:58)
mg    | matching_engine: [ INFO ] Loading file: /tmp/migration-1KfCsO/Zone.tsv (src/te/te_load_textfiles.cc:58)
mg    | matching_engine: [ INFO ] Loading file: /tmp/migration-1KfCsO/ZoneGroup.tsv (src/te/te_load_textfiles.cc:58)
mg    | matching_engine: [ INFO ] Loading file: /tmp/migration-1KfCsO/ZoneRuleBatch.tsv (src/te/te_load_textfiles.cc:58)
mg    | matching_engine: [ INFO ] Loading file: /tmp/migration-1KfCsO/ZoneRuleBatchZone.tsv (src/te/te_load_textfiles.cc:58)
mg    | matching_engine: [ INFO ] Loading file: /tmp/migration-1KfCsO/ZoneRules.tsv (src/te/te_load_textfiles.cc:58)
mg    | matching_engine: [ INFO ] Loaded 10185 records from /tmp/migration-1KfCsO (src/te/te_load_textfiles.cc:174)
mg    | matching_engine: [ INFO ] Set system key for JWT authentication with length 799 (src/te/te_usermanager.cc:221)
mg    | transaction_server: [ INFO ] Connected to upstream at ipc:///opt/MarketGrid/var/sockets/mg_11000_broadcast_queue_12004 (src/te/te_tserver.cc:316)
mg    | transaction_server: [ INFO ] Connected to upstream at ipc:///opt/MarketGrid/var/sockets/mg_11000_transaction_queue_12001 (src/te/te_tserver.cc:317)
mg    | transaction_server: [ INFO ] Connected to upstream at ipc:///opt/MarketGrid/var/sockets/mg_11000_blob_queue_12009 (src/te/te_tserver.cc:318)
mg    | matching_engine: [ INFO ] Checking foreign paths in shared memory (pass 1) (build/generated/te_shm_load.cc:62518)
mg    | matching_engine: [ INFO ] Setting system mode to Initialising (src/te/te_status.cc:380)
mg    | matching_engine: [ INFO ] System time zone is: SYS (src/te/te_statics.cc:779)
mg    | matching_engine: [ INFO ] Setting system mode to Ready (src/te/te_status.cc:380)
mg    | ui_server: [ INFO ] Started Websocket server listening at /opt/MarketGrid/var/sockets/ui_server_websocket
  1. Elided from this output are the messages for every file, converted, migrated and loaded
  2. Note the new version and ABI checksum values for this release (3.13.3-cbd23f57 and 11c395d0adf9d9e96faf37ebf98469672fa93e95 respectively)

Actions

  • Startup MarketGrid engine as normal
    mg    | Starting scenario: mg_migrate
mg    | Starting services: matching_engine, transaction_server, ui_server, nginx
mg    | matching_engine: [ INFO ] MarketGrid 3.13.3-cbd23f57 (ABI: 11c395d0adf9d9e96faf37ebf98469672fa93e95) (src/te/te_engine.cc:47)
mg    | matching_engine: [ INFO ] Cleaning up shared memory instance with key mg_11000 (src/te/te_memory.cc:87)
mg    | transaction_server: [ INFO ] ZMQ version: 4.3.4 (src/te/te_tserver.cc:49)
mg    | matching_engine: [ INFO ] Creating shared memory instance key mg_11000 and size 44798197360 bytes (src/te/te_create.cc:122)
mg    | transaction_server: [ INFO ] Listening on ipc:///opt/MarketGrid/var/sockets/mg_11000_transaction_raw_socket_11200 (src/te/te_tserver.cc:460)
mg    | transaction_server: [ INFO ] Listening on tcp://0.0.0.0:11200 (src/te/te_tserver.cc:460)
mg    | transaction_server: [ INFO ] Listening on ipc:///opt/MarketGrid/var/sockets/mg_11000_transaction_blob_socket_11600 (src/te/te_tserver.cc:460)
mg    | transaction_server: [ INFO ] Listening on tcp://0.0.0.0:11600 (src/te/te_tserver.cc:460)
mg    | transaction_server: [ INFO ] Connecting to ipc:///opt/MarketGrid/var/sockets/mg_11000_broadcast_queue_12004 (src/te/te_tserver.cc:419)
mg    | Running task: ExpandTemplate(nginx/nginx.conf) [nginx]
mg    | matching_engine: [ INFO ] Matching Engine (/opt/MarketGrid/bin/te_engine) Checksum: b1bbc0686b448d37d803e2ca9f18e7 (src/te/te_engine.cc:99)
mg    | matching_engine: [ INFO ] ZMQ Version: 4.3.4 (src/te/te_queue.cc:201)
mg    | matching_engine: [ INFO ] Writing transaction log: /opt/MarketGrid/logs/transactions/20220602-044635/MGTransLog-20220602-044635 (src/te/te_queue.cc:289)
mg    | matching_engine: [ INFO ] Started transaction logger thread as Stand Alone machine (using 10000 buffers) (src/te/te_queue.cc:1667)
  • Start data load from cache directory
      mg    | matching_engine: [ INFO ] Loading from cache directory: /opt/MarketGrid/logs/engine_cache/20220602-044222 (src/te/te_cache.cc:168)
    • detect ABI checksum change
      Migrating engine cache logs with ABI: 7255cac751ffd2284b382851e5087d40b312271e, required ABI: 11c395d0adf9d9e96faf37ebf98469672fa93e95
    • convert text files
      Converting cache files to TSVs using temporary directory: /tmp/migration-1KfCsO
    • read source file
      mg    | log_to_tsv: [ INFO ] Loading log file: /opt/MarketGrid/logs/engine_cache/20220602-044222/Account (src/te/log_to_tsv.cc:226)
    • write temporary file
      mg    | log_to_tsv: [ INFO ] Writing to file: /tmp/migration-1KfCsO/Account.tsv (src/te/log_to_tsv.cc:48)
mg    | log_to_tsv: [ INFO ] Wrote 21 records (src/te/log_to_tsv.cc:238)
    • run migration job
      mg    | matching_engine: [ INFO ] Running cache file migration: /opt/MarketGrid/bin/../migrations/7255cac751ffd2284b382851e5087d40b312271e /tmp/migration-1KfCsO /opt/MarketGrid/bin (src/te/te_cache.cc:216)
  • Start data load from the transformed files
    mg    | matching_engine: [ INFO ] Loading file: /tmp/migration-1KfCsO/Account.tsv (src/te/te_load_textfiles.cc:58)
...
mg    | matching_engine: [ INFO ] Loading file: /tmp/migration-1KfCsO/ZoneRuleBatch.tsv (src/te/te_load_textfiles.cc:58)
mg    | matching_engine: [ INFO ] Loading file: /tmp/migration-1KfCsO/ZoneRuleBatchZone.tsv (src/te/te_load_textfiles.cc:58)
mg    | matching_engine: [ INFO ] Loading file: /tmp/migration-1KfCsO/ZoneRules.tsv (src/te/te_load_textfiles.cc:58)
mg    | matching_engine: [ INFO ] Loaded 10185 records from /tmp/migration-1KfCsO (src/te/te_load_textfiles.cc:174)
  • Complete startup as normal
    mg    | matching_engine: [ INFO ] Set system key for JWT authentication with length 799 (src/te/te_usermanager.cc:221)
mg    | transaction_server: [ INFO ] Connected to upstream at ipc:///opt/MarketGrid/var/sockets/mg_11000_broadcast_queue_12004 (src/te/te_tserver.cc:316)
mg    | transaction_server: [ INFO ] Connected to upstream at ipc:///opt/MarketGrid/var/sockets/mg_11000_transaction_queue_12001 (src/te/te_tserver.cc:317)
mg    | transaction_server: [ INFO ] Connected to upstream at ipc:///opt/MarketGrid/var/sockets/mg_11000_blob_queue_12009 (src/te/te_tserver.cc:318)
mg    | matching_engine: [ INFO ] Checking foreign paths in shared memory (pass 1) (build/generated/te_shm_load.cc:62518)
mg    | matching_engine: [ INFO ] Setting system mode to Initialising (src/te/te_status.cc:380)
mg    | matching_engine: [ INFO ] System time zone is: SYS (src/te/te_statics.cc:779)
mg    | matching_engine: [ INFO ] Setting system mode to Ready (src/te/te_status.cc:380)
mg    | ui_server: [ INFO ] Started Websocket server listening at /opt/MarketGrid/var/sockets/ui_server_websocket

Shutdown Version Next

Shutdowns executed from "Version Next" of MarketGrid will now write their engine cache files in the structure defined in the this new version of the system and they will no longer be able to be used by the previous versions.

Once the first write down is complete, then the migration cycle is complete and the system is in a stable state until the next release is made.

Deployment Specifics

Kubernetes

The shutdown activities for MarketGrid, such as writing out the current cache files, needs to be completed before the engine Pod is terminated by k8s. So the terminationGracePeriodSeconds period should be set or the --grace-period option should be used to give the process sufficient time to complete.

Docker

The shutdown activities for MarketGrid, writing out the current cache files need to be completed before the container is terminated by docker. So it is important to ensure that the shutdown before the software update is graceful.

As noted above this can best be achieved by an ad hoc mg stop rather than relying on the containers graceful shutdown sequence and timing.

MDB Data Migration →

How market database is migrated between different versions of MarketGrid.

Engine Data Migration →

How the engine data is migrated between different versions of MarketGrid.