Migration Activity

The kinds of actions required during a migration can be categorised into two groups;

1. Automatic Migrations
2. Bespoke Actions

Automatic Migration

The migration of Matching Engine cache files is built upon creating a set of tsv files from which the new version of the Matching engine can start.

The Matching Engine is quite flexible in handling new fields and there is no requirement for a new table to have a text file by default. Likewise fields that are removed will be ignored on reload and subsequently not written down for future cache file downloads. As a result when the migration only contains simple changes such as;

• adding new fields
• removing existing fields
• adding tables
• modifying the type fields in such a way that the persisted data is still valid input

then there is no need for the migration script to actually do any work. The Matching engine will know how to handle these changes automatically.

Bespoke Actions

Whilst it's possible to perform any kind of manipulation there are a few kinds of transformation that are more common than others, such as;

• new tables initial data
• transformation of existing fields
• ad hoc values for new fields
• inferred or derived values for new fields
• new business data (a new market or product) that is to be loaded from static data rather than managed via manual entry or other onboarding interfaces

Engine Migration Script

An engine data migration script is just a Linux shell script. As such it has the power to do whatever needs to be done in order to complete the migration.

Script Identification

The name of the migration file that is executed is the same as the source ABI checksum. That is, the name of the shell script that is run to convert the data is the same as the source data ABI checksum.

Version skipping

When a release is created by Carta, a migration script is included to migrate from the immediately previous version of the system.

There is no mechanism to chain together migrations automatically in the event that a given version is skipped. However it is feasible to craft a single migration script that does "accumulate" all the intermediate migrations and allow the migration to effectively jump from a previous version to the current version.

Given that so many migrations can be handled automatically by simply loading the new version from text files derived from the previous version's persisted cache files an empty file with the name of the source version's ABI checksum will be sufficient in most cases to jump from a preceding version.

Execution and Environment

The running script receives two parameters;

1. The location of the source tsv files, say /tmp/migration-1KfCsO
2. The MarketGrid bin directory, normally /opt/MarketGrid/bin

The tsv files are the source data before any conversion is done. They are extracted from the static data messages in the cache files created by the previous shutdown.

The MarketGrid instance that will be started after the migration is complete expects the migrated tsv files to be found in the same location as the source files. Or rather the system will start from the files found in that directory regardless of any transformation they have undergone (or not).

Manual Processing

The principles of the migration process can be used to persist a running engine to disk, convert the cache files to tsv and then perform manual transformation of the resulting dataset to produce a new set of tsv from which the system can be started again.

Such actvities include;

• recovery from software or operational errors that have left the system in a state that cannot be continued
• Fixing the consequences of software defects
• Peforming actions that the existing Business logic in MarketGrid cannot support
• Sanitising production data for deployment in lower environments

These operations can be done within a deployed environment or by taking the files written on a mounted data store and manipulating them in a richer environment. This is particularly important given that the MarketGrid container images generally don't contain a lot in the way of general ETL/programming software.