MDB Data Migration

How market database is migrated between different versions of MarketGrid.

The mdb migration is composed of 2 components:

  1. Setting up the migration script
  2. MDB Migration workflow

Setting up the migration script

Run the comparesyms.q script (located in src/mdb/torq/migration) to generate a skeleton migration script with the following parameters:

  • old - the current sym file to migrate from
  • new - the new sym file to migrate to
  • [createscript] - if specified, creates the migration script, else it will print the differences on the foreground.

q comparesyms.q -old syms/symold.q -new syms/symnew.q -createscript test.q

This will create a checksum folder using the sha1sum algorithm e.g. 3b3253a0dcae5fc976f4b6f7668d1e7ab2b1fe71 and place the migration script there.

The skeleton migration script can then be augmented to define the actual migration logic that is required for this data.

If there are any custom functions used for the migration that could be reused, these should be saved inside the dbmaint folder located at src/mdb/torq/migration/dbmaint. All q scripts inside this folder are loaded in when we run the migration script.

One can run or test the migration script with the following parameters:

  • hdb - the hdb directory to conduct the migration
  • dbmaint - the dbmaint directory to load generic/custom functions
>> q test.q -hdb /data/hdb -dbmaint /opt/MarketGrid/src/mdb/torq/migration/dbmaint

q) main[]

Mdb Migration Workflow

Real time database

Running the mdb migration is automatically detected by the real time database mdb_rdb process.

The real time database script in src/mdb/torq/code/processes/rdb.q executes a function called initchecksum which does the following:

  1. Check if checksum exist in the hdb directory and whether hdbmig folder exist. If checksum does not exist, exit. If the hdbmig directory exist, exit.
  2. Check if the checksum of the current lib/TorQ/sym.q file vs the checksum in the hdb directory /data/hdb/checksum is the same or not, and if its not the same;
  3. Rsync the existing hdb directory /data/hdb to a hdb copy directory /data/hdbcopy.
  4. When the rdb attempts to write down data during migration, it will write the data to /data/rdblog for the migration process to merge the data back into the migrated hdb. The files created will be in the following convention : rdblog_YYYYMMDDDHHMMSS and rdblog_YYYYMMDDDHHMMSS.done.
  5. Start the hdb_migration process.

HDB migration process

The migration process operates on the hdb migration directory : /data/hdbmig and does the following when it starts up:

  1. Check if the hdb migration directory exists or not. If it exists, the process will exit because there could be a migration process already running or a failed migration has occurred which requires the operator/administrator to check and clear.
  2. Move hdbcopy to hdbmig
  3. Load the checksum migration folder based on the checksum file in the hdb migration directory (which is the old checksum value) and run the migration.
  4. Check the rdblog directory to see if there are any files required to be replayed and merged back into the migrated hdb.
  5. Write down the new checksum file
  6. Move hdb to hdbcopy && move hdbmig to hdb
  7. Restart the hdb and gateway processes

Note: if any errors are detected during steps 3, 4 or 5, the hdb migration folder is not removed. A manual intervention is required to check the state of hdbmig.

← stop
File reference →