Test Framework
A guide to using the test framework, with best practice examples.
Introduction
The test framework is a toolset designed to help with functional testing of MarketGrid. It currently supports submitting transactions into the matching engine and confirming the output in shared memory and/or API. In the future it may encompass other types of API testing, performance testing and UI testing.
Testing Overview
The examples in this guide are in the following archive: test-example.tgz Running of these tests in a docker container are explained in Running the tests
Data set
In order to test functionality such as submitting an order, there is a minimal set of data which needs to be present in order for the system to accept and process an order without rejection. The system can load .tsv files which are tab-delimited text files. There are examples of these files in the tests/datasets/core/ directory (of the test-example.tgz archive) which can be used for many simple tests or copied and modified in order to create a new custom dataset for more complex testing. The text files can be edited as plaintext or in spreadsheet applications such as Microsoft Excel. The example dataset used in this guide is the basic.tsv dataset combined with the full-permissions.tsv dataset. basic.tsv contains data to populate a minimum number of records in various tables and prevent missing foreign-key relationships:
# This file contains a basic actor and product hierarchy that can be used within
# test scripts.
# table = Venue
Id ShortName Name Status
1 test-Venue test-venue Active Normal BuyOrdersAccepted SellOrdersAccepted AmendAccepted CancelAccepted
# table = ExternalAccountProvider
Id ShortName Name
1 test-eap test-eap
# table = Enterprise
Id ShortName Name Venue EnterpriseType Status
1 test-enterprise test-enterprise 1 Regular Active Normal BuyOrdersAccepted SellOrdersAccepted AmendAccepted CancelAccepted
# table = Firm
Id ShortName Name Enterprise FirmType Status
1 test-firm test-firm 1 Regular Active Normal BuyOrdersAccepted SellOrdersAccepted AmendAccepted CancelAccepted
# table = Group
Id ShortName Name Firm Status
1 test-group test-group 1 Active Normal
# table = User
Id Firm UserId ShortName Name UserType Status Password
# passwords are 'password' encrypted
1 1 admin admin admin Exchange Active Normal $6$rounds=50000$XUdjniwL6X113buV$31GfdI2VT8UfhSgjEXyhSxIAGU/JLviWtC0DPOZ1tUknF8tSt5juB.LkbE5AYOu8Dex6ax4ovmkVpbaKboFqP1
2 1 test-user test-user test-user Regular Active Normal BuyOrdersAccepted SellOrdersAccepted AmendAccepted CancelAccepted $6$rounds=50000$XUdjniwL6X113buV$31GfdI2VT8UfhSgjEXyhSxIAGU/JLviWtC0DPOZ1tUknF8tSt5juB.LkbE5AYOu8Dex6ax4ovmkVpbaKboFqP1
3 1 test-user-2 test-user-2 test-user-2 Regular Active Normal BuyOrdersAccepted SellOrdersAccepted AmendAccepted CancelAccepted $6$rounds=50000$XUdjniwL6X113buV$31GfdI2VT8UfhSgjEXyhSxIAGU/JLviWtC0DPOZ1tUknF8tSt5juB.LkbE5AYOu8Dex6ax4ovmkVpbaKboFqP1
# table = GroupUser
Id Group User
2 1 2
3 1 3
# table = AccountType
Id ShortName Name
1 test-account-type test-account-type
# table = Account
Id Firm AccountCode Status AccountType
1 1 test-account Active Normal 1
# table = ExternalAccount
Id Firm Account ExternalAccountProvider ExternalAccountCode
1 1 1 1 test-externalaccount
# table = InstrumentGroup
Id ShortName Name Status
1 test-group test-group Active Normal BuyOrdersAccepted SellOrdersAccepted AmendAccepted CancelAccepted
# table = Instrument
Id ShortName Name InstrumentGroup ExternalAccountProvider InstrumentType Status
1 test-instrument test-instrument 1 1 Standard Active Normal BuyOrdersAccepted SellOrdersAccepted AmendAccepted CancelAccepted
2 test-currency test-currency 1 1 Currency Active Normal
# table = TimeZone
Id ShortName Name UTCOffset
1 UTC UTC UTC
# table = Market
Id Venue ShortName Name TimeZone DefaultCurrency Status
1 1 test-market test-market 1 2 Active Normal BuyOrdersAccepted SellOrdersAccepted AmendAccepted CancelAccepted
# table = Session
Id ShortName Name Code Type
1 Continuous Continuous Continuous Continuous
# table = InstrumentMarket
Id ShortName Name Instrument Market Currency SettlementCurrency Session Status OrderType
1 test-im test-im 1 1 2 2 1 Active Normal BuyOrdersAccepted SellOrdersAccepted AmendAccepted CancelAccepted Limit Market Buy Sell GoodtillDate GoodtillCancelled
full-permissions.tsv contains data to allow all permissions by any user for any table and scope:
# table = Permission
Id PermissionAction TableNumber Scope Status
0 All All All Active Normal
More information about the permissions model (TBC)
Writing a test script
The general functionality of a test script is the following:
- Load test data
- Submit transactions to the engine
- Assert replies or data in the engine is as expected
- Print replies or data which can be compared to a baseline.
load < tests/datasets/core/basic.tsv < tests/datasets/core/full-permissions.tsv
echo "Test case for simple order entry. This will be printed to the baseline file."
# Test case for submitting an order and confirming the result
nq <<EOF
# define the OrderNew transaction template
.def OrderNew @userid instrumentmarket side type price totalquantity
# submit OrderNew message using the template
.tx OrderNew test-user test-im Buy Limit+GoodtillCancelled 100 1
# Assert that the OrderNew transaction was successful
.assert _reply.header.result === 0
# print OrderNew transaction reply
.print _reply.body
# Assert that the length of the Order table is 1.
.assert Order.length === 1
# Print the order table with only these specific columns.
.print Order only Id Reference InstrumentMarket Price Currency TotalQuantity TotalBalance
EOF
load < tests/datasets/basic.tsv < tests/datasets/full-permissions.tsv
In the above example script, line 1 loads the two dataset files (in order - basic.tsv first). You can also load a full directory of tsvs by using load /tests/datasets/fullset/
echo "Test case for simple order entry. This will be printed to the baseline file."
Line 3 demonstrates running the bash command echo and printing a string which ends up in the baseline.
nq <<EOF
EOF
Line 5 and Line 18 define the start and end of the nq commands. Anything between these are the same commands as entering in the nq console. Lines starting with # are ignored and can be used as comments.
More help on nq is available at : https://gitlab.com/marketgrid/marketgrid/-/wikis/nq
.def OrderNew @userid instrumentmarket side type price totalquantity
In line 7, .def defines a transaction template for the OrderNew message.
.tx OrderNew test-user test-im Buy Limit+GoodtillCancelled 100 1
Line 9 uses the previous OrderNew template definition to create and send the transaction using the test-user
.assert _reply.header.result === 0
Line 11 asserts that the last transaction reply was successful (i.e. ResultCode of 0)
.print _reply.body
Line 13 prints the body of the last transaction reply - this is the reply of the successful OrderNew message, which includes details such as Order Id, Status, etc.
.assert Order.length === 1
Line 15 asserts that the length of the order table is equal to 1 after the order submission.
.print Order only Id Reference InstrumentMarket Price Currency TotalQuantity TotalBalance
Line 17 prints the order table with a specific set of fields. This is helpful to specify specific fields that are to be tested.
Replacing only with except allows all columns to be printed except those specified. It also can be used with basic regex, such as .print Order except Time* to exclude all columns that start with Time.
Test validation
There are two methods to validate whether a test was successful. These are assertions and baseline comparisons.
Assertions
Assertions can be used in-line during the live test to confirm that a specific value is as expected. This can be useful to capture and confirm the state of the data such as a transaction reply or the length of a table. Assertions can also be used to validate an expected error or rejected transaction. If an assertion fails, the test terminates immediately with an error and no further transactions are submitted.
A few assert examples:
.assert _reply.header.result === EngineError.OrderHitLiftPrice
.assert InstrumentMarket.get(1).ShortName === 'new-shortname'
.assert InstrumentMarket.get(1).Instrument === 2
.assert Order.get(1).TotalBalance === Order.get(1).TotalQuantity - Order.get(1).Matched
l3 = Level3.find(eq('InstrumentMarket', InstrumentMarket.find(eq('ShortName', 'test-im2')).Id));undefined;
.assert l3.ZoneFee === 13
Baseline Comparison
The baseline comparison allows comparing larger blocks of data with an expected result which was committed by the tester previously.
This example of the simple-order baseline shows the output which is compared when re-running the test:
Test case for simple order entry. This will be printed to the baseline file.
Print: _reply.body
ts_outputorder_t {
id: '1',
clordid: '',
status: 'Active Amendable',
totalbalance: 1,
visiblebalance: 1,
hiddenbalance: 0,
type: '',
instrumentmarket: 'test-market:test-im',
side: 'Buy',
origclordid: '',
limit: 0,
matchedquantity: 0,
order_index: 0,
price: 100,
livefor: 0,
source: 'OpsTerminal',
quotereqid: ''
}
Print: Order only Id Reference InstrumentMarket Price Currency TotalQuantity TotalBalance
Id Reference InstrumentMarket Price Currency TotalQuantity TotalBalance
────────────────────────────────────────────────────────────────────────────────
1 1 test-market:test-im 100 test-currency 1 1
A tester should look through this and validate that the data which is being tested is correct before approving the baseline. Once approved, it can be committed to a version control system such as Git, and can become part of a regression test suite.
Running the tests
The test-example.tgz should be extracted and mounted to the /opt/MarketGrid/bin/tests directory inside the docker container - for example:
tar -zxf test-example.tgz
docker run -it --shm-size=8gb -v "$(pwd)"/tests:/opt/MarketGrid/bin/tests registry.cartax.io/platform/meta/marketgrid:latest
# Then to run all the tests (of which there is only one):
mg test --baseline-dir tests/baselines tests/cases
Replace registry.cartax.io/platform/meta/marketgrid:latest with the MarketGrid version which is to be tested.
Analysing test results
The single simple-order test should pass:
[mg@f97632aa0f99 bin]$ mg test --baseline-dir tests/baselines tests/cases
[1/1] [ PASS ] core/engine/order-entry/simple-order (1.428)
Executed 1 test in 1.460s.
[mg@f97632aa0f99 bin]$
Debugging a test
Modify the test (tests/cases/core/engine/order-entry/simple-order, line 9) to place an order of quantity of 2, then re-run the test:
[mg@f97632aa0f99 bin]$ mg test --baseline-dir tests/baselines tests/cases
Not overwriting existing baseline at /opt/MarketGrid/bin/tests/baselines/core/engine/order-entry/simple-order
[1/1] [ FAIL ] core/engine/order-entry/simple-order (1.438)
Test case for simple order entry. This will be printed to the baseline file.
Print: _reply.body
ts_outputorder_t {
id: '1',
clordid: '',
status: 'Active Amendable',
- totalbalance: 2,
+ totalbalance: 1,
- visiblebalance: 2,
+ visiblebalance: 1,
hiddenbalance: 0,
type: '',
instrumentmarket: 'test-market:test-im',
side: 'Buy',
origclordid: '',
limit: 0,
matchedquantity: 0,
order_index: 0,
price: 100,
livefor: 0,
source: 'OpsTerminal',
quotereqid: ''
}
Print: Order only Id Reference InstrumentMarket Price Currency TotalQuantity TotalBalance
Id Reference InstrumentMarket Price Currency TotalQuantity TotalBalance
────────────────────────────────────────────────────────────────────────────────
1 1 test-market:test-im 100 test-currency 1 1
Executed 1 test in 1.461s.
[mg@f97632aa0f99 bin]$
We can now see the test fail because it does not match the baseline.
A single test can be debugged:
mg test tests/cases simple-order --debug
This will run the test and drop the user into a sub-shell (test-run-xxxx) at the point just before the test is terminated. This allows interrogation of the test instance to help debugging or creation of a test. Inside the sub-shell, running nq will open the interactive nq console for interrogation:
test-run-4161 % nq
test-run-4161> Order
Id Status Timestamp Reference InstrumentMarket InstrumentType Market OriginalOrder AmendedOrder PairedOrder RoutingState OwnerFirm OwnerGroup OwnerUser OriginatingUser CounterParty Side Type Currency PriceDecimals QuantityDecimals ValueDecimals SettlementCurrency SettlementVal..
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
1 0x3 2021-02-25 06:07:01.639425200Z 1 test-market:test-im Standard test-market 1 test-firm test-user test-user Buy 0x8000001 test-currency test-currency ..
test-run-4161>
Ctrl+D to leave the test sub-shell. The test instance will be automatically cleaned up & no longer accessible.
Debugging with UI
Run the test with UI service
mg test tests/cases simple-order --debug --ui
Append --branding brand.name to start the UI with desired branding
Browse to UI on address https://localhost:10443
Enable port forwarding for the UI if using a docker container
docker run -it -p 10443:10443 --shm-size=8gb -v "$(pwd)"/tests:/opt/MarketGrid/bin/tests registry.cartax.io/platform/meta/marketgrid:latest
Creating a new test
Copy the simple-order file into a new file called simple-order2 Running the tests again, we can see it is a NEW test.
This will automatically create a new baseline which can be checked in to a Git repository alongside the new test file. If there is no Git repository linked, the test will automatically pass on the 2nd run.
[mg@f97632aa0f99 bin]$ cp tests/cases/core/engine/order-entry/simple-order tests/cases/core/engine/order-entry/simple-order2
[mg@f97632aa0f99 bin]$ mg test --baseline-dir tests/baselines tests/cases
[1/2] [ PASS ] core/engine/order-entry/simple-order (1.588)
[2/2] [ NEW ] core/engine/order-entry/simple-order2 (1.618)
Wrote new baseline for core/engine/order-entry/simple-order2
Executed 2 tests in 1.642s.
[mg@f97632aa0f99 bin]$ mg test --baseline-dir tests/baselines tests/cases
[1/2] [ PASS ] core/engine/order-entry/simple-order2 (1.656)
[2/2] [ PASS ] core/engine/order-entry/simple-order (1.683)
Executed 2 tests in 1.700s.
Creating a new baseline
Remove the directory of existing of baseline tests and run the tests again to create a new baseline
[mg@ca4da7c28661 bin]$ mg test --baseline-dir tests/baselines tests/cases
[1/1] [ NEW ] core/engine/order-entry/simple-order (1.335)
Wrote new baseline for core/engine/order-entry/simple-order
Executed 1 test in 1.365s.
(Optional) Pattern matching test run
Running just mg test will show the optional pattern argument. This allows specifying a pattern such as order which will run all tests with *order* in the filename.
mg test --baseline-dir tests/baselines tests/cases order
Viewing output of a single test
[mg@f97632aa0f99 bin]$ mg test tests/cases order --output
Glossary (WIP):
- nq - the nq server is a nodejs component on which the UI and various other components such as the REST server are built. It supports short and long running scripts, as well as a REPL interface for querying shared memory.
Test fw reference guide (WIP)
.def commands .tx commands Custom test functions (recon_lev_balance) Baseline comparisons vs assertions?
Test fw file layout (WIP)
Directory overview (test folder recursion) Baselines Unit test Group test
Environments & version control (WIP)
Docker vs vm etc Repository