Misordered done/fill Messages within Block

[chris-rr]

Hi @thaaddeus, me again, starting to work on Mango perp market integrations. Have found an apparent issue with the mango-bowl message generation: basically, there are cases where an order is cancelled on the same block as some fills occur for other orders; the cancellation is published after the fills, but based on the traded prices it is clear that the cancellation had actually occurred before the trades.

An example from 20220201 SOL-PERP data:

1. Sell order 184393653760800704710260 added at 99.96

{
    "type": "open",
    "market": "SOL-PERP",
    "timestamp": "2022-02-01T00:07:44.449Z",
    "slot": 118762042,
    "version": 1,
    "orderId": "184393653760800704710260",
    "clientId": "1643674060717",
    "side": "sell",
    "price": "99.96",
    "size": "867.26",
    "account": "CGcrpkxyx92vjyQApsr1jTN6M5PeERKSEaH1zskzccRG",
    "accountSlot": 21,
    "eventTimestamp": "2022-02-01T00:07:40.000Z"
}

2. Sell order 184430547248948123813505 open at 99.98 (slot 118762045)

{
  "type": "open",
  "market": "SOL-PERP",
  "timestamp": "2022-02-01T00:07:45.859Z",
  "slot": 118762045,
  "version": 1,
  "orderId": "184430547248948123813505",
  "clientId": "1643674061723",
  "side": "sell",
  "price": "99.98",
  "size": "867.03",
  "account": "CGcrpkxyx92vjyQApsr1jTN6M5PeERKSEaH1zskzccRG",
  "accountSlot": 21,
  "eventTimestamp": "2022-02-01T00:07:42.000Z"
}

3. Order 184430547248948123813505 traded at 99.98 (slot 118762045) [At this point the order at 99.96 is still open in the book reconstruction, so that this represents an invalid trade]

{
  "type": "fill",
  "market": "SOL-PERP",
  "timestamp": "2022-02-01T00:07:45.859Z",
  "slot": 118762045,
  "version": 1,
  "orderId": "184430547248948123813505",
  "clientId": "1643674061723",
  "side": "sell",
  "price": "99.98",
  "size": "87.68",
  "maker": true,
  "feeCost": -3.5064985599918286,
  "account": "CGcrpkxyx92vjyQApsr1jTN6M5PeERKSEaH1zskzccRG",
  "accountSlot": 21,
  "eventTimestamp": "2022-02-01T00:07:42.000Z"
}

4. Order 184393653760800704710260 done (cancelled) (slot 118762045) [This should be published before the message of 3.]

{
  "type": "done",
  "market": "SOL-PERP",
  "timestamp": "2022-02-01T00:07:45.859Z",
  "slot": 118762045,
  "version": 1,
  "orderId": "184393653760800704710260",
  "clientId": "1643674060717",
  "side": "sell",
  "reason": "canceled",
  "account": "CGcrpkxyx92vjyQApsr1jTN6M5PeERKSEaH1zskzccRG",
  "accountSlot": 21,
  "sizeRemaining": "867.26"
}

Of course, if order 184393653760800704710260 was actually cancelled after the trade of order 184430547248948123813505, the incoming order would have traded against 184393653760800704710260 at 99.96 instead of against 184430547248948123813505 at 99.98.
It just so happens that in this case order 184430547248948123813505 was opened on the same block as these trades occurred, it has been observed in other cases that all involved orders were open before the block with misordered fill/done messages.

Let me know if requiring any other information.

P.S Also just to enquire if the fixes to resolve issues in serum-vial at tardis-dev/serum-vial#29 tardis-dev/serum-vial#30 tardis-dev/serum-vial#31 were also applied here? Haven't encountered any instances to suggest these issues exist here, but just mentioning in case this hasn't been aligned across vial and bowl.

[thaaddeus]

I will look into that, thanks. Serum-vial logic works bit differently to mango-bowl.

[chris-rr]

Hi @thaaddeus, just following up on this?

[thaaddeus]

sorry, no updates on that yet, did not have time to focus on it.