Example orderbook subscribe sol. #19

papadpickle · 2022-02-27T00:47:12Z

this example subscribes to SOL perp market on-chain events for bids, asks and fill.
following info is logged at each update of orderbook as well as fill event:

latest trade
bid-ask values
midprice
spread in bps
market depth at +/-2%

note: in case no fill event received before an orderbook update, the latest trade will be empty. maybe there is a way to get the last traded price(mango-bowl recent trades maybe?), in that case we can add that in future.

snippet:

additions:

a subscription::wssSubscriber class, which can be/is reused to subscribe to given uri. callbacks can be registered by its clients to handle the incoming wss messages.
a subscription::bookSide class, which uses wssSubscriber to subscribe to bids and asks and offers its clients interfaces to get best prices and volume. additionaly it can notify the clients on each update via a callback.
an orderbook::book class which contains two bookSide instances representing bids and asks and offers its clients an interface to get the level1 info.
a subscription::trades class, which uses wssSubscriber to subscribe to market event queue and offers its clients interfaces to get last traded price. additionaly it can notify the clients on each update via a callback.
a orderbook::level1 and orderbook::order structs

minor improvements:

const correctness at solana api
added todo to use scoped enums at existing mango code, didnt change it here as this might require changes in other examples as well.

initial example adapted from max getting lowest ask using wss to subscribe to bids and asks moved the subscription code to own class for reuse added orderbook class which takes book sides and logs the info on updates from either of the book side added a reusable class for wss boilerplate and a subscription class for trade event queue refactored the example so the subscription classes can be reused by other clients atomic snapshot for orderbook struct to avoid potential data race using bsp instead of bps. atomic variables for bookside and added empty calls for book depth getting all leaf nodes and sorting them as per prices const correctness at solana api added market depth logic further refactoring midpoint double replaced atomic with mtx due to issues at macos, can be investigated later simplified the fill reception callback

mschneider · 2022-03-01T19:10:10Z

include/subscriptions/wssSubscriber.hpp

+
+      websocketpp::lib::error_code ec;
+      ws_client::connection_ptr con = c.get_connection(
+          "wss://mango.rpcpool.com/946ef7337da3f5b8d3e4a34e7f88", ec);


add a new constant to mango.hpp

yea, MAINNET.endpoint should work.

mschneider · 2022-03-01T19:26:42Z

[2022-03-01 20:24:16.874] [info] ============Update============
[2022-03-01 20:24:16.874] [info] Latest trade: 9880
[2022-03-01 20:24:16.874] [info] Bid-Ask 9879-9880
[2022-03-01 20:24:16.874] [info] MidPrice: 9879.5
[2022-03-01 20:24:16.874] [info] Spread: 1.01 bps
[2022-03-01 20:24:16.874] [info] Market depth -2%: 1458616
[2022-03-01 20:24:16.874] [info] Market depth +2%: 950953

output should probably display ui units instead of native units, i'll open a separate ticket for this

mschneider · 2022-03-01T19:45:37Z

include/subscriptions/bookSide.hpp

+    if (!newOrders.empty()) {
+      {
+        std::scoped_lock lock(ordersMtx);
+        orders = std::move(newOrders);


newOrders should already be sorted by price as the order book tree data structure is sorted by price

ok. will have to make sure though that the bidside and askside sorting is from midpoint to either ends.

mschneider · 2022-03-01T19:53:28Z

include/subscriptions/bookSide.hpp

+            leafNode->timestamp + leafNode->timeInForce < nowUnix;
+        if (isValid) {
+          newOrders.emplace_back((uint64_t)(leafNode->key >> 64),
+                                 leafNode->quantity);


i have a few issues with the way the code is organized:

parsing the order tree should be independent from how the data is transported (rpc / ws), the code should hence not live under subscriptions

same is true for getVolume

each LeafNode actually contains a lot of valuable information that is discarded here in favor of the very incomplete order struct, this makes this code very hard to re-use (we have an order type in most sdks, it usually contains more data then the LeafNode, ui values for instance)

i think there's value in a generic accountSubscriber<typename T> that stores an std::shared_ptr<T> so data can be accessed concurrently without the use of critical sections. just let a public method return new shared_ptr instances

will ping you on discord to catch up here.

std::shared_ptr so data can be accessed concurrently

where only headers of shared_ptr are atomic. stuff it holds isn't.

mschneider · 2022-03-01T20:04:10Z

tried it locally and the code works well for a few minutes but then one by one the different sockets disconnect. once a single socket disconnects, aggregated data can be invalid (e.g. mid price, after one side of the book stopped updating)

mschneider · 2022-03-01T20:10:13Z

include/mango_v3.hpp

@@ -135,8 +138,10 @@ struct EventQueueHeader {
  uint64_t seqNum;
 };

+// todo: change to scoped enum class


sounds good, want to open a new issue?

* moving book parsing code out of subscriber class

…p logging * some refactoring

papadpickle · 2022-03-02T14:26:03Z

added changes to stop logging and exit in case of problem at any of the wss connections.

* removed unnecessary structs

papadpickle · 2022-03-07T09:06:47Z

re your comment #19 (comment):

parsing of the messages is moved to the classes(BookSide and Trade in mango_v3.hpp) out of subscriptions.
handled at point 1
this is first iteration of parsing the incoming messages and extracted only necessary info. We can fill up/edit the parsing and data struct as we go was my idea. You can create an issue here which can be handled later.
I have refactored the code by adding AccountSubscriber class, removed unnecessary subscriber classes and replacing critical sections with shared_ptr.

I have one point about the structure of the code overall:
we can refactor the monolithic mango_v3.hpp into individual hpp and cpp files to speed up dev work. Now a single line change at those interfaces forces the recompilation of everything. Since we are anyway creating a static lib, we may as well go with seperate files instead of having one huge hpp.
Unless of course you decide to make it a header only lib.

mschneider · 2022-04-01T12:45:08Z

include/subscriptions/orderbook.hpp

+  }
+
+ private:
+  std::shared_ptr<L1Orderbook> level1 = std::make_shared<L1Orderbook>();


nullptr makes sense here to differentiate between waiting for first data update vs empty book

makes sense. will change it.

mschneider · 2022-04-01T12:45:24Z

include/mango_v3.hpp

+      if ((*iter).tag == NodeType::LeafNode) {
+        const auto leafNode =
+            reinterpret_cast<const struct LeafNode *>(&(*iter));
+        const auto now = std::chrono::system_clock::now();


you dont want to recalculate the current time inside the loop

solcpp/examples/ticker.cpp

Line 36 in 9deb4c9

const auto now = std::chrono::system_clock::now();

mango gravy from the initial example lol.
pls create a seperate bug/issue for this.

mschneider · 2022-04-01T12:47:03Z

include/mango_v3.hpp

+
+ private:
+  uint64_t lastSeqNum = INT_MAX;
+  std::shared_ptr<uint64_t> latestTrade = std::make_shared<uint64_t>(0);


why not just an uint64_t? trade at price 0 is impossible after all

will need a mtx/atomic otherwise. same reason why we replaced all mtx'es with shared ptrs.
event based synchronization between getLastTrade and handleMsg.
can change to nullptr though so client of getLastTrade could parse the validity of returned value.

mschneider · 2022-04-01T12:55:51Z

include/mango_v3.hpp

+    // all other messages are event queue updates
+    const std::string method = msg["method"];
+    const int subscription = msg["params"]["subscription"];
+    const int slot = msg["params"]["result"]["context"]["slot"];


this is not correct, slots here are related to the chain consensus not the ringbuffer

well the gravy for mango related code was taken from the original examples from the first commit.

solcpp/examples/accountSubscribe.cpp

Line 69 in dd58f26

const int slot = parsedMsg["params"]["result"]["context"]["slot"];

you can create a bug/issue for this. out of scope for this imo.

mschneider · 2022-04-01T12:58:40Z

include/mango_v3.hpp

+ public:
+  BookSide(Side side) : side(side) {}
+
+  bool handleMsg(const nlohmann::json &msg) {


this should really live in the websocket subscription code, rather than the account type. in fact the code should work the same for all types of account subscriptions.

the first four lines to check json "result" or the whole of msg handling?
first four lines could be moved one layer below to wss subscription code in case we know that each msg has the "result" object.
the other parsing code must be handled by the account itself.

# Conflicts: # examples/CMakeLists.txt # include/mango_v3.hpp # include/solana.hpp

… type itself to interpret

added bookdelay mechanism

…use the struct whichever way it wants

…e_sol

papadpickle · 2022-05-22T22:57:13Z

updated the PR with following changes:

Example orderbook subscribe sol. #19 (comment) -> decoding in the account subscriber itself.
handling bookTimeDelay to filter expired orders
using FillEvent struct for trades.
using LeafNode struct as order struct
minor improvements

minor improvements

mschneider · 2022-06-09T18:31:41Z

include/mango_v3.hpp

+                               std::to_string(decoded.size()) + " expected " +
+                               std::to_string(sizeof(BookSideRaw)));
+    }
+    memcpy(&(*raw), decoded.data(), sizeof(BookSideRaw));


needs critical section from before memcpy to end of iterator lifetime.
concurrent update calls else could cause memory corruption.
another option would be to not update the same buffer and to swap references to a new buffer, so that iterator can still iterate over the "old buffer" and de-allocate it on iterator destruction.

mschneider · 2022-06-09T18:32:22Z

include/mango_v3.hpp

+  uint64_t getVolume(uint64_t price) const {
+    Op operation;
+    uint64_t volume = 0;
+    for (auto&& order : *orders) {


orders needs to be dereferenced once before entering loop

godmodegalactus · 2022-10-10T06:40:39Z

include/subscriptions/wssSubscriber.hpp

+
+    try {
+      websocketpp::lib::error_code ec;
+      ws_client::connection_ptr con = c.get_connection(MAINNET.endpoint, ec);


MAINNET.endpoint should not be statically passed but we should create an endpoint argument in the constructor.

again:

std::shared_ptr so data can be accessed concurrently

if it's so it's a bad thing. Is it?

mschneider reviewed Mar 1, 2022

View reviewed changes

mschneider linked an issue Mar 1, 2022 that may be closed by this pull request

Create advanced example to fetch orderbook via websocket #9

Open

papadpickle mentioned this pull request Mar 1, 2022

mango_v3.hpp, todos re using scoped enum class #22

Open

papadpickle and others added 3 commits March 2, 2022 00:42

* using MAINNET.endpoint instead of magic numbers

8d04f8a

* moving book parsing code out of subscriber class

* added on_close callbacks to handle error in wss connections and sto…

587e1aa

…p logging * some refactoring

style fixup

46f206a

papadpickle and others added 3 commits March 4, 2022 13:30

replaced mtx with shared_ptr trick

2ade336

style fixup

3671454

* added accountSubscriber template class

06c7e5d

* removed unnecessary structs

papadpickle force-pushed the example_orderbook_subscribe_sol branch from c4bca17 to 06c7e5d Compare March 5, 2022 17:41

mschneider reviewed Apr 1, 2022

View reviewed changes

papadpickle and others added 7 commits April 1, 2022 17:50

minor interface change

400efea

Merge branch 'main' into example_orderbook_subscribe_sol

2efe235

# Conflicts: # examples/CMakeLists.txt # include/mango_v3.hpp # include/solana.hpp

style fixup

3a514b0

Merge branch 'main' into example_orderbook_subscribe_sol

c644ad2

merged upstream changes and minor improvement

d014f4c

style fixup

17946c9

merge upstream

701d150

papadpickle and others added 6 commits May 22, 2022 00:19

decoding the message at account subscriber before passing the data to…

a8b9e1a

… type itself to interpret

style fixup

322b78c

corrected expired order culling

f8a589f

added bookdelay mechanism

storing whole fill event in trades instead of only price. client can …

b84d2c4

…use the struct whichever way it wants

using leafNode as order struct itself

155320f

Merge branch 'mschneider:main' into origin/example_orderbook_subscrib…

e625ded

…e_sol

papadpickle added 3 commits May 25, 2022 00:36

added NativeToUi conversion functionality

b77315d

minor improvements

minor log adjustment

d886062

added utest for NativeToUi conversions

7ac7765

mschneider reviewed Jun 9, 2022

View reviewed changes

strict concurrency mtxing

1ade8c2

godmodegalactus reviewed Oct 10, 2022

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Example orderbook subscribe sol. #19

Example orderbook subscribe sol. #19

papadpickle commented Feb 27, 2022 •

edited

Loading

mschneider Mar 1, 2022

papadpickle Mar 1, 2022

mschneider commented Mar 1, 2022

mschneider Mar 1, 2022 •

edited

Loading

papadpickle Mar 1, 2022

mschneider Mar 1, 2022 •

edited

Loading

papadpickle Mar 1, 2022

alexeyneu Sep 29, 2022 •

edited

Loading

mschneider commented Mar 1, 2022

mschneider Mar 1, 2022

papadpickle Mar 1, 2022 •

edited

Loading

papadpickle commented Mar 2, 2022

papadpickle commented Mar 7, 2022

mschneider Apr 1, 2022

papadpickle Apr 1, 2022

mschneider Apr 1, 2022

papadpickle Apr 1, 2022

mschneider Apr 1, 2022 •

edited

Loading

papadpickle Apr 1, 2022

mschneider Apr 1, 2022 •

edited

Loading

papadpickle Apr 1, 2022

mschneider Apr 1, 2022

papadpickle Apr 1, 2022

papadpickle commented May 22, 2022

mschneider Jun 9, 2022

mschneider Jun 9, 2022

godmodegalactus Oct 10, 2022

alexeyneu Oct 10, 2022 •

edited

Loading

Example orderbook subscribe sol. #19

Are you sure you want to change the base?

Example orderbook subscribe sol. #19

Conversation

papadpickle commented Feb 27, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mschneider commented Mar 1, 2022

mschneider Mar 1, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mschneider Mar 1, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alexeyneu Sep 29, 2022 • edited Loading

Choose a reason for hiding this comment

mschneider commented Mar 1, 2022

Choose a reason for hiding this comment

papadpickle Mar 1, 2022 • edited Loading

Choose a reason for hiding this comment

papadpickle commented Mar 2, 2022

papadpickle commented Mar 7, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mschneider Apr 1, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mschneider Apr 1, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

papadpickle commented May 22, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alexeyneu Oct 10, 2022 • edited Loading

Choose a reason for hiding this comment

papadpickle commented Feb 27, 2022 •

edited

Loading

mschneider Mar 1, 2022 •

edited

Loading

mschneider Mar 1, 2022 •

edited

Loading

alexeyneu Sep 29, 2022 •

edited

Loading

papadpickle Mar 1, 2022 •

edited

Loading

mschneider Apr 1, 2022 •

edited

Loading

mschneider Apr 1, 2022 •

edited

Loading

alexeyneu Oct 10, 2022 •

edited

Loading