Building quantitative system: how to deal with system failures
When you are an algorithmic trader, people think you spent your whole day building new fancy strategies that use complex mathematical functions, which is not the case. Instead, quants spent most of their time improving their current infrastructure and scaling their operations.
This is because the greatest danger for a quant shop is the failure of its systems. Miscalculations in the risk engine, systems going down and bugs hiding in hard-to-read code. Quant shops are just like Rome. They aren't built in a day but can be burned in one. So let's talk about a few simple ways to make your systems robust.
Streamline everything
You communicate with exchanges via their API however, every exchange has its own API that requires certain information to be provided with every call you do that another might not have. Let me give you an example by looking at how Binance & FTX expect you to place an order.
| Exchange | Api Call | Number of args |
|------------------------|-----------------|-----------------|
| Binance Spot | /api/v3/order | 14 |
| Binance USD-M futures | /fapi/v1/order | 18 |
| Binance COIN-M futures | /dapi/v1/order | 18 |
| FTX (spot& futs) | /orders | 9 |So for two exchanges, you have four different endpoints that require between 9 to 18 pieces of information to place an order. This is hard to work with, how we at Musca solved this issue is to create our own API wrapper which expects the same arguments for every API call for every exchange and returns the same information in the same format.
There are already some solutions out there like CCXT but their code is messy, does not unify everything and does not support asynchronous requests. Besides that, if you create your own solution it will be easier to add additional exchanges since your unified wrapper/API is tailored to your needs it will be easier to add additional features and debug.
Data isn’t always your friend
Your systems help you make decisions or make them fully on their own based on the data they are receiving. But what if that data is wrong, delayed or interpreted in the wrong way? Let's look at an example.
| Exchange | Market | Price |
|-----------|---------------|----------|
| Binance-f | BTCUSDT-perp | 20520 |
| FTX | BTC-PERP | 20515 |
| Coinbase | BTCUSD | 20514 |
| Coinbase | USDTUSD | 0.99015 |Let’s play a game, what is the USD price of every market you see above? One of them is not like the others, most people would say the Price is the USD price but that is not the case. Since the contract is nominated in USDT the USD value would be 20317.87$ not 20520. Many will not bat an eye at this inefficiency but it's incredibly important that you take it into account.
Let’s go back to the 13th of May 2022, the UST de-pegging from Luna makes people uncomfortable holding stablecoins resulting in a market sell-off for tether (USDT). What would this mean for someone who does not take into account the live prices of stablecoins?
- Live market prices would be incorrect for USDT pairs
- Your internal balance sheet will be incorrect since you still think tether is 1$.
- Every datapoint that relies on USDT market pricing will be incorrect
Monolithic architecture might really send you back to the stone age.
A Monolithic is the most common way new quants start building their systems, the software is developed as a single unit. All functionality, features and modules are developed as a single entity. It's a simple way to create and deploy systems but is horrible in the long run.
Microservice architecture is in my opinion the superior solution for quants. You can scale out each element independently without downtime, integrate new tech easily and components can be individually tested but most importantly the failure of one microservice does not affect other services.
To explain it in a non-technical way think of a monolithic system as a single trader on the floor, he manages your risk, executes the trades and does your analysis. However, when/if this person fails, everything fails. Microservices are a team of people. One person does risk management, the other executes the trades and another does analysis. If one fails that doesn't mean the whole system fails and he can be replaced. Adding new people with a different skill set is easy they only need to learn to communicate with others on the team.
What if the 3rd parties fail?
But would it survive march 2020? This is one of my favourite questions to ask, would my system be able to survive when exchanges fail?
How would you deal with:
- An exchange disabling trading on a specific market you have a position on
- Deposits/Withdrawals disabled
- Frozen blockchains
- API issues
- Exchange UI issues
Some of these are harder than others, do you have an open position but the exchange disabled trading? Hedge it on another exchange.
But what if the API of an exchange fails? Would your systems be able to detect it and take the necessary action? What about disabled deposits? How will you hedge your bets or save yourself from liquidation when you cannot deposit collateral?
They are all crises you have to take into account and require a lot of creative thinking to find an efficient fix.
To conclude:
There are more than enough opportunities in the crypto markets for those who are patient and manage their risk. But you must be able to be there with cash in hand to take advantage of those opportunities. When others are forced to exit positions the best trading opportunities arise, it's up to you to detect and take advantage of them.
Our website: https://musca.capital/
Twitter : https://twitter.com/MuscaCapital
Contact us : https://musca.capital/contact
