Design

Rate limiting

A big issue was the rate limits changing for the CryptoCompare API. I was initially running every hour but had to back it off to once per day.

Profiling

The first round of profiling I tried on a decent dataset revealed the VWAP calculation to be dominating. So we could consider microbenchmarking that function; but also we could try and run it less often!

The thing I wish I could do in my day-job is continuous profiling. It just keeps your in touch with what's going on and sometimes minor changes make dramatic differences in the profiling.

File formats

I developed the platform with cryptos and moving onto stocks via Marketstack and you quickly discover the format can vary wildly. The key difference was the JSON presented by Marketstack was in the opposite time order. So my solution is to download these data as-is and then convert everything to a common format before processing by the C++.

C++ ranges and views

This project has been a great opportunity to explore ranges and views, which have been of great interest to me but I have only used for trivial examples.

This is also an exploration of a more functional style of programming, rather than OOP/iterative. The original implementation used a traditional struct for a price point, but now I've intentionally avoided calling any class methods, instead opting for the standalone equivalent: std::size(x) rather than x.size().

Basic range/view operations are pretty straight-forward, but as soon as you start filtering a range it suddenly becomes more complicated and suddenly you can't just calculate the size of a view without evaluating it.

Floating points

I put a bit of effort into being able to switch between different sized floating point numbers, with the expectation that they might perform differently. They did not. Well, I say that... long double is actually twice as slow, but that's not too much of a shock on a 64-bit architecture.

Parallelism

It's interesting to revert to single threaded occasionally to get another insight into your processing (and on the nightly build it's running in a single thread anyway.) At one point I was only doubling the time taken by running on a single core (rather than 8).

AI coding tools

This is also the first project where I've used Copilot which has been rather enlightening: I've been able to write code extremely quickly and -- probably more importantly -- delete it again as I don't feel an obligation to preserve something I haven't invested much time in.

As a prototyping tool it really is quite revolutionary.

Linux tuning

cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_governors 
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor