Back Filling Data

Backfilling is the process of querying API's or other methods to retrieve historical candlesticks. Every liquidity firm has it's own API and they are largely different. Some will send you json messages that are easily readible and others will send you lists of strings and numbers. It's not consistent for sure but can be done.

Throttling Yourself

Now, when you code a backfiller, the process is simple. Have a start date and an end date to query from. Read the documentation and figure out how quickly you can query an endpoint for the candlesticks and how much data you can send back. In my experience, it's usually about 4 hours of 1 minute data and you can query for the candlestick every 100ms. Some will allow you faster queries and others may be even slower, 300-500ms. What you then find is your code must be throttled down to account for api limits. You are essentially trying not to DDOS their servers so you must throttle yourself.

The Approach

Get a list of symbols you want to pull from. Make a timer that runs at the interval at which you can pull data from so for myself, most will be set to 100ms. Pull the symbol off the list and start querying the data range while increasing your range every loop by 4 hours. Turns out 4 hrs is a good limit and returned json candlestick size limits. Loop until you reach the current day and then repeat with a new symbol. This action will take hours for sure if you have a large list of data. Do it once, I have it set on my server to run as a job in Kubernetes. Once it's done, I won't have to requery history again but in the cast that I do, it's setup to do it for me without me really doing much.