Add rate limiter to ensure 120 API calls per minute limit is not exceeded.

[hmanfarmer]

Is your feature request related to a problem? Please describe.
Per the TDAmeritrade documentation: All private, non-commercial use apps are currently limited to 120 requests per minute on all APIs except for Accounts & Trading
https://developer.tdameritrade.com/content/getting-started

Due to the non-deterministic and asynchronous nature of the TDAmeritrade https API, hitting this limit can happen at unexpected times and I believe that many tda-api users will occasionally run into this rate limit. If those users haven't coded their python script properly for this scenario, bad things might happen.

Describe the solution you'd like
Implement a rate limiter. Something that is fast and precise. Make sure api users are able to turn off the rate limiter if they need too. One discord user recommended implementing the solution using a token bucket rate limiter, as I am not a computer scientist, I am not 100% sure that would be the right solution.
https://en.wikipedia.org/wiki/Token_bucket

To be clear, below are the features that would ideally be in the solutions:

• A way to enable/disable the rate limiter
• Only rate limit API endpoints where a rate limit is enforced by TDAmeritrade. Meaning... don't rate limit 'Accounts & Trading' API endpoints. If this cannot be safely implemented, then rate limit everything.

Describe alternatives you've considered
Users can implement their own rate limiter. The easiest but terribly inefficient way to do this is with time.sleep(0.5).

[nicmcd]

"asynchronous nature"? Is that supported?

[hmanfarmer]

@nicmcd

Maybe asynchronous was not the right word to use, I should of just wrote 'non-deterministic'. I was trying to state that due to the nature of all https APIs, the time it takes to get responses from the API calls can vary quite a bit. A user might think they don't have to worry about hitting the rate limit, then BAM, one day your internet is running fast and the TDA servers are having a good day, and you end up hitting the rate limit even though things were working great for 5 weeks.

[nicmcd]

I agree with that logic. Have you been able to trigger a problem with TDA yet? I've been issuing requests as fast as possible and never got any errors.

[hmanfarmer]

Have you been able to trigger a problem with TDA yet?
Yes. TDA was rate limiting me every once in a while when I was first working on my app. But, now that I am using my APP to screen a couple hundred stocks, I am running into the rate limit all of the time now.

[nicmcd]

Did it come in the form of delayed responses or some type of rejection status?

[hmanfarmer]

@nicmcd a rejection status.

[hmanfarmer]

While implementing my own rate limiting code, I ran into an interesting negative side effect of efficiently implementing rate limiting. I am not sure the solution should try to address this, but maybe the documentation should mention something about it.

If you limit yourself to no more than 120 API calls in 1 minute. But, you send the first 120 calls within a very short period of time, the user will experience a long unnatural pause, waiting for the rest of the minute to elapse.

For example:
Send 120 API calls in 5 seconds.
Rate limiter waits 55 seconds. (This is what will be perceived as unnatural for users)
Send 121st API call.
Rate limiter waits <1 second.
Send 122nd API call.
Rate limiter waits <1 second.
Send 123rd API call.
(Repeat to infinity)

[alexgolec]

"asynchronous nature"? Is that supported?

Currently, no. Unfortunately implementing it at this point would be a rather complex undertaking. You can track this feature here:

#31

[nicmcd]

@hmanfarmer why not simply convert the 120/min to an interval (e.g., 0.5 seconds) then rate limit to that?

[hmanfarmer]

@nicmcd
Simply put, because that would not be the most efficient way to do things.

If you have 120 API calls to make. TDAmeritrade allows you to make 120 API calls as fast as you want. If you rate limited every call to 0.5 seconds, those 120 API calls would take 60 seconds to execute, when they could have executed in maybe 5 seconds.

And, if you have more than 120 API calls... let's say 150. The most efficient way is to let the first 120 calls execute as fast as possible and only wait on the 121st call. But, like I said in my previous comment, for some users, this will produce confounding behavior.

[nicmcd]

@hmanfarmer I understand the concern, but at the same time, I'd be concerned about the state needed to properly rate limit given your scheme across tools. For example, let's say I make two different tools, one that issues N buys, one that issues M sells. If N and M are less than 120 but N+M is greater than 120, this will require state to be saved to a file for rate limiting purposes. Using a fixed interval doesn't require this.

[hmanfarmer]

@nicmcd
Yes, that is true.

I think the best solution to the hypothetical situation you describe is, as you alluded to, for both of those tools to share the same tda-api client object. You are right though, depending on your situation, there might be hoops you need to jump through to share that object.

If maximizing performance isn't a priority, then by all means, use a flat 500ms delay.

I do think the scenario you describe would be less common and I think that finding a way to share the tda-api client object across apps would be easy to solve. There seems to be lots of ways to do that... pickle being one of them.

[hmanfarmer]

For reference, here is the response from TDAmeritrade when you exceed the rate limit. I attempted to call 'get_price_history' on ticker 'KO' and below is what I recieved.

Response status_code: '429'
Response text: "error":"Individual App's transactions per seconds restriction reached. Please contact us with further questions"

[hmanfarmer]

Polarizing1 on discord discovered that the rate limiting is reset every minute versus a rolling rate limit. Meaning you get a whole new set of 120 requests every minute, based on whatever clock they are using (either their system clock or a clock that is started when you begin your session with them). I wanted to make sure Polarizing1 was correct, so I wrote the following test code...

def get_stock_info(ticker):
    instrument = client.search_instruments(ticker,client.Instrument.Projection.SYMBOL_SEARCH)
    return instrument

reqCount = 0
now = datetime.utcnow()

while get_stock_info('NFLX').status_code != 429:
    reqCount = reqCount + 1

afterRateLimited = datetime.utcnow()

print ('Requests Accepted: {}'.format(reqCount))
print ('Elapsed Time: {}ms'.format(((now - afterRateLimited).microseconds)/1000))

#setting to 1 because last call of previous while loop was 1 returned 429 status code.
reqCount = 1
while get_stock_info('NFLX').status_code == 429:
    reqCount = reqCount + 1

after429Period = datetime.utcnow()

print ('Requests Rejected: {}'.format(reqCount))
print ('Elapsed Time: {}ms'.format(((afterRateLimited - after429Period).microseconds)/1000))

#setting to 1 because last call of previous while loop was 1 accepted request.
reqCount = 1
while get_stock_info('NFLX').status_code != 429:
    reqCount = reqCount + 1

after2ndRoundRequests = datetime.utcnow()

print ('Requests Accepted: {}'.format(reqCount))
print ('Elapsed Time: {}ms'.format(((after429Period - after2ndRoundRequests).microseconds)/1000))

The test code confirmed Polarizing1's findings. Here is the output from 2 runs of the test code....

None of these test runs are consecutive, do not draw any conclusions about the time in between the runs
RUN #1 
Requests Accepted: 120
Elapsed Time: 800.889ms
Requests Rejected: 456
Elapsed Time: 202.381ms
Requests Accepted: 120
Elapsed Time: 537.357ms

RUN #2
Requests Accepted: 120
Elapsed Time: 165.696ms
Requests Rejected: 619
Elapsed Time: 64.842ms
Requests Accepted: 120
Elapsed Time: 637.463ms

RUN #3 
Requests Accepted: 209
Elapsed Time: 411.353ms
Requests Rejected: 294
Elapsed Time: 232.891ms
Requests Accepted: 120
Elapsed Time: 594.887ms

RUN #4 
Requests Accepted: 0
Elapsed Time: 781.299ms
Requests Rejected: 423
Elapsed Time: 840.03ms
Requests Accepted: 120
Elapsed Time: 829.892ms

[alexgolec]

So based on this analysis, I think implementing a rate limiter would be counterproductive. If the rate limiter is even slightly out of sync with TDA's underlying rate limit, we will end up either needlessly reducing the number of calls that are permitted or accidentally placing too many and getting a rate limit exception.

So I've been thinking about it, and it seems to me that the real benefit of this change would be save the effort of handling 429 errors. Since tda-api uses the requests library under the hood, you can add retry policies to Client.session according to the following docs:

https://findwork.dev/blog/advanced-usage-python-requests-timeouts-retries-hooks/

[hmanfarmer]

@alexgolec I agree. With the ability to check for 429 errors being so easy, it doesn't make much sense to implement a much more complicated solution. And, I think a rate limiting solution might be impossible to implement properly without more details from tdameritrade on how they are enforcing the rate limiting on their end (details about what clock they are using).

Do you think tda-api should take care of implementing the requests error handling hooks? It would be a nice feature to have (off by default per our other conversations).

[alexgolec]

@hmanfarmer This sounds like a separate issue, no? I propose we close this ticket in favor of one that adds flexible support for timeouts and retry policies, and take discussions on further suggestions to either this ticket:

#71

Or the discord:

https://discord.gg/nfrd9gh

[joeatbayes]

I disagree with just hitting the API and handling the error. Any time you hit a vendors server with what could be a flood of requests when you have reasonable ability to determine that you exceeded their rate limit it should be considered server abuse which is a form of a DDOS attack. When you abuse the vendor's servers they put in all kinds of weird limits that ultimately hurt the clients and cost more. The vendor will inevitably pass these costs back to the customers. To be fair I have published hundreds of API for commercial companies so perhaps I empathize too much with them.

Solving this is super trivial.
Every time you start a request add the timestamp you made the request to a list.
Before you make a request. Remove all items from the list that are older than 1 minute.
If the length of the list is greater or equal to 120 then wait to make your next request.

Note if using multi-threading you will need to add locking to the list.

Note: When using multiple client computers I use an open-source distributed queuing software so all my outbound requests faced with the same limits are aggregated and sent from a single computer. I still had to build the rate limiter but it was less than 40 lines of code including locking. If the rate limit is much higher such as 10,000 requests per second then the logic has to change due to the list management overhead but at these low rates, it is a trivial function.

I would not depend on the 1 minute reset at the servers because most of the API server vendors trying to sell me rate-limiting API frameworks are working to improve the sophistication of their rate-limiting. As a warning. One of the API tool vendors showed me a feature last week where they detect abusers who are flooding their servers with excess requests. They showed me how they can lock out a single customer equivalent of a refresh token for a configurable amount of time such as 10 minutes as a penalty if they are flooding the server with requests. They also showed me a feature where they can lock out the MAC address of the abusing servers. They talked about a feature where they track all the MAC addresses used by the equivalent of an API Token and when any combination of that set of servers using that API Token abuses the rate-limiting they lock out the entire set of servers for a period of time.

[hmanfarmer]

@joeatbayes I agree with you.

Solving this is super trivial.
Every time you start a request add the timestamp you made the request to a list.
Before you make a request. Remove all items from the list that are older than 1 minute.
If the length of the list is greater or equal to 120 then wait to make your next request.

Yes, that would work. The only downside being that you can't eek out the absolute fastest rate. But, I think that is a more than fair tradeoff, especially, since you point out the fact that TDA could always adjust their under the hood implementation while still maintaining their advertised 120 requests per minute rate limit. I think the user base would appreciate a built-in rate limiter.

[wtf911]

So based on this analysis, I think implementing a rate limiter would be counterproductive. If the rate limiter is even slightly out of sync with TDA's underlying rate limit, we will end up either needlessly reducing the number of calls that are permitted or accidentally placing too many and getting a rate limit exception.

So I've been thinking about it, and it seems to me that the real benefit of this change would be save the effort of handling 429 errors. Since tda-api uses the requests library under the hood, you can add retry policies to Client.session according to the following docs:

https://findwork.dev/blog/advanced-usage-python-requests-timeouts-retries-hooks/

@alexgolec Could you kindly give me a code example of enabling retries if I encounter a 429? Right now I'm using your example like so:
c = auth.client_from_token_file(token_path, api_key)

Could you give a code example which would enable retries for "c"?

[MarkOfNY]

Joining very late to this conversation (almost a year later). I am observing very odd behavior with TDA API as of this morning. It seems to only allow requests to be made between a start of every minute and certain number of seconds later. Something like 15-30 seconds past a minute start. In my case, the total number of requests was nowhere near 120 per minute, yet once the seconds crossed 15 or sometimes 30, it'd return that same message, "Too many requests". I don't know if this is temporary issue. Emailed TDA API support.

[kjagadevan1983]

So, this 120 requests per min is specific to the account or specific to IP address?

[SkyHyperV]

API key specific I believe

[SkyHyperV]

How do I suppress and/or catch these messages from the console? I need to sleep and retry an API call but even a bare try/except doesn't catch it.

{'error': "Individual App's transactions per seconds restriction reached. Please contact us with further questions"}

[clone2324]

How do I suppress and/or catch these messages from the console? I need to sleep and retry an API call but even a bare try/except doesn't catch it.

Are you using
except Exception as e: ?