Working with unreliable web APIs
To be clear, you should not compromise on API quality for mission critical things. We do not do this.
Sometimes you have to work with web APIs that have reliability issues and or inconsistent error responses – and changing APIs is not always an option.
Here are a five tips we’ve compiled at Instavest to help you deal with finicky web APIs:
1: Retry loops
This tip only applies to information requests (not changing data).
We had a third party data source that gives us 500 errors and undocumented error responses at high rates (90% of so of requests succeeded). Mathematically speaking, trying twice should give us an error rate of only 1%:
90% + 10% * 90% = 99%
Re-trying 2 times should lower it another order of magnitude:
90% + 10% * 90% + 1% * 90%= 99.9%
success = False
for x in xrange(3):
try:
response = requests.get('https://api.bigco.com/v1/entries/')
if response.status == 200:
success = True
break
except ValueError:
logger.info(“Request failed {} time(s)”.format(x+1))
if not success:
logger.info(“Failed to complete a request for...”)
# Handle error
2: Thorough logging
Each time you have an error for an API request, make it crystal clear what went on so that the API provider can speed up debugging. This includes:
- request URL (including path)
- request headers
- request data parameters (note: hash sensitive things like access tokens)
- http auth parameters (note: hash this too)
- the response text
Your code should log this stuff regardless of a success of failure – this is your audit trail.
3: Set up automatic emails
As you work with more third parties, error reporting can become a lower priority especially if you’ve built a robust application with solid error handling. This does not mean its OK – we have several third party APIs that are not critical to Instavest’s functionality but they help provide better user experiences and therefore are important (e.g. data APIs).
We found that automating emails to tech support teams with detailed error diagnostics (as described above) can be helpful. We also use casual language in the emails so that they don’t think the emails are automated.
request_params = {
'url': 'https://api.bigco.com/entries/.../',
'request_type': 'POST',
'data': {},
'headers': {},
...
}
success = False
for x in xrange(3):
try:
response = execute_request(request_params)
if response.status == 200:
success = True
break
except ValueError as e:
logger.info(“API request {url} failed: {e}, details: {request_params}”.format(**request_params)
if not success:
logger.info(“Failed to complete a request for...”)
send_email_for_bigco_api_error(request_params)
# Handle error
4: Use error reporting tools
We use Sentry to report unhandled exceptions. This is critical because you will never anticipate every error and log reporting tools may have latency (some people periodically grep their logs for exceptions and email alerts).
You should be using this regardless of whether you use third party web APIs or not! Sentry is just one option, there are several well regarded options
Integrating one of these tools usually requires minimal work (it fits in as middleware). So if an uncaught exception occurs, you’re notified (along with frequency and users affected).
5: Track response latency and warn yourself internally
Response times increasing can be a leading indicator of errors. It’s not too difficult to time this in your logic and add it to logs. The Python library, Requests, (has this built in)[http://docs.python-requests.org/en/latest/api/?highlight=elapsed#requests.Response.elapsed]
Start by logging every request to response time delta, and find the distribution curve of these data points.
For example, find the median time response and distribution. When a response time greater than the 95th percentile occurs, give yourself a warning or contact the party responsible for this to warn them. You might help them avoid heavy load errors!
Moreover, this data can help with periphery things like timing heavy data pulls. We run several periodic scripts at Instavest that heavily poll data APIs to calculate interesting statistics for our users. We saw some of these scripts running very slowly – but moving them to times where the third party APIs were more available drastically sped things up.
Understanding this data can help you answer import questions on error handling and user experience.
We hope these tips help you integrate and work better with finicky web APIs. Remember, just because an API isn’t as beautiful as Stripe, doesn’t mean you should ignore it – we’ve had great success working with a variety of APIs at Instavest. Comments? Reach us at hello@instavest.com - we’d love to hear from you.