Dealing with broken APIs

Nowadays, most websites integrate with a handful of APIs (Application Programming Interface). A typical website shows events from Eventbrite, shows the latest tweets from Twitter, automatically adds people into Mailchimp, accepts payments by credit card or via Paypal, etc.

Unfortunately, many APIs are not that well designed. They are usually a marketing afterthought built to “tick a box”. The task of integrating poorly designed and sometimes buggy APIs is left to us (web developers and digital producers). Here is a list of approaches we have taken while working with dozens of broken APIs.

1) Dealing with unresponsive APIs

Some APIs work well … most of the time! Developers easily build the desired features and while you’re testing it, you notice that sometimes the website gets stuck for a while. Refreshing the page “fixes” the issue but you cannot reproduce the issue for another 10 minutes and then it happens again. This is extremely frustrating and contacting the API’s developers turns out to be a waste of time (as they pretend that there is no problem).

Two things can be done by developers to deal with unresponsive APIs:

1.1 - Cache data (both ways) and use the API asynchronously

While we cannot rely on external APIs, we can rely on our good, old relational databases (especially PostgreSQL https://www.postgresql.org/) to cache data. As long as direct feedback from the API isn’t necessary; I would actually recommend to always cache the data so the website is as fast as possible. This is true when getting data from an API and when sending data to an API.

Examples:

Getting data: if you need to list the latest events from an external service, these events shouldn’t be retrieved via the API on every page-load. The events should be regularly retrieved by a task (a CRON job running on the server) and saved into the website’s database. The page would load the events from the database.

Sending data: adding a message to a CRM for each contact form entry via an API. When a user fills out the contact form, that message should be synched in the company CRM. If you try to send that message to the CRM while submitting the form; the CRM API might fail and the end-user would see an error message. It’s much better to simply save the message in the (reliable) database on submit - and then - to asynchronously save it into the CRM as well. This approach is not only faster for the end-user; it also avoids losing the message altogether and it makes it easy to retry to save it in the CRM later in case of failure.

1.2 - Improve visibility for end-users and administrators

In some situations, caching is not possible. That’s the case with payments. If the API you use to handle a payment doesn’t respond; payment-details cannot be saved in the database (for security reasons) and you cannot even assume that these are correct. In other words, the only thing you can do is let the customer know that the payment API is down.

To avoid communication issues, the UX should make it clear that the payment handler is not working and that the team has been notified. You want to avoid unclear messages such as “there is an issue” and be more specific: “The payment platform isn’t responsive. The team has been notified. Please try again later.”. A clear error message will avoid frustration and help the customer-support team.

More visibility is greatly beneficial to website administrators and content editors. For complex websites with various API integrations, it’s great to let them see API statuses and synchronization jobs statuses in the website admin. For example, an administrator wondering why an image hasn’t been imported yet into the CMS could look into the “image import job” screen, see that there is a job in progress (instead of sending emails to the developers).

2) Dealing with unclear and/or outdated documentation

APIs often lack clear documentation. Worse, their documentation is often incorrect (especially the provided examples). As a result, developers cannot design and write their code based on it.

2.1) Zero trust

Even with the best / largest companies; you will often find incoherences in between the documentation and what the API actually does. On top of that, there are probably undocumented features or limitations that are specific to your account. It is therefore important for developers to not trust the API’s documentation but to try it out instead.

2.2) Use CURL before programming

When trying an API; it’s tempting to start coding but it’s actually faster to start with “manual“ experiments (via CURL for example https://curl.haxx.se). Manually testing the API means that, in case of problem:

• We know the issue isn’t in the code
• We can send the CURL command to the API support team
  • In which case they can’t pretend that it’s a programming error
• We can confirm API behaviour changes (same CURL command, different result)
  • The raw CURL commands and their result must be saved
• We can then use that “real” data in automated tests
  • The raw CURL commands and their result must be saved

3) Dealing with inconsistent data

Inconsistent data in APIs is thankfully a rare issue (I can’t remember this happening with any of the GAFA (Google, Apple, Facebook and Amazon) APIs). It’s a rare issue but when you have to deal with it, it’s a nightmare. We generally treat API results as the source of truth. If you use an external API to handle payments, you expect that API to return the correct paid-price (including fees) after submitting the payment - so - what do you do when the returned paid price is negative!?

3.1) Validate everything

If you’re dealing with a clearly broken API, it’s best to validate what it returns and to not save any erroneous data in the database. In other words, it’s highly preferable to fail fast than to have to deal with strange bugs down the line. In the negative paid-price example, you clearly don’t want to show that to a customer in their receipt (you could stop the transaction here or calculate it yourself and ignore the API result).

3.2) Fetch the data again

APIs might return a response quickly before properly treating your request. The negative price in a response might mean that the new price hasn’t been calculated or confirmed yet. If you realise that the API returns data to you before actually checking it (returning nonsense values or mirroring data you’ve provided to it); a good strategy is to ignore the result and to instead re-fetch the same information via different API calls.

3.3) Add plenty of automated-tests with “real” data

Since our job is to automate things, manual testing when developing is clearly insane. There is nothing as helpful as a good suite of tests to deal with complex APIs. That’s especially true with inconsistent ones. Every time you get weird, unexpected results, you should double-check it via CURL (see 2.2 above) and create the corresponding automated test.

4) Dealing with hopeless APIs

Like Gordon Ramsay giving up on a restaurant, sometimes an API is just beyond usage but your website must still run with it. Here are some rare, last-resort solutions which are more political than anything else really.

4.1) Offer an alternative

You must accept payment via an unresponsive API. You could lose half your business doing so. In case of a bug, instead of simply showing “There is a problem with the payment provider…” as mentioned above, also add a link to pay via another platform such as Paypal. The political trick is to show that only in case of failure.

4.2) Block-out full sections

There might be a few pages on your website that rely on one unstable service. If you can detect that the service is down (and show it to the administrators as mentioned in 1.2), you could also block access to all the related pages so that users don’t waste time navigating out-of-date information or filling up long forms for nothing…

Conclusion

API integrations are sometimes harder than necessary. Some APIs simply stop working once in a while in which case you want to cache as much data as possible and be as transparent as possible with all users. The documentation for an API is often incorrect and a lot of manual testing must be done. Sometimes, APIs return incorrect data which cannot be blindly trusted. Again, manual and automated tests are very helpful. Finally, you might even have to avoid using an API altogether while keeping the best user experience possible.

With all that being said, in most cases, we find API integrations fairly straightforward. All it takes is a few experiments, some tests and it all works nicely. Most of this article is based on the few - but very real - broken APIs we had to deal with. Writing about it is therapeutic and it hopefully gives you ideas for your particular “use-cases”.