Issue affecting some API clients
Incident Report for Open Exchange Rates
Postmortem

We experienced an API issue yesterday, after making a small change to our platform in preparation for migrating to our revamped infrastructure next week. This change resulted in an unanticipated permissions conflict, meaning that about 400 clients were unable to access our "latest.json" API endpoint between approximately 03:00 and 06:20 UTC.

Instead of their regular up-to-date exchange rate information, affected clients received an error message stating that their App ID was invalid.

The change we made was in a section of our production environment data pipeline that lies outside of our regular deployment path, and was not picked up by our independent monitoring due to the specific conditions under which it occurred, so we were unable to catch the error ahead of time.

As soon as we received notice that clients were unable to access our latest API endpoint at 06:00 UTC, we investigated, detected and deployed a fix for the error. From 06:20 UTC, affected clients were once again able to access our latest API. Any client who had cached a response locally during that time may have continued to experience errors in their integration until their cache was renewed.

This incident was caused by human error. We have tightened up our pre-deployment review procedures in response, to ensure that it can not happen again.

We apologise that this occurred and sincerely hope it didn't cause severe issues for you or your business.

Please don't hesitate to contact support@openexchangerates.org with any questions or comments.

William Crowcroft - Founder & CEO, Open Exchange Rates

Posted Jul 15, 2016 - 19:47 UTC

Resolved
We have resolved this issue, which affected around 400 users of our service who were unable to fetch rates for approximately 30 minutes, and will write a post-mortem report this afternoon. Thank you for your patience.
Posted Jul 14, 2016 - 06:20 UTC
Identified
We have identified the issue and are deploying a fix.
Posted Jul 14, 2016 - 06:14 UTC
Investigating
We are investigating an issue affecting some users of our API and will post again as soon as we have an update. Apologies for any inconvenience and please contact support@openexchangerates.org with any questions.
Posted Jul 14, 2016 - 06:06 UTC