The issue has been resolved and all messages in push queues have now been flushed and delivered. Note: no messages were lost, but many were significantly delayed in their delivery to endpoints.
Additionally, we've identified and patched the root causes that have been causing delays in push queues over the past few weeks. We've found some edge cases around users creating hundreds of thousands of push queues with no subscribers causing the push processor to get caught up working on nothing for a long time causing delays for other customers. We patched things so that the push processor will bypass queues in those cases. This should significantly improve push queue performance.
We apologize for any application issues this may have caused. We take these issues seriously and are working around the clock to improve the service across the board for users of all paid tiers. That said, if you need guaranteed reliability and performance, and SLA's in place, please contact us about our enterprise dedicated cluster offering.
To discuss dedicated clusters, or anything else about this incident, you can contact us through our in-app messenger, or by emailing support@iron.io.
Posted Jan 18, 2017 - 17:07 PST
Update
We are preparing to deploy fixes to improve the push queue delays, please stand by.
Posted Jan 18, 2017 - 13:29 PST
Identified
We’re working on resolving the issues as quickly as possible.