Resolved -
The queue has cleared, and messages were processed accordingly, no messages were list. We are scheduling a planned modernization of the service's runtime to prevent this class of issue permanently.
We'd like to apologize for any potential inconvenience this issue may have caused.
Aug 12, 13:26 CEST
Monitoring -
We've applied a fix that restored the functionality, and are currently monitoring the export and message queue being processed.
Aug 12, 11:52 CEST
Identified -
An automatic dependency update in an internal background-processing service introduced library versions that are incompatible with the service's current runtime. As a result, the service fails on startup and stopped processing its work queues. The portal itself, sign-in, and all data collection from cash registers are unaffected, and no data has been lost.
We are deploying a corrected build of the service with compatible dependency versions; queued exports and emails will be processed automatically once it is back online, without any action needed from users. Separately, we are scheduling a planned modernization of the service's runtime to prevent this class of issue permanently. This notice will be updated once processing has caught up.
Aug 12, 10:54 CEST
Investigating -
We're currently experiencing delayed data export and notification processing in our cloud platform. Exports started in the portal do not complete within reasonable time - they remain in "pending" state and the export page keeps waiting. Notification and report emails are delayed. No data has been lost: all requests are safely queued and will be processed automatically once service is restored; started exports will complete without needing to be restarted.
Aug 12, 10:50 CEST