Encoded - Notice history

At 17:32 a hardware failure on the primary database server caused some connections to the database to lock. This compounded into an issue resulting in connection exhaustion, which resulted in all database activity halting.

Encoded engineers were alerted at 17:42. Following initial investigations an issue with the primary database server was quickly identified, and a failover to a secondary database was initiated at 17:51. The failover operation was completed at 17:57.

Following the failover, a number of application servers required some additional work to close existing locked connections and initiate new connections to the newly promoted database. This work was completed at 18:09.

We will be performed further investigations into the root cause of the initial primary database issue and will update with an RCA once this has been completed.

If you have any queries regarding this incident, please contact us via the Encoded Service Desk

After monitoring services for some time, we have not seen any further occurances of telephone issues and can confirm this has been resolved.

If you have any queries regarding this incident, please contact us via the Encoded Service Desk

Our upstream provider has been in touch and have informed us they have resolved this incident. Encoded engineers have conducted testing to confirm this and we have been able to process calls again without any issues.

We will continue to monitor services and mark this as resolved once further checks have been completed.

Our upstream supplier has confirmed they are experiencing a service incident and are currently working to restore services.

This incident is only affecting telephone-based services, such as IVR automated payment services and Virtual Terminal services where telephone services are used, such as assisted payments and telephone callback services. All of Encoded's other services are operational.

Our investigtions found that calls are not connecting intermittently, however once a call has successfully connected, we have not observed this being dropped without due cause and services can still be used to complete payments.

We will update this page once we have received updated from our upstream supplier.

We are seeing that calls are intermittently connecting, for calls which do not connect customers may experience engaged tones, or silent calls which disconnect.

We are continuing our investigation on this incident with our upstream provider and will provide an update once available.

We experienced an issue with our SFTP service during a routine patching and rotation of our servers. One of our SFTP servers was assigned an incorrect host key, which may have affected connectivity for some users. This host key should have remained consistent across all SFTP servers, however, it was not applied as intended in this instance. As a result, any automated processes to drop off or collect files from Encoded may have failed due to requiring further action to accept new host keys.

We apologise for any disruption this may have caused. The issue has now been fully resolved and the original host keys have been published.

If you would like to confirm the successful completion, or request that Encoded re-run any missed file imports between 08:45am on the 28th of October and 12:45pm on the 29th of October, please reach out to our Service Desk for assistance.

Thank you for your understanding.

Cashflows have confirmed they have resolved the incident that occured between 14:36 - 14:45.

As of 14:45, transactions are processing correctly on the Cashflows payment gateway. Encoded will continue to monitor transaction processing rates until we have received confirmation from Cashflows that this incident has been resolved and provide an update once received.

Between 14:36 - 14:45 all transactions processed for customers with merchant services with Cashflows would have received a system error, you will receive standard messaging that the transaction has failed, as set out in your specific service design document.