Encoded - INC-135 Encoded Data Centre intermittent connectivity issues – Incident details

All systems operational

INC-135 Encoded Data Centre intermittent connectivity issues

Resolved
Degraded performance
Started 4 months agoLasted about 23 hours

Affected

Data Centres

Degraded performance from 9:27 AM to 5:39 PM, Operational from 5:39 PM to 8:36 AM

Colo 2

Degraded performance from 9:27 AM to 5:39 PM, Operational from 5:39 PM to 8:36 AM

Database

Degraded performance from 9:27 AM to 5:39 PM, Operational from 5:39 PM to 8:36 AM

Web Services

Degraded performance from 9:27 AM to 5:39 PM, Operational from 5:39 PM to 8:36 AM

Email

Degraded performance from 9:27 AM to 5:39 PM, Operational from 5:39 PM to 8:36 AM

Web Payment Services

Degraded performance from 9:27 AM to 5:39 PM, Operational from 5:39 PM to 8:36 AM

Updates
  • Resolved
    Resolved
    After monitoring services overnight we can confirm that this incident has been resolved.
  • Monitoring
    Monitoring

    Our upstream providers have completed the rollbacks and service availability has been restored.

    We will continue monitoring and provide further updates in due course.

  • Update
    Update

    Our upstream providers have confirmed with us that they are bringing forward their rollback of their updates to 17:00 - 18:00 BST on June 12th.

    As previously mentioned, during this rollback it is expected that there will be period of downtime across all services whilst this update is applied and this is expected to last less than 5 minutes.

    We will monitor services and will provide updates once the rollback has been completed.

  • Identified
    Identified

    Our upstream providers have identified the root cause of the intermittent connectivity issues. A rollback of their recent changes is scheduled to take place between 06:00 and 08:00 BST on June 13th.

    We have been advised that this maintenance will involve a brief period of service unavailability across all services, expected to last less than 5 minutes during the rollback window.

    Once the rollback is complete, we will continue monitoring all systems and will provide a further update.

  • Investigating
    Investigating
    Our upstream ISP and data centre provider are currently experiencing intermittent internet connectivity issues following an update completed on June 11th. As a result, Encoded has observed brief interruptions to web-based services overnight, which are continuing intermittently today. In rare cases, customers using web based services may encounter "Error 503: Backend unavailable, connection timeout" messages: * Hosted Payment Pages * Gateway API * Virtual Terminal * Web Modules (e.g., Transactions, Timetables) * IVR (Transaction attempts and API account lookups) These errors typically resolve upon refreshing the page. We are actively working with our upstream partners to identify and resolve the root cause. This page will be updated as soon as more information becomes available.