Encoded - INC-120 - Perimeter Firewall Firmware Update Issue – Incident details

All systems operational

INC-120 - Perimeter Firewall Firmware Update Issue

Resolved
Major outage
Started 9 months agoLasted about 2 hours

Affected

Data Centres

Major outage from 11:02 PM to 12:18 AM, Operational from 12:18 AM to 12:53 AM

Colo 2

Major outage from 11:02 PM to 12:18 AM, Operational from 12:18 AM to 12:53 AM

Web Services

Major outage from 11:02 PM to 12:18 AM, Operational from 12:18 AM to 12:53 AM

Email

Major outage from 11:02 PM to 12:18 AM, Operational from 12:18 AM to 12:53 AM

Web Payment Services

Major outage from 11:02 PM to 12:18 AM, Operational from 12:18 AM to 12:53 AM

Virtual Terminal

Major outage from 11:02 PM to 12:18 AM, Operational from 12:18 AM to 12:53 AM

Updates
  • Resolved
    Resolved

    This incident has now been resolved, and all services have been restored. We apologise for the inconvenience this has caused. We are working with the vendor to understand the root cause of the incident.

  • Investigating
    Investigating

    Unfortunately an issue has been identified during scheduled maintenance (MNT-62) to update the firmware of our perimeter firewalls, which is resulting in degraded performance across all services.

    All services are operational but with degraded performance. You may notice some intermittent connectivity issues or slower than usual response times during this period.

    We are currently working with the firewall vendor's support team to rectify the issue. We have been given an ETA of up to 2 hours for the correct specialised support resource to become available to aid in resolving the issue.

  • Identified
    Identified

    We are currently running through troubleshooting with the firewall vendor, however this has unfortunately caused a number of internal systems to lose connectivity, resulting in outage of services whilst this troubleshooting is undertaken.

    We will keep you updated of progress.