Rackspace hosted Exchange suffered a catastrophic failure starting December 2, 2022 and is still ongoing since 12:37 AM December fourth. At first described as connectivity and login problems, the assistance was eventually updated to reveal that they were handling a security occurrence.
Rackspace Hosted Exchange Issues
The Rackspace system decreased in the morning hours of December 2, 2022. Initially there was no word from Rackspace about what the issue was, much less an ETA of when it would be fixed.
Customers on Buy Twitter Verification reported that Rackspace was not reacting to support e-mails.
This has actually been quite the day with #Rackspace. Every hosted exchange customer has been down for 14 hours or so. Support isn’t reading/responding to tickets. Updates are unhelpful.
I am worried now that they came down with something bad like the ProxyNotShell PoC hack. https://t.co/jchKsAO3Z7
— Joe Sinkwitz (@CygnusSEO) December 2, 2022
A Rackspace customer privately messaged me over social networks on Friday to relate their experience:
“All hosted Exchange customers down over the previous 16 hours.
Unsure the number of business that is, but it’s substantial.
They’re serving a 554 long hold-up bounce so individuals emailing in aren’t knowledgeable about the bounce for several hours.”
The official Rackspace status page offered a running update of the outage but the initial posts had no details aside from there was a blackout and it was being investigated.
The very first official update was on December 2nd at 2:49 AM:
“We are investigating a concern that is impacting our Hosted Exchange environments. More details will be published as they appear.”
Thirteen minutes later Rackspace began calling it a “connectivity concern.”
“We are investigating reports of connection problems to our Exchange environments.
Users may experience an error upon accessing the Outlook Web App (Webmail) and syncing their email client(s).”
By 6:36 AM the Rackspace updates described the ongoing issue as “connection and login issues” then later that afternoon at 1:54 PM Rackspace revealed they were still in the “examination phase” of the blackout, still attempting to determine what went wrong.
And they were still calling it “connectivity and login concerns” in their Cloud Workplace environments at 4:51 PM that afternoon.
Rackspace Recommends Moving to Microsoft 365
4 hours later on Rackspace referred to the situation as a “considerable failure”and started using their consumers totally free Microsoft Exchange Plan 1 licenses on Microsoft 365 as a workaround up until they comprehended the problem and could bring the system back online.
The main guidance mentioned:
“We experienced a significant failure in our Hosted Exchange environment. We proactively closed down the environment to avoid any further problems while we continue work to bring back service. As we continue to overcome the root cause of the problem, we have an alternate service that will re-activate your capability to send out and get e-mails.
At no charge to you, we will be supplying you access to Microsoft Exchange Strategy 1 licenses on Microsoft 365 till further notice.”
Rackspace Hosted Exchange Security Incident
It was not up until almost 24 hours later at 1:57 AM on December 3rd that Rackspace formally announced that their hosted Exchange service was suffering from a security occurrence.
The announcement even more exposed that the Rackspace service technicians had actually powered down and disconnected the Exchange environment.
“After additional analysis, we have actually determined that this is a security incident.
The known effect is isolated to a part of our Hosted Exchange platform. We are taking required actions to evaluate and protect our environments.”
Twelve hours later that afternoon they updated the status page with more details that their security group and outside professionals were still working on fixing the failure.
Was Rackspace Service Affected by a Vulnerability?
Rackspace has not released details of the security event.
A security occasion normally includes a vulnerability and there are two extreme vulnerabilities currently in the wile that were patched in November 2022.
These are the 2 most current vulnerabilities:
Microsoft Exchange Server Server-Side Request Forgery (SSRF) Vulnerability
A Server Side Request Forgery (SSRF) attack allows a hacker to read and alter data on the server.
Microsoft Exchange Server Remote Code Execution Vulnerability
A Remote Code Execution Vulnerability is one in which an assailant is able to run harmful code on a server.
An advisory published in October 2022 described the effect of the vulnerabilities:
“A validated remote aggressor can perform SSRF attacks to escalate opportunities and carry out arbtirary PowerShell code on susceptible Microsoft Exchange servers.
As the attack is targeted against Microsoft Exchange Mail box server, the aggressor can possibly get to other resources via lateral movement into Exchange and Active Directory environments.”
The Rackspace blackout updates have actually not shown what the specific problem was, only that it was a security incident.
The most present status update as of December 4th mentioned that the service is still down and clients are motivated to migrate to the Microsoft 365 service.
Rackspace published the following on December 4, 2022 at 12:37 AM:
“We continue to make progress in addressing the event. The schedule of your service and security of your information is of high significance.
We have committed comprehensive internal resources and engaged first-rate external competence in our efforts to decrease negative impacts to customers.”
It’s possible that the above noted vulnerabilities belong to the security event affecting the Rackspace Hosted Exchange service.
There has actually been no statement of whether client details has been jeopardized. This event is still ongoing.
Included image by Best SMM Panel/Orn Rin