Salesforce engineers roll back change after breaking own cloud for hours today
Services said to be returning to normal from downtime though Tableau Cloud still MIA
Updated If you noticed something funky going on with Salesforce and its software-as-a-service empire today, it's not you: it's recovering from an hours-long outage.
As of writing, Salesforce said at 1829 UTC (1130 PT) in a status update that all of its clouds are returning to normal after suffering about a four-hour downtime.
Well, all except for Salesforce's Tableau and Mulesoft, which remain down or affected to some extent.
The IT breakdown started at 1448 UTC, and we're told by the enterprise software giant this hit "customers across multiple clouds including, Commerce Cloud, Mulesoft, Tableau, Core, Marketing Cloud Account Engagement, Marketing Cloud Intelligence, and Omni Channel."
As a result, "users are unable to log into Salesforce or access any of their services," the biz admitted.
Two hours later, the team said its ClickSoftware, Trailblazer, and Data Cloud products were also affected. At first a third-party cloud provider was thought to be to blame for intermittent networking issues causing Salesforce services to fail.
Then the tech titan said, actually, a cloud provider wasn't at fault, and Salesforce engineers "successfully executed a rollback to mitigate the issue." By 1752 UTC, the biz said its cloud systems were getting back on track, and customers should be able to log in and use their applications as usual, as long as those apps weren't Tableau or Mulesoft.
- Azure SQL Database takes Saturday off on US east coast following network power failure
- Microsoft admits slim staff and broken automation contributed to Azure outage
- Square blames last week's outage on DNS screw-up
- Gartner: Stop worrying and love the cloud, with all its outages and lock-in
Us vultures, as Slack users, noticed a blip in connectivity with that Salesforce-owned chat app, for what it's worth, though Slack is not reporting any substantial outage. It's possible Slack was slightly affected as a knock-on from the outage or efforts to recover from it.
For those relying on the analytics platform Tableau: although Tableau Public has apparently been fully restored, Tableau Cloud is still screwed, we're told, some four hours after a problem was first detected. A status update two hours ago stated:
Tableau Engineers have identified the issue causing the Tableau Cloud outage for all pods. Engineers continue to work on the solution to mitigate user impact. We thank you for your patience; our next update will be when new information becomes available.
Tableau Engineers have identified the issue causing the Tableau Cloud outage for all pods. A solution to mitigate user impact is currently being worked on.
Meanwhile, 20 minutes ago, automation toolkit Mulesoft said its services are recovering after a fix was applied.
We think it's safe to say Salesforce broke part of its back-end systems with an update, causing services and products relying on that infrastructure to fail, and customers unable to log in and use the stuff as expected. Backing out that change has helped fix things. We'll keep you updated. ®
Updated to add at 1920 UTC
It appears Mulesoft has joined the rest of the Salesforce collective in recovering from the earlier outage though Tableau is awaiting the wider deployment of an update to right its services. In an update, the SaaS giant said:
The team continues efforts to fully restore all impacted services to a healthy state. At this time, Tableau is the only remaining impacted service. A fix was deployed to a dev environment and successfully mitigated the impact. That fix is now being rolled out to production.
We've also asked Salesforce for further info, such as how exactly it broke its clouds and what steps are being taken to prevent that from happening again.