Off-Prem

Tencent Cloud to revisit design after circular dependencies slowed emergency API fix

Almost 2,000 customers experienced outages


Tencent Cloud has apologized for an outage that impacted customers last week – an unusual act by a Chinese cloud – and signalled it will review some aspects of its ops in the hope of avoiding future incidents of this nature.

A WeChat post from China's number three cloud – Alibaba Cloud leads the market, ahead of Huawei, with Baidu in fourth place – revealed that on April 8 it updated configuration data for one of its APIs.

The change was a dud, and the API became unavailable. Some Tencent Cloud platform-as-a-service offerings services that rely on it therefore became unreliable, effectively cutting them off from the Tencent Cloud.

The cloud provider was able to fix the mess in just 87 minutes, but has apologized to the 1,957 customers who reported failures.

The triage post explains that the failure was caused by an update to the API that didn't consider compatibility.

Changes to the API's interface protocol meant that apps targeting the old version produced nonsense data that spread across Tencent Cloud and meant the API became unstable.

Tencent Cloud would usually roll back this sort of change. But to do so, it needed the very same API it had just broken.

The cloudy concern admitted that it just didn't test this release properly – it ignored some of its own version change processes, didn't conduct proper sandbox tests, and now realizes its change management processes probably need some work.

The outfit has pledged to redesign bits of its cloud to detect abnormal changes and terminate them before they spread, conduct drills to improve its incident response, and offer alternative APIs should the interfaces fail.

Tencent Cloud is not alone in breaking its own cloud with an update. The Register has reported on similar messes at Google, AWS, and Microsoft.

One important difference, however, is that Tencent Cloud is reputedly so intolerant of outages that it has been known to fire staff responsible for resilience after incidents.

Chinese media suggest that approach may have backfired – sources allege job cuts at Tencent Cloud contributed to this incident. ®

Send us news
2 Comments

US adds web and gaming giant Tencent to list of Chinese military companies

This could be the start of a saga to rival TikTok’s troubles, and embroil Tesla and Microsoft

China to probe US chip subsidies as export curbs rattle allies

Beijing investigating claims of unfair competition in mature semiconductors

Akamai to quit its CDN in China, seemingly not due to trouble from Beijing

Security and cloud compute have so much more upside than the boring business of shifting bits

Biden said to weigh global limits on AI exports in 11th-hour trade war blitz

China faces outright ban while others vie for Uncle Sam's favor

Microsoft invites Chinese software vendors to sell on its marketplace and through its partners

Good luck getting buyers and resellers excited about that

Donald Trump proposes US government acquire half of TikTok, which thanks him and restores service

Incoming president promises to allow ongoing operations for 90 days just as made-in-China app started to go dark

Now Trump's import tariffs could raise the cost of a laptop for Americans by 68%

Make America irate again

Nvidia snaps back at Biden's 'innovation-killing' AI chip export restrictions

'New rule threatens to squander America's hard-won technological advantage' says GPU supremo

TSMC revenue booms and you don’t need AI to figure out why

PLUS: China plans unified APIs; Singapore lets Police run scam victims’ bank accounts; Fujitsu now too cool for aircon

FCC to telcos: By law you must secure your networks from foreign spies. Get on it

Plus: Uncle Sam is cross with this one Chinese biz over Salt Typhoon mega-snooping

Clock ticking for TikTok as US Supreme Court upholds ban

With Biden reportedly planning to skirt enforcement and kick the can to Trump, this saga might still not be over

Brit government contractor CloudKubed enters administration

Home Office, Department for Work and Pensions supplier in hands of FRP Advisory