The Leading News & Information Service For The Facilities, Workplace & Built Environment Community

Monday, 21 October

Hacked - Cut Cables Disconnects UK's Largest Data Centre

One of the largest data hosting and cloud solutions providers, UK Fast's promise of '100% Uptime Guaranteed' was shattered week ending December 15 when a construction worker hacked through cables and disconnected the world.

'We know how important it is that your business is always online, which is why we (UK Fast) have invested in fully-owned Tier 3 standard data centres, enterprise-grade infrastructure and our own super-fast network, giving us guaranteed control of the end-to-end process' - unless of course a pick axe wielding construction worker cuts all the cables...and bang goes that theory.

UK Fast is very aware of its PUE - Power Usage Effectiveness - it is one of its major selling points. Unfortunately, if reports are to be believed, the company, one of the largest data hosting and cloud solutions providers in the UK and a major international player, were not so aware of back-up power systems.

 

Data outage

What UK Fast calls a data outage, customers call something else not entirely polite or publishable. Although UK Fast would have it that most customers were happy with the firm's responsiveness, social media tells a different story with UK Fast being lambasted.

UK Fast operates out of plush headquarter offices in Manchester. Thankfully they are plush as some systems folk (up to forty of them) worked 25 hour shifts to bring services back on stream. A complete loss of power was further complicated by  not being able to reboot.

 

Lawrence Jones MBE

UK Fast owner Lawrence Jones is a bit of a tech mogul, owning a techy magazine and events company BusinessCloud and Secarma the cyber security company. He also owns a Swiss ski hotel and an office fit-out business.

He told reporters that most software was up and running with minutes of the power failure but a VMware auto deployer had become corrupted and needing fixing.

Some hosted sites were down for the full 25 hours of the fix.

Jones also explained that two of its power supplies were cut by workmen (it is unlikely that this level of damage was a solitary pick axe and more likely a major digger). He then cited a back-up generator taking time to kick-in - only a few moments but enough to cause a major headache.

 

Protection

A data centre of this size really should have several power options and an uninterruptible system.

Stephen Peal, a director of Yorkshire based uninterruptible power supplies (UPS) distributor and maintenance company PPS Power said: "“This isn’t a site PPSPower support so I cannot comment on what is in place to ensure power continuity but any type of datacentre should have a UPS system in place to ensure there is no drop in power at all. A generator is only there to keep the UPS batteries charged and therefore support the site for much longer, indefinitely if the fuel tank is kept topped up. Further resilience should be installed on a site like this with UPS backing up UPS this is called N+1. Recently we installed an N+1+1 UPS system in a datacentre smaller than what I assume the UKFast site is.”

In a somewhat contradictory message (in that he claimed already to have such a system in place), Jones, has said UK Fast will be investing in a failover system to ensure a power outage will not happen again. He is promising a full report to customers - especially the 100 or so who were affected for the longest time.

An initial statement from UK Fast, which published a timeline throughout the problem starting on December 12 (updated hourly during the incident read: 'At 10:28 GMT our MANOC 5, 6 and 7 facilities were impacted by instability on the incoming mains power as a result of a civil contractor passing a spike through the main feed. This was work being carried at 0.75km away (not at a UK Fast faility) on the path to our onsite transformer. The UPS system supported the load for its designed time and the generators started. Physical damage to the power cable meant service to the site was unstable as a result, the generator sets failed to synchronize and take over service.

 

@Lawrence_Jones

Jones also took to Twitter to thank his staff. 'Huge thank you to the whole team yesterday for your commitment to our customers. The people in this picture enjoying a welcomed breakfast are some of the volunteers who worked a 25 hour shift

Article written by Brian Shillibeer

Share



Related Articles

And They're Off - Chelmsford Abandoned After Lights Go Out

The all weather horse racing track at Chelmsford City, which predominantly features evening events, was forced to call a halt to activities on Tuesday November 13 as a...

 Read Full Article
Load Bank Testing Could Save Emergency Generation Failure

Simone Bruckner explains why regular load bank testing should be a crucial part of any company’s standby strategy. Companies in general and data centres, in...

 Read Full Article
West Yorks Police & North Yorks Police - No Power Struggle

West Yorkshire Police uses PPSPower for its support generators and uninterruptible power supplies. North Yorkshire Police have the same deal. Initially, the firm held...

 Read Full Article
Power to the People

The majority of PPSPower’s customers are very satisfied with the company’s services and would recommend the business, according to results released in an...

 Read Full Article
Keeping Our Soaps On Screen

For the last year PPS Power has been playing its part in keeping ITV on the screens by providing ongoing maintenance services. The role has also involved engineers...

 Read Full Article
1+1 Makes A Sharp Project Sharper

PPSPower has provided a slick N+1+1 UPS solution for the Sharp Project, an unusual office space recently acquired by Manchester City Council. The space previously...

 Read Full Article
Bat Men Go Robbin'

SPIE has 'knicked' the contract for the Heathrow Terminal 5 central battery systems upgrade from under the noses of all other contenders. The deal will involve...

 Read Full Article
Property Flood Resilience eBook Launches

A new Property Flood Resilience eBook has launched to provide businesses and home owners with over 40 real-life case studies from business and communities that have...

 Read Full Article
Don't Let Driving Cause The Summertime Blues

The RAC is urging drivers to check their vehicles before setting out in the heat - and to be aware that heat can cause tiredness and tiredness can increase the likelihood...

 Read Full Article
Downpours Bring The Roof Down

June 24 saw the Met Office warn of heavy storms - and almost as the alerts went out, Stirling University and the local Tesco felt the full force of nature. The...

 Read Full Article