Cyber Attacks & Recovery

In June of 2018, one of Cyanweb Solution’s shared hosting servers #5 (SV5) was taken offline in a criminal Cyber Attack.   The attack itself was well planned, with the aim of not only destroying all data on that system and but also all of it’s backup drives.

Cyanweb Solutions have generally been a highly reliable and quality hosting service provider for over 16 years – having started up it’s first hosting servers way back in 2002 – though all it took was 1 criminal cyber attack over 20 minutes to almost completely destroy 24 years of hard work.

Many of our customers know me [Jonathan Huckabee], have worked with me, or have received support from me personally over the past 24 years.   As Cyanweb has grown in client base, most times the phone is quiet and things cruise along as normal – though that all changed and I went though one of the darkest times in my career as a Digital Service Provider.

In this review of the events, the aftermath and the recovery process we will detail:

  1. exactly what happened
  2. why we think it happened
  3. what the problems were immediately after the event
  4. the recovery process
  5. what has changed and what we have upgraded

 

So what exactly happened to Cyanweb SV5?

On the 28th on June 2018 I was alerted to a high load on one of our servers SV5.cyanweb.com.au.  Within 15 minutes I was logged in and investigating the issue.  At first it appeared to be a standard denial of service attack with a large range of Swiss IP addresses hammering the server with web page requests across a wide range of hosted sites.

The fact that there were a lot of IP’s grouped from Switzerland was unusual.  I then proceeded to firewall the offending IP’s along with a range of other typical attacking IP’s from Brazil, Vietnam and Russia that appeared also to be involved in the attack.  Shortly thereafter, I noted that this had not produced the expected result of server loads dropping.

Investigation of processes showed about 30-40 php FPM processes running that were still loading the server – but a low IP range incoming.  While digging into these processes I noticed a couple of terminal “screens” open and then a few DD commands popping up – targeting attached backup disk drives.  That was the beginning of the end as that DD command almost instantly wipes disks.

This DD command was run on all our backup disks that were attached and running at the time of the attack.  Panic – and I immediately shut down the server.

Of course when trying to restart the server it did not come back online and I contacted our upstream provider at Servers Australia Pty Ltd to assist.  Their team were soon at the data centre and on the affected server.   All server log folders were wiped so there was no way to trace the attackers from logs.  This is typical attacker behaviour where an automated script runs on restart to wipe all log files.

At first it appeared that only the backup drives were affected in the attack, as the file system on the main drives appeared intact and all home folders were accessible.  I was relieved, but that was short lived, as it was soon discovered that a high speed encryption algorythm was run on the home folders – effectively making the contents of all files in home folders unreadable.  This is where all client site data and email files are stored.

It was also apparent that the DDoS attack was a distraction, effectively masking the PHP processes that were busy destroying the home folder data.

While most home folder data stored on the backup systems and internal client email and web folders on SV5 were effectively destroyed, we were able to shut down the server before all the database data was deleted. Almost all databases were left untouched by the attack.  A few minutes more and it would have been a total loss.

It was very very lucky that I was directly logged into the system when this happened and noticed the screen and DD commands.

The survival of the databases allowed us to recover 90% of client’s web site core database data and rebuild from local and online archive versions of sites.  We were also able to utilise client development data stored at our offices in conjunction with these databases.

Why destroy a small business web hosting server?

The purpose of the attack appears to be commercially motivated rather than a cyber vandalism, terrism or criminal attack.

It is almost impossible to track down an attacker on a small hosting appliance such as cPanel where logs were deleted by the attacker.  Criminal investigation resources are not available to solve such small cases,  so the criminal is almost assured relative safety from discovery.

Why do we suspect a Commercial Cyber Attack?:

  1. while client data was destroyed by fast encryption method – there were no associated ransom note files that are always included in criminal ransom encryption attacks.  Criminals sometimes also copy any data the think may be valuable data first.  There were no ransom files, or contacts with payment demands of any kind.
  2. There were no noted increases in outbound data and we believe that this was simply a well planned seek and destroy attack – meaning the goal was simply to destroy all server and all attached backup drives as quickly as possible.   Hackers usually break into systems to exploit them and engage in further criminal activities (such as spamming or using the compromised system in other attacks) and generally try to remain undetected for as long as possible – this was not the case with SV5.
  3. cyber vandals will usually claim fame after an attack, or deface the web sites hosted on the server with their TAGs for notoriety – again this was not the case and eliminates the amateur, stereotypical “hacker” that most people associate with “hacking”.
  4. anonymous servers housed in Switzerland were used as part of the Cyber attack – these are like Swiss bank accounts and identities of the users / lessors are protected – they cost money and vandals won’t hire Swiss servers to attack someone.  Anyone with money who commits Cybercrimes for commercial profit will take every precaution to avoid discovery, and paying money to make money would make sense and point to paying for anonymous Swiss servers.
  5. immediately after the event a range of media and press outlets were alerted by the attackers – this indicates the intent was to destroy the brand and reputation of Cyanweb as quickly as possible after the attack.  While there were almost 500 “domains” housed on the server there were fewer than 250 clients.  The remaining domains were home to a “domain broker” who buys domains for the sole purpose of selling them again.  Most of our clients know us personally, and would not have started notifying media. We even began to suspect at least one of these “profiteering” press outlets may have been directly in contact with or even part of the brand assassination attempt on Cyanweb Solutions.   No attempts have been made by any press outlet to follow up nor responded to enquiries to fix incorrect facts about the attacks.
  6. within hours many of our clients reported receiving telemarking and targeted email marketing offering “speedy recovery services”. These services including loading up duplicate copies of affected sites downloaded from The Internet Archive.   These were being offered for around $1000 on a commercial yearly contract.  While I don’t think all of these commercial telemarketers were part of the attack, I do suspect at least 2 or 3 were.  These fronts were calling clients way before the attack was made public.  This type of client sign up would net the new provider at least $1000 per client and there would be no wat to tie them to the crime except circumstancial / obvious deductions without proof.  More than a few clients reported getting these calls.  In a way this was lucky as it gave us a possible solution for recovering site from The Internet Archive ourselves.

The motiviation appears to have been purely financial.   This type of cybercrime is profitable – with say just 200 clients – a 5% return is highly likely given the urgency of Email / Web Sites – so even if 10 sign up – that’s $10,000 for 2 hours work for a processional Commercial Cybercriminal Organisation – and unfortunately there are people out there in every industry who have no reservation about destroying others to make a dollar.

This type of attack is like a competitor burning down your supply warehouse then calling all your customers saying they can supply your products to them because you no longer are able to.

After the attack was public, and word had gotten around the induscty via Whirlpool and Tech news outlets, I had calls and emails from numerous small providers who had experienced the exact same type of attack and subsequent vulturing of clients.  While it made me feel better that I was not alone, I had lost a lot of faith in humanity as a whole, as many victims of crime often feel… how could anyone be so… evil?

How did they get in?

I suspect that a hosting client gave an untrustworthy, unscrupulous 3rd party their Cpanel hosting or WordPress login.  Most likely someone who telemarketed them for Seach Engine Optimisation, or some other service and they blindly trusted to work on their web site without understanding the security implications of giving out their account passwords.

This hosting account / wordpress administrator access login was most likely was used to run software on the server to discover a security hole within the server OS or PHP layer. This software or OS flaw allowed them to run scripts as a high level admin.  This assumption is using Occam’s razor, given the nature of the attack and how the encryption was done using PHP processes.

It makes sense that a competitor with hacking knowledge or ties to others would use this direct access to the hosting layer to find a hole and take down a competitor without fear of discovery.

UNDERSTAND: giving someone your cPanel or WordPress Administrator Login gives them permission to run any program on your hosting account – it gives them full access to your user filesystem and to run whatever software they want on your account.

Why were backups destroyed?

The biggest issue with the attack was the fact that the SV5 server’s off-site backups were not running at the time of the attack.  While we had local backups in place, off-site systems were not running at the time.  We did have backups in place, though they were internally connected and were primarily for recovery in the event of hardware or disk failure.

The SV5 server was scheduled for a full migration to a newer server in August / September and resources were being put toward this new server.   The internal backups were thought to be enough in case of hardware failure during this time – with off-site / network backups to be re-instated after the migration.

The SV5 dedicated shared hosting server is a standard cPanel hosting appliance that was thought to be secure, and recoverable given that automated backups and software updates were in place on the appliance.

Unfortunately there was either an undisclosed OS flaw, software or kernel patch that left an attack vector open. This allowed someone with direct login access to probe the server from within, discover the appliance software flaw, escalate their priveledges, and run the software required to perform the data destruction attack.

What have you done to insure this type of data loss can not happen again?

Firstly,  I have taken up a contract with our upstream provider to manage all higher level server software security maintenance and monitoring.  This insures all servers are consistently patched with the latest available patches without need for a direct server administrator, limiting the chances of human error in security management.

Secondly, all remaining and future Cyanweb Servers now have both Local internal backups + full off-site backups running nightly.  This allows for a 24 hour recovery of any server with data loss minimised to the last 24 hours.

Thirdly, all WordPress web sites built by Cyanweb have an additional layer of backup security. All WordPress based web site databases, files and data are backed up to the cloud on a regular basis.

 

The Aftermath & The Road to Recovery

The sigle biggest issue we faced after the loss of the SV5 server was communications overload.  In total about 250 clients were left without their sites and about 200 without email.  All clients who were using 3rd party email such as Office 365 were not affected for more than a day or two and no email was lost on those accounts.

With all these clients calling all at once there was no way for us to take all the calls.  This was the single most damaging aspect of the aftermath as clients were frustrated at not being able to get answers of speak directly to myself.

Our first priority was to get client email flowing again. We immediately started cataloging affected sites and rebuilding accounts and sending clients details and options for setting up email.  Forwarding addresses were setup as quickly as we could.

With such a small team it was difficult and I hired additional staff to deal with the overload.  Within 2 weeks all our staff had quit, citing customer abuse and the pressure was to great for them to handle.

Thankfully by then we had the bulk of accounts back online – though the task now was to start recoving web sites.  Rebuilding 10 years worth of web sites was not going to be easy.

The first task was to prioritise client site recovery based on who had the most to lose by their web site being down.  This had to be done purely on a financial estimation – meaning if your business site was responsible for $10,000 day in sales vs a site that was say $100 day in sales we would prioritise accordingly.

The next step was to catalogue all the sites that we had additional backups for on our development servers.  Luckily we had about 30 sites still on our dev server that could be recovered relatively quickly.

For those sites where no additional backups were available, we devised a recovery plan that would see 4-5 sites recovered per day given our limited staff and financial resources.  This attack took Cyanweb to the absolute edge of The End – though I was committed to my clients to recover and get everyone I could back online.

It took roughly 6 weeks to get the bulk of our clients back online – and after 3 months we managed to retain and recover 70% of all client data.  It would not be until November until we could finally say that things were “back to normal” here at Cyanweb.

While we lost 30% of our customer base that was on SV5 – we had other servers with clients who were not affected by the attack.   Media headlines are wildly incorrect and make the attack to be far worse than it was in the end.

The Damage Done

While Cyanweb has managed to survive the attack, and things are almost back to normal, the financial damage is immense.  I had to borrow up to the hilt to get everyone back online and repurpose our hosting platforms with upgraded server security and new backup systems as a result.  I could have given up, though my commitment to our clients and to the company I spent 24 years of my life working on kept me going at all costs.

Emotionally it was almost unbearable.  I am thankful to all of you in the idustry who called, sent messages and emailed offering support of staff, your time and those that just called to see if I was ok – I still get choked up when I think of how much people cared out there – your support and empathy kept me going.

Still to Do:

We are building a new client support portal where clients will be able to register all their primary and secondary contacts.  This will allow for better communication in any event.  Having a database of all client mobile contacts would have saved us ALL a lot of time, money, and stress!

We will contact all clients with details of how to access and maintain these details as soon as it is ready.

Where are we now with Cyanweb Solutions?

I am now working full time for the Forrest family at their Minderoo Foundation as a support web developer and communications team member.  So at the moment I am working 2 jobs to pay off the debts incurred by the attack, with myself being directly available to Cyanweb customers after hours and weekends only.

Cyanweb are maintaining select clients for digital marketing work and we only take in web development projects as and when we have capacity. Surajit Sur is staying on as our primary web site content manager and web developer, and I am working weekends when I am able and as my workload at Minderoo allows it.   We have Kim Benck working as our content writer and consultant as needed and other casual staff available as needed.

We are primarily taking on high impact projects where we can be of the most help, including project cosulting, re-branding, re-developments, and digital marketing projects.

We are also working close with others in the industry, bringing in talent on a per project basis as needed, and referring work where appropriate.  While we have slowed down a bit,  Cyanweb Solutions is still very much in business and we are still here to help.

Conclusions & Thank You:

I have to stop writing now, but for anyone who made it this far, thanks for taking the time to read this – it was hard for me to revisit the attack after so many months.  I hope this gives everyone a clearer picture of what happened.

We have almost fully recovered and we have taken the necessary steps to never ever have to go through that experience again…

Do please feel free to contact us via email should you have any questions at all.
– Jonathan Huckabee
CEO / Director