Cyber Attacks & Recovery

In June of 2018, Cyanweb Solution’s shared hosting server #5 (SV5) was taken offline in a criminal Cyber Attack. The attack itself was well planned, with the aim of not only destroying all data on that system and but also all of it’s backup drives.

Cyanweb Solutions have generally been a highly reliable and quality hosting service provider for over 16 years – having started up it’s first hosting servers way back in 2002 – though all it took was 1 criminal cyber attack + opportunistic media journalists to almost destroy my company.

Many of our customers know me, have worked with me, or have received support from me personally over the past 24 years. While Cyanweb has grown in client base over 24 years, most times the phone is quiet and things cruise along as normal – though that all changed one evening in June 2018, and I went though one of the darkest times in my life and my career as a Digital Service Provider.

In this review of the events, the aftermath and the recovery process we will detail:

exactly what happened
why we think it happened
what the problems were immediately after the event
the recovery process
what has changed and what we have upgraded

So what exactly happened to Cyanweb SV5?

On the 28th on June 2018 I was alerted to a high load on one of our servers SV5.cyanweb.com.au. Within 15 minutes I was logged in and investigating the issue. At first it appeared to be a standard denial of service attack with a large range of IP addresses hammering the server with web page requests across a wide range of hosted site domains.

DDoS attacks are quite normal and usually easily managed as we had been through this many times in the past. The fact that I noticed straight away there were a lot of IP’s grouped from Switzerland was unusual. I then proceeded to firewall the offending IP’s along with a range of other typical attacking IP’s from Brazil, Vietnam and Russia that appeared also to be involved in the attack. Shortly thereafter, I noted that this had not produced the expected result of server loads dropping.

Investigation of processes showed about 30-40 php FPM processes running that were still loading the server – but a low IP range incoming. While digging into these processes I noticed a couple of terminal “screens” open and then a few DD commands popping up – targeting attached backup disk drives. That was the beginning of the end as that DD command almost instantly wipes disks.

This DD command was run on our backup disks that were attached and running at the time of the attack. Panic – and I immediately shut down the server.

Of course when trying to restart the server it did not come back online and I contacted our upstream provider at Servers Australia Pty Ltd to assist. Their team were soon at the data centre and on the affected server. All server log folders were wiped so there was no way to trace the attackers from logs. This is typical attacker behaviour where an automated script runs on restart to wipe all log files.

At first it appeared that only the backup drives were affected in the attack, as the file system on the main drives appeared intact and all home folders were accessible. I was relieved, but that was short lived, as it was soon discovered that a high speed encryption algorythm was run on the home folders – effectively making the contents of all files in home folders unreadable. This is where all client site data and email files are stored.

It was also apparent that the DDoS attack was a distraction, effectively masking the PHP processes that were busy destroying the home folder data.

While most home folder data stored on the backup systems and internal client email and web folders on SV5 were effectively destroyed, we were able to shut down the server before all the database data was deleted. Almost all databases were left untouched by the attack. A few minutes more and it would have been a total loss.

It was very very lucky that I was directly logged into the system when this happened and noticed the screen and DD commands.

The survival of the databases allowed us to recover 90% of client’s web site core database data and rebuild from local and online archive versions of sites. We were also able to utilise client development data stored at our offices in conjunction with these databases.

Why destroy a small business web hosting server?

The purpose of the attack appears to be commercially motivated rather than a cyber vandalism, terrism or a typical ransomware criminal attack.

Criminal investigation resources are not available to solve such small cases, so these types of commercial / business based ‘white collar cyber criminals’ are almost assured relative safety from discovery.

Why do we suspect a Commercial Cyber Attack?:

while client data was destroyed by fast encryption method – there were no associated ransom note files that are almost always included in criminal ransom encryption attacks. Criminals sometimes also copy any data they think may be valuable data first. There were no ransom files, nor any signs of downloaded data nor follow up contacts with payment demands of any kind.
There were no noted increases in outbound data. We believe that this was simply a well planned commercially motivated seek and destroy attack – meaning the goal was simply to destroy the server and all attached backup drives as quickly as possible at the worst possible time = 2 days before End of Financial Year when taxes and accounting were in full swing… While customers are in a panic, the criminals contact affected businesses and try to ‘sign up’ as many hosting / web clients as possible.
Hackers usually break into systems to exploit them and engage in further criminal activities (such as spamming or using the compromised system in other attacks). Hackers generally try to remain undetected for as long as possible – this was not the case with SV5.
cyber vandals will usually claim fame after an attack, or deface the web sites hosted on the server with their TAGs for notoriety – again this was not the case and eliminates the amateur, stereotypical “hacker” that most people associate with “hacking”.
anonymous servers housed in Switzerland were used as part of the Cyber attack – these are like Swiss bank accounts and identities of the users / lessors are protected – they cost money and vandals won’t hire Swiss servers to attack someone. Anyone with money who commits Cybercrimes for commercial profit will take every precaution to avoid discovery, and paying money to make money would make sense and point to paying for anonymous Swiss servers.
immediately after the event a range of media and press outlets were alerted by the attackers – this indicates the intent was to destroy the brand and reputation of Cyanweb as quickly as possible after the attack. While there were almost 500 “domains” housed on the server there were fewer than 250 clients. The remaining domains were home to a “domain broker” who buys domains for the sole purpose of selling them again.
most of our clients know us personally, and would not have started notifying media. They were tipped off in our best optinion. We even began to suspect at least one of these profiteering press outlets may have been directly in contact with or even part of the brand assassination attempt on Cyanweb Solutions. No attempts have been made by any press outlet to follow up nor responded to enquiries to fix incorrect facts about the attacks. Instead most areas online are simply using the attack to make money for themselves. Certain media outlets used blanket spam techniques on LinkedIn, mobile sms, and email spamming to alert clients of the attack claiming they were just looking for comment… when in fact they were alerting clients as to what happened.
these press outlets appeared to have full contact lists of all clients hosted on the server which was very odd to us. It appeared obvious to us that a lot of preparation had gone into this attack.
the commercial attack + scam: within hours clients reported receiving telemarking and targeted email marketing offering “speedy recovery services”. These services including loading up duplicate copies of affected sites downloaded from The Internet Archive. These were being offered for around $1000 on a commercial yearly contract. These downloadied copies cost less than $30 Australian dollars. While I don’t think _all_ of these commercial telemarketers were part of the attack, we do suspect at least 2 or 3 were fronts for the criminals. Other industry hosting providers that had been hit by similar attacks confirmed with me that their clients were also telemarketed after their servers were attacked.
these criminal business fronts were calling clients before the attack was made public so they had inside knowledge of the attack.
commercial benefits to the criminals involved: each successful ‘sign up’ would net the criminal provider at least $970 per client. There would be no way to tie them to the crime except circumstancial / obvious deductions without proof. More than a few clients reported getting these calls after the event. In a way this was lucky, as it gave us a solution for recovering site from The Internet Archive ourselves.

So, by going over all the circumstantial evidence, the motiviation appears to have been purely for criminal financial gain.

This type of cybercrime is highly profitable – with say just 200 clients – a 5% return is highly likely given the urgency of Email / Web Sites / End of Year timing – so even if just 10 sign up – that’s $9,700 for 2 hours work for a professional Commercial Cybercriminal Organisation

Unfortunately there are people out there in every industry who have no morals nor reservation about destroying others lives to make a dollar.

To put this into perspective:
This type of commercial attack is very much like a bricks and mortar competitor burning down your shop / supply warehouse, then calling all your customers saying they can supply your products to them because you no longer are able to.

After the attack was public, and word had gotten around the industry via Whirlpool and Tech news outlets, I had calls and emails from numerous small providers who had experienced the exact same type of attack and subsequent vulturing of clients.

So how did they get in?

I suspect that a hosting client gave an untrustworthy, unscrupulous 3rd party, most likely a telemarketed SEO or Cheap Site rebuild service, their hosting or WordPress administrator login. Most likely someone who telemarketed them for Seach Engine Optimisation, or promising some other web service. People often blindly trust strangers to work on their web site without references and without understanding the wider security implications of giving out their account passwords.

WE NO LONGER PROVIDE HOSTING OR WORDPRESS ADMINISTRATOR LOGINS to our clients. All administration must go through our support channels.

It’s called hacking the human. Humans are almost always the weakest link in Cyber Security. Tricking someone to divulge their hosting logins or WordPress logins is fairly easy as the promise of making more money on page 1 of Goolge catches a lot of people.

A compromised or disclosed hosting account / wordpress administrator access login was most likely used to load and run software on the server, allowing probing of the server from within in an attemps to discover a security hole within the OS, PHP or other software layer.

It stands to reason that eventually an unpatched or undiscovered Operating System flaw allowed them to run scripts as a high level admin. Given the nature of the attack and how the encryption was done using PHP processes, this ‘insider’ route of attack is much more likely than the Hollywood ‘hacker’ scenario of someone over the web hacking into a server. These types of successful attacks are almost always started from within the system.

Without this type of Username / Password access, compromise of the server would have been next to impossible.

IMPORTANT FOR EVERY BUSINESS OWNER:

Giving someone your cPanel or WordPress Administrator Login generally gives that person permission to run any program on your hosting account – it gives them full access to your user filesystem and to run whatever software they want on your account – even if it’s just a ‘WordPress Administrator Login’. They have full access to all email hosted on the same account, your databases and any information you may have uploaded. Be aware and never disclose your account logins to anyone.

Why were backups destroyed?

The biggest issue with the attack was the fact that the SV5 server’s off-server backups were not running at the time of the attack. We had multiple local and internal backups in place – which were there for recovery in the event of hardware or disk failure while we were preparing that server for migration. An inside hacker deleting the disks was something we were never expecting to happen.

Why no off-site? The SV5 server was scheduled for a full migration to a newer server in August / September and preparations and resources were being put toward this new server. The internal and local backups (2 backups) were thought to be enough in case of hardware failure during this preparation and migration time period – with off-site / network backups to be re-instated after the migration.

The SV5 dedicated shared hosting server is a standard cPanel hosting appliance that was generally thought to be secured, and recoverable given that automated backups and software updates were in place on the appliance.

Unfortunately there was either an undisclosed OS flaw, cPanel software or kernel patch that could not be applied that left an unknown attack vector open. This allowed someone with hosting or WordPress login access to probe the server from within, discover the appliance software flaw, escalate their priveledges, and run the software required to perform the data destruction attack.

What have you done to insure this type of data loss can not happen again?

Firstly, I have taken up a contract with our upstream provider to manage all high level server software security maintenance and monitoring. This insures all servers Kernels are consistently patched with the latest available security patches without need for a direct server administrator, limiting the chances of human error in security management.

Secondly, all remaining and future Cyanweb Servers now have both Local internal backups + full off-server backups running nightly. This allows for a 24 hour recovery of any server with data loss minimised at the very least to the last 24 hours to up to 1 month in the past should there be any data corruption.

Thirdly, all WordPress web sites built by Cyanweb have an additional layer of cloud backup security. All WordPress based web site databases, files and data are backed up to the cloud on a regular basis – further isuring against web site data loss.

Clients should always keep similar off-site / cloud based backups of all their business and email data. If you don’t have backups contact www.Qbit.com.au for assistance.

The Aftermath & The Road to Recovery

The sigle biggest issue we faced after the loss of the SV5 server was communications overload. In total about 250 clients were left without their sites and about 200 without email. All clients who were using 3rd party email such as Office 365 were not affected for more than a day or two and no email was lost on those accounts.

With a large number of clients calling all at once there was no way for us to take all the calls. This was the single most difficult aspect of the aftermath, as clients were frustrated at not being able to get answers or speak directly to myself. It was just not possible.

So, our first priority was to get client email flowing again. We immediately started cataloging affected sites and rebuilding accounts and sending clients details and options for setting up email. Forwarding addresses were setup as quickly as we could process client requests for assistance.

With such a small team it was difficult, and I immediately hired additional staff to deal with the overload.

Unfortunately within 2 weeks all Cyanweb’s staff had quit, citing customer abuse (relatively few of our customers reacted like this – and most were 3rd party / independents that just would not let us get to work and kept calling over and over again – making it more difficult for everyone).

Thankfully by the time everyone on the support team quit we had the bulk of email accounts back online or forwarding to alternate addresses.

The task now was to start recoving the actual web sites. Rebuilding around 10 years worth of web sites was not going to be easy. Luckily many of our advanced business hosting clients were unaffected by the attack.

The next task was to prioritise client site recovery based on who stood to lose the most by their web site being down. This had to be done purely on a financial estimation – meaning if your business site was responsible for $10,000 day in sales vs a site that was say $100 day in sales we would prioritise accordingly.

The next step was to catalogue all the sites that we had additional backups for on our development servers. Luckily we had about 30 sites still on our development server that could be recovered relatively quickly. These were scheduled and uploaded as quickly as possible.

For those sites where no additional backups were available, we devised a site recovery plan that would see 4-5 sites recovered per day given our limited staff and financial resources. This attack took Cyanweb to the absolute edge of The End – though I was committed to getting everyone I could back online no matter the cost.

It took roughly 6 weeks to get the bulk of our clients back online – and after 3 months we managed to retain and recover 70% of all client data. It would not be until November 2018 when we could finally say that things were “back to normal” at Cyanweb.

Of note: media headlines are wildly incorrect and make the attack to be far worse than it was. Granted, at first I thought it was worse that it was as well!

The Damage Done

While Cyanweb and many clients managed to survive the attack (thanks to everyone who helped out and for all of you who showed compassion and patience), and things are almost back to normal, the financial damage to us personally was immense. I had to borrow up to the hilt to get everyone back online and re-think + re-purpose our hosting platforms with additional server security and new backup systems.

The attack cost us around $70,000 in immediate expenses and over 5 year loss of around $150,000 / in lost client hosting revenue. The damage to the company is a total loss in terms of no new work comes in because of the incident and the Internet never forgets.

I would not just let 24 years of my life’s work be wiped out by some jerk criminals… so a rebrand will be forthcoming.

Emotionally the whole thing is almost unbearable. I am thankful to all of my clients who stood by us, and all of you in the idustry who called, sent messages and emailed offering support of staff, your time and those that just called to see if I was ok. I still get emotional when I think of how much clients and even complete strangers from the industry showed their support – you know who you are – and thank you again.

Still to Do:

We are building a new client support portal where clients will be able to register all their primary and secondary contacts. This will allow for better communication in any event. Having a complete database of all client mobile contacts would have saved us ALL a lot of time, money, and stress!

We will contact all clients with details of how to access and maintain these details via the support portal as soon as it is ready.

Where are we now with Cyanweb Solutions?

I am now working at PWD as a senior web developer and consulting team member.

While I am full-time elsewhere – I am making myself available to Cyanweb customers after hours and weekends as needed.

Cyanweb are maintaining select clients for digital marketing work and the team only take in web development projects as and when we have capacity.

Surajit Sur is staying on as our primary web site content manager and web developer, and I am helping out weekends when I am able. We also have Kim Benck working as our content writer and consultant as needed and other casual staff brought into projects when required.

As a general rule, Cyanweb are only taking on high impact projects where we can be of the most help, including project cosulting, re-branding, re-developments, and digital marketing projects.

We are also working close with others in the industry, bringing in talent on a per project basis as needed, and referring work where appropriate. While we have slowed down quite a bit, Cyanweb Solutions is still in business and as always, we are here to help the best we can.

Sometime in 2020 we will be relaunching as a specialty web consulting brand where my 25+ years of experience, both good and bad will be available to those in need of help on web projects and general business and technology processes.

Conclusions & Thank You:

I have to stop writing now, but for anyone who made it this far, thanks for taking the time to read this – it was hard for me emotionally to revisit the attack after so many months. One may not think that this type of crime actually hurts people, but it does – it does a lot – both financially and mentally.

That said, I hope this gives everyone a clearer picture of what we think happened and what we are doing to insure this can never happen again.

Do please feel free to contact us via email should you have any questions at all.
– Jonathan Huckabee
CEO