PageFair security breach has been resolved – here is what you need to know.
Update 10 – 20:57 GMT, 7 November 2015: How to check if you were exposed
You deserve to know if you were placed at risk during this attack. If you did download and install the fake Adobe Flash update shown to you, you need to know that the trojan is on your computer Windows computer has been removed by your anti-virus software.
We have outlined exactly what a vulnerable website visitor would have been shown if they accessed a compromised site in the 83 minute window that the hack occurred. That information is in a separate post, located here.
Update 9 – 19:30 GMT, 6 November 2015: Why this will not happen again.
It is now six days since one of our CDNs was compromised for 83 minutes by a hacker. We have worked hard this week to analyse and disclose what happened to our clients and the world. Thanks to the cooperation of the NanoCore author and the dynamic DNS service Dyno, whatever access the hacker had to infected computers was shut down on Tuesday. In addition, for the last 4 days over 90% of antivirus tools (by market share) are detecting and cleaning the malware.
More detail is below, but here are the summary steps taken are:
- All authentication credentials on all systems (internal and external) have been reset
- All core systems have been reviewed for MFA support, and it has been enabled where possible.
- Reset mails are being isolated into a dedicated email account.
- We are reviewing which IT/SaaS vendors we use based on their support for MFA, login termination and security auditing.
- We are training employees in recognising spear phishing attempts.
Immediate Steps (taken immediately 1st November)
We have always enforced strict policies around the management of passwords – that they be strong, are used for one system only, are stored only in encrypted files, and are never transmitted in cleartext. Access to all servers is via strong key-based authentication and our server infrastructure is deliberately architected to limit the potential attack surface.
In the immediate wake of this attack on early Sunday morning the team set about auditing core systems for access and performed resets on the access credentials to every internal and external system. It is our policy to have Multi-Factor Authentication (MFA) in place on email accounts, but this did not extend to all other systems, including our CDN providers. Immediately after the attack we activated MFA on all critical systems that support it, including CDNs.
Spear phishing is powerful.
The attack started with a very convincing email that was not flagged by Gmail as suspicious. It was faked to look like it came from me (the CEO) to members of staff. It was short, but the subject and content were plausible. It contained a link that appeared to point to a Youtube page, but which actually linked to a faked Google authentication screen customised to the target user with a pre-filled email and avatar. The URL of the authentication screen began with “https://services.google.com”. The only give-away (on a sufficiently large screen) was that the domain name didn’t end there. After the user entered the password, the phishing site also perfectly mimicked Google’s MFA authentication workflow. In short, anyone could have fallen for it.
We are training staff in how this kind of attack works and how to recognise it. We will do the same for all new staff. Unfortunately, training alone cannot totally eliminate the risk of spear phishing. Spear phishing email accounts is powerful because it can be used to leverage access into new systems. Therefore, password reset mails should never be forwarded into any account that is in regular use and which could be compromised through social engineering. We are therefore reconfiguring all systems so that password reset mails are forwarded to a special account, which is not used for any other purpose.
Faked email like this could be flagged as a potential phishing attack by email clients, based on tell-tale signs like:
- it purported to be an internal mail, but was routed via a series of external mail servers
- it contained a link with anchor text containing a URL to a different domain.
Unfortunately, these signals did not automatically flag the message. Perhaps this is something that email clients should look for.
Critical System Security
We are classifying certain SaaS external IT systems as “critical”. These include systems that
- can affect communication between browsers, websites and pagefair
- may contain sensitive information now or in the future or
- is used for staff communication.
We are reviewing all these critical systems for support of (a) MFA (b) the existence of an audit log including IP addresses and user agents of connected clients and (c) the ability to terminate any active logged in sessions. We will consider changing vendor if any system fails to support these security features.
Update 8 – 15:40 GMT November 4, 2015
More antivirus products have updated their signatures to remove the trojan involved in this attack. By combining statistics from VirusTotal and Statista, 92.5% of people using antivirus should have immunity to this trojan.
Update 7 – 18:02 GMT November 3, 2015
Any website visitors who installed the malicious program are now safe from being controlled by the hacker.
The creator of NanoCore, the tool on which the malware was based, has revealed on Twitter that they have terminated the account responsible for the attack. When we reached out to verify the source of these tweets he also stated that:
“The attacker used a heavily modified version of NanoCore (which bypassed our restrictions which prevent this, among other things). We are very sorry that out product was used in such manor and have banned the person responsible… As a result of the attacker being banned, everybody who was affected by this are safe, the file will only connect to his licence, which has been banned. However the file will still be running. Over time Anti-Viruses will detect it and remove it. Malwarebytes does a good job at this.”
Update 6 – 21:38 GMT November 2, 2015
We are provisionally estimating that a relatively low percentage of 2.3% of visitors to the 501 affected publishers during the 83 minutes of the attack would have been placed at risk of infection (although a greater number of visitors would have seen the alert dialog). We hope to refine this estimate during the next 24 hours, but we want to provide background to this calculation now in case it is useful.
Key figures in this calculation are as follows:
- 33 minutes into the attack we updated DNS to bypass the CDN, and 50 minutes later we cut off the attack entirely. Factoring in DNS propagation, 70% of requests during this period would have been routed to the CDN.
- According to statistics in our CDN, only 25% of requests to hackers’ IP addresses succeeded, with the majority returning 504 server errors. On each page load, up to two attempts may have been made to fire the alert dialog. The probability of one of these succeeding was 43% (100% – 75% x 75%).
- Assuming 100% of users shown the modal alert dialog would click “ok”, and given that the hackers were serving the trojan from the same IP addresses that were returning server errors in 25% of cases, the executable file would only succeed in downloading 25% of the time, .
- The trojan targets Windows only, which accounted for 45% of internet traffic in the US in October, according to statscounter.
- According to Microsoft, 24% of PCs worldwide were unprotected by virus scanners in 2014.
- Of the remaining 76%, we calculate that 59% of the marketshare of virus scanners did not detect the trojan when the attack happened (this calculation is based upon virustotal data on what scanners identified the trojan earlier today and opswat data on anti-virus market share).
Multiplying out these percentages (70% x 43% x 25% x 45% x 24% + 70% x 43% x 25% x 45% x 76% x 59%) leads us to an estimate that 2.3% of visitors during the attack period were put at risk.
Once exposed, visitors would also have to choose to run the downloaded executable, while also dismissing security warnings that would have been displayed by Windows about the unverified source of the executable.
Next steps: Our CDN is providing us with a detailed log of all requests against our account during this period. Tomorrow we will perform a more detailed analysis of this log to establish if it corroborates the above mathematics.
Update 5 – 19:36 GMT November 2, 2015
If you believe that you have downloaded the malicious file onto your Windows operating system or you are a publisher that is being contacted by your website visitors about this problem, this update is for you.
We have received reports that the malware in question causes unexpected behaviors in certain Microsoft products such as Word, Excel, and Outlook. If you notice unexpected behavior please use one of the anti-virus programs listed here to scan your system. Choose one of the programs beside the red text (the top-half of the list). This should fix the problem.
If you are in customer support, you can use the following wording:
Thank you for contacting us. To fix this problem we recommend that you scan your system using one of these anti-virus programs, which should detect and remove the malicious file. Choose one of the programs beside the red text (the top-half of the list).
Update 4 – 17:58 GMT November 2, 2015
F-Secure has posted further information of the trojan involved in this incident, including screenshots of what affected users would have seen. They identify the executable as the Nanocore trojan. On Twitter, security researcher @bartblaze has analyzed the file
and believes that it is primarily intended for Bitcoin mining (correction: @bartblaze has clarified that the trojan disables other bitcoin miners during installation).
Update 3 – 16:41 GMT November 2, 2015
The executable connected to this attack is only capable of affecting users of Microsoft Windows. According to VirusTotal, the executable file in this attack would have been correctly identified by many anti-virus products, including Avast, Kaspersky, McAfee, F-Secure, Sophos, AVware, BitDefender and others.
Update 2 – 14:30 GMT November 2, 2015
We have established that only a fraction of the publishers we work with were affected by this attack. The attack affected only versions of the PageFair analytics tag that used the CDN in question. We have established that 501 publishers were affected during the 83 minute period. Most of these publishers are small, with 60% having less than one million page views per month, and 90% having less than ten million page views per month.
Next steps: Regardless of size, every publisher is important to us. We are working to estimate the degree to which visitors to all these websites would have been affected by the attack. We will update again shortly.
Update 1 – 21:30 GMT November 1, 2015
The attack was sophisticated and specifically targeted against PageFair, but it is unacceptable that the hackers could gain access to any of our systems. We identified the breach immediately, but it still took over 80 minutes to fully shut it down. During this time, visitors to websites owned by the publishers who have placed their trust in us were targeted by these hackers.
The damage was mitigated by our standard security practices, but the attackers still gained access. I want to take some time here to describe exactly what happened, how it may have affected some of your visitors, and what we are doing to prevent this from ever happening again.
We will update this post as we establish more facts.
WHAT YOU NEED TO KNOW
We noticed the security breach within 5 minutes, but it took until 01:15 (83 minutes) to fully rectify the situation. After this time visitors were no longer affected.
If you had the free PageFair Analytics code installed on your website yesterday, it is possible that some visitors to your website will have downloaded the malicious executable file. We are directly notifying every publisher who had our code deployed during this time. If we do not reach out to you directly, it means that you were not affected.
WHO WAS AFFECTED?
WHAT WAS NOT AFFECTED
There is no evidence or reason to believe that any core pagefair servers or databases were compromised. No publisher account information, passwords or personal information has been leaked.
- For today, our priority has been to ensure that all systems are fully secure and that all company-wide passwords are reset. This has been done.
- Tomorrow we will audit the level of access to company documents that the hackers may have gained. We do not store any Personally Identifiable Information in any system, but we will advise partners if we have reason to believe any sensitive documents may have been accessed.
- We will analyze which security practices failed and which could be strengthened and adopted to prevent something like this from occurring in future.
- We will continue to post mortem this for the remainder of the week, and will regularly update this post with our findings.
Thanks to our customers who were patient with us during this issue, The Media Trust Company, who worked hard to reach us during the issue, and MaxCDN for being available in real time to help lock the hackers out of our account. We will have more updates tomorrow.
Please ask us any questions in the comments section below or feel free to reach out to us at email@example.com. We will respond to every single email and query that comes our way. We will also be updating our Twitter account as we update this post.