keyboard

Halloween Security Breach

Sean Blanchfield Uncategorized 7 Comments

PageFair security breach has been resolved – here is what you need to know.

Update 10 – 20:57 GMT, 7 November 2015: How to check if you were exposed

You deserve to know if you were placed at risk during this attack.  If you did download and install the fake Adobe Flash update shown to you, you need to know that the trojan is on your computer Windows computer has been removed by your anti-virus software.

We have outlined exactly what a vulnerable website visitor would have been shown if they accessed a compromised site in the 83 minute window that the hack occurred.  That information is in a separate post, located here.

Update 9 – 19:30 GMT, 6 November 2015: Why this will not happen again.

It is now six days since one of our CDNs was compromised for 83 minutes by a hacker.  We have worked hard this week to analyse and disclose what happened to our clients and the world. Thanks to the cooperation of the NanoCore author and the dynamic DNS service Dyno, whatever access the hacker had to infected computers was shut down on Tuesday. In addition, for the last 4 days over 90% of antivirus tools (by market share) are detecting and cleaning the malware.

I want to now take some time to talk about initial steps we’ve taken to stop this from happening again and also share enough detail to be helpful to any other javascript vendor who wants to learn from our experience.

More detail is below, but here are the summary steps taken are:

  1. All authentication credentials on all systems (internal and external) have been reset
  2. All core systems have been reviewed for MFA support, and it has been enabled where possible.
  3. Reset mails are being isolated into a dedicated email account.
  4. We are reviewing which IT/SaaS vendors we use based on their support for MFA, login termination and security auditing.
  5. We are training employees in recognising spear phishing attempts.
  6. We are launching a new open source project to authenticate dynamic javascript.

Immediate Steps (taken immediately 1st November)

We have always enforced strict policies around the management of passwords – that they be strong, are used for one system only, are stored only in encrypted files, and are never transmitted in cleartext.  Access to all servers is via strong key-based authentication and our server infrastructure is deliberately architected to limit the potential attack surface.

In the immediate wake of this attack on early Sunday morning the team set about auditing core systems for access and performed resets on the access credentials to every internal and external system. It is our policy to have Multi-Factor Authentication (MFA) in place on email accounts, but this did not extend to all other systems, including our CDN providers. Immediately after the attack we activated MFA on all critical systems that support it, including CDNs.

Spear phishing is powerful.
The attack started with a very convincing email that was not flagged by Gmail as suspicious. It was faked to look like it came from me (the CEO) to members of staff. It was short, but the subject and content were plausible. It contained a link that appeared to point to a Youtube page, but which actually linked to a faked Google authentication screen customised to the target user with a pre-filled email and avatar. The URL of the authentication screen began with “https://services.google.com”. The only give-away (on a sufficiently large screen) was that the domain name didn’t end there. After the user entered the password, the phishing site also perfectly mimicked Google’s MFA authentication workflow.  In short, anyone could have fallen for it.

We are training staff in how this kind of attack works and how to recognise it. We will do the same for all new staff. Unfortunately, training alone cannot totally eliminate the risk of spear phishing.  Spear phishing email accounts is powerful because it can be used to leverage access into new systems.  Therefore, password reset mails should never be forwarded into any account that is in regular use and which could be compromised through social engineering. We are therefore reconfiguring all systems so that password reset mails are forwarded to a special account, which is not used for any other purpose.

Sidenote:

Faked email like this could be flagged as a potential phishing attack by email clients, based on tell-tale signs like:

  • it purported to be an internal mail, but was routed via a series of external mail servers
  • it contained a link with anchor text containing a URL to a different domain.

Unfortunately, these signals did not automatically flag the message. Perhaps this is something that email clients should look for.

Critical System Security

We are classifying certain SaaS external IT systems as “critical”. These include systems that

  1. can affect communication between browsers, websites and pagefair
  2. may contain sensitive information now or in the future or
  3. is used for staff communication.

We are reviewing all these critical systems for support of (a) MFA (b) the existence of an audit log including IP addresses and user agents of connected clients and (c) the ability to terminate any active logged in sessions. We will consider changing vendor if any system fails to support these security features.

Dynamic Javascript Authentication

Web is full of 3rd party javascript served from CDNs, which are on the periphery of their vendors’ architectures.  The reach of a CDN hijacking attack can be huge, and there are many targets to choose from. Of the 10 or 30 javascripts from different vendors on a typical web page, how many would be secure against a well-researched and targeted hijacking attack?

The core issue is that, unlike applications on a modern operating system, there is no good way to authenticate the origin of javascript files loaded in the browser. Sub-Resource-Integrity (SRI) is a great initiative, but it is not practical for javascript that needs to be updated on the fly (for example to address a security vulnerability, performance issue or feature request). This is going to limit its widespread adoption.

For the modern web to be safe, publishers need a way to authenticate that javascript loaded on their sites is really from the vendors they have relationships with. There needs to be a way to digitally sign a javascript file with a vendor certificate, and validate that on the fly in the web page using the vendor’s public key.  We are going to investigate the feasibility of this as an open source javascript library, and would welcome a discussion with other vendors who would like to adopt it also.

Update 8 – 15:40 GMT November 4, 2015

More antivirus products have updated their signatures to remove the trojan involved in this attack. By combining statistics from VirusTotal and Statista, 92.5% of people using antivirus should have immunity to this trojan.

Update 7 – 18:02 GMT November 3, 2015

Any website visitors who installed the malicious program are now safe from being controlled by the hacker.

The creator of NanoCore, the tool on which the malware was based, has revealed on Twitter that they have terminated the account responsible for the attack. When we reached out to verify the source of these tweets he also stated that:

“The attacker used a heavily modified version of NanoCore (which bypassed our restrictions which prevent this, among other things). We are very sorry that out product was used in such manor and have banned the person responsible… As a result of the attacker being banned, everybody who was affected by this are safe, the file will only connect to his licence, which has been banned. However the file will still be running. Over time Anti-Viruses will detect it and remove it. Malwarebytes does a good job at this.”

Update 6 – 21:38 GMT November 2, 2015

We are provisionally estimating that a relatively low percentage of 2.3% of visitors to the 501 affected publishers during the 83 minutes of the attack would have been placed at risk of infection (although a greater number of visitors would have seen the alert dialog). We hope to refine this estimate during the next 24 hours, but we want to provide background to this calculation now in case it is useful.

Key figures in this calculation are as follows:

  • 33 minutes into the attack we updated DNS to bypass the CDN, and 50 minutes later we cut off the attack entirely. Factoring in DNS propagation, 70% of requests during this period would have been routed to the CDN.
  • According to statistics in our CDN, only 25% of requests to hackers’ IP addresses succeeded, with the majority returning 504 server errors. On each page load, up to two attempts may have been made to fire the alert dialog. The probability of one of these succeeding was 43% (100% – 75% x 75%).
  • Assuming 100% of users shown the modal alert dialog would click “ok”, and given that the hackers were serving the trojan from the same IP addresses that were returning server errors in 25% of cases, the executable file would only succeed in downloading 25% of the time, .
  • The trojan targets Windows only, which accounted for 45% of internet traffic in the US in October, according to statscounter.
  • According to Microsoft, 24% of PCs worldwide were unprotected by virus scanners in 2014.
  • Of the remaining 76%, we calculate that 59% of the marketshare of virus scanners did not detect the trojan when the attack happened (this calculation is based upon virustotal data on what scanners identified the trojan earlier today and opswat data on anti-virus market share).

Multiplying out these percentages (70% x 43% x 25% x 45% x 24% + 70% x 43% x 25% x 45% x 76% x 59%) leads us to an estimate that 2.3% of visitors during the attack period were put at risk.

Once exposed, visitors would also have to choose to run the downloaded executable, while also dismissing security warnings that would have been displayed by Windows about the unverified source of the executable.

Next steps: Our CDN is providing us with a detailed log of all requests against our account during this period. Tomorrow we will perform a more detailed analysis of this log to establish if it corroborates the above mathematics.

Update 5 – 19:36 GMT November 2, 2015

If you believe that you have downloaded the malicious file onto your Windows operating system or you are a publisher that is being contacted by your website visitors about this problem, this update is for you.

We have received reports that the malware in question causes unexpected behaviors in certain Microsoft products such as Word, Excel, and Outlook.  If you notice unexpected behavior please use one of the anti-virus programs listed here to scan your system.  Choose one of the programs beside the red text (the top-half of the list).  This should fix the problem.

If you are in customer support, you can use the following wording:

Thank you for contacting us.  To fix this problem we recommend that you scan your system using one of these anti-virus programs, which should detect and remove the malicious file.  Choose one of the programs beside the red text (the top-half of the list).

Update 4 – 17:58 GMT November 2, 2015

F-Secure has posted further information of the trojan involved in this incident, including screenshots of what affected users would have seen. They identify the executable as the Nanocore trojan. On Twitter, security researcher @bartblaze has analyzed the file and believes that it is primarily intended for Bitcoin mining (correction: @bartblaze has clarified that the trojan disables other bitcoin miners during installation).

Update 3 – 16:41 GMT November 2, 2015

The executable connected to this attack is only capable of affecting users of Microsoft Windows. According to VirusTotal, the executable file in this attack would have been correctly identified by many anti-virus products, including Avast, Kaspersky, McAfee, F-Secure, Sophos, AVware, BitDefender and others.

Update 2 – 14:30 GMT November 2, 2015

We have established that only a fraction of the publishers we work with were affected by this attack. The attack affected only versions of the PageFair analytics tag that used the CDN in question. We have established that 501 publishers were affected during the 83 minute period. Most of these publishers are small, with 60% having less than one million page views per month, and 90% having less than ten million page views per month.

Next steps: Regardless of size, every publisher is important to us. We are working to estimate the degree to which visitors to all these websites would have been affected by the attack. We will update again shortly.

Update 1 – 21:30 GMT November 1, 2015

Core Facts

If you are a publisher using our free analytics service, you have good reason to be very angry and disappointed with us right now. For 83 minutes last night, the PageFair analytics service was compromised by hackers, who succeeded in getting malicious javascript to execute on websites via our service, which prompted some visitors to these websites to download an executable file. I am very sorry that this occurred and would like to assure you that it is no longer happening.

The attack was sophisticated and specifically targeted against PageFair, but it is unacceptable that the hackers could gain access to any of our systems. We identified the breach immediately, but it still took over 80 minutes to fully shut it down.  During this time, visitors to websites owned by the publishers who have placed their trust in us were targeted by these hackers.

The damage was mitigated by our standard security practices, but the attackers still gained access.  I want to take some time here to describe exactly what happened, how it may have affected some of your visitors, and what we are doing to prevent this from ever happening again.

We will update this post as we establish more facts.

WHAT YOU NEED TO KNOW

At 23:52 GMT last night (October 31, 2015) hackers succeeded in executing a spearphishing attack gaining access to a key email account.  The attackers then immediately performed a password reset to hijack PageFair’s account on a Content Distribution Network (CDN) service that we use to serve our analytics javascript tag. They modified the CDN settings so that instead of serving PageFair’s javascript, it served malicious javascript. This intentionally harmful javascript prompted visitors to install a fake Adobe Flash update, which appears to be a botnet trojan that targets Windows (more information on it is now available here). Although many virus scanners will have prevented this file from executing, others may not have been able to correctly detect it.

We noticed the security breach within 5 minutes, but it took until 01:15 (83 minutes) to fully rectify the situation. After this time visitors were no longer affected.

If you had the free PageFair Analytics code installed on your website yesterday, it is possible that some visitors to your website will have downloaded the malicious executable file. We are directly notifying every publisher who had our code deployed during this time.  If we do not reach out to you directly, it means that you were not affected.

WHO WAS AFFECTED?

The malware distributed by the malicious javascript is targeted only at Windows users, and is detected by many anti-virus programs. In addition, not all Windows users accessing your site during the affected period of 83 minutes will have been affected.  Due to caching rules, only visitors who had not been active on your site in the previous 120 minutes would have connected to the CDN.  Also, 33 minutes after the attack started we reconfigured our DNS settings to bypass the CDN entirely. This change began propagating immediately (with a TTL of 60 minutes), and would have prevented many users from ever connecting to the CDN during the attack period.  Finally, at 01:15 GMT, we deleted the CDN “pull zones” in our account, which immediately ended the attack. From that point forward, users were no longer affected.

WHAT WAS NOT AFFECTED

There is no evidence or reason to believe that any core pagefair servers or databases were compromised. No publisher account information, passwords or personal information has been leaked.

WHAT NEXT

  • For today, our priority has been to ensure that all systems are fully secure and that all company-wide passwords are reset.  This has been done.
  • Tomorrow we will audit the level of access to company documents that the hackers may have gained.  We do not store any Personally Identifiable Information in any system, but we will advise partners if we have reason to believe any sensitive documents may have been accessed.
  • We will analyze which security practices failed and which could be strengthened and adopted to prevent something like this from occurring in future.
  • We will continue to post mortem this for the remainder of the week, and will regularly update this post with our findings.

Thanks to our customers who were patient with us during this issue, The Media Trust Company, who worked hard to reach us during the issue, and MaxCDN for being available in real time to help lock the hackers out of our account.  We will have more updates tomorrow.

Please ask us any questions in the comments section below or feel free to reach out to us at [email protected]  We will respond to every single email and query that comes our way.  We will also be updating our Twitter account as we update this post.

  • bvvnbn

    I saw this coming.

  • Benjamin Lim

    Use Subresource Integrity to protect yourself from these types of breaches.

    Links:
    http://security.stackexchange.com/q/74424/53583
    http://limbenjamin.com/articles/verifying-js-integrity.html

    • Eric Lawrence

      That only works if the pagefair scripts never change, which then makes them a sitting duck against the ad-blockers they’re trying to circumvent.

      In a grenade fight between ad-blockers and ad-blocker-blockers, users are just collateral damage.

  • Thanks for the update guys. Just one question though: was at any moment 2FA turned on for your MaxCDN account?

    • PageFair

      Hi Mauricio,

      2FA was active on the relevant email account, but I’m sorry to say that we did not activate it on the MaxCDN account until after the breach. As with all systems we use, access was protected by a very strong, unique and securely stored password that was used only for this account, but this was irrelevant as soon as they gained the ability to receive the relevant password reset mail.

      We have reviewed all other 3rd party systems we have in use to: (a) activate 2FA wherever it is possible, and (b) reset passwords for good measure. We will also review where password reset mails are routed. We are also looking into whether it is possible to authenticate CDN delivered code (SRI is problematic since it would prevent *us* from being able to update the code).

      Sean

  • mikegale

    I’d like to check whether there is any chance that I was exposed. Is there a list of the domain impacted to check against.

  • Kevin O’Brien

    HAHAHA i find this funny , ads are a security risk how can people trust you they cant.