Detect Adblock: Our Secret Sauce

The PageFair Team Adblocking, Uncategorized 32 Comments

How do we do it?

We’re often asked how our adblock detection script works its magic: how do we detect that someone is blocking ads? Most people expect us to guard this secret closely, but the truth is we use an approach that’s widely discussed online. We observe what happens when a web page loads and detect the effects of adblocking plugins.

Understand Ad Blocking

In order to know what effects to look out for we need to understand how adblock stops ads from loading. The first technique used to block ads is to intercept requests from the browser to particular domains or for particular files. Most publishers use hosted ad servers that operate from well known domains; for example Google’s display ads are served from doubleclick.net. The adblock community maintains ‘filter lists’ of these domains that are updated regularly with the latest ad server domains. Filter lists also name particular files for which requests should be blocked regardless of domain; for example any javascript file called ads.js.

The second technique used to block ads is to hide ad-related page elements based on css rules. Publishers carefully design their web pages with space for both content and advertising, but when ads are blocked this could leave large, empty areas on screen. The adblock community’s filter lists specify page elements that should be hidden, for example any element with the ID ‘leaderboard-ad’. Page elements that match standard ad dimensions are also hidden. By hiding these page elements, adblock ensures that the space they would have taken up can be re-used by other parts of the page, such as the main page text. This has the bonus side-effect of also hiding any ads that slip by the first blocking technique.

Choose Your Bait Carefully

With these techniques in mind we insert bait elements into the page that adblock will attempt to block; including a javascript file, an image and an iframe. We then carefully observe what happens when a page loads. onLoad and onError events tell us if they’re successfully retrieved or if requests have been blocked. Their css style tells us if they are visible or have been hidden. We have run these tests billions of times, and have now refined them to the point that we can accurately detect when a user is blocking ads using adblock.

The Devil is in the Detail

As always, there’s more to this than meets the eye. Anyone who’s tried their hand at web development will be familiar with the frustration of cross-browser (in)compatibility and the challenge of staying current with a shifting landscape of browser and plugin technologies. Not to mention the challenge of  building a scalable server infrastructure that can handle vast quantities of analytics traffic in real-time.  We won’t bore you with complaints here though; hopefully you’ve now got enough information to understand what’s going on in the background when you sign up to use our free adblock measurement service.

  • PhasmaFelis

    So has anybody ever tried to address the root cause of adblocker use, i.e. ads are really fucking annoying? I don’t like ads in general, nobody does, but that alone wouldn’t be enough to make me bother to install and maintain AdBlock. What does it is strobing “YOU MAY HAVE ALREADY WON” and animated “one weird trick” scams and softcore porn. This shit is *everywhere*, even on allegedly respectable news sites.

    If you want me to turn off AdBlock, you need to insist on reasonable, non-offensive, non-animated ads. If your ad provider doesn’t do that, get a better one, or lean on yours until they do. If this industry spent one-tenth as much energy pushing ad services for better quality standards as they do wringing their hands about ad blocking, shit would happen.

    • I whole-heartedly agree.

    • oGMo

      Yeah seriously. And if your site refuses to load or whatever due to adblock, it wasn’t worth reading anyway. Your content is not that special. I’ll just go elsewhere.

      (Also the techniques discussed in the article are pretty much nothing special and exactly what I would have expected. Expect the next wave of adblockers to alter reporting for elements they block if this becomes necessary.)

      • Rick Burgess

        How would you suggest paying for the content? are paywalls less annoying?

        • PhasmaFelis

          I would, and did, suggest less offensive ads.

          • Rick Burgess

            Lets be honest though, ad blockers originally came about to block pop-ups because they were horrible. pop-ups are dead on all but adult sites for the most part and what you have now are small text or image ads thats really aren’t a big deal.

            You could also argue that even if they made ads that were less “offensive” you would never see them because of your blocker :p

          • Tijdelijk1

            One of the new sites I sometimes read has replaced pop-ups with Javascirpt adverts that spontaneously starts clips, waltz trough my screen – obscuring the news itself, etc etc.

        • I suggest a simple and universal “tip” service. You would dedicate a certain amount, such as $5 per month, to be used for tips, and every time you click “tip” (a universal and recognizable UI) it’s recorded. Your $5 budget is then split amongst all the tips clicked that month.

          Obviously you could do a certain micro-payment amount as well if required. Or a simple pay-switch (rather than a wall) — the key is to make it a seamless and universal experience, as easy and ubiquitous as this Disqus form, so you’re not inconveniencing users at all.

          Add a “pay wall” and people will not climb over it. Hell no. But make it easy and fun to pay for content, and people will embrace it and feel good about it.

          This is a UX problem, not an economic problem.

          • Toranaga

            marketing is the tax you pay for not being interesting. Pony up!

          • Rick Burgess

            I agree that solution would be better in theory, if people were to actually use it. I fear that the majority of users are used to what they perceive as free content online and will simply not pay if they don’t have to.

            We have the paypal donate type functionality which has been around for years and has a standard (although not nice) UI but I would guess (as i have no data to back it up) that the actual donation rate is pretty low.

            The only way I can see a tipping type service working is if there is some benefit to the user for doing so, much like subscriptions on twitch.tv for example.

          • Snowangelic

            This is flattr : http://flattr.com/

          • Debashisa Jena

            $5 a month! are you mad? I thought $5 for year

          • This has been tested over and over, the cruel reality is that basically nobody pays. Paywalls have failed, except for some english and chinese (mass-languages) exceptions. 🙁

        • oGMo

          First you need actual content. This means not a link chain to some other site or some blathering commentary piece. Hint: If I can skip your site and find the same or very similar content in the next link down the chain, your content is worthless.

          Yes, this means doing real work. This means having something to actually say, some research or something of value you’ve actually done. Then a paywall isn’t even necessary: I’ll subscribe to you even if I block your ads. For instance, I subscribe to places like di.fm and Destructoid which provide real, actual content I can’t find elsewhere.

          If you can’t be bothered to do the work and you just have drivel that no one is even willing to see an ad to read, your “content” doesn’t deserve monetary support. However, if you have quality content, you will get support.

          • Yuri

            That is not borne out by statistics. Sites with high quality journalism have not been very successful with digital subscribership.

          • Hon

            Rick is an employee of pagefair, so he has a vested interest in fucking people with the advertising dick.

      • Pvblivs

        The way I see it, it’s already not necessary. If you already think the content is worthless, then you won’t visit the site and it doesn’t matter if they use detection methods. Ah, but that’s not the reality, is it? You do find value in visiting these sites. You just want to make sure that the content providers make no money for their efforts.

        That’s the reality. If the ads on a site are too obnoxious, I won’t go back to that site. But if a site is worthwhile enough for me to want to read it, I can hardly object that the provider of the content wants to eat.

    • pagefair

      We agree that intrusive ads are bad! In fact we highlighted this issue in a previous blog post ‘Dealing With Adblock: 5 Options That Don’t Work’. The problem is that existing alternatives are bad for both publishers (less revenue) and web users (less access to information). The ad industry is gigantic and sadly change is slow to happen.

    • Justizin

      Having worked at (and left in disgust) an ad-driven company, the answer is simple and clear: The most annoying ads yield the best click-through rates, but I can’t possibly believe they yield the best consumers. Companies like Google who once tried to challenge the shittiest ad strategies are now serving them up, and many companies relying on those ads have offices full of people using AdBlock “because our fucking site doesn’t load otherwise”. I think we should organize to boycott sites with the most visually distracting and CPU intensive ads (FLASH).

      What would astound you, BTW, is that for direct sales, you can barely sign an agreement anymore without a large % of video ads, so sites with no substantial video content have to invent an excuse to have video content and TRY THEIR BEST to distract their users away from the actual site, to watch these 1:00 ads on top of :15 video clips, sometimes bought wholesale.

      It’s abhorrent and it’s a fucking ponzi scheme, but it pays. :/

    • This is already reality for 5+ years, but people like you don’t turn off Adblock because then don’t know and don’t even try, right !? Some major ad networks don’t show low-quality ads, moving ads, flash ads etc. by default. But people still block ALL websites, they simply don’t select wisely. The problem ARE the big sites that still support huge fullscreen flash banners, “loose weight” bullshit etc. and literally force the users to use an adblocker, which then harms also pages that are doing everything right. Adblockers destroy masses of jobs, killing entire industries. This is a serious problem with huge economical dimensions.

    • This isn’t about a lack of innovation, it’s about (like all things) the flow of money.

      There is only so much high quality brand advertising which leaves publishers to use second-tier ad networks to “fill” their inventory so they can make something on the rest of the impressions. And guess who buys all these cheap impressions: all the shady advertisers who want traffic for their scam sites and products and design horrible popup, expanding, giant autoplay ads. A single small network can run through hundreds of thousands of dollars in monthly advertising from these kind of buyers.

      The best solution to all this would be regulation on both ad providers and what can be marketed. Even if it’s just regional (since this industry is global) it can still do a lot. Definitely more than the somewhat powerless industry groups can do today.

  • Pingback: How We Detect Adblock | Rocketboom()

  • kjs3atl

    An even more important reason to run an ad blocker is that the ad networks have become very effective malware distribution mechanisms. I have huge numbers of events like the following (edited slightly for readability):


    GET /7f01baa99716452bda5bba0572c58be9/afr-zone.php HTTP/1.1::
    ~~Accept: text/html, application/xhtml+xml, */*::~~
    Referer: http://www.huffingtonpost.co.uk/_uac/adpage.html::
    ~~Accept-Language: en-US:: ~~
    User-Agent: Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64;Trident/5.0)::
    ~~Accept-Encoding: gzip, deflate::~~
    Host: delivery.globalcdnnode.com::
    ~~Connection: Keep-Alive::~~::~~

    See that referer? HuffPo ad launch page. That host? That’s not a CDN, it’s a malware farm registered to some guy out of RU. That GET? It eventually leads to a Darkleech exploit toolkit.

    We’ve seen these coming from all sorts of legitimate sites and ad distribution networks. We’ve had to block a couple of big ad networks in our web hygiene proxies, and they’ll likely stay blocked until they clean up their content, and that’s probably going to take a lot of big sites to make them.

    Haven’t had a single complaint from the users, tho.

  • ende

    We’ve tried the same technique to detect ad blocker and then show the people an unobtrusive “Please switch off the ad blocker, that’s how we are paid”-banner. It took 3 days for the ad blockers to adjust the rules to not block the bait. It seems the only way to have even such an unobtrusive banner displayed is pay ABP for the exception.

  • flamer96845312

    Good job linking to jQuery’s API when talking about JavaScript events.
    Also, if you start complaining about compatibility issues when talking about a few lines of JS, you must be quite the pro.
    Don’t write “technical” posts to attract clients when they’re only a display of how skillless you are.

  • Locktrap

    My adblocking solution works without any anti-adblocking metrics or scripts figuring out that I’m even blocking ads (as far as I know anyway) except for just one website. That website blocked the content I was trying to access after it determined that it was blocking their ads, but I easily found a way to reach my target content on that website while avoiding the script through just a bit of work.

    Other than that, I have not had a single glitch or hiccup in any of my adblocking.

    I’m using IE9.

  • Pingback: Forget Fiverr Buy $1 Gigs()

  • Martin Steinhauer

    found this today. http://www.bootstrax.com. seems to be a good fix for the ad blocker problem.
    cheers

  • Peytonsdad

    If you don’t like ads stay off the Internet. Or you could offer to pay for the resources you are using by clicking on a donation button now and again.

  • Hon

    Pagefair is an ad-blocking circumvention tool that publishers can use to track readers who’ve taken technological countermeasures to protect their privacy. The company has sold its service to many publishers — including the Economist — by deploying moral arguments about the evils of ad-blocking.

    But on Hallowe’en, for about 90 minutes, Pagefair was hacked and served malware to Economist readers who were using ad-blockers on their computers. Though many ad-brokers have been tricked into serving malware before, Pagefair’s technology to override users’ security measures made it an ideal vector for infecting PCs.

    The Economist was just one of an estimated 500 publishers who served malware during the attack. The attackers tricked PCs into installing Nanocore, a legitimate remote-access program used to allow administrators to control computers over the net. Computers compromised by Nanocore were completely open to attackers; all their files and processes available to them (as well as peripheral sensors like cameras and microphones).

  • Rune Søsted

    Anyone making adds should go flush themselves out the toilet. I dont want adds, I NEVER click adds and I hate distractions, crap that eat my already poor connetion when on missions or exersices.

    If people could just respect I dont want so see adds, and then they can demand money for their website, but dont expect me to pay unless its really great. I hate Adds, everyone that make adds and YOU for helping the loathsome add makers annoy me more.

    Eventually I will simply stop using sites where I cant block the adds, and thats effectively lost custemers for the ACCUAL product the site may try to sell.

  • Венци Конов

    I recommend to anyone using AdBlock that you go ahead and subscribe to some of the popular anti-adblock blocking lists. Block the adblock blocker scripts – I wonder how deep we can get with that.