behindthescenes

The Rules of Adblocking: How Block Lists Work

David Barton AdBlock Leave a Comment

This note is a layman’s guide to how the “block lists” that power adblocking work.

A block list contains tens of thousands of rules that govern how a website should be displayed. There are two types of rules. “Filter” rules define what should be blocked. “Exception” rules define what content should be displayed. The most popular block list, “EasyList” (see the live version here), contains tens of thousands of these filters and exceptions.

A filter can be brutally simple and generic, such as /images/ads/* 1, which might block any images being loaded from the “ads” directory on a website. Or the filter can be more specific, such as ##.top-banner-ad-container 2, which would remove the uppermost ad slot from The Guardian’s website.

Guardian

The screenshots below are taken from the online version of The Irish Independent newspaper. They show how a handful of rules can strip several ads from the page and dramatically change both the experience for the visitor and the revenue for the publisher.

Independent
The right-hand screenshot shows ABP’s display of the elements (in red) that it is removing from the page.3

Eyeo (the company that operates Adblock Plus) is deeply involved in supporting the EasyList community that contributes many of these rules. But although filters are drawn from downloaded block lists that are updated by the adblocking companies and a community of interest that works with them, a user can easily add their own rules too. For instance, one can hide the logo image on the Adblock Plus website simply by adding the line ###logo4 to the ABP custom filters. The result is shown in the screen show below (right hand side).

ABP Filtered

In Firefox, click on ABP and select “Filter Preferences” then “Custom filters” to add filters

Filters and exceptions can be combined in a single rule. This is useful if, for instance, the adblocker wants to remove ads from a specific source on a website but wants to allow other ads through, as in this example where a user was offended by suggestive American Apparel advertising on Daring Fireball. In this case, another user contributed and explained the solution.

SuperUser

While adblocking rules can be complex and subtle 5, the bulk of everyday adblocking is carried out by the simpler, generic filters6. These lines of text indiscriminately strip out millions of ads every day across the Web. Even if a website is not listed in a block list, it is likely that general filters that target ads on all pages will neatly remove that website’s ability to generate revenue.

Whereas “EasyList” is predominantly aimed at English-language websites, there are lists for other countries and regions, such as Germany and France, and for other targets. More aggressive lists target what their authors loosely call “annoyances”, such as social media buttons. These change the appearance – and loading speed – of websites. Other lists, such as the optional EasyPrivacy, are dedicated to protecting privacy or defeating malware. Some of these lists are added by default in independent alternatives to Adblock Plus.

Three ineffective countermeasures

PageFair provides publishers with the technology to serve their ads in a tamper-proof way that adblocking software is unable to circumvent. But publishers have attempted other ways to combat adblocking such as native advertising, domain name rotation, and adblock walls. Each of these three approaches is affected in different ways by this underlying system of filters and exceptions.

Many filters have been designed to block obvious advertising elements from being loaded from third-party advertising servers, and this has led some publishers to mistakenly believe that native advertising offers a way to defeat the adblockers. However, adblockers can easily remove or hide any element on a webpage, even if its nature as an ad is concealed as normal content. Rules can be crafted to target elements of a webpage based on size, position, or just about any aspect of HTML and CSS. The makers of Adblock Plus have long been confident about their ability to block native ads. To understand this, see Eyeo’s post from September 2014 titled “Native schmative … they’re still advertisements”. As PageFair has written elsewhere, there are several reasons to rethink relying on native advertising as a solution to adblocking.

Domain name rotation is another tactic to obfuscate the source of both native and regular advertising content, with some ad rotation networks. Instead of serving ads from a stable third-party server called, e.g., adserver.com, the advertiser uses an ever-changing rotation of domain names so that ads could come from, e.g., nv3k9ynphl8qd83.com one minute and 91ir3c4cw9z9hn8.net the next. This sounds promising when one considers that lists such as EasyList are maintained by a community of mostly unpaid ad dodgers. In theory, domains could – especially as the cost of registering some domains is now relatively cheap – be rotated so quickly and intensively as to overwhelm list authors and prevent list servers from keeping pace with the necessary rule subscription updates.

Unfortunately for both native advertising enthusiasts and networks that rely on the one-trick pony of domain name rotation, adblockers have what amount to nuclear options when they feel that an advertiser or publisher has gone to extreme lengths to evade their rules. Rather than creating rules to examine every element and determine whether it should be blocked or not, adblockers can – as EasyList did last year to Yavli – decide to block all third-party content, thereby making it impossible for affected websites to run any scripts, including analytics and tracking, or load anything at all from an external site.7

So what about adblock walls? The IAB is currently recommending denial of access to content and publishers in both the US and across Europe recently rushed to barricade their content behind adblock walls. But adblockers have been efficiently dealing with these for some time. EasyList has a longstanding policy that an anti-adblock measure can be circumvented if it “limits website functionality or causes significant disruption to browsing” and many adblock walls can be defeated with just a couple of extra filters. Although in some cases the adblockers are successfully encouraged not to automatically add these rules, other adblocking software is not always so reluctant to get involved.

Adblock walls mainly depend on detecting the presence of adblocking software by planting “bait” elements on the page that trigger adblocking filters. In response to this tactic, adblocking software will usually try to evade detection by whitelisting bait elements, leading to a game of cat and mouse. As a result, the only way to reliably detect adblocking software that is actively evading detection is to check if the actual ads actually loaded. For most sites this is problematic because ads usually reside in iframes that the browser security model protects from direct examination. In any case, this approach also requires waiting until all ads have fully loaded before displaying the page – an unacceptable delay on the modern Internet that will probably lead to most users disappearing off into the horizon.

(There is also a strong commercial rationale against Adblock walls for most sites: First, forcing users to turn off their blocker is probably going to lead to their swift exit to somewhere else. Only publishers with exclusive content that is unavailable elsewhere on the web can attempt the wall strategy. If a site cannot sustain a paywall, it is unlikely that it can sustain an adblock wall. Second, the adblocking audience can be advertised to, and represents an invaluable marketing opportunity.)

How PageFair Solves the Problem

PageFair works with major publishers to provide a tamper-proof solution that serves ads to adblockers. It uses a blend of technologies, one of which communicates using a channel to the browser that blocking extensions are unable to monitor or affect. It respects user privacy, limits data consumption, and enforces security by rejecting javascript advertising payloads.

To learn more, read about PageFair’s approach to advertising beyond blocking.

 

Notes

 

  1. This rule simply targets the directory images and subdirectory ads – often used to store advertising content – with the wildcard asterisk character used to tell the adblock software not to show any file from that source. EasyList includes dozens of variations on this rule, e.g. a variety of directories are targeted with /*$image, which would block any image filetype from being loaded.
  2. ## is an instruction to select an element in the HTML code used to render a webpage, in this case an element with the class top-banner-ad-container
  3. To view this window of blocked elements in Firefox click on the ABP icon (if the adblocker is installed) and select “Open blockable content” to view rules.
  4. ## selects an element on the page and the logo element is a div, or division, in the HTML code behind the Adblock Plus page.
  5. Regular expressions can be used to target elements with remarkable accuracy and control, although these are discouraged, as they slow down page rendering because the browser needs to do more work to figure out what to block.
  6. The filter examples above and in the main image behind the title of the post are taken from the current (as of March 21, 2016) rules in EasyList. Adblock Plus provides a helpful guide to writing EasyList rules, if you want to examine any rules in more detail.
  7. It is worth mentioning that Easylist will normally whitelist any javascript that is necessary for the core functioning of the website. This still means that analytics and audience measurement are effectively nuked, and that the website cannot change its interactive features without clearing it with EasyList first. An irksome dialog for any publisher.