Just to be clear: 100 percent of all online traffic is non-human traffic. Humans use specialized tools called “computers” to browse the Internet. Computers are not humans.
Here’s How It Actually Works
A computer running browser software (the “client”) sends a request for information to a computer running server software (the “server”). The computer running the server software answers the request.
When this occurs, another piece of software designed to log (or simply count) the number of client requests and server answers increments a counter, and we (the industry) give that number a name so we can help ourselves, and other humans, interpret what it might mean. The name could be “unique visitors,” “impressions,” “page views” or something else entirely. These are just labels humans use to help describe completed client/server requests.
Why is 100 percent of all online traffic non-human? Because client/server interactions are electronic and exist only when computers are connected to each other on a network (such as the public Internet). Human beings, human eyes, human fingers and human brains are not involved in any part of the transaction – they never have been and they never will be.
Importantly, defrauding a human being who is trying to interpret these numbers using labels that humans can understand does not require any human interaction – it can all be done with bots.
Bots
A “bot” is an application that performs automated tasks. Sadly, bots generate more than half the traffic on the public Internet. This is sometimes called non-human traffic or NHT. It is important to note that there is absolutely no debate over the amount of NHT on the public Internet. None. The Association of National Advertisers believes that advertisers will lose $6.3 billion globally to bots in 2015.
Some bots are benign; some are malicious. In most cases, bots are a bit of both. For example, you might have a benign bot that aggregates pricing from various eCommerce sites and generates data to help consumers make informed decisions. That might be awesome for consumers, but it’s not great for the eCommerce sites being scraped.
There are good bots, like the ones most reputable search engines use to gather information about the web (they usually identify themselves). And there are bad bots, like those that carry out phishing scams to get personal data from consumers or bots that post spam links in comment sections for black hat SEO.
There are the really bad bots, such as the purpose-built ad fraud bots designed specifically to pick up cookies and drive fake ad impressions.
There are criminal bots, such as spyware that collects information about user activities, including passwords, credit card numbers and other valuable personal information or bots designed to carry out DDoS attacks.
Then there are super-unfortunate-you-wish-they-did-not-exist bots, such as the bots that fake social media accounts and ruin social networks, bots that fake tweets, bots that generate content and bots that game search engines and prevent your SEO from doing its job.
Even Good Bots Can Be Very Bad
If you’re paying for caching servers, image servers, video servers, etc., serving content to bots can get expensive, especially since even Google virtually guarantees 56 percent of the digital ad impressions you pay for are never seen.
Viewability and Fraud
Since more than half of all online traffic is non-human, advertisers are paying much higher CPMs and reaching far fewer people than they are being led to believe. This is simply fraud.
It Gets Worse
Remember, impressions are generated by client/server requests. In his article The Definition of an Ad Impression, Reid Tatoris, co-founder of Are You a Human, offers a trinity of ad fraud issues: (1) Broken ads or technical glitches, which account for 15 percent of unseen ads. The fraud is unintentional, but broken ads are a fact of life. (2) Bots, which we know can account for 50 to 60 percent of all traffic. (3) Straight-up fraud. Illegal non-human trafficking may account for 25 percent of all traffic.
According to Tatoris, “We start with the notion that only 15% of impressions ever have the possibility to be seen by a real person. Then, factor in that 54% of ads are not viewable, and you’re left with only 8% of impressions that have the opportunity to be seen by a real person. Let me clarify: that does not mean that 8% of impressions are seen. That means only 8% have the chance to be seen. That’s an unbelievable amount of waste in an industry where metrics are a major selling point.”
This will not stop until someone (the marketers, the government, the justice department) makes it stop because everyone – the ad networks, the traffic sellers, the bot creators, the publishers, the ad agencies, the trading desks, the DMPs, the SSPs, everyone except the marketers – is making money.
We Are Not Doomed – But It’s Tricky
The best sites attract both good and bad bots – lots of them. Amazon gets scraped every day by thousands of bots capturing product and pricing data on behalf of other retailers as well as by comparison-shopping engines. It’s the same with top-tier content sites. The better and more valuable the content, the more bots the site will attract.
This means that buying direct from premium publishers does not guarantee human users or increased viewability. It means you’re likely to run into even more bot traffic. Publishers should not shy away from this reality; it is a fact of life. Rather, they should use it to increase yield by understanding how to manage and price actual human traffic.
Possible Solutions
To work through the problem, I spoke with Are You a Human CEO Ben Trenda. Ben says, “Trying to find and block bots is somewhere between an incomplete solution and a fool’s errand. And yet that approach is all that’s been made available to the industry until now.” It’s easy to understand why – bots are good at impersonating humans, and they are ephemeral, lasting for only a few days. By the time you locate one, the hacker that built it has already improved it and moved it. It’s an iterative game of whac-a-mole, annoying and unwinnable.
The second problem is that real humans get blacklisted as collateral damage. Most botnets utilize malware that infects real users’ laptops. If the bot is identified while it’s on your device, you become blacklisted and can lose access to content or services you want to access.
According to Ben, Are You a Human takes a different approach: the company finds real humans and whitelists them. It’s an elegant solution that got my undivided attention. Ben explained in our conversation, “You wouldn’t design a border security program around trying to find people who aren’t allowed into your country. Instead, you would first issue passports to those who are allowed in. And then maybe do those other things, secondarily.”
Many believe that the ultimate solution will require separate pathways for automated traffic and “real” traffic. This solution will come with its own set of nontrivial problems.
In the meantime, non-human traffic, ad fraud and viewability are huge problems, and it’s time to work together to solve them.