Advertising isn't as effective as you might think; a review of the scholarly literature

The Domain Of The Critique

If you follow the major tech publications, you would think that the advertising of today is a powerful new weapon that desperately needs to be reigned in.

NYT technology reporter Natasha Singer laid it out, “What is at stake here isn’t privacy, the right not to be observed. It’s how companies can use our data to invisibly shunt us in directions that may benefit them more than us.” Matthew Yglesias in Vox likened Facebook to alcoholics and gambling addicts, “Not only is the product bad, but the company is in a deep state of denial about it.” Tech investor and critic Roger McNamee said,

The combination of an advertising model with 2.1 billion personalized Truman Shows on the ubiquitous smartphone is wildly more engaging than any previous platform…and the ads have unprecedented effectiveness.

But if you read perspectives from those in advertising and marketing, then you are sure to get a different take. They lament about the difficulty in identifying users across sites and devices. They are concerned about fraudulent schemes. They worry about ad blockers and ad blindness. A group of them has even brought a case against Facebook, claiming the site is inflating audience numbers by some 400 percent.

Technology hasn’t rid the industry of the fundamental uncertainties. Sometimes people love advertising and share it. Sometimes they despise ads. But people aren’t mindlessly consuming these messages and then going out to buy more stuff. The conversation over advertising supported media desperately needs to be rooted in empirical studies and not simple criticism.

First, a note about the goal of this piece.

In the world of econometrics, there is an idea known as the domain of the model. Individual characteristics only vary so much, so it is wholly inappropriate to make predictions out of this range. Similarly, none of the research laid out below denies that Facebook and Google and other social media sites can have an impact, but commentators should be careful in making claims that go beyond the domain of influence. This piece is an attempt to explore the upper bounds of that impact by piecing together empirical studies on advertising and technology use. By the end, you’ll surely be tired of a phrase that continually pops up: significant but small. Yet, that is the conclusion that should be drawn.

The post that follows is broken down into five sections. First, the traditional perspectives on advertising are laid out. From here, concerns related to individual preferences and measurement problems are detailed. In the third section, the most important part, studies on the effectiveness of advertising and other persuasive technologies are reviewed. Finally, this post ends with a few perspectives of advertisers and then a brief conclusion.

Since this is a work in progress, I would greatly appreciate any additional studies as well as any comments. You can find my contact information here.

Theoretical Perspectives On Advertising

Concerns about advertising are hardly new. Throughout the early twentieth century, sociologists and economists worried that advertising might be too influential. As Kyle Bagwell explained in the single best overview of the economics of advertising, there are three broad visions about what advertising does. The first of these, the persuasive view, views advertising as being able to alter consumers’ tastes, thus creating spurious product differentiation and brand loyalty. As a consequence, the demand for a firm’s product becomes more inelastic, and so advertising results in higher prices. Bagwell explains,

The persuasive approach therefore suggests that advertising can have important anti-competitive effects, as it has no “real” value to consumers, but rather induces artificial product differentiation and results in concentrated markets characterized by high prices and profits.

The persuasive view can be compared to the informative view of advertising as well as the complementary view. The informative view of advertising starts from the position that markets are often inefficient. Consumers don’t know about products and therefore don’t have any opinions about them. Ads thus serve as a response to this information asymmetry, giving consumers information at low cost. The complementary view of advertising doesn’t see advertising as having a persuasive component even though there could be informative parts. Rather, adverts serve to complement the qualities of the product or service and signal a company’s investment in quality.

What is new is the melding of traditional advertising critiques with the study of computers as persuasive technologies, also known as captology, to create a new line of criticism for computer mediated communications. Take for example, professor Zeynep Tufekci’s view of advertising, which can be found in an op-ed in the New York Times,

Ad-based businesses distort our online interactions. People flock to Internet platforms because they help us connect with one another or the world’s bounty of information — a crucial, valuable function. Yet ad-based financing means that the companies have an interest in manipulating our attention on behalf of advertisers, instead of letting us connect as we wish.

Ethan Zuckerman, too, shares this kind of view as he wrote in The Atlantic,

I have come to believe that advertising is the original sin of the web. The fallen state of our Internet is a direct, if unintentional, consequence of choosing advertising as the default model to support online content and services.

The arguments against persuasive advertising have deep moral foundation. As psychologist Michael Billig explained it, everyone worries that “our attention is being seduced by irrelevancies and that there is something inherently counterfeit about these skills of promoting style over substance.” This can be best seen in the often-repeated statement, “if you aren’t paying, then you’re the product.” It is a pithy statement capturing what people find so objectionable about advertising and social media, namely that they are being objectified. Advertising seems to undercut our dignity because it uncuts our ability to choose. But how much does it really change our choices?

Individual Preferences And The Problems Of Measurement

While pervasive, the persuasive approach oftentimes doesn’t align with scholarship.

Indeed, if it were the case that advertising changed consumers’ tastes, which had the end effect of increasing prices, then advertising bans should decrease prices. Yet, the opposite is often true. When Rhode Island lifted its liquor advertising ban, prices went down. Austria, for example, is the only country in the European Union to have an ad tax. When the country moved to harmonize various regional ad taxes in 2000, researchers were able to calculate that the 5 percent tax had raised consumer prices 0.25 percentage points. Advertising restrictions have also been shown to increase prices for breakfast cereal, eyeglasses, and drugs.

In a recent Yale Law Review article, Ramsi Woodcock reflected a common concern, writing that advertising “induce[s] consumers to buy products that they do not in fact prefer.” But lab experiments testing this theory have hardly been conclusive. For an experiment involving a fictitious cell phone company, it seems that ads increased consumers willingness to pay for the product. But on the other hand, the effect was exactly the opposite when people were shown online interstitial ads. John Kwoka’s work on the auto industry found that advertising did have a short term effect on preferences, but these impacts were eventually overtaken when consumers had a better sense of the quality of a new product. Showing advertising to a person can can either lower or raise their willingness to pay for a product, depending on the context.

Still, it is hard to square the widespread belief that ads seduce people with overwhelming evidence that people distrust ads. A survey of 1200 consumers by Marketing Sherpa found that only 39 percent of people trusted banner ads. Social media ads did slightly better and were trusted by only 43 percent of respondents. A report from Ipsos found that 69 percent of people distrust advertisements. A study commissioned by the American Association of Advertising Agencies discovered that just 4 percent of consumers believe advertisers and marketers practice integrity. Overall, the data suggest that people are highly skeptical of ads.

A major contributor to the confusion over advertising comes from the endemic measurement problems. Advertisers aim to connect with people that are most likely to respond to their message, which means there is a bias in the response group that inflates its impact. Here is how one paper described the issue:

Suppose an observational study of [advertising effectiveness] for the Orbitz campaign compares those who saw the ad with those who did not see the ad. Then the exposed group will contain only users who have recently searched for airfares, while the unexposed group will contain a number of users who have not. If the former are more likely to purchase, even in the absence of advertising, then the observational study will overestimate [advertising effectiveness], mistaking correlation for causation.

Consumer reactions to advertising vary widely. Ads might communicate information to one person, trigger a positive emotional response in another, or make someone else skeptical of the product. And for those that were searching for a good or service because they already intended to buy it, advertising might just be a simple reminder. In this scenario, ads have little persuasive impact but will seem otherwise in the data.

How consumer reactions translate into action is yet another area of uncertainty. Individual-level sales are extremely volatile, compounding technical evaluation. Randall A. Lewis and Justin M. Rao put this in perspective in their paper aptly titled, “The Unfavorable Economics of Measuring the Returns to Advertising,” saying bluntly that “most advertisers do not, and indeed some cannot, know the effectiveness of their advertising spend.” They lay out the fundamental problem that, “even if each 30-second television commercial could be randomized at the individual level, it is nearly impossible for a firm to be large enough to afford the ad, but small enough to reliably detect meaningful differences in ROI.” All of the uncertainty and volatility makes measurement a nearly impossible task for everyone in the space.

Uncertainty is reflected in the variability of ad spending. As Lewis and Rao went one to explain, the endemic issues of measurement explain why “similar firms (size, margins, product mix, etc.) operating in the same market often differ in their advertising expenditure by up to an order of magnitude.” It doesn’t make much sense why Etrade would spend 12.6 percent of its revenue on advertising while its competitor TD Ameritrade spent only 1.8 percent, unless neither really understood how effective advertising was. Similarly, why is Fiat-Chrysler putting 1.6 percent of its revenue to ads while Toyota spends just a third of a percent?

Brand reputation could be an important driver of this intraindustry difference. In a study of the hotel industry, Brett Hollenbeck, Sridhar Moorthy, and Davide Proserpio found that online ratings have a causal demand-side effect on ad spending. Hotels with higher ratings on TripAdvisor spend less on advertising than hotels with lower ratings. Summarizing the trends, the authors explained that “the relationship is stronger for independent hotels than for chains, and stronger in less differentiated markets than in more differentiated markets. The former suggests that a strong brand name continues to provide some immunity to reviews and the latter that the advertising response is stronger when ratings are more likely to be pivotal.”

Indeed, the persuasive view of hardly explains why total ad spending as a percentage of GDP was much higher for English-speaking countries like New Zealand (1.33), Australia (1.22), the United Kingdom (1.20) and United States (1.31) as compared to countries in Northern Europe like Sweden (0.77), Norway (0.74), Denmark (0.81), and Finland (0.86). There is at least some cultural or institutional or company level component of advertising that hasn’t yet been discovered.

The Effectiveness Of Advertising And Other Persuasive Technologies

In spite of the problems in measurement, the widespread use of data collection by social media and their growing share of the advertising pie suggests that these companies have a secret sauce. To see this on display, notice how The Intercept documented an insidious example:

The recent document, described as “confidential,” outlines a new advertising service that expands how the social network sells corporations’ access to its users and their lives: Instead of merely offering advertisers the ability to target people based on demographics and consumer preferences, Facebook instead offers the ability to target them based on how they will behave, what they will buy, and what they will think. These capabilities are the fruits of a self-improving, artificial intelligence-powered prediction engine, first unveiled by Facebook in 2016 and dubbed “FBLearner Flow.”

One slide in the document touts Facebook’s ability to “predict future behavior,” allowing companies to target people on the basis of decisions they haven’t even made yet. This would, potentially, give third parties the opportunity to alter a consumer’s anticipated course.

Is Facebook really that accurate at guessing the characteristics of its users? Earlier this year, Pew released a poll that surveyed users on their experience with data collection on the social media site. Interestingly, the surveyors found that Facebook’s affinity groups, which is how the site categorizes and then advertisers to people, are only accurate around two-thirds of the time. In contrast, about a third of “users think Facebook’s listings for them are not on the mark.” More details of Facebook’s inaccuracies are explained in the next section.

A closer look at Facebook’s published research shows that there are limits. In one highly controversial study, Facebook randomly suppressed posts that were either positive or negative. When positive posts were reduced in the News Feed, the percentage of positive words decreased from 3.6 percent to 3.5 percent while the number of negative words increased from 1.6 percent to 1.62 percent. On the other hand, when negative terms were suppressed, the number of negative terms went down from 1.6 percent to 1.53 percent while positive terms jumped up from 3.6 percent 3.66 percent.

So what do these numbers mean? Researchers commonly use Cohen’s d to calculate the impact of a phenomena. Thankfully, Wikipedia offers a nice table of commonly accepted effect sizes and how they are interpreted:

With this in mind, take another read of the results:

When positive posts were reduced in the News Feed, the percentage of positive words in people’s status updates decreased by B = −0.1% compared with control [t(310,044) = −5.63, P < 0.001, Cohen’s d = 0.02], whereas the percentage of words that were negative increased by B = 0.04% (t = 2.71, P = 0.007, d = 0.001). Conversely, when negative posts were reduced, the percent of words that were negative decreased by B = −0.07% [t(310,541) = −5.51, P < 0.001, d = 0.02] and the percentage of words that were positive, conversely, increased by B = 0.06% (t = 2.19, P < 0.003, d = 0.008). [emphasis added]

So yes, Facebook had a statistically significant effect, but it was a very small one.

Facebook also received criticism for their 2010 study testing different go out the vote messages. The resulting study explained that “users who received the social message were 0.39% more likely to vote than users who received no message at all.” As noted in the writeup of the study, “Voter mobilization experiments have shown that most methods of contacting potential voters have small effects (if any) on turnout rates, ranging from 1% to 10%.” By its own admission, Facebook’s efforts were significant, but smaller than nearly every study before it. For some comparison on the magnitude, rain decreases election turnout by about 0.8 percent. When asked to vote on a tax increase to fund schools, people vote about 2 percent more for the referendum if the polling place is a school when controlling for the political views and demographics of a voter.

Political ads might have dominated in the aftermath of 2016 election, but their persuasive effect isn’t especially strong. An analysis of $2 million in ad spending in 2006 combined with targeted polling indicated that televised ads have strong but short-lived effects on voting preferences. Rather than being persuasive, the impacts were found to be “more consistent with psychological models of priming than with models of on-line processing.” In a large scale field experiment (N = 74,102) of individually targeted online ads to get out the vote, Professors Jay Jennings and Katherine Haenschen found that microtargeting can increase turnout, but only by 0.52 percent. While effective, the results were on par with telephone mobilization campaigns. Not surprisingly, when the advocacy group MoveOn boasted that their online ads were able to increase candidate votes, the impact was in line with most other studies at 0.4 percent. Still, these might be high water marks, as political scientist Brendan Nyhan explained in the New York Times as the election season was heating up last year: “In fact, a recent meta-analysis of numerous different forms of campaign persuasion, including in-person canvassing and mail, finds that their average effect in general elections is zero.”

Much like their political cousins, advertising used to sell goods and services isn’t an especially efficient method of influencing people. By combining 751 short term estimates of advertising impacts with 402 long term estimates, professors Sethuraman, Tellis and Briesch discovered that for every dollar spent on traditional advertising, companies will get about 12 cents in higher sales in the short term and about 24 cents in the long run. Online ads seem to be slightly more effective, maxing out at 32 cents for every dollar spent, but those ads can yield nothing in return as well, depending on the campaign. Economists at eBay, for example, found that “brand-keyword ads have no measurable short-term benefits” for sales while “frequent users whose purchasing behavior is not influenced by ads account for most of the advertising expenses, resulting in average returns that are negative.”

An upper bound remains for targeted online advertising as well. Ad personalization increases click through rates (CTR), but the effect is small. In a series of experiments detailed in a 2015 paper, the Cohen’s d for detailed personalization was found to be 0.05. Online ads also help to funnel people towards websites, but those new visitors are about half as likely to convert to a sale as compared to any other random person that comes to the site.

Much like advertising, devices can change habits in some cases, but their power is severely limited. In a recent literature review of the impacts on fitness trackers, for example, the authors admitted that, “it is still unclear whether using a fitness tracker alone leads to long-term behavior change, or if it is dependent on other sources of motivation.” A systematic meta-analysis of health focused social marketers found that the overall “impact of online interventions across all studies was small but statistically significant.” That study calculated a mean Cohen’s d of 0.19. Like the rest of the studies, the impact was significant but small. In a report titled “The Remarkable Unresponsiveness of College Students to Nudging And What We Can Learn from It,” researchers Philip Oreopoulos and Uros Petronijevic reviewed a five-year effort to design online and text-message interventions to improve college achievement. Their conclusion was disheartening, but instructive. None of the interventions significantly influenced academic outcomes.

Humans are stubborn creatures. Not even the most advanced technology can budge us.

A View From Advertisers

Those working in the trenches of advertising and marketing tend to have a different view of the ecosystem. Judy Ungar Franks had this to say:

How bad is it? How much money are advertisers spending on this murky supply chain?… When you add up all the costs associated with the ten different layers, they account for 55% of the cpm (cost-per-thousand) that an advertiser pays for a programmatic ad. This means that for every dollar an advertiser spends in Programmatic Advertising over half (55%) of that dollar never reaches the publisher. It falls into the hands of all the third parties that are required to feed the beast that is the overly complex Programmatic Advertising ecosystem. We now know which half of an advertising investment is wasted. It’s wasted on infrastructure to prop up all those opportunities to buy individual audiences across the entire Programmatic Advertising supply chain.

In part, the layers are needed to keep everyone honest and even with all of the protections, there are still extensive conflicts. Last October, a group of advertisers filed a suit against the social-media giant Facebook, accusing it of overstating the time users spent watching videos. The advertisers claimed that the numbers were inflated by 150 to 900 percent, while the platform contends that the numbers were boosted by only 60 to 80 percent.

Facebook’s inaccuracy in measuring key metrics has been thoroughly documented at MarketingLand. In one two year period, Facebook admitted to misreporting the average watch time of Facebook page videos, the organic reach of Facebook Pages (in two different ways), the ad completion rate, the average time spent reading Instant Articles, the referral traffic to external websites, the iPhone traffic for Instant Articles, ad link clicks, the amount of views of mobile videos, and the number of video views in Instant Articles.

And even if advertisers are awash in accurate information, they still face very real constraints in their campaigns. They are limited to specific time periods and have budgets. Thus, they tend to rely upon immediate conversion probabilities like click through rates. While common, the practice has the effect of “serving ads to consumers who are likely to convert even without ad exposures.”

Aram Zucker-Scharff seems to have summed up a common sentiment in the industry when he said on Twitter: “Literally every time I learn something new about Ad Tech it’s a discovery that one more piece of the stack is bullshit.”

Trying to Grasp The Complete Picture

Users of social media exhibit all sorts of folk theories about platforms, an understandable reaction to opaque system. As a user, it is difficult to understand how moderation decisions are made, how stories, pictures, videos, and ads are ordered, and just how much platform operators know about their product. This drives users to “make sense of content moderation processes by drawing connections between related phenomena, developing non-authoritative conceptions of why and how their content was removed,” as researcher Sarah Myers West explains.

Because of this opacity, platforms often get elevated to a kind of techno-deity. In surveys of users, there is a persistent belief that platforms are “powerful, perceptive, and ultimately unknowable.” The available evidence, detailed above, should dispel this myth. Rather than seeing Google and Facebook as harbingers of a new surveillance apparatus, they should be called out for what they truly are: PanoptiCan’ts.