A Critique of the Acquisio Study on Google Match Types
A few days ago Search Engine Land published the article “Google keyword match type tournament: Place your bets on Exact” by Marc Poirer, which details the results of a study on how match type affects various metrics in an ad campaign.
The catalyst for the study was a recent announcement by Google that “exact match close variants will begin including close variations that share the same meaning as your keyword.” Google included the following chart as an illustration:
To test what this means for digital marketers, Marc and his team at Acquisio set up a “tournament” in which they pitted identical keywords with different match types against each other. Here is how he describes the experiment:
To measure the impact of the exact match relative to other match types we had to put them all under a microscope for the analysis. Aside from match types, we also focused our research on accounts in our database using Acquisio Turing machine learning (my employer) since its campaign level bidding data is unbiased to bids set for different match types of the same keyword.
Between January and October of 2018 there were 978,358 keywords that had at least 2 different match types among the machine learning-managed accounts.* (see research notes below) To make a valid comparison between different match types of the same keyword, they had to be used in pairs on the same day with the exact same bid. Filtering for this constraint reduced the number of candidates to 477,935 keywords.
Since a typical keyword has very little traffic on any given day, many of the keyword match type pairs had statistics on multiple days, which were combined to get a more robust measure. If the same keyword (e.g. “dog”) appeared with different match types in multiple different campaigns, those were considered as individual entries because their bids and other settings could have been very different.
Then they looked at how the same keywords in different match types performed relative to each other across a range of metrics: average position, CTR, CPC, CVR (conversion rate), and CPA. In every round, they found that the narrower the match type, the better the CTR, CVR, and CPA.
The author concludes:
The data is in, the bets have been settled. According to our findings, the cheapest and best-performing match type is exact every time!
The latest exact match update from Google looks promising for search marketers who are hoping to capture more relevant traffic, but hesitant to switch away from phrase or broad match keywords. But if the update hasn’t inspired you to get more granular on match-types than hopefully the data presented here has. Even before the update, exact match keywords were performing better than every other match type and costing campaign managers less.
If true, the study is a vindication of the bid stacking techniques I was taught when I went to work for an agency five years ago. But I remain skeptical.
The problem lies with the experimental design. Broader keywords appear in broader queries, and so the results tell us nothing about how the keywords compare head to head for the same queries. In fact the results may only tell us that less relevant queries provide less qualified traffic, which we already knew. What we really want to know is whether match type affects Quality Score, a term not mentioned in the article.
There has always been a tradeoff between reach and relevance. The author himself concedes:
The one caveat with exact match is low volume, despite impressive ROI. It can be hard (i.e., impossible) to scale your campaigns when you’re cherry picking only the very best queries using exact match.
But the question is not whether an exactly matching query provides more qualified traffic. We know it does. The question is whether exact match keywords have some advantage in auctions over keywords with a broader match type. If it does, that advantage must be in the Quality Score that Google assigns to the keyword for that query.
I don’t know an easy way to measure the impact of match type on Quality Score. It makes some intuitive sense that a narrower keyword will appear in more relevant searches, which should make it perform better, which in turn should garner it a higher QS. On the other hand, a narrower keyword will get fewer impressions and provide Google with less history upon which to base QS. It’s not clear to me which effect is stronger.
Perhaps the search terms report could yield some clue. If exact match keywords earn a higher QS, they should consistently beat broadmatch keywords in queries for which they both qualify. By indicating which keywords fire for which queries, the search term report could help us verify whether exact match keywords have an advantage, but it won’t tell us how to quantify that advantage.
Ultimately I have mixed feelings about attempts to know the mind of Google through experiments like these. On the one hand I think it is great for marketing professionals to think deeply about these issues, test every conjecture, and be prepared to yield to evidence. On the other hand I think there’s a strong tendency to draw the wrong conclusions from data, and we should give more attention to the rational rather than empirical approach.
