Search engine optimization (SEO) requires constant testing to be effective. If you don’t understand what changes drive growth (or why visitors respond the way they do), you’ll waste your time guessing at solutions while your competitors leave you in the dust.

In this post, you’ll learn what SEO testing entails, what challenges you’ll face while trying to collect actionable data from tools, and how to perform tests to improve your most important key performance indicators.

At the end of this post, you’ll see SEO testing’s potential through 3 of my recent test results that turn conventional SEO wisdom on its head.

First, let’s quickly define what SEO testing is and what forms it takes.

What is SEO Testing?

SEO testing refers to the experiments you perform to measure how search engines and users react to your site.

There are many tools you can use to get data about each web page (such as Surfer and Ahrefs), but no tool or SEO checker can tell you what effect changes will have.

The purpose of SEO testing is to manipulate different ranking factors to determine if the changes result in an increase or decrease in traffic or your position in search results.

There are many ways you can test ranking factors, including:

  • Creating different sets of content
  • Adjusting keywords, meta information
  • Trading out different links

Like your high school science classes, SEO testing relies chiefly on utilizing the concepts of experimental groups and control groups.

experimental groups and control groups

The experimental groups are the cases (pages or entire sites) that experience your test variable (such as trying a new link strategy).

The cases in the control groups are untouched.  Their rankings and traffic will give you the reference point to determine whether or not the experiment worked (or not).

Great testing depends on well-built tests.

Instructions on how to create great tests will be covered a little later…

But first, it may help you precisely understand what kind of benefits you can enjoy if you do the work to develop reliable tests.

What Impact Does Testing Have on SEO?

The benefit that testing has on your SEO strategy is immeasurable—no pun intended. It provides all of the following concrete benefits:

Testing Gives You the Power to Be First

Whenever you run an SEO test, you put yourself in a limited category of people on the industry’s cutting edge.

Power to Be First in SEO testing

Some of the most effective SEO tactics ever discovered have been neutered by search engines shortly after they reached the wider community.

If you’re a tester, you get to be in the rare company of people who use the latest techniques while still powerful. That enables you to operate in the most profitable niches.

Testing Provides You With Information You Can Trust

The SEO community is filled with brilliant and helpful people. However, even the most generous of them don’t share everything they’ve learned.

The most accurate information you can gather is the information you collect yourself.

Even when SEOs are forthcoming about their tests’ results, you simply don’t know if the information they’ve gathered has followed all of the right procedures (more on this later).

Testing Helps You Develop a Ranking System That Works

Repeated testing gives you the power to separate what you can and can’t know about SEO, based on experience.

SEO ranking system

This allows you to skip time-wasting techniques and move directly to those with the most significant, most reliable SEO performance impact.

That makes you a far more efficient SEO. This ability is one of the critical differences between amateur SEOs, and the kind who can successfully dominate even the most competitive niches.

Testing Can Make You Algorithm-Proof

If you test SEO changes often enough, you can develop a second-sight for the trends that are guiding search engine updates.

You’ll spot trends before they’re officially announced, and adjust your practices to avoid penalties.

What are the Challenges of Performing Accurate SEO Tests?

The biggest challenge with SEO testing is collecting untainted data. Your results become tainted when they become affected by outside factors or wrong assumptions.

These obstacles make it impossible for you to say which results are a product of your experiments.

seo challenges meme

Below are some of the challenges that will stand in your way when performing SEO testing and solutions on how to overcome them.

Challenge #1: You Must Make Sure Your Variables Are Stable

To understand what’s happening on each web page in the experiment, you need to isolate what changes affect your data. Stabilizing your variables means working to prevent changes from happening that are outside of your test.

single variable testing seo

For example, if you are trying to measure the effect of content upgrades, you need to prevent your link profile from changing during the testing phase.

If a lousy link gets disavowed or a useful link gets added, your site’s changes could easily mask any result of the content upgrades.

Challenge #2: You Need to Make Test Cases as Similar as Possible

The first thing you need to do is make the experimental and control groups nearly identical. This will allow you to start measuring the changes once you start altering the test site.

seo split testing illustration

When you come back to check website changes, you’ll have a better idea of how they happened.

Here are just a few examples of why you’ll have trouble with that:

  • No two web pages can have the same URL, yet the URL is itself a ranking factor.
  • Those links still won’t be the same age if you manage to get the same links for both sites. It’s also highly unlikely that you’ll be able to gather them at an identical velocity.
  • If you create two identical sites for your split test, one is now filled with duplicate content. Duplicate content is a factor that Google can detect and may choose to penalize. In some cases, search engines may refuse to index the new material at all.

Challenge #3: You Need to Expect the “Random Ranking Factor”

The random ranking factor is a phenomenon coined by SEO Terry Kyle (founder of WPX hosting). It is best illustrated with an example…

Imagine that you launch five identical sites or landing pages on the same day. They are in the same niche, use corresponding keywords, and employ the same design style.

You will likely notice the following:

  • Three sites tend toward average performance
  • One site tends toward exceptional performance
  • One site tends toward inexplicable sluggishness

seo testing random ranking factor

This is not a precise rule, but many SEOs have recorded the effect over the years.

For reasons that are nearly impossible to measure efficiently, some sites simply behave as if they are blessed while others act as if they are cursed.

What’s the Solution?

There is one primary solution to the challenges that SEO testing poses.  You need to increase the number of test cases.

One control page is never going to be enough, and neither is one test page.

Adapt to this challenge by creating larger groups for each test group and control group. Have 10 URLs in the control group, and 10 in the experimental group…or better yet, 50.

seo large test cases

Then, you’re going to measure the results by taking the average of the changes in both groups.

Maximizing the number of test cases helps to resolve our three biggest challenges in the following ways:

Keeping variables stable

It’s impossible to completely control external variables in SEO because none of it takes place in a vacuum.

For example, if we have 50 test cases, then we know that a single random backlink hitting one of them isn’t going to throw off our test.

Keeping test cases similar

Once again, creating similar test cases for SEO is nearly impossible. A large amount of test cases helps us smooth out fluctuations.

For example, it helps you control all the minor differences caused by URLs, age, and other factors that play a role in SEO.

Controlling the random ranking factor

With a large number of test cases, we know that we’re at lower risk of our having our results colored by a page that’s uniquely blessed or cursed. The random ranking factor gets averaged out.

We now have enough information to identify sites that behave differently. This will help prevent us from either getting overconfident or giving up on a great technique just because it failed randomly.

Let’s look at all this through an example.

Example Test

Pretend you have a blog in the wellness niche, and you want to test the effect of double-counting keywords in the title tag.

If you have 30 posts with traffic, a simple experiment might involve creating two groups:

  • 15 posts preserved as a control group (titles unchanged)
  • 15 posts to test your idea of an ideal title

wellness website seo test

If you’re curious about how a test like this might play out, don’t miss the ‘clickbait title’ test that will be revealed near the end.

For now, let’s focus on what roles these groups play:

  • Group 1 will help you follow any changes that happen outside your experiment. If your control group experiences sudden SEO shifts, then you know the same effect in your other groups isn’t a result of your tests.
  • Group 2 will test your hypothesis. Once you’ve subtracted any SEO shifts from the control group and random group—you’ll have an idea of the effects of your real changes.

Naturally, testing like this is going to depend on the funding you have. If you can afford it, testing ten sites instead of 5 (or 20 instead of 10) will allow you to create a more accurate average.

If your budget is very limited, you can test across multiple pages with only 1 or 2 sites.

However, there will be more room for error with those numbers than most experienced testers will tolerate.

The more web pages you can create for a test, the more effectively you can cancel out noise. Sometimes, the most effective way to cancel out noise is to remove important factors altogether.

Working with gibberish can help you do that.

The Role of Gibberish

Gibberish (often in the form of lorem ipsum) has become popular as a way of testing SEO factors. In a 2018 roundup, I covered how Kyle Roof created a stir by ranking a site with nothing but gibberish.

It works like this: You create five pages that are optimized for a nonsense keyword, like Fragglebottomcat. Once those pages have been indexed, you can start performing experiments on them to isolate different effects.

For example, you could add schema to just one page and watch to see if that page starts to outrank your other nonsense pages.

gibberish website test sample

You can experiment with applying different links or page speeds. With no real content, you can look at the isolated effect of other factors without interference.

There are limitations, though.

  • It’s difficult to use gibberish to test complex strategies. For example, what’s the effect of a link building campaign with a standard 50 link per month velocity vs. a 200 link per month velocity. Is it 4x better? A live site with real traffic would be a better test case.
  • Google doesn’t apply the same algorithm to all topics. Ranking factors are a lot more complicated for terms related to health or finances than random nonsense words.

As they say in SEO, there’s more than one way to skin a cat.  Steven Kang from SEO Signals Lab noticed the various preferences as well.

steven kang fb comment

Beyond creating more test cases, you can follow some other practices to ensure that your experiments deliver the best data.

Best Practices for SEO Testing

Follow these rules to draw some better data from each SEO test:

Set Aside Enough Time

seo strategy schedule

Beyond your money budget, you need to put a significant time budget in place. Once again, the more time that you can give your experiment, the more sure of the accuracy of any data that you collect.  Some results (especially offsite related) may take months to yield a difference.

Exercise Attention to Detail

If you can’t measure it, you can’t manage it. The more information you’re tracking, the more awareness you’ll have of how sites are different, so you must be tracking even when you’re not testing.

peter drucker quote

I can illustrate this with a real-life example. At one point, I had a great site in the Brazilian testosterone niche. One week, we suddenly jumped from page two to page one.

I was only able to figure out why, because of my tracking. It turns out, some links I snagged from a local citation package were worth a lot more than I imagined it would be.

Thanks to my research, I caught the factor. Thanks to that insight, I now had a new strategy to use for many other sites.

Exercise Discipline

You may want to see results from your experiments (either because they validate an exciting theory or because they make you appear competent). Wanting a result from your data will deeply color your perception of that data. Do your best to approach your results without bias.

mr miyagi quote

Sure, this sounds like a lot of work, when it’s all put together, but the next section will help you understand what it’s all worth. We’ve got the results of 3 recent tests, with massive implications for SEO’s future.

Recent Findings in SEO

Nothing illustrates the impact of SEO testing, like having access to information most of the industry doesn’t yet possess.

The following experiments were performed as personal research. What follows are the theories, tests, and findings from research built using the rules above.

Test #1: Measuring the Effect of Inserting NLP-Friendly Sentences Into the Content

This test involved a look into whether implementing Natural Language Processing (NLP)-friendly language would change the way that Google responded to a site. NLP refers to the ability of software to interpret and manipulate language. This technology is improving but still limited.

Theory: If we can make it easier for NLP to process our content, Google will be more likely to reward us with snippets and better rankings.

We aimed to optimize our content for NLP by doing the following:

  • Echoing back the question
  • Giving the answer
  • Mentioning the correct unit

how to write nlp friendly content

For example, here’s a typical way of answering a question to another human, compared to the NLP-friendly style.

Question: What is the best temperature to brew beer?

Standard answer: “68 to 72 degrees.”

NLP-friendly answer: The best temperature to brew beer is between 68 to 72 degrees Fahrenheit (20 to 22 degrees C).

Test Structure:
  • 50 page were chosen, spread over ten websites
  • Each page had flat rankings
  • NLP-friendly content replaced query answers for a selection of pages, and the sites were submitted for recrawl
  • Finally, we looked at average % increase or decrease in rankings and rich snippets


# of featured snippets4694+104.3%
Average ranks15.211.2+26.3

Implications: NLP-friendly language measurably improves rankings and attracts snippets. The number of snippets we controlled doubled after our work.

Test #2: Is It True That Clickbait Titles Stopped Working in 2022?

clickbait sensational title
This test measured whether the conventional wisdom that clickbait titles were “over” held any water. It involved changing 40 clickbait titles on existing blog posts.

Theory: People have started to avoid clickbait titles, and may prefer more straightforward descriptions of what they’re getting.

We understand clickbait titles as titles that tease exciting results, but may not explain the article’s content:


Clickbait title: 7 Strategies that will increase your traffic by 200%

Straightforward title: How to increase traffic to your blog

Test Structure:
  • We gathered 40 URLs that met the following criteria: Each one:
  • Used a clickbait title (that’s how we normally write all of them)
  • Ranked in the 1-3 spot
  • Had been flat over last 30 days
  • The Click Through Rate (CTR) stats from the past two weeks were gathered from Google Search Console.
  • The SEO titles were changed to straightforward descriptions of what the content was about.


 Total ClicksTotal ImpressionsAverage CTR
40X URLs11,4207,75031,848426,1093.73%2.90%

Implications: The new titles had far fewer clicks. People are still more responsive to clickbait titles than they are too straightforward descriptions of the content. 

Test #3: Does Interlinking Irrelevant Pages Hurt Your Rankings?

This test examined whether irrelevant internal links were noticed or acted upon by Google. The SEO site that inspired this experiment was a beard oil shop interlinked freely with pages that were slightly, but not directly, related (such as beard shampoo).

beard oil starter sample

Theory: Irrelevant internal links hurt the potential to rank

We understand “irrelevant” internal links as those that link to pages with unrelated topics. We were pretty strict in defining links as irrelevant.

For example, for the topic of Beard oil, we purged links from pages that covered:

  • Beard shampoo
  • Beard wax
  • Beard balm

These topics are related to one another, but they don’t serve the same need, so we considered them irrelevant for testing purposes.

Test Structure:
  • Eight real medium-sized websites were chosen
  • 40 URLs were selected from the set. They included:
  • Irrelevant pages: These were chosen because they had no keyword overlap whatsoever. They were linked to our test page.
  • Test pages: These were chosen because they had decent link equity. The irrelevant pages were linked up with standard anchors.

The test involved creating internal links to the irrelevant pages from the test pages. 80 keywords were tracked to determine changes

experimental group test phases


Ranking ImprovementTraffic Improvement% of keywords that went down

Oddly enough, the pages that were given links from irrelevant pages experienced short-term ranking and traffic improvement. Only 2% of all keywords went down.

Implications: Internal links from irrelevant pages are not that harmful, and may even result in short-term improvements.

Success Through Testing Is Within Your Reach

SEO testing is one of the most effective ways to improve your sites and your skill as an SEO. By understanding the role of testing, meeting the challenges head-on, and setting up your tests the right way, you can learn the most effective techniques.

As the tests above showed you, the conventional wisdom about what works for SEO is often flawed, or only applicable in some cases. SEO checker tools will only tell you where your website is now. Testing empowers you to see where it could go.

By taking the time and expense of testing seriously, you can discover a more productive path for your sites, and plan for the changes search engines make. Armed with the knowledge that only you possess, you can carve out a niche for yourself in the SEO world.

If you want to learn more about tested SEO tactics and the strategies that work, you can join The Affiliate Lab. You’ll get access to all my private tests as soon as I have results, and the chance to discuss your own with the most exclusive affiliate SEO Facebook group around.

Got Questions or Comments?

Join the discussion here on Facebook.


Article by

Matt Diggity

Matt is the founder of Diggity Marketing, LeadSpring, The Search Initiative, The Affiliate Lab, and the Chiang Mai SEO Conference. He actually does SEO too.


Want to rank easier, higher and faster?

Sign up and join 30,000+ other subscribers and get SEO test results sent straight to your inbox.

moz sidebar

"One of the most effective SEOs I've ever met"- Cyrus Sheppard

As Seen On...

Search Engine Journal
New York Post
ahrefs brand