YouTubers will often spend dozens of hours writing, shooting , and editing their videos, only for the video to fall flat. Sometimes it’s just the nature of the content. Your audience may not have been as interested in a particular topic or story as you thought. But sometimes, it’s completely fixable: You just need some A/B testing.

What is A/B testing on YouTube?

On YouTube, A/B testing, sometimes called split-testing, consists of an experiment that pits two video titles or thumbnails against each other. The goal is generally to measure the effect of these variations on click-through rate (CTR)—that’s the percentage of people shown your video who click on it. In short, A/B testing on YouTube allows you to see which thumbnail is more alluring to viewers.

Tools like TubeBuddy also allow you to test metadata like tags and description, but most creators will see the most benefit from testing their titles and thumbnails alone.

While CTR is the most commonly tested metric (and one of the most useful), some tools can also measure the effect a title or thumbnail has on things like view duration, subscribers gained, comments left, and likes. YouTube's Test & Compare tool only allows users to measure watch time.

Why you should A/B test on YouTube

At its best, A/B testing can be the difference between a viral hit and a flop. I’ve seen stagnant old videos shoot up hundreds of thousands of views after successful A/B testing. I’ve seen a video ranked #10 shoot up to #1. Sometimes, all it takes is a small bump to your CTR for the algorithm to start distributing your video to new audiences.

Even on videos that are doing reasonably well, creators are leaving viewers (and money) on the table by not optimizing their click-through rate.

How to get started A/B testing

There are a few options. Many YouTubers now have access to YouTube's native ‎Test & Compare tool. But by far the most popular option within the industry is TubeBuddy, which requires a paid subscription. If you're feeling scrappy, you can also do your own A/B test with a spreadsheet.

How to A/B Test on YouTube's native tool, Test & Compare

YouTube calls its A/B tests feature Test & Compare, and you can test three options at once, rather than the usual two. Unlike other tools, Test & Compare won't let you test titles.

The easiest way to find the tool is to go to the thumbnail area when uploading a video, or in the details page once your video is already uploaded. Just click the three-dot menu on the thumbnail image to bring up Test & Compare.

You can check in on the results by once again clicking the three dots on your thumbnail and selecting “view test report.” You can also see the results by clicking on the “Reach” tab of a video’s info, where you’ll see a prominent “Thumbnail test” section. The results are shown as a percentage of total watch time generated from your options. That means the higher the percentage, the better the performance.

When the test is done, YouTube marks a thumbnail “winner” if the results are statistically significant, and “preferred” if they are not.

It's important to note the difference between Test & Compare and other A/B testing options. Test & Compare measures the amount of watch time, and doesn't report on click-through rate. This is an incredibly important metric, but may be frustrating for creators looking for more granular data.

If you need that data and you've got an extra $16.50 a month, TubeBuddy may be a better bet.

How to A/B test on TubeBuddy

To get started, log in to TubeBuddy and click “Create AB Test” in the A/B testing tab. 

It will give you an option to select from your recent videos, or search through your library. From there you can select a thumbnail test or a metadata test.

If you want to test a thumbnail AND title together, or even just a title, you’ll need to run a metadata test. Be warned that testing titles or other metadata can cause your video to be re-indexed by Google, which might hurt videos that get lots of traffic from SEO. A thumbnail-only test won’t do this.

From there, you’ll be asked how long to run your test, or to run until it’s statistically significant. Without getting into a long-winded statistics lesson, experiments are given a confidence percentage from 0–100%, 100% meaning there is total confidence that the results aren’t a fluke, 0% meaning there’s no such confidence. Running your test until it’s statistically significant will mean it runs until the result is 95% confident. 95% is the standard for most science and should be your standard too, even when testing your YouTube CTR.

Testing until you reach statistical significance ensures a more accurate result, but can take a while—sometimes tests will run for months without reaching statistical significance, at which point you should just cancel them and test something else.

Whichever method you use, after a few days of testing, you’ll be able to see preliminary results even before the test completes.

TubeBuddy lets you test multiple variables at once (like a new title and thumbnail), but that can make the results a little muddy. Are people clicking on the new title, or the new thumb? This might not be a problem, especially if your thumbnail uniquely complements your new title. 

How to do a DIY A/B test

When you A/B test with TubeBuddy, the platform just switches your title and thumbnail every 24 hours and reads the same data you have access to through the Creator Studio. That means if you manually switch those variables at the same time every day, you can look at daily changes to click-through rate and watch time and log them. Though if you’re doing this method, I’d encourage you to do a refresher in statistics to make sure you can read the data correctly.

What is a meaningful click-through rate change?

There’s no universally “good” CTR since YouTube is comparing your CTR to similar videos. If your financial advice podcast has a 0.5% lower click-through rate than a similar podcast covering a similar topic, it’s going to hurt your video. While you can’t see your competitors’ CTR, it means that even a percentage point could be the difference between an average video and a breakout hit. 

In general, though, any CTR below 5% should be aggressively tested until it reaches 5% or above. Videos with 8% or above have a good chance to be served widely by the algorithm.

📈 Read more: 3 YouTube analytics numbers you should be measuring — and how to fix them

Common mistakes when A/B testing

Having tools to do A/B testing is just one step—you also have to know what to test in order to get a good result.

Mistake #1: Making minor changes

In marketing lore, there’s tons of stories about how simply changing the color of a button is the key to massive changes in click-through rates and, therefore, success. With A/B testing, it’s easy to assume that changing the font color or changing a word in the title will yield big results.  

There are certainly cases where this is true, but I’ve also run hundreds of tests and seen that small changes usually yield small results. If you have lots of time and money and a big backlog of content, those small changes might be worth the investment. But for most, their effort is better placed elsewhere.

Consider a fictional video titled: 

• “How YouTube Thumbnail A/B Testing Can Improve Your Channel”

Don’t test that title against:

• “YouTube A/B Tests Can Improve Your Channel,” or
• “Can YouTube A/B Tests Improve Your Channel?” 

Try for bigger swings, like:

• “Will This Tool Make Your Video Go Viral?” or 
• “You’re Losing Views by Not A/B Testing”

When a video is really flagging, it’s best to try a new value proposition. Maybe your audience doesn’t think they care about “A/B Tests,” or they don’t know what it means. So you might pivot your language to tell the viewer you’re going to show them a new tool, and they’re missing out if they’re not using it.

The same is true for imagery. You can test different facial expressions all day, and they might yield good results, but sometimes you need to think fresh and use entirely different imagery in an A/B test. 

Mistake #2: Ignoring statistical significance

If TubeBuddy is reporting 50% confidence on any given test, it means there’s a coin-flip chance that the “best” thumbnail is actually the worst thumbnail. If you ran the same test again in a week, you could get the exact opposite result. People might see “80% confidence” and assume the results of a test are true, but in reality it’s saying there’s a 1 in 5 chance that the result is pure coincidence. Researchers aim for a 95% confidence because it means there’s only a 1 in 20 chance that the final result will be bogus.

Similarly, YouTube's Test & Compare might give some creators false confidence with their "Preferred" tag.

Mistake #3: Using clickbait

When chasing click-through rates, it’s important not to forget watch times, which are also displayed in TubeBuddy. Having a high CTR on YouTube but low watch time will kill the video’s chance of being distributed widely. That means when you’re A/B testing, don’t waste your time making clickbait titles that fail to deliver in the content of the video. In general, your title and thumb should offer some kind of value that’s mirrored in the first few seconds of your video.

A/B testing on YouTube FAQ

Can you do A/B testing on YouTube?

You should soon be able to do A/B testing on YouTube, if you can’t already. YouTube has launched its Test & Compare tool for thumbnail tests for many creators. If you don’t yet have access, you can use third-party tools like TubeBuddy to A/B test your video’s thumbnail, title, and other metadata.

Can you A/B test YouTube titles?

Yes, you can A/B test YouTube titles using a third-party tool like TubeBuddy. Keep in mind that A/B testing metadata like titles may lead Google to reindex your video, which can hurt any videos that are getting a lot of search traffic.

How does A/B testing work on YouTube?

A/B testing on YouTube works by showing different audiences different versions of your thumbnail or metadata and determining which version got a higher click-through rate or received more watch-time. You can then use that version on your video going forward in order to improve its performance.