If you can't tell the difference, get rid of it

Yesterday we talked about using blind A/B testing to see whether an idea you have is actually good for the song or just a good but wrong idea wrapped in a gorgeous plugin interface. The latter happens more than we’d like to admit!

And here’s a ps important enough to merit its own post: if your results are inconclusive, you should not go ahead with the treatment.

When you do a blind A/B test, sometimes it will be immediately obvious; one choice will be clearly better. Those are great! It’s nice to have certainty.

But there are lots of situations where what you’re doing might be much more subtle. Perhaps you do a blind A/B, and the first time you choose A. But the second time, you choose B!

If it’s just that your ears needed some time to really hear the change, that’s fine. If you subsequently keep choosing B, that’s valid.

But if you find yourself blindly choosing A sometimes and B other times, and you’re not consistently choosing the same one — get rid of the treatment in question. Because what your ears are telling you is that maybe it’s helping and maybe it’s not. And we only want to be making changes that are definitely helping the music, right?

Rigorously — jamie