The best conversion rate optimization companies start with research, not button colors. They can explain how they find problems, how they prioritize test ideas, how they decide a test is finished, and what happens when results are flat. Before signing with any agency, ask to see their process, a test that failed, their sample size rules, and who owns the data. If the answers are vague, or every case study is a huge win, keep looking.
Hiring a CRO agency can be a smart move. A good one brings experience across many stores, research skills your team may lack, and a testing rhythm that’s hard to build from scratch. A poor one burns months on low-impact tests and leaves you with a slide deck. Here’s how to tell them apart.
Decide what you actually need
Agencies sell different things, so start by deciding which you want.
Full-service programs handle research, test design, development, analysis, and reporting. Good if you have budget but no internal capacity.
Research and strategy engagements deliver an audit and a prioritized roadmap, and your team runs the tests. Good if you have someone who can build tests but needs direction.
Development support builds the variants your team designs. Good if ideas aren’t the bottleneck, but engineering time is.
Many stores do best with a hybrid: an agency for research and strategy, with tests built and run in-house on conversion rate optimization tools the team controls. That keeps learning inside the business and usually costs less.
The questions to ask
“Walk me through your research process.”
Good agencies talk about analytics deep dives, session recordings, heatmaps, on-site surveys, customer interviews, and heuristic reviews before proposing tests. If the first meeting is a list of test ideas based on “best practices,” that’s a warning sign. Ideas should come from your data.
“How do you prioritize test ideas?”
Listen for a framework that weighs evidence and traffic rather than opinion. Some agencies use well-known models. CXL’s PXL framework, for example, scores ideas on things like whether the change is above the fold, whether it’s noticeable within five seconds, whether it runs on a high-traffic page, and whether research supports it. You don’t need them to use PXL specifically, but they should be able to explain why test three comes before test seven.
“How do you decide when a test is done?”
The right answer involves a sample size calculated before launch, minimum runtimes in full weeks, and rules against stopping early when results look good. Ask how they handle tests that don’t reach significance. “We just run them until they win” is the wrong answer.
“How do you check that a test’s data is valid?”
Strong agencies check for sample ratio mismatch, where the planned traffic split comes out uneven. Microsoft’s experimentation team, in a post on diagnosing sample ratio mismatch, says every A/B test there must pass this check before its effects are analyzed, and that about 6% of their tests had an SRM. Ask the agency how often they see it and what they do when they find it.
“Show me a test that lost.”
This is the most revealing question. Every real testing program has plenty of losing and flat tests. An agency that shares only wins is either cherry-picking or not testing rigorously. A good agency will happily explain a loss and what it taught them.
“What metric will you optimize for?”
For ecommerce, the answer should be revenue per visitor or profit-based metrics, with conversion rate as a supporting number. Agencies paid on conversion rate lifts can be tempted toward discount-heavy tests that hurt margin.
“Who owns the data, tests, and learnings?”
Your test history is valuable. Make sure the testing tool account, the results, and the documentation belong to you, and that you keep access if the engagement ends. Ideally, tests run in a tool your team owns, not the agency’s account.
“Who will actually work on our account?”
The people in the sales meeting are often not the people doing the work. Ask to meet the strategist and analyst who’ll run your program, and ask how many other accounts they handle. A brilliant team spread across fifteen clients may give you less attention than a smaller agency with a focused team.
“What if our traffic can’t support many tests?”
This one separates honest agencies from the rest. A store with modest traffic can’t run many properly sized tests per month. A good agency will say so, then suggest bolder tests, template-level tests that pool traffic across many product pages, or more research-driven fixes that don’t need testing at all. An agency that promises a high test count regardless of your traffic is probably planning underpowered tests.
“What happens with SEO?”
Agencies should know Google’s guidance on website testing: no cloaking, rel=”canonical” on variant URLs, temporary 302 redirects rather than 301s, and removing test code once a test ends. If they look blank, that’s a problem.
Red flags
A few patterns should make you cautious. Guaranteed percentage lifts before they’ve seen your data. A plan that starts testing in week one without research. Case studies with huge numbers and no detail about traffic, duration, or method. Contracts that lock you in for long periods with no milestones. And reports full of activity metrics, like number of tests launched, with little about revenue impact or lessons learned.
How to compare shortlisted agencies
Once you have two or three candidates, ask each for a short proposal based on a brief look at your analytics. Compare how they read your store. Did they spot the same main leak? Did they back their ideas with evidence? Are their first proposed tests ambitious enough to matter on your traffic?
If you’re still building a longlist, a roundup of the best conversion rate optimization companies can help you find agencies by region and specialty. Treat any list as a starting point, then let the questions above do the filtering.
Setting up the engagement for success
Once you’ve chosen, agree on a few basics in writing. A clear primary metric. A cadence for reviews, usually every two weeks during active testing. A shared test log you can access at any time. And a named internal owner on your side who joins reviews and makes final decisions.
Plan the handover from day one, too. Even if the agency runs everything now, you’ll want your team to understand the process and keep it going if the engagement ends. Ask the agency to document their research and reasoning, not just results.
Doing more in-house
Many brands pair an agency’s strategy with in-house execution, or move fully in-house once they’ve learned the process. Either way, having CRO software your team can run in-house keeps tests and learnings under your control. CustomFit.ai is one option for D2C and Shopify brands. It’s a no-code A/B testing, personalization and CRO platform with unlimited users, so an agency and your internal team can work in the same account, and there’s a 14-day free trial without a card.
Key takeaways
Choose a CRO agency on process, not promises. Ask how they research, prioritize, size tests, check data validity, and handle losses. Make sure they optimize for revenue, respect SEO rules, and leave you owning the data and learnings. Be wary of any agency that guarantees a lift or only ever shows you wins. And agree on the metric, how often you’ll review results, and how the handover will work before the first test launches, so the value stays in your business when the engagement ends.