SOP 41 — Run one test a week

1. Pick the step first, by arithmetic

Getting better is worth doing because it returns more leads for the same effort. Which step to work on is settled by arithmetic.

Take a funnel with an opt-in, an application and a scheduling step, and work an example on it. The figures below are the ones this page works with throughout.

Step	Rate now	Rate after adding five points	What that does to leads
Opt in for free information	30 percent	35 percent	About a sixth more — a multiplier of 1.16
Apply for what you sell	5 percent	10 percent	Double — a multiplier of 2
Schedule after applying	50 percent	55 percent	A tenth more — a multiplier of 1.1

The same five points added to each step produce wildly different results. Add five to thirty and you get a sixth more; add five to five and you double; add five to fifty and you get a tenth more.

The arithmetic in the first row is rounded down: thirty-five over thirty is a sixteen and two-thirds percent increase, stated here as sixteen, and the multiplier of 1.16 is the one carried through.

The definition you are applying. The constraint is the step where the same absolute change in percentage points produces the largest percentage improvement. So add a fixed number of points to each step, compare the resulting multipliers, and put your effort into the largest. In the example that is the apply step.

And the comparison that tells you it was worth it. If you could take that five percent to twenty percent, you would multiply throughput by four. So the real question is not whether four times more leads is good — it is whether moving one step from five to twenty is easier than hiring four times as many callers. Probably it is, and that is why the effort goes there.

2. One test, one week, one platform

The cadence is one test per week per platform. Four reasons are given for it, in order, and the fourth is probably the most important.

#	The reason
1	Test several things at once on one platform and you never learn which one worked — so you never learn at all
2	Split tests interfere with each other
3	A one-test limit forces you to prioritize
4	Only a single test runs long enough to tell if you can actually get an improvement

On the second reason. Change the landing page and something else at the same time and both changes might move a third thing. You might get more leads and they might be less qualified — and the data can no longer tell you which change did what.

On the third reason. You could run an unlimited number of tests; you do not have unlimited time or unlimited resources. Constraining yourself to one forces you to pick the test you believe will produce the most engaged leads, which is the same discipline as picking the constraint in §1.

On the fourth reason. Too short and you do not collect enough data. Too long and the next constraint goes without the time it could have had. Both errors are named; the balance point is the sample size question in §3.

3. Deciding whether the result is real

You need a sample large enough to support a verdict. The method given is deliberately plain: find a statistical significance calculator, enter the before figures and the after figures, and read off how likely it is that the difference is real. If it is significant, take the winner and start the next test.

What this page cannot give you. No significance threshold, no minimum sample size, and no general minimum or maximum test duration are stated. The instruction is "long enough" and "not too long", with the calculator as the arbiter. This page gives no figure for the sample size a test needs.

The weekly review. One practice here starts one split test per platform on the same day each week, lets it run the week, and on review day does three things:

Pick the winner of each platform's test.
Write the result into a running log of every test run so far, so the next idea starts from what has already been learned rather than from nothing.
Set the next test against whichever version is winning now.

The stop rule in that practice. If four attempts, or a month, pass without the leading version losing, turn to the next constraint. Effort spent improving the same thing brings smaller and smaller returns, and when that effort would earn more elsewhere, that is the point to try something new (SOP 40).

A week is typically long enough for that practice, at the size of its team and its advertising spend. That is a condition on one operation, not a duration for every business. A weekly review on a fixed day is also on SOP 220.

4. What a year of this is worth

One test a week, taken through a year, leaves you with fifty-two improvements. That is the whole argument for the cadence: that many improvements in a year, the claim goes, improve a business a lot.

A worked case. One business grew forty percent over a year and a half while doing nothing new at all. Its quarterly goals were a list of small moves along the path a customer already walked — show rates from sixty-five percent to seventy percent this quarter, click-through rates from 1.1 to 1.2, pickup rates on outbound calling from one figure to a better one. The pickup-rate target has no numbers, so it does not work as a benchmark.

The advantage that accrues is not the individual percentage points. It is that the knowledge of which things mattered stays inside your business, because competitors can see neither the change nor the reason it worked.

5. Running several tests at once, legitimately

The one-test rule is per pathway, not per company. You can run the process across several channels at the same time as long as they are independent.

Allowed	Not allowed
Testing paid advertising on one platform's pages and, in the same week, paid advertising on a different platform's pages	Testing more than one thing inside the same pathway in the same week

Independence is the whole condition. What makes two pathways independent is not established on this page.

6. Prioritize front to back

Within a pathway, work in the order the lead meets things.

The reason is a pattern in the numbers rather than a preference. In general the lowest-percentage steps sit at the front and the higher-percentage steps sit at the back. One percent of people might click an advertisement, and then thirty percent of those will hand over their contact details. A front step is therefore usually the constraint, because it has the largest return for the smallest improvement — which is what makes it the constraint in the first place.

Earlier results sit under this. Most of your attention belongs in the first five seconds of a video, which is SOP 34; fixing those seconds on one video multiplied its views by two hundred.

The order. The call-out, the value elements, the offer, the call to action, the landing page, the headline, the sub-headline, the image.

The list mixes elements of the advertisement with elements of the page the lead arrives on: the headline and the image come after the landing page rather than inside it. Which layer each item belongs to is not established on this page.

Go down the pathway in the order the lead will see it, because that is usually where the most return sits; this is one person's observation, not a law.

7. What this page does not decide for you

These are gaps in the procedure, not omissions from this page.

How many points to add when you compare steps. Five is the figure used above; no rule says five, and the ranking can change with the size of the increment you test.
What to do when steps tie. No tiebreak is given.
How long a test runs. See §3.
Which single change to make inside the chosen step. The page tells you where to test, not what to try.
Whether to test at all below a certain size. The scale rule — do more first under about a million a year in profit — is on SOP 40, and it is stated there as an observation.

Other pages give different answers to questions this page answers. Each is set out here with what it costs.

Which step to work on. Here, add the same points to every step and take the largest multiplier, which always picks the step with the lowest rate. SOP 220, section 7, works on the step lowest relative to its benchmark. This page's way needs no benchmark, but it points at the lowest rate even where that rate is normal for its step; SOP 220's way needs a benchmark for every step, and neither page gives one. This page does not settle which reading is right.
How many tests run at once. Here, one test per week per platform, and never two things in one pathway in the same week. SOP 220, section 7, describes one team that usually runs one or two tests a week, one at step one and one at step two. SOP 119, section 7, puts a large batch of ads out each week with hooks, meats and calls to action varied together. SOP 113, section 7, varies every remix, combines them freely and tests them all. This page's way tells you what each change did, one change a week; theirs finds winners faster among many versions, and you may not know which change carried the result. This page does not settle which reading is right.
The order to test in. Here, the order the lead meets things (section 6). SOP 152, section 4, tests the headline and subheadline first, then the offer, then the image, ranking each test by its impact against the work it takes. This page's way tests the front, where the lowest rates usually sit, first; SOP 152's way runs the cheap, high-impact tests first and leaves the offer, which is more work to change, until after the headline. This page does not settle which reading is right.
Whether a small gain is worth a change. Here, a test a week adds up to fifty-two improvements a year, and the worked case in section 4 grew through small quarterly moves in its rates. SOP 188, sections 2 and 3, holds that a change typically costs about 20 percent up front, especially where people are involved, and gives one owner's minimum of at least a 20 percent gain before changing anything. This page's way keeps taking small gains and takes on whatever drop each change brings; SOP 188's way passes up gains under 20 percent and changes far less often. This page does not settle which reading is right.

8. The checklist

Question	The answer
Who runs this	You, or whoever owns the advertising
What you test	One thing
How often	Once a week
Per what	Per platform, and per pathway
How you choose the step	Add the same number of percentage points to each step; take the largest multiplier
The worked example's steps	Opt in 30 percent, apply 5 percent, schedule 50 percent
Their multipliers after five points	1.16, double, 1.1
Which step wins in the example	Apply
What four times the apply rate does	Multiplies throughput by four
The four reasons for one test at a time	You cannot learn from several; they interfere; it forces prioritizing; only one can run long enough
Which reason ranks most important	The fourth, probably
How you judge a result	A statistical significance calculator, before against after
What a year of the cadence gives you	Fifty-two improvements
When parallel tests are allowed	When the channels are independent
Which end of the pathway to work on	The front, in general
Typical front and back rates given	One percent click; thirty percent give contact details
The order down the pathway	Call-out, value elements, offer, call to action, landing page, headline, sub-headline, image
Minimum sample size	Not established on this page
How long a test should run	No general figure; one practice finds a week typically enough at its own team size and spend
What each review does	In one practice: picks each platform's winner, logs the result, sets the next test against the version winning now
When to stop testing a step	In that practice, the leading version survives four attempts or a month: turn to the next constraint

9. What this page does not cover

Choosing between testing, adding volume and opening a new channel is on SOP 40. The daily volume that feeds any of this is on SOP 52 and SOP 29. The first five seconds of a video are on SOP 34, headlines are on SOP 21, the call-out is on SOP 32 and SOP 33, the middle of an advertisement is on SOP 39, and the page a lead lands on is on SOP 37.

Budgets and bid management are not covered on this page.