Are you buying new growth or just taking credit for it?

Attributed conversions can rise while total growth barely changes. That is worth investigating, but it does not prove the campaign is taking credit for organic demand. Other channels, seasonality, product changes, or reporting gaps could explain the pattern.

The budget question is causal: how many useful outcomes would disappear if this advertising were absent or reduced?

Attribution assigns credit under a set of rules. Incrementality estimates the difference advertising makes against a credible alternative. You need both perspectives, but they answer different questions.

Start with a sentence such as: “Should we continue buying this partner’s new-user placements at the current terms?” That is more useful than “Does marketing work?”

Specify the intervention you can actually test. Removing a partner, reducing one campaign’s spend, and changing an ad’s creative are different experiments. Each produces an answer about its own contrast.

Choose a business outcome measurable in both test and control, such as completed registrations, new paying users, or contribution margin over a defined period. A report containing only conversions attributed to the tested partner cannot, by itself, tell you what the control group did elsewhere.

Define existing-user treatment, the conversion window, and how late outcomes will be handled. If the decision is about acquiring new customers, mixing repeat purchases into the main outcome changes the question.

Also name the smallest effect that would change your decision. A study capable of detecting only a dramatic lift may be unsuitable when a much smaller improvement would justify the spend.

A randomized audience holdout assigns eligible people to an advertising treatment or a control condition. The analysis should follow the assigned groups under the study’s design, rather than comparing people who happened to see or click an ad with those who did not. Exposure and clicking can reflect pre-existing interest.

TikTok describes Conversion Lift as a managed service for eligible accounts, with account representatives helping define the study and establish test and control groups. Its requirements depend on spend, objective, and duration; do not assume it is available as a self-service option for every app. A platform lift study measures only that platform’s advertising, not a partner’s or another channel’s. Ask which app outcomes and measurement coverage your proposed study can support. TikTok’s Conversion Lift guide

A geographic experiment changes advertising across assigned regions and measures the difference in business outcomes. It can be useful when a user-level holdout is impractical. It needs sufficient comparable regions, reliable outcome measurement, and a defensible analysis, not simply one strong city versus one weak city. Google’s research describes random assignment of regions as a way to guard against hidden differences. Google Research on geo experiments

A model-based comparison may estimate what would have happened without a change using historical relationships and comparison markets. Its credibility depends on those relationships holding and on other changes being accounted for. Label the assumptions and the design honestly; a modeled counterfactual is not automatically a randomized experiment.

An ordinary A/B split test answers another question. TikTok’s split-testing tool compares variations using separated audience groups; comparing two ad treatments does not by itself measure advertising against no advertising. TikTok’s split-testing guide

Before authorizing a study, ask the analyst or provider to explain:

• Assignment: what is randomized, and when?
• Coverage: whose outcomes can be measured, and who is missing?
• Power: given the baseline rate and variation, what effect can this design detect?
• Contamination: can control users receive the tested promotion through another account, partner, or device?
• Operational changes: will pricing, onboarding, distribution, or other marketing differ between groups?
• Timing: when does delivery end, when do outcomes mature, and when is the final analysis?

Keep measurement comparable across groups. A design that observes treatment outcomes more completely can generate a misleading apparent difference.

Geographic boundaries are not impermeable either. People travel and delivery can spill across markets. Google’s 2011 geo-experiment paper explicitly discusses ad-serving inaccuracy and travel across geo boundaries as measurement concerns. Google’s geo-experiment methodology

Document foreseeable exclusions and analysis rules before results arrive. Do not repeatedly inspect the numbers and stop at the first attractive result unless the statistical design explicitly supports that stopping rule.

Consider this illustrative teaching example, not a campaign result. Suppose a valid study estimates 80 additional completed registrations from $4,000 of relevant incremental spend. Its stated 95% confidence interval for additional registrations runs from −40 to +200. The interval is invented to demonstrate interpretation, not calculated from an underlying dataset.

The point estimate implies $50 per incremental registration: $4,000 ÷ 80. That number alone is a weak foundation for a growth decision. The interval includes zero and negative effects as well as a useful positive effect.

For a corresponding two-sided test, a 95% interval that contains zero means a zero effect is not rejected at the 5% level. It does not prove the advertising has no effect. NIST’s explanation of confidence intervals and tests

Suppose your own illustrative decision threshold is no more than $40 per additional registration. At $4,000 of spend, that requires at least 100 additional registrations. The point estimate of 80 misses that threshold, while the uncertainty range includes results above it. The study has not settled whether this condition is economically worthwhile.

Do not turn the endpoints into a tidy cost range by dividing spend by −40 and +200. When the interval for additional registrations includes zero, that ratio becomes unstable and has no useful finite interval of that simple form. Ask the analyst for an appropriate uncertainty treatment.

Also separate statistical detectability from business value. A precisely measured positive effect can still cost too much.

Your brief should fit on one page:

• Decision and exact advertising change being tested
• Eligible population, assignment unit, and control condition
• Primary outcome, measurement coverage, and maturity window
• Minimum economically useful effect and planned power assessment
• Exposure limit, duration, and stop conditions
• Contamination checks and concurrent changes to record
• Analysis method, uncertainty reporting, and accountable reviewer

When the report arrives, compare it with that brief. Ask whether assignment held, delivery differed as intended, exclusions changed, and the measured population matches the population you plan to scale. Check whether the reported lift is absolute or relative and whether the cost calculation uses the matching spend contrast.

If a credible experiment is currently infeasible, keep attribution useful for operational decisions and cohort economics. Limit the commercial commitment, document the causal uncertainty, and identify what would make a stronger study possible. A before-and-after chart can suggest a hypothesis; it cannot remove that uncertainty on its own.

The outcome you need is a defensible budget choice for a defined setting. “Promising but unresolved” can be the honest result, provided it leads to a bounded decision rather than an unqualified claim of new growth.