Experiments and measured outcomes
Owners and admins can create a draft with a hypothesis, a fixed key, two to eight named variants and whole-number weights totaling 100%. The first variant is the baseline. Choose one primary outcome and optionally target a segment by platform, build or behaviour. A game can keep 100 experiments and run ten at a time. Members and viewers can read; incoming shared-game access does not include experiments.
Targeting and sample size
A targeted experiment keeps a device once it has matched. Its first assignment is recorded, and a device that later leaves the target segment (a new build, a lower spend) stays in its arm until the experiment stops. Stopping ends every recorded assignment. Choose Include new builds on a build-targeted draft to also enrol devices on builds first seen after the start: the builds already seen are listed at the start, so the experiment needs the event store then, and a game with more than 1,000 builds must target one build.
Set an optional target sample size, 10 to 10,000,000 exposed players per variant. The draft form estimates one for a yes or no outcome: about 16 × p × (1 − p) ÷ d² players per variant to detect a change of d from a baseline rate p, at 80% power and 5% significance. The creator and the studio owner get one email when every variant reaches the target, and one when the experiment stops. Neither email concludes or promotes anything.
Config rules and assignment
Add experiment-variant rules to Remote config before starting. On a config fetch, running experiments assign a variant using the SHA-256 hash of experimentKey + ':' + deviceId, reduced modulo 10,000 and compared with cumulative weights. The result is deterministic for the same key, device and weights; changing weights would change assignments, so started experiments keep their allocation, targeting and outcome fixed. Start a new experiment key for a new test. Device IDs are the unit of assignment, not verified people.
Attach rs_exp_<experiment_key> to gameplay events with the assigned variant name. From Godot 0.5.0, JavaScript 0.7.0, LÖVE 0.7.0, Unity 0.5.0, Unreal 0.5.0, C++ 0.5.0, Odin 0.6.0, iOS 0.6.0 and Android 0.5.0, every SDK does this for you: the first typed config read sends one rs_exposure event and tags the events after it (per session, or per run in C++, iOS and Android), a fetch alone tags nothing, and a well-formed rs_exp_<key> your game sets itself is left alone. On an older SDK, tag events yourself. Variant names start with a lowercase letter and use lowercase letters, digits, underscores or hyphens, up to 40 characters. Malformed exposure labels are removed and counted as reserved-key warnings; the rest of the event stays usable. A config fetch alone is not a measured exposure. Exposure tags add no extra event charge.
Stopping removes assignments from future fetches; cached clients keep their last values until they refresh. Results remain readable. Concluding can name a winner without promotion, or explicitly promote the winner's bound values to config defaults. The review shows before-and-after defaults and sends the reviewed config version. A concurrent change refuses promotion so you can review again. Existing segment rules still take precedence where applicable.
Results and comparisons
Results include real-player events after the first recorded exposure and within the experiment's start/stop window. Players tagged with more than one variant are excluded. Purchases retain historical transaction deduplication and remain in one selected currency. Retention uses complete UTC days after exposure; immature cohorts are excluded. New-player share uses the first-seen registry and excludes players not yet recorded there. Synthetic playtests never enter experiment results.
Comparisons use the labelled player-level outcome: sessions, active days, mean session seconds, errors, currency revenue, transaction counts, resource amounts, progression failures, purchasing/new-player shares, funnel conversion or day-1/day-7 return rates. These are not differences in aggregate DAU, pooled session medians or errors per affected player. Daily charts group outcomes by first exposure day. Experiment results are limited to the latest 90 days, even when raw-event storage is extended, with partial windows identified. Reads cap at 50,000 exposed players, five million scanned rows and 15 seconds; they fail visibly rather than silently sampling.
Continuous outcomes use a Welch t comparison; binary outcomes use a two-proportion z comparison when successes and failures are sufficiently represented. Results show treatment-minus-baseline differences and 95% intervals, with a caution below 100 measured players per arm. Sparse or constant samples do not receive fabricated precision. Intervals are unadjusted for repeated looks and multiple variants, and changes to other game settings can affect the comparison. The system never selects or promotes a winner automatically. Methods: NIST two-sample comparisons and difference of proportions.
Over MCP
MCP list_experiments reads definitions and get_experiment_results reads outcomes. REST GET /api/v1/data/games/{gameId}/experiment-results?experiment_key=... uses the same scoped programmatic token and standard metered rate. These reads do not mutate lifecycle state or config. They are not added to the hosted analyst's prompt. Experiment results and configuration changes do not automatically become studio-memory findings or investigation evidence.