A/B tests and feature flags

The Lab runs A/B experiments and feature flags without cookies: your page asks for a variant or a flag's value, works it out itself, and LogSure counts who saw which and who reached the goal.

Make an experiment

Admins make experiments in the console's Lab, for one app:

• Key: what the page asks for, such as checkout-button: letters, digits, _ and -, up to 60 characters.
• Variants: 2 to 5, each with a key and a weight. The first is the control.
• Goal: a custom event, a page (* matches anything), or a saved funnel.
• Guardrails: other goals that must not get worse, each with whether higher or lower is better.
• Hypothesis: what you expect, for the team.

An experiment starts as a draft. Start it to begin counting, and stop it when you are done. The key and the variants are fixed once it has started.

Ask for the variant

logsure.variant('checkout-button', { id: user.id }).then((v) => {
  if (v === 'green') showGreenButton();
  // null while the experiment is not running: show your default
});

• With an id (your own user ID or any stable string), the same person always gets the same variant: it is a hash of the key and the id, worked out in the page. LogSure never stores the id.
• Without an id, the variant is picked once per page view, so a returning visitor may see both. Pass an id for anything people come back to.
• Counting: the tracker sends one $exposure event and tags the page's later events with exp.<key>. The running experiments are fetched once and cached for a minute.

From a back-end

Send the same with the app's server key: $exposure with props: { exp, variant, unit }, and your goal events with the same unit, an opaque id of your own. To assign like the tracker does, take FNV-1a 32 of key + ":" + id as a share of 2³² and walk the cumulative weights.

curl -sS -X POST 'https://logsure.example.com/_care/event' \
  -H "authorization: Bearer $LOGSURE_KEY" -H 'content-type: application/json' \
  --data '{"app":"example-shop","name":"$exposure",
           "props":{"exp":"checkout-button","variant":"green","unit":"u_1042"}}'

Read the results

• The unit is a visitor and a day: a visitor counts once per UTC day and converts by reaching the goal after seeing a variant, the same day.
• Per variant: visitors, conversions and rate; against the control, the difference and the uplift with their 95% intervals, a p-value and the probability to beat control.
• The verdict: not started, collecting (until every variant has 100 visitors and 10 conversions), keep running, a variant ahead (p below 0.05 and at least a 95% chance to beat control), or the control ahead.
• Sample ratio mismatch: when the split of visitors does not match the weights, the results say so, and should not be trusted.
• Guardrails: each with the same numbers, and the variants that harm it marked.
• Over time and by segment: the rates cumulatively by day, and the results by device, country, browser, system or source.

The same is in the API (lab), the iPhone and Mac apps, the terminal (:lab) and over MCP (list_experiments, get_experiment_results, and with settings access create_experiment, update_experiment, start_experiment, stop_experiment).

Feature flags

Admins make flags under Lab › Flags. A new flag is off until you switch it on.

• Type: boolean, text, number or JSON, fixed once made. Every value the flag serves is of that type; a boolean flag has on and off.
• Values: up to 10, with a default, served when the flag is off and when nothing else applies.
• Kill switch: a flag that is off serves its default to everyone, within about 30 seconds.
• History: every change bumps the flag's version and is in its history and the audit log. Deleting a flag is for the web console only.

How a flag decides

1. Off: the default.
2. Rules, in order: the first whose conditions match (all or any) serves its value. Conditions test the attributes you pass, and id: equals, not equals, in a list, not in a list, contains, starts with, ends with, greater or less than, semver comparisons, and exists. An attribute you didn't pass matches no condition, negated ones included.
3. Without the score (the default for a new flag): the fallthrough value. A score you pass is ignored.
4. With "use score" on: the first range holding the score serves its value. Ranges include both ends: on for scores 10 to 50 means 9 → off, 10 → on, 50 → on, 51 → off. No score, or a score in no range, serves the default.

The score is a number from 0 to 100 that your app chooses, with whatever meaning you like: engagement, a plan tier, a device class, or a random bucket of your own. LogSure never computes or interprets one. Values outside 0 to 100 are clamped, and anything but a number counts as no score.

In the page

<script defer data-app="example-shop" data-flags="true" src="/_care/script.js"></script>

await logsure.flagsReady();
if (logsure.flag('new-checkout', { id: user.id, attrs: { plan: user.plan, version: '2.4.1' }, fallback: false })) {
  showNewCheckout();
}
const headline = logsure.flag('pricing-copy', { score: user.engagement, fallback: 'Start now' }); // a text flag

• The config: the tracker loads the app's flags once from /_care/flags: public, cached for 30 seconds, the same for every visitor, with no visitor data. data-flags="true" loads it with the page.
• logsure.flag(key, { id, score, attrs, fallback }) answers at once with the value in the flag's type, or the fallback until the config has loaded. Your id, score and attributes never leave the page.
• More: logsure.flags(opts) answers every flag, logsure.flagDetail(key, opts) also says which value and why (off, rule, score, fallthrough or default), and logsure.flagsReady() resolves once the config is in.
• Exposures: with the flag's exposure setting on, the first evaluation in a page sends a $flag event with the flag and its value.

On a server

Fetch /_care/flags with the server key and evaluate it the same way, or ask LogSure to:

curl -sS -X POST 'https://logsure.example.com/_care/flags/evaluate' \
  -H "authorization: Bearer $LOGSURE_KEY" -H 'content-type: application/json' \
  --data '{"id":"u_1042","score":30,"attrs":{"plan":"pro"}}'

Nothing sent there is stored.

Measuring a flag

An experiment can measure a flag: pick it under Measure a flag in the experiment editor, and the experiment takes the flag's values as its variants and counts its $flag events as exposures. The flag stays fixed while the experiment runs.

Flags are in the API (lab) and over MCP: list_flags and get_flag (with its history), and with settings access create_flag, update_flag, toggle_flag and set_flag_rollout.