The A/B Testing console in the AccessIQ dashboard.
One experiment engine, three kinds of subject
Every other tool experiments on anonymous keys. AccessIQ experiments on identities — including the non-human ones.
Users
Deterministic, sticky variant assignment with enterprise statistics. A user always sees the same variant, even after weights change.
Organizations
Target an org and every sub-org beneath it with hierarchy-aware rules — “org_path starts with root/acme” reaches the whole tree.
AI Agents
Experiment on agent behavior itself. Variants carry prompt, model and policy config, and agents are targeted by their governed identity — model, trust tier, owning org.
From hypothesis to ship decision
Three steps, no second SDK.
Define variants
Split traffic across variants that carry a flag variation — or, for agents and login journeys, inline prompt / model / policy / journey config.
Target by identity
AccessIQ already knows the tenant, org, role, plan — and for agents, the model and trust tier. Target on any of it, server-side. No second SDK.
Decide with confidence
Sequential (always-valid) statistics, SRM detection, and guardrails tell you SHIP, HOLD, or KEEP RUNNING — with auto-stop when a guardrail is breached.
Enterprise-grade by default
The statistics, governance, and safety rails a regulated B2B team needs.
Always-valid statistics
Sequential (mSPRT) testing lets you peek anytime without inflating false positives. Two-proportion z-tests, Wilson CIs, and Welch’s t-test for revenue and continuous metrics.
Guardrails + auto-stop
Flag protected metrics as guardrails. A statistically significant drop forces a HOLD and stops the experiment automatically — before it hurts.
Sample-ratio mismatch
Built-in SRM detection (chi-square) catches broken assignment before you trust a bad result.
Approvals + audit
Optional approval gates with separation of duties — the creator can’t approve their own experiment — and every lifecycle event is written to the immutable audit log.
Segments + time-series
Break results down by any subject attribute, and watch the cumulative trend per variant over time.
Alerts
Slack-compatible webhooks and email fire the moment there’s a significant winner or a guardrail breach — deduped, so no noise.
Scheduling
Auto start and stop experiments at set times, so launches and wind-downs don’t depend on someone being awake.
One system, no glue
Entitlements, feature flags, and experimentation share the same identity graph. Release, gate by plan, and experiment without stitching tools together.
Wired into the surfaces no one else can reach
Because experimentation lives next to identity, you can test things a flag tool never sees.
Agent behavior experiments
Ship two prompts, two models, or two tool policies and let the winner emerge — with agents bucketed and targeted by their governed identity.
Explore agent identityAuth-journey experiments
A/B the login experience itself — provider order, passwordless-first, step-up — and measure login-success conversion. Fail-open, so login never breaks.
See Agent GuardThe overlap no competitor lives in
Flag vendors have no identity. Auth vendors have no experiments. AccessIQ has both.
| Capability | AccessIQ | Flag vendors | Auth vendors |
|---|---|---|---|
| A/B testing with enterprise statistics | |||
| Target by org hierarchy & role | |||
| Experiment on AI agent behavior | |||
| Experiment on the login journey | |||
| Sequential / always-valid testing | |||
| Built-in immutable audit of every change |
“Flag vendors” = LaunchDarkly / Optimizely. “Auth vendors” = Auth0 / Okta. Based on publicly documented capabilities.
Questions, answered
How is this different from LaunchDarkly or Optimizely?
Those tools experiment on anonymous keys — they have no idea who the user is, what org they belong to, or whether the subject is an AI agent. AccessIQ experiments sit on top of the identity graph, so you can target by org hierarchy, role, plan, or agent trust tier, and you get audit and authorization in the same platform.
What does “experiment on an AI agent” actually mean?
An experiment variant can carry inline config — a prompt, a model choice, a tool policy. When an agent asks the identity service which config to run, it’s sticky-assigned a variant and enriched with its governed identity (model, trust tier, owning org) as targeting attributes. You measure which agent configuration performs better, safely.
How do auth-journey experiments work?
On the login bootstrap, an anonymous visitor is assigned an auth-journey variant — provider order, passwordless-first, step-up policy — and a successful login is recorded as a conversion. Resolution is fail-open: if the experiment engine is ever unavailable, login is never blocked.
Can I peek at results before the experiment ends?
Yes. Sequential (always-valid) testing means you can check results as often as you like without inflating your false-positive rate — no waiting for a fixed sample size.