Feature flags: how flag systems work, and how to keep them from becoming debt

Key points

• Pete Hodgson’s four flag types, release, experiment, ops and permission, differ in how long they live and how often their decision changes. Manage them differently.
• Evaluate in-process where you can: the server SDK holds the rules in memory and evaluation is a function call, not a network round trip.
• Hash the flag key together with a stable user id for percentage rollouts, so a user keeps the same answer and raising the percentage only adds users.
• OpenFeature, a CNCF project, standardizes the evaluation API: providers, an evaluation context, hooks, and a default value returned instead of an exception on any error.
• Every flag needs a safe default written in code, because one day the flag service or its cache will not answer.
• A release flag is debt from the day it is created. Give it an owner and an expiry date, and fail a check when it outlives that date.

What a feature flag system does

A feature flag is a decision point in code whose answer comes from outside the code. The point of it is to separate deploying code from releasing a feature: unfinished work can be merged and shipped dark, then turned on for staff, then for a slice of users, then for everyone, without another deploy. That is what lets teams practice trunk-based development and continuous delivery, covered in what CI/CD is.

Pete Hodgson’s article “Feature Toggles (aka Feature Flags)” on martinfowler.com names the parts. The toggle point is the if in the code. The toggle router decides which way it goes, using toggle configuration and, for dynamic flags, the context of the request. A flag system at scale is those three things made into a service:

• A control plane where people create flags, write targeting rules and change percentages, with permissions and an audit log.
• A distribution layer that gets the current rules to every running app, by streaming or polling, often through relays or edge caches.
• An SDK in each app that evaluates a flag for a context and returns a value, fast and without throwing.

Hodgson also advises keeping flag names out of business logic: put each decision behind a named method, so the code asks features.useNewCheckout(user) rather than reading a flag key in twenty places. That one habit makes both testing and removal much easier.

Four types of flag, managed differently

Hodgson sorts flags along two axes, how long they live and how dynamic their decision is, and finds four categories that behave very differently. Mixing them up is where most flag pain starts: a release flag treated like a permanent setting turns into dead code, and a kill switch treated like a release flag gets deleted just before the incident that needed it.

The lifetime and dynamism columns follow Hodgson’s article; the owner column is common practice rather than his. A useful rule follows from the table: long-lived permission flags are really product configuration and often belong in your own entitlement model, while short-lived release flags belong in the flag system with an expiry date attached.

Where flags are evaluated: server, client, local, remote

Evaluation is the act of turning a flag key and a context into a value. There are two designs, and vendors document them clearly.

• Local evaluation, used by server-side SDKs. LaunchDarkly’s docs say server-side SDKs receive the complete rule set for an SDK key, evaluate flags in-process and, by default, keep a streaming connection open to receive changes. Unleash’s docs say its backend SDKs evaluate locally, cache flag data in memory and sync with the server at intervals, so user data stays inside the SDK. Evaluation costs a function call and keeps working if the flag service goes away.
• Remote evaluation, used by client-side SDKs in browsers and mobile apps. The rules can contain targeting lists, such as account ids or email domains, which should not be shipped to every browser. So the client SDK sends one context to the vendor or to a relay, and receives only the evaluated values for that user. LaunchDarkly gives that as the reason client-side SDKs cannot download the whole rule set; Unleash’s frontend SDKs get evaluated flags from its edge service or frontend API.

What to avoid is a server calling a remote flag API on every request. It adds latency and a hard dependency to the hottest path in the system, and it is what local evaluation exists to remove. If you build your own flag system, copy the shape: rules in memory, pushed or pulled in the background, evaluation as a pure function of rules and context.

Percentage rollouts: deterministic bucketing

A “10 percent rollout” must not mean a coin flip per request, or a user would see the new checkout on one page load and the old one on the next. Instead, the SDK hashes a stable identifier into a bucket and turns the flag on for buckets below the percentage. Unleash documents this as stickiness: it hashes a context field, by default the user id and then the session id, together with a group id that defaults to the flag’s name, using MurmurHash, into a number from 0 to 100. The sketch below does the same with SHA-256 so it runs anywhere.

import hashlib

def bucket(flag_key: str, user_id: str) -> int:
    """Map (flag, user) to a stable bucket in 0..9999."""
    digest = hashlib.sha256(f'{flag_key}:{user_id}'.encode()).digest()
    return int.from_bytes(digest[:8], 'big') % 10_000

def is_enabled(flag_key: str, user_id: str, percent: float) -> bool:
    return bucket(flag_key, user_id) < percent * 100   # 25.0 percent -> buckets 0..2499

users = [f'user-{i}' for i in range(100_000)]
for pct in (1, 10, 25, 50):
    on = sum(is_enabled('new-checkout', u, pct) for u in users)
    print(f'{pct:>3}% rollout: {on:>6} of {len(users)} users ({on / len(users):.2%})')

at_10 = {u for u in users if is_enabled('new-checkout', u, 10)}
at_25 = {u for u in users if is_enabled('new-checkout', u, 25)}
print('everyone in at 10% still in at 25%:', at_10 <= at_25)
print('same answer on a second call:', all(is_enabled('new-checkout', u, 10) == (u in at_10) for u in users))

other = {u for u in users if is_enabled('new-search', u, 10)}
print(f'users in both 10% rollouts: {len(at_10 & other)} (independent flags: about 1,000)')

Running it prints:

  1% rollout:   1016 of 100000 users (1.02%)
 10% rollout:   9937 of 100000 users (9.94%)
 25% rollout:  25212 of 100000 users (25.21%)
 50% rollout:  50090 of 100000 users (50.09%)
everyone in at 10% still in at 25%: True
same answer on a second call: True
users in both 10% rollouts: 1024 (independent flags: about 1,000)

Four properties come out of this, and each one matters in practice:

• Sticky: the same user gets the same answer on every call and every server, with no state stored anywhere.
• Monotonic: raising the percentage from 10 to 25 adds users and removes none, because a user’s bucket does not move. Everyone already on the new path stays on it.
• Independent across flags: because the flag key is part of the hash, two 10 percent rollouts overlap in about 1 percent of users, as with independent coin flips. Hash the user id alone and the same unlucky users get every experiment at once.
• Close, not exact: the counts are within a fraction of a percent of the target at this size. For small audiences, the noise is larger, which matters for experiments.

The identifier is a design decision. Bucket on the user id and a user keeps the same answer across devices; bucket on a session or device id and a logged-out visitor gets a stable answer that may change when they log in. For B2B products, bucketing on the account id keeps a whole team on the same version. Changing the salt, here the flag key, reshuffles everyone, which you want for a new experiment and never in the middle of a rollout.

Targeting rules and the evaluation context

Percentage rollouts are usually the last step of a rule list. A typical flag evaluates, in order: is the flag on at all; is this user individually targeted; does the user match a rule, such as plan is enterprise or country is in a list or email ends with the company domain; and if nothing matched, the default rollout, which can be a percentage. The first match wins. Rules reference attributes, so the SDK needs those attributes at evaluation time.

That bundle of attributes is the evaluation context. OpenFeature defines it as ambient information for flag evaluation and gives it a targetingKey, the identifier of the subject being evaluated, which providers use for fractional rollouts and per-user overrides, plus custom fields. Context can be set at several levels and is merged, with later levels winning: global, then per transaction or request, then per client, then per call, then anything a before hook adds. Put only what rules need into the context. With local evaluation it never leaves the process; with remote evaluation it is sent to whoever evaluates.

OpenFeature: a standard evaluation API

OpenFeature is a vendor-neutral specification and set of SDKs for flag evaluation. It was accepted into the CNCF in June 2022 and moved to the Incubating level in November 2023. The idea is the same as with logging or tracing APIs: application code calls one API, and a provider plugs in the actual flag backend, a vendor, an open-source server or a file.

• Providers: the API sets a default provider, can bind other providers to named domains, calls the provider’s initialize and shutdown, and offers a set-and-wait call that returns once the provider is ready or in error.
• Evaluation API: typed methods for boolean, string, number and structure flags, each taking a flag key, a default value, an optional evaluation context and options. Detail methods also return the variant, a reason, an error code and flag metadata.
• Defaults on failure: requirement 1.4.10 says evaluation methods must not throw or terminate abnormally, and must return the default value when something goes wrong, with the error code filled in.
• Hooks: code that runs at before, after, error and finally stages around each evaluation, registered globally, per client, per call or by the provider. before hooks can add context; the after-stages run in reverse order, like a stack. Hooks are where logging, metrics, tracing and validation go.

Here is what an evaluation looks like through an OpenFeature-style API. The names follow the JavaScript server SDK; other languages expose the same shape.

import { OpenFeature, type Hook } from '@openfeature/server-sdk';
import { VendorProvider } from './flags/provider'; // any OpenFeature-compatible provider

// Runs around every evaluation: record which variant was served, and why.
const recordVariant: Hook = {
  after(hookContext, details) {
    console.info('flag_evaluated', { flag: details.flagKey, variant: details.variant, reason: details.reason });
  },
  error(hookContext, err) {
    console.warn('flag_fell_back_to_default', { flag: hookContext.flagKey, error: String(err) });
  },
};

await OpenFeature.setProviderAndWait(new VendorProvider());
OpenFeature.addHooks(recordVariant);
const flags = OpenFeature.getClient('checkout');

export async function checkoutVersion(user: { id: string; plan: string; country: string }) {
  const details = await flags.getBooleanDetails('new-checkout', false, {
    targetingKey: user.id, // the stable id a percentage rollout hashes
    plan: user.plan,
    country: user.country,
  });
  // Does not throw: if the provider is down or the flag is missing, value is the default (false).
  return details.value ? 'v2' : 'v1';
}

The spec also says evaluation methods should not write logs themselves, because they run on hot paths, which is why logging lives in a hook you choose to add. Swapping vendors then means changing the provider line, not every call site.

Propagation, caching and safe defaults

A flag change travels from the control plane to every SDK. With streaming, a server SDK holds an open connection and gets changes as they happen; with polling, it fetches the rules on an interval and may be stale for up to one interval. Large deployments add relays or edge services that hold the rules close to the apps, so thousands of instances do not all connect to the vendor. Each SDK keeps the last rules it received in memory, which is a cache with all the usual questions about freshness and invalidation; caching strategies covers those trade-offs in general.

Plan for three failure cases separately:

• The flag service goes down while apps are running. With local evaluation, SDKs keep serving the last rules they have. Unleash’s docs say initialized clients keep working without a connection to the server. Changes made during the outage arrive when the connection returns.
• An app starts while the flag service is down. It has no rules yet. Some SDKs can bootstrap from a file or environment, which Unleash documents; otherwise every flag returns the default value written in the code.
• The flag is missing or the rules are malformed. OpenFeature-style SDKs return the default and report an error code instead of throwing.

So the default in code is the behavior you get when everything else fails, and it should be the safe one. For a release flag that is usually off, the old path. For a kill switch it is the normal, not-killed state, so an outage of the flag service does not switch off a working feature. For a permission flag it is the least access. Write the default deliberately at every call site, or better, once per flag behind the named decision method.

Testing flag combinations and auditing changes

Ten boolean flags make 1,024 combinations, and nobody tests them all. Hodgson’s advice is to test the configuration you expect to be live, the fallback configuration with the new flags off, and a configuration with all flags on to catch regressions in future releases; most flags do not interact, so full combinatorial testing is rarely needed. For flags that do interact, test those pairs explicitly. In unit tests, inject flag values through a test provider or the decision object instead of calling the real service.

A flag change is a production change that skips the deploy pipeline, so it needs the record a deploy would leave. Keep an audit log of who changed which flag, in which environment, from what to what, when and why, and require approval for flags that touch money, permissions or data. When an incident starts, the first question is often “what changed?”, and flag changes belong on the same timeline as deploys. Record the served variant on logs and traces through an after hook, the way LLM observability and tracing attaches model and prompt versions to spans, so a bad variant can be found from the trace.

Flag debt: expiry dates, linting and cleanup

Every flag is a branch the code carries until someone deletes it. A release flag left at 100 percent is dead code with a runtime dependency, and old flags make the code harder to read and test. Hodgson calls flags inventory with a carrying cost. It is technical debt in its plainest form, and it accumulates quietly because removing a flag is never urgent.

• Create the removal task with the flag. Hodgson suggests adding it to the backlog the moment a release flag is introduced.
• Give short-lived flags an expiry date and an owner. Hodgson describes time bombs: a test that fails, or an app that refuses to start, once a flag has outlived its date.
• Cap the inventory. Hodgson suggests a limit on how many flags may exist at once, so adding one means removing another.
• Lint for stale flags. Search the code for flag keys and compare with the flag service: keys in code that the service no longer knows, flags at 100 percent or 0 percent for a long time, and flags with no evaluations recently are all candidates for removal.
• Automate the deletion. Uber’s Piranha tool rewrites code to remove a stale flag and sends the change to the flag’s author. The authors report that from Dec 2017 to May 2019 it generated cleanup diffs for 1,381 flags, 17 percent of all flags, and that 65 percent of those diffs landed without changes.

Removal has an order: roll the flag fully to the winning value, wait until no evaluations of the old value happen, delete the toggle point and the losing branch in code, deploy, and only then archive the flag in the service. Deleting the flag in the service first turns every remaining call site into a default-value evaluation, which may not be the value users currently see.

BYD Tech builds web apps and their backends: the code lives in the client’s GitHub from day one, the founders are on the calls, and AI-assisted work is reviewed by an engineer.