Skip to content

Benchmark FFE evaluation on the scale suites - #13485

Draft
leoromanovsky wants to merge 7 commits into
masterfrom
leo.romanovsky/ffe-scale-suite-benchmark
Draft

leoromanovsky wants to merge 7 commits into
masterfrom
leo.romanovsky/ffe-scale-suite-benchmark

Conversation

@leoromanovsky

@leoromanovsky leoromanovsky commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

What Does This Do

Adds JMH benchmarks that measure FFE flag evaluation on the opt-in scale and scale-json suites of the shared fixtures, and sets a baseline for master:

  • ScaleSuiteBenchmark evaluates the cases of each case file through the provider and reports time and allocation for each evaluation.
  • ScaleSuiteLoadBenchmark measures how long each suite's configuration takes to parse, and prints the heap that the parsed configuration retains.
  • ScaleSuiteTest runs the same cases in the unit tests, so a change that resolves a large configuration wrongly fails CI instead of only a manual benchmark run.

This is the base of a stack. Three sibling PRs are stacked on it, and each one reports its before/after numbers against the baseline below:

This PR has no change to shipped code. The submodule bump also brings in fixture main commits that add 8 core cases (one time-bounded holdout flag); CanonicalFixtureTest passes on all of them.

Motivation

We need flag evaluation in Java to win a performance-sensitive customer, and this PR is the foundation: a repeatable measurement for every change that follows. The canonical fixtures are small, so they do not show where an evaluation's cost grows with configuration size: list length, allocation count, rule depth, shard count, regex input length, or JSON value size. The suites in DataDog/ffe-system-test-data#78 cover these dimensions at N and N × M, up to JSON values of 4 MB. This PR runs our evaluator over them, so every performance change shows its effect on the same inputs, on the same machine.

Additional Notes

What the benchmark covers.

  • @Param cases holds all 17 case files: scale/flags, numeric, one-of, one-of-many, one-of-small-numbers, regex, regex-many, rules, rules-sweep, semver, splits, time-windows, variations, wide-context, and scale-json/json-shapes, json-sizes, json-values. The fixture evaluation-suites/scale/AGENTS.md describes each one.
  • With index=all, the operations go through the cases of a file in turn, so a row is the unweighted mean over the 2 to 13 cases of the file, and its slowest case dominates it: the json-sizes row is mostly its 4 MB case. A change that targets one case should also report that case with -p index=<n>; the setup prints the index of each case.
  • The setup evaluates every case once, and the teardown again after the measurement. Both throw on a wrong value or reason, on an error code, or on an expectation the check does not handle, so a run that resolves a case wrongly fails instead of reporting a number. All 108 cases pass on master, including the sharded ones.
  • ScaleSuiteTest evaluates every case twice in the unit tests, and fails when a case file of the fixtures is missing from the benchmark's list.
  • The separate evaluation-suites/regex suite (dialect and backtracking cases, not scale/regex above) is not here, because the JDK engine does not finish its backtracking cases. Evaluate feature flag regex conditions in linear time #13487 adds it.

Decisions.

  • Compare JSON numbers by value. master returns JSON integers as Double. Return read-only JSON flag values with exact integers #13484 makes them exact, and the core fixture Add JSON number precision fixture ffe-system-test-data#77 checks the type. Comparing by value lets one benchmark run on every PR in the stack. This check guards the measurement; CanonicalFixtureTest is the conformance check of the types.
  • Measure load as well as evaluation. An index, compiled patterns, or prebuilt JSON values move work to load time. The load benchmark shows that cost next to the evaluation gain. The retained-heap figure is the used heap after a full collection, with and without the parsed configuration, after a first parse that initializes the parser and as the smallest of three samples, so it is approximate (it repeats within a few KB).
  • Pin to the fixture PR head. The submodule points to the head of Add opt-in scale, scale-json, and regex evaluation suites ffe-system-test-data#78 (ec0ec05). Repin to its merge commit before this PR merges.
  • Rerun on a fixture change. The jmh task declares the suites as an input, so a repin reruns the benchmarks instead of reporting the task up to date.
  • No new dependency. The case files are read with jackson-core, which the module already uses.

Baseline on master. AWS workspace, Intel Xeon Platinum 8175M, 16 vCPUs, JDK 21.0.10. JMH AverageTime, 1 fork, 3 warm-up and 5 measurement iterations of 2 s, -prof gc. Each row is the mean over the cases of the file. The time and allocation figures predate the latest commit, which leaves the measured evaluation unchanged; the retained heap uses the revised method. The one-of-small-numbers row and the scale load row were measured later, after the fixtures added that file and its 2 flags; the earlier scale parse time was 95.8 ms, and 2 more flags out of 10,030 do not explain the difference, so treat the parse time as within this machine's run-to-run variation. The json-sizes row and the scale-json load row were measured again after the fixtures replaced the 10 MB value with a 4 MB one, with three forks while the machine was heavily loaded (load average about 70 on 16 vCPUs): their allocation and retained heap are exact, but compare their times only with runs made back to back on the same machine. One fork suits allocation and large time differences; for a time difference under about 15%, run three forks (-Pjmh.forks=3, or -f 3 with the JMH jar).

Case file Time per evaluation Allocation per evaluation
scale/flags (10,000 flags) 215 ns 330 B
scale/wide-context 685 ns 368 B
scale/time-windows 801 ns 368 B
scale/numeric 2.79 µs 1,968 B
scale/rules 3.29 µs 1,904 B
scale/one-of-small-numbers 12.5 µs 15,424 B
scale/semver 12.8 µs 15,971 B
scale/regex-many 21.0 µs 42,164 B
scale/splits 26.7 µs 42,267 B
scale/rules-sweep 28.5 µs 392 B
scale/variations 53.1 µs 28,067 B
scale/one-of 137 µs 98,869 B
scale/one-of-many 195 µs 393 B
scale/regex 1.82 ms 12,201 B
scale-json/json-values 36.0 µs 63,736 B
scale-json/json-shapes 1.11 ms 3.3 MB
scale-json/json-sizes 5.55 ms 6.9 MB

Through the client's context. The rows above pass the provider one reused context (evaluate). The OpenFeature client passes a new LayeredEvaluationContext on each call instead, and evaluateLayered measures that. It adds 48 B per evaluation for most cases. A rule on id costs much more when the context has no id, because the layered context builds its key set again on each call:

scale/wide-context case evaluate evaluateLayered
scale-id-wide-context, match (index 2) 1.24 µs, 368 B 11.4 µs, 11,296 B
scale-id-wide-context, miss (index 3) 1.12 µs, 368 B 13.9 µs, 11,360 B
Suite Configuration Parse time Allocated by one parse Retained heap
scale 7.8 MB, 10,030 flags 111 ms 90.5 MB 24.2 MB
scale-json 7.7 MB, 10 flags 74.6 ms 64.6 MB 38.2 MB

The stacked PRs address json-*, one-of*, and regex*. The baseline also shows the next candidates. splits allocates 42 KB per call (a new MessageDigest and hex string for each shard). semver allocates 16 KB (the attribute is parsed again for each condition). variations scans 1,000 allocations.

Run all case files, with both evaluation methods:

./gradlew :dd-openfeature:jmh -Pjmh.includes=ScaleSuiteBenchmark,ScaleSuiteLoadBenchmark -Pjmh.profilers=gc

Only the rows of the main table (evaluate), in about half the time:

./gradlew :dd-openfeature:jmh '-Pjmh.includes=ScaleSuiteBenchmark.evaluate$' -Pjmh.profilers=gc

Run one case:

./gradlew :dd-openfeature:jmhJar
java -jar dd-openfeature/build/libs/dd-openfeature-*-jmh.jar ScaleSuiteBenchmark -p cases=scale/splits -p index=0 -prof gc

Contributor Checklist

Jira ticket: [PROJ-IDENT]

@leoromanovsky leoromanovsky added type: feature Enhancements and improvements tag: performance Performance related changes tag: no release notes Changes to exclude from release notes tag: ai generated Largely based on code generated by an AI or LLM comp: openfeature OpenFeature labels Nov 17, 2026
@datadog-datadog-us1-prod

datadog-datadog-us1-prod Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 82.45% (+22.10%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 72d60ca | Docs | Give us feedback!

@dd-octo-sts

dd-octo-sts Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 14.86 s 14.84 s [-0.7%; +1.1%] (no difference)
startup:insecure-bank:tracing:Agent 13.64 s 13.70 s [-1.1%; +0.4%] (no difference)
startup:petclinic:appsec:Agent 17.14 s 16.44 s [+0.0%; +8.4%] (maybe worse)
startup:petclinic:iast:Agent 16.97 s 17.07 s [-1.5%; +0.4%] (no difference)
startup:petclinic:profiling:Agent 16.12 s 16.75 s [-7.8%; +0.3%] (no difference)
startup:petclinic:sca:Agent 17.09 s 16.98 s [-0.4%; +1.7%] (no difference)
startup:petclinic:tracing:Agent 16.11 s 15.76 s [-1.9%; +6.5%] (no difference)

Commit: 72d60cae · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

leoromanovsky and others added 2 commits November 17, 2026 00:47
Add ScaleSuiteBenchmark, which evaluates each case of the opt-in scale suite of the shared FFE fixtures. The setup checks every expected value and reason first. Pin ffe-system-test-data to the scale suite PR head.
The scale benchmark sat on the JSON value change and checked JSON
integers by type, so it could not set a baseline on master. It now
compares JSON numbers by value, covers every case file of the scale and
scale-json suites, can isolate one case with the index parameter, and
prints each case's index. A load benchmark measures parsing each suite's
configuration and prints the heap it retains, so changes that move work
to load time show their cost.

The submodule points to the scale-suite fixture branch, which adds the
N x M dimensions and JSON values up to 10 MB.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@leoromanovsky
leoromanovsky force-pushed the leo.romanovsky/ffe-scale-suite-benchmark branch from 5a15d5c to c18784a Compare November 17, 2026 00:54
@leoromanovsky
leoromanovsky changed the base branch from leo.romanovsky/ffe-json-frozen-values to master November 17, 2026 00:54
@leoromanovsky leoromanovsky changed the title Benchmark the FFE scale evaluation suite Benchmark FFE evaluation on the scale suites Nov 17, 2026
@leoromanovsky leoromanovsky added the crucible: reviewing A Crucible review has started; this PR has reviewer attention. label Nov 17, 2026 — with ddtool CLI
@leoromanovsky leoromanovsky removed the crucible: reviewing A Crucible review has started; this PR has reviewer attention. label Nov 17, 2026
The scale benchmarks failed the forbidden API check on System.out, and
nothing outside a manual JMH run checked the scale cases, so a sibling
change that resolves a large configuration wrongly would go unnoticed.

- Add ScaleSuiteTest: evaluate every case of the scale and scale-json
  suites twice, and fail when the benchmark misses a case file. The
  cases, their check and the suite reader move to the test sources and
  the benchmarks reuse them; the reader uses JsonReading.
- Allow System.out in the benchmark setups (@SuppressForbidden).
- Check every case again after the measurement, fail on an error code,
  and reject expectations the check does not handle.
- Rerun the jmh task when the fixtures change.
- Measure retained heap after a first parse, as the smallest of three
  signed samples.
- Describe a row as the unweighted mean over the cases of a file, say
  that hooks are not measured and when to run three forks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copy link
Copy Markdown
Contributor Author

Reminder: before merge, repin the fixture submodule to the commit that DataDog/ffe-system-test-data#78 lands as on main, and confirm the scale suites are present there (the checklist item in the description). Two things depend on that order:

Check after the repin:

git -C dd-smoke-tests/openfeature/src/test/resources/ffe-system-test-data fetch origin main
git -C dd-smoke-tests/openfeature/src/test/resources/ffe-system-test-data merge-base --is-ancestor HEAD origin/main && echo on-main
./gradlew :dd-openfeature:test --tests com.datadog.openfeature.ScaleSuiteTest

leoromanovsky added a commit that referenced this pull request Nov 17, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The fixture PR now checks that the committed scale files match the
generator and fixes its docs. The case and configuration files are
unchanged, so the benchmark numbers do not change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
leoromanovsky added a commit that referenced this pull request Nov 17, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
leoromanovsky added a commit that referenced this pull request Nov 17, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
leoromanovsky added a commit that referenced this pull request Nov 17, 2026
Brings in ScaleSuiteTest, the benchmark hardening, and the fixtures pin
f425d578. ScaleSuiteTest now also checks the four case files of the
regex suite, so the cross-SDK regex contract runs in CI. ScaleCase
accepts an expected errorCode, so the PARSE_ERROR cases of
unsupported-syntax.json are compared, not skipped. semantics and
unsupported-syntax are checked but not benchmarked.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The fixtures add a case file with ONE_OF lists of the integers 1 to
1000 and of the powers of two up to 2^52, which an index keyed on exact
doubles must still match. Pin them and benchmark the file; its 8 cases
pass on master.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
leoromanovsky added a commit that referenced this pull request Nov 17, 2026
Brings in the scale/one-of-small-numbers case file and the fixtures pin
08e86993. The benchmark and ScaleSuiteTest keep both the new case file
and the regex suite.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The OpenFeature client passes the provider a new LayeredEvaluationContext
on each call, while the benchmark reused one MutableContext. A rule on
`id` when the context has no `id` reads the layered context's key set,
which it builds again on each call, so the benchmark did not show what
applications pay for it.

evaluateLayered wraps each case's context the way the client does, as
ExperimentationOptOutBenchmark.evaluateFreshLayeredContext does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
leoromanovsky added a commit that referenced this pull request Nov 17, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
leoromanovsky added a commit that referenced this pull request Nov 17, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
leoromanovsky added a commit that referenced this pull request Nov 17, 2026
Brings in the evaluateLayered benchmark, which evaluates with the fresh
layered context that the OpenFeature client builds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The fixtures replace the 10 MB JSON flag with a 4 MB one, which brings
the scale-json configuration from 14 MB down to 8 MB. The other cases
are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
leoromanovsky added a commit that referenced this pull request Nov 17, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp: openfeature OpenFeature tag: ai generated Largely based on code generated by an AI or LLM tag: no release notes Changes to exclude from release notes tag: performance Performance related changes type: feature Enhancements and improvements

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant