Repository navigation
Benchmark FFE evaluation on the scale suites - #13485
Draft
leoromanovsky wants to merge 7 commits into
Draft
leoromanovsky wants to merge 7 commits into
leoromanovsky wants to merge 7 commits into
Conversation
Contributor
🟢 Java Benchmark SLOs — All performance SLOs passed
PR vs. master results
Commit: Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion. |
Add ScaleSuiteBenchmark, which evaluates each case of the opt-in scale suite of the shared FFE fixtures. The setup checks every expected value and reason first. Pin ffe-system-test-data to the scale suite PR head.
The scale benchmark sat on the JSON value change and checked JSON integers by type, so it could not set a baseline on master. It now compares JSON numbers by value, covers every case file of the scale and scale-json suites, can isolate one case with the index parameter, and prints each case's index. A load benchmark measures parsing each suite's configuration and prints the heap it retains, so changes that move work to load time show their cost. The submodule points to the scale-suite fixture branch, which adds the N x M dimensions and JSON values up to 10 MB. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
leoromanovsky
force-pushed
the
leo.romanovsky/ffe-scale-suite-benchmark
branch
from
November 17, 2026 00:54
5a15d5c to
c18784a
Compare
leoromanovsky
changed the base branch from
leo.romanovsky/ffe-json-frozen-values
to
master
November 17, 2026 00:54
4 tasks done
This was referenced Nov 17, 2026
This was referenced Nov 17, 2026
The scale benchmarks failed the forbidden API check on System.out, and nothing outside a manual JMH run checked the scale cases, so a sibling change that resolves a large configuration wrongly would go unnoticed. - Add ScaleSuiteTest: evaluate every case of the scale and scale-json suites twice, and fail when the benchmark misses a case file. The cases, their check and the suite reader move to the test sources and the benchmarks reuse them; the reader uses JsonReading. - Allow System.out in the benchmark setups (@SuppressForbidden). - Check every case again after the measurement, fail on an error code, and reject expectations the check does not handle. - Rerun the jmh task when the fixtures change. - Measure retained heap after a first parse, as the smallest of three signed samples. - Describe a row as the unweighted mean over the cases of a file, say that hooks are not measured and when to run three forks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
Author
|
Reminder: before merge, repin the fixture submodule to the commit that DataDog/ffe-system-test-data#78 lands as on
Check after the repin: |
leoromanovsky
added a commit
that referenced
this pull request
Nov 17, 2026
The fixture PR now checks that the committed scale files match the generator and fixes its docs. The case and configuration files are unchanged, so the benchmark numbers do not change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
leoromanovsky
added a commit
that referenced
this pull request
Nov 17, 2026
leoromanovsky
added a commit
that referenced
this pull request
Nov 17, 2026
leoromanovsky
added a commit
that referenced
this pull request
Nov 17, 2026
Brings in ScaleSuiteTest, the benchmark hardening, and the fixtures pin f425d578. ScaleSuiteTest now also checks the four case files of the regex suite, so the cross-SDK regex contract runs in CI. ScaleCase accepts an expected errorCode, so the PARSE_ERROR cases of unsupported-syntax.json are compared, not skipped. semantics and unsupported-syntax are checked but not benchmarked. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The fixtures add a case file with ONE_OF lists of the integers 1 to 1000 and of the powers of two up to 2^52, which an index keyed on exact doubles must still match. Pin them and benchmark the file; its 8 cases pass on master. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
leoromanovsky
added a commit
that referenced
this pull request
Nov 17, 2026
Brings in the scale/one-of-small-numbers case file and the fixtures pin 08e86993. The benchmark and ScaleSuiteTest keep both the new case file and the regex suite. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The OpenFeature client passes the provider a new LayeredEvaluationContext on each call, while the benchmark reused one MutableContext. A rule on `id` when the context has no `id` reads the layered context's key set, which it builds again on each call, so the benchmark did not show what applications pay for it. evaluateLayered wraps each case's context the way the client does, as ExperimentationOptOutBenchmark.evaluateFreshLayeredContext does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
leoromanovsky
added a commit
that referenced
this pull request
Nov 17, 2026
leoromanovsky
added a commit
that referenced
this pull request
Nov 17, 2026
leoromanovsky
added a commit
that referenced
this pull request
Nov 17, 2026
Brings in the evaluateLayered benchmark, which evaluates with the fresh layered context that the OpenFeature client builds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The fixtures replace the 10 MB JSON flag with a 4 MB one, which brings the scale-json configuration from 14 MB down to 8 MB. The other cases are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
leoromanovsky
added a commit
that referenced
this pull request
Nov 17, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What Does This Do
Adds JMH benchmarks that measure FFE flag evaluation on the opt-in
scaleandscale-jsonsuites of the shared fixtures, and sets a baseline formaster:ScaleSuiteBenchmarkevaluates the cases of each case file through the provider and reports time and allocation for each evaluation.ScaleSuiteLoadBenchmarkmeasures how long each suite's configuration takes to parse, and prints the heap that the parsed configuration retains.ScaleSuiteTestruns the same cases in the unit tests, so a change that resolves a large configuration wrongly fails CI instead of only a manual benchmark run.This is the base of a stack. Three sibling PRs are stacked on it, and each one reports its before/after numbers against the baseline below:
ONE_OFandNOT_ONE_OFlists when the configuration loadsThis PR has no change to shipped code. The submodule bump also brings in fixture
maincommits that add 8 core cases (one time-bounded holdout flag);CanonicalFixtureTestpasses on all of them.Motivation
We need flag evaluation in Java to win a performance-sensitive customer, and this PR is the foundation: a repeatable measurement for every change that follows. The canonical fixtures are small, so they do not show where an evaluation's cost grows with configuration size: list length, allocation count, rule depth, shard count, regex input length, or JSON value size. The suites in DataDog/ffe-system-test-data#78 cover these dimensions at N and N × M, up to JSON values of 4 MB. This PR runs our evaluator over them, so every performance change shows its effect on the same inputs, on the same machine.
Additional Notes
What the benchmark covers.
@Param casesholds all 17 case files:scale/flags,numeric,one-of,one-of-many,one-of-small-numbers,regex,regex-many,rules,rules-sweep,semver,splits,time-windows,variations,wide-context, andscale-json/json-shapes,json-sizes,json-values. The fixtureevaluation-suites/scale/AGENTS.mddescribes each one.index=all, the operations go through the cases of a file in turn, so a row is the unweighted mean over the 2 to 13 cases of the file, and its slowest case dominates it: thejson-sizesrow is mostly its 4 MB case. A change that targets one case should also report that case with-p index=<n>; the setup prints the index of each case.master, including the sharded ones.ScaleSuiteTestevaluates every case twice in the unit tests, and fails when a case file of the fixtures is missing from the benchmark's list.evaluation-suites/regexsuite (dialect and backtracking cases, notscale/regexabove) is not here, because the JDK engine does not finish its backtracking cases. Evaluate feature flag regex conditions in linear time #13487 adds it.Decisions.
masterreturns JSON integers asDouble. Return read-only JSON flag values with exact integers #13484 makes them exact, and the core fixture Add JSON number precision fixture ffe-system-test-data#77 checks the type. Comparing by value lets one benchmark run on every PR in the stack. This check guards the measurement;CanonicalFixtureTestis the conformance check of the types.ec0ec05). Repin to its merge commit before this PR merges.main.jmhtask declares the suites as an input, so a repin reruns the benchmarks instead of reporting the task up to date.jackson-core, which the module already uses.Baseline on
master. AWS workspace, Intel Xeon Platinum 8175M, 16 vCPUs, JDK 21.0.10. JMH AverageTime, 1 fork, 3 warm-up and 5 measurement iterations of 2 s,-prof gc. Each row is the mean over the cases of the file. The time and allocation figures predate the latest commit, which leaves the measured evaluation unchanged; the retained heap uses the revised method. Theone-of-small-numbersrow and thescaleload row were measured later, after the fixtures added that file and its 2 flags; the earlierscaleparse time was 95.8 ms, and 2 more flags out of 10,030 do not explain the difference, so treat the parse time as within this machine's run-to-run variation. Thejson-sizesrow and thescale-jsonload row were measured again after the fixtures replaced the 10 MB value with a 4 MB one, with three forks while the machine was heavily loaded (load average about 70 on 16 vCPUs): their allocation and retained heap are exact, but compare their times only with runs made back to back on the same machine. One fork suits allocation and large time differences; for a time difference under about 15%, run three forks (-Pjmh.forks=3, or-f 3with the JMH jar).scale/flags(10,000 flags)scale/wide-contextscale/time-windowsscale/numericscale/rulesscale/one-of-small-numbersscale/semverscale/regex-manyscale/splitsscale/rules-sweepscale/variationsscale/one-ofscale/one-of-manyscale/regexscale-json/json-valuesscale-json/json-shapesscale-json/json-sizesThrough the client's context. The rows above pass the provider one reused context (
evaluate). The OpenFeature client passes a newLayeredEvaluationContexton each call instead, andevaluateLayeredmeasures that. It adds 48 B per evaluation for most cases. A rule onidcosts much more when the context has noid, because the layered context builds its key set again on each call:scale/wide-contextcaseevaluateevaluateLayeredscale-id-wide-context, match (index 2)scale-id-wide-context, miss (index 3)scalescale-jsonThe stacked PRs address
json-*,one-of*, andregex*. The baseline also shows the next candidates.splitsallocates 42 KB per call (a newMessageDigestand hex string for each shard).semverallocates 16 KB (the attribute is parsed again for each condition).variationsscans 1,000 allocations.Run all case files, with both evaluation methods:
Only the rows of the main table (
evaluate), in about half the time:Run one case:
Contributor Checklist
type:and (comp:orinst:) labels in addition to any other useful labelsclose,fix, or any linking keywords when referencing an issueUse
solvesinstead, and assign the PR milestone to the issueJira ticket: [PROJ-IDENT]