Skip to content

feat: add metrics hook - #344

Open
danjuv wants to merge 10 commits into
open-feature:mainfrom
danjuv:feat/add-metrics-hook
Open

danjuv wants to merge 10 commits into
open-feature:mainfrom
danjuv:feat/add-metrics-hook

Conversation

@danjuv

@danjuv danjuv commented Feb 18, 2026 •

Copy link
Copy Markdown
Contributor

This PR

  • Adds an OpenFeature hook to collect OpenTelemetry metrics according to the OFEP
  • Refactors this package to split into multiple files. I am happy to consolidate all of these files into init.py if we prefer this refactoring to happen in a separate PR

Related Issues

Fixes #80

Notes

  • The OFEP proposes feature_flag.variant but the TracingHook and SemConv uses feature_flag.result.variant. For consistency I've deferred to what TracingHook uses.
  • Other SDKs pass in an OTEL In-Memory Reader to test the Hook, since this package doesn't have a dependency on the otel-sdk, I've avoided that and mocked the metrics interface instead. This is consistent with the TracingHook tests

@gemini-code-assist

Copy link
Copy Markdown
Contributor

Summary of Changes

Hello @danjuv, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces a new OpenTelemetry Metrics Hook to capture detailed metrics related to feature flag evaluations, aligning with the OpenFeature Enhancement Proposal (OFEP). Alongside this new functionality, the existing OpenTelemetry integration has been refactored into a more modular structure, separating tracing and metrics concerns into distinct files and centralizing common constants. This enhances the clarity, maintainability, and extensibility of the OpenTelemetry integration within the OpenFeature hooks.

Highlights

  • OpenTelemetry Metrics Hook: Implemented a new OpenFeature hook to collect OpenTelemetry metrics for flag evaluations, adhering to the OFEP specification.
  • Code Refactoring: The OpenTelemetry integration package was refactored by splitting the existing TracingHook and introducing a new MetricsHook into separate, dedicated files for better organization.
  • Standardized Constants: Introduced a constants.py file to centralize and standardize OpenTelemetry attribute and metric names, improving maintainability and consistency across the hooks.
Changelog
  • hooks/openfeature-hooks-opentelemetry/src/openfeature/contrib/hook/opentelemetry/init.py
    • Updated to import MetricsHook and TracingHook from their new respective modules.
    • Modified __all__ to export both MetricsHook and TracingHook.
    • Removed the inline TracingHook implementation and related imports.
  • hooks/openfeature-hooks-opentelemetry/src/openfeature/contrib/hook/opentelemetry/constants.py
    • Added a new file defining Attributes and Metrics classes to hold standardized OpenTelemetry attribute keys and metric names.
  • hooks/openfeature-hooks-opentelemetry/src/openfeature/contrib/hook/opentelemetry/metric.py
    • Added a new file containing the MetricsHook class, which implements OpenFeature hook methods (before, after, error, finally_after) to record various OpenTelemetry metrics (active, error, success, request totals) during flag evaluations.
  • hooks/openfeature-hooks-opentelemetry/src/openfeature/contrib/hook/opentelemetry/trace.py
    • Added a new file containing the TracingHook class, which was moved from __init__.py.
    • Updated attribute references to use the new constants.Attributes class.
  • hooks/openfeature-hooks-opentelemetry/tests/test_metric.py
    • Added a new test file with unit tests for the MetricsHook, verifying its behavior for before, after, error, and finally_after methods.
  • hooks/openfeature-hooks-opentelemetry/tests/test_otel.py
    • Renamed the existing test file for tracing to test_trace.py.
    • Updated test function names to include _trace prefix for clarity.
Activity
  • Fixed issue Implement OpenTelemetry Metrics hook #80.
  • The author noted that the OFEP proposes feature_flag.variant but deferred to feature_flag.result.variant for consistency with existing TracingHook and SemConv.
  • The author opted to mock the metrics interface for testing instead of using an OTEL In-Memory Reader, aligning with how the Tracing Hook is tested.
  • The author expressed wariness about potential high cardinality if exception messages are assigned to every error but followed the OFEP proposal.
  • The author duplicated semconv strings due to the feature flag conventions being in a private _incubating module in opentelemetry-python.
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

@gemini-code-assist

Copy link
Copy Markdown
Contributor

Warning

Gemini encountered an error creating the review. You can try again by commenting /gemini review.

@danjuv
danjuv force-pushed the feat/add-metrics-hook branch 2 times, most recently from aea7eb3 to e3229fa Compare February 18, 2026 14:18
@danjuv

danjuv commented Feb 18, 2026

Copy link
Copy Markdown
Contributor Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a new MetricsHook for OpenTelemetry and refactors the existing TracingHook into a more organized file structure. The refactoring is well-executed and improves maintainability. The new MetricsHook is a great addition. I've identified a potential high-cardinality issue with an error attribute that could impact monitoring systems, and a minor issue where a string literal is used instead of a defined constant. My review includes suggestions to address these points.

) -> None:
attributes = {
Attributes.OTEL_FLAG_KEY: hook_context.flag_key,
"exception": str(exception).lower(),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Using str(exception) as a metric attribute value can lead to high cardinality, which is an anti-pattern for metrics and can cause issues with your monitoring backend (e.g., performance, cost). The OpenFeature Enhancement Proposal (OFEP) for metric hooks suggests using error.type with a low-cardinality value.

To align with best practices and the OFEP, I suggest using the exception's type name instead of its string representation. This will keep the cardinality of the error.type attribute low, addressing the concern you raised in the PR description.

Suggested change
"exception": str(exception).lower(),
Attributes.OTEL_ERROR_TYPE: type(exception).__name__,

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with this in principle, but the OFEP specifically states to use the message

@toddbaert @beeme1mr can you confirm?

{
"feature_flag.key": "flag_key",
"feature_flag.provider.name": "test-provider",
"exception": "test error",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

This test should be updated to reflect the recommended change in MetricsHook.error to use error.type with the exception class name. This ensures the test validates the low-cardinality attribute.

Suggested change
"exception": "test error",
"error.type": "Exception",

@danjuv

danjuv commented Feb 18, 2026

Copy link
Copy Markdown
Contributor Author

Something seems wrong with CI, doesn't seem related to this PR

@danjuv
danjuv marked this pull request as ready for review February 18, 2026 15:53
@danjuv
danjuv requested review from a team as code owners February 18, 2026 15:53
@danjuv
danjuv marked this pull request as draft February 19, 2026 12:30
@codecov

codecov Bot commented Feb 20, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (80da6b9) to head (1e4989c).
⚠️ Report is 7 commits behind head on main.

Additional details and impacted files
@@             Coverage Diff             @@
##             main      #344      +/-   ##
===========================================
+ Coverage   96.55%   100.00%   +3.44%     
===========================================
  Files           5         6       +1     
  Lines         174       241      +67     
===========================================
+ Hits          168       241      +73     
+ Misses          6         0       -6     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Signed-off-by: Danju Visvanathan <danju.visvanathan@gmail.com>
Signed-off-by: Danju Visvanathan <danju.visvanathan@gmail.com>
Signed-off-by: Danju Visvanathan <danju.visvanathan@gmail.com>
Signed-off-by: Danju Visvanathan <danju.visvanathan@gmail.com>
Signed-off-by: Danju Visvanathan <danju.visvanathan@gmail.com>
Signed-off-by: Danju Visvanathan <danju.visvanathan@gmail.com>
@danjuv
danjuv force-pushed the feat/add-metrics-hook branch from c7a7fc1 to f9fb358 Compare February 20, 2026 13:07
@danjuv
danjuv marked this pull request as ready for review February 20, 2026 13:08
Signed-off-by: Danju Visvanathan <danju.visvanathan@gmail.com>
@danjuv

danjuv commented Feb 22, 2026

Copy link
Copy Markdown
Contributor Author

/gemini summary

@gemini-code-assist

Copy link
Copy Markdown
Contributor

Summary of Changes

This pull request significantly enhances the OpenTelemetry integration by introducing a new metrics hook for feature flag evaluations, aligning with the OpenFeature Enhancement Proposal. It also improves the overall structure and maintainability of the OpenTelemetry hooks by refactoring the existing tracing functionality into a dedicated module and centralizing common constants.

Highlights

  • OpenTelemetry Metrics Hook: Added a new OpenTelemetry Metrics Hook to collect flag evaluation metrics, adhering to the OpenFeature Enhancement Proposal (OFEP).
  • Code Refactoring: Refactored the OpenTelemetry integration package by splitting TracingHook and introducing MetricsHook into separate, dedicated files for improved modularity.
  • Standardized Constants: Introduced a constants.py file to centralize and standardize OpenTelemetry attribute and metric names, enhancing maintainability.
Changelog
  • hooks/openfeature-hooks-opentelemetry/src/openfeature/contrib/hook/opentelemetry/init.py
    • Updated to import MetricsHook and TracingHook from their new modules and modified all to export both.
  • hooks/openfeature-hooks-opentelemetry/src/openfeature/contrib/hook/opentelemetry/constants.py
    • Added a new file defining Attributes and Metrics classes for standardized OpenTelemetry attribute keys and metric names.
  • hooks/openfeature-hooks-opentelemetry/src/openfeature/contrib/hook/opentelemetry/metric.py
    • Added a new file containing the MetricsHook class, which implements OpenFeature hook methods to record various OpenTelemetry metrics during flag evaluations.
  • hooks/openfeature-hooks-opentelemetry/src/openfeature/contrib/hook/opentelemetry/trace.py
    • Added a new file containing the TracingHook class, which was moved from init.py, and updated attribute references to use the new constants.Attributes class.
  • hooks/openfeature-hooks-opentelemetry/tests/test_metric.py
    • Added a new test file with unit tests for the MetricsHook, verifying its behavior for before, after, error, and finally_after methods.
  • hooks/openfeature-hooks-opentelemetry/tests/test_trace.py
    • Renamed the existing test file for tracing to test_trace.py and updated test function names to include a _trace prefix for clarity.
Activity
  • The pull request addresses and fixes issue Implement OpenTelemetry Metrics hook #80.
  • The author chose feature_flag.result.variant for consistency, despite the OFEP proposing feature_flag.variant.
  • The author opted to mock the metrics interface for testing, consistent with the TracingHook tests, rather than using an OTEL In-Memory Reader.
  • The author raised concerns about potential high cardinality if exception messages are assigned to every error but followed the OFEP proposal.
  • The author noted that semconv strings were duplicated because feature flag conventions are in a private _incubating module in opentelemetry-python.
  • A bot provided an initial summary of the changes.
  • The bot encountered an error when attempting to create a review.
  • The author manually requested a review from the bot.
  • The author commented on unrelated CI issues.
  • Codecov reported 100% coverage for the changes.
  • A bot suggested changing the exception attribute in metrics to error.type with the exception's class name to avoid high cardinality.
  • The author questioned the bot's suggestion regarding high cardinality, citing the OFEP's recommendation to use the exception message.
  • A bot suggested using the OTEL_ERROR_MESSAGE constant for an error message attribute.
  • A bot suggested updating a test case to reflect the proposed change for error.type.

danjuv added 3 commits March 19, 2026 02:10
Signed-off-by: Danju Visvanathan <danju.visvanathan@gmail.com>
Signed-off-by: Danju Visvanathan <danju.visvanathan@gmail.com>
Signed-off-by: Danju Visvanathan <danju.visvanathan@gmail.com>
aepfli added a commit to orange-case/JoustMania that referenced this pull request Oct 18, 2026
…Talk dashboards, Grafana folders (#1427) (#1480)

## Summary

This is step 6, the last step of the Grafana consolidation (#1427). It
does three things:
- adds the **standard OpenFeature OTel hooks** in Go and Python;
- rebuilds feature-flags.json as **Flags / OpenFeature**;
- adds the one-screen **Talk** dashboard (uid `talk`).

It also puts the 6 dashboards into Grafana folders (Game, Agent,
Platform and Talk) and fixes the docs.

## Spec changes

Spec changes:
- telemetry.md §6:
- a new "OpenFeature OTel hooks" row covering the metric names, labels
and cardinality rules, plus the span event;
  - the game-coordinator row loses `game_flag_evaluations_total`;
  - the metric counts are now 160 names and 183 definitions.
- telemetry.md §6.1: the Grafana consumer cell lists 6 dashboards in 4
folders, plus the Flags / OpenFeature and Talk rows.
- telemetry.md §6.2: the `game_flag_evaluations_total` row is removed.
- telemetry.md §10.3: Grafana consumers.
- UC-035 step 6: adds flags, the talk stage view and the folders.
- FR-139 traceability: the dashboard name.
- specs-inventory D-T5 is resolved.
- flags.md: line reference.

## Changes

**OpenFeature OTel hooks**

- **Go agent:**
`github.com/open-feature/go-sdk-contrib/hooks/open-telemetry`
**v0.3.7**, with both the metrics hook and the traces hook, registered
on the agent's flag client (`services/agent/flags/otelhooks.go`).
- v0.3.8 needs go-sdk 1.18, which removes `GetApiInstance`. v0.3.7 works
with the pinned go-sdk 1.17.2 and OTel 1.44, so `go.mod` gains only that
module.
- The MeterProvider gets a view that drops the hook's free-text
`error.message` label from `feature_flag.evaluation_error_total`.
- **Python:** `openfeature-hooks-opentelemetry` **0.3.1** is the latest
release and is already locked. It ships only `TracingHook`, which stays
registered. Its metrics hook is still an open upstream PR
(open-feature/python-sdk-contrib#344, idle since March).
- New `lib/openfeature_otel_hooks.py` (`MetricsHook`) records the Go
hook's four instruments with the same names and attributes.
- It is registered with `TracingHook` in `init_flag_domain` and uses the
global MeterProvider that `lib.otel_metrics.init_metrics` sets.
- **Cardinality:**
  - The targeting key and the context never become labels.
- The flag service generates a fresh variant name per write
(`r<epoch>_<n>` for a scoped value, `o<epoch>_<n>` for an overlay). On
metrics these become `scoped_value` / `overlay_value` in both languages;
spans keep the real name.
- **Metric names in VictoriaMetrics**, verified through a real
collector:
- `feature_flag_evaluation_requests_total{feature_flag_key,
feature_flag_provider_name}`
- `feature_flag_evaluation_success_total{…, feature_flag_result_variant,
reason}`
- `feature_flag_evaluation_error_total{feature_flag_key,
feature_flag_provider_name}`
  - `feature_flag_evaluation_active_count{feature_flag_key}`
  - Each also carries `job` and `otel_scope_name`.
- **Existing metrics:**
- `game_flag_evaluations_total` is **removed**. It counted a subset of
the same evaluations, and maintainer decision 6 said to replace the
custom counter. Its call sites, test, catalog row and docs are updated.
- `flag_eval_errors_total{flag_key, reason}` is **kept**. It is the only
one with the error reason; it also counts exceptions outside the SDK and
suppresses the startup race.
- `game_flag_configuration_changes_total` is **kept**, because it counts
config-change events, not evaluations.

**Dashboards**

- **Flags / OpenFeature** (uid `feature-flags` kept; folder Platform; 28
panels and 4 rows). The game-loop Hz panels are dropped; Game & Players
has them.
  - Rows:
- **Flag evaluations:** by flag, by flag and variant, by reason, by
service, and a time-range table.
- **Errors and fallbacks:** the standard error counter,
`flag_eval_errors_total` with `or vector(0)`, error share, and fallbacks
over 1h.
- **Flag service:** writes by set, result and op; rejected writes by
reason; live games registered; sync streams (red at 0); snapshot size;
experiments and game scopes expired; persist errors; corrupt state;
propagation and write p95; rejected sync calls; scope values; overlays.
- **flagd:** up, CPU, memory and goroutines, plus the coordinator's
config changes.
- **flagd reality check:** with `--metrics-exporter otel`, flagd v0.16
pushed no OTLP metrics to VictoriaMetrics in the smoke stack, even with
live RPC evaluations. The :8014 scrape (Go runtime and process metrics)
is all that arrives, so that is what the flagd row shows.
- **Talk: Two Standards** (uid `talk`; folder Talk; 6 panels; no
variables; refresh 5 s; dark):
  - live games (real / shadow);
  - decisions per minute by backend and outcome (span metrics);
  - interventions applied per minute by type;
  - OpenFeature evaluations per second by flag;
  - flag-service writes per minute by set and result;
  - a Jaeger table of `agent.decide` traces.
- The talk runbook now links it (`/grafana/d/talk?kiosk`) and the Flags
dashboard.
- **Folders:** one file provider per folder, and
`services/grafana/dashboards/{game,agent,platform,talk}/`. Links and the
SPA's Metrics-tab embed use uids, so they are unchanged. A throwaway
Grafana 13.0.2 confirms the embed URL
`/d/controller-overview/controller-overview?…&kiosk` returns 200.

**Docs**

- `observability-quickstart.md`: the queries for
`game_configured_update_frequency_hz` and `game_loop_latency_ms_bucket`
are replaced with defined metrics (finding 7). The `localhost:8080` URLs
are left alone (#1469).
- Runbook and doc paths moved to `dashboards/agent/…`.
- feature-flags.md metrics section, agent-act-runbook, flag-service.md.

**Tests**

- `tests/contract/test_grafana_provisioning.py`, new. It checks:
- every dashboard is in exactly one provisioned folder, with the
expected folder map;
  - the folder uids;
  - dashboard uids are unique;
- the SPA embed and every `/grafana/d/<uid>` link in `docs/` (research
notes excepted) exist;
  - the Talk dashboard's shape.
- `test_openfeature_hook_metrics_are_the_ones_consumed`: the dashboard
`feature_flag_evaluation_*` names are exactly the hook constants.
- The dashboard scan is now recursive.
- Go `flags/otelhooks_test.go` (UC-035) runs the real contrib hook
through an in-memory provider. It checks the bounded variant, that
`error.message` is dropped, that `feature_flag.context.id` never becomes
a label, and the span events.
- Python `lib/tests/test_openfeature_otel_hooks.py` (UC-035).

## Test Plan

- [x] `make lint`
- [x] `make test-contract`: 95 passed.
- [x] Spec gate `scripts/aiup/spec-lint-gate.sh`: 0 errors, 0 warnings.
- [x] `go vet ./...` and `go test -race ./...` in services/agent: all
packages ok.
- [x] lib pytest: 420 passed. game_coordinator pytest: 1476 passed.
- [x] `python -m json.tool` on all 6 dashboards; YAML load of the
provisioning file.
- [x] Throwaway smoke stack (127.0.0.1 ports, removed afterwards; the
maintainer stack was not touched): collector built from
`images/otel-collector`, VictoriaMetrics, the real `flag-service` image,
and flagd v0.16 syncing from it. In that stack:
- Real Python (`lib.feature_flags` + hooks) and real Go
(`flags.SetupFlagd` + hooks + `MetricViews`) evaluations were exported
through the collector.
- Real flag-service writes produced generated variants (`r884e4932_1`,
`o884e4932_2`), which reached VictoriaMetrics as `scoped_value` /
`overlay_value`.
- The span-metrics, `game_active` and `game_interventions_total` series
were synthetic.
- **Every query on Flags / OpenFeature (30 PromQL targets) and Talk (6
PromQL targets) returned data.** No query on any of the 6 dashboards
returned an error.
- Grafana loaded the 4 folders with the expected dashboards, and a Talk
query through the datasource returned frames.
- [ ] Integration tests: CI.
- Dashboard SPA: not touched (no vitest needed).

## Rollout

- The **agent** image needs a rebuild for the Go hooks (`go.mod`). The
**Python service** images need a rebuild for the new lib module.
- Grafana itself needs no rebuild, but the `services/grafana` bind
mounts change layout, so **restart Grafana** after pulling: `docker
compose up -d --force-recreate grafana`.
- On the dev stack, run with `IMAGE_TAG=latest` or `--build`. `.env`
pins `IMAGE_TAG=0.7.0`, and the Pi stays on 0.7.0 until release 0.8.0.

## Deferred / notes

- Python metrics hook: once python-sdk-contrib#344 is released, swap
`lib/openfeature_otel_hooks.MetricsHook` for the upstream one. Its
proposed names (`feature_flag.evaluation.request_total`, …) differ from
the Go hook's today, so the dashboards would need to follow.
- `services/agent/decision/decision.go` still emits a hand-rolled
`feature_flag` event on `agent.decision`. Its comment claims the Python
TracingHook uses that name, but 0.3.1 emits `feature_flag.evaluation`.
That is a contract change (telemetry.md §4.1), so it is left for a
follow-up.
- `provider.go` uses the deprecated `openfeature.GetApiInstance`
(pre-existing). It is fixed by the go-sdk 1.18 bump that hook v0.3.8
would need.
- Talk time range: I read "30s or shorter … auto-refresh" as refresh ≤
30 s, so refresh is 5 s with a 15 min window. A literal 30 s window
would leave the 1-minute rate panels empty.

## Related Issues

Part of #1427

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Implement OpenTelemetry Metrics hook

2 participants