Repository navigation
Conversation
Summary of ChangesHello @danjuv, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request introduces a new OpenTelemetry Metrics Hook to capture detailed metrics related to feature flag evaluations, aligning with the OpenFeature Enhancement Proposal (OFEP). Alongside this new functionality, the existing OpenTelemetry integration has been refactored into a more modular structure, separating tracing and metrics concerns into distinct files and centralizing common constants. This enhances the clarity, maintainability, and extensibility of the OpenTelemetry integration within the OpenFeature hooks. Highlights
Changelog
Activity
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
|
Warning Gemini encountered an error creating the review. You can try again by commenting |
aea7eb3 to
e3229fa
Compare
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request introduces a new MetricsHook for OpenTelemetry and refactors the existing TracingHook into a more organized file structure. The refactoring is well-executed and improves maintainability. The new MetricsHook is a great addition. I've identified a potential high-cardinality issue with an error attribute that could impact monitoring systems, and a minor issue where a string literal is used instead of a defined constant. My review includes suggestions to address these points.
| ) -> None: | ||
| attributes = { | ||
| Attributes.OTEL_FLAG_KEY: hook_context.flag_key, | ||
| "exception": str(exception).lower(), |
There was a problem hiding this comment.
Using str(exception) as a metric attribute value can lead to high cardinality, which is an anti-pattern for metrics and can cause issues with your monitoring backend (e.g., performance, cost). The OpenFeature Enhancement Proposal (OFEP) for metric hooks suggests using error.type with a low-cardinality value.
To align with best practices and the OFEP, I suggest using the exception's type name instead of its string representation. This will keep the cardinality of the error.type attribute low, addressing the concern you raised in the PR description.
| "exception": str(exception).lower(), | |
| Attributes.OTEL_ERROR_TYPE: type(exception).__name__, |
There was a problem hiding this comment.
I agree with this in principle, but the OFEP specifically states to use the message
@toddbaert @beeme1mr can you confirm?
| { | ||
| "feature_flag.key": "flag_key", | ||
| "feature_flag.provider.name": "test-provider", | ||
| "exception": "test error", |
There was a problem hiding this comment.
|
Something seems wrong with CI, doesn't seem related to this PR |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #344 +/- ##
===========================================
+ Coverage 96.55% 100.00% +3.44%
===========================================
Files 5 6 +1
Lines 174 241 +67
===========================================
+ Hits 168 241 +73
+ Misses 6 0 -6 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Signed-off-by: Danju Visvanathan <danju.visvanathan@gmail.com>
Signed-off-by: Danju Visvanathan <danju.visvanathan@gmail.com>
Signed-off-by: Danju Visvanathan <danju.visvanathan@gmail.com>
Signed-off-by: Danju Visvanathan <danju.visvanathan@gmail.com>
Signed-off-by: Danju Visvanathan <danju.visvanathan@gmail.com>
Signed-off-by: Danju Visvanathan <danju.visvanathan@gmail.com>
c7a7fc1 to
f9fb358
Compare
Signed-off-by: Danju Visvanathan <danju.visvanathan@gmail.com>
|
/gemini summary |
Summary of ChangesThis pull request significantly enhances the OpenTelemetry integration by introducing a new metrics hook for feature flag evaluations, aligning with the OpenFeature Enhancement Proposal. It also improves the overall structure and maintainability of the OpenTelemetry hooks by refactoring the existing tracing functionality into a dedicated module and centralizing common constants. Highlights
Changelog
Activity
|
Signed-off-by: Danju Visvanathan <danju.visvanathan@gmail.com>
Signed-off-by: Danju Visvanathan <danju.visvanathan@gmail.com>
Signed-off-by: Danju Visvanathan <danju.visvanathan@gmail.com>
…Talk dashboards, Grafana folders (#1427) (#1480) ## Summary This is step 6, the last step of the Grafana consolidation (#1427). It does three things: - adds the **standard OpenFeature OTel hooks** in Go and Python; - rebuilds feature-flags.json as **Flags / OpenFeature**; - adds the one-screen **Talk** dashboard (uid `talk`). It also puts the 6 dashboards into Grafana folders (Game, Agent, Platform and Talk) and fixes the docs. ## Spec changes Spec changes: - telemetry.md §6: - a new "OpenFeature OTel hooks" row covering the metric names, labels and cardinality rules, plus the span event; - the game-coordinator row loses `game_flag_evaluations_total`; - the metric counts are now 160 names and 183 definitions. - telemetry.md §6.1: the Grafana consumer cell lists 6 dashboards in 4 folders, plus the Flags / OpenFeature and Talk rows. - telemetry.md §6.2: the `game_flag_evaluations_total` row is removed. - telemetry.md §10.3: Grafana consumers. - UC-035 step 6: adds flags, the talk stage view and the folders. - FR-139 traceability: the dashboard name. - specs-inventory D-T5 is resolved. - flags.md: line reference. ## Changes **OpenFeature OTel hooks** - **Go agent:** `github.com/open-feature/go-sdk-contrib/hooks/open-telemetry` **v0.3.7**, with both the metrics hook and the traces hook, registered on the agent's flag client (`services/agent/flags/otelhooks.go`). - v0.3.8 needs go-sdk 1.18, which removes `GetApiInstance`. v0.3.7 works with the pinned go-sdk 1.17.2 and OTel 1.44, so `go.mod` gains only that module. - The MeterProvider gets a view that drops the hook's free-text `error.message` label from `feature_flag.evaluation_error_total`. - **Python:** `openfeature-hooks-opentelemetry` **0.3.1** is the latest release and is already locked. It ships only `TracingHook`, which stays registered. Its metrics hook is still an open upstream PR (open-feature/python-sdk-contrib#344, idle since March). - New `lib/openfeature_otel_hooks.py` (`MetricsHook`) records the Go hook's four instruments with the same names and attributes. - It is registered with `TracingHook` in `init_flag_domain` and uses the global MeterProvider that `lib.otel_metrics.init_metrics` sets. - **Cardinality:** - The targeting key and the context never become labels. - The flag service generates a fresh variant name per write (`r<epoch>_<n>` for a scoped value, `o<epoch>_<n>` for an overlay). On metrics these become `scoped_value` / `overlay_value` in both languages; spans keep the real name. - **Metric names in VictoriaMetrics**, verified through a real collector: - `feature_flag_evaluation_requests_total{feature_flag_key, feature_flag_provider_name}` - `feature_flag_evaluation_success_total{…, feature_flag_result_variant, reason}` - `feature_flag_evaluation_error_total{feature_flag_key, feature_flag_provider_name}` - `feature_flag_evaluation_active_count{feature_flag_key}` - Each also carries `job` and `otel_scope_name`. - **Existing metrics:** - `game_flag_evaluations_total` is **removed**. It counted a subset of the same evaluations, and maintainer decision 6 said to replace the custom counter. Its call sites, test, catalog row and docs are updated. - `flag_eval_errors_total{flag_key, reason}` is **kept**. It is the only one with the error reason; it also counts exceptions outside the SDK and suppresses the startup race. - `game_flag_configuration_changes_total` is **kept**, because it counts config-change events, not evaluations. **Dashboards** - **Flags / OpenFeature** (uid `feature-flags` kept; folder Platform; 28 panels and 4 rows). The game-loop Hz panels are dropped; Game & Players has them. - Rows: - **Flag evaluations:** by flag, by flag and variant, by reason, by service, and a time-range table. - **Errors and fallbacks:** the standard error counter, `flag_eval_errors_total` with `or vector(0)`, error share, and fallbacks over 1h. - **Flag service:** writes by set, result and op; rejected writes by reason; live games registered; sync streams (red at 0); snapshot size; experiments and game scopes expired; persist errors; corrupt state; propagation and write p95; rejected sync calls; scope values; overlays. - **flagd:** up, CPU, memory and goroutines, plus the coordinator's config changes. - **flagd reality check:** with `--metrics-exporter otel`, flagd v0.16 pushed no OTLP metrics to VictoriaMetrics in the smoke stack, even with live RPC evaluations. The :8014 scrape (Go runtime and process metrics) is all that arrives, so that is what the flagd row shows. - **Talk: Two Standards** (uid `talk`; folder Talk; 6 panels; no variables; refresh 5 s; dark): - live games (real / shadow); - decisions per minute by backend and outcome (span metrics); - interventions applied per minute by type; - OpenFeature evaluations per second by flag; - flag-service writes per minute by set and result; - a Jaeger table of `agent.decide` traces. - The talk runbook now links it (`/grafana/d/talk?kiosk`) and the Flags dashboard. - **Folders:** one file provider per folder, and `services/grafana/dashboards/{game,agent,platform,talk}/`. Links and the SPA's Metrics-tab embed use uids, so they are unchanged. A throwaway Grafana 13.0.2 confirms the embed URL `/d/controller-overview/controller-overview?…&kiosk` returns 200. **Docs** - `observability-quickstart.md`: the queries for `game_configured_update_frequency_hz` and `game_loop_latency_ms_bucket` are replaced with defined metrics (finding 7). The `localhost:8080` URLs are left alone (#1469). - Runbook and doc paths moved to `dashboards/agent/…`. - feature-flags.md metrics section, agent-act-runbook, flag-service.md. **Tests** - `tests/contract/test_grafana_provisioning.py`, new. It checks: - every dashboard is in exactly one provisioned folder, with the expected folder map; - the folder uids; - dashboard uids are unique; - the SPA embed and every `/grafana/d/<uid>` link in `docs/` (research notes excepted) exist; - the Talk dashboard's shape. - `test_openfeature_hook_metrics_are_the_ones_consumed`: the dashboard `feature_flag_evaluation_*` names are exactly the hook constants. - The dashboard scan is now recursive. - Go `flags/otelhooks_test.go` (UC-035) runs the real contrib hook through an in-memory provider. It checks the bounded variant, that `error.message` is dropped, that `feature_flag.context.id` never becomes a label, and the span events. - Python `lib/tests/test_openfeature_otel_hooks.py` (UC-035). ## Test Plan - [x] `make lint` - [x] `make test-contract`: 95 passed. - [x] Spec gate `scripts/aiup/spec-lint-gate.sh`: 0 errors, 0 warnings. - [x] `go vet ./...` and `go test -race ./...` in services/agent: all packages ok. - [x] lib pytest: 420 passed. game_coordinator pytest: 1476 passed. - [x] `python -m json.tool` on all 6 dashboards; YAML load of the provisioning file. - [x] Throwaway smoke stack (127.0.0.1 ports, removed afterwards; the maintainer stack was not touched): collector built from `images/otel-collector`, VictoriaMetrics, the real `flag-service` image, and flagd v0.16 syncing from it. In that stack: - Real Python (`lib.feature_flags` + hooks) and real Go (`flags.SetupFlagd` + hooks + `MetricViews`) evaluations were exported through the collector. - Real flag-service writes produced generated variants (`r884e4932_1`, `o884e4932_2`), which reached VictoriaMetrics as `scoped_value` / `overlay_value`. - The span-metrics, `game_active` and `game_interventions_total` series were synthetic. - **Every query on Flags / OpenFeature (30 PromQL targets) and Talk (6 PromQL targets) returned data.** No query on any of the 6 dashboards returned an error. - Grafana loaded the 4 folders with the expected dashboards, and a Talk query through the datasource returned frames. - [ ] Integration tests: CI. - Dashboard SPA: not touched (no vitest needed). ## Rollout - The **agent** image needs a rebuild for the Go hooks (`go.mod`). The **Python service** images need a rebuild for the new lib module. - Grafana itself needs no rebuild, but the `services/grafana` bind mounts change layout, so **restart Grafana** after pulling: `docker compose up -d --force-recreate grafana`. - On the dev stack, run with `IMAGE_TAG=latest` or `--build`. `.env` pins `IMAGE_TAG=0.7.0`, and the Pi stays on 0.7.0 until release 0.8.0. ## Deferred / notes - Python metrics hook: once python-sdk-contrib#344 is released, swap `lib/openfeature_otel_hooks.MetricsHook` for the upstream one. Its proposed names (`feature_flag.evaluation.request_total`, …) differ from the Go hook's today, so the dashboards would need to follow. - `services/agent/decision/decision.go` still emits a hand-rolled `feature_flag` event on `agent.decision`. Its comment claims the Python TracingHook uses that name, but 0.3.1 emits `feature_flag.evaluation`. That is a contract change (telemetry.md §4.1), so it is left for a follow-up. - `provider.go` uses the deprecated `openfeature.GetApiInstance` (pre-existing). It is fixed by the go-sdk 1.18 bump that hook v0.3.8 would need. - Talk time range: I read "30s or shorter … auto-refresh" as refresh ≤ 30 s, so refresh is 5 s with a 15 min window. A literal 30 s window would leave the 1-minute rate panels empty. ## Related Issues Part of #1427 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This PR
Related Issues
Fixes #80
Notes
feature_flag.variantbut the TracingHook and SemConv usesfeature_flag.result.variant. For consistency I've deferred to what TracingHook uses.