Skip to content
OttomanDeveloperPublic

About

On-device hand gesture recognition + facial emotion detection for Flutter. 13 gestures, 7 emotions, 21 hand landmarks, face contours, blink/nod detection, attention scoring. 25-30 FPS, zero cloud. Android + iOS.

Topics

Resources

Stars

4 stars

Watchers

0 watching

Forks

Latest commit

 

History

44 Commits

Folders and files

Repository files navigation

vision_ai

vision_ai

On-device hand gesture, facial emotion, animal, and body-pose detection for Flutter. Runs at 25-30 FPS with zero cloud dependencies. Android + iOS.

pub package license


What You Can Build

Use Case Features Used
Sign language interpreter 13 gestures + custom finger patterns + hand landmarks
Driver drowsiness alert Blink detection + attention scoring + head nod detection
Touchless kiosk control Hand motion tracking + gesture recognition + two-hand interaction
Online proctoring Attention scoring + face tracking + head nod/shake
Fitness rep counter / form checker 33-point body pose + joint-angle, rep, and posture analytics
Posture & fall monitor Body pose torso/limb geometry → posture classification + fall detection
Interactive children's app Emotion detection + gesture games + two-hand clap detection
Accessibility controller Custom gestures mapped to app actions + blink-to-click
Live streaming reactions Real-time emotion overlay + gesture-triggered effects
Face-to-face distance monitor Face distance estimation + attention level
AR filter trigger Face contours + landmarks + emotion-driven overlays
Pet monitoring camera Animal detection + tracking IDs + zone alerts ("off the couch")
Wildlife / trail cam Animal detection + counting + pet-vs-wildlife grouping
Breed identifier Animal detection + breed/species classification

Packages

Package Description pub.dev
vision_ai Core plugin — camera, native ML engine (hand, face, animal), detectors, platform channels. Bundles no models (0.3.0+). pub
vision_ai_models Bundles the hand + face models and resolves their runtime paths pub
vision_ai_animals Animal detection + breed classification, analytics, overlays, bundled animal models pub
vision_ai_pose Body pose (33-point skeleton) + fitness/posture analytics, skeleton overlay, bundled pose model pub
vision_ai_flutter UI widgets — camera view, skeleton painters, label overlays (hand/face) pub

Models (0.3.0+): the core plugin no longer bundles ML models — each is loaded from a path you supply via ModelPaths. Add vision_ai_models (hand/face), vision_ai_animals (animal), or vision_ai_pose (body pose) to ship only the models your app uses.


Feature Overview

Hand Detection Face Detection

Hand Detection

  • 13 built-in gestures — fist, open palm, peace, thumbs up/down, pointing up, I love you, ok, counting 1-5
  • Custom gestures — define any finger pattern with wildcards
  • 21 hand landmarks — normalized [0,1] image coords + world coordinates in meters
  • Per-finger tracking — extended/closed for all 5 fingers
  • Hand bounding box — computed from landmark min/max
  • Motion tracking — speed, direction (8 compass), velocity components
  • Two-hand interaction — pinch, clap, hands touching
  • Gesture filtering — allow/deny lists, per-gesture confidence thresholds
  • World measurements — real-world distances in cm (pinch gap, hand span)

Face Detection

  • 7 emotion classes — happy, sad, angry, surprised, disgusted, fearful, neutral
  • 15 face contour types — full face mesh outline, eyes, lips, eyebrows, nose, cheeks
  • 10 landmark points — eyes, nose, mouth corners, ears, cheeks
  • Face tracking — stable IDs across frames
  • Blink detection — per-eye with duration in ms
  • Head nod/shake — yes/no gesture from Euler angle oscillations
  • Distance estimation — camera-to-face distance via pinhole model (cm + zones)
  • Attention scoring — eye openness + face orientation + head stability → 0-100% score
  • Accurate mode — ML Kit high-quality detection for distant/angled faces

Animal Detection (vision_ai_animals)

  • Detect & locate — bounding box + COCO class (cat, dog, bird, horse, cow, …) + score, via MediaPipe Object Detector
  • Breed / species — optional crop→classify on each detection (EfficientNet-Lite / ImageNet)
  • Tracking IDs — stable cross-frame IDs via a Dart IoU tracker
  • Analytics — counting, enter/exit events, motion + activity level, proximity, zone alerts
  • Pet-vs-wildlife grouping + overlay widgets (AnimalCameraView)
  • Bring your own model — swap in any compatible TFLite detector/classifier

Body Pose (vision_ai_pose)

  • 33-point skeleton — normalized landmarks (+ visibility/presence) and world landmarks (metres) per frame, via MediaPipe Pose Landmarker
  • Joint angles — elbow/knee/hip/shoulder (L+R), 2D or 3D
  • Rep counter — angle-oscillation state machine with hysteresis (pick any joint)
  • Posture classifier — upright / leaning / crouching / lying down
  • Fall detector — horizontal torso + rapid hip drop
  • Skeleton overlay (PoseCameraView) + rep-counter badge
  • Bring your own model / variant — swap the bundled full for lite / heavy or a custom .task

Performance

  • 25-30 FPS on mid-range devices
  • GPU acceleration with automatic CPU fallback
  • Buffer pooling to minimize GC pressure
  • Configurable emission throttling
  • Ships only the models you use (0.3.0+ de-bundled model delivery)
  • 100% on-device — no server, no API keys, no internet

Platform Support

Platform Status Min Version Notes
Android Stable API 24 (Android 7.0) Tested on Samsung Galaxy A15, multiple devices
iOS Stable iOS 12.0 Verified on a physical device (iPhone 16 Pro Max); broader device coverage welcome
Web Planned — MediaPipe WASM + TFJS feasible, not yet implemented

Quick Start

# pubspec.yaml
dependencies:
  vision_ai: ^0.4.0
  vision_ai_models: ^0.1.0   # bundled hand/face models (core ships none)
  vision_ai_flutter: ^0.4.0  # optional: pre-built overlay widgets
  # vision_ai_animals: ^0.1.0  # add for animal detection
  # vision_ai_pose: ^0.1.0     # add for body pose detection

Android Setup

<!-- android/app/src/main/AndroidManifest.xml -->
<uses-permission android:name="android.permission.CAMERA" />

Release builds: MediaPipe crashes with R8 code shrinking enabled. Add this to android/app/build.gradle.kts:

android {
    buildTypes {
        release {
            isMinifyEnabled = false
            isShrinkResources = false
        }
    }
}

iOS Setup

<!-- ios/Runner/Info.plist -->
<key>NSCameraUsageDescription</key>
<string>Camera access is needed for hand gesture and face detection.</string>

Minimal Example

import 'package:vision_ai/vision_ai.dart';
import 'package:vision_ai_models/vision_ai_models.dart';

// Core bundles no models — load the bundled hand/face paths once.
final models = await VisionAiModels.ensureLoaded();

final vision = VisionAi(
  hand: HandConfig(maxHands: 2),
  face: FaceConfig(detectEmotion: true),
  models: models, // required
);

final textureId = await vision.start();

// Render camera preview
Texture(textureId: textureId);

// Listen to results
vision.results.listen((result) {
  final hand = result.primaryHand;
  if (hand != null) {
    print('Gesture: ${hand.gesture.name} (${(hand.gestureConfidence * 100).toStringAsFixed(0)}%)');
  }
  
  final face = result.primaryFace;
  if (face != null) {
    print('Emotion: ${face.emotion.name}');
  }
});

// Clean up
await vision.stop();
await vision.dispose();

Animal Detection

import 'package:vision_ai_animals/vision_ai_animals.dart';

final vision = await AnimalVision.create(
  animal: const AnimalConfig(detectBreed: true),
);
final textureId = await vision.start();

vision.results.listen((result) {
  for (final a in result.animals) {
    print('${a.label} ${(a.confidence * 100).round()}% ${a.breed ?? ''}');
  }
});

Body Pose

import 'package:vision_ai_pose/vision_ai_pose.dart';

final vision = await PoseVision.create(pose: const PoseConfig());
final textureId = await vision.start();

final reps = RepCounter(angleSelector: (p) => PoseAngles.of(p).leftKnee);
vision.results.listen((result) {
  final pose = result.primaryPose;
  if (pose != null) print('reps: ${reps.update(pose).count}');
});

For detailed API documentation, see the vision_ai, vision_ai_animals, and vision_ai_pose package READMEs.


Architecture

Architecture

All ML inference runs on-device:

  • Hand gestures: MediaPipe Gesture Recognizer (~8MB, GPU delegate, LIVE_STREAM mode)
  • Face detection: Google ML Kit Face Detection (bundled by platform)
  • Emotion: TFLite CNN on FER2013 (~2MB, 7 classes)
  • Animal detection: MediaPipe Object Detector (EfficientDet-Lite0, LIVE_STREAM) + Image Classifier for breed
  • Body pose: MediaPipe Pose Landmarker (full variant ~9MB, LIVE_STREAM) — 33 landmarks + world landmarks

Camera frames are processed natively (CameraX on Android, AVFoundation on iOS). Only lightweight results (landmarks, labels, scores) cross the platform channel — raw frame data never leaves the native side.

Model delivery (0.3.0+): the core plugin bundles no models — every model is loaded from a file path supplied at runtime via ModelPaths. Opt-in asset packages (vision_ai_models for hand/face, vision_ai_animals for animal) bundle the weights, copy them to app storage on first use, and hand back the paths. An app links only the models it enables.

Threading model:

  • Camera frames arrive on a dedicated background thread/queue
  • MediaPipe runs async (result via callback)
  • ML Kit runs synchronously on the same thread
  • Results are dispatched to the main/UI thread for Flutter's EventSink
  • All ML resources are closed on the processing thread to avoid racing with in-flight inference

Try It Before Using It

The package ships with a full demo app with a settings panel for every feature. No code needed — just run and toggle:

git clone https://github.com/OttomanDeveloper/vision_ai.git
cd vision_ai/packages/vision_ai/example
flutter run

The example includes toggles for hand/face detection, all 6 Dart detectors, overlay visibility, camera settings, gesture filtering, accurate mode, and more. Settings are grouped into cards (Hand Detection, Face Detection, Camera, Overlays) that show/hide related options when you toggle the parent feature on or off.

Tap the paw icon in the app bar to open the Animals screen — live animal detection with breed labels, a running per-class counter, and a zone-alert demo (powered by vision_ai_animals). Tap the pose icon for the Body Pose screen — a live skeleton overlay, a rep-counter demo (pick a joint), and a posture + fall readout (powered by vision_ai_pose).


Contributing

We welcome contributions, especially for:

iOS device coverage

iOS is verified working on a physical device (iPhone 16 Pro Max). Help us broaden coverage across more iOS models:

  1. Run the example app on your iOS device (hand/face + the Animals screen)
  2. Report anything off on GitHub Issues with the ios label and your device model

Other Ways to Help

  • Report bugs with logs and device info
  • Suggest new features or gesture patterns
  • Improve the emotion model (the current FER2013 model has limited accuracy for disgust/fear)
  • Help with web platform support (MediaPipe WASM + TFJS)
  • Add unit tests for Dart-only detectors

Development Setup

This is a melos monorepo:

# Install melos
dart pub global activate melos

# Bootstrap all packages
melos bootstrap

# Run analyzer across all packages
melos run analyze

# Run tests
melos run test

License

Apache 2.0 — see LICENSE and NOTICE.

If you fork or redistribute, you must retain the copyright notice and state any changes made.

About

On-device hand gesture recognition + facial emotion detection for Flutter. 13 gestures, 7 emotions, 21 hand landmarks, face contours, blink/nod detection, attention scoring. 25-30 FPS, zero cloud. Android + iOS.

Topics

Resources

Stars

4 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages