On-device hand gesture, facial emotion, animal, and body-pose detection for Flutter. Runs at 25-30 FPS with zero cloud dependencies. Android + iOS.
| Use Case | Features Used |
|---|---|
| Sign language interpreter | 13 gestures + custom finger patterns + hand landmarks |
| Driver drowsiness alert | Blink detection + attention scoring + head nod detection |
| Touchless kiosk control | Hand motion tracking + gesture recognition + two-hand interaction |
| Online proctoring | Attention scoring + face tracking + head nod/shake |
| Fitness rep counter / form checker | 33-point body pose + joint-angle, rep, and posture analytics |
| Posture & fall monitor | Body pose torso/limb geometry → posture classification + fall detection |
| Interactive children's app | Emotion detection + gesture games + two-hand clap detection |
| Accessibility controller | Custom gestures mapped to app actions + blink-to-click |
| Live streaming reactions | Real-time emotion overlay + gesture-triggered effects |
| Face-to-face distance monitor | Face distance estimation + attention level |
| AR filter trigger | Face contours + landmarks + emotion-driven overlays |
| Pet monitoring camera | Animal detection + tracking IDs + zone alerts ("off the couch") |
| Wildlife / trail cam | Animal detection + counting + pet-vs-wildlife grouping |
| Breed identifier | Animal detection + breed/species classification |
| Package | Description | pub.dev |
|---|---|---|
vision_ai |
Core plugin — camera, native ML engine (hand, face, animal), detectors, platform channels. Bundles no models (0.3.0+). | |
vision_ai_models |
Bundles the hand + face models and resolves their runtime paths | |
vision_ai_animals |
Animal detection + breed classification, analytics, overlays, bundled animal models | |
vision_ai_pose |
Body pose (33-point skeleton) + fitness/posture analytics, skeleton overlay, bundled pose model | |
vision_ai_flutter |
UI widgets — camera view, skeleton painters, label overlays (hand/face) |
Models (0.3.0+): the core plugin no longer bundles ML models — each is loaded from a path you supply via
ModelPaths. Addvision_ai_models(hand/face),vision_ai_animals(animal), orvision_ai_pose(body pose) to ship only the models your app uses.
|
|
|
- 13 built-in gestures — fist, open palm, peace, thumbs up/down, pointing up, I love you, ok, counting 1-5
- Custom gestures — define any finger pattern with wildcards
- 21 hand landmarks — normalized [0,1] image coords + world coordinates in meters
- Per-finger tracking — extended/closed for all 5 fingers
- Hand bounding box — computed from landmark min/max
- Motion tracking — speed, direction (8 compass), velocity components
- Two-hand interaction — pinch, clap, hands touching
- Gesture filtering — allow/deny lists, per-gesture confidence thresholds
- World measurements — real-world distances in cm (pinch gap, hand span)
- 7 emotion classes — happy, sad, angry, surprised, disgusted, fearful, neutral
- 15 face contour types — full face mesh outline, eyes, lips, eyebrows, nose, cheeks
- 10 landmark points — eyes, nose, mouth corners, ears, cheeks
- Face tracking — stable IDs across frames
- Blink detection — per-eye with duration in ms
- Head nod/shake — yes/no gesture from Euler angle oscillations
- Distance estimation — camera-to-face distance via pinhole model (cm + zones)
- Attention scoring — eye openness + face orientation + head stability → 0-100% score
- Accurate mode — ML Kit high-quality detection for distant/angled faces
Animal Detection (vision_ai_animals)
- Detect & locate — bounding box + COCO class (cat, dog, bird, horse, cow, …) + score, via MediaPipe Object Detector
- Breed / species — optional crop→classify on each detection (EfficientNet-Lite / ImageNet)
- Tracking IDs — stable cross-frame IDs via a Dart IoU tracker
- Analytics — counting, enter/exit events, motion + activity level, proximity, zone alerts
- Pet-vs-wildlife grouping + overlay widgets (
AnimalCameraView) - Bring your own model — swap in any compatible TFLite detector/classifier
Body Pose (vision_ai_pose)
- 33-point skeleton — normalized landmarks (+ visibility/presence) and world landmarks (metres) per frame, via MediaPipe Pose Landmarker
- Joint angles — elbow/knee/hip/shoulder (L+R), 2D or 3D
- Rep counter — angle-oscillation state machine with hysteresis (pick any joint)
- Posture classifier — upright / leaning / crouching / lying down
- Fall detector — horizontal torso + rapid hip drop
- Skeleton overlay (
PoseCameraView) + rep-counter badge - Bring your own model / variant — swap the bundled
fullforlite/heavyor a custom.task
- 25-30 FPS on mid-range devices
- GPU acceleration with automatic CPU fallback
- Buffer pooling to minimize GC pressure
- Configurable emission throttling
- Ships only the models you use (0.3.0+ de-bundled model delivery)
- 100% on-device — no server, no API keys, no internet
| Platform | Status | Min Version | Notes |
|---|---|---|---|
| Android | Stable | API 24 (Android 7.0) | Tested on Samsung Galaxy A15, multiple devices |
| iOS | Stable | iOS 12.0 | Verified on a physical device (iPhone 16 Pro Max); broader device coverage welcome |
| Web | Planned | — | MediaPipe WASM + TFJS feasible, not yet implemented |
# pubspec.yaml
dependencies:
vision_ai: ^0.4.0
vision_ai_models: ^0.1.0 # bundled hand/face models (core ships none)
vision_ai_flutter: ^0.4.0 # optional: pre-built overlay widgets
# vision_ai_animals: ^0.1.0 # add for animal detection
# vision_ai_pose: ^0.1.0 # add for body pose detection<!-- android/app/src/main/AndroidManifest.xml -->
<uses-permission android:name="android.permission.CAMERA" />Release builds: MediaPipe crashes with R8 code shrinking enabled. Add this to android/app/build.gradle.kts:
android {
buildTypes {
release {
isMinifyEnabled = false
isShrinkResources = false
}
}
}<!-- ios/Runner/Info.plist -->
<key>NSCameraUsageDescription</key>
<string>Camera access is needed for hand gesture and face detection.</string>import 'package:vision_ai/vision_ai.dart';
import 'package:vision_ai_models/vision_ai_models.dart';
// Core bundles no models — load the bundled hand/face paths once.
final models = await VisionAiModels.ensureLoaded();
final vision = VisionAi(
hand: HandConfig(maxHands: 2),
face: FaceConfig(detectEmotion: true),
models: models, // required
);
final textureId = await vision.start();
// Render camera preview
Texture(textureId: textureId);
// Listen to results
vision.results.listen((result) {
final hand = result.primaryHand;
if (hand != null) {
print('Gesture: ${hand.gesture.name} (${(hand.gestureConfidence * 100).toStringAsFixed(0)}%)');
}
final face = result.primaryFace;
if (face != null) {
print('Emotion: ${face.emotion.name}');
}
});
// Clean up
await vision.stop();
await vision.dispose();import 'package:vision_ai_animals/vision_ai_animals.dart';
final vision = await AnimalVision.create(
animal: const AnimalConfig(detectBreed: true),
);
final textureId = await vision.start();
vision.results.listen((result) {
for (final a in result.animals) {
print('${a.label} ${(a.confidence * 100).round()}% ${a.breed ?? ''}');
}
});import 'package:vision_ai_pose/vision_ai_pose.dart';
final vision = await PoseVision.create(pose: const PoseConfig());
final textureId = await vision.start();
final reps = RepCounter(angleSelector: (p) => PoseAngles.of(p).leftKnee);
vision.results.listen((result) {
final pose = result.primaryPose;
if (pose != null) print('reps: ${reps.update(pose).count}');
});For detailed API documentation, see the vision_ai, vision_ai_animals, and vision_ai_pose package READMEs.
All ML inference runs on-device:
- Hand gestures: MediaPipe Gesture Recognizer (~8MB, GPU delegate, LIVE_STREAM mode)
- Face detection: Google ML Kit Face Detection (bundled by platform)
- Emotion: TFLite CNN on FER2013 (~2MB, 7 classes)
- Animal detection: MediaPipe Object Detector (EfficientDet-Lite0, LIVE_STREAM) + Image Classifier for breed
- Body pose: MediaPipe Pose Landmarker (full variant ~9MB, LIVE_STREAM) — 33 landmarks + world landmarks
Camera frames are processed natively (CameraX on Android, AVFoundation on iOS). Only lightweight results (landmarks, labels, scores) cross the platform channel — raw frame data never leaves the native side.
Model delivery (0.3.0+): the core plugin bundles no models — every model is loaded from a file path supplied at runtime via ModelPaths. Opt-in asset packages (vision_ai_models for hand/face, vision_ai_animals for animal) bundle the weights, copy them to app storage on first use, and hand back the paths. An app links only the models it enables.
Threading model:
- Camera frames arrive on a dedicated background thread/queue
- MediaPipe runs async (result via callback)
- ML Kit runs synchronously on the same thread
- Results are dispatched to the main/UI thread for Flutter's EventSink
- All ML resources are closed on the processing thread to avoid racing with in-flight inference
The package ships with a full demo app with a settings panel for every feature. No code needed — just run and toggle:
git clone https://github.com/OttomanDeveloper/vision_ai.git
cd vision_ai/packages/vision_ai/example
flutter runThe example includes toggles for hand/face detection, all 6 Dart detectors, overlay visibility, camera settings, gesture filtering, accurate mode, and more. Settings are grouped into cards (Hand Detection, Face Detection, Camera, Overlays) that show/hide related options when you toggle the parent feature on or off.
Tap the paw icon in the app bar to open the Animals screen — live animal detection with breed labels, a running per-class counter, and a zone-alert demo (powered by vision_ai_animals). Tap the pose icon for the Body Pose screen — a live skeleton overlay, a rep-counter demo (pick a joint), and a posture + fall readout (powered by vision_ai_pose).
We welcome contributions, especially for:
iOS is verified working on a physical device (iPhone 16 Pro Max). Help us broaden coverage across more iOS models:
- Run the example app on your iOS device (hand/face + the Animals screen)
- Report anything off on GitHub Issues with the
ioslabel and your device model
- Report bugs with logs and device info
- Suggest new features or gesture patterns
- Improve the emotion model (the current FER2013 model has limited accuracy for disgust/fear)
- Help with web platform support (MediaPipe WASM + TFJS)
- Add unit tests for Dart-only detectors
This is a melos monorepo:
# Install melos
dart pub global activate melos
# Bootstrap all packages
melos bootstrap
# Run analyzer across all packages
melos run analyze
# Run tests
melos run testApache 2.0 — see LICENSE and NOTICE.
If you fork or redistribute, you must retain the copyright notice and state any changes made.