🎓 IISc CP330 · Edge AI Course Project

Real-Time Edge AI
Semantic Mapping

Assistive navigation for the visually impaired — YOLOv8n at 24.7 FPS on Raspberry Pi 5 + Hailo-8 NPU. 5-stage model compression, IMU sensor fusion, live 2D polar semantic map. Fully offline. No cloud. No GPU.

View on GitHub ← Back to Portfolio

Hardware

Edge Hardware Stack

Four components working together — fully portable on a USB-C power bank.

🖥️

Raspberry Pi 5

8 GB RAM · Pre/post-processing, display, sensor fusion

⚡

Hailo-8 AI HAT

13 TOPS · M.2 PCIe · YOLOv8n HEF inference

📡

Arduino Nicla Vision

LSM6DSOX IMU · Gyroscope yaw at 50 Hz over USB serial

📷

TNBA1392 Camera

CSI ribbon · RGB 640×480 · Direct Pi camera interface

Optimization Pipeline

5-Stage Model Compression

From a 6 MB FP32 model to a 4.5 MB Hailo HEF — with accuracy actually improving at the end.

📦 FP32 BaselineYOLOv8n fine-tune

→

🔢 PTQ INT8No retraining

→

🧪 QAT INT8Simulated quant.

→

✂️ L1 Pruning20% filters removed

→

🚀 Hailo HEF4.5 MB · RPi5

💡 Why pruning improves accuracy

Removing the 20% of filters with the lowest L1-norm eliminates the filters that contribute the most quantization noise — giving the INT8 quantizer cleaner weight distributions to round. mAP50 jumped from 0.6093 (PTQ) to 0.6793 (Pruned) — an 11.5% improvement without any new training data.

Benchmarks

Optimization Results

All stages measured on the same 20-class COCO128 validation set.

Stage	mAP50	FPS (T4)	Size
Baseline FP32	0.6269	120.5	6.23 MB
PTQ INT8	0.6093	214.4	5.57 MB
QAT INT8	0.5944	186.6	5.65 MB
L1 Pruned INT8	0.6793	218.3	5.65 MB	✓ Deployed
Hailo HEF on RPi5	—	24.7	4.50 MB	✓ Edge

System Architecture

BackboneYOLOv8n CSPDarknet

Classes20 indoor-relevant COCO

InferenceHailo HEF → 6 raw tensors

DFL decodeRPi5 CPU (post-process)

NMSconf > 0.40, IoU < 0.45

Clusters8/8 Hailo clusters used

2D Semantic Map

IMU update rate50 Hz gyroscope

Heading formulaIMU yaw + camera FOV offset

Distance estimateReference height heuristic

SmoothingEMA λ = 0.7

Object memory10 second persistence

Map size520×520 px OpenCV canvas

Runtime

How It Works on Device

Six steps from camera frame to live polar map — all running on the Raspberry Pi 5.

Camera Capture

Picamera2 grabs 640×480 RGB frame from TNBA1392 via CSI ribbon at every cycle.

Hailo Inference

HEF runs on Hailo-8 NPU via M.2 PCIe. Returns 6 raw tensors (3 box + 3 class) for DFL decode.

DFL Decode + NMS

RPi5 CPU decodes Distribution Focal Loss outputs across 3 strides. Class-wise NMS filters duplicates.

IMU Yaw Stream

Nicla Vision streams calibrated gyro yaw at 50 Hz over USB serial. Background thread reads continuously.

Sensor Fusion

Each detection gets: heading = IMU yaw + camera FOV offset. Distance from bounding-box height heuristic. EMA-smoothed.

Dual Display

Window 1: detection feed with labeled boxes + FPS. Window 2: top-down 2D polar map with 1 m range rings.

YOLOv8PyTorchHailo DFC HailoRTPicamera2OpenCV MicroPythonpyserial NumPypyttsx3

Team

CP330 Project Team

IISc Bengaluru · Department of Computational and Data Sciences
Instructor: Dr. Pandarasamy Arjunan

Karney Jayanath

SR No. 26831

Model training & optimization — YOLOv8n fine-tuning, PTQ, QAT, L1 pruning, KD ablation, Colab pipeline, benchmarking

Shreevathsa K S

SR No. 25905

RPi5 inference pipeline — Hailo HEF deployment, HailoRT integration, DFL decoder, NMS, Picamera2 capture, bounding-box rendering

Rajneesh Babu ✦

SR No. 26058

2D semantic mapping — Nicla Vision IMU firmware (MicroPython), gyro calibration, yaw streaming, polar map, EMA smoothing, distance estimation

Full Code, Report & Models

Training notebook, HEF model, RPi5 inference script, Nicla Vision firmware, and IEEE-format report — all in the repo.