← All projects

Robotics engineering internship · Jul–Oct 2026

AstairoPets

Robot hardware and multi-model perception

AstairoPets is a pet companion robot that brings sensing, interaction and feeding into one platform. My work covered the robot’s electrical system architecture and the vision software that turns video into structured observations of the home, people and pets.

Technologies: RK3566, ESP32-S3, Python, PyTorch, YOLO, RTMPose, GRU, ONNX, RKNN

Recorded development demo combining household objects, human pose and pet pose. The image illustrates an earlier integration stage.
Recorded development demo combining household objects, human pose and pet pose. The image illustrates an earlier integration stage.

Hardware system and connections

I designed the system around an RK3566 host for perception and interaction, with an ESP32-S3 handling local sensing and actuation. The architecture covers wheel drive, encoders, pan-tilt servos, feeding, an IR-CUT camera, radar, odor sensing, audio and a display.

Four revisions of component selection and connection documentation translated that architecture into schematics, pin assignments and wiring tables. Separate 5 V system and actuator branches support the modules, while a regulated 12 V branch supplies the host. INA260 provides whole-system voltage and current monitoring.

The fourth revision added odor sensing without disturbing the audio interface. I reassigned the display reset pin to release UART9 and specified bidirectional 3.3 V/5 V level translation for the odor module.

Fourth-revision system schematic covering power, communication, sensing and actuation.
Fourth-revision system schematic covering power, communication, sensing and actuation.

A unified perception pipeline

The vision pipeline combines a 17-class household/pet-supply detector, 20-keypoint cat/dog pose, RTMPose human pose and causal pet tracking. Each branch owns a distinct output type, avoiding duplicate public detections of people, cats and dogs.

Raw predictions remain available for inspection. A separate tracked stream carries stable IDs, smoothed coordinates and explicit observed, carried or missing states. Downstream behavior models consume that consistent track history rather than unrelated frame-level predictions.

Behavior decisions across models

Static classification provides a quick estimate of a cat’s current state. RGB-GRU and Pose-GRU branches use aligned two-second windows to add motion history. The integrated temporal path uses fixed-average fusion; a learned quality gate remains an offline comparison.

Eating decisions also consider nearby dish evidence. Causal smoothing and hysteresis limit abrupt label changes. A confidence-based policy selects a primary behavior from static and temporal predictions while preserving both source scores and the selection reason.

Earlier eating-rule demonstration using a 2.4-second RGB window. The current RGB/Pose fusion uses aligned two-second windows.
Earlier eating-rule demonstration using a 2.4-second RGB window. The current RGB/Pose fusion uses aligned two-second windows.

Model selection and edge delivery

I compared six static classification architectures and selected MobileNet-V3-Large for its accuracy and compute tradeoff. On the 289-image development-test subset after exact-image duplicate removal, it reached 82.70% accuracy and 85.87% Macro-F1.

For RK3566 deployment, I prepared FP16 RKNN and ONNX assets and versioned Linux releases with dependency locks, integrity checks and rollback support. Server conversion acceptance on 104 real inputs produced the same seven-class top-1 prediction as ONNX for every input.

Shared-protocol, seven-class development evaluation after exact-image duplicate removal. These results do not establish new-scene accuracy or physical-board performance.
ArchitectureParametersDevelopment accuracy
ResNet-1811.18 M78.55%
EfficientNet-B04.02 M83.04%
MobileNet-V3-Large4.21 M82.70%
© 2026 Yifan Ye
Built with Astrofy by Manuel Ernesto