Perception‑Guided Manipulation on Franka Panda

Perception-Guided Manipulation on Franka Panda

This project demonstrates a perception-driven manipulation pipeline developed for Samsung SSI’s laboratory automation. A Franka Emika Panda 7-DoF arm with an eye-in-hand Intel RealSense D435i performs autonomous pick-and-place guided by vision and analytically validated kinematics.

Context

Repetitive lab handling—sample transfers, vial sorting, tool exchange—demands precise placement under variable lighting and clutter. The goal was to perceive, localize, and manipulate objects without fixed fiducials or taught poses, reducing setup time and operator intervention.

Approach

1. Hand‑Eye Calibration
   A Python/OpenCV workflow establishes the rigid transform between the RealSense and the end effector. For each pose, synchronized images and EE poses are recorded, PnP is solved, and cv2.calibrateHandEye() yields the camera-to-gripper transform with millimeter-level repeatability.
2. Perception & Pose Estimation
   RGB-D fusion produces 6-DoF object poses in the camera frame. Using the calibrated transform, poses are mapped into the robot base frame for grasp alignment, approach vector generation, and visual refinement.
3. Analytical / Numerically Stable IK
   A hybrid IK stack (closed-form where applicable; damped least-squares fallback) generates smooth, collision-aware trajectories within joint and torque limits. Each solver path is covered by pytest for deterministic behavior across the workspace.
4. Custom Middleware & Execution
   Instead of ROS 2, a lightweight middleware I built handles device abstraction, trajectory streaming, safety interlocks, and telemetry. It exposes clear interfaces for perception, planning, and control, enabling low-latency loops and straightforward swap-in of new perception or IK modules without changing the motion layer.

Results

The calibrated pipeline achieved < 2 mm positional error between predicted and measured grasp points and cut setup time for new object configurations by roughly an order of magnitude compared to manual teaching. In internal SSI trials, cycle time per pick-and-place dropped from ~18 s to under 7 s with robust performance across multiple object geometries.

Impact

The system forms a reusable template for perception-guided manipulation at SSI—combining rigorous calibration, validated kinematics, and modular middleware. It now serves as a foundation for future cells with additional sensors, multi-SKU parts, or dual-arm coordination.