Short answer: a camera or depth sensor locates an object and estimates its position and orientation, software converts that position from the camera's frame of reference into the robot's frame through calibration, a planner chooses where and how to grip the object, and a control loop carries out the motion while checking, through feedback, whether the grasp actually succeeded. Every one of those stages can fail in its own way, which is why a working system also needs an explicit response for when something does not go as planned.
What does the camera actually sense?
Most picking systems use either a standard camera paired with software that infers depth, or a dedicated depth sensor that measures distance directly, often using structured light, stereo vision, or time-of-flight. The output is usually a point cloud or a depth image: a grid of points in space, each with a position, rather than a single photograph. From this data, the system has to work out which points belong to the object of interest, which belong to the surface underneath it, and which belong to something else entirely.
Lighting, reflective surfaces, transparent materials and clutter all make this harder than it sounds. A sensor that works reliably on a plain, dry, evenly lit object in a demonstration may perform much worse on a wet, reflective, or partially obscured one on a real line. This is one reason a working prototype under known conditions is not the same claim as a validated system for production use.
Why does calibration matter so much?
The camera sees the world in its own frame of reference, essentially a coordinate system centred on the lens. The robot moves in a different frame of reference, centred on its own base or its gripper. Calibration is the process of establishing the fixed relationship between these two frames, so that a position the camera reports as "here" can be translated correctly into a position the robot can actually move to.
Get this wrong, even by a small amount, and the robot will reach for a point that is close to the object but not on it. Calibration is usually done by showing the camera a known pattern, such as a checkerboard, from several angles and positions, and using the geometry of those known points to solve for the transformation between the two frames. This has to be redone whenever the camera or the gripper moves relative to each other, since the whole calculation depends on that relationship staying fixed.
How does the system decide what it is looking at?
Once the raw sensor data exists, detection is the step that identifies the object and estimates its pose: its position and its orientation in three dimensions. Simpler approaches match the sensed shape against a known model of the object. More flexible approaches use a trained model to recognise a category of object, such as "box" or "bottle", even with some variation between individual items. Either way, the output is an estimate, not a certainty, and a well-built system carries a confidence measure alongside that estimate rather than treating it as exact.
How is a grasp chosen once the object is found?
Grasp planning takes the object's estimated pose and works out where the gripper can actually make contact: which surfaces are reachable, wide enough, and stable enough to hold without the object slipping or rotating unexpectedly. This step also has to respect the robot's own limits, such as how far its arm can reach and rotate, and check that the planned motion will not collide with the object's surroundings.
What happens in the control loop?
- Sense: capture the current image or depth data of the workspace.
- Detect: identify the object and estimate its pose, with a confidence value.
- Transform: convert that pose from the camera's frame into the robot's frame, using the fixed calibration.
- Plan: choose a grasp point and a motion path that reaches it without collision.
- Move: execute the motion under closed-loop control, adjusting as the arm approaches the object.
- Verify: check feedback, such as gripper force or a second sensor reading, to confirm the object was actually picked up.
- Handle the exception: if any step's confidence is too low, or verification fails, stop and route to a defined fallback rather than guessing.
That last step is not an afterthought. A system that always assumes success will eventually drop, crush, or miss an object with no record of what happened.
| Stage | A common failure at this stage |
|---|---|
| Sensing | Glare, reflection or low contrast hides the object |
| Calibration | A shifted camera or gripper throws off every position estimate |
| Detection | A partially occluded or unfamiliar object is misidentified |
| Grasp planning | A reachable point is chosen that is not actually stable |
| Motion and verification | The object slips or is not detected as gripped |
When this is the wrong choice
Vision-guided picking is the wrong first step when the objects and their positions are fixed and predictable enough that a simpler, fixed motion would do the job without a camera at all; adding vision there is extra cost and extra failure points for no real benefit. It is also premature when there has been no testing under the actual conditions of the workspace, including lighting, material variation and clutter, since a system validated only in a clean demonstration setting does not tell you how it will behave on a real line. Physical deployment always needs engineering and safety validation beyond a working concept.
How we approach it at Ribhu Labs
We map the full loop, sensing, calibration, detection, planning, motion and verification, before committing to hardware, and we test the conditions that actually affect performance rather than only the easy case. Failures are made visible to an operator rather than hidden inside the system. Read more on the Robotics & vision page, see the layered mechanism up close in the mechanism study in the Lab, an illustrative browser demonstration rather than a working robot, and start with the robotics essentials lesson for a short introduction to the same loop.