3D Robot Perception

What makes robot perception different from computer vision: a camera you control is a different problem from a dataset you are handed, and the toolkit the rest of the course runs on: rigid-body state, the geometry that turns a 3D world into pixels and back, sensor physics, and what a learned function can add to any of it. The classical-vision and deep-learning background is taught here, in class, across these five meetings; the bridge module notes remain the written reference and carry the self-checks.

• L01What embodiment changesCourse information and logistics. Then: closed-loop versus open-loop perception, latency, partial observability, metric scale, and the asymmetric cost of being wrong. Perception as an action: a robot produces its dataset rather than receiving it. Then the crash course opens: the pinhole camera, homogeneous coordinates, intrinsics and extrinsics, the camera matrix and its eleven degrees of freedom, distortion, and calibration as an estimation problem.Slides →annotated PDFwrite-in PDF
• L02The camera matrix, and why a pose needs a groupThe pinhole camera assembled end to end: intrinsics, extrinsics, the camera matrix and its eleven degrees of freedom, estimating it from correspondences, reprojection error, radial and tangential distortion, and calibration as an estimation problem. Then the question the rest of the unit answers: three numbers cannot parameterise a rotation without a singularity somewhere, so what kind of object is a pose? The four group axioms, and what calling SO(3) a group buys you.Slides →annotated PDFwrite-in PDF
• L03Rigid-body state, and two viewsSO(3) and SE(3) as groups rather than as matrices. The tangent space and the hat map that identifies it with R³, where the exponential comes from, and twists. Uncertainty as a tangent-space covariance, the adjoint, and the lever arm that turns base rotation error into sensor position error. Frames, transform trees, and PnP optimised on the manifold. Then two views: the epipolar constraint from coplanarity, the essential and fundamental matrices and where each inverse intrinsic in F comes from, normalisation, the five steps of the eight-point algorithm, and the argument that makes F rank 2. Finally, which points can be matched at all: flat regions, edges and corners.Slides →Tutorial: rotations and manifolds →annotated PDFwrite-in PDF
• L04Robust estimation, and 3D sensingCorrespondence first: invariance and what a feature has to survive, the feature transform, image pyramids, and matching as a distance with the ratio test. RANSAC and its iteration count. Triangulation as reprojection error minimised over the point. Structure from motion step by step, from two-view initialisation to PnP registration, and bundle adjustment with its factor graph. Visual SLAM: keyframes, tracking and local mapping, and loop closure. Then stereo: depth in metres from a measured baseline.Slides →annotated PDFwrite-in PDF
• L05How a neural network learnsFirst the depth sensing L04 did not reach: LiDAR scans, metric versus relative depth, and where learned depth fits. Then neural networks from the start, for anyone who has never trained one: a neuron as a line with a weight and a bias, a loss, gradient descent and the learning rate, and one training step worked by hand and redrawn as backpropagation on a computation graph. Why stacked linear layers collapse, activation functions, softmax and cross-entropy, batches and epochs, training, validation and test sets and how to read their loss curves, and overfitting.Slides →Tutorial: neural networks →annotated PDFwrite-in PDF

The representation catalog, then the two reconstruction paradigms currently competing: per-scene optimization and feed-forward inference. Running through both is the question of which measurement to take next.

• L06The representation catalog, and learning on itFirst the deep learning L05 did not reach: vanishing gradients and normalisation, then convolutional networks, from filters, stride and padding to pooling, dropout and the encoder–decoder shape of a depth network. Then point clouds, voxels and octrees, occupancy in log-odds, TSDF and ESDF, meshes, implicit SDF and occupancy, 3D Gaussians, organized by what the robot has to do with them. Then the networks that consume them, introduced through the papers that proposed them: PointNet and PointNet++ and why a function of a set must ignore the order of its points, then sparse convolution and MinkowskiNet.Slides →Tutorial: neural networks →annotated PDFwrite-in PDF
• L07Registration and map representationsICP, its point-to-plane variant, global registration, and robust alignment, you align before you fuse. Then two ways to hold a map, side by side. Volumetric: KinectFusion to Voxblox, TSDF integration and weighting, ESDFs and why the planner wants distance rather than occupancy. Neural: NeRF, positional encoding, Instant-NGP hash grids, implicit SLAM with iMAP and NICE-SLAM. Then the comparison: drift, memory, and the cost of a map you cannot query in closed form.Slides →annotated PDFwrite-in PDF
• L08Active perception GuestChoosing the next measurement rather than accepting the one you get. Next-best-view selection, Fisher information and uncertainty quantification over a radiance field, information-driven exploration, and active mapping that closes the loop with the SLAM front end. Where the sensing budget goes on a robot that cannot look everywhere.Wen Jiang
• L09From NeRF to Gaussian splatting SLAMFrom NeRF to Gaussian splatting to SLAM, one step at a time. NeRF's volume rendering and why it is slow; 3D Gaussian splatting as the same blending with the network replaced by a list of Gaussians; then Gaussian Splatting SLAM, which builds the map and finds the camera poses live, compared side by side with SplaTAM. Last, when a splat map is worth its cost on a real platform.Slides →annotated PDFwrite-in PDF
• L10Feed-forward 3D foundation modelsAttention and the transformer, built slowly from the ground up: tokens, attention as a soft lookup, queries, keys and values, worked by hand, and the transformer block, then images as tokens. Then DUSt3R in depth: pointmaps, why the second pointmap contains the camera pose, cross-attention as matching, the loss, and global alignment. Last, briefly, what was built on it: MASt3R, CUT3R, VGGT, and the SLAM systems on top of them.Slides →annotated PDFwrite-in PDF

Geometry becomes structure: instances, relations, hierarchy, and a spatial memory a planner or language model can query.

• L11Lifting 2D foundation models into 3DSAM and SAM 2, CLIP and DINOv2, then Grounding DINO, OWL-ViT, LSeg and OpenSeg, SigLIP, BLIP-2 and RAM, and which to choose for which robot task. Back-projection, visibility, multi-view fusion and association. Open-vocabulary 3D segmentation: OpenScene, ConceptFusion. Why 3D labels are scarce and 2D priors are the workaround.Slides →annotated PDFwrite-in PDF
• L12Object-centric perceptionWhat a robot grasp is and why it needs the object's pose. Pose from matched points: Procrustes, point pair features, ICP. Render-and-compare pose estimation and tracking, FoundationPose. Scoring pose: ADD, ADD-S, BOP. Shape completion, articulated and deformable objects, parts and affordances.Slides →annotated PDFwrite-in PDF
• L133D scene graphs, open-set and task-drivenThe scene graph as a data structure: nodes, edges, and the mesh–object–place–room hierarchy. ConceptGraphs in depth, then Clio, HOV-SG and OpenGraph. Memory, multi-robot bandwidth and change detection. Then graphs and memories as language-model context: SayPlan, ReMEmbR, and embodied question answering with OpenEQA.Slides →annotated PDFwrite-in PDF
• L143D scene graphs, construction GuestNodes, edges, and hierarchy from mesh to objects to places to rooms to buildings. Real-time incremental construction, and optimizing a scene graph under loop closure.Yun Chang
• L15Spatial memory and language groundingScene graphs and memories as language-model context: SayPlan, ReMEmbR, and embodied question answering with OpenEQA. What a language model can and cannot infer about space, from text, images and video (VSI-Bench), what improved in 2025 and 2026, GPT-6 Astra on robots, and three ways to compute the geometry for it.Slides →annotated PDFwrite-in PDF

From a static model of what is there to a predictive model of what happens next under the robot's own actions.

• L16Latent dynamicsWhat a world model is and what it is conditioned on. PlaNet and Dreamer. JEPA: predicting in representation space rather than pixels, from I-JEPA to planning with V-JEPA 2. Model-based RL, planning in latent space, and the compounding-error problem.
• L174D and occupancy forecastingOccupancy networks and occupancy prediction for autonomous driving. Point-cloud forecasting, scene flow, 4D reconstruction of dynamic scenes. Where the driving and manipulation literatures diverge.
• L18Uncertainty, failure, and evaluationCalibration, ensembles, conformal prediction, out-of-distribution detection, stale spatial memory. Then: why FVD and PSNR do not predict task success, and what decision-centric evaluation looks like.
• L19Generative 3D and video world models GuestLearned 3D content generation. Action-conditioned and controllable video prediction. World foundation models for physical AI, and the gap between a plausible video and a usable simulator.Jun Gao

Closing the loop. The recurring question: what does the policy actually need from perception, and how much of the pipeline survives end-to-end learning?

• L20VLAs and the perception–action interfaceRT-2, OpenVLA, and flow-matching policies. What representation survives inside an end-to-end model. The modular-versus-monolithic argument, taken seriously in both directions.
• L21Navigation, exploration, and instruction followingSemantic mapping for navigation. Frontier and information-theoretic exploration. ObjectNav and vision-language navigation. Scene graphs as the planning substrate.
• L223D-aware manipulation policies GuestMulti-view rendering as a policy input, point-cloud policies, and diffusion as an action generator. What view and geometry a manipulation policy actually needs.Ankit Goyal
• L23Deployment, data, and consequencesSim-to-real and domain randomization. Data engines and large robot datasets. Privacy and consent for persistent 3D memory of the places people live and work.