Bridge modules

A camera maps a world point to a pixel by a rigid transform into the camera frame, a perspective divide, and an affine map into pixels:

Four intrinsic parameters that matter: focal lengths fx, fy in pixels and the principal point cx, cy. The principal point sits a few pixels off the image centre. It is usually within a few pixels of it, and assuming otherwise puts a small systematic bend in every reconstruction you build.

Real lenses are not pinholes. The standard correction is radial plus tangential, applied in normalised coordinates before K:

Calibration estimates all of this from images of a known target. Zhang's method uses a planar checkerboard at several orientations: each view gives a homography between the board plane and the image, each homography constrains the image of the absolute conic, and enough of them determine K. Then everything is refined by minimising reprojection error over intrinsics, distortion, and the per-view board poses.

ReadSzeliski §2.1.4 (projective camera), §11.1.4 (calibration). CMU 16-385 Camera models, both lectures.

Given the same scene point in two images, the two rays must intersect. That single constraint is the whole of two-view geometry. In pixels it is the fundamental matrix; in calibrated normalised coordinates it is the essential matrix:

The rank-2 condition is not decoration: it is what makes every point in one image map to a line in the other rather than a point. Estimating F without enforcing it gives epipolar lines that fail to meet at a common epipole.

Decomposing E by SVD gives four candidate (R, t) pairs: two rotations times two translation signs. Only one puts triangulated points in front of both cameras; that cheirality test is how you pick it. Triangulate a single point and check the sign of both depths.

ReadHartley & Zisserman ch. 9–11 is the reference; Szeliski §11.3 is faster. CMU 16-385 Two-view geometry. TUM Multiple View Geometry lectures 5–7 if you want it derived slowly on video.

Rectify two views so epipolar lines are corresponding image rows, and matching becomes a 1D search along a row. Disparity d, the horizontal shift of the match, gives depth directly:

Differentiate. This is the most consequential equation in Module A:

Disparity error is roughly constant across the image. It is a matching problem, not a distance problem. Depth standard deviation therefore grows with the square of range, and depth variance with the fourth power. A stereo rig accurate to a centimetre at 1 m is accurate to about 25 cm at 5 m, with nothing changed but the distance.

The same shape holds, for different reasons, across the depth sensors in Unit 0: structured light degrades with range as the projected pattern spreads; time-of-flight is more nearly constant in range but fails on dark and specular surfaces; and every one of them is worse at grazing incidence.

ReadSzeliski §12.1–12.4. CMU 16-385 Stereo. Nguyen et al., Modeling Kinect Sensor Noise (3DIMPVT 2012) for a measured axial-noise model of a real depth camera.

Detection finds repeatable points. The Harris criterion looks at the structure tensor of local gradients: two large eigenvalues means a corner, one means an edge, none means flat, and a corner is exactly the case where a small shift in any direction changes the patch, which is what makes it locatable.

Description summarises the neighbourhood so it can be recognised from another viewpoint. SIFT builds a gradient-orientation histogram over a scale- and rotation-normalised patch; ORB pairs a FAST detector with a binary BRIEF descriptor compared by Hamming distance, which is far cheaper and is what you find inside real-time SLAM systems. Learned front-ends (SuperPoint, and the detector-free matchers that followed) now beat both on hard viewpoint change.

Matching is where the errors come from. Two standard filters: Lowe's ratio test, rejecting a match when the best and second-best distances are too close (≈ 0.8), and mutual nearest neighbour, keeping a pair only if each is the other's best. Both are cheap and both remove far more bad matches than they cost you good ones.

ReadSzeliski §7.1–7.2. CMU 16-385 Feature and corner detection and Feature descriptors and matching.

Least squares assumes every measurement is a measurement. Correspondence sets violate that badly: a wrong match is not a noisy measurement of the right answer, it is a measurement of something else entirely, and one of them can drag a fit arbitrarily far.

RANSAC handles this by hypothesis and verification: draw the minimum number of points that determine a model, fit, count how many of the rest agree, keep the best, refit on its consensus set. The iteration count follows from wanting to draw one all-inlier sample with probability p:

Note how violently N depends on s and w. This is why minimal solvers matter: a 4-point solver is tractable where an 8-point one is not, at the same inlier rate.

The continuous cousin is a robust loss, Huber, Tukey, which down-weights large residuals instead of hard-classifying them. Robust losses need an initialisation good enough that inliers already dominate; RANSAC does not. In practice you use RANSAC to get there and a robust loss to refine.

ReadSzeliski §8.1.4. CMU 16-385 Image homographies covers RANSAC in the setting it is usually first met.

Three estimation problems, distinguished by what is known and what is being solved for.

Triangulation, poses known, find the point. Two rays never quite meet under noise, so the linear (DLT) solution minimises an algebraic residual and the optimal one minimises reprojection error in both images. Always check cheirality: a triangulated point behind a camera is a sign the correspondence is wrong, not that the point is unusual.

PnP, points known, find the pose. Three correspondences give up to four solutions (P3P) and a fourth point disambiguates. Wrapped in RANSAC, this is how a system relocalises against an existing map.

Bundle adjustment, solve for everything at once, minimising total reprojection error over all poses and all points:

The problem is enormous and almost entirely sparse: pose i and point j interact only if j was seen in view i. Exploiting that structure, eliminating the points via the Schur complement and solving the much smaller pose system, is what makes it tractable at all.

The modern framing is a factor graph: variables are poses and landmarks, factors are measurements, and each factor carries a noise model. Marginalising the landmarks out leaves a pose graph, poses connected by relative-pose constraints. That is exactly the structure of Lab 1 Part E, and it is why the loop-closure covariance you supply there is not a formality: it sets how the correction is distributed around the loop.

ReadSzeliski §11.4. Dellaert & Kaess, Factor Graphs for Robot Perception §1–3, the reference for Unit 1, listed under Resources. CMU 16-822 covers the geometry end in depth.

A network is a composition of differentiable operations. Evaluating it builds a computation graph whose nodes are operations and whose edges carry intermediate values. Backpropagation is the chain rule applied to that graph in reverse topological order: seed ∂L/∂L = 1 at the loss and push gradients backward, each node turning the gradient of its output into the gradient of its inputs.

Why reverse mode? Cost scales with the number of outputs, not inputs. Forward mode costs one pass per input variable; reverse mode costs one forward and one backward pass regardless of how many parameters there are. With a scalar loss and ten million parameters, that difference is the entire field.

The price is memory: the backward pass needs values cached from the forward pass, so activation memory grows with depth and batch size. That is what gradient checkpointing trades back, recompute instead of store, and it is usually the first thing to reach for when Lab 2 runs out of GPU memory.

When a hand-written gradient is wrong, check it numerically before debugging anything else:

ReadOlah, Calculus on Computational Graphs, the clearest short treatment. Szeliski §5.3.5–5.3.6. PyTorch's autograd tutorial for the mechanics.

A convolution slides a small learned kernel over the input, sharing weights across position. That weight sharing buys two things: far fewer parameters than a dense layer, and translation equivariance, shift the input, and the output shifts with it. For images, where a cat is a cat wherever it appears, that is exactly the right prior.

Receptive field grows linearly with depth at stride 1, and multiplicatively once you stride or dilate. Normalisation matters more than it looks: batch norm computes statistics across the batch, which breaks without warning when the batch is small or when examples have different numbers of elements, both of which are true for point clouds. Group and layer norm do not have that dependence, which is why you see them in 3D architectures.

Now make it 3D. A dense 3D convolution over a 10×10×3 m room at 1 cm is 3×10⁸ voxels, of which the observed surface occupies well under one percent. Nearly all the compute is spent multiplying zeros, and memory is worse.

Sparse convolution is the answer: store only occupied voxels in a hash table, and define the convolution to produce output only where there is input. MinkowskiNet is the standard implementation, and point transformers are the attention-based alternative. The speedup in practice is smaller than the occupancy ratio suggests, gather/scatter over a hash table has poor memory locality compared to a dense tensor, and that gap is a real cost, not an implementation defect.

ReadSzeliski §5.4. Stanford CS231n's convolutional-networks notes are still the best introduction. Then Choy et al., 4D Spatio-Temporal ConvNets, which is on the Unit 1 reading list.

Attention lets every element look at every other element and decide what to take:

Queries ask, keys advertise, values carry content. The √d_k is not cosmetic: without it the dot products grow with dimension, the softmax saturates, and gradients vanish. Multiple heads run this in parallel in different learned subspaces so one head can track geometry while another tracks appearance.

The property that matters for this course is that self-attention is permutation-equivariant. Reorder the inputs and the outputs reorder identically; nothing in the operation assumes a grid, a sequence, or any neighbourhood structure. A point cloud is an unordered set, so attention fits it natively where convolution has to be told what "adjacent" means. Positional information is then injected explicitly, and in 3D that means encoding coordinates, which is a modelling choice rather than a given.

The cost is O(n²d) in the number of elements. A room at useful resolution has far too many points for that, which is why practical point transformers restrict attention to neighbourhoods or serialise points along a space-filling curve to get locality back cheaply. Recognising that these are all workarounds for one quadratic is most of what you need to read the architectures in Units 1 and 4.

ReadVaswani et al., Attention Is All You Need, then The Annotated Transformer, which walks the paper line by line as runnable code. Dive into Deep Learning ch. 11 if you want it worked more slowly.

Optimisers. SGD with momentum still generalises best on vision tasks given a tuned schedule; AdamW is more forgiving and is what you should reach for first on an unfamiliar problem. The "W" matters, decoupled weight decay behaves as regularisation should, whereas Adam's original L2 term interacts with the adaptive scaling in ways that make the decay coefficient mean something different at every parameter.

Learning rate is the hyperparameter worth your tuning time; nothing else is close. Warm up over the first few hundred steps, then decay, cosine is a reasonable default. If a run diverges in the first epoch, the learning rate is too high before it is anything else.

Reading a curve. Training loss falling while validation rises is overfitting: more augmentation, more regularisation, or early stopping. Both flat and high is underfitting or a learning rate far off. Loss spiking irregularly usually means the rate is too high or a few batches contain something pathological, and in 3D, "pathological" is often a scan with almost no points in it.

Reproducibility. Seed Python, NumPy, and Torch, and record the seed. Note that seeding gives you reproducibility, not determinism: several CUDA kernels are nondeterministic regardless, and forcing determinism costs speed. Report which one you had.

ReadPyTorch's Learn the Basics for the mechanics. Dive into Deep Learning ch. 12 for optimisation. Michigan's ROB 498 DeepRob has a practical treatment aimed at robotics specifically.