3D Perception Labs

00 — How labs work

Read this once before the lab

The lab is implemented in Python and evaluated on real data. It compares 3D representations against a task a robot actually has to do.

The starter repository provides the conventions, the evaluation metrics, and the deliverable file formats; what it leaves out is every function you are graded on. Read rob498/se3.py before you write anything. It fixes the pose, twist, and quaternion conventions that the lab and every grading script assume, and it provides the SE(3) exponential, logarithm, and adjoint, so you never write your own. Use them rather than a pose helper from another library: a convention mismatch fails silently and shows up only in your numbers.

The lab is worth 100 points and 30% of your course grade. It is graded on one principle: a working pipeline with an honest evaluation beats a broken pipeline with an optimistic one. Reporting that your method failed, with evidence for why, earns most of the credit. Reporting a number you did not actually measure earns none of it.

What you submit

A .zip on Canvas containing your code, plus a report PDF submitted separately so it can be annotated. Page limits are firm; appendices do not extend them. Include a README.md stating exactly how to reproduce each number in your report.

What you may use

Standard scientific Python, PyTorch, Open3D, and the specific libraries named in the lab. You may not import a library that implements the core algorithm you were asked to write. Use one to check your answer and say so in the report.

Collaboration

Discuss freely. Write and code alone. If you discussed a problem with someone, name them at the top of your report — this costs you nothing and its absence, when we can see it, costs a lot.

Generative AI

Permitted, per the course policy. You are responsible for everything you submit: if you cannot explain a line of your code in office hours, it does not count as yours. Note in your README where you used it substantially.

Reporting numbers

Every quantitative claim needs the conditions attached: which sequence, how many trials, what seed, what hardware. A table row that reads VGGT: 0.03 is worth nothing. One that reads VGGT, eval scene, 64 views, Sim(3)-aligned, 5 random view subsets, median ATE 0.031 m (IQR 0.024–0.049), 41 s mean, A40 is worth full marks even if the number is bad.

01 — Environment

Compute and data

The lab needs a GPU with at least 16 GB of memory; use the course allocation on Great Lakes. Colab will not carry a 3DGS fit or a long VGGT forward pass, and you will lose a day discovering that.

One conda environment covers the lab: conda env create -f environment.yml && conda activate rob498. The lab ships a data/ fetch script. Run it the day the lab is released, not the night before it is due — the Lab 1 ScanNet++ subset requires you to have accepted the dataset terms, which takes time to approve.

Lab 1: a ScanNet++ subset, plus held-out views and a ground-truth mesh for the evaluation scene.

02 — Starter code

Scaffolding, not solutions

The lab comes with a starter repository: the rob498 package (SE(3) conventions, evaluation metrics, deliverable file formats, summary tables), the lab modules, contract tests, and a runner CLI. It is on GitHub → bucherb/rob-3d-perception-lab. Clone it and follow its README, which takes you from a fresh clone to a passing pytest:

git clone https://github.com/bucherb/rob-3d-perception-lab.git
cd rob-3d-perception-lab

The algorithms are not in it. Every function you are graded on raises NotImplementedError with its part number and point value in the docstring. The split follows the policy above: bookkeeping, data structures and evaluation are provided, and the algorithms are not.

Two things in there are worth reading before you write any code. The rob498/se3.py docstring fixes the conventions the lab and every grading script assume — poses map child to parent, twists are ordered (v, omega) with translation first, quaternions are scalar-last — and the module provides exp_se3, log_se3, and adjoint so you do not write them. And lab1/tests/test_contracts.py checks those conventions against your code: it cannot tell you your segmenter is any good, but it will tell you that your poses are in the wrong convention, or that your deliverable files will not parse — which is where people lose points on work they actually did.

Tests for functions you have not written yet skip rather than fail, so pytest lab1/tests -v stays useful from the first day to the last.

LAB 1 Out Oct 7 · Due Fri Nov 20, 11:59pm

Representations, compared honestly

Build the same scene three ways — a learned point-cloud segmenter, a per-scene Gaussian splat, and a feed-forward transformer reconstruction — then evaluate all three on a robot task rather than on rendering quality. The ranking will change. Your job is to explain why.

The trap in this lab

It is easy to produce three pipelines and a table of PSNR values and conclude that the highest number wins. That submission fails Part E, which is 40 of the lab's 100 points. The question is not which representation reconstructs best. It is which one a robot can use, and those are different questions with different answers.

Part B — Learned segmentation

25 pts

B112 pts

Train a sparse-convolution network (MinkowskiNet) or a point transformer (PTv3) for semantic segmentation on the provided ScanNet++ subset. Report overall mIoU and per-class IoU. You may start from released weights; if you do, say so and report both fine-tuned and zero-shot numbers.

B27 pts

Ablate voxel size over at least 2 cm, 5 cm, and 10 cm. Plot mIoU and inference latency against voxel size. Identify which classes degrade fastest as resolution coarsens and explain what they have in common.

B36 pts

Your downstream task in Part E cares about a small subset of classes. Report IoU restricted to those, and comment on whether mIoU was a good proxy for what you actually needed.

Part C — Per-scene optimization

20 pts

C18 pts

Fit a 3D Gaussian Splatting model to the evaluation scene. Report PSNR, SSIM, and LPIPS on the held-out views, the final Gaussian count, wall-clock fit time, and peak GPU memory.

C26 pts

Extract a surface or occupancy estimate from your splats. Document the procedure precisely: opacity threshold, how you treat anisotropic Gaussians, and how you handle the floaters that a splat fit always produces. Show the extracted surface against the ground-truth mesh.

C36 pts

Report accuracy and completeness of that surface against ground truth, at the fixed threshold the lab sets (default 0.05 m, --recon-threshold) — the same one used for the feed-forward geometry in Part D3, so splat and feed-forward numbers are comparable within this lab. Comment on the relationship between your PSNR and your geometric accuracy. They are not the same ordering, and the gap is the finding.

Part D — Feed-forward reconstruction

15 pts

D15 pts

Run VGGT on the same images, supplying no poses and no intrinsics. Report wall-clock time and peak memory, and note how these scale as you vary the number of input views.

D25 pts

Compare the recovered camera poses against ground truth after Sim(3) alignment. Report ATE and mean rotation error. Note the scale factor you had to apply and what that means for a robot that needs metric distances.

D35 pts

Evaluate the predicted geometry against the ground-truth mesh with the same accuracy and completeness metrics, at the same threshold, as Part C3. Identify the regions where it fails and characterize them — textureless, specular, thin structure, or beyond the view distribution.

Part E — The downstream task

40 pts

The task is collision checking: given a position in the room, would the robot hit something there? You answer it once from the ground-truth mesh, which gives the right answers, and then once from each representation you built. The representation a robot should trust is the one whose answers agree with the ground truth, and above all the one that does not call a collision free.

How the collision check works

The robot is a sphere of radius --robot-radius (default 0.15 m). A configuration is just the sphere’s centre, a point in the room, and it is in collision if the sphere touches the scene. That is the same question as “is the scene within r of this point?”, so every representation only has to answer one query: is anything within r of x? Use the same r everywhere.

Sample configurations (provided). downstream.sample_task_configurations(scene, n=2000, seed=..., robot_radius_m=r) gives you the 2,000 positions to test. It samples the room and keeps only positions some camera actually saw, so the inside of a cabinet or the space behind a wall is never scored. Report the seed and the radius.
Label them from ground truth (E1). label_ground_truth_occupancy: a configuration is in collision when its unsigned distance to the ground-truth mesh is at most r. Do not test inside versus outside; a scanned room is not watertight, so that test fails.
Write one occupancy query per representation (E2). Each is a subclass of OccupancyQuery in lab1/downstream.py, with is_occupied(points) and memory_mb(): the segmented sensor cloud from Part B, the splats from Part C, the VGGT reconstruction from Part D (after fixing its scale), and a TSDF as the baseline. Each class’s docstring says what the query means for that representation and where the traps are.
Evaluate them identically. evaluate_representations runs every query on the same points and reports false-positive and false-negative rates, query latency and memory. It is provided and you should not change it.

E18 pts

Define your metric formally before you run anything, and state what a false negative and a false positive each cost a robot that relies on your collision check. They are not symmetric, and your evaluation should reflect that.

E218 pts

Evaluate all three representations on that metric. Hold the query procedure fixed across representations so the comparison is about the representation and not your query code.

E310 pts

Find the flip. Identify at least one case where the representation that scored best on reconstruction fidelity scores worse on the task, and explain the mechanism. If you genuinely cannot find one, say so and show the evidence — a well-supported negative result earns full credit here; an unsupported claim that no flip exists earns none.

E44 pts

Write a one-paragraph recommendation: which representation for which robot task, and under what compute budget. Write it as if to an engineer who has to pick one by Friday.

Lab 1 deliverables

Report PDF, 8 pages maximum.
One summary table with representations as rows and columns for fidelity, memory, build time, query latency, and your task metric. This table is the centre of the report; build everything else around it.
Code archive with the exact configs and seeds used for each run.