\[\argmin_{\{\mathbf{T}_i\},\,\{\mathbf{X}_j\}} \; \sum_{i}\sum_{j \in \mathcal{V}(i)} \rho\!\left( \big\| \pi\!\left(\mathbf{K}, \mathbf{T}_i, \mathbf{X}_j\right) - \mathbf{u}_{ij} \big\|^2_{\boldsymbol{\Sigma}_{ij}} \right)\]

Every camera pose and every 3D point, against every observation, in one nonlinear least-squares problem.
$\mathbf{K}$ is shared; only $\mathbf{R}_t$ and $\mathbf{T}_t$ change from frame to frame.
$\rho$ is a robust kernel such as Huber, so a surviving outlier bends the answer instead of breaking it.

OpenCV sfm module reconstruction of the Sagrada Familia: green sparse 3D points of the towers, surrounded by a loop of yellow camera frusta.

Image: OpenCV contrib sfm module docs, Apache-2.0