Bootstrapping Dense 3D Segmentations from Sparse 2D Annotations

Motivation

Consider a fresh 3D microscopy volume with no annotations. The goal is a dense instance segmentation of some specific structure of interest: cells, organelles, neurites, or other objects. The natural first step is to try existing deep learning models.

Sometimes they work reasonably well, especially when the data resembles what the model was trained on. That is a good sign: it means the model can likely be fine-tuned. But often the data differs enough in imaging modality, tissue type, resolution, or staining that off-the-shelf models produce unusable results.

Segmentation foundation models (SAM², microSAM³, CellSAM⁴, Cellpose⁵, etc.) can help in 2D, but they produce sparse segmentations, lack 3D consistency, and frequently miss the specific structures of interest. The same features that make a dataset scientifically interesting are exactly what generalist models tend to get wrong.

This leads to the same bottleneck every time: manual annotation of objects of interest to generate training data. Dense 3D annotation of even a small volume can take hundreds to thousands of expert hours. For example, dense annotation of hippocampal neuropil in a 180 μm³ EM volume required approximately 2,000 hours of expert effort ⁶.

We asked: how sparse can the annotations be?

We developed a 2D→3D method that generates dense 3D instance segmentations from sparse 2D annotations. Ten minutes of non-expert painting on a few 2D sections is sufficient to bootstrap 3D segmentations approaching the quality of models trained on dense expert ground-truth on matched evaluation volumes, a 1,000-fold reduction in annotation time. The approach works across diverse imaging modalities, biological targets, and segmentation tasks, and requires no domain-specific pretraining.

Results

Dense 3D segmentations from sparse 2D annotations

We validated the 2D→3D method across six publicly available datasets: HARRIS-15⁶, FIB-25⁷, CREMI-A/B/C⁸, and EPI⁹. Each dataset contains two dense ground-truth volumes. We systematically varied annotation density from single objects to dense labels.

Across all datasets, ten minutes of non-expert sparse annotation on a single section was sufficient to generate a dense 3D segmentation comparable in quality to one based on expert annotation requiring 1,000× more time.

On HARRIS-15, segmentation errors against the dense baseline drop with increasing amounts of annotation, with diminishing returns at each additional order of magnitude of annotation time.

In practical terms, the 2D→3D bootstrap path approaches the quality of direct dense annotation while saving 1-6 months of calendar time and $25k-$125k of expert annotation labor.

Generalization across imaging modalities and segmentation tasks

We tested the 2D→3D method on five additional publicly available datasets spanning distinct imaging modalities, biological structures, and segmentation tasks. Across all five, the pipeline produced visually coherent 3D segmentations from minimal annotation effort.

• LICONN¹⁰: Expansion-microscopy volumes of mouse somatosensory cortex (effective voxel size ~10 × 10 × 20 nm, x × y × z). 10 minutes of SAM²-assisted sparse annotation on a few sections for boundary segmentation.
• PRISM¹¹: 18-channel light microscopy of mouse hippocampus (voxel size 35 × 35 × 80 nm, x × y × z). Ten sections of manual ground-truth annotation for boundary segmentation.
• CREMI-C⁸: Serial-section TEM of Drosophila neuropil through an axon tract (voxel size 4 × 4 × 40 nm, x × y × z). Two sections of manual annotation for synaptic cleft segmentation.
• MitoEM-H¹²: Multi-beam SEM of human cortex (voxel size 8 × 8 × 30 nm, x × y × z). Two sections of manual annotation for mitochondria instance segmentation.
• Fluo-C2DL-Huh7¹³: Live cell laser confocal imaging of Huh7 hepatocarcinoma cells (pixel size 0.65 × 0.65 μm, x × y). Two frames of manual annotation for 2D+t cell segmentation.

Comparison with existing tools

To contextualize the method against existing general-purpose segmentation tools, we compared it with Cellpose + uSegment3D^5,14 on the EPI dataset. Cellpose was applied to all 540 images of the EPI test volume, and uSegment3D was used to merge the 2D segmentations into a 3D consensus segmentation. For the 2D→3D method, we evaluated three levels of supervision: dense ground-truth labels, a single densely labelled section, and sparse SAM-generated labels on 3 images in 5 minutes of human time. The 2D→3D method with sparse SAM labels achieved a usable segmentation from 5 minutes of human effort, approximately two orders of magnitude less annotation for a modest decrease in segmentation accuracy.

Bootstrapping reduces total reconstruction cost

Unproofread 2D→3D segmentations serve as pseudo ground-truth to bootstrap dedicated 3D segmentation models. We evaluated whether segmentations from different amounts of sparse training data could bootstrap subsequent 3D models without any manual proofreading. Across all six datasets, bootstrapped 3D models trained on 2D→3D pseudo ground-truth approached the quality of dense 3D baselines at all sparsity levels.

Pseudo ground-truth from 10 minutes of non-expert annotation on a single section yielded bootstrapped segmentations requiring only approximately 2-3 additional edits per micron (in HARRIS-15) compared to those from dense expert annotations of entire volumes (1,000× more annotation time). Sparser approaches increase the subsequent proofreading burden, but the reduction in upstream annotation yields an order-of-magnitude improvement in total reconstruction time.

Cost-benefit analysis

We estimated the total reconstruction time as the sum of manual annotation time, machine computation time, and estimated proofreading time for HARRIS-15. Proofreading time was estimated at 0.1 minutes per split correction and 1, 3, or 10 minutes per merge correction, spanning the range of reported proofreading throughputs. All times are in Person Workdays (PWD) at 5 hours per day.

How It Works

The pipeline operates in two stages. Both use standard U-Net¹⁵ architectures and are lightweight enough to run on consumer hardware (<3 GB GPU memory).

Stage 1: Sparse 2D → Dense 2D

A user creates sparse 2D annotations on one or a few sections, either manually or using a foundation model (e.g., SAM²). A 2D U-Net is trained on these sparse labels to predict dense local shape descriptors (LSDs)¹⁶ and affinities^17,18. A masked loss restricts supervision to labeled regions, preventing the network from learning background features from unannotated instances. At inference, the 2D network is applied section-by-section to produce a stack of dense 2D predictions.

Synthetic 3D Label Generation

Synthetic 3D training data is generated using three distinct strategies to simulate diverse biological structures: morphological dilation from random seeds, section-wise dilation of speckled binary arrays, and watershed on Gaussian-filtered random peaks. A fourth ensemble strategy combines all three equally. These synthetic labels serve as the source of both inputs and targets for the Stage 2 network below and are generated on-the-fly during training.

Stage 2: Stacked 2D → 3D

A lightweight 3D U-Net is pre-trained on the synthetic 3D labels described above. It learns to map noisy stacked 2D LSDs to clean 3D affinities, with both inputs and targets simulated from the synthetic labels. Because the 3D network only ever sees stacked 2D predictions as input, it acquires z-axis connectivity priors entirely from the synthetic data, without requiring any 3D annotation from the target dataset.

Inference

At inference, the two trained U-Nets run sequentially on a target volume. The 2D U-Net is applied section-by-section to produce stacked 2D LSDs, which the 3D U-Net converts to 3D affinities. Seeded watershed on the Stage 2 affinities produces the final 3D instance segmentation.

Get Started

Napari Plugin

The napari-bootstrapper plugin provides an interactive graphical interface for applying the 2D→3D method to small volumes. Create sparse annotations directly in napari, generate dense volumetric segmentations, and perform basic proofreading and post-processing. Works seamlessly with foundation models and the napari plugin ecosystem.

pip install napari-bootstrapper

Bootstrapper CLI

The complete framework is open-source. It includes modules, scripts, and a command line interface for all components of the workflow: data preparation, 2D→3D models, LSD models, training pipelines, blockwise parallel inference and post-processing, evaluation, and error identification. Designed to scale to large volumes using lightweight distributed algorithms that do not require high-performance computing clusters.

pip install bootstrapper
# or
git clone https://github.com/ucsdmanorlab/bootstrapper.git
cd bootstrapper && pip install -e .

Links

• ucsdmanorlab/bootstrapper : Full pipeline (CLI, training, inference, evaluation)
• ucsdmanorlab/napari-bootstrapper : Napari plugin for interactive 2D→3D
• funkelab/gunpowder : Data loading and augmentation
• funkelab/lsd : Local Shape Descriptors

Takeaways

• We present a general-purpose method for generating dense 3D segmentations from sparse 2D annotations. It works across diverse imaging modalities, biological targets, and segmentation tasks. No domain-specific pretraining or high-performance computing is required.
• Specialist 3D models trained on dense ground-truth achieve the highest accuracy within their domain. This method does not replace them. It addresses the bottleneck that precedes them: generating the dense training data needed for both model development and evaluation.
• In our experiments, bootstrapped 3D models trained on unproofread pseudo ground-truth approached the quality of models trained on dense expert annotations, with orders-of-magnitude less human effort.

Complementary to foundation models

The pipeline is complementary to segmentation foundation models. SAM, microSAM, or CellSAM outputs serve directly as sparse input labels for the 2D→3D framework. As 2D foundation models improve, so does every segmentation bootstrapped through them, with no changes to the pipeline itself.

Toward autonomous bootstrapping

The bootstrapper CLI is fully scriptable. AI agents can independently orchestrate the entire bootstrapping loop: selecting annotations, training models, running inference, monitoring LSD and segmentation inconsistencies, and iterating. This paradigm, where agents explore multiple analysis paths in parallel and retain those that meet quality thresholds, could dramatically accelerate ground-truth generation across many volumes simultaneously.

Dense voxel ground-truth remains the gold standard for 3D microscopy segmentation. This method accelerates convergence to that standard by reducing the total cost of generating training data. For laboratories with limited resources, it transforms an intractable annotation problem into a manageable one.