Get the latest tech news

Segment Anything Model and Friends

The Segment Anything Model (SAM) and its successors made a significant leap forward in computer vision, particularly in image and video segmentation. Along with SAM’s innovative approach to promptable segmentation, the literature rapidly evolved to address key challenges like computational efficiency and real-time performance. FastSAM, MobileSAM, and EfficientSAM each brought unique optimizations, dramatically reducing model size and inference time while maintaining competitive performance.

An important clarification to make here is that SAM is inherently different from prior works in multi-task segmentation systems, where a model is pre-trained and evaluated on a fixed set of tasks. SAM uses a transformer backbone ( worst-case quadratic complexity in terms of tokens), at higher input resolutions it has a heavy computational resource demand, which presents a hurdle to its practical deployment, especially in real-time applications. By directly training a CNN detector on only 2% (1/50) of the SA-1B dataset, the authors achieved comparable performance to SAM, but with drastically reduced computational and resource demands, enabling real-time application.

Get the Android app