Transformers Support from Huggingface with Rastervision

[agtech223]

Hi,

I am wondering, if there are any tutorial or guide available for implementing vision transformers such as SAM2 from Huggingface with rastervision library? Thank you.

[AdeelH]

Depends on the model and what you are trying to do. In general, you can

• Use any pretrained model to make predictions on geospatial imagery using Raster Vision
• Use any model architecture to train on geospatial imagery using Raster Vision, as long as its outputs can be understood by Raster Vision

I'm not sure how it will work with SAM since it's a promptable model and not a normal segmentation model. Can you share more details about what you are trying to do?

[agtech223]

Thanks for prompt response. I am trying to delineate agriculture field boundaries with finetuned SAM. I have point prompts as well as segmentation masks for my area. There is good library SamGEO, which is handling all the prompts and everything, but unfortunately does not support finetuning.

[AdeelH]

RV can train/finetune normal semantic segmentation models that take in an image and output a probability raster with one band per class (from a fixed set of classes). If the SAM finetuning is different from this (e.g. requiring the use of point prompts etc.) then that is not supported. You might still find RV useful for reading/sampling training data though.