More and better Transforms

Changes

New transforms

• Add PhaseShift Transform (#1714) by @pearguacamole
• Add FDATransform (#1734) by @vectorvp

Switch to version-independent torchvision transforms.

• If torchvision transforms v2 are available, they are used. Otherwise torchvision transforms v1 are used. For details see this comment.
• Add Transform for DetCon + MultiViewTransformV2 for torchvision.transforms.v2 (#1737)

Typing, naming & docstring improvements

• Type data/_utils (#1740), data/_helpers (#1742) and tests/models (#1744) by @vectorvp
• Cleanup: docstrings in the lightly/data subpackage (#1741) by @ChiragAgg5k
• Refactor: Update naming and remove unused package from AmplitudeRescaleTransform (#1732) by @vectorvp

Other

• Fix DINOProjectionHead BatchNorm Handling (#1729)
• Add masked average pooling for pooling with segmentation masks (DetCon)(#1739)

Many thanks to all of our contributors!

Models

• AIM: Scalable Pre-training of Large Autoregressive Image Models
• Barlow Twins: Self-Supervised Learning via Redundancy Reduction, 2021
• Bootstrap your own latent: A new approach to self-supervised Learning, 2020
• DCL: Decoupled Contrastive Learning, 2021
• DenseCL: Dense Contrastive Learning for Self-Supervised Visual Pre-Training, 2021
• DINO: Emerging Properties in Self-Supervised Vision Transformers, 2021
• FastSiam: Resource-Efficient Self-supervised Learning on a Single GPU, 2022
• I-JEPA: Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture, 2023
• MAE: Masked Autoencoders Are Scalable Vision Learners, 2021
• MSN: Masked Siamese Networks for Label-Efficient Learning, 2022
• MoCo: Momentum Contrast for Unsupervised Visual Representation Learning, 2019
• NNCLR: Nearest-Neighbor Contrastive Learning of Visual Representations, 2021
• PMSN: Prior Matching for Siamese Networks, 2022
• SimCLR: A Simple Framework for Contrastive Learning of Visual Representations, 2020
• SimMIM: A Simple Framework for Masked Image Modeling, 2021
• SimSiam: Exploring Simple Siamese Representation Learning, 2020
• SMoG: Unsupervised Visual Representation Learning by Synchronous Momentum Grouping, 2022
• SwAV: Unsupervised Learning of Visual Features by Contrasting Cluster Assignments, M. Caron, 2020
• TiCo: Transformation Invariance and Covariance Contrast for Self-Supervised Visual Representation Learning, 2022
• VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning, Bardes, A. et. al, 2022
• VICRegL: VICRegL: Self-Supervised Learning of Local Visual Features, 2022

More Transforms, Typing and Docs Improvements

Changes

New transforms

• Added RFFT2D and IRFFT2D transforms @snehilchatterjee
• Add RandomFrequencyMaskTransform @payo101
• Add GaussianMixtureMaskTransform @snehilchatterjee
• Add AmplitudeRescaleTransform @payo101
• Better support for both torchvision.transforms v1 and v2 without warnings/errors.

Added and updated docstrings

• Many improvements by @Prathamesh010, @ayush22iitbhu, @ChiragAgg5k @HarshitVashisht11

Docs improvements

• Improvements of the README.md @bhargavshirin and @kushal34712 @eltociear @Mefisto04 @ayush22iitbhu
• Improvements of other parts of the the docs and tutorials @jizhang02
• Fix examples on Windows @snehilchatterjee
• Improve CONTRIBUTING.md @Prathamesh010
• Added a back to top button for easier navigation @hackit-coder

More and better typing

• Testing typing for all python versions
• Typing of serve.py @ishaanagw
• Cleanup: _image.py and _utils.py file in data subpackage @ChiragAgg5k

Better formatting

• Move classes and public functions to top of file @fadkeabhi and @SauravMaheshkar

Other

• Print Aggregate Metrics at End of Benchmarks as Markdown Table @EricLiclair

Many thanks to all of our contributors!

Models

• AIM: Scalable Pre-training of Large Autoregressive Image Models
• Barlow Twins: Self-Supervised Learning via Redundancy Reduction, 2021
• Bootstrap your own latent: A new approach to self-supervised Learning, 2020
• DCL: Decoupled Contrastive Learning, 2021
• DenseCL: Dense Contrastive Learning for Self-Supervised Visual Pre-Training, 2021
• DINO: Emerging Properties in Self-Supervised Vision Transformers, 2021
• FastSiam: Resource-Efficient Self-supervised Learning on a Single GPU, 2022
• I-JEPA: Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture, 2023
• MAE: Masked Autoencoders Are Scalable Vision Learners, 2021
• MSN: Masked Siamese Networks for Label-Efficient Learning, 2022
• MoCo: Momentum Contrast for Unsupervised Visual Representation Learning, 2019
• NNCLR: Nearest-Neighbor Contrastive Learning of Visual Representations, 2021
• PMSN: Prior Matching for Siamese Networks, 2022
• SimCLR: A Simple Framework for Contrastive Learning of Visual Representations, 2020
• SimMIM: A Simple Framework for Masked Image Modeling, 2021
• SimSiam: Exploring Simple Siamese Representation Learning, 2020
• SMoG: Unsupervised Visual Representation Learning by Synchronous Momentum Grouping, 2022
• SwAV: Unsupervised Learning of Visual Features by Contrasting Cluster Assignments, M. Caron, 2020
• TiCo: Transformation Invariance and Covariance Contrast for Self-Supervised Visual Representation Learning, 2022
• VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning, Bardes, A. et. al, 2022
• VICRegL: VICRegL: Self-Supervised Learning of Local Visual Features, 2022

Support python 3.12

• Support python 3.12, thanks @MalteEbner
• update cosine warmup scheduler, thanks @guarin

Models

• AIM: Scalable Pre-training of Large Autoregressive Image Models
• Barlow Twins: Self-Supervised Learning via Redundancy Reduction, 2021
• Bootstrap your own latent: A new approach to self-supervised Learning, 2020
• DCL: Decoupled Contrastive Learning, 2021
• DenseCL: Dense Contrastive Learning for Self-Supervised Visual Pre-Training, 2021
• DINO: Emerging Properties in Self-Supervised Vision Transformers, 2021
• FastSiam: Resource-Efficient Self-supervised Learning on a Single GPU, 2022
• I-JEPA: Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture, 2023
• MAE: Masked Autoencoders Are Scalable Vision Learners, 2021
• MSN: Masked Siamese Networks for Label-Efficient Learning, 2022
• MoCo: Momentum Contrast for Unsupervised Visual Representation Learning, 2019
• NNCLR: Nearest-Neighbor Contrastive Learning of Visual Representations, 2021
• PMSN: Prior Matching for Siamese Networks, 2022
• SimCLR: A Simple Framework for Contrastive Learning of Visual Representations, 2020
• SimMIM: A Simple Framework for Masked Image Modeling, 2021
• SimSiam: Exploring Simple Siamese Representation Learning, 2020
• SMoG: Unsupervised Visual Representation Learning by Synchronous Momentum Grouping, 2022
• SwAV: Unsupervised Learning of Visual Features by Contrasting Cluster Assignments, M. Caron, 2020
• TiCo: Transformation Invariance and Covariance Contrast for Self-Supervised Visual Representation Learning, 2022
• VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning, Bardes, A. et. al, 2022
• VICRegL: VICRegL: Self-Supervised Learning of Local Visual Features, 2022

enhanced library compatibility

• Use TiCoTransform Everywhere
• Refactor DINOLoss to not use center module
• Add CenterCrop to val transform

Dependencies

• Make library compatible with torch 1.10, torchvision 0.11, and pytorch lightning 1.6 (by using uv), thanks @guarin

Docs

• Add notebooks, thanks @SauravMaheshkar
• Add Timm Backbone Tutorial, thanks @SauravMaheshkar
• Further docs and tutorial improvements

Models

• AIM: Scalable Pre-training of Large Autoregressive Image Models
• Barlow Twins: Self-Supervised Learning via Redundancy Reduction, 2021
• Bootstrap your own latent: A new approach to self-supervised Learning, 2020
• DCL: Decoupled Contrastive Learning, 2021
• DenseCL: Dense Contrastive Learning for Self-Supervised Visual Pre-Training, 2021
• DINO: Emerging Properties in Self-Supervised Vision Transformers, 2021
• FastSiam: Resource-Efficient Self-supervised Learning on a Single GPU, 2022
• I-JEPA: Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture, 2023
• MAE: Masked Autoencoders Are Scalable Vision Learners, 2021
• MSN: Masked Siamese Networks for Label-Efficient Learning, 2022
• MoCo: Momentum Contrast for Unsupervised Visual Representation Learning, 2019
• NNCLR: Nearest-Neighbor Contrastive Learning of Visual Representations, 2021
• PMSN: Prior Matching for Siamese Networks, 2022
• SimCLR: A Simple Framework for Contrastive Learning of Visual Representations, 2020
• SimMIM: A Simple Framework for Masked Image Modeling, 2021
• SimSiam: Exploring Simple Siamese Representation Learning, 2020
• SMoG: Unsupervised Visual Representation Learning by Synchronous Momentum Grouping, 2022
• SwAV: Unsupervised Learning of Visual Features by Contrasting Cluster Assignments, M. Caron, 2020
• TiCo: Transformation Invariance and Covariance Contrast for Self-Supervised Visual Representation Learning, 2022
• VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning, Bardes, A. et. al, 2022
• VICRegL: VICRegL: Self-Supervised Learning of Local Visual Features, 2022

NumPy 2 support and new IBOTPatchLoss, KoLeoLoss

• Added IBOTPatchLoss, KoLeoLoss and block masking, thanks @guarin
• Allow learnable positional embeddings and boolean masking in masked vision transformer
• Refactor IJEPA to use timm, thanks @radiradev

Dependencies

• Allow NumPy 2, thanks @adamjstewart
• Removed lightning-bolts dependency

Docs

• Add finetuning tutorial, thanks @SauravMaheshkar
• Fix MoCo link in DenseCL docs and further docs and tutorial improvements

Models

• AIM: Scalable Pre-training of Large Autoregressive Image Models
• Barlow Twins: Self-Supervised Learning via Redundancy Reduction, 2021
• Bootstrap your own latent: A new approach to self-supervised Learning, 2020
• DCL: Decoupled Contrastive Learning, 2021
• DenseCL: Dense Contrastive Learning for Self-Supervised Visual Pre-Training, 2021
• DINO: Emerging Properties in Self-Supervised Vision Transformers, 2021
• FastSiam: Resource-Efficient Self-supervised Learning on a Single GPU, 2022
• I-JEPA: Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture, 2023
• MAE: Masked Autoencoders Are Scalable Vision Learners, 2021
• MSN: Masked Siamese Networks for Label-Efficient Learning, 2022
• MoCo: Momentum Contrast for Unsupervised Visual Representation Learning, 2019
• NNCLR: Nearest-Neighbor Contrastive Learning of Visual Representations, 2021
• PMSN: Prior Matching for Siamese Networks, 2022
• SimCLR: A Simple Framework for Contrastive Learning of Visual Representations, 2020
• SimMIM: A Simple Framework for Masked Image Modeling, 2021
• SimSiam: Exploring Simple Siamese Representation Learning, 2020
• SMoG: Unsupervised Visual Representation Learning by Synchronous Momentum Grouping, 2022
• SwAV: Unsupervised Learning of Visual Features by Contrasting Cluster Assignments, M. Caron, 2020
• TiCo: Transformation Invariance and Covariance Contrast for Self-Supervised Visual Representation Learning, 2022
• VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning, Bardes, A. et. al, 2022
• VICRegL: VICRegL: Self-Supervised Learning of Local Visual Features, 2022

DenseCL, SSL model improvements

• Adds the DenseCL: Dense Contrastive Learning for Self-Supervised Visual Pre-Training method. See the docs.
• Add TiCoTransform, thanks @radiradev!
• Improvements to the pre-commit hooks, thanks @SauravMaheshkar!
• Fix memory bank issue when using gather_distributed=True and training on a single GPU
• Fix student head update in DINO benchmark
• Various improvements to MaskedVisionTransformer
• Renaming of Lightly SSL to LightlySSL

Models

• AIM: Scalable Pre-training of Large Autoregressive Image Models
• Barlow Twins: Self-Supervised Learning via Redundancy Reduction, 2021
• Bootstrap your own latent: A new approach to self-supervised Learning, 2020
• DCL: Decoupled Contrastive Learning, 2021
• DenseCL: Dense Contrastive Learning for Self-Supervised Visual Pre-Training, 2021
• DINO: Emerging Properties in Self-Supervised Vision Transformers, 2021
• FastSiam: Resource-Efficient Self-supervised Learning on a Single GPU, 2022
• I-JEPA: Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture, 2023
• MAE: Masked Autoencoders Are Scalable Vision Learners, 2021
• MSN: Masked Siamese Networks for Label-Efficient Learning, 2022
• MoCo: Momentum Contrast for Unsupervised Visual Representation Learning, 2019
• NNCLR: Nearest-Neighbor Contrastive Learning of Visual Representations, 2021
• PMSN: Prior Matching for Siamese Networks, 2022
• SimCLR: A Simple Framework for Contrastive Learning of Visual Representations, 2020
• SimMIM: A Simple Framework for Masked Image Modeling, 2021
• SimSiam: Exploring Simple Siamese Representation Learning, 2020
• SMoG: Unsupervised Visual Representation Learning by Synchronous Momentum Grouping, 2022
• SwAV: Unsupervised Learning of Visual Features by Contrasting Cluster Assignments, M. Caron, 2020
• TiCo: Transformation Invariance and Covariance Contrast for Self-Supervised Visual Representation Learning, 2022
• VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning, Bardes, A. et. al, 2022
• VICRegL: VICRegL: Self-Supervised Learning of Local Visual Features, 2022

pydantic2 compatibility

• Lightly is now compatible with pydantic2
• migrated to pyproject

Models

• AIM: Scalable Pre-training of Large Autoregressive Image Models
• Barlow Twins: Self-Supervised Learning via Redundancy Reduction, 2021
• Bootstrap your own latent: A new approach to self-supervised Learning, 2020
• DCL: Decoupled Contrastive Learning, 2021
• DINO: Emerging Properties in Self-Supervised Vision Transformers, 2021
• FastSiam: Resource-Efficient Self-supervised Learning on a Single GPU, 2022
• I-JEPA: Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture, 2023
• MAE: Masked Autoencoders Are Scalable Vision Learners, 2021
• MSN: Masked Siamese Networks for Label-Efficient Learning, 2022
• MoCo: Momentum Contrast for Unsupervised Visual Representation Learning, 2019
• NNCLR: Nearest-Neighbor Contrastive Learning of Visual Representations, 2021
• PMSN: Prior Matching for Siamese Networks, 2022
• SimCLR: A Simple Framework for Contrastive Learning of Visual Representations, 2020
• SimMIM: A Simple Framework for Masked Image Modeling, 2021
• SimSiam: Exploring Simple Siamese Representation Learning, 2020
• SMoG: Unsupervised Visual Representation Learning by Synchronous Momentum Grouping, 2022
• SwAV: Unsupervised Learning of Visual Features by Contrasting Cluster Assignments, M. Caron, 2020
• TiCo: Transformation Invariance and Covariance Contrast for Self-Supervised Visual Representation Learning, 2022
• VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning, Bardes, A. et. al, 2022
• VICRegL: VICRegL: Self-Supervised Learning of Local Visual Features, 2022

Numpy version compatibility

Two changes w.r.t numpy version 2:

• Make lightly itself support numpy version 2: #1561
• Disallow numpy 2.0 in the requirements, as torchvision is not yet compatible with numpy 2: #1562

For more context, see #1558

Models

• AIM: Scalable Pre-training of Large Autoregressive Image Models
• Barlow Twins: Self-Supervised Learning via Redundancy Reduction, 2021
• Bootstrap your own latent: A new approach to self-supervised Learning, 2020
• DCL: Decoupled Contrastive Learning, 2021
• DINO: Emerging Properties in Self-Supervised Vision Transformers, 2021
• FastSiam: Resource-Efficient Self-supervised Learning on a Single GPU, 2022
• I-JEPA: Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture, 2023
• MAE: Masked Autoencoders Are Scalable Vision Learners, 2021
• MSN: Masked Siamese Networks for Label-Efficient Learning, 2022
• MoCo: Momentum Contrast for Unsupervised Visual Representation Learning, 2019
• NNCLR: Nearest-Neighbor Contrastive Learning of Visual Representations, 2021
• PMSN: Prior Matching for Siamese Networks, 2022
• SimCLR: A Simple Framework for Contrastive Learning of Visual Representations, 2020
• SimMIM: A Simple Framework for Masked Image Modeling, 2021
• SimSiam: Exploring Simple Siamese Representation Learning, 2020
• SMoG: Unsupervised Visual Representation Learning by Synchronous Momentum Grouping, 2022
• SwAV: Unsupervised Learning of Visual Features by Contrasting Cluster Assignments, M. Caron, 2020
• TiCo: Transformation Invariance and Covariance Contrast for Self-Supervised Visual Representation Learning, 2022
• VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning, Bardes, A. et. al, 2022
• VICRegL: VICRegL: Self-Supervised Learning of Local Visual Features, 2022

v1.5.7

Tiny improvements

• Increase download timeout for json files (#1556)
• Migrate coverage and mypy configuration to pyproject.toml #1549). Many thanks to @SauravMaheshkar for this improvement!

Models

• AIM: Scalable Pre-training of Large Autoregressive Image Models
• Barlow Twins: Self-Supervised Learning via Redundancy Reduction, 2021
• Bootstrap your own latent: A new approach to self-supervised Learning, 2020
• DCL: Decoupled Contrastive Learning, 2021
• DINO: Emerging Properties in Self-Supervised Vision Transformers, 2021
• FastSiam: Resource-Efficient Self-supervised Learning on a Single GPU, 2022
• I-JEPA: Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture, 2023
• MAE: Masked Autoencoders Are Scalable Vision Learners, 2021
• MSN: Masked Siamese Networks for Label-Efficient Learning, 2022
• MoCo: Momentum Contrast for Unsupervised Visual Representation Learning, 2019
• NNCLR: Nearest-Neighbor Contrastive Learning of Visual Representations, 2021
• PMSN: Prior Matching for Siamese Networks, 2022
• SimCLR: A Simple Framework for Contrastive Learning of Visual Representations, 2020
• SimMIM: A Simple Framework for Masked Image Modeling, 2021
• SimSiam: Exploring Simple Siamese Representation Learning, 2020
• SMoG: Unsupervised Visual Representation Learning by Synchronous Momentum Grouping, 2022
• SwAV: Unsupervised Learning of Visual Features by Contrasting Cluster Assignments, M. Caron, 2020
• TiCo: Transformation Invariance and Covariance Contrast for Self-Supervised Visual Representation Learning, 2022
• VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning, Bardes, A. et. al, 2022
• VICRegL: VICRegL: Self-Supervised Learning of Local Visual Features, 2022

v1.5.6

Changes

• Allow lightly-serve to run securely via https by passing ssl_cert and ssl_key

Models

• AIM: Scalable Pre-training of Large Autoregressive Image Models
• Barlow Twins: Self-Supervised Learning via Redundancy Reduction, 2021
• Bootstrap your own latent: A new approach to self-supervised Learning, 2020
• DCL: Decoupled Contrastive Learning, 2021
• DINO: Emerging Properties in Self-Supervised Vision Transformers, 2021
• FastSiam: Resource-Efficient Self-supervised Learning on a Single GPU, 2022
• I-JEPA: Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture, 2023
• MAE: Masked Autoencoders Are Scalable Vision Learners, 2021
• MSN: Masked Siamese Networks for Label-Efficient Learning, 2022
• MoCo: Momentum Contrast for Unsupervised Visual Representation Learning, 2019
• NNCLR: Nearest-Neighbor Contrastive Learning of Visual Representations, 2021
• PMSN: Prior Matching for Siamese Networks, 2022
• SimCLR: A Simple Framework for Contrastive Learning of Visual Representations, 2020
• SimMIM: A Simple Framework for Masked Image Modeling, 2021
• SimSiam: Exploring Simple Siamese Representation Learning, 2020
• SMoG: Unsupervised Visual Representation Learning by Synchronous Momentum Grouping, 2022
• SwAV: Unsupervised Learning of Visual Features by Contrasting Cluster Assignments, M. Caron, 2020
• TiCo: Transformation Invariance and Covariance Contrast for Self-Supervised Visual Representation Learning, 2022
• VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning, Bardes, A. et. al, 2022
• VICRegL: VICRegL: Self-Supervised Learning of Local Visual Features, 2022