I’ve been doing some experimentation with smaller models and embeddings, including distilbert/distilbert-base-uncased-finetuned-sst-2-english and cardiffnlp/twitter-roberta-base-sentiment-latest as binary sentiment classifiers and google/vit-base-patch16-224 as an image classifier.
Also GoogleNews-vectors-negative300 and fasttext-wiki-news-subwords-300 for embeddings to try and find semantically similar words and concepts.