Skip to content

Latest commit

 

History

History
1911 lines (1280 loc) · 103 KB

Multimodel.md

File metadata and controls

1911 lines (1280 loc) · 103 KB

Recognition to Cognition Networks https://visualcommonsense.com https://github.com/rowanz/r2c

CVPR 2018 - Regularizing RNNs for Caption Generation by Reconstructing The Past with The Present https://github.com/chenxinpeng/ARNet

Beyond Narrative Description: Generating Poetry from Images by Multi-Adversarial Training https://github.com/bei21/img2poem

Dataset and starting code for visual entailment dataset https://arxiv.org/abs/1811.10582 https://github.com/necla-ml/SNLI-VE

Mapping Images to Scene Graphs with Permutation-Invariant Structured Prediction https://github.com/shikorab/SceneGraph

【视觉/语言预训练模型最新进展】'Recent Advances in Vision and Language PreTrained Models (VL-PTMs)'

https://github.com/yuewang-cuhk/awesome-vision-language-pretraining-papers

Aligning Visual Regions and Textual Concepts for Semantic-Grounded Image Representations https://github.com/fenglinliu98/MIA

Tensorflow implementation of "A Structured Self-Attentive Sentence Embedding" https://github.com/flrngel/Self-Attentive-tensorflow

This repository contains the reference code for the paper Show, Control and Tell: A Framework for Generating Controllable and Grounded Captions (CVPR 2019). https://github.com/aimagelab/show-control-and-tell

The Code for ICME2019 Grand Challenge: Short Video Understanding (Single Model Ranks 6th) https://github.com/guoday/ICME2019-CTR

【基于Transformer的图像自动描述PyTorch/Fairseq扩展】 https://github.com/krasserm/fairseq-image-captioning

Code for Neural Inverse Knitting: From Images to Manufacturing Instructions https://github.com/xionluhnis/neural_inverse_knitting

Code for paper "Attention on Attention for Image Captioning". ICCV 2019 https://arxiv.org/abs/1908.06954 https://github.com/husthuaan/AoANet

Learning to Evaluate Image Captioning. CVPR 2018 https://github.com/richardaecn/cvpr18-caption-eval

Vision-Language Pre-training for Image Captioning and Question Answering https://github.com/LuoweiZhou/VLP

Official Tensorflow Implementation of the paper "Bidirectional Attentive Fusion with Context Gating for Dense Video Captioning" in CVPR 2018, with code, model and prediction results. https://github.com/JaywongWang/DenseVideoCaptioning

A PyTorch implementation of Transformer in "Attention is All You Need" https://arxiv.org/abs/1706.03762 https://github.com/dreamgonfly/Transformer-pytorch

《ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Data》 https://www.arxiv-vanity.com/papers/2001.07966/

【结合BERT的图片描述生成】’Image Captioning System - BERT + Image Captioning' https://github.com/ajamjoom/Image-Captions

M^2: Meshed-Memory Transformer for Image Captioning https://github.com/aimagelab/meshed-memory-transformer

Video Grounding and Captioning

https://github.com/facebookresearch/grounded-video-description

Reformer:高效的Transformer https://github.com/google/trax/tree/master/trax/models/reformer

ICCV研讨会的中英文视频描述大赛 http://vatex.org/main/index.html

Cooperative Vision-and-Dialog Navigation https://github.com/mmurray/cvdn

Auto-Encoding Scene Graphs for Image Captioning, CVPR 2019 https://github.com/yangxuntu/SGAE

A PyTorch implementation of the Transformer model from "Attention Is All You Need". https://github.com/phohenecker/pytorch-transformer

【MMF:基于PyTorch的视觉/语言研究模块化框架,可方便进行VQA、图像描述、视觉对话、仇恨检测和其他视觉/语言任务的研究】 https://github.com/facebookresearch/mmf

基于transformers的图像Instagram标题生成 https://github.com/antoninodimaggio/Hugging-Captions

Implementation of 'X-Linear Attention Networks for Image Captioning' [CVPR 2020] https://github.com/JDAI-CV/image-captioning

Train Scene Graph Generation for Visual Genome and GQA in PyTorch >= 1.2 with improved zero and few-shot generalization. Paper: "Graph Density-Aware Losses for Novel Compositions in Scene Graph Generation" https://github.com/bknyaz/sgg

Code and Resources for the Transformer Encoder Reasoning Network (TERN) - https://arxiv.org/abs/2004.09144 https://github.com/mesnico/TERN

Code for ACL 2020 paper "Dense-Caption Matching and Frame-Selection Gating for Temporal Localization in VideoQA." https://github.com/hyounghk/VideoQADenseCapFrameGate-ACL2020

[ACL 2020] PyTorch code for MART: Memory-Augmented Recurrent Transformer for Coherent Video Paragraph Captioning https://github.com/jayleicn/recurrent-transformer

PyTorch code for: Learning to Generate Grounded Visual Captions without Localization Supervision https://github.com/chihyaoma/cyclical-visual-captioning

Say As You Wish: Fine-grained Control of Image Caption Generation with Abstract Scene Graphs https://github.com/cshizhe/asg2cap

Fine-grained Video-Text Retrieval with Hierarchical Graph Reasoning https://github.com/cshizhe/hgr_v2t

The repository of ECCV 2020 paper Active Visual Information Gathering for Vision-Language Navigation https://github.com/HanqingWangAI/Active_VLN

Code for the CVPR 2020 oral paper: Weakly Supervised Visual Semantic Parsing https://github.com/alirezazareian/vspnet https://arxiv.org/abs/2001.02359

Learning Visual Representations with Caption Annotations https://arxiv.org/abs/2008.01392

Towards Unique and Informative Captioning of Images https://github.com/princetonvisualai/SPICE-U

计算机如何做到“看图说话”?在没有对应的训练数据的情况下,模型能否准确描述测试图像中新出现的各种类别的物体?微软 Azure 认知服务团队和微软研究院的研究员提出了全新解决方案视觉词表预训练 (Visual Vocabulary Pre-training)。该方法在 nocaps 挑战中取得了新的 SOTA,并首次超越人类表现 https://weibo.com/ttarticle/p/show?id=2309404559989682602061

一个视频的文本摘要生成项目,输入一段视频,通过深度学习网络和人工智能程序识别视频主要表达的意思 https://github.com/CaptainEven/VideoCaption

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale https://github.com/google-research/vision_transformer

CapWAP: Captioning with a Purpose

https://arxiv.org/abs/2011.04264 https://github.com/google-research/language/tree/master/language/capwap

Transformer视觉表示学习文献资源列表 https://github.com/alohays/awesome-visual-representation-learning-with-transformers

Online Bag-of-Visual-Words Generation for Unsupervised Representation Learning https://github.com/valeoai/obow

VinVL: Making Visual Representations Matter in Vision-Language Models https://arxiv.org/abs/2101.00529

Cross-Document Language Modeling https://github.com/aviclu/CD-LM https://arxiv.org/abs/2101.00406

Transformers in Vision: A Survey https://arxiv.org/abs/2101.01169

视觉Transformer相关工作列表 https://github.com/dk-liang/Awesome-Visual-Transformer

实例教程:(PyTorch)从头实现Transformer https://colab.research.google.com/drive/1swXWW5sOLW8zSZBaQBYcGQkQ_Bje_bmI

Cross-Modal Contrastive Learning for Text-to-Image Generation https://arxiv.org/abs/2101.04702

Is Attention Better Than Matrix Decomposition? https://openreview.net/forum?id=1FvkSpWosOl

Pre-training without Natural Images https://arxiv.org/abs/2101.08515

胸片为例的医学图像自动描述 https://towardsdatascience.com/medical-image-captioning-on-chest-x-rays-a43561a6871d

图:完全理解Transformer的注意力机制 https://imgur.com/gallery/vuw15aL

视频描述生成相关文献列表

https://github.com/tgc1997/Awesome-Video-Captioning

Keras实例:Vision Transformer图像分类 https://keras.io/examples/vision/image_classification_with_vision_transformer/

CPTR: Full Transformer Network for Image Captioning

https://arxiv.org/abs/2101.10804

(Colab)基于CLIP的Unsplash图片语义搜索 https://github.com/haltakov/natural-language-image-search

Language-Mediated, Object-Centric Representation Learning https://arxiv.org/abs/2012.15814

Vx2Text: End-to-End Learning of Video-Based Text Generation From Multimodal Inputs https://arxiv.org/abs/2101.12059

Bottleneck Transformers for Visual Recognition https://arxiv.org/abs/2101.11605

用TensorFlow Serving快速部署Transformers https://huggingface.co/blog/tf-serving

Video Transformer Network https://arxiv.org/abs/2102.00719

Decoupling the Role of Data, Attention, and Losses in Multimodal Transformers https://arxiv.org/abs/2102.00529

(Colab)结合CLIP的文本-图像生成 https://cdn.openai.com/papers/Learning_Transferable_Visual_Models_From_Natural_Language_Supervision.pdf

Learning to Discretely Compose Reasoning Module Networks for Video Captioning

https://github.com/tgc1997/RMN

Comprehensive Image Captioning via Scene Graph Decomposition https://github.com/YiwuZhong/Sub-GC

Look and Modify: Modification Networks for Image Captioning https://github.com/fawazsammani/look-and-modify

Understanding the Difficulty of Training Transformers https://github.com/LiyuanLucasLiu/Transformer-Clinic

Latent Normalizing Flows for Many-to-Many Cross Domain Mappings https://github.com/visinf/lnfmm

用编程方式理解Transformers

https://towardsdatascience.com/understanding-transformers-the-programming-way-f8ed22d112b2

transformer的十七大变种优化 https://mp.weixin.qq.com/s/k5XkAwcXkyi9ymFUIej_iw

Transformers与图神经网络 https://github.com/chaitjo/gated-graph-transformers

Transformer语言模型可视化解析

https://jalammar.github.io/explaining-transformers/

Zelda Rose:Transformer模型训练器,支持分布式训练 https://github.com/LoicGrobol/zeldarose

Unifying Vision-and-Language Tasks via Text Generation https://github.com/j-min/VL-T5

Source code for "Bi-modal Transformer for Dense Video Captioning" https://github.com/v-iashin/BMT

Visual Relation Grounding in Videos https://github.com/doc-doc/vRGV

MMT: Multi-modal Transformer for Video Retrieval https://github.com/gabeur/mmt

Research code for EMNLP 2020 paper "HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training" https://github.com/linjieli222/HERO

Zelda Rose:Transformer模型训练器,支持分布式训练 https://github.com/LoicGrobol/zeldarose

Implementation of Vision Transformer in PyTorch, a new model to achieve SOTA in vision classification with using transformer style encoders https://github.com/gupta-abhay/ViT

Data Movement Is All You Need: A Case Study on Optimizing Transformers. https://github.com/spcl/substation

VisualCOMET: Reasoning about the Dynamic Context of a Still Image

https://github.com/jamespark3922/visual-comet

This is the official Pytorch implementation of Length-Adaptive Transformer. https://github.com/clovaai/length-adaptive-transformer

AR-Net: Adaptive Resolution Network for Efficient Video Understanding https://github.com/mengyuest/AR-Net

Code base for WaveTransformer: A novel architecture for automated audio captioning https://github.com/haantran96/wavetransformer

The official source code for the paper Consensus-Aware Visual-Semantic Embedding for Image-Text Matching (ECCV 2020) https://github.com/BruceW91/CVSE

Research Code for NeurIPS 2020 Spotlight paper "Large-Scale Adversarial Training for Vision-and-Language Representation Learning": LXMERT adversarial training part https://github.com/zhegan27/LXMERT-AdvTrain

COOT: Cooperative Hierarchical Transformer for Video-Text Representation Learning https://github.com/gingsi/coot-videotext

A PyTorch implementation of the paper - "Synthesizer: Rethinking Self-Attention in Transformer Models" https://github.com/10-zin/Synthesizer

iPerceive: Applying Common-Sense Reasoning to Multi-Modal Dense Video Captioning and Video Question Answering https://github.com/amanchadha/iPerceive

Video Feature Extraction Code for EMNLP 2020 paper "HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training" https://github.com/linjieli222/HERO_Video_Feature_Extractor

Big Bird: Transformers for Longer Sequences https://github.com/google-research/bigbird

Code and data for the framework in "Multimodal Pretraining Unmasked: Unifying the Vision and Language BERTs", arXiv preprint arXiv:2011.15124. https://github.com/e-bug/volta

Self-Supervised Learning by Cross-Modal Audio-Video Clustering https://github.com/HumamAlwassel/XDC

Multi-Modal Reasoning Graph for Scene-Text Based Fine-Grained Image Classification and Retrieval https://github.com/AndresPMD/GCN_classification

Implementation of RealFormer using pytorch https://github.com/cloneofsimo/RealFormer-pytorch

Tensorflow implementation of the Vision Transformer (An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale) https://github.com/emla2805/vision-transformer

(Pytorch) Visual Transformers: Token-based Image Representation and Processing for Computer Vision: https://github.com/tahmid0007/VisualTransformers

Code and data for the project "Visually grounded continual learning of compositional semantics" https://github.com/INK-USC/VisCOLL

Connecting Vision and Language with Localized Narratives

https://google.github.io/localized-narratives/ https://github.com/google/localized-narratives

An implementation of Performer, a linear attention-based transformer, in Pytorch https://github.com/lucidrains/performer-pytorch

DSNet: A Flexible Detect-to-Summarize Network for Video Summarization https://github.com/li-plus/DSNet

Learning to ground explanations of affect for visual art. https://github.com/optas/artemis

Keras Implementation of Vision Transformer (An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale) https://github.com/tuvovan/Vision_Transformer_Keras

Implementation of "Multimodal Text Style Transfer for Outdoor Vision-and-Language Navigation" https://github.com/VegB/VLN-Transformer

Code of paper: A Recurrent Vision-and-Language BERT for Navigation https://github.com/YicongHong/Recurrent-VLN-BERT

Implementation of Bottleneck Transformer in Pytorch https://github.com/lucidrains/bottleneck-transformer-pytorch

Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet https://github.com/yitu-opensource/T2T-ViT

ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision https://arxiv.org/abs/2102.03334

Learning Spatiotemporal Features via Video and Text Pair Discrimination https://github.com/MCG-NJU/CPD-Video

The implementation of paper UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generation. https://github.com/microsoft/UniVL

Implementation of Feedback Transformer in Pytorch https://github.com/lucidrains/feedback-transformer-pytorch

code of the paper "Vision-Language Navigation with Multi-granularity Observation and Auxiliary Reasoning Tasks" https://github.com/ZhuFengdaaa/MG-AuxRN

We rank the 1st in DSTC8 Audio-Visual Scene-Aware Dialog competition. This is the source code for our AAAI2020-DSTC8-AVSD paper "Bridging Text and Video: A Universal Multimodal Transformer for Video-Audio Scene-Aware Dialog". https://github.com/ictnlp/DSTC8-AVSD

Implementation of TimeSformer, a pure attention-based solution for video classification https://github.com/lucidrains/TimeSformer-pytorch

Tensorflow implementation of the Vision Transformer (An Image is Worth 16x16 Words: Transformer https://github.com/ashishpatel26/Vision-Transformer-Keras-Tensorflow-Pytorch-Examples

基于CLIP用自然语言搜索Youtube视频 https://github.com/haltakov/natural-language-youtube-search

Less is More: ClipBERT for Video-and-Language Learning via Sparse Sampling https://github.com/jayleicn/ClipBERT

Is Space-Time Attention All You Need for Video Understanding? https://arxiv.org/abs/2102.05095

Training Vision Transformers for Image Retrieval https://arxiv.org/abs/2102.05644

【自注意力分类图】“A Visual Self-Attention Taxonomy” https://paperswithcode.com/newsletter/4/

Optimizing Inference Performance of Transformers on CPUs https://arxiv.org/abs/2102.06621

Transformer计算机视觉应用研究列表 https://github.com/DirtyHarryLYL/Transformer-in-Vision

Transformers Interpret:Transformer模型解释工具 https://github.com/cdpierse/transformers-interpret

我们能否用自然语言模型描述图片视觉差异?最近本组的EACL 2021工作发现,必须从语义上理解单个图片,同时对多个图片进行对比,才可以很好地用文字描述视觉上的细微差别。L2C: Describing Visual Differences Needs Semantic Understanding of Individuals https://arxiv.org/abs/2102.01860

Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts https://arxiv.org/abs/2102.08981

Transformer及其应用参考资料汇编 https://elvissaravia.substack.com/p/learn-about-transformers-a-recipe

Inferring spatial relations from textual descriptions of images https://arxiv.org/abs/2102.00997 https://github.com/ixa-ehu/rec-coco

The source code of ACL 2020 paper: "Cross-Modality Relevance for Reasoning on Language and Vision" https://arxiv.org/abs/2005.06035 https://github.com/HLR/Cross_Modality_Relevance

Text-to-Image Generation Grounded by Fine-Grained User Attention https://github.com/google-research/trecs_image_generation

Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions

https://github.com/whai362/PVT

Position Information in Transformers: An Overview https://arxiv.org/abs/2102.11090

Do Transformer Modifications Transfer Across Implementations and Applications? https://arxiv.org/abs/2102.11972

Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions

https://github.com/whai362/PVT https://arxiv.org/abs/2102.12122

Zero-Shot Text-to-Image Generation https://github.com/openai/DALL-E https://arxiv.org/abs/2102.12092

Transformer 一篇就够了 - 记录了学习Transformer过程中的一些疑问和解答,并且实现Transformer的全过程 https://github.com/BSlience/transformer-all-in-one

大规模Transformer图像识别 https://ai.googleblog.com/2020/12/transformers-for-image-recognition-at.html https://github.com/google-research/vision_transformer

FastFormers:CPU上的233倍速Transformer推理

https://medium.com/ai-in-plain-english/fastformers-233x-faster-transformers-inference-on-cpu-4c0b7a720e1

Towards Accurate and Compact Architectures via Neural Architecture Transformer https://arxiv.org/abs/2102.10301

Do We Really Need Explicit Position Encodings for Vision Transformers? https://arxiv.org/abs/2102.10882 https://github.com/Meituan-AutoML/CPVT

Transformer及其NLP应用的直观介绍 https://theaisummer.com/transformer/

OmniNet: Omnidirectional Representations from Transformers https://www.arxiv-vanity.com/papers/2103.01075

Duel-Level Collaborative Transformer for Image Captioning https://github.com/luo3300612/image-captioning-DLCT

《Coordinate Attention for Efficient Mobile Network Design》(CVPR 2021) github.com/Andrew-Qibin/CoordAttention

Energy-Based Learning for Scene Graph Generation github.com/mods333/energy-based-scene-graph

长程Transformers论文解读 https://huggingface.co/blog/long-range-transformers

Improving Image Captioning Evaluation by Considering Inter References Variance(ACL 2020) https://github.com/ck0123/improved-bertscore-for-image-captioning-evaluation

《Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth》 https://github.com/twistedcubic/attention-rank-collapse

WenLan: Bridging Vision and Language by Large-Scale Multi-Modal Pre-Training https://www.arxiv-vanity.com/papers/2103.06561

Perspectives and Prospects on Transformer Architecture for Cross-Modal Tasks with Language and Vision https://www.arxiv-vanity.com/papers/2103.04037

基于OpenAI CLIP模型的Unsplash图片语义搜索 github.com/haofanwang/natural-language-joint-query-search

DeepViT: Towards Deeper Vision Transformer https://www.arxiv-vanity.com/papers/2103.11886

《An Image is Worth 16x16 Words, What is a Video Worth?》 https://www.arxiv-vanity.com/papers/2103.13915 github.com/Alibaba-MIIL/STAM

Can Vision Transformers Learn without Natural Images? https://www.arxiv-vanity.com/papers/2103.13023

Meta-DETR: Few-Shot Object Detection via Unified Image-Level Meta-Learning https://www.arxiv-vanity.com/papers/2103.11731

《Dodrio: Exploring Transformer Models with Interactive Visualization》 github.com/poloclub/dodrio

《A Practical Survey on Faster and Lighter Transformers》 https://www.arxiv-vanity.com/papers/2103.14636

multimodal:面向VQA和图像描述研究的多模态数据集和视觉特征库,可直接用pip安装

github.com/cdancette/multimodal

《VinVL: Revisiting Visual Representations in Vision-Language Models》(ECCV 2021) github.com/pzzhang/VinVL

《Scaling Local Self-Attention For Parameter Efficient Visual Backbones》(2021)

github.com/lucidrains/halonet-pytorch

《RoFormer: Transformer with Rotary Position Embeddings》(2021) github.com/ZhuiyiTechnology/roformer

《Swin Transformer: Hierarchical Vision Transformer using Shifted Windows》(2021) github.com/microsoft/Swin-Transformer

《Diverse Image Captioning with Context-Object Split Latent Spaces》(NeurIPS 2020) github.com/visinf/cos-cvae

《Self-Attention Attribution: Interpreting Information Interactions Inside Transformer》(AAAI 2021) github.com/YRdddream/attattr

《An Image is Worth 16x16 Words, What is a Video Worth?》(2021)

github.com/lucidrains/STAM-pytorch

CvT: Introducing Convolutions to Vision Transformers https://arxiv.org/abs/2103.15808

ViViT: A Video Vision Transformer https://www.arxiv-vanity.com/papers/2103.15691

Thinking Fast and Slow: Efficient Text-to-Visual Retrieval with Transformers https://www.arxiv-vanity.com/papers/2103.16553

Rethinking Spatial Dimensions of Vision Transformers github.com/naver-ai/pit

Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encoding https://www.arxiv-vanity.com/papers/2104.00298

Going deeper with Image Transformers https://www.arxiv-vanity.com/papers/2103.17239

视觉Transformer的大爆发 https://paperswithcode.com/newsletter/7/

On the Adversarial Robustness of Visual Transformers https://www.arxiv-vanity.com/papers/2103.15670

Towards General Purpose Vision Systems https://www.arxiv-vanity.com/papers/2104.00743

FasterTransformer:更快的FasterTransformer,包括Transformer相关优化、GPT、BERT等 github.com/NVIDIA/FasterTransformer

多种视觉Transformer的PyTorch实现 github.com/rosinality/vision-transformers-pytorch

Language-based Video Editing via Multi-Modal Multi-Level Transformer https://www.arxiv-vanity.com/papers/2104.01122

LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inference https://www.arxiv-vanity.com/papers/2104.01136

屠榜各大 CV 任务,微软开源的 Swin Transformer 有多强? https://weibo.com/ttarticle/p/show?id=2309404626713857622141#related https://www.zhihu.com/question/437495132/answer/1800881612

Swin Transformer: Hierarchical Vision Transformer using Shifted Windows (2021) github.com/berniwal/swin-transformer-pytorch github.com/SwinTransformer/Swin-Transformer-Object-Detection

Fourier Image Transformer github.com/juglab/FourierImageTransformer

视觉-语言相关资源大列表 github.com/sangminwoo/awesome-vision-and-language

句子-视频段定位相关文献与基准列表 github.com/Soldelli/Awesome-Temporal-Language-Grounding-in-Videos

SiT: Self-supervised vIsion Transformer github.com/Sara-Ahmed/SiT

PyTorchVideo:Facebook面向视频理解研究的深度学习库 github.com/facebookresearch/pytorchvideo

OpenAI DALLE复现 github.com/kobiso/DALLE-reproduction

LXMERT: Learning Cross-Modality Encoder Representations from Transformers github.com/yangxuntu/lxmertcatt

TransFG: A Transformer Architecture for Fine-grained Recognition (2021) github.com/TACJu/TransFG

Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformers(2021) github.com/hila-chefer/Transformer-MM-Explainability

UniMoCo: Unsupervised, Semi-Supervised and Full-Supervised Visual Representation Learning github.com/dddzg/unimoco

ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biases github.com/facebookresearch/convit

《DeepViT: Towards Deeper Vision Transformer》(2021) github.com/zhoudaquan/dvit_repo

《Visual Semantic Role Labeling for Video Understanding》(CVPR 2021) github.com/TheShadow29/VidSitu

《Blow the Dog Whistle: A Chinese Dataset for Cant Understanding with Common Sense and World Knowledge》(NAACL 2021) github.com/JetRunner/dogwhistle

《Location-Sensitive Visual Recognition with Cross-IOU Loss》(2021) github.com/Duankaiwen/LSNet

《CeiT : Convolutional enhanced image Transformer》(2021) github.com/rishikksh20/CeiT-pytorch

《Syntax-Aware Action Targeting for Video Captioning》(CVPR 2020) github.com/SydCaption/SAAT

VideoGPT: Video Generation using VQ-VAE and Transformers github.com/wilson1yan/VideoGPT

RoFormer: Enhanced Transformer with Rotary Position Embedding https://www.arxiv-vanity.com/papers/2104.09864

EET:针对Transformer-based大模型和长序列场景的高性能PyTorch推理插件 github.com/NetEase-FuXi/EET

Lightning Transfomers:基于Pytorch Lightning, Transformers, Hydra做高性能研究的集成界面,提供了灵活接口用于训练和微调最新Transformer模型 github.com/PyTorchLightning/lightning-transformers

NLP实操手册: 基于Transformer的深度学习架构的应用指南(综述) https://www.arxiv-vanity.com/papers/2104.10640

MaX-DeepLab:面向端到端全景分割的双通路Transformer https://arxiv.org/abs/2012.00759

Hierarchical Cross-Modal Agent for Robotics Vision-and-Language Navigation https://www.arxiv-vanity.com/papers/2104.10674

《So-ViT: Mind Visual Tokens for Vision Transformer》(2021) github.com/jiangtaoxie/So-ViT

VideoGPT: Video Generation using VQ-VAE and Transformers (2021) github.com/wilson1yan/VideoGPT

Learning the Best Pooling Strategy for Visual Semantic Embedding (CVPR 2021) github.com/woodfrog/vse_infty

Improving Vision-and-Language Navigation with Image-Text Pairs from the Web (ECCV 2020) github.com/arjunmajum/vln-bert

Zero-Shot Detection via Vision and Language Knowledge Distillation https://www.arxiv-vanity.com/papers/2104.13921

面向视觉的Transformer iaml-it.github.io/posts/2021-04-28-transformers-in-vision/

DINO与PAWS:训练速度提高10倍的自监督Transformer,无监督视觉分割、分类新水准 https://ai.facebook.com/blog/dino-paws-computer-vision-with-self-supervised-transformers-and-10x-more-efficient-training/

交叉描述:图像与文本的语义相似性

https://www.aclweb.org/anthology/2021.eacl-main.249/ https://ai.googleblog.com/2021/05/crisscrossed-captions-semantic.html

OpenAI-CLIP:CLIP的简单PyTorch实现 github.com/moein-shariatnia/OpenAI-CLIP

Transformers、BERT、ELMo、GPT1论文笔记注释版 github.com/shreyansh26/Annotated-ML-Papers

Relative Positional Encoding for Transformers with Linear Complexity https://www.arxiv-vanity.com/papers/2105.08399/

跨模态对比学习文本-图像生成 https://arxiv.org/abs/2101.04702

CogView: Mastering Text-to-Image Generation via Transformers github.com/THUDM/CogView

Transformers-Tutorials:HuggingFace Transformers库实例教程集 github.com/NielsRogge/Transformers-Tutorials

train-CLIP:PyTorch Lightning实现的OpenAI's CLIP训练方案 github.com/Zasder3/train-CLIP

ConViT:用软卷积归纳偏差改进视觉Transformer https://wandb.ai/wandb_fc/pytorch-image-models/reports/ConViT-Improving-Vision-Transformers-with-Soft-Convolutional-Inductive-Biases--Vmlldzo3MjIxMDk

Less is More: Pay Less Attention in Vision Transformers https://www.arxiv-vanity.com/papers/2105.14217/

Not All Images are Worth 16x16 Words: Dynamic Vision Transformers with Adaptive Sequence Length github.com/blackfeather-wang/Dynamic-Vision-Transformer

Token Labeling: Training a 85.4% Top-1 Accuracy Vision Transformer with 56M Parameters on ImageNet github.com/zihangJiang/TokenLabeling

《Twins: Revisiting the Design of Spatial Attention in Vision Transformers》 github.com/Meituan-AutoML/Twins

LightSeq: A High Performance Inference Library for Transformers github.com/bytedance/lightseq

《X-LXMERT: Paint, Caption and Answer Questions with Multi-Modal Transformers》(EMNLP 2020) github.com/allenai/x-lxmert

《ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision》(ICML 2021) github.com/dandelin/vilt

《LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inference》 github.com/facebookresearch/LeViT

《Neuro-Symbolic Representations for Video Captioning: A Case for Leveraging Inductive Biases for Vision and Language》(2020) github.com/hassanhub/R3Transformer

《Beyond Categorical Label Representations for Image Classification》(ICLR 2021) github.com/BoyuanChen/label_representations 图像描述其实是图像分类的扩展,是否能吸取这篇论文中的想法,在标签表示方面创新?

《Referring Image Segmentation via Cross-Modal Progressive Comprehension》(CVPR 2020) github.com/spyflying/CMPC-Refseg

《Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learning》(CVPR 2021) github.com/researchmm/soho

《Vision Transformers are Robust Learners》(2021) github.com/sayakpaul/robustness-vit

《CvT: Introducing Convolutions to Vision Transformers》(2021) github.com/microsoft/CvT

《ResT: An Efficient Transformer for Visual Recognition》(2021) github.com/wofmanaf/ResT

《CrossViT : Cross-Attention Multi-Scale Vision Transformer for Image Classification》(2021) github.com/rishikksh20/CrossViT-pytorch

《Not All Images are Worth 16x16 Words: Dynamic Vision Transformers with Adaptive Sequence Length》(2021) github.com/blackfeather-wang/Dynamic-Vision-Transformer

《You Only Look at One Sequence: Rethinking Transformer in Vision through Object Detection》(2021) github.com/hustvl/YOLOS

《CvT: Introducing Convolutions to Vision Transformers》(2021) github.com/leoxiaobin/CvT

DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification github.com/raoyongming/DynamicViT

An Attention Free Transformer https://www.arxiv-vanity.com/papers/2105.14103

When Vision Transformers Outperform ResNets without Pretraining or Strong Data Augmentations https://www.arxiv-vanity.com/papers/2106.01548

教程:如何创建和训练视觉Transformer https://theaisummer.com/hugging-face-vit/

Semantic Correspondence with Transformers github.com/SunghwanHong/CATs

Beyond Self-attention: External Attention using Two Linear Layers for Visual Tasks https://www.arxiv-vanity.com/papers/2105.02358/

X-volution: On the unification of convolution and self-attention https://www.arxiv-vanity.com/papers/2106.02253/

Attention Free Transformer github.com/rish-16/aft-pytorch

各种注意力机制的PyTorch实现 github.com/xmu-xiaoma666/External-Attention-pytorch

On Improving Adversarial Transferability of Vision Transformers github.com/Muzammal-Naseer/Improving-Adversarial-Transferability-of-Vision-Transformers

MERLOT: Multimodal Neural Script Knowledge Models https://www.arxiv-vanity.com/papers/2106.02636/

Refiner: Refining Self-attention for Vision Transformers github.com/zhoudaquan/Refiner_ViT

A Survey of Transformers https://www.arxiv-vanity.com/papers/2106.04554

Scaling Vision Transformers https://www.arxiv-vanity.com/papers/2106.04560

Demystifying Local Vision Transformer: Sparse Connectivity, Weight Sharing, and Dynamic Weight https://www.arxiv-vanity.com/papers/2106.04263

Transformed CNNs: recasting pre-trained convolutional layers with self-attention https://www.arxiv-vanity.com/papers/2106.05795

微软联合UCSB、UCSC、UNC等单位合作推出的视频与语言理解基准测试集 VALUE: Video-and-Language Understanding Evaluation https://arxiv.org/pdf/2106.04632.pdf

Space-time Mixing Attention for Video Transformer https://www.arxiv-vanity.com/papers/2106.05968

《CAT: Cross Attention in Vision Transformer》(2021) github.com/linhezheng19/CAT

Keras实例:Transformer视频分类 https://keras.io/examples/vision/video_transformers/

Towards General Purpose Vision Systems:面向通用视觉的任务不可知视觉语言架构 github.com/allenai/gpv-1

Invertible Attention github.com/Schwartz-Zha/InvertibleAttention

《Episodic Transformer for Vision-and-Language Navigation》 github.com/alexpashevich/E.T.

How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers https://www.arxiv-vanity.com/papers/2106.10270

Memory-efficient Transformers via Top-k Attention github.com/ag1988/top_k_attention

XCiT: Cross-Covariance Image Transformers github.com/facebookresearch/xcit

Stable, Fast and Accurate: Kernelized Attention with Relative Positional Encoding https://www.arxiv-vanity.com/papers/2106.12566

Towards Long-Form Video Understanding github.com/chaoyuaw/lvu

Video Swin Transformer github.com/SwinTransformer/Video-Swin-Transformer

Early Convolutions Help Transformers See Better https://www.arxiv-vanity.com/papers/2106.14881/

CLIPDraw: Exploring Text-to-Drawing Synthesis through Language-Image Encoders https://arxiv.org/abs/2106.14843 https://colab.research.google.com/github/kvfrans/clipdraw/blob/main/clipdraw.ipynb

Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under Data Augmentation https://arxiv.org/abs/2107.00644

CLIP-It! Language-Guided Video Summarization https://arxiv.org/abs/2107.00650

Focal Self-attention for Local-Global Interactions in Vision Transformers https://arxiv.org/abs/2107.00641

CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windows https://arxiv.org/abs/2107.00652

Rethinking Positional Encoding github.com/osiriszjq/Rethinking-positional-encoding

Long-Short Transformer: Efficient Transformers for Language and Vision https://arxiv.org/abs/2107.02192

Transformers与卷积网络的结合 https://ai.facebook.com/blog/computer-vision-combining-transformers-and-convolutional-neural-networks https://arxiv.org/abs/2103.10697

CMT: Convolutional Neural Networks Meet Vision Transformers https://arxiv.org/abs/2107.06263

VidLanKD: Improving Language Understanding via Video-Distilled Knowledge Transfer github.com/zinengtang/VidLanKD

From Show to Tell: A Survey on Image Captioning https://arxiv.org/abs/2107.06912

How Much Can CLIP Benefit Vision-and-Language Tasks? github.com/clip-vil/CLIP-ViL

Awesome Text-to-Image:文本-图像合成相关资源大列表 github.com/Yutong-Zhou-cv/Awesome-Text-to-Image

《An Empirical Study of Training Self-Supervised Vision Transformers》(2021) github.com/CupidJay/MoCov3-pytorch

《Towards Long-Form Video Understanding》(CVPR 2021) github.com/chaoyuaw/lvu

《Video Swin Transformer》(2021) github.com/SwinTransformer/Video-Swin-Transformer

《Shuffle Transformer: Rethinking Spatial Shuffle for Vision Transformer》(2021) github.com/mulinmeng/Shuffle-Transformer

Align before Fuse: Vision and Language Representation Learning with Momentum Distillation github.com/salesforce/ALBEF/

memery:用自然语言进行大规模图片数据集语义搜索 github.com/deepfates/memery

Playground for CLIP-like models github.com/kevinzakka/clip_playground

《Contextual Transformer Networks for Visual Recognition》 github.com/JDAI-CV/CoTNet

Transformer视觉研究文献列表 github.com/Yangzhangcst/Transformer-in-Computer-Vision

OpenCLIP:CLIP的开源实现(PyTorch) #TODO github.com/mlfoundations/open_clip

ACL 2021 | 时空可控的图片描述生成 https://weibo.com/ttarticle/p/show?id=2309404664630223176058 https://aclanthology.org/2021.acl-long.157.pdf

CLIP:用无监督NLP提高患者护理连续性的大规模标注数据集,包含超过100,000个带有标注的句子,在718个完整的出院摘要中对每个句子进行标记,指定这个句子是否包含一个后续行动项 https://www.asapp.com/blog/introducing-clip-a-dataset-to-improve-continuity-of-patient-care-with-unsupervised-nlp/

ReFormer: The Relational Transformer for Image Captioning #READ https://arxiv.org/abs/2107.14178

Hugging Face新上线的DALL·E mini:根据文字提示自动生成图片 https://huggingface.co/spaces/flax-community/dalle-mini

Rethinking and Improving Relative Position Encoding for Vision Transformer github.com/microsoft/AutoML/tree/main/iRPE

DETR注释版 amaarora.github.io/2021/07/26/annotateddetr.html

Cycle-Consistent Inverse GAN for Text-to-Image Synthesis 条件styleGAN文本-》图像 https://arxiv.org/abs/2108.01361

Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformer https://arxiv.org/abs/2108.01390

Vision Transformer with Progressive Sampling https://arxiv.org/abs/2108.01684

PSViT: Better Vision Transformer via Token Pooling and Attention Sharing https://arxiv.org/abs/2108.03428

The Right to Talk: An Audio-Visual Transformer Approach github.com/uark-cviu/Right2Talk

RaftMLP: Do MLP-based Models Dream of Winning Over Computer Vision? github.com/okojoalg/raft-mlp

计算机视觉Transformer应用文献资源集 github.com/Yutong-Zhou-cv/Awesome-Transformer-in-CV

Billion-Scale Pretraining with Vision Transformers for Multi-Task Visual Representations https://arxiv.org/abs/2108.05887

Image Retrieval on Real-life Images with Pre-trained Vision-and-Language Models https://arxiv.org/abs/2108.04024

多尺度视觉Transformer:视觉数据建模架构 https://ai.facebook.com/blog/multiscale-vision-transformers-an-architecture-for-modeling-visual-data/

tldr-transformers:Transformer论文笔记集 github.com/will-thompson-k/tldr-transformers

Mobile-Former: Bridging MobileNet and Transformer https://arxiv.org/abs/2108.05895

Screen2Words: Automatic Mobile UI Summarization with Multimodal Learning github.com/google-research/google-research/tree/master/screen2words

ROSITA: Enhancing Vision-and-Language Semantic Alignments via Cross- and Intra-modal Knowledge Integration github.com/MILVLG/rosita

Visual Distant Supervision for Scene Graph Generation ICCV 2021 CV的场景图构建与NLP中的关系抽取任务很相似,能把RE常用的Distant Supervision思想迁移到CV场景图构建上 github.com/thunlp/VisualDS

Vision-Attention-Papers:视觉注意力机制总结

github.com/MenghaoGuo/Awesome-Vision-Attentions

理解einsum:从头实现多头自注意力Transformer https://theaisummer.com/einsum-attention/

Searching for Efficient Multi-Stage Vision Transformers github.com/yilunliao/vit-search

Learning to Prompt for Vision-Language Models github.com/KaiyangZhou/CoOp

Exploring and Improving Mobile Level Vision Transformers https://arxiv.org/abs/2108.13015

SentenceTransformers:最先进的句子、文本和图像嵌入Python框架 https://www.sbert.net/

ConvMLP: Hierarchical Convolutional MLPs for Vision github.com/SHI-Labs/Convolutional-MLPs

Data Efficient Masked Language Modeling for Vision and Language github.com/yonatanbitton/data_efficient_masked_language_modeling_for_vision_and_language

EfficientCLIP: Efficient Cross-Modal Pre-training by Ensemble Confident Learning and Language Modeling https://arxiv.org/abs/2109.04699

Vision-and-Language or Vision-for-Language? On Cross-Modal Influence in Multimodal Transformers github.com/e-bug/cross-modal-ablation

VQGAN-CLIP-Docker:Docker化部署的零样本文本-图像生成VQGAN+CLIP github.com/kcosta42/VQGAN-CLIP-Docker

xGQA: Cross-Lingual Visual Question Answering github.com/Adapter-Hub/xGQA

Optimum:大规模Transformer优化工具包 github.com/huggingface/optimum

Audio-Visual Speech Recognition is Worth 32×32×8 Voxels https://arxiv.org/abs/2109.09536

Vision-Transformer-papers:Vision Transformer (ViT)相关工作文献集

github.com/NielsRogge/Vision-Transformer-papers

Contrastive Language-Image Forensic Search:基于CLIP的对比语言-图像视频语义检索 github.com/johanmodin/clifs

CogView: Mastering Text-to-Image Generation via Transformers., http://keg.cs.tsinghua.edu.cn/jietang/ https://agc.platform.baai.ac.cn/CogView/index.html

《MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer》 https://arxiv.org/abs/2110.02178

CLIP prefix captioning.:基于CLIP的简单图像描述模型 #TODO

github.com/rmokady/CLIP_prefix_caption

「多模态内容理解」算法框架:Lichee。 框架采用分层的思想组织模型训练流程,包含了数据处理、预训练模型、常见模型以及模型加速等模块。 github.com/Tencent/Lichee ​​​​

微软图灵团队发布了具有25亿参数且可以执行94种语言的图像-语言任务的通用图像语言表示模型:Turing Bletchley。其拥有图像编码器和通用语言编码器,可分别对输入图像和文本进行矢量化,进而使得在语义上相似的图像和文本能够相互对齐。该模型展现出了独特的强大功能,并且在图像语言理解上取得了突破性进步。其表现已经优于多个SOTA模型,如ALIGN、MULE、SMALR等

https://www.microsoft.com/en-us/research/blog/turing-bletchley-a-universal-image-language-representation-model-by-microsoft/

Machine-in-the-Loop Rewriting for Creative Image Captioning https://arxiv.org/abs/2111.04193

A Survey of Visual Transformers https://arxiv.org/abs/2111.06091

Transformer应用汇总:从NLP到CV ithub.com/IbrahimSobh/Transformers

《Attention Mechanisms in Computer Vision: A Survey》 github.com/MenghaoGuo/Awesome-Vision-Attentions

ClipCap: CLIP Prefix for Image Captioning github.com/rmokady/CLIP_prefix_caption

TransMix: Attend to Mix for Vision Transformers github.com/Beckschen/TransMix

Transparent Human Evaluation for Image Captioning https://arxiv.org/abs/2111.08940

RedCaps: web-curated image-text data created by the people, for the people https://www.arxiv-vanity.com/papers/2111.11431/

Semi-Supervised Vision Transformers #IDEA 半监督 补全 直觉 知识 https://arxiv.org/abs/2111.11067

VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modeling github.com/tsujuifu/pytorch_violet

Scaling Up Vision-Language Pre-training for Image Captioning https://arxiv.org/abs/2111.12233

微软与北京大学联手,在 GitHub 开源了一个多模态预训练模型:「NÜWA(女娲)」,可实现文本/草图转图像、图像补全、文字/草图转视频等任务,功能异常强大。 GitHub:github.com/microsoft/NUWA

Efficient Video Transformers with Spatial-Temporal Token Selection https://arxiv.org/abs/2111.11591

Adaptive Fourier Neural Operators: Efficient Token Mixers for Transformers https://arxiv.org/abs/2111.13587

MURAL:跨语言多模态多任务检索 #NSFC https://ai.googleblog.com/2021/11/mural-multimodal-multi-task-retrieval.html

Vector Quantized Diffusion Model for Text-to-Image Synthesis github.com/microsoft/VQ-Diffusion

Improved Multiscale Vision Transformers for Classification and Detection https://arxiv.org/abs/2112.01526

Learning to Detect Every Thing in an Open World https://arxiv.org/abs/2112.01698

通过Tokenize学习提高视觉Transformer的效率和精度 https://arxiv.org/pdf/2106.11297.pdf https://ai.googleblog.com/2021/12/improving-vision-transformer-efficiency.html

CMA-CLIP: Cross-Modality Attention CLIP for Image-Text Classification https://arxiv.org/abs/2112.03562

Grounded Language-Image Pre-training https://arxiv.org/abs/2112.03857

Prompting Visual-Language Models for Efficient Video Understanding https://arxiv.org/abs/2112.04478

MLP Architectures for Vision-and-Language Modeling: An Empirical Study https://arxiv.org/abs/2112.04453 https://github.com/easonnie/mlp-vil

CLIP-Lite: Information Efficient Visual Representation Learning from Textual Annotations https://arxiv.org/abs/2112.07133

Everything at Once -- Multi-modal Fusion Transformer for Video Retrieval https://arxiv.org/abs/2112.04446

AdaViT: Adaptive Tokens for Efficient Vision Transformer https://arxiv.org/abs/2112.07658

Deep ViT Features as Dense Visual Descriptors https://arxiv.org/abs/2112.05814

SwinBERT: End-to-End Transformers with Sparse Attention for Video Captioning https://arxiv.org/abs/2111.13196

Keras实例教程:视觉Transformer的Token化学习 https://keras.io/examples/vision/token_learner/

RegionCLIP: Region-based Language-Image Pretraining https://arxiv.org/abs/2112.09106

CLIPfa: Connecting Farsi Text and Images github.com/sajjjadayobi/CLIPfa

PyTorch实现的CLIP引导的目标检测边框排序 github.com/bes-dev/pytorch_clip_bbox

DenseCLIP: Extract Free Dense Labels from CLIP https://arxiv.org/abs/2112.01071

面向计算机视觉的Transformer:架构、技巧与提升 https://theaisummer.com/transformers-computer-vision/?hss_channel=tw-1259466268505243649

Natural Language Descriptions of Deep Features https://openreview.net/forum?id=NudBMY-tzDr

多语言图像描述

github.com/gchhablani/multilingual-image-captioning

ERNIE-ViLG: Unified Generative Pre-training for Bidirectional Vision-Language Generation https://arxiv.org/abs/2112.15283

A Simple Baseline for Zero-shot Semantic Segmentation with Pre-trained Vision-language Model https://arxiv.org/abs/2112.14757

Language as Queries for Referring Video Object Segmentation https://arxiv.org/abs/2201.00487

Detecting Twenty-thousand Classes using Image-level Supervision https://arxiv.org/abs/2201.02605

Language-driven Semantic Segmentation https://arxiv.org/abs/2201.03546

Transformer4Vision:面向视觉任务的Transformer相关工作列表 github.com/quanghuy0497/Transformer4Vision

A Thousand Words Are Worth More Than a Picture: Natural Language-Centric Outside-Knowledge Visual Question Answering https://arxiv.org/abs/2201.05299

CLIP-Event: Connecting Text and Images with Event Structures https://arxiv.org/abs/2201.05078

MeMViT: Memory-Augmented Multiscale Vision Transformer for Efficient Long-Term Video Recognition https://arxiv.org/abs/2201.08383

Awesome CLIP:CLIP相关研究大列表】’Awesome CLIP - Awesome list for research on CLIP (Contrastive Language-Image Pre-Training). github.com/yzhuoning/Awesome-CLIP

CM3: A Causal Masked Multimodal Model of the Internet https://arxiv.org/abs/2201.07520

Kears实例教程:视频视觉Transformer(ViViT) https://keras.io/examples/vision/vivit/

CLIP-TD: CLIP Targeted Distillation for Vision-Language Tasks https://arxiv.org/abs/2201.05729

Natural Language Descriptions of Deep Visual Features https://arxiv.org/abs/2201.11114

Convolutional Xformers for Vision https://arxiv.org/abs/2201.10271

Keras实例教程:基于TabTransformer的结构化数据学习 https://keras.io/examples/structured_data/tabtransformer/

VC-GPT: Visual Conditioned GPT for End-to-End Generative Vision-and-Language Pre-training #Done https://arxiv.org/abs/2201.12723

BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation #Done https://arxiv.org/abs/2201.12086 https://github.com/salesforce/BLIP

Vision-Language Pre-Training with Triple Contrastive Learning https://arxiv.org/abs/2202.10401

A Survey of Vision-Language Pre-Trained Models #Done https://arxiv.org/abs/2202.10936

4D-Net: 3D和图像输入多模态对齐学习 https://ai.googleblog.com/2022/02/4d-net-learning-multi-modal-alignment.html

CLIP-GEN: Language-Free Training of a Text-to-Image Generator with CLIP #Done https://arxiv.org/abs/2203.00386

微软亚洲研究院多模态模型NÜWA:以自然语言创造视觉内容!此前我们曾提出了一个问题:从文字脚本生成创意视频一共分几步?微软亚洲研究院在视频生成预训练模型的基础上进行再创新,开发了多模态的 NÜWA(Neural visUal World creAtion)模型。通过自然语言指令,NÜWA 可以实现文本、图像、视频之间的生成、转换和编辑,帮助视觉内容创作者降低技术门槛,提高创造力。同时,开发者也可以利用 NÜWA 构建基于 AI 的视觉内容创造平台。 https://weibo.com/ttarticle/p/show?id=2309404743014097486309

解释Transformers里面所谓的Self-Attention https://openreview.net/pdf?id=MmujBClawFo 主旨是Self-Attention其实就是在计算sequence的self-expression。其实想法非常简单。Self-expressive是我们最早在2008年做motion sequences聚类就提出来过的 有趣的是那篇文章也正是研究了sequence有遮挡时如何complete的问题(也就是用填空方式self-supervise训练Transformers的思想)。而且正因为聚类的本质在于压缩,所以这篇论文也正好用到了lossy coding编码,比较了编码量与刻画稀疏的1-范数以及刻画相关性的矩阵rank(秩)的关系。我们现在知道优化编码量可以把深度网络以白盒子的形式推导出来 -- 而每一层的算子就是数据在做auto-regression。所以2008年的那篇小论文应该是同时联系到稀疏、低秩、压缩编码、非监督聚类(学习)、深度网络、以及Self-Attention的文章(想想都很神奇)。近年很多深度学习涉及到的思想概念其实极其简单,而且想法早已被发现和用过。只是通过大量的算力和数据放大出来,再换上吸引眼球的新名字,就变成了神秘的“人工智能”。

计算机视觉中的自注意力机制】,谷歌伯克利166页ppt教程 本教程将介绍自注意力机制在计算机视觉中的应用。Self-Attention在NLP中被广泛采用,完全注意的Transformer模型已经在很大程度上取代了RNN,现在被用于最先进的语言理解模型,如GPT、BERT、XLNet、T5、Electra和Meena。因此,人们对研究自注意力是否能在计算机视觉中产生同样巨大而深远的影响产生了极大的兴趣。然而,由于视觉任务与语言任务具有不同的性质,因此许多研究都致力于探索自注意力在视觉模型中的最佳应用方式。本教程将涵盖视觉中自注意力的许多不同应用,以便让用户对这个子领域有一个广泛而精确的理解。 https://icml.cc/media/icml-2021/Slides/10842.pdf

LoopITR: Combining Dual and Cross Encoder Architectures for Image-Text Retrieval https://arxiv.org/abs/2203.05465

Conditional Prompt Learning for Vision-Language Models https://www.arxiv-vanity.com/papers/2203.05557/

StyleBabel: Artistic Style Tagging and Captioning https://arxiv.org/abs/2203.05321

CLIP Models are Few-shot Learners: Empirical Studies on VQA and Visual Entailment https://arxiv.org/abs/2203.07190

Awesome Prompting Papers in Computer Vision:计算机视觉/视觉-语言学模型基于提示方法论文列表 github.com/ttengwang/Awesome_Prompting_Papers_in_Computer_Vision

Integrating Language Guidance into Vision-based Deep Metric Learning https://arxiv.org/abs/2203.08543

Three things everyone should know about Vision Transformers https://arxiv.org/abs/2203.09795

MetaFormer: A Unified Meta Framework for Fine-Grained Recognition https://arxiv.org/abs/2203.02751

Attention Bottlenecks for Multimodal Fusion https://arxiv.org/abs/2107.00135

WuDaoMM: A large-scale Multi-Modal Dataset for Pre-training models https://arxiv.org/abs/2203.11480 WuDaoMM - 基于中文图文对的多模态数据,全量数据集约有6.5亿图文对,包含强相关数据5千万对和弱相关数据6亿对 github.com/BAAI-WuDao/WuDaoMM

CLIP-as-service:用CLIP将图像和句子嵌入到固定长度向量

github.com/jina-ai/clip-as-service

Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors https://arxiv.org/abs/2203.13131

TorchMultimodal:用于大规模训练最先进多模态多任务模型的PyTorch库 github.com/facebookresearch/multimodal

Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directions https://arxiv.org/abs/2203.12667

视觉-语言模型:多模态深度学习探索 https://theaisummer.com/vision-language-models/

imgbeddings:无需PyTorch/TensorFlow用CLIP生成图像嵌入向量的Python包 github.com/minimaxir/imgbeddings

ClipCap-Chinese:基于ClipCap的看图说话Image Caption模型

github.com/yangjianxin1/ClipCap-Chinese https://arxiv.org/pdf/2111.09734.pdf

CLIP on Wheels: Zero-Shot Object Navigation as Object Localization and Exploration https://arxiv.org/abs/2203.10421

solo-learn:Pytorch Lightning视觉表示自监督学习库 github.com/vturrisi/solo-learn

OpenCLIP:CLIP的开源实现(PyTorch)】’OpenCLIP - An open source implementation of CLIP. github.com/mlfoundations/open_clip

DALL·E 2:DALL·E的新发布版本,可根据自然语言描述创建逼真的图像和艺术 https://openai.com/dall-e-2/

Hierarchical Text-Conditional Image Generation with CLIP Latents https://cdn.openai.com/papers/dall-e-2.pdf

multisearch:不只是搜索框,视觉信息+文字查询的跨模态搜索新范式 https://blog.google/products/search/multisearch/

Unleashing Vanilla Vision Transformer with Masked Image Modeling for Object Detection https://arxiv.org/abs/2204.02964

Unified Contrastive Learning in Image-Text-Label Space https://arxiv.org/abs/2204.03610

Image Retrieval from Contextual Descriptions https://arxiv.org/abs/2203.15867

FALCON: Fast Visual Concept Learning by Integrating Images, Linguistic descriptions, and Conceptual Relations https://arxiv.org/abs/2203.16639

MMF:面向视觉-语言多模态研究的模块化框架

github.com/facebookresearch/mmf

CLIP Benchmark:CLIP类模型评价基准 github.com/LAION-AI/CLIP_benchmark

pyglide:基于GLIDE根据文本生成图片的命令行工具 github.com/afiaka87/pyglide

用单个视觉语言模型搞定多个任务 https://www.deepmind.com/blog/tackling-multiple-tasks-with-a-single-visual-language-model

DALL·E 2工作原理通俗解析 http://adityaramesh.com/posts/dalle2/dalle2.html

dalle-playground:基于DALL-E Mini的文本生成图片试练场 github.com/saharmor/dalle-playground

Answer-Me: Multi-Task Open-Vocabulary Visual Question Answering https://arxiv.org/abs/2205.00949

CLAP - Contrastive Language-Audio Pretraining github.com/LAION-AI/CLAP

PyramidCLIP: Hierarchical Feature Alignment for Vision-language Model Pretraining https://arxiv.org/abs/2204.14095

CoCa: Contrastive Captioners are Image-Text Foundation Models https://arxiv.org/abs/2205.01917

Language Models Can See: Plugging Visual Controls in Text Generation https://arxiv.org/abs/2205.02655

All You May Need for VQA are Image Captions https://arxiv.org/abs/2205.01883

Beyond a Pre-Trained Object Detector: Cross-Modal Textual and Visual Context for Image Captioning https://arxiv.org/abs/2205.04363

Data Determines Distributional Robustness in Contrastive Language Image Pre-training (CLIP) https://arxiv.org/abs/2205.01397

OpenAI and the road to text-guided image generation: DALL·E, CLIP, GLIDE, DALL·E 2 (unCLIP) https://blog.inten.to/openai-and-the-road-to-text-guided-image-generation-dall-e-clip-glide-dall-e-2-unclip-c6e28f7194ea

A Generalist Agent 全能型智能体,用单个通用智能体Gato就可以玩雅达利游戏、对图像进行描述、聊天、用真正的机器手臂堆积木等等。 https://arxiv.org/abs/2205.06175

Learning Visual Styles from Audio-Visual Associations https://arxiv.org/abs/2205.05072

A CLIP-Hitchhiker's Guide to Long Video Retrieval https://arxiv.org/abs/2205.08508

Training Vision-Language Transformers from Captions Alone https://arxiv.org/abs/2205.09256

Language Models with Image Descriptors are Strong Few-Shot Video-Language Learners https://arxiv.org/abs/2205.10747

Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding https://arxiv.org/abs/2205.11487

X-ViT: High Performance Linear Vision Transformer without Softmax https://arxiv.org/abs/2205.13805

GIT: A Generative Image-to-text Transformer for Vision and Language https://arxiv.org/abs/2205.14100

Fine-grained Image Captioning with CLIP Reward https://arxiv.org/abs/2205.13115

VL-BEiT: Generative Vision-Language Pretraining https://arxiv.org/abs/2206.01127

Multimodal Masked Autoencoders Learn Transferable Representations https://arxiv.org/abs/2205.14204

A-OKVQA: A Benchmark for Visual Question Answering using World Knowledge https://arxiv.org/abs/2206.01718

【book_writer:AI写作助手】’book_writer' by Sung Kim GitHub: github.com/hunkim/book_writer

Revisiting the "Video" in Video-Language Understanding https://arxiv.org/abs/2206.01720

Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Experts https://arxiv.org/abs/2206.02770

【PaddleMM:多模态学习工具包,以百度 PaddlePaddle 平台为主,兼容 PyTorch 提供 torch 版本,旨在于提供模态联合学习和跨模态学习算法模型库,为处理图片文本等多模态数据提供高效的解决方案,助力多模态学习应用落地】’PaddleMM - Multi-Modal learning toolkit based on PaddlePaddle and PyTorch, supporting multiple applications such as multi-modal classification, cross-modal retrieval and image caption.' by njustkmg GitHub: github.com/njustkmg/PaddleMM

【多模态学习相关资源大列表】’Awesome Multimodality - A Survey on multimodal learning research.' by Sanctuary GitHub: github.com/Yutong-Zhou-cv/Awesome-Multimodality

[CV]《GLIPv2: Unifying Localization and Vision-Language Understanding》H Zhang, P Zhang, X Hu, Y Chen, L H Li, X Dai, L Wang, L Yuan, J Hwang, J Gao [University of Washington & Microsoft & UCLA] (2022) https://arxiv.org/abs/2206.05836

[CL]《Language Models are General-Purpose Interfaces》Y Hao, H Song, L Dong, S Huang, Z Chi, W Wang, S Ma, F Wei [Microsoft Research] (2022) https://arxiv.org/abs/2206.06336

[CV]《Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Tasks》J Lu, C Clark, R Zellers, R Mottaghi, A Kembhavi [Allen Institute for AI] (2022) https://arxiv.org/abs/2206.08916

[CV]《Prefix Language Models are Unified Modal Learners》S Diao, W Zhou, X Zhang, J Wang [The Hong Kong University of Science and Technology & ByteDance AI Lab & Shanghai Jiao Tong University] (2022) https://arxiv.org/abs/2206.07699

[CV]《Bridge-Tower: Building Bridges Between Encoders in Vision-Language Representation Learning》X Xu, C Wu, S Rosenman, V Lal, N Duan [Microsoft Research Asia & Intel Labs] (2022) https://arxiv.org/abs/2206.08657

[CV]《Global Context Vision Transformers》A Hatamizadeh, H Yin, J Kautz, P Molchanov [NVIDIA] (2022) https://arxiv.org/abs/2206.09959

[CV]《Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbone》Z Dou, A Kamath, Z Gan, P Zhang, J Wang, L Li, Z Liu, C Liu, Y LeCun, N Peng, J Gao, L Wang [Microsoft & University of California, Los Angeles & New York University] (2022) https://arxiv.org/abs/2206.07643

【视觉-语言预训练(VLP)最新进展】'Recent Advances in Vision-and-Language Pre-training (VLP) - Recent Advances in Vision and Language Pre-training (VLP)' by Phellon Chen GitHub: github.com/phellonchen/awesome-Vision-and-Language-Pre-training

【Enso:具有视觉和文本双重表示的交互式编程语言】’Enso - Hybrid visual and textual functional programming.' GitHub: github.com/enso-org/enso

[CV]《Towards Counterfactual Image Manipulation via CLIP》Y Yu, F Zhan, R Wu, J Zhang, S Lu, M Cui, X Xie, X Hua, C Miao [Nanyang Technological University & Max Planck Institute for Informatics & Alibaba Group] (2022) https://arxiv.org/abs/2207.02812

【Chinese-CLIP:CLIP模型的中文版,用大规模中文数据进行训练(~2亿图文对),旨在帮助用户实现中文领域的跨模态检索、图像表示等】'Chinese-CLIP - Chinese version of CLIP which achieves Chinese cross-modal retrieval and representation generation.' by OFA Sys GitHub: github.com/OFA-Sys/Chinese-CLIP

✅[CV]《Bootstrapped Masked Autoencoders for Vision BERT Pretraining》X Dong, J Bao, T Zhang, D Chen, W Zhang, L Yuan, D Chen, F Wen, N Yu [University of Science and Technology of China & Microsoft Research Asia & Microsoft Cloud + AI] (2022) https://arxiv.org/abs/2207.07116

✅[CV]《Multimodal Open-Vocabulary Video Classification via Pre-Trained Vision and Language Models》R Qian, Y Li, Z Xu, M Yang, S Belongie, Y Cui [Google Research & University of Copenhagen] (2022) https://arxiv.org/abs/2207.07646

✅[CV]《TokenMix: Rethinking Image Mixing for Data Augmentation in Vision Transformers》J Liu, B Liu, H Zhou, H Li, Y Liu [CUHK & SenseTime Research & CAS] (2022) https://arxiv.org/abs/2207.08409

✅[CV]《LAVA: Language Audio Vision Alignment for Contrastive Video Pre-Training》S Gurram, A Fang, D Chan, J Canny [UC Berkeley] (2022) https://arxiv.org/abs/2207.08024

[CV]《Semantic Abstraction: Open-World 3D Scene Understanding from 2D Vision-Language Models》H Ha, S Song [Columbia University] (2022) https://arxiv.org/abs/2207.11514

[CV]《Learning Visual Representation from Modality-Shared Contrastive Language-Image Pre-training》H You, L Zhou, B Xiao, N Codella, Y Cheng, R Xu, S Chang, L Yuan [Columbia University & Microsoft Cloud and AI & Microsoft Research] (2022) https://arxiv.org/abs/2207.12661

[CL]《RealTime QA: What's the Answer Right Now?》J Kasai, K Sakaguchi, Y Takahashi, R L Bras, A Asai, X Yu, D Radev, N A. Smith, Y Choi, K Inui [University of Washington & Allen Institute for AI & Tohoku University & Yale University] (2022) https://arxiv.org/abs/2207.13332

[CV]《WinoGAViL: Gamified Association Benchmark to Challenge Vision-and-Language Models》Y Bitton, N B Guetta, R Yosef, Y Elovici, M Bansal, G Stanovsky, R Schwartz [The Hebrew University of Jerusalem & Ben Gurion University & University of North Carolina at Chapel Hill] (2022) https://arxiv.org/abs/2207.12576

✅[CV]《Retrieval-Augmented Transformer for Image Captioning》S Sarto, M Cornia, L Baraldi, R Cucchiara [University of Modena and Reggio Emilia] (2022) https://arxiv.org/abs/2207.13162

【Multi-modal-Deep-Learning:多模态深度学习进展追踪】’Multi-modal-Deep-Learning' by Jingfeng Yang GitHub: github.com/JingfengYang/Multi-modal-Deep-Learning

[CV]《Curriculum Learning for Data-Efficient Vision-Language Alignment》T Srinivasan, X Ren, J Thomason [University of Southern California] (2022) https://arxiv.org/abs/2207.14525

[CV]《Video Question Answering with Iterative Video-Text Co-Tokenization》A Piergiovanni, K Morton, W Kuo, M S. Ryoo, A Angelova [Google Research & MIT] (2022) https://arxiv.org/abs/2208.00934

[CV]《An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion》R Gal, Y Alaluf, Y Atzmon, O Patashnik, A H. Bermano, G Chechik, D Cohen-Or [Tel-Aviv University & NVIDIA] (2022) https://arxiv.org/abs/2208.01618

✅[CV]《Masked Vision and Language Modeling for Multi-modal Representation Learning》G Kwon, Z Cai, A Ravichandran, E Bas, R Bhotika, S Soatto [AWS AI Labs] (2022) https://arxiv.org/abs/2208.02131

[CV]《Expanding Language-Image Pretrained Models for General Video Recognition》B Ni, H Peng, M Chen, S Zhang, G Meng, J Fu, S Xiang, H Ling [Microsoft Research & Chinese Academy of Sciences & Stony Brook University & University of Rochester] (2022) https://arxiv.org/abs/2208.02816

[CL]《Prompt Tuning for Generative Multimodal Pretrained Models》H Yang, J Lin, A Yang, P Wang, C Zhou, H Yang [DAMO Academy] (2022) https://arxiv.org/abs/2208.02532

✅[CV]《Masked Feature Prediction for Self-Supervised Visual Pre-Training》C Wei, H Fan, S Xie, C Wu, A Yuille, C Feichtenhofer [Facebook AI Research & Johns Hopkins University] (2022) https://arxiv.org/abs/2112.09133

大家主要用的vision-language pretraining的model是啥? vinvl

[CV]《Text-to-Image Generation via Implicit Visual Guidance and Hypernetwork》X Yuan, Z Lin, J Kuen, J Zhang, J Collomosse [University of Chicago & Adobe Research] (2022) https://arxiv.org/abs/2208.08493

【Chinese-CLIP:CLIP模型的中文版本,使用大规模中文数据进行训练(~2亿图文对),旨在帮助用户实现中文领域的跨模态检索、图像表示等】'Chinese-CLIP - Chinese version of CLIP which achieves Chinese cross-modal retrieval and representation generation.' by billjie1 GitHub: github.com/billjie1/Chinese-CLIP

[CV]《Discovering Bugs in Vision Models using Off-the-shelf Image Generation and Captioning》O Wiles, I Albuquerque, S Gowal [DeepMind] (2022) https://arxiv.org/abs/2208.08831

【MinImagen:Imagen文本图像生成模型的最小化实现】’MinImagen: A minimal implementation of the Imagen text-to-image model' by AssemblyAI GitHub: github.com/AssemblyAI-Examples/MinImagen

[CV]《Understanding Attention for Vision-and-Language Tasks》F Cao, S C Han, S Long, C Xu, J Poon [The University of Sydney] (2022) https://arxiv.org/abs/2208.08104

[CV]《Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks》W Wang, H Bao, L Dong, J Bjorck, Z Peng, Q Liu, K Aggarwal, O K Mohammed, S Singhal, S Som, F Wei [Microsoft Corporation] (2022) https://arxiv.org/abs/2208.10442

[CV]《FashionVQA: A Domain-Specific Visual Question Answering System》M Wang, A Mahjoubfar, A Joshi [Target Corporation] (2022) https://arxiv.org/abs/2208.11253

[CV]《MaskCLIP: Masked Self-Distillation Advances Contrastive Language-Image Pretraining》X Dong, Y Zheng, J Bao, T Zhang, D Chen, H Yang, M Zeng, W Zhang, L Yuan, D Chen, F Wen, N Yu [University of Science and Technology of China & Xiamen University & Microsoft Research Asia & Microsoft Cloud + AI] (2022) https://arxiv.org/abs/2208.12262

[CV]《Opal: Multimodal Image Generation for News Illustration》V Liu, H Qiao, L Chilton [Columbia University & University of Toronto] (2022) https://arxiv.org/abs/2204.09007

[CV]《DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation》N Ruiz, Y Li, V Jampani, Y Pritch, M Rubinstein, K Aberman [Google Research] (2022) https://arxiv.org/abs/2208.12242

微软亚洲研究院联合微软图灵团队推出了最新升级的 BEiT-3 预训练模型[毕业帽],在广泛的视觉及视觉-语言任务上,包括目标检测(COCO)、实例分割(COCO)、语义分割(ADE20K)、图像分类(ImageNet)、视觉推理(NLVR2)、视觉问答(VQAv2)、图片描述生成(COCO)和跨模态检索(Flickr30K,COCO)等,实现了 SOTA 的迁移性能。BEiT-3 创新的设计和出色的表现为多模态研究打开了新思路,也预示着 AI 大一统渐露曙光。 https://weibo.com/ttarticle/p/show?id=2309404808248644468949

【Stable Diffusion模型相关资源大列表】’Awesome Stable-Diffusion - Curated list of awesome resources for the Stable Diffusion AI Model.' GitHub: github.com/awesome-stable-diffusion/awesome-stable-diffusion

CogVideo,文本到视频生成的预训练模型,94亿参数。CogVideo将预训练文本到图像生成模型(CogView2)有效地利用到文本到视频生成模型,并使用了多帧率分层训练策略。 https://models.aminer.cn/cogvideo/

'Experiments with Stable Diffusion' by Justin GitHub: github.com/justinpinkney/stable-diffusion

'Peacasso - UI interface for experimenting with multimodal (text, image) models (stable diffusion).' by Victor Dibia GitHub: github.com/victordibia/peacasso

[CV]《What does a platypus look like? Generating customized prompts for zero-shot image classification》S Pratt, R Liu, A Farhadi [University of Washington & Google Research] (2022) https://arxiv.org/abs/2209.03320 https://github.com/sarahpratt/CuPL

【Stable Diffusion长文本图像生成】’long_stable_diffusion - Long Stable Diffusion: Long-form text to images’ by Sharon Zhou GitHub: github.com/sharonzhou/long_stable_diffusion

[CV]《An Empirical Study of End-to-End Video-Language Transformers with Masked Visual Modeling》T Fu, L Li, Z Gan, K Lin, W Y Wang, L Wang, Z Liu [UC Santa Barbara & Microsoft] (2022) https://arxiv.org/abs/2209.01540

[CV]《Pre-training image-language transformers for open-vocabulary tasks》A Piergiovanni, W Kuo, A Angelova [Google Research] (2022) https://arxiv.org/abs/2209.04372

【视觉-语言预训练相关资源大列表】’Awesome Vision-and-Language Pre-Training' by Zhihong Chen GitHub: github.com/zhjohnchan/awesome-vision-and-language-pretraining

【Deforum Stable Diffusion Local Version:Deforum Stable Diffusion本地版,可生成动画】’Deforum Stable Diffusion Local Version - Local version of Deforum Stable Diffusion, supports txt settings file input and animation features!' by Eddu Hu GitHub: github.com/HelixNGC7293/DeforumStableDiffusionLocal

[LG]《Diffusion Models: A Comprehensive Survey of Methods and Applications》L Yang, Z Zhang, S Hong, R Xu, Y Zhao, Y Shao, W Zhang, M Yang, B Cui [Peking University & University of California, Los Angeles & CMU & BUPT & Mila & University of California at Merced] (2022) https://arxiv.org/abs/2209.00796v5

[CV]《CLIP-ViP: Adapting Pre-trained Image-Text Model to Video-Language Representation Alignment》H Xue, Y Sun, B Liu, J Fu, R Song, H Li, J Luo [University of Science and Technology of China & Renmin University of China & Microsoft Research Asia & University of Rochester] (2022) https://arxiv.org/abs/2209.06430 https://github.com/microsoft/XPretrain/tree/main/CLIP-ViP

[CV]《PaLI: A Jointly-Scaled Multilingual Language-Image Model》X Chen, X Wang, S Changpinyo, A Piergiovanni... [Google Research] (2022) https://arxiv.org/abs/2209.06794

[CL]《Towards Multi-Lingual Visual Question Answering》S Changpinyo, L Xue, I Szpektor, A V. Thapliyal, J Amelot, X Chen, R Soricut [Google Research] (2022) https://arxiv.org/abs/2209.05401

'txt2img2img for Stable Diffusion - Improve the editability of any Stability Diffusion subject while retaining a high degree of likeness' by ThereforeGames GitHub: github.com/ThereforeGames/txt2img2img

Cogview第2个版本CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformers https://www.aminer.cn/pub/626f3dd05aee126c0f8f76a7/cogview-faster-and-better-text-to-image-generation-via-hierarchical-transformers https://models.aminer.cn/CogView/index.html

[CV]《OmniVL:One Foundation Model for Image-Language and Video-Language Tasks》J Wang, D Chen, Z Wu, C Luo, L Zhou, Y Zhao, Y Xie, C Liu, Y Jiang, L Yuan [Fudan University & Microsoft Cloud + AI & Microsoft Research Asia] (2022) https://arxiv.org/abs/2209.07526

[CV]《X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrieval》Y Ma, G Xu, X Sun, M Yan, J Zhang, R Ji [Xiamen University & DAMO Academy, Alibaba Group] (2022) https://arxiv.org/abs/2207.07285

[CV]《Diffusion Models in Vision: A Survey》F Croitoru, V Hondru, R T Ionescu, M Shah [University of Bucharest & University of Central Florida] (2022) https://arxiv.org/abs/2209.04747

[CV]《StoryDALL-E: Adapting Pretrained Text-to-Image Transformers for Story Continuation》A Maharana, D Hannan, M Bansal [UNC Chapel Hill] (2022) https://arxiv.org/abs/2209.06192

[CV]《LAVIS: A Library for Language-Vision Intelligence》D Li, J Li, H Le, G Wang, S Savarese, S C.H. Hoi [Salesforce Research] (2022) https://arxiv.org/abs/2209.09019

[CV]《I2DFormer: Learning Image to Document Attention for Zero-Shot Image Classification》M F Naeem, Y Xian, L V Gool, F Tombari [ETH Zurich & TUM] (2022) https://arxiv.org/abs/2209.10304

[CV]《Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models》M Shu, W Nie, D Huang, Z Yu, T Goldstein, A Anandkumar, C Xiao [University of Maryland & NVIDIA] (2022) https://arxiv.org/abs/2209.07511

[CV]《UniColor: A Unified Framework for Multi-Modal Colorization with Transformer》Z Huang, N Zhao, J Liao [City University of Hong Kong & University of Bath] (2022) https://arxiv.org/abs/2209.11223

[CV]《Semantic scene descriptions as an objective of human vision》A Doerig, T C Kietzmann, E Allen, Y Wu, T Naselaris, K Kay, I Charest [Donders Institute for Brain & University of Minnesota] (2022) https://arxiv.org/abs/2209.11737

[CV]《UniCLIP: Unified Framework for Contrastive Language-Image Pre-training》J Lee, J Kim, H Shon, B Kim, S H Kim, H Lee, J Kim [LG AI Research & KAIST] (2022) https://arxiv.org/abs/2209.13430

[CV]《Towards Multimodal Multitask Scene Understanding Models for Indoor Mobile Agents》Y H Tsai, H Goh, A Farhadi, J Zhang [Apple] (2022) https://arxiv.org/abs/2209.13156

[CV]《TVLT: Textless Vision-Language Transformer》Z Tang, J Cho, Y Nie, M Bansal [UNC Chapel Hill] (2022) https://arxiv.org/abs/2209.14156

[CL]《SpeechCLIP: Integrating Speech with Pre-Trained Vision and Language Model》Y Shih, H Wang, H Chang, L Berry, H Lee, D Harwath [National Taiwan University & The University of Texas at Austin] (2022) https://arxiv.org/abs/2210.00705

[CV]《Paraphrasing Is All You Need for Novel Object Captioning》C Yang, Y H Tsai, W Fan, R Salakhutdinov, L Morency, Y F Wang [UCLA & CMU & National Taiwan University] (2022) https://arxiv.org/abs/2209.12343

【(CVPR2022 Tutorial)视觉-语言预训练最新进展】《CVPR2022 Tutorial: Recent Advanced in Vision-and-Language Pre-training》 vlp-tutorial.github.io/2022/

[CL]《Linearly Mapping from Image to Text Space》J Merullo, L Castricato, C Eickhoff, E Pavlick [Brown University] (2022) https://arxiv.org/abs/2209.15162

[CV]《F-VLM: Open-Vocabulary Object Detection upon Frozen Vision and Language Models》W Kuo, Y Cui, X Gu, A Piergiovanni, A Angelova [Google Research] (2022) https://arxiv.org/abs/2209.15639

【TencentPretrain:用于对文本、图像、语音等模态数据进行预训练和微调的工具包】'TencentPretrain - Tencent Pre-training framework in PyTorch & Pre-trained Model Zoo' by Tencent GitHub: github.com/Tencent/TencentPretrain

[CV]《Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP》F Liang, B Wu, X Dai, K Li, Y Zhao, H Zhang, P Zhang, P Vajda, D Marculescu [The University of Texas at Austin & Meta Reality Labs & Cruise] (2022) https://arxiv.org/abs/2210.04150

【多模态强化学习相关文献资源大列表】’Awesome Multi-Modal Reinforcement Learning - A curated list of Multi-Modal Reinforcement Learning resources' by OpenDILab GitHub: github.com/opendilab/awesome-multi-modal-reinforcement-learning

[CV]《Vision-Language Pre-training: Basics, Recent Advances, and Future Trends》Z Gan, L Li, C Li, L Wang, Z Liu, J Gao [Microsoft Corporation] (2022) https://arxiv.org/abs/2210.09263

'clip-as-service - Embed/reason/rank images and sentences with CLIP models' by Jina AI GitHub: github.com/jina-ai/clip-as-service

不需要任何标注,在翻译、图像标题生成等生成类任务的自动评估上取得了SOTA https://arxiv.org/pdf/2210.05035.pdf

[CL]《Text-Only Training for Image Captioning using Noise-Injected CLIP》D Nukrai, R Mokady, A Globerson [Tel Aviv University] (2022) https://arxiv.org/abs/2211.00575

[CL]《BLOOM: A 176B-Parameter Open-Access Multilingual Language Model》T L Scao, A Fan, C Akiki, E Pavlick, S Ilić, D Hesslow, R Castagné... (2022) https://arxiv.org/abs/2211.05100

[CV]《CLOP: Video-and-Language Pre-Training with Knowledge Regularizations》G Li, H Yang, F He, Z Feng, Y Lyu, H Wu, H Wang [Baidu Inc] (2022) https://arxiv.org/abs/2211.03314

【VideoX - 多模态视频内容理解模型集】’VideoX - a collection of video cross-modal models' by Microsoft GitHub: github.com/microsoft/VideoX

[CV]《I Can't Believe There's No Images! Learning Visual Tasks Using only Language Data》S Gu, C Clark, A Kembhavi [Allen Institute for Artificial Intelligence] https://arxiv.org/abs/2211.09778

[CV]《Uni-Perceiver v2: A Generalist Model for Large-Scale Vision and Vision-Language Tasks》H Li, J Zhu, X Jiang, X Zhu, H Li, C Yuan, X Wang, Y Qiao, X Wang, W Wang, J Dai [The Chinese University of Hong Kong & Xi’an Jiaotong University & Tsinghua University & SenseTime Research] (2022) https://arxiv.org/abs/2211.09808

[CV]《Retrieval-Augmented Multimodal Language Modeling》M Yasunaga, A Aghajanyan, W Shi, R James, J Leskovec, P Liang, M Lewis, L Zettlemoyer, W Yih [Stanford University & Meta AI & University of Washington] (2022) https://arxiv.org/abs/2211.12561

[CV]《Human or Machine? Turing Tests for Vision and Language》M Zhang, G Dellaferrera, A Sikarwar, M Armendariz... [CFAR and I2R & IBM Research & Harvard Medical School...] (2022) https://arxiv.org/abs/2211.13087

[CV]《Exploiting Category Names for Few-Shot Classification with Vision-Language Models》T Xiao, Z Wang, L Cao, J Yu, S Dai, M Yang [Google & University of California, Merced & Apple] (2022) https://arxiv.org/abs/2211.16594

Vision-and-Language Navigation github.com/nuaa-nlp/paper-reading

'CLIP-Chinese:中文多模态对比学习CLIP预训练模型' by Yang JianXin GitHub: github.com/yangjianxin1/CLIP-Chinese

【阿里达摩院发布的中文版CLIP模型,旨在帮助用户快速实现中文领域的图文特征&相似度计算、跨模态检索、零样本图片分类等任务】’Chinese-CLIP - Chinese version of CLIP which achieves Chinese cross-modal retrieval and representation generation.' by OFA-Sys GitHub: github.com/OFA-Sys/Chinese-CLIP

[CV]《 Unifying (统一) Vision, Text, and Layout for Universal Document Processing》Z Tang, Z Yang, G Wang, Y Fang, Y Liu, C Zhu, M Zeng, C Zhang, M Bansal [Microsoft Azure & University of North Carolina at Chapel Hill] (2022) https://arxiv.org/abs/2212.02623

【OFASys:用于构建通用模型的多模态多任务学习系统】'OFASys: A Multi-Modal Multi-Task Learning System for Building Generalist Models' by OFA Sys GitHub: github.com/OFA-Sys/OFASys

【Big-Interleaved-Dataset:开源多模态数据集】’Big-Interleaved-Dataset - Big-Interleaved-Dataset' by LAION-AI GitHub: github.com/LAION-AI/Big-Interleaved-Dataset

[CV]《REVEAL: Retrieval-Augmented Visual-Language Pre-Training with Multi-Source Multimodal Knowledge Memory》Z Hu, A Iscen, C Sun, Z Wang, K Chang, Y Sun, C Schmid, D A. Ross, A Fathi [Google Research & University of California, Los Angeles] (2022) https://arxiv.org/abs/2212.05221

【Data2vec 2.0: 视觉、语音和文本高效自监督学习】《Data2vec 2.0: Highly efficient self-supervised learning for vision, speech and text》 GitHub: github.com/facebookresearch/fairseq/tree/main/examples/data2vec

[CV]《VindLU: A Recipe for Effective Video-and-Language Pretraining》F Cheng, X Wang, J Lei, D Crandall, M Bansal, G Bertasius [UNC Chapel Hill & Indiana University] (2022) https://arxiv.org/abs/2212.05051

[CV]《Generalized Decoding for Pixel, Image, and Language》X Zou, Z Dou, J Yang, Z Gan, L Li, C Li, X Dai, H Behl, J Wang, L Yuan, N Peng, L Wang, Y J Lee, J Gao [Microsoft & University of Wisconsin-Madison & UCLA] (2022) https://arxiv.org/abs/2212.11270

【Multimodal Machine Learning for Music (MML4Music):音乐多模态机器学习相关学术资源列表】’Multimodal Machine Learning for Music (MML4Music) - List of academic resources on Multimodal ML for Music' by Ilaria Manco GitHub: github.com/ilaria-manco/multimodal-ml-music

[CV]《LayoutDETR: Detection Transformer Is a Good Multimodal Layout Designer》N Yu, C Chen, Z Chen, R Meng, G Wu, P Josel, J C Niebles, C Xiong, R Xu [Salesforce Research] (2022) https://arxiv.org/abs/2212.09877

【以计算机视觉和多模态为重点的参数高效迁移学习论文列表】’Awesome-Parameter-Efficient-Transfer-Learning - A collection of parameter-efficient transfer learning papers focusing on computer vision and multimodal domains.' by Jiang Haojun GitHub: github.com/jianghaojun/Awesome-Parameter-Efficient-Transfer-Learning

[CV]《Filtering, Distillation, and Hard Negatives for Vision-Language Pre-Training》F Radenovic, A Dubey, A Kadian, T Mihaylov, S Vandenhende, Y Patel, Y Wen, V Ramanathan, D Mahajan [Meta AI] (2023) https://arxiv.org/abs/2301.02280

[CV]《HierVL: Learning Hierarchical Video-Language Embeddings》K Ashutosh, R Girdhar, L Torresani, K Grauman [Meta AI & UT Austin] (2023) https://arxiv.org/abs/2301.02311

[CV]《What You Say Is What You Show: Visual Narration Detection in Instructional Videos》K Ashutosh, R Girdhar, L Torresani, K Grauman [Meta AI & UT Austin] (2023) https://arxiv.org/abs/2301.02307

[CV]《All in Tokens: Unifying Output Space of Visual Tasks via Soft Token》J Ning, C Li, Z Zhang, Z Geng, Q Dai, K He, H Hu [Microsoft Research Asia & Huazhong University of Science and Technology & ...] (2023) https://arxiv.org/abs/2301.02229

迷你书(270+页)《多模态深度学习》 https://arxiv.org/abs/2301.04856

[CV]《Zorro: the masked multimodal transformer》A Recasens, J Lin, J Carreira, D Jaegle, L Wang, J Alayrac, P Luc, A Miech, L Smaira, R Hemsley, A Zisserman [DeepMind] (2023) https://arxiv.org/abs/2301.09595

【深入视觉-语言模型】《A Dive into Vision-Language Models》 https://huggingface.co/blog/vision_language_pretraining

【MMRec:多模态推荐工具箱】'MMRec - A Toolbox for MultiModal Recommendation. Integrating 10+ Models...' enoche GitHub: github.com/enoche/MMRec

[CV]《Re-ViLM: Retrieval-Augmented Visual Language Model for Zero and Few-Shot Image Captioning》Z Yang, W Ping, Z Liu, V Korthikanti, W Nie, D Huang, L Fan, Z Yu, S Lan, B Li, M Liu, Y Zhu, M Shoeybi, B Catanzaro, C Xiao, A Anandkumar [NVIDIA & UIUC] (2023) https://arxiv.org/abs/2302.04858

'TaiSu(太素)- a large-scale Chinese multimodal dataset(亿级大规模中文视觉语言预训练数据集)’ TaiSu-data GitHub: github.com/ksOAn6g5/TaiSu

【多模态预训练大模型相关文献资源列表】’MultiModal_BigModels_Survey - A continuously updated paper list for multi-modal pre-trained big models' Xiao Wang GitHub: github.com/wangxiao5791509/MultiModal_BigModels_Survey

一种改进视觉语言模型的新方法,用于提升其理解物体计数的能力,在下游任务中产生更可靠的结果。引入一种新的计数基准,CountBench,并展示了对 CLIP 和 BASIC 模型的重大改进。[CV]《Teaching CLIP to Count to Ten》R Paiss, A Ephrat, O Tov, S Zada, I Mosseri, M Irani, T Dekel [Google Research] (2023) https://arxiv.org/abs/2302.12066

[LG]《Aligning Text-to-Image Models using Human Feedback》K Lee, H Liu, M Ryu, O Watkins, Y Du, C Boutilier, P Abbeel, M Ghavamzadeh, S S Gu [Google Research & UC Berkeley] (2023) 利用人工反馈对文本到图像模型进行微调,以改善图像-文本对齐,可作为研究从人工反馈中学习以改进文本到图像模型的起点。https://arxiv.org/abs/2302.12192

【UForm:多模态推理库,用于将多种语言的文本、图像以及音频、视频和文档编码到一个共享的向量空间】'UForm - Multi-Modal Inference Library For Semantic Search Applications and Mid-Fusion Vision-Language Transformers' Unum GitHub: github.com/unum-cloud/uform

[CL]《Language Is Not All You Need: Aligning Perception with Language Models》S Huang, L Dong, W Wang, Y Hao, S Singhal… [Microsoft] (2023) 多模态大型语言模型 Kosmos-1 能感知一般模态,进行上下文学习,并遵循指令,在语言和多模态任务上取得了令人印象深刻的表现,无需进行微调,这表明将语言和感知相结合,是迈向通用人工智能的关键一步。 https://arxiv.org/abs/2302.14045

[CV]《Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning》A Yang, A Nagrani, P H Seo, A Miech, J Pont-Tuset, I Laptev, J Sivic, C Schmid [Google Research] (2023) https://arxiv.org/abs/2302.14115 Vid2Seq 是一种用于稠密视频描述的视觉语言模型,利用未标记叙述视频进行大规模预训练,在各种基准上取得了最先进的结果。

[CV]《Prismer: A Vision-Language Model with An Ensemble of Experts》S Liu, L Fan, E Johns, Z Yu, C Xiao, A Anandkumar [Imperial College London & NVIDIA] (2023) Prismer 是一个数据高效且参数高效的视觉语言模型,利用领域专家组合来实现高效的多模态生成。 https://arxiv.org/abs/2303.02506

[CV]《ViperGPT: Visual Inference via Python Execution for Reasoning》D Surís, S Menon, C Vondrick [Columbia University] (2023) ViperGPT是一个框架,结合了视觉和语言模型,用代码生成来解决复杂的视觉查询,实现了最先进的结果。https://arxiv.org/abs/2303.08128

[CV]《Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models》C Wu, S Yin, W Qi, X Wang, Z Tang, N Duan [Microsoft Research Asia] (2023) Visual ChatGPT 是一个结合了视觉基础模型的系统,使用户能够超越语言格式与 ChatGPT 交互,解决复杂的视觉任务。 https://arxiv.org/abs/2303.04671

'visual-chatgpt-zh - Visual Chatgpt 支持中文版本' Shawn Wu GitHub: github.com/wxj630/visual-chatgpt-zh

[CV]《HiCLIP: Contrastive Language-Image Pretraining with Hierarchy-aware Attention》S Geng, J Yuan, Y Tian, Y Chen, Y Zhang [Rutgers University & ByteDance Inc] (2023) https://arxiv.org/abs/2303.02995 HiCLIP 在 CLIP 的基础上进行了改进,纳入了层次感知的注意力,以无监督方式捕捉图像和文本中传达的高层次和细粒度的语义的层次性。

[CV]《Learning Object-Language Alignments for Open-Vocabulary Object Detection》C Lin, P Sun, Y Jiang, P Luo, L Qu, G Haffari, Z Yuan, J Cai [Monash University & The University of Hong Kong & ByteDance] (2023) 从语言中学习开放词表目标检测具有挑战性,所提出的 VLDet 从图像-标题对中学习免标注的物体-语言对齐,在 COCO 和 LVIS 上实现了新的最先进水平。 https://openreview.net/forum?id=mjHlitXvReu&continueFlag=45b7de259cb193d01394f25ca952a334

【用图片生成stable diffusion提示(图像描述)】“Image to Prompt - Go from image to stable diffusion prompt in 30s” https://imagetoprompt.com/

'Visual OpenLLM - 基于开源模型, 以交互方式连接不同视觉模型的开源工具' visual-openllm GitHub: github.com/visual-openllm/visual-openllm

'OFA-Chinese:中文多模态统一预训练模型 - transformers结构的中文OFA模型' Yang JianXin GitHub: github.com/yangjianxin1/OFA-Chinese

【OpenFlamingo:一个开源框架,允许训练和评估大型多模态模型(LMM),是DeepMind的Flamingo模型的一个复现,包括Python框架、大规模多模态数据集、上下文学习评估基准和基于LLaMA的OpenFlamingo-9B模型的第一个版本,旨在推动多模态机器学习的进展】《Announcing OpenFlamingo: An open-source framework for training vision-language models with in-context learning | LAION》 github.com/mlfoundations/open_flamingo

genmo.ai 这是一个极其富有创造力的多模态聊天机器人,除了能接收图片输入之外,它还可以自动生成、编辑图片与视频。 跟现在所流行的 #AI绘图# 工具 Midjourney 不同,它更像是一个 "交互式" 助手,能通过用户的指令反馈,及时更新与调整图像内容。 微博视频中演示了 Genmo 的几个基础功能:

  1. 通过对话,将夜空图像转为动态图像;
  2. 融合图像与视频,做成电影,用淡入淡出转场效果;
  3. 生成更高分辨率,视野更宽的视频; 同样,Genmo 也需要先加入 waitlist,等待通过才能使用。 除了上述提到的这些功能,你还可以用它来进行一些比较简单的短视频创作,实时修改图片,生成多种不同风格的图片等。

Scenex https://scenex.jina.ai/ 给一张图片,它用自然语言描述这张图片由什么内容。

这个模型可以根据图片生成文字描述,甚至可以就图片回答一些简单的问题。 🔗 huggingface.co/spaces/Salesforce/BLIP ​​​ https://blog.salesforceairesearch.com/blip-bootstrapping-language-image-pretraining/

【ChatGPT + BLIP2 + OFA + GRIT + Segment Anything + ControlNet 图像的文本段描述】’Transform Image Into Unique Paragraph - Transform Image into Unique Paragraph with ChatGPT, BLIP2, OFA, GRIT, Segment Anything, ControlNet.' Show Lab GitHub: github.com/showlab/Image2Paragraph

一个类似于GPT-4的但是开源的多模态实现,可以执行复杂的视觉语言任务,例如: ✍️ 为图片赋诗(想起@instagram四哥) 🔍 指出问题并提供解决方案 🌐 草图 ➡️生成 网站 🔗 minigpt-4.github.io 🔗 github.com/Vision-CAIR/MiniGPT-4

Ask-Anything是一个简单而有趣的视频聊天工具,可以针对视频的内容提问,然后机器人会为你解答。 🔗 github.com/OpenGVLab/Ask-Anything

【GPT4Tools: 一个智能系统,可以自动决定、控制和利用不同的视觉基础模型,允许用户在对话过程中与图像进行交互】'GPT4Tools: Teaching LLM to Use Tools via Self-instruction - GPT4Tools is an intelligent system that can automatically deciding, controlling, and utilizing different visual foundation models, allowing the user to interact with images during a conversation.' Lin Song GitHub: github.com/StevenGrove/GPT4Tools

RAM: Relate-Anything-Model RAM 这个项目能分析图像中对象之间的关系 这是一个将Meta的Segment-Anything 网页链接 模型与ECCV'22论文相结合的演示: Panoptic Scene Graph Generation 网页链接 。 Relate Anything Model能够将一幅图像作为输入,并利用SAM来识别图像中的对象。然后RAM可以分析图像中对象之间的关系。 🔗github.com/Luodian/RelateAnything

'MiniGPT-4-ZH - MiniGPT-4 中文部署翻译' Ori GitHub: github.com/RiseInRose/MiniGPT-4-ZH

【Graphit: 一个新模型,可以在一个框架内执行各种图像编辑任务,通过对文本到图像扩散模型进行额外训练来编辑图像,结合了文本到图像扩散模型,如Stable diffusion和unCLIP,可以实现广泛的图像编辑功能】'Graphit: A Unified Framework for Diverse Image Editing Tasks - Official Pytorch implementation of "Graphit: A Unified Framework for Diverse Image Editing Tasks"' NAVER/LINE Vision GitHub: github.com/navervision/Graphit

LLaVA: Large Language and Vision Assistant 大型语言和视觉助理 一个轻量级、多模式的GPT-4 官方Demo:🔗llava-vl.github.io🔗 Repo:🔗github.com/haotian-liu/LLaVA 【多模态语言-图像数据集、LLaVA模型及在线Demo:利用语言模型生成多模态语言-图像指令遵循数据,并用这些数据训练出大型多模态模型LLaVA,用于通用的视觉和语言理解。用语言模型GPT-4生成多模态指令遵循数据,并在HuggingFace Dataset上公开了15.8万条样本;将预训练的CLIP ViT-L/14视觉编码器和大型语言模型LLaMA连接起来,并采用了两阶段的指令微调过程;在一个合成多模态指令遵循数据集上,LLaVA表现出了令人印象深刻的多模态聊天能力,有时甚至展现出了多模态GPT-4的行为,并获得了85.1%相对于GPT-4的得分;在Science QA数据集上,LLaVA和GPT-4的协同达到了92.53%的新的最佳准确率】“Visual Instruction Tuning - LLaVA: Large Language and Vision Assistant”

【Dolphin:基于大型语言模型的通用视频交互平台,旨在为视频理解、处理和生成构建一个聊天机器人。主要功能包括:视频处理:用moviepy实现基本的视频剪辑、添加字幕、提取音频、添加音频等功能,还包括视频转换为姿态/深度/边缘等功能;视频生成:文本到视频、姿态/深度和文本到视频、视频pix2pix等;视频交互:使用先进的语言模型处理自然语言查询,并生成相关的视频回复,支持多种场景和上下文】’Dolphin - General video interaction platform based on LLMs' kaleido-lab GitHub: github.com/kaleido-lab/dolphin

【MaMMUT是一个紧凑的、具有2B参数的多模态模型,由一个图像编码器和一个文本解码器组成,在图像-文本检索、文本-图像检索、视觉问答、视频问答、视频描述生成、开放词汇检测等多个视觉语言任务中取得了SOTA或竞争性表现。该模型使用Decoder-only架构,通过两遍学习的方法来训练生成任务和对比任务,从而同时实现了两种任务类型。在不需要自适应的情况下,MaMMUT在图像-文本和文本-图像检索任务中的效果都超过了之前的SOTA模型】《MaMMUT: A simple vision-encoder text-decoder architecture for multimodal tasks – Google AI Blog》 https://ai.googleblog.com/2023/05/mammut-simple-vision-encoder-text.html

【mPLUG-Owl: 基于LLaMA和ViT并增加视觉摘要模块,构建多模态模块化语言模型,采用两阶段训练范式,第一阶段固定文本模块,微调训练视觉模块学习视觉语义知识,第二阶段采用增加少量参数的LoRA模块,并混合纯文本和多模态指令数据联合训练。mPLUG-Owl构建了一个公平比较的多模态指令评测集,评测效果较最近的miniGPT4和LLaVA取得一定提升,并涌现了些多语言、多图理解以及文档理解等能力】'mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality' X-PLUG GitHub: github.com/X-PLUG/mPLUG-Owl

MetaAI 重磅开源 ImageBind,可让模型跨 6 种不同的模态(图像、文本、音频、深度、热能和 IMU 数据)进行联动! 基于该项目,开发者可以「开箱即用」实现包括跨模态检索、使用算术合成模态、跨模态检测和生成等各类新兴应用。 详细介绍:MetaAI 在 GitHub 丢出重磅炸弹,让 AI 再次成功进化! 借助 ImageBind,则可以做到直接通过声音来直接生成图像。这使得 AI 能够更加深入了解人类情感,理解他们的喜怒哀乐,进而为人类提供更好的服务。 当你举起手机,录制一个海边日落的视频时,AI 便能自动根据视频内容来生成文案和字幕,并匹配上合适的背景音乐。 甚至 AI 还有可能通过一首歌,直接为歌手生成一段视频 MV。 此举将为 AIGC 技术带来更为广泛的应用场景,一大波更为有趣、实用的 AI 项目也即将来袭。 GitHub:github.com/facebookresearch/ImageBind

【用LoRA微调ImageBind】’Unofficial ImageBind Fine-tuning with LoRA - Fine-tuning "ImageBind One Embedding Space to Bind Them All" with LoRA' Fares Abawi GitHub: github.com/fabawi/ImageBind-LoRA

清华开源的多模态语言模型VisualGLM-6B来了! 地址:github.com/THUDM/VisualGLM-6B VisualGLM-6B 是一个开源的,支持图像、中文和英文的多模态对话语言模型,语言模型基于 ChatGLM-6B,具有 62 亿参数;图像部分通过训练 BLIP2-Qformer 构建起视觉模型与语言模型的桥梁,整体模型共78亿参数。

【Oryx Video-ChatGPT:一个大型视频语言模型,基于专用的视频编码器和大型语言模型(LLM) ,支持视频理解和视频对话】'Oryx Video-ChatGPT - Video-ChatGPT is a large vision-language model with a dedicated video-encoder and large language model (LLM), enabling video understanding and conversation about videos.' ORYX GitHub: github.com/mbzuai-oryx/Video-ChatGPT

PandaGPT,整合了Meta的ImageNet和开源大语言模型(LLM)Vicuna,实现了LLM的多模态输入和输出。 项目首页:panda-gpt.github.io🔗 代码库:github.com/yxuansu/PandaGPT

【ImageBind揭秘:Meta发布的最新模型ImageBind,是一种多模态AI模型,能理解六种不同类型的输入,并将它们结合起来,以与我们相似的方式看待世界。该模型可以与生成式AI解决方案一起使用,从声音中创建图像,将声音和图像组合成融合了两个概念的新图像,或者提供与给定图像相关的音频。通过这种高效、低成本的方法,ImageBind有望成为AI创新的先驱。该模型的关键在于创造了一个联合嵌入空间,将不同的输入形式结合到一个独特的潜空间中,使机器能以与我们类似的方式理解世界】《Unveiling ImageBind: Pictures That Speak, Hear, and Feel with Meta’s New Model | Medium》 http://aicoco.net/s/33

【Macaw-LLM: 一种多模态语言模型,无缝结合图像、视频、音频和文本数据,为语言建模带来突破,集成了CLIP、Whisper和LLaMA等先进模型】'Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integration - Macaw-LLM: Multi-Modal Language Modeling with Image, Video, Audio, and Text Integration' Chenyang Lyu GitHub: github.com/lyuchenyang/Macaw-LLM

【RWKV实现】’RWKV - implementation of the pretrained RWKV model, with numerically-stable Triton kernels and code for doing LoRA fine-tuning' Ben Bolte GitHub: github.com/codekansas/rwkv

【nanoRWKV:RWKV语言模型的迷你版实现】’nanoRWKV - minimal implementation of RWKV language model following nanoGPT' Hannibal046 GitHub: github.com/Hannibal046/nanoRWKV

'VoxelGPT - AI assistant that can query visual datasets, search the FiftyOne docs, and answer general computer vision questions' Voxel51 GitHub: github.com/voxel51/voxelgpt

【God app:将图像生成、视频生成、音频生成和通用自然语言处理整合到一个界面中,只需一个提示,即可完成所有需求】'God app - an experimental project using OpenAI GPT Plugins and Replicate to combine all AI APis into one' by Nader Dabit GitHub: github.com/dabit3/openai-functions-god-app

【SEEChat - 一见多模态对话模型,一个侧重视觉能力的多模态对话大模型,基于单模态专家缝合路线,重点是将视觉能力与文本对话能力相集成】’SEEChat - Multimodal chatbot with computer vision capabilities integrated' 360 AI Research Institute GitHub: github.com/360CVGroup/SEEChat

'Visual-Chinese-LLaMA-Alpaca(VisualCLA)- 基于中文LLaMA&Alpaca大模型项目开发的多模态中文大模型。VisualCLA在中文LLaMA/Alpaca模型上增加了图像编码等模块,使LLaMA模型可以接收视觉信息' by Ziqing Yang GitHub: github.com/airaria/Visual-Chinese-LLaMA-Alpaca

【VisualRWKV:RWKV语言模型的视觉增强版本,使RWKV能够处理各种视觉任务】'VisualRWKV - the visual-enhanced version of the RWKV language model, enabling RWKV to handle various visual tasks.' Haowen Hou GitHub: github.com/howard-hou/VisualRWKV

'PaddleMIX:基于飞桨的跨模态大模型开发套件,聚合图像、文本、视频等多种模态,覆盖视觉语言预训练,文生图,文生视频等丰富的跨模态任务。提供开箱即用的开发体验,同时满足开发者灵活定制需求,探索通用人工智能' by PaddlePaddle GitHub: github.com/PaddlePaddle/PaddleMIX

【AI fashion assistant:将计算机视觉模型和LLM结合起来,以实现高级图像数据集查询】’AI fashion assistant - Our idea is to combine the power of computer vision model and LLMs. We use YOLO, CLIP and DINOv2 to extract high-level features from images. We pass the prompt, along with the extracted features, to LLM, allowing for advanced image dataset queries.' Piotr Skalski GitHub: github.com/SkalskiP/fashion-assistant

【LLM Engine:一个开源引擎,用于微调和提供大型语言模型的服务,是定制和提供LLM的最简单方式】’LLM Engine - Scale LLM Engine public repository' Scale GitHub: github.com/scaleapi/llm-engine

【多模态推理相关文献资源列表】’Awesome Multimodal Reasoning - Collection of papers and resources on Multimodal Reasoning, including Vision-Language Models, Multimodal Chain-of-Thought, Visual Inference, and others.' Armando Fortes GitHub: github.com/atfortes/Awesome-Multimodal-Reasoning

【Starlight Vision:多模态AI模型,可以使用文本、图像或视频片段作为输入,生成高质量的新颖视频内容。该模型利用先进的深度学习技术合成逼真而令人印象深刻的视频,可用于电影制作、广告、虚拟现实等多种应用】'Starlight Vision - A multi-modal AI Model that can generate high quality novel videos with text, images, or video clips.' Eternal Reclaimer GitHub: github.com/kyegomez/StarlightVision

【Transformer网络架构全面解析:详细解析了Transformer的各个关键组成部分,从注意力机制到编-解码器结构,探讨了利用Transformer的大型语言模型在自然语言处理之外的应用,探讨了该架构当前面临的挑战以及未来的发展方向,文章还提供了一份开源实现和其他补充资源的精选列表】《The Transformer Blueprint: A Holistic Guide to the Transformer Neural Network Architecture》 https://deeprevision.github.io/posts/001-transformer/

综述了新兴的视觉和语言基础模型,强调了它们在计算机视觉中的重要性和潜力。 https://arxiv.org/abs/2307.13721

'Chinese LLaVA - 支持中英文双语视觉-文本对话的开源可商用多模态模型' LinkSoul-AI GitHub: github.com/LinkSoul-AI/Chinese-LLaVA

探究了文本模型中的多模态神经元,发现这些神经元能够将图像信息转换为文本描述,并对图像描述生成产生系统性影响。 https://arxiv.org/abs/2308.01544

'Heron - 可无缝集成多种图像/视频和语言模型的库. 此外, 它还提供在各种数据集上训练的预训练权重' by Turing GitHub: github.com/turingmotors/heron

【blip-caption:用Salesforce BLIP实现的命令行图像描述生成工具】’blip-caption - Generate captions for images with Salesforce BLIP' Simon Willison GitHub: github.com/simonw/blip-caption

【多模态和大型多模态模型 (LMM)】

  • 多模态让模型能处理和生成不同模态的数据,如文本、图像、音频等,这对实际应用的AI系统至关重要。
  • LMM将额外的模态集成到大型语言模型中,例如CLIP(文本+图像)、Flamingo(文本+图像+视频)。
  • LMM的关键组件包括每个模态的编码器,多模态嵌入的对齐,以及用于文本生成的语言模型。
  • CLIP通过对网页规模的文本-图像对进行对比学习来对齐多模态嵌入,支持零样本分类和检索。
  • Flamingo通过Perceiver Resampler和GATED XATTN-DENSE等技术微调语言模型,以基于文本和视觉条件化地生成文本。
  • LMM的前沿研究关注融入更多模态,指令遵循,使用适配器的高效训练,多模态生成,以及评估基准的建立。
  • 多模态系统可以处理多种格式的数据,如文本、图像、音频等,这对真实世界的AI应用至关重要。
  • LMM通过在大型语言模型中加入图像等其他模态,扩展了其能力。
  • 多模态系统通常包含专门的编码器、跨模态嵌入对齐方法,以及生成模块。
  • CLIP通过自然语言监督和对比学习得到强大的视觉编码器,为后续LMM奠定基础。
  • Flamingo在CLIP基础上加入语言模型,可以生成文本,使用新的网络结构使语言模型可以处理多模态输入。
  • 当前的研究方向包括支持更多数据格式、使LMM遵循指令、用Adapter使训练更高效等。
  • 生成多模态输出是下一个重要方向,需要定义统一的中间表示。
  • 多模态系统仍在早期,但其重要性日益凸显。LLM的表现依赖基础语言模型。 《Multimodality and Large Multimodal Models (LMMs)》 https://huyenchip.com/2023/10/10/multimodal.html

CogVLM 是一个170亿参数的开源视觉语言模型,利用视觉专家模块深度整合语言编码和视觉编码,在 10来个项权威跨模态基准上取得了 SOTA 性能,可惜有2/3个指标还不是最好。好像可以把夜间的校门照片“自强不息,厚德载物”识别出来清华校园,大家试试自己的学校能否识别出来。哈哈,模型开源在github上了,欢迎大家去玩玩。 http://36.103.203.44:7861/

【UI-Act:一个 Transformer 模型,用于使用图形用户界面与计算机进行交互,其动机是为了无缝集成到人机工作流中,模型可以自然地使用专家的人工演示进行训练】’UI-Act - An AI agent for interacting with a computer using the graphical user interface' Tobias Norlund GitHub: github.com/TobiasNorlund/UI-Act

【Fuyu-8b多模态模型在线Demo】《Fuyu Multimodal - a Hugging Face Space by adept》 https://huggingface.co/spaces/adept/fuyu-8b-demo

提出一种新的目标位置建模方法(pixel2emb),通过将位置嵌入编码为特定标记,实现了对不同位置格式的支持,提高了多模态对话中的视觉理解能力。 https://arxiv.org/abs/2311.04498 [CV]《NExT-Chat: An LMM for Chat, Detection and Segmentation》A Zhang, L Zhao, C Xie, Y Zheng, W Ji, T Chua [National University of Singapore & Alibaba Group] (2023)

Video-LLaVA:北大开源的视觉语言大模型 地址:github.com/PKU-YuanGroup/Video-LLaVA 可以理解图片和理解视频内容。