upload

.gitattributes

@@ -33,3 +33,6 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+*.png filter=lfs diff=lfs merge=lfs -text
+*.jpg filter=lfs diff=lfs merge=lfs -text
+*.jpeg filter=lfs diff=lfs merge=lfs -text

.mv

@@ -0,0 +1 @@
+Revision:v1.0.0,CreatedAt:1666686679

README.md

@@ -0,0 +1,197 @@
+---
+tasks:
+- visual-question-answering
+
+widgets:
+  - task: visual-question-answering
+    inputs:
+      - type: image
+        name: image
+        title: 图片
+        validator:
+          max_size: 10M
+          max_resolution: 5000*5000
+      - type: text
+        name: question
+        title: 问题
+    examples:
+      - name: 1
+        title: 示例1
+        inputs:
+          - name: image
+            data: https://alice-open.oss-cn-zhangjiakou.aliyuncs.com/mPLUG/image_mplug_vqa_5.jpg
+          - name: question
+            data: what name is this guy?
+      - name: 2
+        title: 示例2
+        inputs:
+          - name: image
+            data: https://alice-open.oss-cn-zhangjiakou.aliyuncs.com/mPLUG/image_mplug_vqa_4.jpg
+          - name: question
+            data: what is the name of the planet?
+      - name: 3
+        title: 示例3
+        inputs:
+          - name: image
+            data: https://alice-open.oss-cn-zhangjiakou.aliyuncs.com/mPLUG/image_mplug_vqa_1.jpg
+          - name: question
+            data: what airline owns this plane?
+      - name: 4
+        title: 示例4
+        inputs:
+          - name: image
+            data: http://xingchen-data.oss-cn-zhangjiakou.aliyuncs.com/maas/visual-question-answering/visual_question_answering.png
+          - name: question
+            data: what is grown on the plant?
+      - name: 5
+        title: 示例5
+        inputs:
+          - name: image
+            data: https://alice-open.oss-cn-zhangjiakou.aliyuncs.com/mPLUG/image_mplug_vqa_3.jpg
+          - name: question
+            data: What do you call the devices on top of the pole?
+      - name: 6
+        title: 示例6
+        inputs:
+          - name: image
+            data:  https://alice-open.oss-cn-zhangjiakou.aliyuncs.com/mPLUG/image_mplug_vqa_2.jpg
+          - name: question
+            data: what does this machine do?
+    inferencespec:
+      cpu: 4
+      memory: 12000
+      gpu: 1
+      gpu_memory: 16000
+
+model-type:
+- mplug
+
+domain:
+- multi-modal
+
+frameworks:
+- pytorch
+
+backbone:
+- transformer
+
+containers:
+
+metrics:
+- accuracy
+
+license: Apache License 2.0
+
+finetune-support: True
+
+language: 
+- en
+
+tags:
+- transformer
+- Alibaba
+- volume:abs/2205.12005
+
+datasets:
+  train:
+  - 14M image-text pairs(google cc, mscoco, vg, sbu)
+  - VQA
+  test: 
+  - VQA test set
+  - modelscope/vqa_trial
+---
+
+# 视觉问答介绍
+视觉问答：给定一个问题和图片，通过图片信息来给出答案。需要模型具备多模态理解的能力，目前主流的方法大多是基于多模态预训练，最为知名的视觉问答数据集包括：VQA，GQA等。
+
+## 模型描述
+
+本任务是mPLUG，在英文VQA数据集进行finetune的视觉问答下游任务。mPLUG模型是统一理解和生成的多模态基础模型，该模型提出了基于skip-connections的高效跨模态融合框架。其中，mPLUG在VQA上支持开放阈生成，达到开放阈生成的SOTA，详见：[mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connections](https://arxiv.org/abs/2205.12005)
+
+![mplug](./resources/model.png)
+
+模型生成结果如下图所示：
+
+![vqa_case](./resources/case.png)
+
+
+## 期望模型使用方式以及适用范围
+本模型主要用于给问题和对应图片生成答案。用户可以自行尝试各种输入文档。具体调用方式请参考代码示例。
+
+### 如何使用
+在安装完成MaaS-lib之后即可使用visual-question-answering的能力 （注意：模型运行约需占用 9G 内存）
+
+#### 代码范例
+```python
+from modelscope.pipelines import pipeline
+from modelscope.utils.constant import Tasks
+
+model_id = 'damo/mplug_visual-question-answering_coco_large_en'
+input_vqa = {
+    'image': 'https://alice-open.oss-cn-zhangjiakou.aliyuncs.com/mPLUG/image_mplug_vqa.jpg',
+    'question': 'What is the woman doing?',
+}
+
+pipeline_vqa = pipeline(Tasks.visual_question_answering, model=model_id)
+print(pipeline_vqa(input_vqa))
+
+```
+
+### 模型局限性以及可能的偏差
+模型在数据集上训练，有可能产生一些偏差，请用户自行评测后决定如何使用。
+
+## 训练数据介绍
+本模型训练数据集是VQA，数据集包含83k图片， 具体数据可以[下载](https://visualqa.org/)
+
+## 模型训练流程
+
+### 预处理
+训练数据集需要包含 image，question，answer，以下为使用截取的部分 coco_caption 数据集进行的训练代码预处理示例：
+```python
+datadict = MsDataset.load('coco_captions_small_slice')
+self.train_dataset = MsDataset(datadict['train'].to_hf_dataset().map(
+    lambda _: {
+        'question': 'what the picture describes?'
+    }).rename_column('image:FILE',
+                     'image').rename_column('answer:Value', 'answer'))
+self.test_dataset = MsDataset(datadict['test'].to_hf_dataset().map(
+    lambda _: {
+        'question': 'what the picture describes?'
+    }).rename_column('image:FILE',
+                     'image').rename_column('answer:Value', 'answer'))
+```
+
+### 训练
+以下为使用 modelscope 中的 trainer 进行训练的代码示例：
+```python
+kwargs = dict(
+    model='damo/mplug_visual-question-answering_coco_large_en',
+    train_dataset=self.train_dataset,
+    eval_dataset=self.test_dataset,
+    max_epochs=self.max_epochs,
+    work_dir=self.tmp_dir)
+
+trainer: EpochBasedTrainer = build_trainer(
+    name=Trainers.nlp_base_trainer, default_args=kwargs)
+trainer.train()
+```
+
+## 数据评估及结果
+mPLUG在VQA数据集，同等规模和预训练数据的模型中取得SOTA，VQA榜单上排名前列
+
+![mplug_vqa_score](./resources/vqa_exp.png)
+
+![vqa_leaderboard](./resources/vqa.png)
+### 相关论文以及引用信息
+如果我们的模型对您有帮助，请您引入我们的文章：
+```BibTeX
+@inproceedings{li2022mplug,
+      title={mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connections}, 
+      author={Li, Chenliang and Xu, Haiyang and Tian, Junfeng and Wang, Wei and Yan, Ming and Bi, Bin and Ye, Jiabo and Chen, Hehong and Xu, Guohai and Cao, Zheng and Zhang, Ji and Huang, Songfang and Huang, Fei and Zhou, Jingren and Luo Si},
+      year={2022},
+      journal={arXiv}
+}
+```
+
+
+

config.yaml

@@ -0,0 +1,39 @@
+bert_config: 'config_bert.json'
+
+image_res: 504
+batch_size_train: 128
+vision_width: 1024
+distill: True
+clip_name: "ViT-L-14"
+batch_size_test: 64 
+k_test: 128
+
+alpha: 0.4
+warm_up: True
+
+eos: '[SEP]'
+
+optimizer: {opt: adamW, lr1: 3e-5, lr2: 5e-6, weight_decay: 0.02}
+schedular: {sched: cosine, lr: 3e-5, epochs: 8, min_lr: 1e-6, decay_rate: 1, warmup_lr: 1e-5, warmup_epochs: 4, cooldown_epochs: 0}
+
+# predictor
+min_length: 1
+max_length: 10
+beam_size: 5
+add_ocr: False
+add_object: False
+text_encoder: 'bert-base-uncased'
+text_decoder: 'bert-base-uncased'
+
+# clip
+clip_embed_dim: 768
+clip_image_resolution: 224
+clip_vision_layers: 24
+clip_vision_width: 1024
+clip_vision_patch_size: 14
+clip_context_length: 77
+clip_vocab_size: 49408
+clip_transformer_width: 768
+clip_transformer_heads: 12
+clip_transformer_layers: 12
+

config_bert.json

@@ -0,0 +1,27 @@
+{
+  "architectures": [
+    "BertForMaskedLM"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "layer_norm_eps": 1e-12,
+  "max_position_embeddings": 512,
+  "model_type": "bert",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "pad_token_id": 0,
+  "type_vocab_size": 2,
+  "vocab_size": 30522,
+  "encoder_width": 768,
+  "add_cross_attention": false,
+  "use_cache":false,
+  "gradient_checkpointing": false,
+  "text_encoder_layers": 6,
+  "fusion_layers": 6,
+  "text_decode_layers": 12,
+  "stride_layer": 6
+}

configuration.json

@@ -0,0 +1,60 @@
+{
+    "framework": "pytorch",
+    "task": "visual-question-answering",
+    "preprocessor": {
+        "type": "mplug-tasks-preprocessor"
+    },
+    "model": {
+        "type": "mplug"
+    },
+    "pipeline": {
+        "type": "visual-question-answering"
+    },
+    "train": {
+        "work_dir": "/tmp",
+        "max_epochs": 3,
+        "dataloader": {
+            "batch_size_per_gpu": 2,
+            "workers_per_gpu": 1
+        },
+        "optimizer": {
+            "type": "SGD",
+            "lr": 0.01,
+            "options": {
+                "grad_clip": {
+                    "max_norm": 2.0
+                }
+            }
+        },
+        "lr_scheduler": {
+            "type": "StepLR",
+            "step_size": 2,
+            "options": {
+                "warmup": {
+                    "type": "LinearWarmup",
+                    "warmup_iters": 2
+                }
+            }
+        },
+        "hooks": [{
+            "type": "CheckpointHook",
+            "interval": 1
+        }, {
+            "type": "TextLoggerHook",
+            "interval": 1
+        }, {
+            "type": "IterTimerHook"
+        }, {
+            "type": "EvaluationHook",
+            "interval": 1
+        }]
+    },
+    "evaluation": {
+        "dataloader": {
+            "batch_size_per_gpu": 2,
+            "workers_per_gpu": 1,
+            "shuffle": false
+        }
+    }
+}
+

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:702b1fcb760b1a2bb4ab7f7b0d6f540aad438716f9fdb602f85e121d5a371332
+size 2989186229