forked from ailab/MiniCPM-Llama3-V-2_5
first commit
This commit is contained in:
commit
53bbdcfc8a
|
|
@ -0,0 +1,37 @@
|
||||||
|
*.7z filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.arrow filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.bin filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.ftz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.gz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.h5 filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.joblib filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.model filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.npy filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.npz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.onnx filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.ot filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.parquet filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pb filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pickle filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pkl filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.rar filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||||
|
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tar filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tflite filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tgz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.wasm filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.xz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.zip filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.zst filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.png filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.gif filter=lfs diff=lfs merge=lfs -text
|
||||||
|
|
@ -0,0 +1,212 @@
|
||||||
|
---
|
||||||
|
pipeline_tag: visual-question-answering
|
||||||
|
language:
|
||||||
|
- en
|
||||||
|
- zh
|
||||||
|
datasets:
|
||||||
|
- HaoyeZhang/RLAIF-V-Dataset
|
||||||
|
---
|
||||||
|
|
||||||
|
|
||||||
|
<h1>A GPT-4V Level Multimodal LLM on Your Phone</h1>
|
||||||
|
|
||||||
|
[GitHub](https://github.com/OpenBMB/MiniCPM-V) | [Demo](https://huggingface.co/spaces/openbmb/MiniCPM-Llama3-V-2_5) | <a href="https://github.com/OpenBMB/MiniCPM-V/blob/main/docs/wechat.md" target="_blank"> WeChat</a>
|
||||||
|
|
||||||
|
|
||||||
|
## News <!-- omit in toc -->
|
||||||
|
|
||||||
|
#### 📌 Pinned
|
||||||
|
|
||||||
|
* [2024.05.28] 🚀🚀🚀 MiniCPM-Llama3-V 2.5 now fully supports its feature in llama.cpp and ollama! Please pull the latest code **of our provided forks** ([llama.cpp](https://github.com/OpenBMB/llama.cpp/blob/minicpm-v2.5/examples/minicpmv/README.md), [ollama](https://github.com/OpenBMB/ollama/tree/minicpm-v2.5)). GGUF models in various sizes are available [here](https://huggingface.co/openbmb/MiniCPM-Llama3-V-2_5-gguf/tree/main). We are working hard to merge PRs into official repositories. Please stay tuned! You can visit our [GitHub](https://github.com/OpenBMB/MiniCPM-V) repository for more information!
|
||||||
|
* [2024.05.28] 💫 We now support LoRA fine-tuning for MiniCPM-Llama3-V 2.5, using only 2 V100 GPUs! See more statistics [here](https://github.com/OpenBMB/MiniCPM-V/tree/main/finetune#model-fine-tuning-memory-usage-statistics).
|
||||||
|
* [2024.05.23] 🔍 We've released a comprehensive comparison between Phi-3-vision-128k-instruct and MiniCPM-Llama3-V 2.5, including benchmarks evaluations, multilingual capabilities, and inference efficiency 🌟📊🌍🚀. Click [here](https://github.com/OpenBMB/MiniCPM-V/blob/main/docs/compare_with_phi-3_vision.md) to view more details.
|
||||||
|
* [2024.05.23] 🔥🔥🔥 MiniCPM-V tops GitHub Trending and HuggingFace Trending! Our demo, recommended by Hugging Face Gradio’s official account, is available [here](https://huggingface.co/spaces/openbmb/MiniCPM-Llama3-V-2_5). Come and try it out!
|
||||||
|
|
||||||
|
<br>
|
||||||
|
|
||||||
|
* [2024.06.03] Now, you can run MiniCPM-Llama3-V 2.5 on multiple low VRAM GPUs(12 GB or 16 GB) by distributing the model's layers across multiple GPUs. For more details, Check this [link](https://github.com/OpenBMB/MiniCPM-V/blob/main/docs/inference_on_multiple_gpus.md).
|
||||||
|
* [2024.05.25] MiniCPM-Llama3-V 2.5 now supports streaming outputs and customized system prompts. Try it at [here](#usage)
|
||||||
|
* [2024.05.24] We release the [MiniCPM-Llama3-V 2.5 gguf](https://huggingface.co/openbmb/MiniCPM-Llama3-V-2_5-gguf), which supports [llama.cpp](https://github.com/OpenBMB/MiniCPM-V/tree/main?tab=readme-ov-file#inference-with-llamacpp) inference and provides a 6~8 token/s smooth decoding on mobile phones. Try it now!
|
||||||
|
* [2024.05.20] We open-soure MiniCPM-Llama3-V 2.5, it has improved OCR capability and supports 30+ languages, representing the first end-side MLLM achieving GPT-4V level performance! We provide [efficient inference](#deployment-on-mobile-phone) and [simple fine-tuning](https://github.com/OpenBMB/MiniCPM-V/blob/main/finetune/readme.md). Try it now!
|
||||||
|
|
||||||
|
|
||||||
|
## Model Summary
|
||||||
|
|
||||||
|
**MiniCPM-Llama3-V 2.5** is the latest model in the MiniCPM-V series. The model is built on SigLip-400M and Llama3-8B-Instruct with a total of 8B parameters. It exhibits a significant performance improvement over MiniCPM-V 2.0. Notable features of MiniCPM-Llama3-V 2.5 include:
|
||||||
|
|
||||||
|
- 🔥 **Leading Performance.**
|
||||||
|
MiniCPM-Llama3-V 2.5 has achieved an average score of 65.1 on OpenCompass, a comprehensive evaluation over 11 popular benchmarks. **With only 8B parameters, it surpasses widely used proprietary models like GPT-4V-1106, Gemini Pro, Claude 3 and Qwen-VL-Max** and greatly outperforms other Llama 3-based MLLMs.
|
||||||
|
|
||||||
|
- 💪 **Strong OCR Capabilities.**
|
||||||
|
MiniCPM-Llama3-V 2.5 can process images with any aspect ratio and up to 1.8 million pixels (e.g., 1344x1344), achieving an **700+ score on OCRBench, surpassing proprietary models such as GPT-4o, GPT-4V-0409, Qwen-VL-Max and Gemini Pro**. Based on recent user feedback, MiniCPM-Llama3-V 2.5 has now enhanced full-text OCR extraction, table-to-markdown conversion, and other high-utility capabilities, and has further strengthened its instruction-following and complex reasoning abilities, enhancing multimodal interaction experiences.
|
||||||
|
|
||||||
|
- 🏆 **Trustworthy Behavior.**
|
||||||
|
Leveraging the latest [RLAIF-V](https://github.com/RLHF-V/RLAIF-V/) method (the newest technology in the [RLHF-V](https://github.com/RLHF-V) [CVPR'24] series), MiniCPM-Llama3-V 2.5 exhibits more trustworthy behavior. It achieves **10.3%** hallucination rate on Object HalBench, lower than GPT-4V-1106 (13.6%), achieving the best-level performance within the open-source community. [Data released](https://huggingface.co/datasets/openbmb/RLAIF-V-Dataset).
|
||||||
|
|
||||||
|
- 🌏 **Multilingual Support.**
|
||||||
|
Thanks to the strong multilingual capabilities of Llama 3 and the cross-lingual generalization technique from [VisCPM](https://github.com/OpenBMB/VisCPM), MiniCPM-Llama3-V 2.5 extends its bilingual (Chinese-English) multimodal capabilities to **over 30 languages including German, French, Spanish, Italian, Russian etc.** [All Supported Languages](./assets/minicpm-llama-v-2-5_languages.md).
|
||||||
|
|
||||||
|
- 🚀 **Efficient Deployment.**
|
||||||
|
MiniCPM-Llama3-V 2.5 systematically employs **model quantization, CPU optimizations, NPU optimizations and compilation optimizations**, achieving high-efficiency deployment on edge devices. For mobile phones with Qualcomm chips, we have integrated the NPU acceleration framework QNN into llama.cpp for the first time. After systematic optimization, MiniCPM-Llama3-V 2.5 has realized a **150-fold acceleration in multimodal large model end-side image encoding** and a **3-fold increase in language decoding speed**.
|
||||||
|
|
||||||
|
- 💫 **Easy Usage.**
|
||||||
|
MiniCPM-Llama3-V 2.5 can be easily used in various ways: (1) [llama.cpp](https://github.com/OpenBMB/llama.cpp/blob/minicpm-v2.5/examples/minicpmv/README.md) and [ollama](https://github.com/OpenBMB/ollama/tree/minicpm-v2.5/examples/minicpm-v2.5) support for efficient CPU inference on local devices, (2) [GGUF](https://huggingface.co/openbmb/MiniCPM-Llama3-V-2_5-gguf) format quantized models in 16 sizes, (3) efficient [LoRA](https://github.com/OpenBMB/MiniCPM-V/tree/main/finetune#lora-finetuning) fine-tuning with only 2 V100 GPUs, (4) [streaming output](https://huggingface.co/openbmb/MiniCPM-Llama3-V-2_5#usage), (5) quick local WebUI demo setup with [Gradio](https://github.com/OpenBMB/MiniCPM-V/blob/main/web_demo_2.5.py) and [Streamlit](https://github.com/OpenBMB/MiniCPM-V/blob/main/web_demo_streamlit-2_5.py), and (6) interactive demos on [HuggingFace Spaces](https://huggingface.co/spaces/openbmb/MiniCPM-Llama3-V-2_5).
|
||||||
|
|
||||||
|
### Evaluation <!-- omit in toc -->
|
||||||
|
|
||||||
|
Results on TextVQA, DocVQA, OCRBench, OpenCompass MultiModal Avg , MME, MMBench, MMMU, MathVista, LLaVA Bench, RealWorld QA, Object HalBench.
|
||||||
|
|
||||||
|
<div align="center">
|
||||||
|
<img src="https://cdn-uploads.huggingface.co/production/uploads/64abc4aa6cadc7aca585dddf/v2KE3wqQgM05ZW3dH2wbx.png" width="110%" />
|
||||||
|
</div>
|
||||||
|
|
||||||
|
|
||||||
|
Evaluation results of multilingual LLaVA Bench
|
||||||
|
<div align="center">
|
||||||
|
<img src="assets/minicpmv-llama3-v2.5/llavabench_compare.png" width="110%" />
|
||||||
|
</div>
|
||||||
|
|
||||||
|
|
||||||
|
### Examples <!-- omit in toc -->
|
||||||
|
|
||||||
|
<table align="center">
|
||||||
|
<p align="center">
|
||||||
|
<img src="assets/minicpmv-llama3-v2.5/cases_all.png" width=95%/>
|
||||||
|
</p>
|
||||||
|
</table>
|
||||||
|
|
||||||
|
We deploy MiniCPM-Llama3-V 2.5 on end devices. The demo video is the raw screen recording on a Xiaomi 14 Pro without edition.
|
||||||
|
|
||||||
|
<table align="center">
|
||||||
|
<p align="center">
|
||||||
|
<img src="assets/gif_cases/ticket.gif" width=40% style="display:inline-block;"/>
|
||||||
|
<img src="assets/gif_cases/meal_plan.gif" width=40% style="display:inline-block;"/>
|
||||||
|
</p>
|
||||||
|
</table>
|
||||||
|
|
||||||
|
<table align="center">
|
||||||
|
<p align="center">
|
||||||
|
<img src="assets/gif_cases/1-4.gif" width=80%/>
|
||||||
|
</p>
|
||||||
|
</table>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
## Demo
|
||||||
|
Click here to try out the Demo of [MiniCPM-Llama3-V 2.5](https://huggingface.co/spaces/openbmb/MiniCPM-Llama3-V-2_5).
|
||||||
|
|
||||||
|
## Deployment on Mobile Phone
|
||||||
|
Coming soon.
|
||||||
|
|
||||||
|
## Usage
|
||||||
|
Inference using Huggingface transformers on NVIDIA GPUs. Requirements tested on python 3.10：
|
||||||
|
```
|
||||||
|
Pillow==10.1.0
|
||||||
|
torch==2.1.2
|
||||||
|
torchvision==0.16.2
|
||||||
|
transformers==4.40.0
|
||||||
|
sentencepiece==0.1.99
|
||||||
|
```
|
||||||
|
|
||||||
|
```python
|
||||||
|
# test.py
|
||||||
|
import torch
|
||||||
|
from PIL import Image
|
||||||
|
from transformers import AutoModel, AutoTokenizer
|
||||||
|
|
||||||
|
model = AutoModel.from_pretrained('openbmb/MiniCPM-Llama3-V-2_5', trust_remote_code=True, torch_dtype=torch.float16)
|
||||||
|
model = model.to(device='cuda')
|
||||||
|
|
||||||
|
tokenizer = AutoTokenizer.from_pretrained('openbmb/MiniCPM-Llama3-V-2_5', trust_remote_code=True)
|
||||||
|
model.eval()
|
||||||
|
|
||||||
|
image = Image.open('xx.jpg').convert('RGB')
|
||||||
|
question = 'What is in the image?'
|
||||||
|
msgs = [{'role': 'user', 'content': question}]
|
||||||
|
|
||||||
|
res = model.chat(
|
||||||
|
image=image,
|
||||||
|
msgs=msgs,
|
||||||
|
tokenizer=tokenizer,
|
||||||
|
sampling=True, # if sampling=False, beam_search will be used by default
|
||||||
|
temperature=0.7,
|
||||||
|
# system_prompt='' # pass system_prompt if needed
|
||||||
|
)
|
||||||
|
print(res)
|
||||||
|
|
||||||
|
## if you want to use streaming, please make sure sampling=True and stream=True
|
||||||
|
## the model.chat will return a generator
|
||||||
|
res = model.chat(
|
||||||
|
image=image,
|
||||||
|
msgs=msgs,
|
||||||
|
tokenizer=tokenizer,
|
||||||
|
sampling=True,
|
||||||
|
temperature=0.7,
|
||||||
|
stream=True
|
||||||
|
)
|
||||||
|
|
||||||
|
generated_text = ""
|
||||||
|
for new_text in res:
|
||||||
|
generated_text += new_text
|
||||||
|
print(new_text, flush=True, end='')
|
||||||
|
```
|
||||||
|
|
||||||
|
Please look at [GitHub](https://github.com/OpenBMB/MiniCPM-V) for more detail about usage.
|
||||||
|
|
||||||
|
|
||||||
|
## Inference with llama.cpp<a id="llamacpp"></a>
|
||||||
|
MiniCPM-Llama3-V 2.5 can run with llama.cpp now! See our fork of [llama.cpp](https://github.com/OpenBMB/llama.cpp/tree/minicpm-v2.5/examples/minicpmv) for more detail.
|
||||||
|
|
||||||
|
|
||||||
|
## Int4 quantized version
|
||||||
|
Download the int4 quantized version for lower GPU memory (8GB) usage: [MiniCPM-Llama3-V-2_5-int4](https://huggingface.co/openbmb/MiniCPM-Llama3-V-2_5-int4).
|
||||||
|
|
||||||
|
## MiniCPM-V 2.0 <!-- omit in toc -->
|
||||||
|
Please see the info about MiniCPM-V 2.0 [here](https://huggingface.co/openbmb/MiniCPM-V-2).
|
||||||
|
|
||||||
|
## License
|
||||||
|
#### Model License
|
||||||
|
* The code in this repo is released under the [Apache-2.0](https://github.com/OpenBMB/MiniCPM/blob/main/LICENSE) License.
|
||||||
|
* The usage of MiniCPM-V series model weights must strictly follow [MiniCPM Model License.md](https://github.com/OpenBMB/MiniCPM/blob/main/MiniCPM%20Model%20License.md).
|
||||||
|
* The models and weights of MiniCPM are completely free for academic research. after filling out a ["questionnaire"](https://modelbest.feishu.cn/share/base/form/shrcnpV5ZT9EJ6xYjh3Kx0J6v8g) for registration, are also available for free commercial use.
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
#### Statement
|
||||||
|
* As an LLM, MiniCPM-Llama3-V 2.5 generates contents by learning a large mount of texts, but it cannot comprehend, express personal opinions or make value judgement. Anything generated by MiniCPM-Llama3-V 2.5 does not represent the views and positions of the model developers
|
||||||
|
* We will not be liable for any problems arising from the use of the MinCPM-V open Source model, including but not limited to data security issues, risk of public opinion, or any risks and problems arising from the misdirection, misuse, dissemination or misuse of the model.
|
||||||
|
|
||||||
|
## Other Multimodal Projects from Our Team
|
||||||
|
|
||||||
|
[VisCPM](https://github.com/OpenBMB/VisCPM/tree/main) | [RLHF-V](https://github.com/RLHF-V/RLHF-V) | [LLaVA-UHD](https://github.com/thunlp/LLaVA-UHD) | [RLAIF-V](https://github.com/RLHF-V/RLAIF-V)
|
||||||
|
|
||||||
|
## Citation
|
||||||
|
|
||||||
|
If you find our work helpful, please consider citing the following papers
|
||||||
|
|
||||||
|
```bib
|
||||||
|
@article{yu2023rlhf,
|
||||||
|
title={Rlhf-v: Towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback},
|
||||||
|
author={Yu, Tianyu and Yao, Yuan and Zhang, Haoye and He, Taiwen and Han, Yifeng and Cui, Ganqu and Hu, Jinyi and Liu, Zhiyuan and Zheng, Hai-Tao and Sun, Maosong and others},
|
||||||
|
journal={arXiv preprint arXiv:2312.00849},
|
||||||
|
year={2023}
|
||||||
|
}
|
||||||
|
@article{viscpm,
|
||||||
|
title={Large Multilingual Models Pivot Zero-Shot Multimodal Learning across Languages},
|
||||||
|
author={Jinyi Hu and Yuan Yao and Chongyi Wang and Shan Wang and Yinxu Pan and Qianyu Chen and Tianyu Yu and Hanghao Wu and Yue Zhao and Haoye Zhang and Xu Han and Yankai Lin and Jiao Xue and Dahai Li and Zhiyuan Liu and Maosong Sun},
|
||||||
|
journal={arXiv preprint arXiv:2308.12038},
|
||||||
|
year={2023}
|
||||||
|
}
|
||||||
|
@article{xu2024llava-uhd,
|
||||||
|
title={{LLaVA-UHD}: an LMM Perceiving Any Aspect Ratio and High-Resolution Images},
|
||||||
|
author={Xu, Ruyi and Yao, Yuan and Guo, Zonghao and Cui, Junbo and Ni, Zanlin and Ge, Chunjiang and Chua, Tat-Seng and Liu, Zhiyuan and Huang, Gao},
|
||||||
|
journal={arXiv preprint arXiv:2403.11703},
|
||||||
|
year={2024}
|
||||||
|
}
|
||||||
|
@article{yu2024rlaifv,
|
||||||
|
title={RLAIF-V: Aligning MLLMs through Open-Source AI Feedback for Super GPT-4V Trustworthiness},
|
||||||
|
author={Yu, Tianyu and Zhang, Haoye and Yao, Yuan and Dang, Yunkai and Chen, Da and Lu, Xiaoman and Cui, Ganqu and He, Taiwen and Liu, Zhiyuan and Chua, Tat-Seng and Sun, Maosong},
|
||||||
|
journal={arXiv preprint arXiv:2405.17220},
|
||||||
|
year={2024},
|
||||||
|
}
|
||||||
|
```
|
||||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
|
|
@ -0,0 +1,54 @@
|
||||||
|
{
|
||||||
|
"_name_or_path": "openbmb/MiniCPM-Llama3-V-2_5",
|
||||||
|
"architectures": [
|
||||||
|
"MiniCPMV"
|
||||||
|
],
|
||||||
|
"attention_bias": false,
|
||||||
|
"attention_dropout": 0.0,
|
||||||
|
"auto_map": {
|
||||||
|
"AutoConfig": "configuration_minicpm.MiniCPMVConfig",
|
||||||
|
"AutoModel": "modeling_minicpmv.MiniCPMV",
|
||||||
|
"AutoModelForCausalLM": "modeling_minicpmv.MiniCPMV"
|
||||||
|
},
|
||||||
|
"batch_vision_input": true,
|
||||||
|
"bos_token_id": 128000,
|
||||||
|
"drop_vision_last_layer": false,
|
||||||
|
"eos_token_id": 128001,
|
||||||
|
"hidden_act": "silu",
|
||||||
|
"hidden_size": 4096,
|
||||||
|
"image_size": 448,
|
||||||
|
"initializer_range": 0.02,
|
||||||
|
"intermediate_size": 14336,
|
||||||
|
"max_position_embeddings": 8192,
|
||||||
|
"mm_use_im_start_end": true,
|
||||||
|
"model_type": "minicpmv",
|
||||||
|
"num_attention_heads": 32,
|
||||||
|
"num_hidden_layers": 32,
|
||||||
|
"num_key_value_heads": 8,
|
||||||
|
"patch_size": 14,
|
||||||
|
"pretraining_tp": 1,
|
||||||
|
"query_num": 96,
|
||||||
|
"rms_norm_eps": 1e-05,
|
||||||
|
"rope_scaling": null,
|
||||||
|
"rope_theta": 500000.0,
|
||||||
|
"slice_config": {
|
||||||
|
"max_slice_nums": 9,
|
||||||
|
"patch_size": 14,
|
||||||
|
"model_type": "minicpmv"
|
||||||
|
},
|
||||||
|
"slice_mode": true,
|
||||||
|
"tie_word_embeddings": false,
|
||||||
|
"torch_dtype": "float16",
|
||||||
|
"transformers_version": "4.40.0",
|
||||||
|
"use_cache": false,
|
||||||
|
"vision_config": {
|
||||||
|
"hidden_size": 1152,
|
||||||
|
"image_size": 980,
|
||||||
|
"intermediate_size": 4304,
|
||||||
|
"model_type": "idefics2",
|
||||||
|
"num_attention_heads": 16,
|
||||||
|
"num_hidden_layers": 27,
|
||||||
|
"patch_size": 14
|
||||||
|
},
|
||||||
|
"vocab_size": 128256
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1,113 @@
|
||||||
|
# coding=utf-8
|
||||||
|
# Copyright 2022 EleutherAI and the HuggingFace Inc. team. All rights reserved.
|
||||||
|
#
|
||||||
|
# This code is based on EleutherAI's GPT-NeoX library and the GPT-NeoX
|
||||||
|
# and OPT implementations in this library. It has been modified from its
|
||||||
|
# original forms to accommodate minor architectural differences compared
|
||||||
|
# to GPT-NeoX and OPT used by the Meta AI team that trained the model.
|
||||||
|
#
|
||||||
|
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
# you may not use this file except in compliance with the License.
|
||||||
|
# You may obtain a copy of the License at
|
||||||
|
#
|
||||||
|
# http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
#
|
||||||
|
# Unless required by applicable law or agreed to in writing, software
|
||||||
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
# See the License for the specific language governing permissions and
|
||||||
|
# limitations under the License.
|
||||||
|
""" MiniCPM model configuration"""
|
||||||
|
import os
|
||||||
|
from typing import Union
|
||||||
|
|
||||||
|
from transformers.utils import logging
|
||||||
|
from transformers import LlamaConfig, PretrainedConfig
|
||||||
|
from transformers.models.idefics2.modeling_idefics2 import Idefics2VisionConfig
|
||||||
|
|
||||||
|
logger = logging.get_logger(__name__)
|
||||||
|
|
||||||
|
|
||||||
|
class MiniCPMVSliceConfig(PretrainedConfig):
|
||||||
|
model_type = "minicpmv"
|
||||||
|
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
patch_size=14,
|
||||||
|
max_slice_nums=9,
|
||||||
|
scale_resolution=448,
|
||||||
|
**kwargs,
|
||||||
|
):
|
||||||
|
super().__init__(**kwargs)
|
||||||
|
self.patch_size = patch_size
|
||||||
|
self.max_slice_nums = max_slice_nums
|
||||||
|
self.scale_resolution = scale_resolution
|
||||||
|
|
||||||
|
@classmethod
|
||||||
|
def from_pretrained(cls, pretrained_model_name_or_path: Union[str, os.PathLike], **kwargs) -> "PretrainedConfig":
|
||||||
|
cls._set_token_in_kwargs(kwargs)
|
||||||
|
|
||||||
|
config_dict, kwargs = cls.get_config_dict(pretrained_model_name_or_path, **kwargs)
|
||||||
|
|
||||||
|
if config_dict.get("model_type") == "minicpmv":
|
||||||
|
config_dict = config_dict["slice_config"]
|
||||||
|
|
||||||
|
if "model_type" in config_dict and hasattr(cls, "model_type") and config_dict["model_type"] != cls.model_type:
|
||||||
|
logger.warning(
|
||||||
|
f"You are using a model of type {config_dict['model_type']} to instantiate a model of type "
|
||||||
|
f"{cls.model_type}. This is not supported for all configurations of models and can yield errors."
|
||||||
|
)
|
||||||
|
|
||||||
|
return cls.from_dict(config_dict, **kwargs)
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
class MiniCPMVConfig(LlamaConfig):
|
||||||
|
model_type = "minicpmv"
|
||||||
|
keys_to_ignore_at_inference = ["past_key_values"]
|
||||||
|
|
||||||
|
default_vision_config = {
|
||||||
|
"hidden_size": 1152,
|
||||||
|
"image_size": 980,
|
||||||
|
"intermediate_size": 4304,
|
||||||
|
"model_type": "idefics2",
|
||||||
|
"num_attention_heads": 16,
|
||||||
|
"num_hidden_layers": 27,
|
||||||
|
"patch_size": 14,
|
||||||
|
}
|
||||||
|
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
use_cache=True,
|
||||||
|
query_num=64,
|
||||||
|
image_size=448,
|
||||||
|
drop_vision_last_layer=True,
|
||||||
|
batch_vision_input=True,
|
||||||
|
slice_config=None,
|
||||||
|
vision_config=None,
|
||||||
|
**kwargs,
|
||||||
|
):
|
||||||
|
self.use_cache = use_cache
|
||||||
|
self.query_num = query_num
|
||||||
|
self.image_size = image_size
|
||||||
|
self.drop_vision_last_layer = drop_vision_last_layer
|
||||||
|
self.batch_vision_input = batch_vision_input
|
||||||
|
|
||||||
|
if slice_config is None:
|
||||||
|
self.slice_config = MiniCPMVSliceConfig(max_slice_nums=1)
|
||||||
|
else:
|
||||||
|
self.slice_config = MiniCPMVSliceConfig(**slice_config)
|
||||||
|
self.slice_mode = True
|
||||||
|
|
||||||
|
# same as HuggingFaceM4/siglip-so400m-14-980-flash-attn2-navit
|
||||||
|
if vision_config is None:
|
||||||
|
self.vision_config = Idefics2VisionConfig(**self.default_vision_config)
|
||||||
|
logger.info("vision_config is None, using default vision config")
|
||||||
|
elif isinstance(vision_config, dict):
|
||||||
|
self.vision_config = Idefics2VisionConfig(**vision_config)
|
||||||
|
elif isinstance(vision_config, Idefics2VisionConfig):
|
||||||
|
self.vision_config = vision_config
|
||||||
|
|
||||||
|
self.patch_size = self.vision_config.patch_size
|
||||||
|
|
||||||
|
super().__init__(**kwargs)
|
||||||
|
|
@ -0,0 +1,6 @@
|
||||||
|
{
|
||||||
|
"_from_model_config": true,
|
||||||
|
"bos_token_id": 128000,
|
||||||
|
"eos_token_id": 128001,
|
||||||
|
"transformers_version": "4.40.0"
|
||||||
|
}
|
||||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
|
|
@ -0,0 +1,748 @@
|
||||||
|
{
|
||||||
|
"metadata": {
|
||||||
|
"total_size": 34148369344
|
||||||
|
},
|
||||||
|
"weight_map": {
|
||||||
|
"llm.lm_head.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"llm.model.embed_tokens.weight": "model-00001-of-00007.safetensors",
|
||||||
|
"llm.model.layers.0.input_layernorm.weight": "model-00001-of-00007.safetensors",
|
||||||
|
"llm.model.layers.0.mlp.down_proj.weight": "model-00001-of-00007.safetensors",
|
||||||
|
"llm.model.layers.0.mlp.gate_proj.weight": "model-00001-of-00007.safetensors",
|
||||||
|
"llm.model.layers.0.mlp.up_proj.weight": "model-00001-of-00007.safetensors",
|
||||||
|
"llm.model.layers.0.post_attention_layernorm.weight": "model-00001-of-00007.safetensors",
|
||||||
|
"llm.model.layers.0.self_attn.k_proj.weight": "model-00001-of-00007.safetensors",
|
||||||
|
"llm.model.layers.0.self_attn.o_proj.weight": "model-00001-of-00007.safetensors",
|
||||||
|
"llm.model.layers.0.self_attn.q_proj.weight": "model-00001-of-00007.safetensors",
|
||||||
|
"llm.model.layers.0.self_attn.v_proj.weight": "model-00001-of-00007.safetensors",
|
||||||
|
"llm.model.layers.1.input_layernorm.weight": "model-00001-of-00007.safetensors",
|
||||||
|
"llm.model.layers.1.mlp.down_proj.weight": "model-00001-of-00007.safetensors",
|
||||||
|
"llm.model.layers.1.mlp.gate_proj.weight": "model-00001-of-00007.safetensors",
|
||||||
|
"llm.model.layers.1.mlp.up_proj.weight": "model-00001-of-00007.safetensors",
|
||||||
|
"llm.model.layers.1.post_attention_layernorm.weight": "model-00001-of-00007.safetensors",
|
||||||
|
"llm.model.layers.1.self_attn.k_proj.weight": "model-00001-of-00007.safetensors",
|
||||||
|
"llm.model.layers.1.self_attn.o_proj.weight": "model-00001-of-00007.safetensors",
|
||||||
|
"llm.model.layers.1.self_attn.q_proj.weight": "model-00001-of-00007.safetensors",
|
||||||
|
"llm.model.layers.1.self_attn.v_proj.weight": "model-00001-of-00007.safetensors",
|
||||||
|
"llm.model.layers.10.input_layernorm.weight": "model-00003-of-00007.safetensors",
|
||||||
|
"llm.model.layers.10.mlp.down_proj.weight": "model-00003-of-00007.safetensors",
|
||||||
|
"llm.model.layers.10.mlp.gate_proj.weight": "model-00003-of-00007.safetensors",
|
||||||
|
"llm.model.layers.10.mlp.up_proj.weight": "model-00003-of-00007.safetensors",
|
||||||
|
"llm.model.layers.10.post_attention_layernorm.weight": "model-00003-of-00007.safetensors",
|
||||||
|
"llm.model.layers.10.self_attn.k_proj.weight": "model-00003-of-00007.safetensors",
|
||||||
|
"llm.model.layers.10.self_attn.o_proj.weight": "model-00003-of-00007.safetensors",
|
||||||
|
"llm.model.layers.10.self_attn.q_proj.weight": "model-00003-of-00007.safetensors",
|
||||||
|
"llm.model.layers.10.self_attn.v_proj.weight": "model-00003-of-00007.safetensors",
|
||||||
|
"llm.model.layers.11.input_layernorm.weight": "model-00003-of-00007.safetensors",
|
||||||
|
"llm.model.layers.11.mlp.down_proj.weight": "model-00003-of-00007.safetensors",
|
||||||
|
"llm.model.layers.11.mlp.gate_proj.weight": "model-00003-of-00007.safetensors",
|
||||||
|
"llm.model.layers.11.mlp.up_proj.weight": "model-00003-of-00007.safetensors",
|
||||||
|
"llm.model.layers.11.post_attention_layernorm.weight": "model-00003-of-00007.safetensors",
|
||||||
|
"llm.model.layers.11.self_attn.k_proj.weight": "model-00003-of-00007.safetensors",
|
||||||
|
"llm.model.layers.11.self_attn.o_proj.weight": "model-00003-of-00007.safetensors",
|
||||||
|
"llm.model.layers.11.self_attn.q_proj.weight": "model-00003-of-00007.safetensors",
|
||||||
|
"llm.model.layers.11.self_attn.v_proj.weight": "model-00003-of-00007.safetensors",
|
||||||
|
"llm.model.layers.12.input_layernorm.weight": "model-00003-of-00007.safetensors",
|
||||||
|
"llm.model.layers.12.mlp.down_proj.weight": "model-00003-of-00007.safetensors",
|
||||||
|
"llm.model.layers.12.mlp.gate_proj.weight": "model-00003-of-00007.safetensors",
|
||||||
|
"llm.model.layers.12.mlp.up_proj.weight": "model-00003-of-00007.safetensors",
|
||||||
|
"llm.model.layers.12.post_attention_layernorm.weight": "model-00003-of-00007.safetensors",
|
||||||
|
"llm.model.layers.12.self_attn.k_proj.weight": "model-00003-of-00007.safetensors",
|
||||||
|
"llm.model.layers.12.self_attn.o_proj.weight": "model-00003-of-00007.safetensors",
|
||||||
|
"llm.model.layers.12.self_attn.q_proj.weight": "model-00003-of-00007.safetensors",
|
||||||
|
"llm.model.layers.12.self_attn.v_proj.weight": "model-00003-of-00007.safetensors",
|
||||||
|
"llm.model.layers.13.input_layernorm.weight": "model-00003-of-00007.safetensors",
|
||||||
|
"llm.model.layers.13.mlp.down_proj.weight": "model-00003-of-00007.safetensors",
|
||||||
|
"llm.model.layers.13.mlp.gate_proj.weight": "model-00003-of-00007.safetensors",
|
||||||
|
"llm.model.layers.13.mlp.up_proj.weight": "model-00003-of-00007.safetensors",
|
||||||
|
"llm.model.layers.13.post_attention_layernorm.weight": "model-00003-of-00007.safetensors",
|
||||||
|
"llm.model.layers.13.self_attn.k_proj.weight": "model-00003-of-00007.safetensors",
|
||||||
|
"llm.model.layers.13.self_attn.o_proj.weight": "model-00003-of-00007.safetensors",
|
||||||
|
"llm.model.layers.13.self_attn.q_proj.weight": "model-00003-of-00007.safetensors",
|
||||||
|
"llm.model.layers.13.self_attn.v_proj.weight": "model-00003-of-00007.safetensors",
|
||||||
|
"llm.model.layers.14.input_layernorm.weight": "model-00004-of-00007.safetensors",
|
||||||
|
"llm.model.layers.14.mlp.down_proj.weight": "model-00004-of-00007.safetensors",
|
||||||
|
"llm.model.layers.14.mlp.gate_proj.weight": "model-00003-of-00007.safetensors",
|
||||||
|
"llm.model.layers.14.mlp.up_proj.weight": "model-00004-of-00007.safetensors",
|
||||||
|
"llm.model.layers.14.post_attention_layernorm.weight": "model-00004-of-00007.safetensors",
|
||||||
|
"llm.model.layers.14.self_attn.k_proj.weight": "model-00003-of-00007.safetensors",
|
||||||
|
"llm.model.layers.14.self_attn.o_proj.weight": "model-00003-of-00007.safetensors",
|
||||||
|
"llm.model.layers.14.self_attn.q_proj.weight": "model-00003-of-00007.safetensors",
|
||||||
|
"llm.model.layers.14.self_attn.v_proj.weight": "model-00003-of-00007.safetensors",
|
||||||
|
"llm.model.layers.15.input_layernorm.weight": "model-00004-of-00007.safetensors",
|
||||||
|
"llm.model.layers.15.mlp.down_proj.weight": "model-00004-of-00007.safetensors",
|
||||||
|
"llm.model.layers.15.mlp.gate_proj.weight": "model-00004-of-00007.safetensors",
|
||||||
|
"llm.model.layers.15.mlp.up_proj.weight": "model-00004-of-00007.safetensors",
|
||||||
|
"llm.model.layers.15.post_attention_layernorm.weight": "model-00004-of-00007.safetensors",
|
||||||
|
"llm.model.layers.15.self_attn.k_proj.weight": "model-00004-of-00007.safetensors",
|
||||||
|
"llm.model.layers.15.self_attn.o_proj.weight": "model-00004-of-00007.safetensors",
|
||||||
|
"llm.model.layers.15.self_attn.q_proj.weight": "model-00004-of-00007.safetensors",
|
||||||
|
"llm.model.layers.15.self_attn.v_proj.weight": "model-00004-of-00007.safetensors",
|
||||||
|
"llm.model.layers.16.input_layernorm.weight": "model-00004-of-00007.safetensors",
|
||||||
|
"llm.model.layers.16.mlp.down_proj.weight": "model-00004-of-00007.safetensors",
|
||||||
|
"llm.model.layers.16.mlp.gate_proj.weight": "model-00004-of-00007.safetensors",
|
||||||
|
"llm.model.layers.16.mlp.up_proj.weight": "model-00004-of-00007.safetensors",
|
||||||
|
"llm.model.layers.16.post_attention_layernorm.weight": "model-00004-of-00007.safetensors",
|
||||||
|
"llm.model.layers.16.self_attn.k_proj.weight": "model-00004-of-00007.safetensors",
|
||||||
|
"llm.model.layers.16.self_attn.o_proj.weight": "model-00004-of-00007.safetensors",
|
||||||
|
"llm.model.layers.16.self_attn.q_proj.weight": "model-00004-of-00007.safetensors",
|
||||||
|
"llm.model.layers.16.self_attn.v_proj.weight": "model-00004-of-00007.safetensors",
|
||||||
|
"llm.model.layers.17.input_layernorm.weight": "model-00004-of-00007.safetensors",
|
||||||
|
"llm.model.layers.17.mlp.down_proj.weight": "model-00004-of-00007.safetensors",
|
||||||
|
"llm.model.layers.17.mlp.gate_proj.weight": "model-00004-of-00007.safetensors",
|
||||||
|
"llm.model.layers.17.mlp.up_proj.weight": "model-00004-of-00007.safetensors",
|
||||||
|
"llm.model.layers.17.post_attention_layernorm.weight": "model-00004-of-00007.safetensors",
|
||||||
|
"llm.model.layers.17.self_attn.k_proj.weight": "model-00004-of-00007.safetensors",
|
||||||
|
"llm.model.layers.17.self_attn.o_proj.weight": "model-00004-of-00007.safetensors",
|
||||||
|
"llm.model.layers.17.self_attn.q_proj.weight": "model-00004-of-00007.safetensors",
|
||||||
|
"llm.model.layers.17.self_attn.v_proj.weight": "model-00004-of-00007.safetensors",
|
||||||
|
"llm.model.layers.18.input_layernorm.weight": "model-00004-of-00007.safetensors",
|
||||||
|
"llm.model.layers.18.mlp.down_proj.weight": "model-00004-of-00007.safetensors",
|
||||||
|
"llm.model.layers.18.mlp.gate_proj.weight": "model-00004-of-00007.safetensors",
|
||||||
|
"llm.model.layers.18.mlp.up_proj.weight": "model-00004-of-00007.safetensors",
|
||||||
|
"llm.model.layers.18.post_attention_layernorm.weight": "model-00004-of-00007.safetensors",
|
||||||
|
"llm.model.layers.18.self_attn.k_proj.weight": "model-00004-of-00007.safetensors",
|
||||||
|
"llm.model.layers.18.self_attn.o_proj.weight": "model-00004-of-00007.safetensors",
|
||||||
|
"llm.model.layers.18.self_attn.q_proj.weight": "model-00004-of-00007.safetensors",
|
||||||
|
"llm.model.layers.18.self_attn.v_proj.weight": "model-00004-of-00007.safetensors",
|
||||||
|
"llm.model.layers.19.input_layernorm.weight": "model-00004-of-00007.safetensors",
|
||||||
|
"llm.model.layers.19.mlp.down_proj.weight": "model-00004-of-00007.safetensors",
|
||||||
|
"llm.model.layers.19.mlp.gate_proj.weight": "model-00004-of-00007.safetensors",
|
||||||
|
"llm.model.layers.19.mlp.up_proj.weight": "model-00004-of-00007.safetensors",
|
||||||
|
"llm.model.layers.19.post_attention_layernorm.weight": "model-00004-of-00007.safetensors",
|
||||||
|
"llm.model.layers.19.self_attn.k_proj.weight": "model-00004-of-00007.safetensors",
|
||||||
|
"llm.model.layers.19.self_attn.o_proj.weight": "model-00004-of-00007.safetensors",
|
||||||
|
"llm.model.layers.19.self_attn.q_proj.weight": "model-00004-of-00007.safetensors",
|
||||||
|
"llm.model.layers.19.self_attn.v_proj.weight": "model-00004-of-00007.safetensors",
|
||||||
|
"llm.model.layers.2.input_layernorm.weight": "model-00001-of-00007.safetensors",
|
||||||
|
"llm.model.layers.2.mlp.down_proj.weight": "model-00001-of-00007.safetensors",
|
||||||
|
"llm.model.layers.2.mlp.gate_proj.weight": "model-00001-of-00007.safetensors",
|
||||||
|
"llm.model.layers.2.mlp.up_proj.weight": "model-00001-of-00007.safetensors",
|
||||||
|
"llm.model.layers.2.post_attention_layernorm.weight": "model-00001-of-00007.safetensors",
|
||||||
|
"llm.model.layers.2.self_attn.k_proj.weight": "model-00001-of-00007.safetensors",
|
||||||
|
"llm.model.layers.2.self_attn.o_proj.weight": "model-00001-of-00007.safetensors",
|
||||||
|
"llm.model.layers.2.self_attn.q_proj.weight": "model-00001-of-00007.safetensors",
|
||||||
|
"llm.model.layers.2.self_attn.v_proj.weight": "model-00001-of-00007.safetensors",
|
||||||
|
"llm.model.layers.20.input_layernorm.weight": "model-00005-of-00007.safetensors",
|
||||||
|
"llm.model.layers.20.mlp.down_proj.weight": "model-00005-of-00007.safetensors",
|
||||||
|
"llm.model.layers.20.mlp.gate_proj.weight": "model-00005-of-00007.safetensors",
|
||||||
|
"llm.model.layers.20.mlp.up_proj.weight": "model-00005-of-00007.safetensors",
|
||||||
|
"llm.model.layers.20.post_attention_layernorm.weight": "model-00005-of-00007.safetensors",
|
||||||
|
"llm.model.layers.20.self_attn.k_proj.weight": "model-00004-of-00007.safetensors",
|
||||||
|
"llm.model.layers.20.self_attn.o_proj.weight": "model-00004-of-00007.safetensors",
|
||||||
|
"llm.model.layers.20.self_attn.q_proj.weight": "model-00004-of-00007.safetensors",
|
||||||
|
"llm.model.layers.20.self_attn.v_proj.weight": "model-00004-of-00007.safetensors",
|
||||||
|
"llm.model.layers.21.input_layernorm.weight": "model-00005-of-00007.safetensors",
|
||||||
|
"llm.model.layers.21.mlp.down_proj.weight": "model-00005-of-00007.safetensors",
|
||||||
|
"llm.model.layers.21.mlp.gate_proj.weight": "model-00005-of-00007.safetensors",
|
||||||
|
"llm.model.layers.21.mlp.up_proj.weight": "model-00005-of-00007.safetensors",
|
||||||
|
"llm.model.layers.21.post_attention_layernorm.weight": "model-00005-of-00007.safetensors",
|
||||||
|
"llm.model.layers.21.self_attn.k_proj.weight": "model-00005-of-00007.safetensors",
|
||||||
|
"llm.model.layers.21.self_attn.o_proj.weight": "model-00005-of-00007.safetensors",
|
||||||
|
"llm.model.layers.21.self_attn.q_proj.weight": "model-00005-of-00007.safetensors",
|
||||||
|
"llm.model.layers.21.self_attn.v_proj.weight": "model-00005-of-00007.safetensors",
|
||||||
|
"llm.model.layers.22.input_layernorm.weight": "model-00005-of-00007.safetensors",
|
||||||
|
"llm.model.layers.22.mlp.down_proj.weight": "model-00005-of-00007.safetensors",
|
||||||
|
"llm.model.layers.22.mlp.gate_proj.weight": "model-00005-of-00007.safetensors",
|
||||||
|
"llm.model.layers.22.mlp.up_proj.weight": "model-00005-of-00007.safetensors",
|
||||||
|
"llm.model.layers.22.post_attention_layernorm.weight": "model-00005-of-00007.safetensors",
|
||||||
|
"llm.model.layers.22.self_attn.k_proj.weight": "model-00005-of-00007.safetensors",
|
||||||
|
"llm.model.layers.22.self_attn.o_proj.weight": "model-00005-of-00007.safetensors",
|
||||||
|
"llm.model.layers.22.self_attn.q_proj.weight": "model-00005-of-00007.safetensors",
|
||||||
|
"llm.model.layers.22.self_attn.v_proj.weight": "model-00005-of-00007.safetensors",
|
||||||
|
"llm.model.layers.23.input_layernorm.weight": "model-00005-of-00007.safetensors",
|
||||||
|
"llm.model.layers.23.mlp.down_proj.weight": "model-00005-of-00007.safetensors",
|
||||||
|
"llm.model.layers.23.mlp.gate_proj.weight": "model-00005-of-00007.safetensors",
|
||||||
|
"llm.model.layers.23.mlp.up_proj.weight": "model-00005-of-00007.safetensors",
|
||||||
|
"llm.model.layers.23.post_attention_layernorm.weight": "model-00005-of-00007.safetensors",
|
||||||
|
"llm.model.layers.23.self_attn.k_proj.weight": "model-00005-of-00007.safetensors",
|
||||||
|
"llm.model.layers.23.self_attn.o_proj.weight": "model-00005-of-00007.safetensors",
|
||||||
|
"llm.model.layers.23.self_attn.q_proj.weight": "model-00005-of-00007.safetensors",
|
||||||
|
"llm.model.layers.23.self_attn.v_proj.weight": "model-00005-of-00007.safetensors",
|
||||||
|
"llm.model.layers.24.input_layernorm.weight": "model-00005-of-00007.safetensors",
|
||||||
|
"llm.model.layers.24.mlp.down_proj.weight": "model-00005-of-00007.safetensors",
|
||||||
|
"llm.model.layers.24.mlp.gate_proj.weight": "model-00005-of-00007.safetensors",
|
||||||
|
"llm.model.layers.24.mlp.up_proj.weight": "model-00005-of-00007.safetensors",
|
||||||
|
"llm.model.layers.24.post_attention_layernorm.weight": "model-00005-of-00007.safetensors",
|
||||||
|
"llm.model.layers.24.self_attn.k_proj.weight": "model-00005-of-00007.safetensors",
|
||||||
|
"llm.model.layers.24.self_attn.o_proj.weight": "model-00005-of-00007.safetensors",
|
||||||
|
"llm.model.layers.24.self_attn.q_proj.weight": "model-00005-of-00007.safetensors",
|
||||||
|
"llm.model.layers.24.self_attn.v_proj.weight": "model-00005-of-00007.safetensors",
|
||||||
|
"llm.model.layers.25.input_layernorm.weight": "model-00006-of-00007.safetensors",
|
||||||
|
"llm.model.layers.25.mlp.down_proj.weight": "model-00006-of-00007.safetensors",
|
||||||
|
"llm.model.layers.25.mlp.gate_proj.weight": "model-00005-of-00007.safetensors",
|
||||||
|
"llm.model.layers.25.mlp.up_proj.weight": "model-00005-of-00007.safetensors",
|
||||||
|
"llm.model.layers.25.post_attention_layernorm.weight": "model-00006-of-00007.safetensors",
|
||||||
|
"llm.model.layers.25.self_attn.k_proj.weight": "model-00005-of-00007.safetensors",
|
||||||
|
"llm.model.layers.25.self_attn.o_proj.weight": "model-00005-of-00007.safetensors",
|
||||||
|
"llm.model.layers.25.self_attn.q_proj.weight": "model-00005-of-00007.safetensors",
|
||||||
|
"llm.model.layers.25.self_attn.v_proj.weight": "model-00005-of-00007.safetensors",
|
||||||
|
"llm.model.layers.26.input_layernorm.weight": "model-00006-of-00007.safetensors",
|
||||||
|
"llm.model.layers.26.mlp.down_proj.weight": "model-00006-of-00007.safetensors",
|
||||||
|
"llm.model.layers.26.mlp.gate_proj.weight": "model-00006-of-00007.safetensors",
|
||||||
|
"llm.model.layers.26.mlp.up_proj.weight": "model-00006-of-00007.safetensors",
|
||||||
|
"llm.model.layers.26.post_attention_layernorm.weight": "model-00006-of-00007.safetensors",
|
||||||
|
"llm.model.layers.26.self_attn.k_proj.weight": "model-00006-of-00007.safetensors",
|
||||||
|
"llm.model.layers.26.self_attn.o_proj.weight": "model-00006-of-00007.safetensors",
|
||||||
|
"llm.model.layers.26.self_attn.q_proj.weight": "model-00006-of-00007.safetensors",
|
||||||
|
"llm.model.layers.26.self_attn.v_proj.weight": "model-00006-of-00007.safetensors",
|
||||||
|
"llm.model.layers.27.input_layernorm.weight": "model-00006-of-00007.safetensors",
|
||||||
|
"llm.model.layers.27.mlp.down_proj.weight": "model-00006-of-00007.safetensors",
|
||||||
|
"llm.model.layers.27.mlp.gate_proj.weight": "model-00006-of-00007.safetensors",
|
||||||
|
"llm.model.layers.27.mlp.up_proj.weight": "model-00006-of-00007.safetensors",
|
||||||
|
"llm.model.layers.27.post_attention_layernorm.weight": "model-00006-of-00007.safetensors",
|
||||||
|
"llm.model.layers.27.self_attn.k_proj.weight": "model-00006-of-00007.safetensors",
|
||||||
|
"llm.model.layers.27.self_attn.o_proj.weight": "model-00006-of-00007.safetensors",
|
||||||
|
"llm.model.layers.27.self_attn.q_proj.weight": "model-00006-of-00007.safetensors",
|
||||||
|
"llm.model.layers.27.self_attn.v_proj.weight": "model-00006-of-00007.safetensors",
|
||||||
|
"llm.model.layers.28.input_layernorm.weight": "model-00006-of-00007.safetensors",
|
||||||
|
"llm.model.layers.28.mlp.down_proj.weight": "model-00006-of-00007.safetensors",
|
||||||
|
"llm.model.layers.28.mlp.gate_proj.weight": "model-00006-of-00007.safetensors",
|
||||||
|
"llm.model.layers.28.mlp.up_proj.weight": "model-00006-of-00007.safetensors",
|
||||||
|
"llm.model.layers.28.post_attention_layernorm.weight": "model-00006-of-00007.safetensors",
|
||||||
|
"llm.model.layers.28.self_attn.k_proj.weight": "model-00006-of-00007.safetensors",
|
||||||
|
"llm.model.layers.28.self_attn.o_proj.weight": "model-00006-of-00007.safetensors",
|
||||||
|
"llm.model.layers.28.self_attn.q_proj.weight": "model-00006-of-00007.safetensors",
|
||||||
|
"llm.model.layers.28.self_attn.v_proj.weight": "model-00006-of-00007.safetensors",
|
||||||
|
"llm.model.layers.29.input_layernorm.weight": "model-00006-of-00007.safetensors",
|
||||||
|
"llm.model.layers.29.mlp.down_proj.weight": "model-00006-of-00007.safetensors",
|
||||||
|
"llm.model.layers.29.mlp.gate_proj.weight": "model-00006-of-00007.safetensors",
|
||||||
|
"llm.model.layers.29.mlp.up_proj.weight": "model-00006-of-00007.safetensors",
|
||||||
|
"llm.model.layers.29.post_attention_layernorm.weight": "model-00006-of-00007.safetensors",
|
||||||
|
"llm.model.layers.29.self_attn.k_proj.weight": "model-00006-of-00007.safetensors",
|
||||||
|
"llm.model.layers.29.self_attn.o_proj.weight": "model-00006-of-00007.safetensors",
|
||||||
|
"llm.model.layers.29.self_attn.q_proj.weight": "model-00006-of-00007.safetensors",
|
||||||
|
"llm.model.layers.29.self_attn.v_proj.weight": "model-00006-of-00007.safetensors",
|
||||||
|
"llm.model.layers.3.input_layernorm.weight": "model-00002-of-00007.safetensors",
|
||||||
|
"llm.model.layers.3.mlp.down_proj.weight": "model-00002-of-00007.safetensors",
|
||||||
|
"llm.model.layers.3.mlp.gate_proj.weight": "model-00002-of-00007.safetensors",
|
||||||
|
"llm.model.layers.3.mlp.up_proj.weight": "model-00002-of-00007.safetensors",
|
||||||
|
"llm.model.layers.3.post_attention_layernorm.weight": "model-00002-of-00007.safetensors",
|
||||||
|
"llm.model.layers.3.self_attn.k_proj.weight": "model-00001-of-00007.safetensors",
|
||||||
|
"llm.model.layers.3.self_attn.o_proj.weight": "model-00001-of-00007.safetensors",
|
||||||
|
"llm.model.layers.3.self_attn.q_proj.weight": "model-00001-of-00007.safetensors",
|
||||||
|
"llm.model.layers.3.self_attn.v_proj.weight": "model-00001-of-00007.safetensors",
|
||||||
|
"llm.model.layers.30.input_layernorm.weight": "model-00006-of-00007.safetensors",
|
||||||
|
"llm.model.layers.30.mlp.down_proj.weight": "model-00006-of-00007.safetensors",
|
||||||
|
"llm.model.layers.30.mlp.gate_proj.weight": "model-00006-of-00007.safetensors",
|
||||||
|
"llm.model.layers.30.mlp.up_proj.weight": "model-00006-of-00007.safetensors",
|
||||||
|
"llm.model.layers.30.post_attention_layernorm.weight": "model-00006-of-00007.safetensors",
|
||||||
|
"llm.model.layers.30.self_attn.k_proj.weight": "model-00006-of-00007.safetensors",
|
||||||
|
"llm.model.layers.30.self_attn.o_proj.weight": "model-00006-of-00007.safetensors",
|
||||||
|
"llm.model.layers.30.self_attn.q_proj.weight": "model-00006-of-00007.safetensors",
|
||||||
|
"llm.model.layers.30.self_attn.v_proj.weight": "model-00006-of-00007.safetensors",
|
||||||
|
"llm.model.layers.31.input_layernorm.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"llm.model.layers.31.mlp.down_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"llm.model.layers.31.mlp.gate_proj.weight": "model-00006-of-00007.safetensors",
|
||||||
|
"llm.model.layers.31.mlp.up_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"llm.model.layers.31.post_attention_layernorm.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"llm.model.layers.31.self_attn.k_proj.weight": "model-00006-of-00007.safetensors",
|
||||||
|
"llm.model.layers.31.self_attn.o_proj.weight": "model-00006-of-00007.safetensors",
|
||||||
|
"llm.model.layers.31.self_attn.q_proj.weight": "model-00006-of-00007.safetensors",
|
||||||
|
"llm.model.layers.31.self_attn.v_proj.weight": "model-00006-of-00007.safetensors",
|
||||||
|
"llm.model.layers.4.input_layernorm.weight": "model-00002-of-00007.safetensors",
|
||||||
|
"llm.model.layers.4.mlp.down_proj.weight": "model-00002-of-00007.safetensors",
|
||||||
|
"llm.model.layers.4.mlp.gate_proj.weight": "model-00002-of-00007.safetensors",
|
||||||
|
"llm.model.layers.4.mlp.up_proj.weight": "model-00002-of-00007.safetensors",
|
||||||
|
"llm.model.layers.4.post_attention_layernorm.weight": "model-00002-of-00007.safetensors",
|
||||||
|
"llm.model.layers.4.self_attn.k_proj.weight": "model-00002-of-00007.safetensors",
|
||||||
|
"llm.model.layers.4.self_attn.o_proj.weight": "model-00002-of-00007.safetensors",
|
||||||
|
"llm.model.layers.4.self_attn.q_proj.weight": "model-00002-of-00007.safetensors",
|
||||||
|
"llm.model.layers.4.self_attn.v_proj.weight": "model-00002-of-00007.safetensors",
|
||||||
|
"llm.model.layers.5.input_layernorm.weight": "model-00002-of-00007.safetensors",
|
||||||
|
"llm.model.layers.5.mlp.down_proj.weight": "model-00002-of-00007.safetensors",
|
||||||
|
"llm.model.layers.5.mlp.gate_proj.weight": "model-00002-of-00007.safetensors",
|
||||||
|
"llm.model.layers.5.mlp.up_proj.weight": "model-00002-of-00007.safetensors",
|
||||||
|
"llm.model.layers.5.post_attention_layernorm.weight": "model-00002-of-00007.safetensors",
|
||||||
|
"llm.model.layers.5.self_attn.k_proj.weight": "model-00002-of-00007.safetensors",
|
||||||
|
"llm.model.layers.5.self_attn.o_proj.weight": "model-00002-of-00007.safetensors",
|
||||||
|
"llm.model.layers.5.self_attn.q_proj.weight": "model-00002-of-00007.safetensors",
|
||||||
|
"llm.model.layers.5.self_attn.v_proj.weight": "model-00002-of-00007.safetensors",
|
||||||
|
"llm.model.layers.6.input_layernorm.weight": "model-00002-of-00007.safetensors",
|
||||||
|
"llm.model.layers.6.mlp.down_proj.weight": "model-00002-of-00007.safetensors",
|
||||||
|
"llm.model.layers.6.mlp.gate_proj.weight": "model-00002-of-00007.safetensors",
|
||||||
|
"llm.model.layers.6.mlp.up_proj.weight": "model-00002-of-00007.safetensors",
|
||||||
|
"llm.model.layers.6.post_attention_layernorm.weight": "model-00002-of-00007.safetensors",
|
||||||
|
"llm.model.layers.6.self_attn.k_proj.weight": "model-00002-of-00007.safetensors",
|
||||||
|
"llm.model.layers.6.self_attn.o_proj.weight": "model-00002-of-00007.safetensors",
|
||||||
|
"llm.model.layers.6.self_attn.q_proj.weight": "model-00002-of-00007.safetensors",
|
||||||
|
"llm.model.layers.6.self_attn.v_proj.weight": "model-00002-of-00007.safetensors",
|
||||||
|
"llm.model.layers.7.input_layernorm.weight": "model-00002-of-00007.safetensors",
|
||||||
|
"llm.model.layers.7.mlp.down_proj.weight": "model-00002-of-00007.safetensors",
|
||||||
|
"llm.model.layers.7.mlp.gate_proj.weight": "model-00002-of-00007.safetensors",
|
||||||
|
"llm.model.layers.7.mlp.up_proj.weight": "model-00002-of-00007.safetensors",
|
||||||
|
"llm.model.layers.7.post_attention_layernorm.weight": "model-00002-of-00007.safetensors",
|
||||||
|
"llm.model.layers.7.self_attn.k_proj.weight": "model-00002-of-00007.safetensors",
|
||||||
|
"llm.model.layers.7.self_attn.o_proj.weight": "model-00002-of-00007.safetensors",
|
||||||
|
"llm.model.layers.7.self_attn.q_proj.weight": "model-00002-of-00007.safetensors",
|
||||||
|
"llm.model.layers.7.self_attn.v_proj.weight": "model-00002-of-00007.safetensors",
|
||||||
|
"llm.model.layers.8.input_layernorm.weight": "model-00003-of-00007.safetensors",
|
||||||
|
"llm.model.layers.8.mlp.down_proj.weight": "model-00003-of-00007.safetensors",
|
||||||
|
"llm.model.layers.8.mlp.gate_proj.weight": "model-00002-of-00007.safetensors",
|
||||||
|
"llm.model.layers.8.mlp.up_proj.weight": "model-00002-of-00007.safetensors",
|
||||||
|
"llm.model.layers.8.post_attention_layernorm.weight": "model-00003-of-00007.safetensors",
|
||||||
|
"llm.model.layers.8.self_attn.k_proj.weight": "model-00002-of-00007.safetensors",
|
||||||
|
"llm.model.layers.8.self_attn.o_proj.weight": "model-00002-of-00007.safetensors",
|
||||||
|
"llm.model.layers.8.self_attn.q_proj.weight": "model-00002-of-00007.safetensors",
|
||||||
|
"llm.model.layers.8.self_attn.v_proj.weight": "model-00002-of-00007.safetensors",
|
||||||
|
"llm.model.layers.9.input_layernorm.weight": "model-00003-of-00007.safetensors",
|
||||||
|
"llm.model.layers.9.mlp.down_proj.weight": "model-00003-of-00007.safetensors",
|
||||||
|
"llm.model.layers.9.mlp.gate_proj.weight": "model-00003-of-00007.safetensors",
|
||||||
|
"llm.model.layers.9.mlp.up_proj.weight": "model-00003-of-00007.safetensors",
|
||||||
|
"llm.model.layers.9.post_attention_layernorm.weight": "model-00003-of-00007.safetensors",
|
||||||
|
"llm.model.layers.9.self_attn.k_proj.weight": "model-00003-of-00007.safetensors",
|
||||||
|
"llm.model.layers.9.self_attn.o_proj.weight": "model-00003-of-00007.safetensors",
|
||||||
|
"llm.model.layers.9.self_attn.q_proj.weight": "model-00003-of-00007.safetensors",
|
||||||
|
"llm.model.layers.9.self_attn.v_proj.weight": "model-00003-of-00007.safetensors",
|
||||||
|
"llm.model.norm.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"resampler.attn.in_proj_bias": "model-00007-of-00007.safetensors",
|
||||||
|
"resampler.attn.in_proj_weight": "model-00007-of-00007.safetensors",
|
||||||
|
"resampler.attn.out_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"resampler.attn.out_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"resampler.kv_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"resampler.ln_kv.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"resampler.ln_kv.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"resampler.ln_post.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"resampler.ln_post.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"resampler.ln_q.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"resampler.ln_q.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"resampler.proj": "model-00007-of-00007.safetensors",
|
||||||
|
"resampler.query": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.embeddings.patch_embedding.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.embeddings.patch_embedding.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.embeddings.position_embedding.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.0.layer_norm1.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.0.layer_norm1.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.0.layer_norm2.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.0.layer_norm2.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.0.mlp.fc1.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.0.mlp.fc1.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.0.mlp.fc2.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.0.mlp.fc2.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.0.self_attn.k_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.0.self_attn.k_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.0.self_attn.out_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.0.self_attn.out_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.0.self_attn.q_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.0.self_attn.q_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.0.self_attn.v_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.0.self_attn.v_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.1.layer_norm1.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.1.layer_norm1.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.1.layer_norm2.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.1.layer_norm2.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.1.mlp.fc1.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.1.mlp.fc1.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.1.mlp.fc2.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.1.mlp.fc2.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.1.self_attn.k_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.1.self_attn.k_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.1.self_attn.out_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.1.self_attn.out_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.1.self_attn.q_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.1.self_attn.q_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.1.self_attn.v_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.1.self_attn.v_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.10.layer_norm1.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.10.layer_norm1.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.10.layer_norm2.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.10.layer_norm2.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.10.mlp.fc1.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.10.mlp.fc1.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.10.mlp.fc2.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.10.mlp.fc2.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.10.self_attn.k_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.10.self_attn.k_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.10.self_attn.out_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.10.self_attn.out_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.10.self_attn.q_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.10.self_attn.q_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.10.self_attn.v_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.10.self_attn.v_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.11.layer_norm1.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.11.layer_norm1.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.11.layer_norm2.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.11.layer_norm2.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.11.mlp.fc1.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.11.mlp.fc1.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.11.mlp.fc2.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.11.mlp.fc2.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.11.self_attn.k_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.11.self_attn.k_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.11.self_attn.out_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.11.self_attn.out_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.11.self_attn.q_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.11.self_attn.q_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.11.self_attn.v_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.11.self_attn.v_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.12.layer_norm1.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.12.layer_norm1.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.12.layer_norm2.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.12.layer_norm2.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.12.mlp.fc1.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.12.mlp.fc1.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.12.mlp.fc2.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.12.mlp.fc2.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.12.self_attn.k_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.12.self_attn.k_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.12.self_attn.out_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.12.self_attn.out_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.12.self_attn.q_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.12.self_attn.q_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.12.self_attn.v_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.12.self_attn.v_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.13.layer_norm1.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.13.layer_norm1.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.13.layer_norm2.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.13.layer_norm2.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.13.mlp.fc1.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.13.mlp.fc1.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.13.mlp.fc2.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.13.mlp.fc2.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.13.self_attn.k_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.13.self_attn.k_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.13.self_attn.out_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.13.self_attn.out_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.13.self_attn.q_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.13.self_attn.q_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.13.self_attn.v_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.13.self_attn.v_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.14.layer_norm1.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.14.layer_norm1.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.14.layer_norm2.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.14.layer_norm2.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.14.mlp.fc1.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.14.mlp.fc1.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.14.mlp.fc2.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.14.mlp.fc2.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.14.self_attn.k_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.14.self_attn.k_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.14.self_attn.out_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.14.self_attn.out_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.14.self_attn.q_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.14.self_attn.q_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.14.self_attn.v_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.14.self_attn.v_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.15.layer_norm1.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.15.layer_norm1.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.15.layer_norm2.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.15.layer_norm2.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.15.mlp.fc1.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.15.mlp.fc1.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.15.mlp.fc2.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.15.mlp.fc2.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.15.self_attn.k_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.15.self_attn.k_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.15.self_attn.out_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.15.self_attn.out_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.15.self_attn.q_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.15.self_attn.q_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.15.self_attn.v_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.15.self_attn.v_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.16.layer_norm1.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.16.layer_norm1.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.16.layer_norm2.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.16.layer_norm2.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.16.mlp.fc1.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.16.mlp.fc1.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.16.mlp.fc2.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.16.mlp.fc2.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.16.self_attn.k_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.16.self_attn.k_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.16.self_attn.out_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.16.self_attn.out_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.16.self_attn.q_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.16.self_attn.q_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.16.self_attn.v_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.16.self_attn.v_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.17.layer_norm1.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.17.layer_norm1.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.17.layer_norm2.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.17.layer_norm2.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.17.mlp.fc1.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.17.mlp.fc1.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.17.mlp.fc2.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.17.mlp.fc2.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.17.self_attn.k_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.17.self_attn.k_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.17.self_attn.out_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.17.self_attn.out_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.17.self_attn.q_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.17.self_attn.q_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.17.self_attn.v_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.17.self_attn.v_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.18.layer_norm1.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.18.layer_norm1.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.18.layer_norm2.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.18.layer_norm2.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.18.mlp.fc1.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.18.mlp.fc1.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.18.mlp.fc2.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.18.mlp.fc2.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.18.self_attn.k_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.18.self_attn.k_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.18.self_attn.out_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.18.self_attn.out_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.18.self_attn.q_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.18.self_attn.q_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.18.self_attn.v_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.18.self_attn.v_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.19.layer_norm1.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.19.layer_norm1.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.19.layer_norm2.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.19.layer_norm2.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.19.mlp.fc1.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.19.mlp.fc1.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.19.mlp.fc2.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.19.mlp.fc2.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.19.self_attn.k_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.19.self_attn.k_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.19.self_attn.out_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.19.self_attn.out_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.19.self_attn.q_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.19.self_attn.q_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.19.self_attn.v_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.19.self_attn.v_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.2.layer_norm1.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.2.layer_norm1.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.2.layer_norm2.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.2.layer_norm2.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.2.mlp.fc1.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.2.mlp.fc1.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.2.mlp.fc2.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.2.mlp.fc2.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.2.self_attn.k_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.2.self_attn.k_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.2.self_attn.out_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.2.self_attn.out_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.2.self_attn.q_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.2.self_attn.q_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.2.self_attn.v_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.2.self_attn.v_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.20.layer_norm1.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.20.layer_norm1.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.20.layer_norm2.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.20.layer_norm2.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.20.mlp.fc1.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.20.mlp.fc1.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.20.mlp.fc2.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.20.mlp.fc2.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.20.self_attn.k_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.20.self_attn.k_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.20.self_attn.out_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.20.self_attn.out_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.20.self_attn.q_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.20.self_attn.q_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.20.self_attn.v_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.20.self_attn.v_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.21.layer_norm1.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.21.layer_norm1.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.21.layer_norm2.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.21.layer_norm2.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.21.mlp.fc1.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.21.mlp.fc1.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.21.mlp.fc2.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.21.mlp.fc2.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.21.self_attn.k_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.21.self_attn.k_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.21.self_attn.out_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.21.self_attn.out_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.21.self_attn.q_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.21.self_attn.q_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.21.self_attn.v_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.21.self_attn.v_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.22.layer_norm1.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.22.layer_norm1.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.22.layer_norm2.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.22.layer_norm2.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.22.mlp.fc1.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.22.mlp.fc1.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.22.mlp.fc2.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.22.mlp.fc2.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.22.self_attn.k_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.22.self_attn.k_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.22.self_attn.out_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.22.self_attn.out_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.22.self_attn.q_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.22.self_attn.q_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.22.self_attn.v_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.22.self_attn.v_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.23.layer_norm1.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.23.layer_norm1.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.23.layer_norm2.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.23.layer_norm2.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.23.mlp.fc1.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.23.mlp.fc1.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.23.mlp.fc2.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.23.mlp.fc2.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.23.self_attn.k_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.23.self_attn.k_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.23.self_attn.out_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.23.self_attn.out_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.23.self_attn.q_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.23.self_attn.q_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.23.self_attn.v_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.23.self_attn.v_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.24.layer_norm1.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.24.layer_norm1.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.24.layer_norm2.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.24.layer_norm2.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.24.mlp.fc1.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.24.mlp.fc1.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.24.mlp.fc2.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.24.mlp.fc2.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.24.self_attn.k_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.24.self_attn.k_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.24.self_attn.out_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.24.self_attn.out_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.24.self_attn.q_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.24.self_attn.q_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.24.self_attn.v_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.24.self_attn.v_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.25.layer_norm1.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.25.layer_norm1.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.25.layer_norm2.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.25.layer_norm2.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.25.mlp.fc1.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.25.mlp.fc1.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.25.mlp.fc2.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.25.mlp.fc2.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.25.self_attn.k_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.25.self_attn.k_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.25.self_attn.out_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.25.self_attn.out_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.25.self_attn.q_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.25.self_attn.q_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.25.self_attn.v_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.25.self_attn.v_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.26.layer_norm1.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.26.layer_norm1.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.26.layer_norm2.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.26.layer_norm2.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.26.mlp.fc1.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.26.mlp.fc1.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.26.mlp.fc2.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.26.mlp.fc2.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.26.self_attn.k_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.26.self_attn.k_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.26.self_attn.out_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.26.self_attn.out_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.26.self_attn.q_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.26.self_attn.q_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.26.self_attn.v_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.26.self_attn.v_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.3.layer_norm1.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.3.layer_norm1.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.3.layer_norm2.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.3.layer_norm2.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.3.mlp.fc1.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.3.mlp.fc1.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.3.mlp.fc2.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.3.mlp.fc2.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.3.self_attn.k_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.3.self_attn.k_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.3.self_attn.out_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.3.self_attn.out_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.3.self_attn.q_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.3.self_attn.q_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.3.self_attn.v_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.3.self_attn.v_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.4.layer_norm1.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.4.layer_norm1.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.4.layer_norm2.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.4.layer_norm2.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.4.mlp.fc1.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.4.mlp.fc1.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.4.mlp.fc2.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.4.mlp.fc2.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.4.self_attn.k_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.4.self_attn.k_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.4.self_attn.out_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.4.self_attn.out_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.4.self_attn.q_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.4.self_attn.q_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.4.self_attn.v_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.4.self_attn.v_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.5.layer_norm1.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.5.layer_norm1.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.5.layer_norm2.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.5.layer_norm2.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.5.mlp.fc1.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.5.mlp.fc1.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.5.mlp.fc2.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.5.mlp.fc2.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.5.self_attn.k_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.5.self_attn.k_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.5.self_attn.out_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.5.self_attn.out_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.5.self_attn.q_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.5.self_attn.q_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.5.self_attn.v_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.5.self_attn.v_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.6.layer_norm1.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.6.layer_norm1.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.6.layer_norm2.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.6.layer_norm2.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.6.mlp.fc1.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.6.mlp.fc1.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.6.mlp.fc2.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.6.mlp.fc2.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.6.self_attn.k_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.6.self_attn.k_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.6.self_attn.out_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.6.self_attn.out_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.6.self_attn.q_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.6.self_attn.q_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.6.self_attn.v_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.6.self_attn.v_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.7.layer_norm1.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.7.layer_norm1.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.7.layer_norm2.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.7.layer_norm2.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.7.mlp.fc1.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.7.mlp.fc1.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.7.mlp.fc2.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.7.mlp.fc2.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.7.self_attn.k_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.7.self_attn.k_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.7.self_attn.out_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.7.self_attn.out_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.7.self_attn.q_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.7.self_attn.q_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.7.self_attn.v_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.7.self_attn.v_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.8.layer_norm1.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.8.layer_norm1.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.8.layer_norm2.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.8.layer_norm2.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.8.mlp.fc1.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.8.mlp.fc1.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.8.mlp.fc2.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.8.mlp.fc2.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.8.self_attn.k_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.8.self_attn.k_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.8.self_attn.out_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.8.self_attn.out_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.8.self_attn.q_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.8.self_attn.q_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.8.self_attn.v_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.8.self_attn.v_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.9.layer_norm1.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.9.layer_norm1.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.9.layer_norm2.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.9.layer_norm2.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.9.mlp.fc1.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.9.mlp.fc1.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.9.mlp.fc2.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.9.mlp.fc2.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.9.self_attn.k_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.9.self_attn.k_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.9.self_attn.out_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.9.self_attn.out_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.9.self_attn.q_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.9.self_attn.q_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.9.self_attn.v_proj.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.encoder.layers.9.self_attn.v_proj.weight": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.post_layernorm.bias": "model-00007-of-00007.safetensors",
|
||||||
|
"vpm.post_layernorm.weight": "model-00007-of-00007.safetensors"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1,696 @@
|
||||||
|
import math
|
||||||
|
from typing import List, Optional
|
||||||
|
import json
|
||||||
|
import torch
|
||||||
|
import torchvision
|
||||||
|
from threading import Thread
|
||||||
|
from copy import deepcopy
|
||||||
|
from PIL import Image
|
||||||
|
from torchvision import transforms
|
||||||
|
from transformers import LlamaTokenizer, LlamaPreTrainedModel, LlamaForCausalLM, AutoModel, PreTrainedTokenizerFast, TextIteratorStreamer
|
||||||
|
from transformers.models.idefics2.modeling_idefics2 import Idefics2VisionTransformer
|
||||||
|
|
||||||
|
from .configuration_minicpm import MiniCPMVConfig
|
||||||
|
from .resampler import Resampler
|
||||||
|
|
||||||
|
IMAGENET_INCEPTION_MEAN = (0.5, 0.5, 0.5) # timm.data.IMAGENET_INCEPTION_MEAN
|
||||||
|
IMAGENET_INCEPTION_STD = (0.5, 0.5, 0.5) # timm.data.IMAGENET_INCEPTION_STD
|
||||||
|
|
||||||
|
class MiniCPMVPreTrainedModel(LlamaPreTrainedModel):
|
||||||
|
config_class = MiniCPMVConfig
|
||||||
|
|
||||||
|
|
||||||
|
class MiniCPMV(MiniCPMVPreTrainedModel):
|
||||||
|
def __init__(self, config):
|
||||||
|
super().__init__(config)
|
||||||
|
|
||||||
|
self.llm = LlamaForCausalLM(config)
|
||||||
|
self.vpm = self.init_vision_module()
|
||||||
|
self.vision_dim = self.vpm.embed_dim
|
||||||
|
self.embed_dim = self.llm.config.hidden_size
|
||||||
|
self.resampler = self.init_resampler(self.embed_dim, self.vision_dim)
|
||||||
|
self.transform = self.init_transform()
|
||||||
|
|
||||||
|
def init_vision_module(self):
|
||||||
|
# same as HuggingFaceM4/siglip-so400m-14-980-flash-attn2-navit
|
||||||
|
model = Idefics2VisionTransformer(self.config.vision_config)
|
||||||
|
if self.config.drop_vision_last_layer:
|
||||||
|
model.encoder.layers = model.encoder.layers[:-1]
|
||||||
|
|
||||||
|
setattr(model, 'embed_dim', model.embeddings.embed_dim)
|
||||||
|
setattr(model, 'patch_size', model.embeddings.patch_size)
|
||||||
|
|
||||||
|
return model
|
||||||
|
|
||||||
|
def init_resampler(self, embed_dim, vision_dim):
|
||||||
|
return Resampler(
|
||||||
|
num_queries=self.config.query_num,
|
||||||
|
embed_dim=embed_dim,
|
||||||
|
num_heads=embed_dim // 128,
|
||||||
|
kv_dim=vision_dim,
|
||||||
|
adaptive=True
|
||||||
|
)
|
||||||
|
|
||||||
|
def init_transform(self):
|
||||||
|
return transforms.Compose(
|
||||||
|
[
|
||||||
|
transforms.ToTensor(),
|
||||||
|
transforms.Normalize(
|
||||||
|
mean=IMAGENET_INCEPTION_MEAN, std=IMAGENET_INCEPTION_STD
|
||||||
|
),
|
||||||
|
]
|
||||||
|
)
|
||||||
|
|
||||||
|
def get_vllm_embedding(self, data):
|
||||||
|
if 'vision_hidden_states' not in data:
|
||||||
|
dtype = self.vpm.embeddings.position_embedding.weight.dtype
|
||||||
|
device = self.vpm.embeddings.position_embedding.weight.device
|
||||||
|
tgt_sizes = data['tgt_sizes']
|
||||||
|
pixel_values_list = data['pixel_values']
|
||||||
|
vision_hidden_states = []
|
||||||
|
all_pixel_values = []
|
||||||
|
img_cnt = []
|
||||||
|
for pixel_values in pixel_values_list:
|
||||||
|
img_cnt.append(len(pixel_values))
|
||||||
|
all_pixel_values.extend([i.flatten(end_dim=1).permute(1, 0) for i in pixel_values])
|
||||||
|
|
||||||
|
# exist image
|
||||||
|
if all_pixel_values:
|
||||||
|
tgt_sizes = torch.vstack(tgt_sizes).type(torch.int32)
|
||||||
|
|
||||||
|
if self.config.batch_vision_input:
|
||||||
|
max_patches = torch.max(tgt_sizes[:, 0] * tgt_sizes[:, 1])
|
||||||
|
|
||||||
|
all_pixel_values = torch.nn.utils.rnn.pad_sequence(all_pixel_values, batch_first=True,
|
||||||
|
padding_value=0.0)
|
||||||
|
B, L, _ = all_pixel_values.shape
|
||||||
|
all_pixel_values = all_pixel_values.permute(0, 2, 1).reshape(B, 3, -1, L)
|
||||||
|
|
||||||
|
patch_attn_mask = torch.zeros((B, 1, max_patches), dtype=torch.bool, device=device)
|
||||||
|
for i in range(B):
|
||||||
|
patch_attn_mask[i, :tgt_sizes[i][0] * tgt_sizes[i][1]] = True
|
||||||
|
|
||||||
|
vision_embedding = self.vpm(all_pixel_values.type(dtype), patch_attention_mask=patch_attn_mask).last_hidden_state
|
||||||
|
vision_embedding = self.resampler(vision_embedding, tgt_sizes)
|
||||||
|
else:
|
||||||
|
# get vision_embedding foreach
|
||||||
|
vision_embedding = []
|
||||||
|
for single_tgt_size, single_pixel_values in zip(tgt_sizes, all_pixel_values):
|
||||||
|
single_pixel_values = single_pixel_values.unsqueeze(0)
|
||||||
|
B, L, _ = single_pixel_values.shape
|
||||||
|
single_pixel_values = single_pixel_values.permute(0, 2, 1).reshape(B, 3, -1, L)
|
||||||
|
single_vision_embedding = self.vpm(single_pixel_values.type(dtype)).last_hidden_state
|
||||||
|
single_vision_embedding = self.resampler(single_vision_embedding, single_tgt_size.unsqueeze(0))
|
||||||
|
vision_embedding.append(single_vision_embedding)
|
||||||
|
vision_embedding = torch.vstack(vision_embedding)
|
||||||
|
|
||||||
|
start = 0
|
||||||
|
for pixel_values in pixel_values_list:
|
||||||
|
img_cnt = len(pixel_values)
|
||||||
|
if img_cnt > 0:
|
||||||
|
vision_hidden_states.append(vision_embedding[start: start + img_cnt])
|
||||||
|
start += img_cnt
|
||||||
|
else:
|
||||||
|
vision_hidden_states.append([])
|
||||||
|
else: # no image
|
||||||
|
if self.training:
|
||||||
|
dummy_image = torch.zeros(
|
||||||
|
(1, 3, 224, 224),
|
||||||
|
device=device, dtype=dtype
|
||||||
|
)
|
||||||
|
tgt_sizes = torch.Tensor([[(224 // self.config.patch_size), math.ceil(224 / self.config.patch_size)]]).type(torch.int32)
|
||||||
|
dummy_feature = self.resampler(self.vpm(dummy_image).last_hidden_state, tgt_sizes)
|
||||||
|
else:
|
||||||
|
dummy_feature = []
|
||||||
|
for _ in range(len(pixel_values_list)):
|
||||||
|
vision_hidden_states.append(dummy_feature)
|
||||||
|
|
||||||
|
else:
|
||||||
|
vision_hidden_states = data['vision_hidden_states']
|
||||||
|
|
||||||
|
if hasattr(self.llm.config, 'scale_emb'):
|
||||||
|
vllm_embedding = self.llm.model.embed_tokens(data['input_ids']) * self.llm.config.scale_emb
|
||||||
|
else:
|
||||||
|
vllm_embedding = self.llm.model.embed_tokens(data['input_ids'])
|
||||||
|
|
||||||
|
vision_hidden_states = [i.type(vllm_embedding.dtype) if isinstance(
|
||||||
|
i, torch.Tensor) else i for i in vision_hidden_states]
|
||||||
|
|
||||||
|
bs = len(data['input_ids'])
|
||||||
|
for i in range(bs):
|
||||||
|
cur_vs_hs = vision_hidden_states[i]
|
||||||
|
if len(cur_vs_hs) > 0:
|
||||||
|
cur_vllm_emb = vllm_embedding[i]
|
||||||
|
cur_image_bound = data['image_bound'][i]
|
||||||
|
if len(cur_image_bound) > 0:
|
||||||
|
image_indices = torch.stack(
|
||||||
|
[torch.arange(r[0], r[1], dtype=torch.long) for r in cur_image_bound]
|
||||||
|
).to(vllm_embedding.device)
|
||||||
|
|
||||||
|
cur_vllm_emb.scatter_(0, image_indices.view(-1, 1).repeat(1, cur_vllm_emb.shape[-1]),
|
||||||
|
cur_vs_hs.view(-1, cur_vs_hs.shape[-1]))
|
||||||
|
elif self.training:
|
||||||
|
cur_vllm_emb += cur_vs_hs[0].mean() * 0
|
||||||
|
|
||||||
|
return vllm_embedding, vision_hidden_states
|
||||||
|
|
||||||
|
def forward(self, data, **kwargs):
|
||||||
|
vllm_embedding, vision_hidden_states = self.get_vllm_embedding(data)
|
||||||
|
position_ids = data["position_ids"]
|
||||||
|
if position_ids.dtype != torch.int64:
|
||||||
|
position_ids = position_ids.long()
|
||||||
|
|
||||||
|
return self.llm(
|
||||||
|
input_ids=None,
|
||||||
|
position_ids=position_ids,
|
||||||
|
inputs_embeds=vllm_embedding,
|
||||||
|
**kwargs
|
||||||
|
)
|
||||||
|
|
||||||
|
def _convert_to_tensors(
|
||||||
|
self, tokenizer, input_ids, max_inp_length: Optional[int] = None
|
||||||
|
):
|
||||||
|
if max_inp_length is not None:
|
||||||
|
input_ids = input_ids[:max_inp_length]
|
||||||
|
input_ids = torch.tensor(input_ids, dtype=torch.int32)
|
||||||
|
|
||||||
|
image_start_tokens = torch.where(input_ids == tokenizer.im_start_id)[0]
|
||||||
|
# 跳过 im_start
|
||||||
|
image_start_tokens += 1
|
||||||
|
image_end_tokens = torch.where(input_ids == tokenizer.im_end_id)[0]
|
||||||
|
valid_image_nums = max(len(image_start_tokens), len(image_end_tokens))
|
||||||
|
image_bound = torch.hstack(
|
||||||
|
[
|
||||||
|
image_start_tokens[:valid_image_nums].unsqueeze(-1),
|
||||||
|
image_end_tokens[:valid_image_nums].unsqueeze(-1),
|
||||||
|
]
|
||||||
|
)
|
||||||
|
|
||||||
|
model_input = {}
|
||||||
|
model_input["input_ids"] = input_ids.unsqueeze(0).to(self.device)
|
||||||
|
model_input["image_bound"] = image_bound
|
||||||
|
|
||||||
|
return model_input
|
||||||
|
|
||||||
|
def _process_list(
|
||||||
|
self, tokenizer, input_id_list, max_inp_length: Optional[int] = None
|
||||||
|
):
|
||||||
|
pad_keys = ["input_ids"]
|
||||||
|
input_tensors = []
|
||||||
|
for input_ids in input_id_list:
|
||||||
|
input_tensors.append(
|
||||||
|
self._convert_to_tensors(tokenizer, input_ids, max_inp_length)
|
||||||
|
)
|
||||||
|
padded = {}
|
||||||
|
for key in pad_keys:
|
||||||
|
padded[key] = pad(input_tensors, key, padding_side="left").to(self.device)
|
||||||
|
padded["image_bound"] = [i["image_bound"] for i in input_tensors]
|
||||||
|
return padded
|
||||||
|
|
||||||
|
def _decode(self, inputs_embeds, tokenizer, **kwargs):
|
||||||
|
terminators = [
|
||||||
|
tokenizer.eos_token_id,
|
||||||
|
tokenizer.convert_tokens_to_ids("<|eot_id|>")
|
||||||
|
]
|
||||||
|
output = self.llm.generate(
|
||||||
|
inputs_embeds=inputs_embeds,
|
||||||
|
pad_token_id=0,
|
||||||
|
eos_token_id=terminators,
|
||||||
|
**kwargs
|
||||||
|
)
|
||||||
|
return self._decode_text(output, tokenizer)
|
||||||
|
|
||||||
|
def _decode_stream(self, inputs_embeds, tokenizer, **kwargs):
|
||||||
|
terminators = [
|
||||||
|
tokenizer.eos_token_id,
|
||||||
|
tokenizer.convert_tokens_to_ids("<|eot_id|>")
|
||||||
|
]
|
||||||
|
streamer = TextIteratorStreamer(tokenizer=tokenizer)
|
||||||
|
generation_kwargs = {
|
||||||
|
'inputs_embeds': inputs_embeds,
|
||||||
|
'pad_token_id': 0,
|
||||||
|
'eos_token_id': terminators,
|
||||||
|
'streamer': streamer
|
||||||
|
}
|
||||||
|
generation_kwargs.update(kwargs)
|
||||||
|
|
||||||
|
thread = Thread(target=self.llm.generate, kwargs=generation_kwargs)
|
||||||
|
thread.start()
|
||||||
|
|
||||||
|
return streamer
|
||||||
|
|
||||||
|
def _decode_text(self, result_ids, tokenizer):
|
||||||
|
result_text = []
|
||||||
|
for result in result_ids:
|
||||||
|
result = result[result != 0]
|
||||||
|
if result[0] == tokenizer.bos_id:
|
||||||
|
result = result[1:]
|
||||||
|
if result[-1] == tokenizer.eos_id or result[-1] == tokenizer.eot_id:
|
||||||
|
result = result[:-1]
|
||||||
|
result_text.append(tokenizer.decode(result).strip())
|
||||||
|
return result_text
|
||||||
|
|
||||||
|
def slice_image(self, image):
|
||||||
|
return slice_image(
|
||||||
|
image,
|
||||||
|
self.config.slice_config.max_slice_nums,
|
||||||
|
self.config.slice_config.scale_resolution,
|
||||||
|
self.config.slice_config.patch_size,
|
||||||
|
)
|
||||||
|
|
||||||
|
def get_slice_image_placeholder(self, image, tokenizer):
|
||||||
|
image_placeholder = (
|
||||||
|
tokenizer.im_start
|
||||||
|
+ tokenizer.unk_token * self.config.query_num
|
||||||
|
+ tokenizer.im_end
|
||||||
|
)
|
||||||
|
|
||||||
|
slice_images = []
|
||||||
|
|
||||||
|
source_image, patches, best_grid = slice_image(
|
||||||
|
image,
|
||||||
|
self.config.slice_config.max_slice_nums,
|
||||||
|
self.config.slice_config.scale_resolution,
|
||||||
|
self.config.slice_config.patch_size,
|
||||||
|
)
|
||||||
|
|
||||||
|
slice_images.append(source_image)
|
||||||
|
final_placeholder = image_placeholder
|
||||||
|
|
||||||
|
if len(patches) > 0:
|
||||||
|
for i in range(len(patches)):
|
||||||
|
for j in range(len(patches[0])):
|
||||||
|
slice_images.append(patches[i][j])
|
||||||
|
|
||||||
|
final_placeholder += get_grid_placeholder(
|
||||||
|
tokenizer, best_grid, self.config.query_num
|
||||||
|
)
|
||||||
|
|
||||||
|
return slice_images, final_placeholder
|
||||||
|
|
||||||
|
def reshape_by_patch(self, image_tensor):
|
||||||
|
"""
|
||||||
|
:param image_tensor: shape [3, H, W]
|
||||||
|
:param patch_size:
|
||||||
|
:return: [3, patch_size, HW/patch_size]
|
||||||
|
"""
|
||||||
|
patch_size = self.config.patch_size
|
||||||
|
patches = torch.nn.functional.unfold(
|
||||||
|
image_tensor,
|
||||||
|
(patch_size, patch_size),
|
||||||
|
stride=(patch_size, patch_size)
|
||||||
|
)
|
||||||
|
|
||||||
|
patches = patches.reshape(image_tensor.size(0), patch_size, patch_size, -1)
|
||||||
|
patches = patches.permute(0, 1, 3, 2).reshape(image_tensor.size(0), patch_size, -1)
|
||||||
|
return patches
|
||||||
|
|
||||||
|
def generate(
|
||||||
|
self,
|
||||||
|
input_id_list=None,
|
||||||
|
img_list=None,
|
||||||
|
tgt_sizes=None,
|
||||||
|
tokenizer=None,
|
||||||
|
max_inp_length: Optional[int] = None,
|
||||||
|
vision_hidden_states=None,
|
||||||
|
return_vision_hidden_states=False,
|
||||||
|
stream=False,
|
||||||
|
**kwargs
|
||||||
|
):
|
||||||
|
|
||||||
|
assert input_id_list is not None
|
||||||
|
bs = len(input_id_list)
|
||||||
|
if img_list == None:
|
||||||
|
img_list = [[] for i in range(bs)]
|
||||||
|
assert bs == len(img_list)
|
||||||
|
|
||||||
|
model_inputs = self._process_list(tokenizer, input_id_list, max_inp_length)
|
||||||
|
|
||||||
|
if vision_hidden_states is None:
|
||||||
|
pixel_values = []
|
||||||
|
for i in range(bs):
|
||||||
|
img_inps = []
|
||||||
|
for img in img_list[i]:
|
||||||
|
img_inps.append(img.to(self.device))
|
||||||
|
if img_inps:
|
||||||
|
pixel_values.append(img_inps)
|
||||||
|
else:
|
||||||
|
pixel_values.append([])
|
||||||
|
model_inputs["pixel_values"] = pixel_values
|
||||||
|
model_inputs['tgt_sizes'] = tgt_sizes
|
||||||
|
else:
|
||||||
|
model_inputs["vision_hidden_states"] = vision_hidden_states
|
||||||
|
|
||||||
|
with torch.inference_mode():
|
||||||
|
(
|
||||||
|
model_inputs["inputs_embeds"],
|
||||||
|
vision_hidden_states,
|
||||||
|
) = self.get_vllm_embedding(model_inputs)
|
||||||
|
|
||||||
|
if stream:
|
||||||
|
result = self._decode_stream(model_inputs["inputs_embeds"], tokenizer, **kwargs)
|
||||||
|
else:
|
||||||
|
result = self._decode(model_inputs["inputs_embeds"], tokenizer, **kwargs)
|
||||||
|
|
||||||
|
if return_vision_hidden_states:
|
||||||
|
return result, vision_hidden_states
|
||||||
|
|
||||||
|
return result
|
||||||
|
|
||||||
|
def chat(
|
||||||
|
self,
|
||||||
|
image,
|
||||||
|
msgs,
|
||||||
|
tokenizer,
|
||||||
|
vision_hidden_states=None,
|
||||||
|
max_new_tokens=1024,
|
||||||
|
sampling=True,
|
||||||
|
max_inp_length=2048,
|
||||||
|
system_prompt='',
|
||||||
|
stream=False,
|
||||||
|
**kwargs
|
||||||
|
):
|
||||||
|
if isinstance(msgs, str):
|
||||||
|
msgs = json.loads(msgs)
|
||||||
|
|
||||||
|
copy_msgs = deepcopy(msgs)
|
||||||
|
assert len(copy_msgs) > 0, 'msgs is empty'
|
||||||
|
assert sampling or not stream, 'if use stream mode, make sure sampling=True'
|
||||||
|
|
||||||
|
if image is not None and isinstance(copy_msgs[0]['content'], str):
|
||||||
|
copy_msgs[0]['content'] = [image, copy_msgs[0]['content']]
|
||||||
|
|
||||||
|
images = []
|
||||||
|
tgt_sizes = []
|
||||||
|
for i, msg in enumerate(copy_msgs):
|
||||||
|
role = msg["role"]
|
||||||
|
content = msg["content"]
|
||||||
|
assert role in ["user", "assistant"]
|
||||||
|
if i == 0:
|
||||||
|
assert role == "user", "The role of first msg should be user"
|
||||||
|
if isinstance(content, str):
|
||||||
|
content = [content]
|
||||||
|
|
||||||
|
cur_msgs = []
|
||||||
|
for c in content:
|
||||||
|
if isinstance(c, Image.Image):
|
||||||
|
image = c
|
||||||
|
if self.config.slice_mode:
|
||||||
|
slice_images, image_placeholder = self.get_slice_image_placeholder(
|
||||||
|
image, tokenizer
|
||||||
|
)
|
||||||
|
cur_msgs.append(image_placeholder)
|
||||||
|
for slice_image in slice_images:
|
||||||
|
slice_image = self.transform(slice_image)
|
||||||
|
H, W = slice_image.shape[1:]
|
||||||
|
images.append(self.reshape_by_patch(slice_image))
|
||||||
|
tgt_sizes.append(torch.Tensor([H // self.config.patch_size, W // self.config.patch_size]).type(torch.int32))
|
||||||
|
else:
|
||||||
|
images.append(self.transform(image))
|
||||||
|
cur_msgs.append(
|
||||||
|
tokenizer.im_start
|
||||||
|
+ tokenizer.unk_token * self.config.query_num
|
||||||
|
+ tokenizer.im_end
|
||||||
|
)
|
||||||
|
elif isinstance(c, str):
|
||||||
|
cur_msgs.append(c)
|
||||||
|
|
||||||
|
|
||||||
|
msg['content'] = '\n'.join(cur_msgs)
|
||||||
|
if tgt_sizes:
|
||||||
|
tgt_sizes = torch.vstack(tgt_sizes)
|
||||||
|
|
||||||
|
if system_prompt:
|
||||||
|
sys_msg = {'role': 'system', 'content': system_prompt}
|
||||||
|
copy_msgs = [sys_msg] + copy_msgs
|
||||||
|
|
||||||
|
input_ids = tokenizer.apply_chat_template(copy_msgs, tokenize=True, add_generation_prompt=False)
|
||||||
|
|
||||||
|
if sampling:
|
||||||
|
generation_config = {
|
||||||
|
"top_p": 0.8,
|
||||||
|
"top_k": 100,
|
||||||
|
"temperature": 0.7,
|
||||||
|
"do_sample": True,
|
||||||
|
"repetition_penalty": 1.05
|
||||||
|
}
|
||||||
|
else:
|
||||||
|
generation_config = {
|
||||||
|
"num_beams": 3,
|
||||||
|
"repetition_penalty": 1.2,
|
||||||
|
}
|
||||||
|
|
||||||
|
generation_config.update(
|
||||||
|
(k, kwargs[k]) for k in generation_config.keys() & kwargs.keys()
|
||||||
|
)
|
||||||
|
|
||||||
|
with torch.inference_mode():
|
||||||
|
res, vision_hidden_states = self.generate(
|
||||||
|
input_id_list=[input_ids],
|
||||||
|
max_inp_length=max_inp_length,
|
||||||
|
img_list=[images],
|
||||||
|
tgt_sizes=[tgt_sizes],
|
||||||
|
tokenizer=tokenizer,
|
||||||
|
max_new_tokens=max_new_tokens,
|
||||||
|
vision_hidden_states=vision_hidden_states,
|
||||||
|
return_vision_hidden_states=True,
|
||||||
|
stream=stream,
|
||||||
|
**generation_config
|
||||||
|
)
|
||||||
|
|
||||||
|
if stream:
|
||||||
|
def stream_gen():
|
||||||
|
for text in res:
|
||||||
|
text = text.replace(tokenizer.eot_token, '').replace(tokenizer.eos_token, '')
|
||||||
|
yield text
|
||||||
|
return stream_gen()
|
||||||
|
|
||||||
|
else:
|
||||||
|
answer = res[0]
|
||||||
|
return answer
|
||||||
|
|
||||||
|
|
||||||
|
class PreTrainedTokenizerFastWrapper(PreTrainedTokenizerFast):
|
||||||
|
def __init__(self, **kwargs):
|
||||||
|
super().__init__(**kwargs)
|
||||||
|
self.eot_token = "<|eot_id|>"
|
||||||
|
self.im_start = "<image>"
|
||||||
|
self.im_end = "</image>"
|
||||||
|
self.ref_start = "<ref>"
|
||||||
|
self.ref_end = "</ref>"
|
||||||
|
self.box_start = "<box>"
|
||||||
|
self.box_end = "</box>"
|
||||||
|
self.quad_start = "<quad>"
|
||||||
|
self.quad_end = "</quad>"
|
||||||
|
self.slice_start = "<slice>"
|
||||||
|
self.slice_end = "</slice>"
|
||||||
|
|
||||||
|
@property
|
||||||
|
def eos_id(self):
|
||||||
|
return self.eos_token_id
|
||||||
|
|
||||||
|
@property
|
||||||
|
def bos_id(self):
|
||||||
|
return self.bos_token_id
|
||||||
|
|
||||||
|
@property
|
||||||
|
def unk_id(self):
|
||||||
|
return self.unk_token_id
|
||||||
|
|
||||||
|
@property
|
||||||
|
def eot_id(self):
|
||||||
|
return self.convert_tokens_to_ids(self.eot_token)
|
||||||
|
|
||||||
|
@property
|
||||||
|
def im_start_id(self):
|
||||||
|
return self.convert_tokens_to_ids(self.im_start)
|
||||||
|
|
||||||
|
@property
|
||||||
|
def im_end_id(self):
|
||||||
|
return self.convert_tokens_to_ids(self.im_end)
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def escape(text: str) -> str:
|
||||||
|
return text
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def unescape(text: str) -> str:
|
||||||
|
return text
|
||||||
|
|
||||||
|
|
||||||
|
def pad(orig_items, key, max_length=None, padding_value=0, padding_side="left"):
|
||||||
|
items = []
|
||||||
|
if isinstance(orig_items[0][key], list):
|
||||||
|
assert isinstance(orig_items[0][key][0], torch.Tensor)
|
||||||
|
for it in orig_items:
|
||||||
|
for tr in it[key]:
|
||||||
|
items.append({key: tr})
|
||||||
|
else:
|
||||||
|
assert isinstance(orig_items[0][key], torch.Tensor)
|
||||||
|
items = orig_items
|
||||||
|
|
||||||
|
batch_size = len(items)
|
||||||
|
shape = items[0][key].shape
|
||||||
|
dim = len(shape)
|
||||||
|
assert dim <= 3
|
||||||
|
if max_length is None:
|
||||||
|
max_length = 0
|
||||||
|
max_length = max(max_length, max(item[key].shape[-1] for item in items))
|
||||||
|
min_length = min(item[key].shape[-1] for item in items)
|
||||||
|
dtype = items[0][key].dtype
|
||||||
|
|
||||||
|
if dim == 1:
|
||||||
|
return torch.cat([item[key] for item in items], dim=0)
|
||||||
|
elif dim == 2:
|
||||||
|
if max_length == min_length:
|
||||||
|
return torch.cat([item[key] for item in items], dim=0)
|
||||||
|
tensor = torch.zeros((batch_size, max_length), dtype=dtype) + padding_value
|
||||||
|
else:
|
||||||
|
tensor = (
|
||||||
|
torch.zeros((batch_size, max_length, shape[-1]), dtype=dtype)
|
||||||
|
+ padding_value
|
||||||
|
)
|
||||||
|
|
||||||
|
for i, item in enumerate(items):
|
||||||
|
if dim == 2:
|
||||||
|
if padding_side == "left":
|
||||||
|
tensor[i, -len(item[key][0]) :] = item[key][0].clone()
|
||||||
|
else:
|
||||||
|
tensor[i, : len(item[key][0])] = item[key][0].clone()
|
||||||
|
elif dim == 3:
|
||||||
|
if padding_side == "left":
|
||||||
|
tensor[i, -len(item[key][0]) :, :] = item[key][0].clone()
|
||||||
|
else:
|
||||||
|
tensor[i, : len(item[key][0]), :] = item[key][0].clone()
|
||||||
|
|
||||||
|
return tensor
|
||||||
|
|
||||||
|
|
||||||
|
def slice_image(
|
||||||
|
image, max_slice_nums=9, scale_resolution=448, patch_size=14, never_split=False
|
||||||
|
):
|
||||||
|
original_size = image.size
|
||||||
|
original_width, original_height = original_size
|
||||||
|
log_ratio = math.log(original_width / original_height)
|
||||||
|
ratio = original_width * original_height / (scale_resolution * scale_resolution)
|
||||||
|
multiple = min(math.ceil(ratio), max_slice_nums)
|
||||||
|
|
||||||
|
source_image = None
|
||||||
|
best_grid = None
|
||||||
|
patches = []
|
||||||
|
|
||||||
|
if multiple <= 1 or never_split:
|
||||||
|
# dont need to slice, upsample
|
||||||
|
best_size = find_best_resize(
|
||||||
|
original_size, scale_resolution, patch_size, allow_upscale=True
|
||||||
|
)
|
||||||
|
source_image = image.resize(best_size, Image.Resampling.BICUBIC)
|
||||||
|
else:
|
||||||
|
candidate_split_grids_nums = []
|
||||||
|
for i in [multiple - 1, multiple, multiple + 1]:
|
||||||
|
if i == 1 or i > max_slice_nums:
|
||||||
|
continue
|
||||||
|
candidate_split_grids_nums.append(i)
|
||||||
|
|
||||||
|
# source image, down-sampling and ensure divided by patch_size
|
||||||
|
best_resize = find_best_resize(original_size, scale_resolution, patch_size)
|
||||||
|
source_image = image.copy().resize(best_resize, Image.Resampling.BICUBIC)
|
||||||
|
candidate_grids = []
|
||||||
|
|
||||||
|
# find best grid
|
||||||
|
for split_grids_nums in candidate_split_grids_nums:
|
||||||
|
m = 1
|
||||||
|
while m <= split_grids_nums:
|
||||||
|
if split_grids_nums % m == 0:
|
||||||
|
candidate_grids.append([m, split_grids_nums // m])
|
||||||
|
m += 1
|
||||||
|
|
||||||
|
best_grid = [1, 1]
|
||||||
|
min_error = float("inf")
|
||||||
|
for grid in candidate_grids:
|
||||||
|
error = abs(log_ratio - math.log(grid[0] / grid[1]))
|
||||||
|
if error < min_error:
|
||||||
|
best_grid = grid
|
||||||
|
min_error = error
|
||||||
|
|
||||||
|
refine_size = get_refine_size(
|
||||||
|
original_size, best_grid, scale_resolution, patch_size, allow_upscale=True
|
||||||
|
)
|
||||||
|
|
||||||
|
refine_image = image.resize(refine_size, Image.Resampling.BICUBIC)
|
||||||
|
patches = split_to_patches(refine_image, best_grid)
|
||||||
|
|
||||||
|
return source_image, patches, best_grid
|
||||||
|
|
||||||
|
|
||||||
|
def ensure_divide(length, patch_size):
|
||||||
|
return max(round(length / patch_size) * patch_size, patch_size)
|
||||||
|
|
||||||
|
|
||||||
|
def find_best_resize(original_size, scale_resolution, patch_size, allow_upscale=False):
|
||||||
|
width, height = original_size
|
||||||
|
if (width * height > scale_resolution * scale_resolution) or allow_upscale:
|
||||||
|
r = width / height
|
||||||
|
height = int(scale_resolution / math.sqrt(r))
|
||||||
|
width = int(height * r)
|
||||||
|
best_width = ensure_divide(width, patch_size)
|
||||||
|
best_height = ensure_divide(height, patch_size)
|
||||||
|
return (best_width, best_height)
|
||||||
|
|
||||||
|
|
||||||
|
def get_refine_size(
|
||||||
|
original_size, grid, scale_resolution, patch_size, allow_upscale=False
|
||||||
|
):
|
||||||
|
width, height = original_size
|
||||||
|
grid_x, grid_y = grid
|
||||||
|
|
||||||
|
refine_width = ensure_divide(width, grid_x)
|
||||||
|
refine_height = ensure_divide(height, grid_y)
|
||||||
|
|
||||||
|
grid_width = refine_width / grid_x
|
||||||
|
grid_height = refine_height / grid_y
|
||||||
|
|
||||||
|
best_grid_size = find_best_resize(
|
||||||
|
(grid_width, grid_height),
|
||||||
|
scale_resolution,
|
||||||
|
patch_size,
|
||||||
|
allow_upscale=allow_upscale,
|
||||||
|
)
|
||||||
|
|
||||||
|
refine_size = (best_grid_size[0] * grid_x, best_grid_size[1] * grid_y)
|
||||||
|
|
||||||
|
return refine_size
|
||||||
|
|
||||||
|
|
||||||
|
def split_to_patches(image, grid):
|
||||||
|
patches = []
|
||||||
|
width, height = image.size
|
||||||
|
grid_x = int(width / grid[0])
|
||||||
|
grid_y = int(height / grid[1])
|
||||||
|
|
||||||
|
for i in range(0, height, grid_y):
|
||||||
|
images = []
|
||||||
|
for j in range(0, width, grid_x):
|
||||||
|
box = (j, i, j + grid_x, i + grid_y)
|
||||||
|
patch = image.crop(box)
|
||||||
|
images.append(patch)
|
||||||
|
patches.append(images)
|
||||||
|
|
||||||
|
return patches
|
||||||
|
|
||||||
|
|
||||||
|
def get_grid_placeholder(tokenizer, grid, query_num):
|
||||||
|
image_placeholder = (
|
||||||
|
tokenizer.im_start + tokenizer.unk_token * query_num + tokenizer.im_end
|
||||||
|
)
|
||||||
|
|
||||||
|
cols = grid[0]
|
||||||
|
rows = grid[1]
|
||||||
|
slices = []
|
||||||
|
for i in range(rows):
|
||||||
|
lines = []
|
||||||
|
for j in range(cols):
|
||||||
|
lines.append(image_placeholder)
|
||||||
|
slices.append("".join(lines))
|
||||||
|
slice_placeholder = tokenizer.slice_start + "\n".join(slices) + tokenizer.slice_end
|
||||||
|
return slice_placeholder
|
||||||
|
|
@ -0,0 +1,163 @@
|
||||||
|
from functools import partial
|
||||||
|
import numpy as np
|
||||||
|
|
||||||
|
import torch
|
||||||
|
from torch import nn
|
||||||
|
from torch.nn.init import trunc_normal_
|
||||||
|
|
||||||
|
def get_2d_sincos_pos_embed(embed_dim, image_size):
|
||||||
|
"""
|
||||||
|
image_size: image_size or (image_height, image_width)
|
||||||
|
return:
|
||||||
|
pos_embed: [image_height, image_width, embed_dim]
|
||||||
|
"""
|
||||||
|
if isinstance(image_size, int):
|
||||||
|
grid_h_size, grid_w_size = image_size, image_size
|
||||||
|
else:
|
||||||
|
grid_h_size, grid_w_size = image_size[0], image_size[1]
|
||||||
|
|
||||||
|
grid_h = np.arange(grid_h_size, dtype=np.float32)
|
||||||
|
grid_w = np.arange(grid_w_size, dtype=np.float32)
|
||||||
|
grid = np.meshgrid(grid_w, grid_h) # here w goes first
|
||||||
|
grid = np.stack(grid, axis=0)
|
||||||
|
|
||||||
|
pos_embed = get_2d_sincos_pos_embed_from_grid(embed_dim, grid)
|
||||||
|
return pos_embed
|
||||||
|
|
||||||
|
|
||||||
|
def get_2d_sincos_pos_embed_from_grid(embed_dim, grid):
|
||||||
|
assert embed_dim % 2 == 0
|
||||||
|
|
||||||
|
# use half of dimensions to encode grid_h
|
||||||
|
emb_h = get_1d_sincos_pos_embed_from_grid_new(embed_dim // 2, grid[0]) # (H, W, D/2)
|
||||||
|
emb_w = get_1d_sincos_pos_embed_from_grid_new(embed_dim // 2, grid[1]) # (H, W, D/2)
|
||||||
|
|
||||||
|
emb = np.concatenate([emb_h, emb_w], axis=-1) # (H, W, D)
|
||||||
|
return emb
|
||||||
|
|
||||||
|
|
||||||
|
def get_1d_sincos_pos_embed_from_grid_new(embed_dim, pos):
|
||||||
|
"""
|
||||||
|
embed_dim: output dimension for each position
|
||||||
|
pos: a list of positions to be encoded: size (H, W)
|
||||||
|
out: (H, W, D)
|
||||||
|
"""
|
||||||
|
assert embed_dim % 2 == 0
|
||||||
|
omega = np.arange(embed_dim // 2, dtype=np.float32)
|
||||||
|
omega /= embed_dim / 2.
|
||||||
|
omega = 1. / 10000 ** omega # (D/2,)
|
||||||
|
|
||||||
|
out = np.einsum('hw,d->hwd', pos, omega) # (H, W, D/2), outer product
|
||||||
|
|
||||||
|
emb_sin = np.sin(out) # (H, W, D/2)
|
||||||
|
emb_cos = np.cos(out) # (H, W, D/2)
|
||||||
|
|
||||||
|
emb = np.concatenate([emb_sin, emb_cos], axis=-1) # (H, W, D)
|
||||||
|
return emb
|
||||||
|
|
||||||
|
|
||||||
|
class Resampler(nn.Module):
|
||||||
|
"""
|
||||||
|
A 2D perceiver-resampler network with one cross attention layers by
|
||||||
|
given learnable queries and 2d sincos pos_emb
|
||||||
|
Outputs:
|
||||||
|
A tensor with the shape of (batch_size, num_queries, embed_dim)
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
num_queries,
|
||||||
|
embed_dim,
|
||||||
|
num_heads,
|
||||||
|
kv_dim=None,
|
||||||
|
norm_layer=partial(nn.LayerNorm, eps=1e-6),
|
||||||
|
adaptive=False,
|
||||||
|
max_size=(70, 70),
|
||||||
|
):
|
||||||
|
super().__init__()
|
||||||
|
self.num_queries = num_queries
|
||||||
|
self.embed_dim = embed_dim
|
||||||
|
self.num_heads = num_heads
|
||||||
|
self.adaptive = adaptive
|
||||||
|
self.max_size = max_size
|
||||||
|
|
||||||
|
self.query = nn.Parameter(torch.zeros(self.num_queries, embed_dim))
|
||||||
|
trunc_normal_(self.query, std=.02)
|
||||||
|
|
||||||
|
if kv_dim is not None and kv_dim != embed_dim:
|
||||||
|
self.kv_proj = nn.Linear(kv_dim, embed_dim, bias=False)
|
||||||
|
else:
|
||||||
|
self.kv_proj = nn.Identity()
|
||||||
|
|
||||||
|
self.attn = nn.MultiheadAttention(embed_dim, num_heads)
|
||||||
|
self.ln_q = norm_layer(embed_dim)
|
||||||
|
self.ln_kv = norm_layer(embed_dim)
|
||||||
|
|
||||||
|
self.ln_post = norm_layer(embed_dim)
|
||||||
|
self.proj = nn.Parameter((embed_dim ** -0.5) * torch.randn(embed_dim, embed_dim))
|
||||||
|
|
||||||
|
self._set_2d_pos_cache(self.max_size)
|
||||||
|
self.apply(self._init_weights)
|
||||||
|
|
||||||
|
def _set_2d_pos_cache(self, max_size, device='cpu'):
|
||||||
|
pos_embed = torch.from_numpy(get_2d_sincos_pos_embed(self.embed_dim, max_size)).float().to(device)
|
||||||
|
self.register_buffer("pos_embed", pos_embed, persistent=False)
|
||||||
|
|
||||||
|
def _adjust_pos_cache(self, tgt_sizes, device):
|
||||||
|
max_h = torch.max(tgt_sizes[:, 0])
|
||||||
|
max_w = torch.max(tgt_sizes[:, 1])
|
||||||
|
if max_h > self.max_size[0] or max_w > self.max_size[1]:
|
||||||
|
self.max_size = [max(max_h, self.max_size[0]), max(max_w, self.max_size[1])]
|
||||||
|
self._set_2d_pos_cache(self.max_size, device)
|
||||||
|
|
||||||
|
def _init_weights(self, m):
|
||||||
|
if isinstance(m, nn.Linear):
|
||||||
|
trunc_normal_(m.weight, std=.02)
|
||||||
|
if isinstance(m, nn.Linear) and m.bias is not None:
|
||||||
|
nn.init.constant_(m.bias, 0)
|
||||||
|
elif isinstance(m, nn.LayerNorm):
|
||||||
|
nn.init.constant_(m.bias, 0)
|
||||||
|
nn.init.constant_(m.weight, 1.0)
|
||||||
|
|
||||||
|
def forward(self, x, tgt_sizes=None):
|
||||||
|
assert x.shape[0] == tgt_sizes.shape[0]
|
||||||
|
bs = x.shape[0]
|
||||||
|
|
||||||
|
device = x.device
|
||||||
|
dtype = x.dtype
|
||||||
|
|
||||||
|
patch_len = tgt_sizes[:, 0] * tgt_sizes[:, 1]
|
||||||
|
|
||||||
|
self._adjust_pos_cache(tgt_sizes, device=device)
|
||||||
|
|
||||||
|
max_patch_len = torch.max(patch_len)
|
||||||
|
key_padding_mask = torch.zeros((bs, max_patch_len), dtype=torch.bool, device=device)
|
||||||
|
|
||||||
|
pos_embed = []
|
||||||
|
for i in range(bs):
|
||||||
|
tgt_h, tgt_w = tgt_sizes[i]
|
||||||
|
pos_embed.append(self.pos_embed[:tgt_h, :tgt_w, :].reshape((tgt_h * tgt_w, -1)).to(dtype)) # patches * D
|
||||||
|
key_padding_mask[i, patch_len[i]:] = True
|
||||||
|
|
||||||
|
pos_embed = torch.nn.utils.rnn.pad_sequence(
|
||||||
|
pos_embed, batch_first=True, padding_value=0.0).permute(1, 0, 2) # BLD => L * B * D
|
||||||
|
|
||||||
|
x = self.kv_proj(x) # B * L * D
|
||||||
|
x = self.ln_kv(x).permute(1, 0, 2) # L * B * D
|
||||||
|
|
||||||
|
q = self.ln_q(self.query) # Q * D
|
||||||
|
|
||||||
|
out = self.attn(
|
||||||
|
self._repeat(q, bs), # Q * B * D
|
||||||
|
x + pos_embed, # L * B * D + L * B * D
|
||||||
|
x,
|
||||||
|
key_padding_mask=key_padding_mask)[0]
|
||||||
|
# out: Q * B * D
|
||||||
|
x = out.permute(1, 0, 2) # B * Q * D
|
||||||
|
|
||||||
|
x = self.ln_post(x)
|
||||||
|
x = x @ self.proj
|
||||||
|
return x
|
||||||
|
|
||||||
|
def _repeat(self, query, N: int):
|
||||||
|
return query.unsqueeze(1).repeat(1, N, 1)
|
||||||
|
|
@ -0,0 +1,24 @@
|
||||||
|
{
|
||||||
|
"bos_token": {
|
||||||
|
"content": "<|begin_of_text|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false
|
||||||
|
},
|
||||||
|
"eos_token": {
|
||||||
|
"content": "<|end_of_text|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false
|
||||||
|
},
|
||||||
|
"pad_token": "!",
|
||||||
|
"unk_token": {
|
||||||
|
"content": "<unk>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false
|
||||||
|
}
|
||||||
|
}
|
||||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
Loading…
Reference in New Issue