first commit
This commit is contained in:
xxl
parent
5e3b8dc818
commit
ab40b3f9b5
|
|
@ -0,0 +1,94 @@
|
||||||
|
The CogAgent License
|
||||||
|
|
||||||
|
1. Definitions
|
||||||
|
|
||||||
|
"Licensor" refers to the CogAgent model team distributing its software.
|
||||||
|
"Software" refers to the CogAgent model parameters provided under this license.
|
||||||
|
|
||||||
|
2. License Grant
|
||||||
|
|
||||||
|
Subject to the terms and conditions of this license, the Licensor hereby grants you a non-exclusive, worldwide, non-transferable, non-sublicensable, revocable, royalty-free copyright license.
|
||||||
|
This license permits you to use all open-source models in this repository for free for academic research purposes. Users wishing to use the model for commercial purposes must complete registration [here](https://open.bigmodel.cn/mla/form). Registered users may use the model for commercial activities free of charge but must comply with all terms and conditions of this license.
|
||||||
|
The above copyright statement and this license statement must be included in all copies or significant portions of the software.
|
||||||
|
If you distribute or provide materials from THUDM/Zhipu AI regarding the CogAgent open-source model (or any derivative works thereof) or use any materials (including all open-source models in the CogAgent series) in products or services, you must:
|
||||||
|
|
||||||
|
(A) Provide a copy of this agreement with any such materials from THUDM/Zhipu AI;
|
||||||
|
(B) Prominently display “Built with CogAgent” on relevant websites, user interfaces, blog posts, about pages, or product documentation.
|
||||||
|
If you use materials from THUDM/Zhipu AI's CogAgent open-source model to create, train, fine-tune, or otherwise improve distributed or publicly available AI models, you must also prepend “CogAgent” to the names of any such AI models.
|
||||||
|
|
||||||
|
3. Restrictions
|
||||||
|
|
||||||
|
You must comply with applicable laws, regulations, ethical standards, and other requirements in your jurisdiction when using this software. You must independently obtain permissions, licenses, or other access rights required by third-party software/applications and make prudent and independent judgments on all operational decisions. You must not use the software or implement the following actions in an improper manner:
|
||||||
|
(1) Use, copy, modify, merge, publish, distribute, or create derivative works of this software, in whole or in part, for any military or illegal purposes;
|
||||||
|
(2) Engage in activities that harm national security, public interest, social morals, or infringe upon others' trade secrets, intellectual property, reputation, portrait rights, property rights, or other rights and interests;
|
||||||
|
(3) Use the software for fraud, phishing, spamming, misleading, bullying, harassment, discrimination, hate promotion, or dissemination of false information;
|
||||||
|
(4) Use the software to make automated high-risk decisions in fields such as health, education, credit, finance, or critical infrastructure management, which significantly impact individual or societal safety, rights, or welfare;
|
||||||
|
(5) Rely on the software to perform major operations, including but not limited to monetary transactions, large purchases, placing irreversible orders, or publishing content detrimental to others' rights or social ethics;
|
||||||
|
(6) Use the software in services requiring subject qualifications or professional review, or as a substitute for professional services in fields such as medicine, law, journalism, education, or financial investment;
|
||||||
|
(7) Use the software dishonestly, claim or imply AI-generated content is human-created, or disguise human-created works as AI-generated content;
|
||||||
|
(8) Engage in illegal network intrusion, disrupt normal network functionality, steal network data, or deliberately spread malicious programs or viruses that harm network security and order;
|
||||||
|
(9) Collect personal information unlawfully or use the software in a way that infringes upon any third party’s personal information protection rights or privacy.
|
||||||
|
The Licensor bears no responsibility for your actions while using this software, and you shall assume all resulting liabilities.
|
||||||
|
|
||||||
|
4. Disclaimer
|
||||||
|
|
||||||
|
The software is provided "as is" without any express or implied warranties, including but not limited to warranties of merchantability, fitness for a particular purpose, or non-infringement.
|
||||||
|
The Licensor does not guarantee the content or operations executed by the software are entirely accurate, reliable, functional, timely, secure, error-free, uninterrupted, or continuously stable. The Licensor is not liable for risks arising from your operational errors or software defects.
|
||||||
|
In no event shall the authors or copyright holders be liable for any claims, damages, or other liabilities, whether in contract, tort, or otherwise, arising from, out of, or in connection with the software or the use or other dealings in the software.
|
||||||
|
|
||||||
|
5. Limitation of Liability
|
||||||
|
|
||||||
|
To the maximum extent permitted by applicable law, in no event and under no legal theory shall the Licensor be liable for any direct, indirect, special, incidental, exemplary, or consequential damages, or any other commercial losses, even if the Licensor has been advised of the possibility of such damages.
|
||||||
|
|
||||||
|
6. Dispute Resolution
|
||||||
|
|
||||||
|
This license is governed by and construed in accordance with the laws of the People's Republic of China. Any disputes arising out of or in connection with this license shall be submitted to the People's Court of Haidian District, Beijing.
|
||||||
|
Please note that this license may be updated to a more comprehensive version. For any questions about the license or copyright, please contact us at license@zhipuai.cn or opensource@zhipuai.cn.
|
||||||
|
|
||||||
|
CogAgent系列模型开源协议
|
||||||
|
|
||||||
|
1. 定义
|
||||||
|
|
||||||
|
“许可方”是指分发其软件的 CogAgent系列 模型团队。
|
||||||
|
“软件”是指根据本许可提供的 CogAgent系列 模型参数。
|
||||||
|
|
||||||
|
2. 许可授予
|
||||||
|
|
||||||
|
根据本许可的条款和条件，许可方特此授予您非排他性、全球性、不可转让、不可再许可、可撤销、免版税的版权许可。
|
||||||
|
本许可仅允许您免费使用本仓库中的所有开源模型进行学术研究，对于希望将模型用于商业目的的用户，需在[这里](https://open.bigmodel.cn/mla/form)完成登记。经过登记的用户可以免费使用本模型进行商业活动，但必须遵守本许可的所有条款和条件。
|
||||||
|
上述版权声明和本许可声明应包含在本软件的所有副本或重要部分中。
|
||||||
|
如果您分发或提供 THUDM / 智谱AI 关于 CogAgent系列开源模型的材料（或其任何衍生作品），或使用其中任何材料（包括 CogAgent系列的所有开源模型）的产品或服务，您应:
|
||||||
|
|
||||||
|
(A) 随任何此类 THUDM / 智谱AI 材料提供本协议的副本；
|
||||||
|
(B) 在相关网站、用户界面、博客文章、关于页面或产品文档上突出显示 “Built with CogAgent”。
|
||||||
|
如果您使用 THUDM / 智谱AI的 CogAgent系列开源模型的材料来创建、训练、微调或以其他方式改进已分发或可用的 AI 模型，您还应在任何此类 AI 模型名称的开头添加 “CogAgent”。
|
||||||
|
|
||||||
|
3. 限制
|
||||||
|
|
||||||
|
您在使用中应遵循使用地所适用的法律法规政策、道德规范等要求，在操作应用程序时自行取得第三方软件/应用所需的操作权限、授权或其他准入要求并对所有操作决策进行独立审慎的判断。您不得以以下不当方式使用软件或利用软件实施以下行为：
|
||||||
|
(1) 出于任何军事或非法目的使用、复制、修改、合并、发布、分发、复制或创建本软件的全部或部分衍生作品；
|
||||||
|
(2) 利用本软件从事任何危害国家安全和国家统一，危害社会公共利益及公序良俗，侵犯他人商业秘密、知识产权、名誉权、肖像权、财产权等权益的行为；
|
||||||
|
(3) 用于欺诈、诈骗、发送垃圾短信/邮件、误导、欺凌、骚扰、歧视、宣扬仇恨、传播虚假信息等途径；
|
||||||
|
(4) 利用软件实施任何决策行为，如在健康、教育、信贷、金融、关键基础设施管理等对个人及社会的安全、权利或福祉有重大影响的领域做出高风险的自动化决策；
|
||||||
|
(5) 依赖本软件执行任何重大的操作，包括但不限于资金交易、大额消费、下单不可撤销的订单、发布有损他人权益或社会公德的消息等；
|
||||||
|
(6) 用于任何对主体资格有要求或需要专业人员审查的服务中，或作为专业服务的替代品，包括但不限于医疗、律师、新闻、教育、投资理财等专业领域；
|
||||||
|
(7) 以不诚实的方式使用，主张或声称人工智能的生成物是人类的作品，或将人类的作品伪装为人工智能的生成物；
|
||||||
|
(8) 非法侵入网络、干扰网络正常功能、窃取网络数据、故意传播恶意程序或病毒等危害网络安全和网络秩序的活动；
|
||||||
|
(9) 违法采集他人个人信息，或以可能侵犯任何第三方个人信息保护权及隐私的方式使用本软件。
|
||||||
|
许可方不对您使用本软件的行为承担任何责任，由此产生的责任将由您自行承担。
|
||||||
|
|
||||||
|
4. 免责声明
|
||||||
|
|
||||||
|
本软件“按原样”提供，不提供任何明示或暗示的保证，包括但不限于对适销性、特定用途的适用性和非侵权性的保证。
|
||||||
|
许可方不保证软件生成的内容及执行的操作百分百准确可靠、功能可用、及时、安全、无错误、不受干扰、无中断、持续稳定、不存在任何故障，AI并不能真正像人类一样理解您输入的内容及指令，如果由于您的操作失误或AI的缺陷导致的风险，许可方不承担相应的责任。
|
||||||
|
在任何情况下，作者或版权持有人均不对任何索赔、损害或其他责任负责，无论是在合同诉讼、侵权行为还是其他方面，包括但不限于由软件或软件的使用引起、利用软件进行的交易或与软件相关引起的问题。
|
||||||
|
|
||||||
|
5. 责任限制
|
||||||
|
|
||||||
|
除适用法律禁止的范围外，在任何情况下且根据任何法律理论，无论是基于侵权行为、疏忽、合同、责任或其他原因，任何许可方均不对您承担任何直接、间接、特殊、偶然、示范性、
|
||||||
|
或间接损害，或任何其他商业损失，即使许可人已被告知此类损害的可能性。
|
||||||
|
|
||||||
|
6. 争议解决
|
||||||
|
|
||||||
|
本许可受中华人民共和国法律管辖并按其解释。 因本许可引起的或与本许可有关的任何争议应提交北京市海淀区人民法院。
|
||||||
|
请注意，许可证可能会更新到更全面的版本。 有关许可和版权的任何问题，请通过 license@zhipuai.cn 或 opensource@zhipuai.cn 与我们联系。
|
||||||
94
README.md
94
README.md
|
|
@ -1,3 +1,93 @@
|
||||||
# CogAgent-9B-2024122_a14079275235536896916920
|
---
|
||||||
|
frameworks:
|
||||||
|
- Pytorch
|
||||||
|
license: other
|
||||||
|
domain:
|
||||||
|
- nlp
|
||||||
|
|
||||||
CogAgent-9B-2024122
|
language:
|
||||||
|
- cn
|
||||||
|
- en
|
||||||
|
---
|
||||||
|
# CogAgent
|
||||||
|
|
||||||
|
<p style="text-align: center;">
|
||||||
|
<p align="center">
|
||||||
|
<a href="https://github.com/THUDM/CogAgent">🌐 Github </a> |
|
||||||
|
<a href="https://huggingface.co/spaces/THUDM-HF-SPACE/CogAgent-Demo">🤗 Huggingface Space</a> |
|
||||||
|
<a href="https://cogagent.aminer.cn/blog#/articles/cogagent-9b-20241220-technical-report-en">📄 Technical Report </a> |
|
||||||
|
<a href="https://arxiv.org/abs/2312.08914">📜 arxiv paper </a>
|
||||||
|
</p>
|
||||||
|
|
||||||
|
## 关于模型
|
||||||
|
|
||||||
|
`CogAgent-9B-2024122` 模型基于 [GLM-4V-9B](https://huggingface.co/THUDM/glm-4v-9b)
|
||||||
|
双语开源VLM基座模型，通过数据的采集与优化、多阶段训练与策略改进等方法，`CogAgent-9B-20241220` 在GUI
|
||||||
|
感知、推理预测准确性、动作空间完善性、任务的普适和泛化性上得到了大幅提升，能够接受中英文双语的屏幕截图和语言交互。
|
||||||
|
此版CogAgent模型已被应用于智谱AI的 [GLM-PC产品](https://cogagent.aminer.cn/home)
|
||||||
|
。我们希望这版模型的发布能够帮助到学术研究者们和开发者们，一起推进基于视觉语言往我们的模型的 GUI agent 的研究和应用。
|
||||||
|
|
||||||
|
## 运行模型
|
||||||
|
|
||||||
|
<p>请前往我们的 <a href="https://github.com/THUDM/CogAgent">github</a> 查看具体的运行示例，以及模型提示词拼接部分 <strong style="color: red;">(这直接影响模型是否正常运行)</strong>。</p>
|
||||||
|
|
||||||
|
其中，特别注意提示词拼接过程。
|
||||||
|
您可以参考 [app/client.py#L115](https://github.com/THUDM/CogAgent/blob/e3ca6f4dc94118d3dfb749f195cbb800ee4543ce/app/client.py#L115)
|
||||||
|
拼接用户输入提示词。
|
||||||
|
``` python
|
||||||
|
|
||||||
|
current_platform = identify_os() # "Mac" or "WIN" or "Mobile"，注意大小写
|
||||||
|
platform_str = f"(Platform: {current_platform})\n"
|
||||||
|
format_str = "(Answer in Action-Operation-Sensitive format.)\n" # You can use other format to replace "Action-Operation-Sensitive"
|
||||||
|
|
||||||
|
history_str = "\nHistory steps: "
|
||||||
|
for index, (grounded_op_func, action) in enumerate(zip(history_grounded_op_funcs, history_actions)):
|
||||||
|
history_str += f"\n{index}. {grounded_op_func}\t{action}" # start from 0.
|
||||||
|
|
||||||
|
query = f"Task: {task}{history_str}\n{platform_str}{format_str}" # Be careful about the \n
|
||||||
|
|
||||||
|
```
|
||||||
|
|
||||||
|
一个最简用户输入拼接代码如下所示：
|
||||||
|
|
||||||
|
```
|
||||||
|
"Task: Search for doors, click doors on sale and filter by brands \"Mastercraft\".\nHistory steps: \n0. CLICK(box=[[352,102,786,139]], element_info='Search')\tLeft click on the search box located in the middle top of the screen next to the Menards logo.\n1. TYPE(box=[[352,102,786,139]], text='doors', element_info='Search')\tIn the search input box at the top, type 'doors'.\n2. CLICK(box=[[787,102,809,139]], element_info='SEARCH')\tLeft click on the magnifying glass icon next to the search bar to perform the search.\n3. SCROLL_DOWN(box=[[0,209,998,952]], step_count=5, element_info='[None]')\tScroll down the page to see the available doors.\n4. CLICK(box=[[280,708,710,809]], element_info='Doors on Sale')\tClick the \"Doors On Sale\" button in the middle of the page to view the doors that are currently on sale.\n(Platform: WIN)\n(Answer in Action-Operation format.)\n"
|
||||||
|
```
|
||||||
|
|
||||||
|
拼接后的python字符串形如：
|
||||||
|
|
||||||
|
``` python
|
||||||
|
"Task: Search for doors, click doors on sale and filter by brands \"Mastercraft\".\nHistory steps: \n0. CLICK(box=[[352,102,786,139]], element_info='Search')\tLeft click on the search box located in the middle top of the screen next to the Menards logo.\n1. TYPE(box=[[352,102,786,139]], text='doors', element_info='Search')\tIn the search input box at the top, type 'doors'.\n2. CLICK(box=[[787,102,809,139]], element_info='SEARCH')\tLeft click on the magnifying glass icon next to the search bar to perform the search.\n3. SCROLL_DOWN(box=[[0,209,998,952]], step_count=5, element_info='[None]')\tScroll down the page to see the available doors.\n4. CLICK(box=[[280,708,710,809]], element_info='Doors on Sale')\tClick the \"Doors On Sale\" button in the middle of the page to view the doors that are currently on sale.\n(Platform: WIN)\n(Answer in Action-Operation format.)\n"
|
||||||
|
```
|
||||||
|
|
||||||
|
由于篇幅较长，若您想仔细了解每个字段的含义和表示，请参考[github](https://github.com/THUDM/CogAgent)。
|
||||||
|
|
||||||
|
## 先前的工作
|
||||||
|
|
||||||
|
在2023年11月，我们发布了CogAgent的第一代模型，现在，你可以在 [CogVLM&CogAgent官方仓库](https://github.com/THUDM/CogVLM)
|
||||||
|
找到相关代码和权重地址。
|
||||||
|
|
||||||
|
<div align="center">
|
||||||
|
<img src=https://raw.githubusercontent.com/THUDM/CogAgent/refs/heads/main/assets/cogagent_function_cn.jpg width=70% />
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<table>
|
||||||
|
<tr>
|
||||||
|
<td>
|
||||||
|
<h2> CogVLM </h2>
|
||||||
|
<p> 📖 Paper: <a href="https://arxiv.org/abs/2311.03079">CogVLM: Visual Expert for Pretrained Language Models</a></p>
|
||||||
|
<p><b>CogVLM</b> 是一个强大的开源视觉语言模型（VLM）。CogVLM-17B拥有100亿的视觉参数和70亿的语言参数，支持490*490分辨率的图像理解和多轮对话。</p>
|
||||||
|
<p><b>CogVLM-17B 17B在10个经典的跨模态基准测试中取得了最先进的性能</b>包括NoCaps, Flicker30k captioning, RefCOCO, RefCOCO+, RefCOCOg, Visual7W, GQA, ScienceQA, VizWiz VQA 和 TDIUC 基准测试。</p>
|
||||||
|
</td>
|
||||||
|
<td>
|
||||||
|
<h2> CogAgent </h2>
|
||||||
|
<p> 📖 Paper: <a href="https://arxiv.org/abs/2312.08914">CogAgent: A Visual Language Model for GUI Agents </a></p>
|
||||||
|
<p><b>CogAgent</b> 是一个基于CogVLM改进的开源视觉语言模型。CogAgent-18B拥有110亿的视觉参数和70亿的语言参数, <b>支持1120*1120分辨率的图像理解。在CogVLM的能力之上，它进一步拥有了GUI图像Agent的能力。</b></p>
|
||||||
|
<p> <b>CogAgent-18B 在9个经典的跨模态基准测试中实现了最先进的通用性能，</b>包括 VQAv2, OK-VQ, TextVQA, ST-VQA, ChartQA, infoVQA, DocVQA, MM-Vet, 和 POPE 测试基准。它在包括AITW和Mind2Web在内的GUI操作数据集上显著超越了现有的模型。</p>
|
||||||
|
</td>
|
||||||
|
</tr>
|
||||||
|
</table>
|
||||||
|
|
||||||
|
## 协议
|
||||||
|
|
||||||
|
模型权重的使用请遵循 [Model License](LICENSE)。
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,68 @@
|
||||||
|
{
|
||||||
|
"_name_or_path": "THUDM/cogagent-9b-20241220",
|
||||||
|
"add_bias_linear": false,
|
||||||
|
"add_qkv_bias": true,
|
||||||
|
"apply_query_key_layer_scaling": true,
|
||||||
|
"apply_residual_connection_post_layernorm": false,
|
||||||
|
"architectures": [
|
||||||
|
"ChatGLMForConditionalGeneration"
|
||||||
|
],
|
||||||
|
"attention_dropout": 0.0,
|
||||||
|
"attention_softmax_in_fp32": true,
|
||||||
|
"auto_map": {
|
||||||
|
"AutoConfig": "configuration_chatglm.ChatGLMConfig",
|
||||||
|
"AutoModel": "modeling_chatglm.ChatGLMForConditionalGeneration",
|
||||||
|
"AutoModelForCausalLM": "modeling_chatglm.ChatGLMForConditionalGeneration",
|
||||||
|
"AutoModelForSeq2SeqLM": "modeling_chatglm.ChatGLMForConditionalGeneration",
|
||||||
|
"AutoModelForSequenceClassification": "modeling_chatglm.ChatGLMForSequenceClassification"
|
||||||
|
},
|
||||||
|
"bias_dropout_fusion": true,
|
||||||
|
"boi_token_id": 151339,
|
||||||
|
"classifier_dropout": null,
|
||||||
|
"eoi_token_id": 151340,
|
||||||
|
"eos_token_id": [
|
||||||
|
151329,
|
||||||
|
151336,
|
||||||
|
151338
|
||||||
|
],
|
||||||
|
"ffn_hidden_size": 13696,
|
||||||
|
"fp32_residual_connection": false,
|
||||||
|
"hidden_dropout": 0.0,
|
||||||
|
"hidden_size": 4096,
|
||||||
|
"kv_channels": 128,
|
||||||
|
"layernorm_epsilon": 1e-05,
|
||||||
|
"model_type": "chatglm",
|
||||||
|
"multi_query_attention": true,
|
||||||
|
"multi_query_group_num": 2,
|
||||||
|
"num_attention_heads": 32,
|
||||||
|
"num_layers": 40,
|
||||||
|
"original_rope": false,
|
||||||
|
"pad_token_id": 151329,
|
||||||
|
"padded_vocab_size": 151552,
|
||||||
|
"post_layer_norm": true,
|
||||||
|
"pre_seq_len": null,
|
||||||
|
"prefix_projection": false,
|
||||||
|
"rmsnorm": true,
|
||||||
|
"rope_ratio": 1,
|
||||||
|
"rotary_percent": 0.5,
|
||||||
|
"seq_length": 8192,
|
||||||
|
"tie_word_embeddings": false,
|
||||||
|
"torch_dtype": "bfloat16",
|
||||||
|
"transformers_version": "4.48.0.dev0",
|
||||||
|
"use_cache": true,
|
||||||
|
"vision_config": {
|
||||||
|
"dropout_prob": 0.0,
|
||||||
|
"hidden_act": "gelu",
|
||||||
|
"hidden_size": 1792,
|
||||||
|
"image_size": 1120,
|
||||||
|
"in_channels": 3,
|
||||||
|
"intermediate_size": 15360,
|
||||||
|
"layer_norm_eps": 1e-06,
|
||||||
|
"num_heads": 16,
|
||||||
|
"num_hidden_layers": 63,
|
||||||
|
"num_positions": 6401,
|
||||||
|
"patch_size": 14,
|
||||||
|
"scaling_factor": 8
|
||||||
|
},
|
||||||
|
"vocab_size": 151552
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1,66 @@
|
||||||
|
from transformers import PretrainedConfig
|
||||||
|
|
||||||
|
|
||||||
|
class ChatGLMConfig(PretrainedConfig):
|
||||||
|
model_type = "chatglm"
|
||||||
|
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
num_layers=28,
|
||||||
|
padded_vocab_size=65024,
|
||||||
|
hidden_size=4096,
|
||||||
|
ffn_hidden_size=13696,
|
||||||
|
kv_channels=128,
|
||||||
|
num_attention_heads=32,
|
||||||
|
seq_length=2048,
|
||||||
|
hidden_dropout=0.0,
|
||||||
|
classifier_dropout=None,
|
||||||
|
attention_dropout=0.0,
|
||||||
|
layernorm_epsilon=1e-5,
|
||||||
|
rmsnorm=True,
|
||||||
|
apply_residual_connection_post_layernorm=False,
|
||||||
|
post_layer_norm=True,
|
||||||
|
add_bias_linear=False,
|
||||||
|
add_qkv_bias=False,
|
||||||
|
bias_dropout_fusion=True,
|
||||||
|
multi_query_attention=False,
|
||||||
|
multi_query_group_num=1,
|
||||||
|
rope_ratio=1,
|
||||||
|
apply_query_key_layer_scaling=True,
|
||||||
|
attention_softmax_in_fp32=True,
|
||||||
|
fp32_residual_connection=False,
|
||||||
|
pre_seq_len=None,
|
||||||
|
prefix_projection=False,
|
||||||
|
boi_token_id=None,
|
||||||
|
eoi_token_id=None,
|
||||||
|
**kwargs
|
||||||
|
):
|
||||||
|
self.num_layers = num_layers
|
||||||
|
self.vocab_size = padded_vocab_size
|
||||||
|
self.padded_vocab_size = padded_vocab_size
|
||||||
|
self.hidden_size = hidden_size
|
||||||
|
self.ffn_hidden_size = ffn_hidden_size
|
||||||
|
self.kv_channels = kv_channels
|
||||||
|
self.num_attention_heads = num_attention_heads
|
||||||
|
self.seq_length = seq_length
|
||||||
|
self.hidden_dropout = hidden_dropout
|
||||||
|
self.classifier_dropout = classifier_dropout
|
||||||
|
self.attention_dropout = attention_dropout
|
||||||
|
self.layernorm_epsilon = layernorm_epsilon
|
||||||
|
self.rmsnorm = rmsnorm
|
||||||
|
self.apply_residual_connection_post_layernorm = apply_residual_connection_post_layernorm
|
||||||
|
self.post_layer_norm = post_layer_norm
|
||||||
|
self.add_bias_linear = add_bias_linear
|
||||||
|
self.add_qkv_bias = add_qkv_bias
|
||||||
|
self.bias_dropout_fusion = bias_dropout_fusion
|
||||||
|
self.multi_query_attention = multi_query_attention
|
||||||
|
self.multi_query_group_num = multi_query_group_num
|
||||||
|
self.rope_ratio = rope_ratio
|
||||||
|
self.apply_query_key_layer_scaling = apply_query_key_layer_scaling
|
||||||
|
self.attention_softmax_in_fp32 = attention_softmax_in_fp32
|
||||||
|
self.fp32_residual_connection = fp32_residual_connection
|
||||||
|
self.pre_seq_len = pre_seq_len
|
||||||
|
self.prefix_projection = prefix_projection
|
||||||
|
self.boi_token_id = boi_token_id
|
||||||
|
self.eoi_token_id = eoi_token_id
|
||||||
|
super().__init__(**kwargs)
|
||||||
|
|
@ -0,0 +1,13 @@
|
||||||
|
{
|
||||||
|
"do_sample": true,
|
||||||
|
"eos_token_id": [
|
||||||
|
151329,
|
||||||
|
151336,
|
||||||
|
151338
|
||||||
|
],
|
||||||
|
"max_length": 8192,
|
||||||
|
"pad_token_id": 151329,
|
||||||
|
"temperature": 0.8,
|
||||||
|
"top_p": 0.8,
|
||||||
|
"transformers_version": "4.48.0.dev0"
|
||||||
|
}
|
||||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
|
|
@ -0,0 +1,374 @@
|
||||||
|
import regex as re
|
||||||
|
import base64
|
||||||
|
import os
|
||||||
|
import json
|
||||||
|
import tiktoken
|
||||||
|
import torch
|
||||||
|
from torch import TensorType
|
||||||
|
from typing import List, Optional, Union, Dict, Any
|
||||||
|
from torchvision import transforms
|
||||||
|
from transformers import PreTrainedTokenizer
|
||||||
|
from transformers.utils import PaddingStrategy
|
||||||
|
from transformers.tokenization_utils_base import EncodedInput, BatchEncoding
|
||||||
|
|
||||||
|
|
||||||
|
class ChatGLM4Tokenizer(PreTrainedTokenizer):
|
||||||
|
vocab_files_names = {"vocab_file": "tokenizer.model"}
|
||||||
|
model_input_names = ["input_ids", "attention_mask", "position_ids"]
|
||||||
|
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
vocab_file,
|
||||||
|
padding_side="left",
|
||||||
|
clean_up_tokenization_spaces=False,
|
||||||
|
encode_special_tokens=False,
|
||||||
|
image_size=None,
|
||||||
|
**kwargs,
|
||||||
|
):
|
||||||
|
self.name = "GLM4Tokenizer"
|
||||||
|
self.vocab_file = vocab_file
|
||||||
|
pat_str = "(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?\\p{L}+|\\p{N}{1,3}| ?[^\\s\\p{L}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+"
|
||||||
|
self.pat_str = re.compile(pat_str)
|
||||||
|
self.encode_special_tokens = encode_special_tokens
|
||||||
|
self.image_size = image_size
|
||||||
|
|
||||||
|
mergeable_ranks = {}
|
||||||
|
with open(vocab_file) as f:
|
||||||
|
for line in f:
|
||||||
|
token, rank = line.strip().split()
|
||||||
|
rank = int(rank)
|
||||||
|
token = base64.b64decode(token)
|
||||||
|
mergeable_ranks[token] = rank
|
||||||
|
|
||||||
|
self.mergeable_ranks = mergeable_ranks
|
||||||
|
|
||||||
|
self.tokenizer = tiktoken.Encoding(
|
||||||
|
name="my_tokenizer",
|
||||||
|
pat_str=pat_str,
|
||||||
|
mergeable_ranks=mergeable_ranks,
|
||||||
|
special_tokens={},
|
||||||
|
)
|
||||||
|
self.decoder = {rank: token for token, rank in mergeable_ranks.items()}
|
||||||
|
self.n_words = len(self.decoder)
|
||||||
|
|
||||||
|
super().__init__(
|
||||||
|
padding_side=padding_side,
|
||||||
|
clean_up_tokenization_spaces=clean_up_tokenization_spaces,
|
||||||
|
**kwargs,
|
||||||
|
)
|
||||||
|
|
||||||
|
@property
|
||||||
|
def vocab_size(self):
|
||||||
|
return self.n_words
|
||||||
|
|
||||||
|
def get_vocab(self):
|
||||||
|
"""Returns vocab as a dict"""
|
||||||
|
vocab = {self._convert_id_to_token(i): i for i in range(self.vocab_size)}
|
||||||
|
vocab.update(self.added_tokens_encoder)
|
||||||
|
return vocab
|
||||||
|
|
||||||
|
def convert_tokens_to_string(self, tokens: List[Union[bytes, str, int]]) -> str:
|
||||||
|
"""
|
||||||
|
Converts a sequence of tokens in a single string.
|
||||||
|
"""
|
||||||
|
text = ""
|
||||||
|
temp = b""
|
||||||
|
for t in tokens:
|
||||||
|
if isinstance(t, int):
|
||||||
|
t = chr(t)
|
||||||
|
if isinstance(t, str):
|
||||||
|
if temp:
|
||||||
|
text += temp.decode("utf-8", errors="replace")
|
||||||
|
elif isinstance(t, bytes):
|
||||||
|
temp += t
|
||||||
|
else:
|
||||||
|
raise TypeError("token should only be of type int, bytes or str")
|
||||||
|
if temp:
|
||||||
|
text += temp.decode("utf-8", errors="replace")
|
||||||
|
return text
|
||||||
|
|
||||||
|
def _tokenize(self, text, **kwargs):
|
||||||
|
tokens = []
|
||||||
|
ids = self.tokenizer.encode(text)
|
||||||
|
for t in ids:
|
||||||
|
tokens.append(self.decoder[t])
|
||||||
|
return tokens
|
||||||
|
|
||||||
|
def _convert_token_to_id(self, token):
|
||||||
|
"""Converts a token (str) in an id using the vocab."""
|
||||||
|
return self.mergeable_ranks[token]
|
||||||
|
|
||||||
|
def _convert_id_to_token(self, index):
|
||||||
|
"""Converts an index (integer) in a token (str) using the vocab."""
|
||||||
|
return self.decoder.get(index, "")
|
||||||
|
|
||||||
|
def save_vocabulary(self, save_directory, filename_prefix=None):
|
||||||
|
"""
|
||||||
|
Save the vocabulary and special tokens file to a directory.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
save_directory (`str`):
|
||||||
|
The directory in which to save the vocabulary.
|
||||||
|
filename_prefix (`str`, *optional*):
|
||||||
|
An optional prefix to add to the named of the saved files.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
`Tuple(str)`: Paths to the files saved.
|
||||||
|
"""
|
||||||
|
if os.path.isdir(save_directory):
|
||||||
|
vocab_file = os.path.join(
|
||||||
|
save_directory, self.vocab_files_names["vocab_file"]
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
vocab_file = save_directory
|
||||||
|
|
||||||
|
with open(self.vocab_file, "rb") as fin:
|
||||||
|
proto_str = fin.read()
|
||||||
|
|
||||||
|
with open(vocab_file, "wb") as writer:
|
||||||
|
writer.write(proto_str)
|
||||||
|
|
||||||
|
return (vocab_file,)
|
||||||
|
|
||||||
|
def get_prefix_tokens(self):
|
||||||
|
prefix_tokens = [
|
||||||
|
self.convert_tokens_to_ids("[gMASK]"),
|
||||||
|
self.convert_tokens_to_ids("<sop>"),
|
||||||
|
]
|
||||||
|
return prefix_tokens
|
||||||
|
|
||||||
|
def build_single_message(
|
||||||
|
self, role, metadata, message, tokenize=True, message_prefix=None
|
||||||
|
):
|
||||||
|
assert role in ["system", "user", "assistant", "observation"], role
|
||||||
|
if tokenize:
|
||||||
|
role_tokens = [
|
||||||
|
self.convert_tokens_to_ids(f"<|{role}|>")
|
||||||
|
] + self.tokenizer.encode(f"{metadata}\n", disallowed_special=())
|
||||||
|
message_tokens = self.tokenizer.encode(message, disallowed_special=())
|
||||||
|
if message_prefix is not None:
|
||||||
|
message_tokens = message_prefix + message_tokens
|
||||||
|
tokens = role_tokens + message_tokens
|
||||||
|
return tokens
|
||||||
|
else:
|
||||||
|
return str(f"<|{role}|>{metadata}\n{message}")
|
||||||
|
|
||||||
|
def apply_chat_template(
|
||||||
|
self,
|
||||||
|
conversation: Union[
|
||||||
|
List[Dict[str, str]], List[List[Dict[str, str]]], "Conversation"
|
||||||
|
],
|
||||||
|
add_generation_prompt: bool = False,
|
||||||
|
tokenize: bool = True,
|
||||||
|
padding: bool = False,
|
||||||
|
truncation: bool = False,
|
||||||
|
max_length: Optional[int] = None,
|
||||||
|
return_tensors: Optional[Union[str, TensorType]] = None,
|
||||||
|
return_dict: bool = False,
|
||||||
|
tokenizer_kwargs: Optional[Dict[str, Any]] = None,
|
||||||
|
add_special_tokens: bool = True,
|
||||||
|
**kwargs,
|
||||||
|
) -> Union[str, List[int], List[str], List[List[int]], BatchEncoding]:
|
||||||
|
if return_dict and not tokenize:
|
||||||
|
raise ValueError(
|
||||||
|
"`return_dict=True` is incompatible with `tokenize=False`, because there is no dict "
|
||||||
|
"of tokenizer outputs to return."
|
||||||
|
)
|
||||||
|
|
||||||
|
def handle_single_conversation(conversation):
|
||||||
|
input_ids = self.get_prefix_tokens() if add_special_tokens else []
|
||||||
|
input_message = "[gMASK]<sop>" if add_special_tokens else ""
|
||||||
|
input_image = None
|
||||||
|
transform = transforms.Compose(
|
||||||
|
[
|
||||||
|
transforms.Resize(
|
||||||
|
(self.image_size, self.image_size),
|
||||||
|
interpolation=transforms.InterpolationMode.BICUBIC,
|
||||||
|
),
|
||||||
|
transforms.ToTensor(),
|
||||||
|
transforms.Normalize(
|
||||||
|
(0.48145466, 0.4578275, 0.40821073),
|
||||||
|
(0.26862954, 0.26130258, 0.27577711),
|
||||||
|
),
|
||||||
|
]
|
||||||
|
)
|
||||||
|
for item in conversation:
|
||||||
|
message = ""
|
||||||
|
message_prefix = None
|
||||||
|
if item.get("image"):
|
||||||
|
assert input_image is None, "Multiple images are not supported"
|
||||||
|
input_image = transform(item["image"])
|
||||||
|
message_prefix = self.convert_tokens_to_ids(
|
||||||
|
["<|begin_of_image|>", "<|endoftext|>", "<|end_of_image|>"]
|
||||||
|
)
|
||||||
|
if item.get("content"):
|
||||||
|
message += item["content"]
|
||||||
|
if message or message_prefix:
|
||||||
|
input = self.build_single_message(
|
||||||
|
item["role"],
|
||||||
|
item.get("metadata", ""),
|
||||||
|
message,
|
||||||
|
tokenize=tokenize,
|
||||||
|
message_prefix=message_prefix,
|
||||||
|
)
|
||||||
|
if tokenize:
|
||||||
|
input_ids.extend(input)
|
||||||
|
else:
|
||||||
|
input_message += input
|
||||||
|
if add_generation_prompt:
|
||||||
|
if tokenize:
|
||||||
|
input_ids.extend([self.convert_tokens_to_ids("<|assistant|>"), 198]) # 198 is \n in the vocab
|
||||||
|
else:
|
||||||
|
input_message += "<|assistant|>\n"
|
||||||
|
return {
|
||||||
|
"input": input_ids if tokenize else input_message,
|
||||||
|
"image": input_image,
|
||||||
|
}
|
||||||
|
|
||||||
|
# Main logic to handle different conversation formats
|
||||||
|
if isinstance(conversation, list) and all(
|
||||||
|
isinstance(i, dict) for i in conversation
|
||||||
|
):
|
||||||
|
result = handle_single_conversation(conversation)
|
||||||
|
input_ids = result["input"]
|
||||||
|
input_images = [result["image"]]
|
||||||
|
elif isinstance(conversation, list) and all(
|
||||||
|
isinstance(i, list) for i in conversation
|
||||||
|
):
|
||||||
|
results = [handle_single_conversation(c) for c in conversation]
|
||||||
|
input_ids = [item["input"] for item in results]
|
||||||
|
input_images = [item["image"] for item in results]
|
||||||
|
elif hasattr(conversation, "messages"):
|
||||||
|
result = handle_single_conversation(conversation.messages)
|
||||||
|
input_ids = result["input"]
|
||||||
|
input_images = [result["image"]]
|
||||||
|
else:
|
||||||
|
raise ValueError("Invalid conversation format")
|
||||||
|
|
||||||
|
if tokenize:
|
||||||
|
output = self.batch_encode_plus(
|
||||||
|
[input_ids] if isinstance(input_ids[0], int) else input_ids,
|
||||||
|
padding=padding,
|
||||||
|
truncation=truncation,
|
||||||
|
max_length=max_length,
|
||||||
|
return_tensors=return_tensors,
|
||||||
|
is_split_into_words=True,
|
||||||
|
add_special_tokens=False,
|
||||||
|
)
|
||||||
|
if return_dict:
|
||||||
|
found_image = False
|
||||||
|
for image in input_images:
|
||||||
|
if image is not None:
|
||||||
|
found_image = True
|
||||||
|
break
|
||||||
|
if found_image:
|
||||||
|
output["images"] = torch.stack(input_images)
|
||||||
|
return output
|
||||||
|
else:
|
||||||
|
return output["input_ids"]
|
||||||
|
else:
|
||||||
|
return input_ids
|
||||||
|
|
||||||
|
def build_inputs_with_special_tokens(
|
||||||
|
self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None
|
||||||
|
) -> List[int]:
|
||||||
|
"""
|
||||||
|
Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and
|
||||||
|
adding special tokens. A BERT sequence has the following format:
|
||||||
|
|
||||||
|
- single sequence: `[CLS] X [SEP]`
|
||||||
|
- pair of sequences: `[CLS] A [SEP] B [SEP]`
|
||||||
|
|
||||||
|
Args:
|
||||||
|
token_ids_0 (`List[int]`):
|
||||||
|
List of IDs to which the special tokens will be added.
|
||||||
|
token_ids_1 (`List[int]`, *optional*):
|
||||||
|
Optional second list of IDs for sequence pairs.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
`List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens.
|
||||||
|
"""
|
||||||
|
prefix_tokens = self.get_prefix_tokens()
|
||||||
|
token_ids_0 = prefix_tokens + token_ids_0
|
||||||
|
if token_ids_1 is not None:
|
||||||
|
token_ids_0 = (
|
||||||
|
token_ids_0 + token_ids_1 + [self.convert_tokens_to_ids("<eos>")]
|
||||||
|
)
|
||||||
|
return token_ids_0
|
||||||
|
|
||||||
|
def _pad(
|
||||||
|
self,
|
||||||
|
encoded_inputs: Union[Dict[str, EncodedInput], BatchEncoding],
|
||||||
|
max_length: Optional[int] = None,
|
||||||
|
padding_strategy: PaddingStrategy = PaddingStrategy.DO_NOT_PAD,
|
||||||
|
pad_to_multiple_of: Optional[int] = None,
|
||||||
|
padding_side: Optional[str] = None,
|
||||||
|
return_attention_mask: Optional[bool] = None,
|
||||||
|
) -> dict:
|
||||||
|
"""
|
||||||
|
Pad encoded inputs (on left/right and up to predefined length or max length in the batch)
|
||||||
|
|
||||||
|
Args:
|
||||||
|
encoded_inputs:
|
||||||
|
Dictionary of tokenized inputs (`List[int]`) or batch of tokenized inputs (`List[List[int]]`).
|
||||||
|
max_length: maximum length of the returned list and optionally padding length (see below).
|
||||||
|
Will truncate by taking into account the special tokens.
|
||||||
|
padding_strategy: PaddingStrategy to use for padding.
|
||||||
|
|
||||||
|
- PaddingStrategy.LONGEST Pad to the longest sequence in the batch
|
||||||
|
- PaddingStrategy.MAX_LENGTH: Pad to the max length (default)
|
||||||
|
- PaddingStrategy.DO_NOT_PAD: Do not pad
|
||||||
|
The tokenizer padding sides are defined in self.padding_side:
|
||||||
|
|
||||||
|
- 'left': pads on the left of the sequences
|
||||||
|
- 'right': pads on the right of the sequences
|
||||||
|
pad_to_multiple_of: (optional) Integer if set will pad the sequence to a multiple of the provided value.
|
||||||
|
This is especially useful to enable the use of Tensor Core on NVIDIA hardware with compute capability
|
||||||
|
`>= 7.5` (Volta).
|
||||||
|
return_attention_mask:
|
||||||
|
(optional) Set to False to avoid returning attention mask (default: set to model specifics)
|
||||||
|
"""
|
||||||
|
# Load from model defaults
|
||||||
|
assert self.padding_side == "left"
|
||||||
|
|
||||||
|
required_input = encoded_inputs[self.model_input_names[0]]
|
||||||
|
seq_length = len(required_input)
|
||||||
|
|
||||||
|
if padding_strategy == PaddingStrategy.LONGEST:
|
||||||
|
max_length = len(required_input)
|
||||||
|
|
||||||
|
if (
|
||||||
|
max_length is not None
|
||||||
|
and pad_to_multiple_of is not None
|
||||||
|
and (max_length % pad_to_multiple_of != 0)
|
||||||
|
):
|
||||||
|
max_length = ((max_length // pad_to_multiple_of) + 1) * pad_to_multiple_of
|
||||||
|
|
||||||
|
needs_to_be_padded = (
|
||||||
|
padding_strategy != PaddingStrategy.DO_NOT_PAD
|
||||||
|
and len(required_input) != max_length
|
||||||
|
)
|
||||||
|
|
||||||
|
# Initialize attention mask if not present.
|
||||||
|
if "attention_mask" not in encoded_inputs:
|
||||||
|
encoded_inputs["attention_mask"] = [1] * seq_length
|
||||||
|
|
||||||
|
if "position_ids" not in encoded_inputs:
|
||||||
|
encoded_inputs["position_ids"] = list(range(seq_length))
|
||||||
|
|
||||||
|
if needs_to_be_padded:
|
||||||
|
difference = max_length - len(required_input)
|
||||||
|
|
||||||
|
if "attention_mask" in encoded_inputs:
|
||||||
|
encoded_inputs["attention_mask"] = [0] * difference + encoded_inputs[
|
||||||
|
"attention_mask"
|
||||||
|
]
|
||||||
|
if "position_ids" in encoded_inputs:
|
||||||
|
encoded_inputs["position_ids"] = [0] * difference + encoded_inputs[
|
||||||
|
"position_ids"
|
||||||
|
]
|
||||||
|
encoded_inputs[self.model_input_names[0]] = [
|
||||||
|
self.pad_token_id
|
||||||
|
] * difference + required_input
|
||||||
|
|
||||||
|
return encoded_inputs
|
||||||
Binary file not shown.
|
|
@ -0,0 +1,134 @@
|
||||||
|
{
|
||||||
|
"auto_map": {
|
||||||
|
"AutoTokenizer": [
|
||||||
|
"tokenization_chatglm.ChatGLM4Tokenizer",
|
||||||
|
null
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"added_tokens_decoder": {
|
||||||
|
"151329": {
|
||||||
|
"content": "<|endoftext|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151330": {
|
||||||
|
"content": "[MASK]",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151331": {
|
||||||
|
"content": "[gMASK]",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151332": {
|
||||||
|
"content": "[sMASK]",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151333": {
|
||||||
|
"content": "<sop>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151334": {
|
||||||
|
"content": "<eop>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151335": {
|
||||||
|
"content": "<|system|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151336": {
|
||||||
|
"content": "<|user|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151337": {
|
||||||
|
"content": "<|assistant|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151338": {
|
||||||
|
"content": "<|observation|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151339": {
|
||||||
|
"content": "<|begin_of_image|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151340": {
|
||||||
|
"content": "<|end_of_image|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151341": {
|
||||||
|
"content": "<|begin_of_video|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151342": {
|
||||||
|
"content": "<|end_of_video|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"additional_special_tokens": ["<|endoftext|>", "[MASK]", "[gMASK]", "[sMASK]", "<sop>", "<eop>", "<|system|>",
|
||||||
|
"<|user|>", "<|assistant|>", "<|observation|>", "<|begin_of_image|>", "<|end_of_image|>",
|
||||||
|
"<|begin_of_video|>", "<|end_of_video|>"],
|
||||||
|
"clean_up_tokenization_spaces": false,
|
||||||
|
"do_lower_case": false,
|
||||||
|
"eos_token": "<|endoftext|>",
|
||||||
|
"pad_token": "<|endoftext|>",
|
||||||
|
"model_max_length": 8192,
|
||||||
|
"padding_side": "left",
|
||||||
|
"remove_space": false,
|
||||||
|
"tokenizer_class": "ChatGLM4Tokenizer",
|
||||||
|
"image_size": 1120
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1,199 @@
|
||||||
|
import torch
|
||||||
|
from torch import nn
|
||||||
|
from argparse import Namespace
|
||||||
|
import torch.nn.functional as F
|
||||||
|
from transformers.activations import ACT2FN
|
||||||
|
import math
|
||||||
|
from torch.nn import LayerNorm
|
||||||
|
|
||||||
|
|
||||||
|
def standard_attention(
|
||||||
|
query_layer, key_layer, value_layer, scaling_attention_score=True
|
||||||
|
):
|
||||||
|
if scaling_attention_score:
|
||||||
|
query_layer = query_layer / math.sqrt(query_layer.shape[-1])
|
||||||
|
attention_scores = torch.matmul(query_layer, key_layer.transpose(-1, -2))
|
||||||
|
attention_probs = F.softmax(attention_scores, dim=-1)
|
||||||
|
context_layer = torch.matmul(attention_probs, value_layer)
|
||||||
|
return context_layer
|
||||||
|
|
||||||
|
|
||||||
|
def attention_fn_default(
|
||||||
|
query_layer, key_layer, value_layer, scaling_attention_score=True
|
||||||
|
):
|
||||||
|
if int(torch.__version__.split(".")[0]) >= 2 and scaling_attention_score:
|
||||||
|
attn_output = torch.nn.functional.scaled_dot_product_attention(
|
||||||
|
query_layer,
|
||||||
|
key_layer,
|
||||||
|
value_layer,
|
||||||
|
attn_mask=None,
|
||||||
|
dropout_p=0.0,
|
||||||
|
is_causal=False,
|
||||||
|
)
|
||||||
|
return attn_output
|
||||||
|
else:
|
||||||
|
return standard_attention(
|
||||||
|
query_layer,
|
||||||
|
key_layer,
|
||||||
|
value_layer,
|
||||||
|
scaling_attention_score=scaling_attention_score,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class PatchEmbedding(nn.Module):
|
||||||
|
def __init__(self, config):
|
||||||
|
super().__init__()
|
||||||
|
self.proj = nn.Conv2d(
|
||||||
|
config.in_channels,
|
||||||
|
config.hidden_size,
|
||||||
|
kernel_size=config.patch_size,
|
||||||
|
stride=config.patch_size,
|
||||||
|
)
|
||||||
|
self.cls_embedding = nn.Parameter(torch.zeros(1, config.hidden_size))
|
||||||
|
self.position_embedding = nn.Embedding(config.num_positions, config.hidden_size)
|
||||||
|
|
||||||
|
def forward(self, images: "tensor(B, C, H, W)") -> "tensor(B, L, D)":
|
||||||
|
x = self.proj(images)
|
||||||
|
x = x.flatten(2).transpose(1, 2)
|
||||||
|
x += self.position_embedding.weight[1:, :].unsqueeze(0)
|
||||||
|
return x
|
||||||
|
|
||||||
|
|
||||||
|
class Attention(nn.Module):
|
||||||
|
def __init__(self, config):
|
||||||
|
super().__init__()
|
||||||
|
self.num_heads = config.num_heads
|
||||||
|
head_dim = config.hidden_size // config.num_heads
|
||||||
|
self.scale = head_dim**-0.5
|
||||||
|
self.query_key_value = nn.Linear(config.hidden_size, config.hidden_size * 3)
|
||||||
|
self.dense = nn.Linear(config.hidden_size, config.hidden_size)
|
||||||
|
self.output_dropout = torch.nn.Dropout(config.dropout_prob)
|
||||||
|
|
||||||
|
def forward(self, x: "tensor(B, L, D)") -> "tensor(B, L, D)":
|
||||||
|
B, L, _ = x.shape
|
||||||
|
qkv = self.query_key_value(x)
|
||||||
|
qkv = qkv.reshape(B, L, self.num_heads, 3, -1).permute(
|
||||||
|
3, 0, 2, 1, 4
|
||||||
|
) # 3, B, H, L, D
|
||||||
|
q, k, v = qkv[0], qkv[1], qkv[2]
|
||||||
|
|
||||||
|
out = attention_fn_default(q, k, v)
|
||||||
|
output = self.dense(out.transpose(1, 2).reshape(B, L, -1))
|
||||||
|
output = self.output_dropout(output)
|
||||||
|
return output
|
||||||
|
|
||||||
|
def attention(self, q, k, v):
|
||||||
|
attn_weights = torch.matmul(q * self.scale, k.transpose(-2, -1))
|
||||||
|
attn_weights = attn_weights.softmax(dim=-1)
|
||||||
|
output = torch.matmul(attn_weights, v)
|
||||||
|
return output
|
||||||
|
|
||||||
|
|
||||||
|
class MLP(nn.Module):
|
||||||
|
def __init__(self, config):
|
||||||
|
super().__init__()
|
||||||
|
self.config = config
|
||||||
|
self.activation_fn = ACT2FN[config.hidden_act]
|
||||||
|
self.fc1 = nn.Linear(config.hidden_size, config.intermediate_size)
|
||||||
|
self.fc2 = nn.Linear(config.intermediate_size, config.hidden_size)
|
||||||
|
|
||||||
|
def forward(self, x: torch.Tensor) -> torch.Tensor:
|
||||||
|
x = self.fc1(x)
|
||||||
|
x = self.activation_fn(x)
|
||||||
|
x = self.fc2(x)
|
||||||
|
return x
|
||||||
|
|
||||||
|
|
||||||
|
class TransformerLayer(nn.Module):
|
||||||
|
def __init__(self, config):
|
||||||
|
super().__init__()
|
||||||
|
self.input_layernorm = LayerNorm(config.hidden_size, eps=config.layer_norm_eps)
|
||||||
|
self.attention = Attention(config)
|
||||||
|
self.mlp = MLP(config)
|
||||||
|
self.post_attention_layernorm = LayerNorm(
|
||||||
|
config.hidden_size, eps=config.layer_norm_eps
|
||||||
|
)
|
||||||
|
|
||||||
|
def forward(self, hidden_states):
|
||||||
|
attention_input = hidden_states
|
||||||
|
attention_output = self.input_layernorm(self.attention(attention_input))
|
||||||
|
hidden_states = attention_input + attention_output
|
||||||
|
mlp_input = hidden_states
|
||||||
|
mlp_output = self.post_attention_layernorm(self.mlp(mlp_input)).to(
|
||||||
|
mlp_input.device
|
||||||
|
)
|
||||||
|
output = mlp_input + mlp_output
|
||||||
|
return output
|
||||||
|
|
||||||
|
|
||||||
|
class Transformer(nn.Module):
|
||||||
|
def __init__(self, config):
|
||||||
|
super().__init__()
|
||||||
|
self.layers = nn.ModuleList(
|
||||||
|
[TransformerLayer(config) for _ in range(config.num_hidden_layers)]
|
||||||
|
)
|
||||||
|
|
||||||
|
def forward(self, hidden_states):
|
||||||
|
for layer_module in self.layers:
|
||||||
|
hidden_states = layer_module(hidden_states)
|
||||||
|
return hidden_states
|
||||||
|
|
||||||
|
|
||||||
|
class GLU(nn.Module):
|
||||||
|
def __init__(self, config, in_features):
|
||||||
|
super().__init__()
|
||||||
|
self.linear_proj = nn.Linear(in_features, config.hidden_size, bias=False)
|
||||||
|
self.norm1 = nn.LayerNorm(config.hidden_size)
|
||||||
|
self.act1 = nn.GELU()
|
||||||
|
self.act2 = nn.functional.silu
|
||||||
|
self.dense_h_to_4h = nn.Linear(
|
||||||
|
config.hidden_size, config.ffn_hidden_size, bias=False
|
||||||
|
)
|
||||||
|
self.gate_proj = nn.Linear(
|
||||||
|
config.hidden_size, config.ffn_hidden_size, bias=False
|
||||||
|
)
|
||||||
|
self.dense_4h_to_h = nn.Linear(
|
||||||
|
config.ffn_hidden_size, config.hidden_size, bias=False
|
||||||
|
)
|
||||||
|
|
||||||
|
def forward(self, x):
|
||||||
|
x = self.linear_proj(x)
|
||||||
|
x = self.act1(self.norm1(x))
|
||||||
|
x = self.act2(self.gate_proj(x)) * self.dense_h_to_4h(x)
|
||||||
|
x = self.dense_4h_to_h(x)
|
||||||
|
return x
|
||||||
|
|
||||||
|
|
||||||
|
class EVA2CLIPModel(nn.Module):
|
||||||
|
def __init__(self, config):
|
||||||
|
super().__init__()
|
||||||
|
vision_config = Namespace(**config.vision_config)
|
||||||
|
self.patch_embedding = PatchEmbedding(vision_config)
|
||||||
|
self.transformer = Transformer(vision_config)
|
||||||
|
self.linear_proj = GLU(config, in_features=config.hidden_size)
|
||||||
|
self.conv = nn.Conv2d(
|
||||||
|
in_channels=vision_config.hidden_size,
|
||||||
|
out_channels=config.hidden_size,
|
||||||
|
kernel_size=2,
|
||||||
|
stride=2,
|
||||||
|
)
|
||||||
|
self.boi = nn.Parameter(torch.zeros(1, 1, config.hidden_size))
|
||||||
|
self.eoi = nn.Parameter(torch.zeros(1, 1, config.hidden_size))
|
||||||
|
self.scaling_factor = vision_config.scaling_factor
|
||||||
|
|
||||||
|
def forward(self, images: "tensor(B, C, H, W)") -> "tensor(B, L, D)":
|
||||||
|
x = self.patch_embedding(images)
|
||||||
|
x = self.transformer(x)
|
||||||
|
|
||||||
|
b, s, h = x.shape
|
||||||
|
grid_size = int(s**0.5)
|
||||||
|
x = x.view(b, grid_size, grid_size, h).permute(0, 3, 1, 2)
|
||||||
|
x = self.conv(x)
|
||||||
|
|
||||||
|
x = x.flatten(2).transpose(1, 2)
|
||||||
|
x = self.linear_proj(x)
|
||||||
|
boi = self.boi.expand(x.shape[0], -1, -1).to(x.device)
|
||||||
|
eoi = self.eoi.expand(x.shape[0], -1, -1).to(x.device)
|
||||||
|
x = torch.cat((boi, x, eoi), dim=1)
|
||||||
|
|
||||||
|
return x
|
||||||
Loading…
Reference in New Issue