first commit
This commit is contained in:
xxl
parent
b04ad6c1c6
commit
c6563764ed
389
README.md
389
README.md
|
|
@ -1,3 +1,388 @@
|
|||
# bge-reranker-v2-minicpm-layerwise_a13590573183266816374692
|
||||
---
|
||||
license: apache-2.0
|
||||
pipeline_tag: text-classification
|
||||
tags:
|
||||
- transformers
|
||||
- sentence-transformers
|
||||
language:
|
||||
- multilingual
|
||||
---
|
||||
|
||||
bge-reranker-v2-minicpm-layerwise
|
||||
# Reranker
|
||||
|
||||
**More details please refer to our Github: [FlagEmbedding](https://github.com/FlagOpen/FlagEmbedding/tree/master).**
|
||||
|
||||
- [Model List](#model-list)
|
||||
- [Usage](#usage)
|
||||
- [Fine-tuning](#fine-tune)
|
||||
- [Evaluation](#evaluation)
|
||||
- [Citation](#citation)
|
||||
|
||||
Different from embedding model, reranker uses question and document as input and directly output similarity instead of embedding.
|
||||
You can get a relevance score by inputting query and passage to the reranker.
|
||||
And the score can be mapped to a float value in [0,1] by sigmoid function.
|
||||
|
||||
|
||||
## Model List
|
||||
|
||||
| Model | Base model | Language | layerwise | feature |
|
||||
|:--------------------------------------------------------------------------|:--------:|:-----------------------------------------------------------------------------------------------------------------------------------:|:----------------------------------------------------------------------------------------------:|:----------------------------------------------------------------------------------------------:|
|
||||
| [BAAI/bge-reranker-base](https://huggingface.co/BAAI/bge-reranker-base) | [xlm-roberta-base](https://huggingface.co/xlm-roberta-base) | Chinese and English | - | Lightweight reranker model, easy to deploy, with fast inference. |
|
||||
| [BAAI/bge-reranker-large](https://huggingface.co/BAAI/bge-reranker-large) | [xlm-roberta-large](https://huggingface.co/FacebookAI/xlm-roberta-large) | Chinese and English | - | Lightweight reranker model, easy to deploy, with fast inference. |
|
||||
| [BAAI/bge-reranker-v2-m3](https://huggingface.co/BAAI/bge-reranker-v2-m3) | [bge-m3](https://huggingface.co/BAAI/bge-m3) | Multilingual | - | Lightweight reranker model, possesses strong multilingual capabilities, easy to deploy, with fast inference. |
|
||||
| [BAAI/bge-reranker-v2-gemma](https://huggingface.co/BAAI/bge-reranker-v2-gemma) | [gemma-2b](https://huggingface.co/google/gemma-2b) | Multilingual | - | Suitable for multilingual contexts, performs well in both English proficiency and multilingual capabilities. |
|
||||
| [BAAI/bge-reranker-v2-minicpm-layerwise](https://huggingface.co/BAAI/bge-reranker-v2-minicpm-layerwise) | [MiniCPM-2B-dpo-bf16](https://huggingface.co/openbmb/MiniCPM-2B-dpo-bf16) | Multilingual | 8-40 | Suitable for multilingual contexts, performs well in both English and Chinese proficiency, allows freedom to select layers for output, facilitating accelerated inference. |
|
||||
|
||||
|
||||
You can select the model according your senario and resource.
|
||||
- For **multilingual**, utilize [BAAI/bge-reranker-v2-m3](https://huggingface.co/BAAI/bge-reranker-v2-m3) and [BAAI/bge-reranker-v2-gemma](https://huggingface.co/BAAI/bge-reranker-v2-gemma)
|
||||
|
||||
- For **Chinese or English**, utilize [BAAI/bge-reranker-v2-m3](https://huggingface.co/BAAI/bge-reranker-v2-m3) and [BAAI/bge-reranker-v2-minicpm-layerwise](https://huggingface.co/BAAI/bge-reranker-v2-minicpm-layerwise).
|
||||
|
||||
- For **efficiency**, utilize [BAAI/bge-reranker-v2-m3](https://huggingface.co/BAAI/bge-reranker-v2-m3) and the low layer of [BAAI/bge-reranker-v2-minicpm-layerwise](https://huggingface.co/BAAI/bge-reranker-v2-minicpm-layerwise).
|
||||
|
||||
- For better performance, recommand [BAAI/bge-reranker-v2-minicpm-layerwise](https://huggingface.co/BAAI/bge-reranker-v2-minicpm-layerwise) and [BAAI/bge-reranker-v2-gemma](https://huggingface.co/BAAI/bge-reranker-v2-gemma)
|
||||
|
||||
## Usage
|
||||
### Using FlagEmbedding
|
||||
|
||||
```
|
||||
pip install -U FlagEmbedding
|
||||
```
|
||||
|
||||
#### For normal reranker (bge-reranker-base / bge-reranker-large / bge-reranker-v2-m3 )
|
||||
|
||||
Get relevance scores (higher scores indicate more relevance):
|
||||
|
||||
```python
|
||||
from FlagEmbedding import FlagReranker
|
||||
reranker = FlagReranker('BAAI/bge-reranker-v2-m3', use_fp16=True) # Setting use_fp16 to True speeds up computation with a slight performance degradation
|
||||
|
||||
score = reranker.compute_score(['query', 'passage'])
|
||||
print(score) # -5.65234375
|
||||
|
||||
# You can map the scores into 0-1 by set "normalize=True", which will apply sigmoid function to the score
|
||||
score = reranker.compute_score(['query', 'passage'], normalize=True)
|
||||
print(score) # 0.003497010252573502
|
||||
|
||||
scores = reranker.compute_score([['what is panda?', 'hi'], ['what is panda?', 'The giant panda (Ailuropoda melanoleuca), sometimes called a panda bear or simply panda, is a bear species endemic to China.']])
|
||||
print(scores) # [-8.1875, 5.26171875]
|
||||
|
||||
# You can map the scores into 0-1 by set "normalize=True", which will apply sigmoid function to the score
|
||||
scores = reranker.compute_score([['what is panda?', 'hi'], ['what is panda?', 'The giant panda (Ailuropoda melanoleuca), sometimes called a panda bear or simply panda, is a bear species endemic to China.']], normalize=True)
|
||||
print(scores) # [0.00027803096387751553, 0.9948403768236574]
|
||||
```
|
||||
|
||||
#### For LLM-based reranker
|
||||
|
||||
```python
|
||||
from FlagEmbedding import FlagLLMReranker
|
||||
reranker = FlagLLMReranker('BAAI/bge-reranker-v2-gemma', use_fp16=True) # Setting use_fp16 to True speeds up computation with a slight performance degradation
|
||||
# reranker = FlagLLMReranker('BAAI/bge-reranker-v2-gemma', use_bf16=True) # You can also set use_bf16=True to speed up computation with a slight performance degradation
|
||||
|
||||
score = reranker.compute_score(['query', 'passage'])
|
||||
print(score)
|
||||
|
||||
scores = reranker.compute_score([['what is panda?', 'hi'], ['what is panda?', 'The giant panda (Ailuropoda melanoleuca), sometimes called a panda bear or simply panda, is a bear species endemic to China.']])
|
||||
print(scores)
|
||||
```
|
||||
|
||||
#### For LLM-based layerwise reranker
|
||||
|
||||
```python
|
||||
from FlagEmbedding import LayerWiseFlagLLMReranker
|
||||
reranker = LayerWiseFlagLLMReranker('BAAI/bge-reranker-v2-minicpm-layerwise', use_fp16=True) # Setting use_fp16 to True speeds up computation with a slight performance degradation
|
||||
# reranker = LayerWiseFlagLLMReranker('BAAI/bge-reranker-v2-minicpm-layerwise', use_bf16=True) # You can also set use_bf16=True to speed up computation with a slight performance degradation
|
||||
|
||||
score = reranker.compute_score(['query', 'passage'], cutoff_layers=[28]) # Adjusting 'cutoff_layers' to pick which layers are used for computing the score.
|
||||
print(score)
|
||||
|
||||
scores = reranker.compute_score([['what is panda?', 'hi'], ['what is panda?', 'The giant panda (Ailuropoda melanoleuca), sometimes called a panda bear or simply panda, is a bear species endemic to China.']], cutoff_layers=[28])
|
||||
print(scores)
|
||||
```
|
||||
|
||||
### Using Huggingface transformers
|
||||
|
||||
#### For normal reranker (bge-reranker-base / bge-reranker-large / bge-reranker-v2-m3 )
|
||||
|
||||
Get relevance scores (higher scores indicate more relevance):
|
||||
|
||||
```python
|
||||
import torch
|
||||
from transformers import AutoModelForSequenceClassification, AutoTokenizer
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained('BAAI/bge-reranker-v2-m3')
|
||||
model = AutoModelForSequenceClassification.from_pretrained('BAAI/bge-reranker-v2-m3')
|
||||
model.eval()
|
||||
|
||||
pairs = [['what is panda?', 'hi'], ['what is panda?', 'The giant panda (Ailuropoda melanoleuca), sometimes called a panda bear or simply panda, is a bear species endemic to China.']]
|
||||
with torch.no_grad():
|
||||
inputs = tokenizer(pairs, padding=True, truncation=True, return_tensors='pt', max_length=512)
|
||||
scores = model(**inputs, return_dict=True).logits.view(-1, ).float()
|
||||
print(scores)
|
||||
```
|
||||
|
||||
#### For LLM-based reranker
|
||||
|
||||
```python
|
||||
import torch
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||
|
||||
def get_inputs(pairs, tokenizer, prompt=None, max_length=1024):
|
||||
if prompt is None:
|
||||
prompt = "Given a query A and a passage B, determine whether the passage contains an answer to the query by providing a prediction of either 'Yes' or 'No'."
|
||||
sep = "\n"
|
||||
prompt_inputs = tokenizer(prompt,
|
||||
return_tensors=None,
|
||||
add_special_tokens=False)['input_ids']
|
||||
sep_inputs = tokenizer(sep,
|
||||
return_tensors=None,
|
||||
add_special_tokens=False)['input_ids']
|
||||
inputs = []
|
||||
for query, passage in pairs:
|
||||
query_inputs = tokenizer(f'A: {query}',
|
||||
return_tensors=None,
|
||||
add_special_tokens=False,
|
||||
max_length=max_length * 3 // 4,
|
||||
truncation=True)
|
||||
passage_inputs = tokenizer(f'B: {passage}',
|
||||
return_tensors=None,
|
||||
add_special_tokens=False,
|
||||
max_length=max_length,
|
||||
truncation=True)
|
||||
item = tokenizer.prepare_for_model(
|
||||
[tokenizer.bos_token_id] + query_inputs['input_ids'],
|
||||
sep_inputs + passage_inputs['input_ids'],
|
||||
truncation='only_second',
|
||||
max_length=max_length,
|
||||
padding=False,
|
||||
return_attention_mask=False,
|
||||
return_token_type_ids=False,
|
||||
add_special_tokens=False
|
||||
)
|
||||
item['input_ids'] = item['input_ids'] + sep_inputs + prompt_inputs
|
||||
item['attention_mask'] = [1] * len(item['input_ids'])
|
||||
inputs.append(item)
|
||||
return tokenizer.pad(
|
||||
inputs,
|
||||
padding=True,
|
||||
max_length=max_length + len(sep_inputs) + len(prompt_inputs),
|
||||
pad_to_multiple_of=8,
|
||||
return_tensors='pt',
|
||||
)
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained('BAAI/bge-reranker-v2-gemma')
|
||||
model = AutoModelForCausalLM.from_pretrained('BAAI/bge-reranker-v2-gemma')
|
||||
yes_loc = tokenizer('Yes', add_special_tokens=False)['input_ids'][0]
|
||||
model.eval()
|
||||
|
||||
pairs = [['what is panda?', 'hi'], ['what is panda?', 'The giant panda (Ailuropoda melanoleuca), sometimes called a panda bear or simply panda, is a bear species endemic to China.']]
|
||||
with torch.no_grad():
|
||||
inputs = get_inputs(pairs, tokenizer)
|
||||
scores = model(**inputs, return_dict=True).logits[:, -1, yes_loc].view(-1, ).float()
|
||||
print(scores)
|
||||
```
|
||||
|
||||
#### For LLM-based layerwise reranker
|
||||
|
||||
```python
|
||||
import torch
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||
|
||||
def get_inputs(pairs, tokenizer, prompt=None, max_length=1024):
|
||||
if prompt is None:
|
||||
prompt = "Given a query A and a passage B, determine whether the passage contains an answer to the query by providing a prediction of either 'Yes' or 'No'."
|
||||
sep = "\n"
|
||||
prompt_inputs = tokenizer(prompt,
|
||||
return_tensors=None,
|
||||
add_special_tokens=False)['input_ids']
|
||||
sep_inputs = tokenizer(sep,
|
||||
return_tensors=None,
|
||||
add_special_tokens=False)['input_ids']
|
||||
inputs = []
|
||||
for query, passage in pairs:
|
||||
query_inputs = tokenizer(f'A: {query}',
|
||||
return_tensors=None,
|
||||
add_special_tokens=False,
|
||||
max_length=max_length * 3 // 4,
|
||||
truncation=True)
|
||||
passage_inputs = tokenizer(f'B: {passage}',
|
||||
return_tensors=None,
|
||||
add_special_tokens=False,
|
||||
max_length=max_length,
|
||||
truncation=True)
|
||||
item = tokenizer.prepare_for_model(
|
||||
[tokenizer.bos_token_id] + query_inputs['input_ids'],
|
||||
sep_inputs + passage_inputs['input_ids'],
|
||||
truncation='only_second',
|
||||
max_length=max_length,
|
||||
padding=False,
|
||||
return_attention_mask=False,
|
||||
return_token_type_ids=False,
|
||||
add_special_tokens=False
|
||||
)
|
||||
item['input_ids'] = item['input_ids'] + sep_inputs + prompt_inputs
|
||||
item['attention_mask'] = [1] * len(item['input_ids'])
|
||||
inputs.append(item)
|
||||
return tokenizer.pad(
|
||||
inputs,
|
||||
padding=True,
|
||||
max_length=max_length + len(sep_inputs) + len(prompt_inputs),
|
||||
pad_to_multiple_of=8,
|
||||
return_tensors='pt',
|
||||
)
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained('BAAI/bge-reranker-v2-minicpm-layerwise', trust_remote_code=True)
|
||||
model = AutoModelForCausalLM.from_pretrained('BAAI/bge-reranker-v2-minicpm-layerwise', trust_remote_code=True, torch_dtype=torch.bfloat16)
|
||||
model = model.to('cuda')
|
||||
model.eval()
|
||||
|
||||
pairs = [['what is panda?', 'hi'], ['what is panda?', 'The giant panda (Ailuropoda melanoleuca), sometimes called a panda bear or simply panda, is a bear species endemic to China.']]
|
||||
with torch.no_grad():
|
||||
inputs = get_inputs(pairs, tokenizer).to(model.device)
|
||||
all_scores = model(**inputs, return_dict=True, cutoff_layers=[28])
|
||||
all_scores = [scores[:, -1].view(-1, ).float() for scores in all_scores[0]]
|
||||
print(all_scores)
|
||||
```
|
||||
|
||||
## Fine-tune
|
||||
|
||||
### Data Format
|
||||
|
||||
Train data should be a json file, where each line is a dict like this:
|
||||
|
||||
```
|
||||
{"query": str, "pos": List[str], "neg":List[str], "prompt": str}
|
||||
```
|
||||
|
||||
`query` is the query, and `pos` is a list of positive texts, `neg` is a list of negative texts, `prompt` indicates the relationship between query and texts. If you have no negative texts for a query, you can random sample some from the entire corpus as the negatives.
|
||||
|
||||
See [toy_finetune_data.jsonl](https://github.com/FlagOpen/FlagEmbedding/tree/master/FlagEmbedding/llm_reranker/toy_finetune_data.jsonl) for a toy data file.
|
||||
|
||||
### Train
|
||||
|
||||
You can fine-tune the reranker with the following code:
|
||||
|
||||
**For llm-based reranker**
|
||||
|
||||
```shell
|
||||
torchrun --nproc_per_node {number of gpus} \
|
||||
-m FlagEmbedding.llm_reranker.finetune_for_instruction.run \
|
||||
--output_dir {path to save model} \
|
||||
--model_name_or_path google/gemma-2b \
|
||||
--train_data ./toy_finetune_data.jsonl \
|
||||
--learning_rate 2e-4 \
|
||||
--num_train_epochs 1 \
|
||||
--per_device_train_batch_size 1 \
|
||||
--gradient_accumulation_steps 16 \
|
||||
--dataloader_drop_last True \
|
||||
--query_max_len 512 \
|
||||
--passage_max_len 512 \
|
||||
--train_group_size 16 \
|
||||
--logging_steps 1 \
|
||||
--save_steps 2000 \
|
||||
--save_total_limit 50 \
|
||||
--ddp_find_unused_parameters False \
|
||||
--gradient_checkpointing \
|
||||
--deepspeed stage1.json \
|
||||
--warmup_ratio 0.1 \
|
||||
--bf16 \
|
||||
--use_lora True \
|
||||
--lora_rank 32 \
|
||||
--lora_alpha 64 \
|
||||
--use_flash_attn True \
|
||||
--target_modules q_proj k_proj v_proj o_proj
|
||||
```
|
||||
|
||||
**For llm-based layerwise reranker**
|
||||
|
||||
```shell
|
||||
torchrun --nproc_per_node {number of gpus} \
|
||||
-m FlagEmbedding.llm_reranker.finetune_for_layerwise.run \
|
||||
--output_dir {path to save model} \
|
||||
--model_name_or_path openbmb/MiniCPM-2B-dpo-bf16 \
|
||||
--train_data ./toy_finetune_data.jsonl \
|
||||
--learning_rate 2e-4 \
|
||||
--num_train_epochs 1 \
|
||||
--per_device_train_batch_size 1 \
|
||||
--gradient_accumulation_steps 16 \
|
||||
--dataloader_drop_last True \
|
||||
--query_max_len 512 \
|
||||
--passage_max_len 512 \
|
||||
--train_group_size 16 \
|
||||
--logging_steps 1 \
|
||||
--save_steps 2000 \
|
||||
--save_total_limit 50 \
|
||||
--ddp_find_unused_parameters False \
|
||||
--gradient_checkpointing \
|
||||
--deepspeed stage1.json \
|
||||
--warmup_ratio 0.1 \
|
||||
--bf16 \
|
||||
--use_lora True \
|
||||
--lora_rank 32 \
|
||||
--lora_alpha 64 \
|
||||
--use_flash_attn True \
|
||||
--target_modules q_proj k_proj v_proj o_proj \
|
||||
--start_layer 8 \
|
||||
--head_multi True \
|
||||
--head_type simple \
|
||||
--lora_extra_parameters linear_head
|
||||
```
|
||||
|
||||
Our rerankers are initialized from [google/gemma-2b](https://huggingface.co/google/gemma-2b) (for llm-based reranker) and [openbmb/MiniCPM-2B-dpo-bf16](https://huggingface.co/openbmb/MiniCPM-2B-dpo-bf16) (for llm-based layerwise reranker), and we train it on a mixture of multilingual datasets:
|
||||
|
||||
- [bge-m3-data](https://huggingface.co/datasets/Shitao/bge-m3-data)
|
||||
- [quora train data](https://huggingface.co/datasets/quora)
|
||||
- [fever train data](https://fever.ai/dataset/fever.html)
|
||||
|
||||
## Evaluation
|
||||
|
||||
- llama-index.
|
||||
|
||||
|
||||
|
||||
|
||||
- BEIR.
|
||||
|
||||
rereank the top 100 results from bge-en-v1.5 large.
|
||||
|
||||
|
||||
|
||||
rereank the top 100 results from e5 mistral 7b instruct.
|
||||
|
||||
|
||||
|
||||
- CMTEB-retrieval.
|
||||
It rereank the top 100 results from bge-zh-v1.5 large.
|
||||
|
||||
|
||||
|
||||
- miracl (multi-language).
|
||||
It rereank the top 100 results from bge-m3.
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
## Citation
|
||||
|
||||
If you find this repository useful, please consider giving a star and citation
|
||||
|
||||
```bibtex
|
||||
@misc{li2023making,
|
||||
title={Making Large Language Models A Better Foundation For Dense Retrieval},
|
||||
author={Chaofan Li and Zheng Liu and Shitao Xiao and Yingxia Shao},
|
||||
year={2023},
|
||||
eprint={2312.15503},
|
||||
archivePrefix={arXiv},
|
||||
primaryClass={cs.CL}
|
||||
}
|
||||
@misc{chen2024bge,
|
||||
title={BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation},
|
||||
author={Jianlv Chen and Shitao Xiao and Peitian Zhang and Kun Luo and Defu Lian and Zheng Liu},
|
||||
year={2024},
|
||||
eprint={2402.03216},
|
||||
archivePrefix={arXiv},
|
||||
primaryClass={cs.CL}
|
||||
}
|
||||
```
|
||||
|
|
@ -0,0 +1,38 @@
|
|||
{
|
||||
"_name_or_path": "BAAI/bge-reranker-v2-minicpm-layerwise",
|
||||
"architectures": [
|
||||
"LayerWiseMiniCPMForCausalLM"
|
||||
],
|
||||
"attention_bias": false,
|
||||
"attention_dropout": 0.0,
|
||||
"auto_map": {
|
||||
"AutoConfig": "BAAI/bge-reranker-v2-minicpm-layerwise--configuration_minicpm_reranker.LayerWiseMiniCPMConfig",
|
||||
"AutoModel": "BAAI/bge-reranker-v2-minicpm-layerwise--modeling_minicpm_reranker.LayerWiseMiniCPMModel",
|
||||
"AutoModelForCausalLM": "BAAI/bge-reranker-v2-minicpm-layerwise--modeling_minicpm_reranker.LayerWiseMiniCPMForCausalLM"
|
||||
},
|
||||
"bos_token_id": 1,
|
||||
"dim_model_base": 256,
|
||||
"eos_token_id": 2,
|
||||
"head_multi": true,
|
||||
"head_type": "simple",
|
||||
"hidden_act": "silu",
|
||||
"hidden_size": 2304,
|
||||
"initializer_range": 0.1,
|
||||
"intermediate_size": 5760,
|
||||
"max_position_embeddings": 2048,
|
||||
"model_type": "minicpm",
|
||||
"num_attention_heads": 36,
|
||||
"num_hidden_layers": 40,
|
||||
"num_key_value_heads": 36,
|
||||
"pretraining_tp": 1,
|
||||
"rms_norm_eps": 1e-05,
|
||||
"rope_scaling": null,
|
||||
"rope_theta": 10000.0,
|
||||
"scale_depth": 1.4,
|
||||
"scale_emb": 12,
|
||||
"start_layer": 8,
|
||||
"torch_dtype": "bfloat16",
|
||||
"transformers_version": "4.38.1",
|
||||
"use_cache": false,
|
||||
"vocab_size": 122753
|
||||
}
|
||||
|
|
@ -0,0 +1 @@
|
|||
{"framework": "pytorch", "task": "text-classification", "allow_remote": true}
|
||||
|
|
@ -0,0 +1,209 @@
|
|||
# coding=utf-8
|
||||
# Copyright 2022 EleutherAI and the HuggingFace Inc. team. All rights reserved.
|
||||
#
|
||||
# This code is based on EleutherAI's GPT-NeoX library and the GPT-NeoX
|
||||
# and OPT implementations in this library. It has been modified from its
|
||||
# original forms to accommodate minor architectural differences compared
|
||||
# to GPT-NeoX and OPT used by the Meta AI team that trained the model.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
""" MiniCPM model configuration"""
|
||||
|
||||
from transformers.configuration_utils import PretrainedConfig
|
||||
from transformers.utils import logging
|
||||
|
||||
|
||||
logger = logging.get_logger(__name__)
|
||||
|
||||
MINICPM_PRETRAINED_CONFIG_ARCHIVE_MAP = {}
|
||||
|
||||
|
||||
class LayerWiseMiniCPMConfig(PretrainedConfig):
|
||||
r"""
|
||||
This is the configuration class to store the configuration of a [`MiniCPMModel`]. It is used to instantiate an MiniCPM
|
||||
model according to the specified arguments, defining the model architecture. Instantiating a configuration with the
|
||||
defaults will yield a similar configuration to that of the MiniCPM-7B.
|
||||
|
||||
Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the
|
||||
documentation from [`PretrainedConfig`] for more information.
|
||||
|
||||
|
||||
Args:
|
||||
vocab_size (`int`, *optional*, defaults to 32000):
|
||||
Vocabulary size of the MiniCPM model. Defines the number of different tokens that can be represented by the
|
||||
`inputs_ids` passed when calling [`MiniCPMModel`]
|
||||
hidden_size (`int`, *optional*, defaults to 4096):
|
||||
Dimension of the hidden representations.
|
||||
intermediate_size (`int`, *optional*, defaults to 11008):
|
||||
Dimension of the MLP representations.
|
||||
num_hidden_layers (`int`, *optional*, defaults to 32):
|
||||
Number of hidden layers in the Transformer decoder.
|
||||
num_attention_heads (`int`, *optional*, defaults to 32):
|
||||
Number of attention heads for each attention layer in the Transformer decoder.
|
||||
num_key_value_heads (`int`, *optional*):
|
||||
This is the number of key_value heads that should be used to implement Grouped Query Attention. If
|
||||
`num_key_value_heads=num_attention_heads`, the model will use Multi Head Attention (MHA), if
|
||||
`num_key_value_heads=1 the model will use Multi Query Attention (MQA) otherwise GQA is used. When
|
||||
converting a multi-head checkpoint to a GQA checkpoint, each group key and value head should be constructed
|
||||
by meanpooling all the original heads within that group. For more details checkout [this
|
||||
paper](https://arxiv.org/pdf/2305.13245.pdf). If it is not specified, will default to
|
||||
`num_attention_heads`.
|
||||
hidden_act (`str` or `function`, *optional*, defaults to `"silu"`):
|
||||
The non-linear activation function (function or string) in the decoder.
|
||||
max_position_embeddings (`int`, *optional*, defaults to 2048):
|
||||
The maximum sequence length that this model might ever be used with. MiniCPM 1 supports up to 2048 tokens,
|
||||
MiniCPM 2 up to 4096, CodeMiniCPM up to 16384.
|
||||
initializer_range (`float`, *optional*, defaults to 0.02):
|
||||
The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
|
||||
rms_norm_eps (`float`, *optional*, defaults to 1e-06):
|
||||
The epsilon used by the rms normalization layers.
|
||||
use_cache (`bool`, *optional*, defaults to `True`):
|
||||
Whether or not the model should return the last key/values attentions (not used by all models). Only
|
||||
relevant if `config.is_decoder=True`.
|
||||
pad_token_id (`int`, *optional*):
|
||||
Padding token id.
|
||||
bos_token_id (`int`, *optional*, defaults to 1):
|
||||
Beginning of stream token id.
|
||||
eos_token_id (`int`, *optional*, defaults to 2):
|
||||
End of stream token id.
|
||||
pretraining_tp (`int`, *optional*, defaults to 1):
|
||||
Experimental feature. Tensor parallelism rank used during pretraining. Please refer to [this
|
||||
document](https://huggingface.co/docs/transformers/parallelism) to understand more about it. This value is
|
||||
necessary to ensure exact reproducibility of the pretraining results. Please refer to [this
|
||||
issue](https://github.com/pytorch/pytorch/issues/76232).
|
||||
tie_word_embeddings (`bool`, *optional*, defaults to `False`):
|
||||
Whether to tie weight embeddings
|
||||
rope_theta (`float`, *optional*, defaults to 10000.0):
|
||||
The base period of the RoPE embeddings.
|
||||
rope_scaling (`Dict`, *optional*):
|
||||
Dictionary containing the scaling configuration for the RoPE embeddings. Currently supports two scaling
|
||||
strategies: linear and dynamic. Their scaling factor must be a float greater than 1. The expected format is
|
||||
`{"type": strategy name, "factor": scaling factor}`. When using this flag, don't update
|
||||
`max_position_embeddings` to the expected new maximum. See the following thread for more information on how
|
||||
these scaling strategies behave:
|
||||
https://www.reddit.com/r/LocalMiniCPM/comments/14mrgpr/dynamically_scaled_rope_further_increases/. This is an
|
||||
experimental feature, subject to breaking API changes in future versions.
|
||||
attention_bias (`bool`, defaults to `False`, *optional*, defaults to `False`):
|
||||
Whether to use a bias in the query, key, value and output projection layers during self-attention.
|
||||
attention_dropout (`float`, *optional*, defaults to 0.0):
|
||||
The dropout ratio for the attention probabilities.
|
||||
|
||||
```python
|
||||
>>> from transformers import MiniCPMModel, MiniCPMConfig
|
||||
|
||||
>>> # Initializing a MiniCPM minicpm-7b style configuration
|
||||
>>> configuration = MiniCPMConfig()
|
||||
|
||||
>>> # Initializing a model from the minicpm-7b style configuration
|
||||
>>> model = MiniCPMModel(configuration)
|
||||
|
||||
>>> # Accessing the model configuration
|
||||
>>> configuration = model.config
|
||||
```"""
|
||||
|
||||
model_type = "minicpm"
|
||||
keys_to_ignore_at_inference = ["past_key_values"]
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
vocab_size=32000,
|
||||
hidden_size=4096,
|
||||
intermediate_size=11008,
|
||||
num_hidden_layers=32,
|
||||
num_attention_heads=32,
|
||||
num_key_value_heads=None,
|
||||
hidden_act="silu",
|
||||
max_position_embeddings=2048,
|
||||
initializer_range=0.02,
|
||||
rms_norm_eps=1e-6,
|
||||
use_cache=True,
|
||||
pad_token_id=None,
|
||||
bos_token_id=1,
|
||||
eos_token_id=2,
|
||||
pretraining_tp=1,
|
||||
tie_word_embeddings=True,
|
||||
rope_theta=10000.0,
|
||||
rope_scaling=None,
|
||||
attention_bias=False,
|
||||
attention_dropout=0.0,
|
||||
scale_emb=1,
|
||||
dim_model_base=1,
|
||||
scale_depth=1,
|
||||
start_layer=8,
|
||||
head_multi=True,
|
||||
head_type="simple",
|
||||
**kwargs,
|
||||
):
|
||||
self.vocab_size = vocab_size
|
||||
self.max_position_embeddings = max_position_embeddings
|
||||
self.hidden_size = hidden_size
|
||||
self.intermediate_size = intermediate_size
|
||||
self.num_hidden_layers = num_hidden_layers
|
||||
self.num_attention_heads = num_attention_heads
|
||||
|
||||
# for backward compatibility
|
||||
if num_key_value_heads is None:
|
||||
num_key_value_heads = num_attention_heads
|
||||
|
||||
self.num_key_value_heads = num_key_value_heads
|
||||
self.hidden_act = hidden_act
|
||||
self.initializer_range = initializer_range
|
||||
self.rms_norm_eps = rms_norm_eps
|
||||
self.pretraining_tp = pretraining_tp
|
||||
self.use_cache = use_cache
|
||||
self.rope_theta = rope_theta
|
||||
self.rope_scaling = rope_scaling
|
||||
self._rope_scaling_validation()
|
||||
self.attention_bias = attention_bias
|
||||
self.attention_dropout = attention_dropout
|
||||
self.scale_emb = scale_emb
|
||||
self.dim_model_base = dim_model_base
|
||||
self.scale_depth = scale_depth
|
||||
|
||||
self.start_layer = start_layer
|
||||
self.head_multi = head_multi
|
||||
self.head_type = head_type
|
||||
|
||||
super().__init__(
|
||||
pad_token_id=pad_token_id,
|
||||
bos_token_id=bos_token_id,
|
||||
eos_token_id=eos_token_id,
|
||||
tie_word_embeddings=tie_word_embeddings,
|
||||
**kwargs,
|
||||
)
|
||||
try:
|
||||
import flash_attn
|
||||
self._attn_implementation = "flash_attention_2"
|
||||
except:
|
||||
pass
|
||||
|
||||
def _rope_scaling_validation(self):
|
||||
"""
|
||||
Validate the `rope_scaling` configuration.
|
||||
"""
|
||||
if self.rope_scaling is None:
|
||||
return
|
||||
|
||||
if not isinstance(self.rope_scaling, dict) or len(self.rope_scaling) != 2:
|
||||
raise ValueError(
|
||||
"`rope_scaling` must be a dictionary with with two fields, `type` and `factor`, "
|
||||
f"got {self.rope_scaling}"
|
||||
)
|
||||
rope_scaling_type = self.rope_scaling.get("type", None)
|
||||
rope_scaling_factor = self.rope_scaling.get("factor", None)
|
||||
if rope_scaling_type is None or rope_scaling_type not in ["linear", "dynamic"]:
|
||||
raise ValueError(
|
||||
f"`rope_scaling`'s type field must be one of ['linear', 'dynamic'], got {rope_scaling_type}"
|
||||
)
|
||||
if rope_scaling_factor is None or not isinstance(rope_scaling_factor, float) or rope_scaling_factor <= 1.0:
|
||||
raise ValueError(f"`rope_scaling`'s factor field must be a float > 1, got {rope_scaling_factor}")
|
||||
|
|
@ -0,0 +1,8 @@
|
|||
{
|
||||
"bos_token_id": 1,
|
||||
"do_sample": true,
|
||||
"eos_token_id": 2,
|
||||
"temperature": 0.8,
|
||||
"top_p": 0.8,
|
||||
"transformers_version": "4.38.1"
|
||||
}
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
|
|
@ -0,0 +1,402 @@
|
|||
{
|
||||
"metadata": {
|
||||
"total_size": 10899827712
|
||||
},
|
||||
"weight_map": {
|
||||
"lm_head.0.linear_head.weight": "model-00003-of-00003.safetensors",
|
||||
"lm_head.1.linear_head.weight": "model-00003-of-00003.safetensors",
|
||||
"lm_head.10.linear_head.weight": "model-00003-of-00003.safetensors",
|
||||
"lm_head.11.linear_head.weight": "model-00003-of-00003.safetensors",
|
||||
"lm_head.12.linear_head.weight": "model-00003-of-00003.safetensors",
|
||||
"lm_head.13.linear_head.weight": "model-00003-of-00003.safetensors",
|
||||
"lm_head.14.linear_head.weight": "model-00003-of-00003.safetensors",
|
||||
"lm_head.15.linear_head.weight": "model-00003-of-00003.safetensors",
|
||||
"lm_head.16.linear_head.weight": "model-00003-of-00003.safetensors",
|
||||
"lm_head.17.linear_head.weight": "model-00003-of-00003.safetensors",
|
||||
"lm_head.18.linear_head.weight": "model-00003-of-00003.safetensors",
|
||||
"lm_head.19.linear_head.weight": "model-00003-of-00003.safetensors",
|
||||
"lm_head.2.linear_head.weight": "model-00003-of-00003.safetensors",
|
||||
"lm_head.20.linear_head.weight": "model-00003-of-00003.safetensors",
|
||||
"lm_head.21.linear_head.weight": "model-00003-of-00003.safetensors",
|
||||
"lm_head.22.linear_head.weight": "model-00003-of-00003.safetensors",
|
||||
"lm_head.23.linear_head.weight": "model-00003-of-00003.safetensors",
|
||||
"lm_head.24.linear_head.weight": "model-00003-of-00003.safetensors",
|
||||
"lm_head.25.linear_head.weight": "model-00003-of-00003.safetensors",
|
||||
"lm_head.26.linear_head.weight": "model-00003-of-00003.safetensors",
|
||||
"lm_head.27.linear_head.weight": "model-00003-of-00003.safetensors",
|
||||
"lm_head.28.linear_head.weight": "model-00003-of-00003.safetensors",
|
||||
"lm_head.29.linear_head.weight": "model-00003-of-00003.safetensors",
|
||||
"lm_head.3.linear_head.weight": "model-00003-of-00003.safetensors",
|
||||
"lm_head.30.linear_head.weight": "model-00003-of-00003.safetensors",
|
||||
"lm_head.31.linear_head.weight": "model-00003-of-00003.safetensors",
|
||||
"lm_head.32.linear_head.weight": "model-00003-of-00003.safetensors",
|
||||
"lm_head.4.linear_head.weight": "model-00003-of-00003.safetensors",
|
||||
"lm_head.5.linear_head.weight": "model-00003-of-00003.safetensors",
|
||||
"lm_head.6.linear_head.weight": "model-00003-of-00003.safetensors",
|
||||
"lm_head.7.linear_head.weight": "model-00003-of-00003.safetensors",
|
||||
"lm_head.8.linear_head.weight": "model-00003-of-00003.safetensors",
|
||||
"lm_head.9.linear_head.weight": "model-00003-of-00003.safetensors",
|
||||
"model.embed_tokens.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.0.input_layernorm.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.0.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.0.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.0.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.0.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.0.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.0.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.0.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.0.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.1.input_layernorm.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.1.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.1.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.1.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.1.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.1.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.1.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.1.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.1.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.10.input_layernorm.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.10.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.10.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.10.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.10.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.10.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.10.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.10.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.10.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.11.input_layernorm.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.11.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.11.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.11.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.11.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.11.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.11.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.11.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.11.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.12.input_layernorm.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.12.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.12.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.12.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.12.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.12.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.12.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.12.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.12.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.13.input_layernorm.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.13.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.13.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.13.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.13.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.13.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.13.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.13.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.13.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.14.input_layernorm.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.14.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.14.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.14.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.14.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.14.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.14.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.14.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.14.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.15.input_layernorm.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.15.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.15.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.15.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.15.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.15.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.15.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.15.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.15.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.16.input_layernorm.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.16.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.16.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.16.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.16.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.16.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.16.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.16.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.16.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.17.input_layernorm.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.17.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.17.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.17.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.17.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.17.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.17.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.17.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.17.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.18.input_layernorm.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.18.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.18.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.18.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.18.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.18.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.18.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.18.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.18.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.19.input_layernorm.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.19.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.19.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.19.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.19.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.19.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.19.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.19.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.19.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.2.input_layernorm.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.2.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.2.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.2.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.2.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.2.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.2.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.2.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.2.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.20.input_layernorm.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.20.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.20.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.20.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.20.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.20.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.20.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.20.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.20.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.21.input_layernorm.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.21.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.21.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.21.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.21.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.21.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.21.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.21.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.21.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.22.input_layernorm.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.22.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.22.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.22.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.22.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.22.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.22.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.22.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.22.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.23.input_layernorm.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.23.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.23.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.23.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.23.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.23.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.23.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.23.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.23.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.24.input_layernorm.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.24.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.24.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.24.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.24.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.24.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.24.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.24.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.24.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.25.input_layernorm.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.25.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.25.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.25.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.25.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.25.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.25.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.25.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.25.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.26.input_layernorm.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.26.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.26.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.26.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.26.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.26.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.26.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.26.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.26.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.27.input_layernorm.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.27.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.27.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.27.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.27.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.27.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.27.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.27.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.27.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.28.input_layernorm.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.28.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.28.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.28.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.28.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.28.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.28.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.28.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.28.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.29.input_layernorm.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.29.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.29.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.29.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.29.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.29.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.29.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.29.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.29.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.3.input_layernorm.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.3.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.3.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.3.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.3.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.3.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.3.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.3.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.3.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.30.input_layernorm.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.30.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.30.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.30.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.30.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.30.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.30.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.30.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.30.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.31.input_layernorm.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.31.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.31.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.31.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.31.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.31.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.31.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.31.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.31.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.32.input_layernorm.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.32.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.32.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.32.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.32.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.32.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.32.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.32.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.32.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.33.input_layernorm.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.33.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.33.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.33.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.33.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.33.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.33.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.33.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.33.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.34.input_layernorm.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.34.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.34.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.34.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.34.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.34.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.34.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.34.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.34.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.35.input_layernorm.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.35.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.35.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.35.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.35.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.35.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.35.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.35.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.35.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.36.input_layernorm.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.36.mlp.down_proj.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.36.mlp.gate_proj.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.36.mlp.up_proj.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.36.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.36.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.36.self_attn.o_proj.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.36.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
|
||||
"model.layers.36.self_attn.v_proj.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.37.input_layernorm.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.37.mlp.down_proj.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.37.mlp.gate_proj.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.37.mlp.up_proj.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.37.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.37.self_attn.k_proj.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.37.self_attn.o_proj.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.37.self_attn.q_proj.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.37.self_attn.v_proj.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.38.input_layernorm.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.38.mlp.down_proj.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.38.mlp.gate_proj.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.38.mlp.up_proj.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.38.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.38.self_attn.k_proj.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.38.self_attn.o_proj.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.38.self_attn.q_proj.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.38.self_attn.v_proj.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.39.input_layernorm.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.39.mlp.down_proj.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.39.mlp.gate_proj.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.39.mlp.up_proj.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.39.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.39.self_attn.k_proj.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.39.self_attn.o_proj.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.39.self_attn.q_proj.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.39.self_attn.v_proj.weight": "model-00003-of-00003.safetensors",
|
||||
"model.layers.4.input_layernorm.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.4.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.4.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.4.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.4.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.4.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.4.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.4.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.4.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.5.input_layernorm.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.5.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.5.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.5.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.5.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.5.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.5.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.5.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.5.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.6.input_layernorm.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.6.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.6.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.6.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.6.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.6.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.6.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.6.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.6.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.7.input_layernorm.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.7.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.7.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.7.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.7.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.7.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.7.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.7.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.7.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.8.input_layernorm.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.8.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.8.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.8.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.8.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.8.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.8.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.8.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.8.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.9.input_layernorm.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.9.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.9.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.9.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.9.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.9.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.9.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.9.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.layers.9.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
|
||||
"model.norm.weight": "model-00003-of-00003.safetensors"
|
||||
}
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
|
|
@ -0,0 +1,30 @@
|
|||
{
|
||||
"bos_token": {
|
||||
"content": "<s>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
"eos_token": {
|
||||
"content": "</s>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
"pad_token": {
|
||||
"content": "<unk>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
"unk_token": {
|
||||
"content": "<unk>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
}
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
Binary file not shown.
|
|
@ -0,0 +1,42 @@
|
|||
{
|
||||
"add_bos_token": true,
|
||||
"add_eos_token": false,
|
||||
"added_tokens_decoder": {
|
||||
"0": {
|
||||
"content": "<unk>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"1": {
|
||||
"content": "<s>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"2": {
|
||||
"content": "</s>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
}
|
||||
},
|
||||
"bos_token": "<s>",
|
||||
"chat_template": "{% for message in messages %}{% if message['role'] == 'user' %}{{'<用户>' + message['content'].strip() + '<AI>'}}{% else %}{{message['content'].strip()}}{% endif %}{% endfor %}",
|
||||
"clean_up_tokenization_spaces": false,
|
||||
"eos_token": "</s>",
|
||||
"legacy": true,
|
||||
"model_max_length": 1000000000000000019884624838656,
|
||||
"pad_token": "<unk>",
|
||||
"sp_model_kwargs": {},
|
||||
"spaces_between_special_tokens": false,
|
||||
"tokenizer_class": "LlamaTokenizer",
|
||||
"unk_token": "<unk>",
|
||||
"use_default_system_prompt": false
|
||||
}
|
||||
Loading…
Reference in New Issue