ailab
/
jina-embeddings-v2-base-zh
1
0
Fork 1
Code Issues Pull Requests Packages Projects Releases Wiki Activity
jina-embeddings-v2-base-zh/README.md

1254 lines
33 KiB
Markdown
Raw Normal View History

first commit
2024-08-21 14:59:50 +08:00
---
tags:
- sentence-transformers
- feature-extraction
- sentence-similarity
- mteb
- transformers
- transformers.js
inference: false
license: apache-2.0
language:
- en
- zh
model-index:
- name: jina-embeddings-v2-base-zh
results:
- task:
type: STS
dataset:
type: C-MTEB/AFQMC
name: MTEB AFQMC
config: default
split: validation
revision: None
metrics:
- type: cos_sim_pearson
value: 48.51403119231363
- type: cos_sim_spearman
value: 50.5928547846445
- type: euclidean_pearson
value: 48.750436310559074
- type: euclidean_spearman
value: 50.50950238691385
- type: manhattan_pearson
value: 48.7866189440328
- type: manhattan_spearman
value: 50.58692402017165
- task:
type: STS
dataset:
type: C-MTEB/ATEC
name: MTEB ATEC
config: default
split: test
revision: None
metrics:
- type: cos_sim_pearson
value: 50.25985700105725
- type: cos_sim_spearman
value: 51.28815934593989
- type: euclidean_pearson
value: 52.70329248799904
- type: euclidean_spearman
value: 50.94101139559258
- type: manhattan_pearson
value: 52.6647237400892
- type: manhattan_spearman
value: 50.922441325406176
- task:
type: Classification
dataset:
type: mteb/amazon_reviews_multi
name: MTEB AmazonReviewsClassification (zh)
config: zh
split: test
revision: 1399c76144fd37290681b995c656ef9b2e06e26d
metrics:
- type: accuracy
value: 34.944
- type: f1
value: 34.06478860660109
- task:
type: STS
dataset:
type: C-MTEB/BQ
name: MTEB BQ
config: default
split: test
revision: None
metrics:
- type: cos_sim_pearson
value: 65.15667035488342
- type: cos_sim_spearman
value: 66.07110142081
- type: euclidean_pearson
value: 60.447598102249714
- type: euclidean_spearman
value: 61.826575796578766
- type: manhattan_pearson
value: 60.39364279354984
- type: manhattan_spearman
value: 61.78743491223281
- task:
type: Clustering
dataset:
type: C-MTEB/CLSClusteringP2P
name: MTEB CLSClusteringP2P
config: default
split: test
revision: None
metrics:
- type: v_measure
value: 39.96714175391701
- task:
type: Clustering
dataset:
type: C-MTEB/CLSClusteringS2S
name: MTEB CLSClusteringS2S
config: default
split: test
revision: None
metrics:
- type: v_measure
value: 38.39863566717934
- task:
type: Reranking
dataset:
type: C-MTEB/CMedQAv1-reranking
name: MTEB CMedQAv1
config: default
split: test
revision: None
metrics:
- type: map
value: 83.63680381780644
- type: mrr
value: 86.16476190476192
- task:
type: Reranking
dataset:
type: C-MTEB/CMedQAv2-reranking
name: MTEB CMedQAv2
config: default
split: test
revision: None
metrics:
- type: map
value: 83.74350667859487
- type: mrr
value: 86.10388888888889
- task:
type: Retrieval
dataset:
type: C-MTEB/CmedqaRetrieval
name: MTEB CmedqaRetrieval
config: default
split: dev
revision: None
metrics:
- type: map_at_1
value: 22.072
- type: map_at_10
value: 32.942
- type: map_at_100
value: 34.768
- type: map_at_1000
value: 34.902
- type: map_at_3
value: 29.357
- type: map_at_5
value: 31.236000000000004
- type: mrr_at_1
value: 34.259
- type: mrr_at_10
value: 41.957
- type: mrr_at_100
value: 42.982
- type: mrr_at_1000
value: 43.042
- type: mrr_at_3
value: 39.722
- type: mrr_at_5
value: 40.898
- type: ndcg_at_1
value: 34.259
- type: ndcg_at_10
value: 39.153
- type: ndcg_at_100
value: 46.493
- type: ndcg_at_1000
value: 49.01
- type: ndcg_at_3
value: 34.636
- type: ndcg_at_5
value: 36.278
- type: precision_at_1
value: 34.259
- type: precision_at_10
value: 8.815000000000001
- type: precision_at_100
value: 1.474
- type: precision_at_1000
value: 0.179
- type: precision_at_3
value: 19.73
- type: precision_at_5
value: 14.174000000000001
- type: recall_at_1
value: 22.072
- type: recall_at_10
value: 48.484
- type: recall_at_100
value: 79.035
- type: recall_at_1000
value: 96.15
- type: recall_at_3
value: 34.607
- type: recall_at_5
value: 40.064
- task:
type: PairClassification
dataset:
type: C-MTEB/CMNLI
name: MTEB Cmnli
config: default
split: validation
revision: None
metrics:
- type: cos_sim_accuracy
value: 76.7047504509922
- type: cos_sim_ap
value: 85.26649874800871
- type: cos_sim_f1
value: 78.13528724646915
- type: cos_sim_precision
value: 71.57587548638132
- type: cos_sim_recall
value: 86.01823708206688
- type: dot_accuracy
value: 70.13830426939266
- type: dot_ap
value: 77.01510412382171
- type: dot_f1
value: 73.56710042713817
- type: dot_precision
value: 63.955094991364426
- type: dot_recall
value: 86.57937806873977
- type: euclidean_accuracy
value: 75.53818400481059
- type: euclidean_ap
value: 84.34668448241264
- type: euclidean_f1
value: 77.51741608613047
- type: euclidean_precision
value: 70.65614777756399
- type: euclidean_recall
value: 85.85457096095394
- type: manhattan_accuracy
value: 75.49007817197835
- type: manhattan_ap
value: 84.40297506704299
- type: manhattan_f1
value: 77.63185324160932
- type: manhattan_precision
value: 70.03949595636637
- type: manhattan_recall
value: 87.07037643207856
- type: max_accuracy
value: 76.7047504509922
- type: max_ap
value: 85.26649874800871
- type: max_f1
value: 78.13528724646915
- task:
type: Retrieval
dataset:
type: C-MTEB/CovidRetrieval
name: MTEB CovidRetrieval
config: default
split: dev
revision: None
metrics:
- type: map_at_1
value: 69.178
- type: map_at_10
value: 77.523
- type: map_at_100
value: 77.793
- type: map_at_1000
value: 77.79899999999999
- type: map_at_3
value: 75.878
- type: map_at_5
value: 76.849
- type: mrr_at_1
value: 69.44200000000001
- type: mrr_at_10
value: 77.55
- type: mrr_at_100
value: 77.819
- type: mrr_at_1000
value: 77.826
- type: mrr_at_3
value: 75.957
- type: mrr_at_5
value: 76.916
- type: ndcg_at_1
value: 69.44200000000001
- type: ndcg_at_10
value: 81.217
- type: ndcg_at_100
value: 82.45
- type: ndcg_at_1000
value: 82.636
- type: ndcg_at_3
value: 77.931
- type: ndcg_at_5
value: 79.655
- type: precision_at_1
value: 69.44200000000001
- type: precision_at_10
value: 9.357
- type: precision_at_100
value: 0.993
- type: precision_at_1000
value: 0.101
- type: precision_at_3
value: 28.1
- type: precision_at_5
value: 17.724
- type: recall_at_1
value: 69.178
- type: recall_at_10
value: 92.624
- type: recall_at_100
value: 98.209
- type: recall_at_1000
value: 99.684
- type: recall_at_3
value: 83.772
- type: recall_at_5
value: 87.882
- task:
type: Retrieval
dataset:
type: C-MTEB/DuRetrieval
name: MTEB DuRetrieval
config: default
split: dev
revision: None
metrics:
- type: map_at_1
value: 25.163999999999998
- type: map_at_10
value: 76.386
- type: map_at_100
value: 79.339
- type: map_at_1000
value: 79.39500000000001
- type: map_at_3
value: 52.959
- type: map_at_5
value: 66.59
- type: mrr_at_1
value: 87.9
- type: mrr_at_10
value: 91.682
- type: mrr_at_100
value: 91.747
- type: mrr_at_1000
value: 91.751
- type: mrr_at_3
value: 91.267
- type: mrr_at_5
value: 91.527
- type: ndcg_at_1
value: 87.9
- type: ndcg_at_10
value: 84.569
- type: ndcg_at_100
value: 87.83800000000001
- type: ndcg_at_1000
value: 88.322
- type: ndcg_at_3
value: 83.473
- type: ndcg_at_5
value: 82.178
- type: precision_at_1
value: 87.9
- type: precision_at_10
value: 40.605000000000004
- type: precision_at_100
value: 4.752
- type: precision_at_1000
value: 0.488
- type: precision_at_3
value: 74.9
- type: precision_at_5
value: 62.96000000000001
- type: recall_at_1
value: 25.163999999999998
- type: recall_at_10
value: 85.97399999999999
- type: recall_at_100
value: 96.63000000000001
- type: recall_at_1000
value: 99.016
- type: recall_at_3
value: 55.611999999999995
- type: recall_at_5
value: 71.936
- task:
type: Retrieval
dataset:
type: C-MTEB/EcomRetrieval
name: MTEB EcomRetrieval
config: default
split: dev
revision: None
metrics:
- type: map_at_1
value: 48.6
- type: map_at_10
value: 58.831
- type: map_at_100
value: 59.427
- type: map_at_1000
value: 59.44199999999999
- type: map_at_3
value: 56.383
- type: map_at_5
value: 57.753
- type: mrr_at_1
value: 48.6
- type: mrr_at_10
value: 58.831
- type: mrr_at_100
value: 59.427
- type: mrr_at_1000
value: 59.44199999999999
- type: mrr_at_3
value: 56.383
- type: mrr_at_5
value: 57.753
- type: ndcg_at_1
value: 48.6
- type: ndcg_at_10
value: 63.951
- type: ndcg_at_100
value: 66.72200000000001
- type: ndcg_at_1000
value: 67.13900000000001
- type: ndcg_at_3
value: 58.882
- type: ndcg_at_5
value: 61.373
- type: precision_at_1
value: 48.6
- type: precision_at_10
value: 8.01
- type: precision_at_100
value: 0.928
- type: precision_at_1000
value: 0.096
- type: precision_at_3
value: 22.033
- type: precision_at_5
value: 14.44
- type: recall_at_1
value: 48.6
- type: recall_at_10
value: 80.10000000000001
- type: recall_at_100
value: 92.80000000000001
- type: recall_at_1000
value: 96.1
- type: recall_at_3
value: 66.10000000000001
- type: recall_at_5
value: 72.2
- task:
type: Classification
dataset:
type: C-MTEB/IFlyTek-classification
name: MTEB IFlyTek
config: default
split: validation
revision: None
metrics:
- type: accuracy
value: 47.36437091188918
- type: f1
value: 36.60946954228577
- task:
type: Classification
dataset:
type: C-MTEB/JDReview-classification
name: MTEB JDReview
config: default
split: test
revision: None
metrics:
- type: accuracy
value: 79.5684803001876
- type: ap
value: 42.671935929201524
- type: f1
value: 73.31912729103752
- task:
type: STS
dataset:
type: C-MTEB/LCQMC
name: MTEB LCQMC
config: default
split: test
revision: None
metrics:
- type: cos_sim_pearson
value: 68.62670112113864
- type: cos_sim_spearman
value: 75.74009123170768
- type: euclidean_pearson
value: 73.93002595958237
- type: euclidean_spearman
value: 75.35222935003587
- type: manhattan_pearson
value: 73.89870445158144
- type: manhattan_spearman
value: 75.31714936339398
- task:
type: Reranking
dataset:
type: C-MTEB/Mmarco-reranking
name: MTEB MMarcoReranking
config: default
split: dev
revision: None
metrics:
- type: map
value: 31.5372713650176
- type: mrr
value: 30.163095238095238
- task:
type: Retrieval
dataset:
type: C-MTEB/MMarcoRetrieval
name: MTEB MMarcoRetrieval
config: default
split: dev
revision: None
metrics:
- type: map_at_1
value: 65.054
- type: map_at_10
value: 74.156
- type: map_at_100
value: 74.523
- type: map_at_1000
value: 74.535
- type: map_at_3
value: 72.269
- type: map_at_5
value: 73.41
- type: mrr_at_1
value: 67.24900000000001
- type: mrr_at_10
value: 74.78399999999999
- type: mrr_at_100
value: 75.107
- type: mrr_at_1000
value: 75.117
- type: mrr_at_3
value: 73.13499999999999
- type: mrr_at_5
value: 74.13499999999999
- type: ndcg_at_1
value: 67.24900000000001
- type: ndcg_at_10
value: 77.96300000000001
- type: ndcg_at_100
value: 79.584
- type: ndcg_at_1000
value: 79.884
- type: ndcg_at_3
value: 74.342
- type: ndcg_at_5
value: 76.278
- type: precision_at_1
value: 67.24900000000001
- type: precision_at_10
value: 9.466
- type: precision_at_100
value: 1.027
- type: precision_at_1000
value: 0.105
- type: precision_at_3
value: 27.955999999999996
- type: precision_at_5
value: 17.817
- type: recall_at_1
value: 65.054
- type: recall_at_10
value: 89.113
- type: recall_at_100
value: 96.369
- type: recall_at_1000
value: 98.714
- type: recall_at_3
value: 79.45400000000001
- type: recall_at_5
value: 84.06
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (zh-CN)
config: zh-CN
split: test
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
metrics:
- type: accuracy
value: 68.1977135171486
- type: f1
value: 67.23114308718404
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (zh-CN)
config: zh-CN
split: test
revision: 7d571f92784cd94a019292a1f45445077d0ef634
metrics:
- type: accuracy
value: 71.92669804976462
- type: f1
value: 72.90628475628779
- task:
type: Retrieval
dataset:
type: C-MTEB/MedicalRetrieval
name: MTEB MedicalRetrieval
config: default
split: dev
revision: None
metrics:
- type: map_at_1
value: 49.2
- type: map_at_10
value: 54.539
- type: map_at_100
value: 55.135
- type: map_at_1000
value: 55.19199999999999
- type: map_at_3
value: 53.383
- type: map_at_5
value: 54.142999999999994
- type: mrr_at_1
value: 49.2
- type: mrr_at_10
value: 54.539
- type: mrr_at_100
value: 55.135999999999996
- type: mrr_at_1000
value: 55.19199999999999
- type: mrr_at_3
value: 53.383
- type: mrr_at_5
value: 54.142999999999994
- type: ndcg_at_1
value: 49.2
- type: ndcg_at_10
value: 57.123000000000005
- type: ndcg_at_100
value: 60.21300000000001
- type: ndcg_at_1000
value: 61.915
- type: ndcg_at_3
value: 54.772
- type: ndcg_at_5
value: 56.157999999999994
- type: precision_at_1
value: 49.2
- type: precision_at_10
value: 6.52
- type: precision_at_100
value: 0.8009999999999999
- type: precision_at_1000
value: 0.094
- type: precision_at_3
value: 19.6
- type: precision_at_5
value: 12.44
- type: recall_at_1
value: 49.2
- type: recall_at_10
value: 65.2
- type: recall_at_100
value: 80.10000000000001
- type: recall_at_1000
value: 93.89999999999999
- type: recall_at_3
value: 58.8
- type: recall_at_5
value: 62.2
- task:
type: Classification
dataset:
type: C-MTEB/MultilingualSentiment-classification
name: MTEB MultilingualSentiment
config: default
split: validation
revision: None
metrics:
- type: accuracy
value: 63.29333333333334
- type: f1
value: 63.03293854259612
- task:
type: PairClassification
dataset:
type: C-MTEB/OCNLI
name: MTEB Ocnli
config: default
split: validation
revision: None
metrics:
- type: cos_sim_accuracy
value: 75.69030860855442
- type: cos_sim_ap
value: 80.6157833772759
- type: cos_sim_f1
value: 77.87524366471735
- type: cos_sim_precision
value: 72.3076923076923
- type: cos_sim_recall
value: 84.37170010559663
- type: dot_accuracy
value: 67.78559826746074
- type: dot_ap
value: 72.00871467527499
- type: dot_f1
value: 72.58722247394654
- type: dot_precision
value: 63.57142857142857
- type: dot_recall
value: 84.58289334741288
- type: euclidean_accuracy
value: 75.20303194369248
- type: euclidean_ap
value: 80.98587256415605
- type: euclidean_f1
value: 77.26396917148362
- type: euclidean_precision
value: 71.03631532329496
- type: euclidean_recall
value: 84.68848996832101
- type: manhattan_accuracy
value: 75.20303194369248
- type: manhattan_ap
value: 80.93460699513219
- type: manhattan_f1
value: 77.124773960217
- type: manhattan_precision
value: 67.43083003952569
- type: manhattan_recall
value: 90.07391763463569
- type: max_accuracy
value: 75.69030860855442
- type: max_ap
value: 80.98587256415605
- type: max_f1
value: 77.87524366471735
- task:
type: Classification
dataset:
type: C-MTEB/OnlineShopping-classification
name: MTEB OnlineShopping
config: default
split: test
revision: None
metrics:
- type: accuracy
value: 87.00000000000001
- type: ap
value: 83.24372135949511
- type: f1
value: 86.95554191530607
- task:
type: STS
dataset:
type: C-MTEB/PAWSX
name: MTEB PAWSX
config: default
split: test
revision: None
metrics:
- type: cos_sim_pearson
value: 37.57616811591219
- type: cos_sim_spearman
value: 41.490259084930045
- type: euclidean_pearson
value: 38.9155043692188
- type: euclidean_spearman
value: 39.16056534305623
- type: manhattan_pearson
value: 38.76569892264335
- type: manhattan_spearman
value: 38.99891685590743
- task:
type: STS
dataset:
type: C-MTEB/QBQTC
name: MTEB QBQTC
config: default
split: test
revision: None
metrics:
- type: cos_sim_pearson
value: 35.44858610359665
- type: cos_sim_spearman
value: 38.11128146262466
- type: euclidean_pearson
value: 31.928644189822457
- type: euclidean_spearman
value: 34.384936631696554
- type: manhattan_pearson
value: 31.90586687414376
- type: manhattan_spearman
value: 34.35770153777186
- task:
type: STS
dataset:
type: mteb/sts22-crosslingual-sts
name: MTEB STS22 (zh)
config: zh
split: test
revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80
metrics:
- type: cos_sim_pearson
value: 66.54931957553592
- type: cos_sim_spearman
value: 69.25068863016632
- type: euclidean_pearson
value: 50.26525596106869
- type: euclidean_spearman
value: 63.83352741910006
- type: manhattan_pearson
value: 49.98798282198196
- type: manhattan_spearman
value: 63.87649521907841
- task:
type: STS
dataset:
type: C-MTEB/STSB
name: MTEB STSB
config: default
split: test
revision: None
metrics:
- type: cos_sim_pearson
value: 82.52782476625825
- type: cos_sim_spearman
value: 82.55618986168398
- type: euclidean_pearson
value: 78.48190631687673
- type: euclidean_spearman
value: 78.39479731354655
- type: manhattan_pearson
value: 78.51176592165885
- type: manhattan_spearman
value: 78.42363787303265
- task:
type: Reranking
dataset:
type: C-MTEB/T2Reranking
name: MTEB T2Reranking
config: default
split: dev
revision: None
metrics:
- type: map
value: 67.36693873615643
- type: mrr
value: 77.83847701797939
- task:
type: Retrieval
dataset:
type: C-MTEB/T2Retrieval
name: MTEB T2Retrieval
config: default
split: dev
revision: None
metrics:
- type: map_at_1
value: 25.795
- type: map_at_10
value: 72.258
- type: map_at_100
value: 76.049
- type: map_at_1000
value: 76.134
- type: map_at_3
value: 50.697
- type: map_at_5
value: 62.324999999999996
- type: mrr_at_1
value: 86.634
- type: mrr_at_10
value: 89.792
- type: mrr_at_100
value: 89.91900000000001
- type: mrr_at_1000
value: 89.923
- type: mrr_at_3
value: 89.224
- type: mrr_at_5
value: 89.608
- type: ndcg_at_1
value: 86.634
- type: ndcg_at_10
value: 80.589
- type: ndcg_at_100
value: 84.812
- type: ndcg_at_1000
value: 85.662
- type: ndcg_at_3
value: 82.169
- type: ndcg_at_5
value: 80.619
- type: precision_at_1
value: 86.634
- type: precision_at_10
value: 40.389
- type: precision_at_100
value: 4.93
- type: precision_at_1000
value: 0.513
- type: precision_at_3
value: 72.104
- type: precision_at_5
value: 60.425
- type: recall_at_1
value: 25.795
- type: recall_at_10
value: 79.565
- type: recall_at_100
value: 93.24799999999999
- type: recall_at_1000
value: 97.595
- type: recall_at_3
value: 52.583999999999996
- type: recall_at_5
value: 66.175
- task:
type: Classification
dataset:
type: C-MTEB/TNews-classification
name: MTEB TNews
config: default
split: validation
revision: None
metrics:
- type: accuracy
value: 47.648999999999994
- type: f1
value: 46.28925837008413
- task:
type: Clustering
dataset:
type: C-MTEB/ThuNewsClusteringP2P
name: MTEB ThuNewsClusteringP2P
config: default
split: test
revision: None
metrics:
- type: v_measure
value: 54.07641891287953
- task:
type: Clustering
dataset:
type: C-MTEB/ThuNewsClusteringS2S
name: MTEB ThuNewsClusteringS2S
config: default
split: test
revision: None
metrics:
- type: v_measure
value: 53.423702062353954
- task:
type: Retrieval
dataset:
type: C-MTEB/VideoRetrieval
name: MTEB VideoRetrieval
config: default
split: dev
revision: None
metrics:
- type: map_at_1
value: 55.7
- type: map_at_10
value: 65.923
- type: map_at_100
value: 66.42
- type: map_at_1000
value: 66.431
- type: map_at_3
value: 63.9
- type: map_at_5
value: 65.225
- type: mrr_at_1
value: 55.60000000000001
- type: mrr_at_10
value: 65.873
- type: mrr_at_100
value: 66.36999999999999
- type: mrr_at_1000
value: 66.381
- type: mrr_at_3
value: 63.849999999999994
- type: mrr_at_5
value: 65.17500000000001
- type: ndcg_at_1
value: 55.7
- type: ndcg_at_10
value: 70.621
- type: ndcg_at_100
value: 72.944
- type: ndcg_at_1000
value: 73.25399999999999
- type: ndcg_at_3
value: 66.547
- type: ndcg_at_5
value: 68.93599999999999
- type: precision_at_1
value: 55.7
- type: precision_at_10
value: 8.52
- type: precision_at_100
value: 0.958
- type: precision_at_1000
value: 0.098
- type: precision_at_3
value: 24.733
- type: precision_at_5
value: 16
- type: recall_at_1
value: 55.7
- type: recall_at_10
value: 85.2
- type: recall_at_100
value: 95.8
- type: recall_at_1000
value: 98.3
- type: recall_at_3
value: 74.2
- type: recall_at_5
value: 80
- task:
type: Classification
dataset:
type: C-MTEB/waimai-classification
name: MTEB Waimai
config: default
split: test
revision: None
metrics:
- type: accuracy
value: 84.54
- type: ap
value: 66.13603199670062
- type: f1
value: 82.61420654584116
---
<!-- TODO: add evaluation results here -->
<br><br>
<p align="center">
<img src="https://aeiljuispo.cloudimg.io/v7/https://cdn-uploads.huggingface.co/production/uploads/603763514de52ff951d89793/AFoybzd5lpBQXEBrQHuTt.png?w=200&h=200&f=face" alt="Finetuner logo: Finetuner helps you to create experiments in order to improve embeddings on search tasks. It accompanies you to deliver the last mile of performance-tuning for neural search applications." width="150px">
</p>
<p align="center">
<b>The text embedding set trained by <a href="https://jina.ai/"><b>Jina AI</b></a>.</b>
</p>
## Quick Start
The easiest way to starting using `jina-embeddings-v2-base-zh` is to use Jina AI's [Embedding API](https://jina.ai/embeddings/).
## Intended Usage & Model Info
`jina-embeddings-v2-base-zh` is a Chinese/English bilingual text **embedding model** supporting **8192 sequence length**.
It is based on a BERT architecture (JinaBERT) that supports the symmetric bidirectional variant of [ALiBi](https://arxiv.org/abs/2108.12409) to allow longer sequence length.
We have designed it for high performance in mono-lingual & cross-lingual applications and trained it specifically to support mixed Chinese-English input without bias.
Additionally, we provide the following embedding models:
`jina-embeddings-v2-base-zh` 是支持中英双语的**文本向量**模型，它支持长达**8192字符**的文本编码。
该模型的研发基于BERT架构(JinaBERT)，JinaBERT是在BERT架构基础上的改进，首次将[ALiBi](https://arxiv.org/abs/2108.12409)应用到编码器架构中以支持更长的序列。
不同于以往的单语言/多语言向量模型，我们设计双语模型来更好的支持单语言（中搜中）以及跨语言（中搜英）文档检索。
除此之外，我们也提供其它向量模型:
- [`jina-embeddings-v2-small-en`](https://huggingface.co/jinaai/jina-embeddings-v2-small-en): 33 million parameters.
- [`jina-embeddings-v2-base-en`](https://huggingface.co/jinaai/jina-embeddings-v2-base-en): 137 million parameters.
- [`jina-embeddings-v2-base-zh`](https://huggingface.co/jinaai/jina-embeddings-v2-base-zh): 161 million parameters Chinese-English Bilingual embeddings **(you are here)**.
- [`jina-embeddings-v2-base-de`](https://huggingface.co/jinaai/jina-embeddings-v2-base-de): 161 million parameters German-English Bilingual embeddings.
- [`jina-embeddings-v2-base-es`](): Spanish-English Bilingual embeddings (soon).
- [`jina-embeddings-v2-base-code`](https://huggingface.co/jinaai/jina-embeddings-v2-base-code): 161 million parameters code embeddings.
## Data & Parameters
The data and training details are described in this [technical report](https://arxiv.org/abs/2402.17016).
## Usage
**<details><summary>Please apply mean pooling when integrating the model.</summary>**
<p>
### Why mean pooling?
`mean poooling` takes all token embeddings from model output and averaging them at sentence/paragraph level.
It has been proved to be the most effective way to produce high-quality sentence embeddings.
We offer an `encode` function to deal with this.
However, if you would like to do it without using the default `encode` function:
```python
import torch
import torch.nn.functional as F
from transformers import AutoTokenizer, AutoModel
def mean_pooling(model_output, attention_mask):
token_embeddings = model_output[0]
input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)
sentences = ['How is the weather today?', '今天天气怎么样?']
tokenizer = AutoTokenizer.from_pretrained('jinaai/jina-embeddings-v2-base-zh')
model = AutoModel.from_pretrained('jinaai/jina-embeddings-v2-base-zh', trust_remote_code=True, torch_dtype=torch.bfloat16)
encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')
with torch.no_grad():
model_output = model(**encoded_input)
embeddings = mean_pooling(model_output, encoded_input['attention_mask'])
embeddings = F.normalize(embeddings, p=2, dim=1)
```
</p>
</details>
You can use Jina Embedding models directly from transformers package.
```python
!pip install transformers
import torch
from transformers import AutoModel
from numpy.linalg import norm
cos_sim = lambda a,b: (a @ b.T) / (norm(a)*norm(b))
model = AutoModel.from_pretrained('jinaai/jina-embeddings-v2-base-zh', trust_remote_code=True, torch_dtype=torch.bfloat16)
embeddings = model.encode(['How is the weather today?', '今天天气怎么样?'])
print(cos_sim(embeddings[0], embeddings[1]))
```
If you only want to handle shorter sequence, such as 2k, pass the `max_length` parameter to the `encode` function:
```python
embeddings = model.encode(
['Very long ... document'],
max_length=2048
)
```
If you want to use the model together with the [sentence-transformers package](https://github.com/UKPLab/sentence-transformers/), make sure that you have installed the latest release and set `trust_remote_code=True` as well:
```python
!pip install -U sentence-transformers
from sentence_transformers import SentenceTransformer
from numpy.linalg import norm
cos_sim = lambda a,b: (a @ b.T) / (norm(a)*norm(b))
model = SentenceTransformer('jinaai/jina-embeddings-v2-base-zh', trust_remote_code=True)
embeddings = model.encode(['How is the weather today?', '今天天气怎么样?'])
print(cos_sim(embeddings[0], embeddings[1]))
```
Using the its latest release (v2.3.0) sentence-transformers also supports Jina embeddings (Please make sure that you are logged into huggingface as well):
```python
!pip install -U sentence-transformers
from sentence_transformers import SentenceTransformer
from sentence_transformers.util import cos_sim
model = SentenceTransformer(
"jinaai/jina-embeddings-v2-base-zh", # switch to en/zh for English or Chinese
trust_remote_code=True
)
# control your input sequence length up to 8192
model.max_seq_length = 1024
embeddings = model.encode([
'How is the weather today?',
'今天天气怎么样?'
])
print(cos_sim(embeddings[0], embeddings[1]))
```
## Alternatives to Using Transformers Package
1. _Managed SaaS_: Get started with a free key on Jina AI's [Embedding API](https://jina.ai/embeddings/).
2. _Private and high-performance deployment_: Get started by picking from our suite of models and deploy them on [AWS Sagemaker](https://aws.amazon.com/marketplace/seller-profile?id=seller-stch2ludm6vgy).
## Use Jina Embeddings for RAG
According to the latest blog post from [LLamaIndex](https://blog.llamaindex.ai/boosting-rag-picking-the-best-embedding-reranker-models-42d079022e83),
> In summary, to achieve the peak performance in both hit rate and MRR, the combination of OpenAI or JinaAI-Base embeddings with the CohereRerank/bge-reranker-large reranker stands out.
<img src="https://miro.medium.com/v2/resize:fit:4800/format:webp/1*ZP2RVejCZovF3FDCg-Bx3A.png" width="780px">
## Trouble Shooting
**Loading of Model Code failed**
If you forgot to pass the `trust_remote_code=True` flag when calling `AutoModel.from_pretrained` or initializing the model via the `SentenceTransformer` class, you will receive an error that the model weights could not be initialized.
This is caused by tranformers falling back to creating a default BERT model, instead of a jina-embedding model:
```bash
Some weights of the model checkpoint at jinaai/jina-embeddings-v2-base-zh were not used when initializing BertModel: ['encoder.layer.2.mlp.layernorm.weight', 'encoder.layer.3.mlp.layernorm.weight', 'encoder.layer.10.mlp.wo.bias', 'encoder.layer.5.mlp.wo.bias', 'encoder.layer.2.mlp.layernorm.bias', 'encoder.layer.1.mlp.gated_layers.weight', 'encoder.layer.5.mlp.gated_layers.weight', 'encoder.layer.8.mlp.layernorm.bias', ...
```
**User is not logged into Huggingface**
The model is only availabe under [gated access](https://huggingface.co/docs/hub/models-gated).
This means you need to be logged into huggingface load load it.
If you receive the following error, you need to provide an access token, either by using the huggingface-cli or providing the token via an environment variable as described above:
```bash
OSError: jinaai/jina-embeddings-v2-base-zh is not a local folder and is not a valid model identifier listed on 'https://huggingface.co/models'
If this is a private repository, make sure to pass a token having permission to this repo with `use_auth_token` or log in with `huggingface-cli login` and pass `use_auth_token=True`.
```
## Contact
Join our [Discord community](https://discord.jina.ai) and chat with other community members about ideas.
## Citation
If you find Jina Embeddings useful in your research, please cite the following paper:
```
@article{mohr2024multi,
title={Multi-Task Contrastive Learning for 8192-Token Bilingual Text Embeddings},
author={Mohr, Isabelle and Krimmel, Markus and Sturua, Saba and Akram, Mohammad Kalim and Koukounas, Andreas and G{\"u}nther, Michael and Mastrapas, Georgios and Ravishankar, Vinit and Mart{\'\i}nez, Joan Fontanals and Wang, Feng and others},
journal={arXiv preprint arXiv:2402.17016},
year={2024}
}
```