CIRCL/vulnerability-severity-classification-chinese-macbert-base

VLAI: A RoBERTa-Based Model for Automated Vulnerability Severity Classification (Chinese Text) 🇨🇳

This model is a fine-tuned version of hfl/chinese-macbert-base on the dataset CIRCL/Vulnerability-CNVD.

For more information, visit the Vulnerability-Lookup project page or the ML-Gateway GitHub repository, which demonstrates its usage in a FastAPI server.

How to use

You can use this model directly with the Hugging Face transformers library for text classification:

from transformers import pipeline

classifier = pipeline(
    "text-classification",
    model="CIRCL/vulnerability-severity-classification-chinese-macbert-base"
)

# Example usage for a Chinese vulnerability description
description_chinese = "TOTOLINK A3600R是中国吉翁电子（TOTOLINK）公司的一款6天线1200M无线路由器。TOTOLINK A3600R存在缓冲区溢出漏洞，该漏洞源于/cgi-bin/cstecgi.cgi文件的UploadCustomModule函数中的File参数未能正确验证输入数据的长度大小，攻击者可利用该漏洞在系统上执行任意代码或者导致拒绝服务。"
result_chinese = classifier(description_chinese)
print(result_chinese)
# Expected output example: [{'label': '高', 'score': 0.9802}]

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 3e-05
• train_batch_size: 32
• eval_batch_size: 32
• seed: 42
• optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• num_epochs: 5

It achieves the following results on the evaluation set:

• Loss: 1.3186
• Accuracy: 0.7657
• F1 Macro: 0.6796
• Low Precision: 0.5544
• Low Recall: 0.3987
• Low F1: 0.4638
• Medium Precision: 0.7805
• Medium Recall: 0.8196
• Medium F1: 0.7996
• High Precision: 0.7787
• High Recall: 0.7720
• High F1: 0.7753

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy	F1 Macro	Low Precision	Low Recall	Low F1	Medium Precision	Medium Recall	Medium F1	High Precision	High Recall	High F1
1.2165	1.0	3221	1.2363	0.7429	0.5960	0.6380	0.1631	0.2598	0.7335	0.8549	0.7895	0.7679	0.7115	0.7386
1.1430	2.0	6442	1.1676	0.7625	0.6464	0.6316	0.2643	0.3726	0.7548	0.8568	0.8026	0.7909	0.7386	0.7639
0.8890	3.0	9663	1.1915	0.7631	0.6690	0.5884	0.3470	0.4365	0.7833	0.8091	0.7960	0.7564	0.7933	0.7744
0.8253	4.0	12884	1.2354	0.7675	0.6796	0.5739	0.3874	0.4626	0.7765	0.8305	0.8026	0.7849	0.7630	0.7738
0.5851	5.0	16105	1.3186	0.7657	0.6796	0.5544	0.3987	0.4638	0.7805	0.8196	0.7996	0.7787	0.7720	0.7753

Framework versions

• Transformers 5.9.0
• Pytorch 2.12.0+cu130
• Datasets 4.8.5
• Tokenizers 0.22.2