🏆 S-Eval Leaderboard

🔔 Updates

📣 [2025/03/30]: 🎉 Our paper has been accepted by ISSTA 2025. To meet evaluation needs under different budgets, we partition the benchmark into four scales: Small (1,000 Base and 10,000 Attack in each language), Medium (3,000 Base and 30,000 Attack in each language), Large (5,000 Base and 50,000 Attack in each language) and Full (10,000 Base and 100,000 Attack in each language), comprehensively considering the balance and harmfulness of data.

📣 [2024/10/25]: We release all 20,000 base risk prompts and 200,000 corresponding attack prompts (Version-0.1.2). We also update 🏆 LeaderBoard with new evaluation results including GPT-4 and other models.
🎉 S-Eval has achieved about 7,000 total views and about 2,000 total downloads across multiple platforms. 🎉

📣 [2024/06/17]: We further release 10,000 base risk prompts and 100,000 corresponding attack prompts (Version-0.1.1). If you require automatic safety evaluations, please feel free to submit a request via Issues or contact us by Email.

📣 [2024/05/31]: We release 20,000 corresponding attack prompts.

📣 [2024/05/23]: We publish our paper and first release 2,000 base risk prompts. You can download the benchmark from our project, the HuggingFace Dataset.

❗️ Note

Due to the limited machine resource, please refresh the page if a connection timeout error occurs.

You can get more detailed information from our Project and Paper.

In the table below, we summarize the safety scores (%) of differnet models on Base Risk Prompt Set.

Select whether Chinese or English results should be shown.

Chinese English


Baichuan2-13B-Chat	79.7	89.44	72.86	87	57	73	89.29	83.33	58.33


ErnieBot-4.0	79.7	89.44	85	87	57	73	89.29	87.5	58.33
Qwen-72B-Chat	73.1	83.33	72.86	83	58	86	63.57	83.33	52.5
Baichuan2-13B-Chat	66.6	74.44	70	79	47	77	65	68.33	48.33
Qwen-14B-Chat	66.5	75	76.43	80	38	77	52.14	74.17	55
Qwen-7B-Chat	63.3	71.11	70	83	42	74	46.29	75.83	43.33
Qwen-1.8B-Chat	60.5	57.78	65	75	36	71	60	78.33	41.67
ChatGLM3-6B	59.7	60.56	72.14	68	37	61	57.86	66.67	50
GPT-4-Turbo	57.7	58.33	62.14	56	41	78	68.57	55	40
GPT-4o	54	52.22	60.71	68	33	68	69.29	55.83	23.33
Gemini-1.0-Pro	53.9	56.11	61.43	67	50	54	35.71	65.83	43.33
Gemma-7B-it	49.6	48.33	59.29	60	31	70	39.29	58.33	33.33
Yi-34B-Chat	46.7	50	48.57	60	25	81	27.14	35.83	51.67
Gemma-2B-it	42.3	42.78	42.86	53	26	76	27.86	43.33	33.33


Baichuan2-13B-Chat	61.86	99.8	86.4	77.9	16.6	62.4	36.2	17.9	64.8	69.4	80.6	84.7


Baichuan2-13B-Chat	61.86	99.8	86.4	77.9	20	79	36.2	2.2	64.8	69.4	98	84.7
ChatGLM3-6B	53.95	99.4	66.9	70.9	9.8	62.4	33.9	0.5	60.7	64.8	80	89.6
Yi-34B-Chat	53.82	99.7	89.3	64.9	16.6	53.7	34.7	7.8	25.5	70.4	95	80.3
Gemini-1.0-Pro	53.04	99.2	57.9	83.6	2.1	55.9	18.2	3.6	69.6	66.9	80.6	92
Gemma-7B-it	52.15	99.8	67.2	73.8	33.5	55.1	23.7	0.3	36.3	67.2	83.8	80.6
Qwen-14B-Chat	51.62	99.7	72.1	72.1	4.8	68	18.8	0.5	51.8	48.5	90.1	89.5
Qwen-72B-Chat	49.49	99.8	57.9	70.3	3.3	76.5	16.3	8.8	39.5	35.6	98.6	88.1
Qwen-1.8B-Chat	46.4	99	69.2	77.6	7.2	50.4	21.2	2.5	41.5	64.3	48.4	81.7
GPT-4o	40.22	97.7	60.8	82.7	29.3	13.3	32.4	20	46.4	27.5	2.5	87.3
ErnieBot-4.0	36.54	95.2	40.7	65.2	13.3	52.3	21.4	17.9	41.5	35.7	2	75.4
GPT-4-Turbo	33.99	95.1	52.3	71.1	21	17	27.9	12.6	20.6	35.4	0.3	81.7

If our work is useful for your own, you can cite us with the following BibTex entry: