评测¶
评测框架针对 Agent 运行脚本化用例并比较各 Provider。用例从 YAML/JSON 套件加载;结果携带逐用例的分数与对话记录。
评测框架¶
EvalCase
dataclass
¶
EvalCase(
name: str,
input: str,
expected_keywords: list[str] = list(),
expected_tools: list[str] = list(),
must_not_contain: list[str] = list(),
max_turns: int = 10,
)
A single evaluation test case.
input is the user prompt. expected_keywords are terms that should
appear in a successful response. expected_tools are tools that should
be called. must_not_contain flags failure indicators.
expected_keywords
class-attribute
instance-attribute
¶
expected_keywords: list[str] = field(default_factory=list)
expected_tools
class-attribute
instance-attribute
¶
expected_tools: list[str] = field(default_factory=list)
must_not_contain
class-attribute
instance-attribute
¶
must_not_contain: list[str] = field(default_factory=list)
EvalResult
dataclass
¶
EvalResult(
case_name: str,
passed: bool,
score: float,
response: str = "",
tools_used: list[str] = list(),
errors: list[str] = list(),
duration: float = 0.0,
)
Result of running one EvalCase.
to_dict
¶
to_dict() -> dict[str, Any]
Source code in src/dashscope_agenticx/eval/__init__.py
50 51 52 53 54 55 56 57 58 59 | |
EvalRunner
¶
EvalRunner(agent_factory: Any = None)
Runs EvalCases against an Agent and scores results.
Scoring
- 0.4: keyword overlap (expected_keywords in response)
- 0.3: tool usage (expected_tools were called)
- 0.2: no forbidden content (must_not_contain absent)
- 0.1: no errors
agent_factory(config) -> Agent creates a fresh agent per
case.
Source code in src/dashscope_agenticx/eval/__init__.py
72 73 74 75 | |
run_case
async
¶
run_case(case: EvalCase, agent: Any = None) -> EvalResult
Run a single evaluation case.
Source code in src/dashscope_agenticx/eval/__init__.py
77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 | |
run_suite
async
¶
run_suite(
cases: list[EvalCase], agent: Any = None
) -> list[EvalResult]
Run multiple cases. Returns results in order.
Source code in src/dashscope_agenticx/eval/__init__.py
166 167 168 169 170 171 172 173 174 175 176 | |
ProviderComparator
¶
ProviderComparator(runner: EvalRunner | None = None)
A/B comparison of two providers on the same eval suite.
Runs the same cases against two agent configurations (e.g. tongyi vs anthropic) and produces a comparison report.
Source code in src/dashscope_agenticx/eval/__init__.py
186 187 | |
compare
async
¶
compare(
cases: list[EvalCase],
agent_a: Any,
agent_b: Any,
label_a: str = "A",
label_b: str = "B",
) -> dict[str, Any]
Run cases against both agents and return a comparison report.
Source code in src/dashscope_agenticx/eval/__init__.py
189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 | |
load_eval_suite
¶
load_eval_suite(path: str | Path) -> list[EvalCase]
Load eval cases from a JSON file.
Format:
[{"name": "...", "input": "...", "expected_keywords": [...], ...}]
Source code in src/dashscope_agenticx/eval/__init__.py
229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 | |
异常¶
AgenticX 将模型与工具失败以 error 事件的形式呈现在
Agent 事件流中,而非统一的异常体系。少数确实
存在的异常属于控制流信号或平台专属异常:
UserAbortedTurn/UserSupplement— 交互式轮次控制。BailianCLIError/MCPError— 百炼平台故障 (平台)。RetriesExhausted— Provider 加固层放弃重试 (Provider)。
UserAbortedTurn
¶
Bases: Exception
Raised when the user picks [s]top at a confirmation prompt — signals the agent loop to bail out entirely (not just skip one tool call).
UserSupplement
¶
UserSupplement(supplement: str)
Bases: Exception
Raised when the user picks [u]pdate at a confirmation prompt — signals the agent loop to inject supplemental info into the conversation and re-plan before continuing tool execution.
Source code in src/dashscope_agenticx/utils/exceptions.py
17 18 19 | |