Evaluation
Evaluate on realsecurity work.
A useful benchmark needs repeatable tasks, ground truth, and reported failure cases. Model inference is still being connected, so results are not published yet.
| Task family | Evidence required |
|---|---|
| Shellcode offset mapping | Check offset reasoning against a known binary and debugger trace. |
| PowerShell deobfuscation | Compare decoded stages with a manually verified sample. |
| API authorization test planning | Review proposed cases, then execute them against an authorized fixture. |
| Ghidra dump triage | Check explanations against annotated decompiler output. |
No performance, refusal-rate, or latency score is claimed until the evaluation is run and published.