Servers / mcp-eval-harness
mcp-eval-harness
Contract tests for MCP servers: expected tools, schema checks and golden fixtures, with CLI exit codes that can gate a release.
Owner-built project. Status is maintainer-reported; inclusion is not an independent audit. Version information may lag upstream releases. Listing standards · Read as Markdown
A shared test harness for MCP servers. The mcp-eval CLI loads a YAML or JSON suite, connects to a stdio or Streamable HTTP target, runs contracts through the official MCP client, compares results with golden fixtures and JSON Schemas, and exits non-zero on a failed contract. It checks the published interface rather than judging answer quality.
Review and maturity
No dated review has been recorded for this listing.
What it does
- Checks exact tool, prompt and resource lists, plus advertised input schemas
- Golden comparisons with only_paths and ignore_paths for timestamps and request ids
- Error-path assertions for tool errors and JSON-RPC error codes, over stdio or Streamable HTTP
- Golden files change only with an explicit --update-goldens flag, never implicitly
Current limitations
- Version 0.1.0 on main. Publish-ready, but not published to npm or tagged; install from git.
- Prompt and resource listing contracts are supported; content goldens for those surfaces are deferred.
- This is not an LLM judge or a hosted service.