# mcp-eval-harness

Contract tests for MCP servers: expected tools, schema checks and golden fixtures, with CLI exit codes that can gate a release.

A shared test harness for MCP servers. The mcp-eval CLI loads a YAML or JSON suite, connects to a stdio or Streamable HTTP target, runs contracts through the official MCP client, compares results with golden fixtures and JSON Schemas, and exits non-zero on a failed contract. It checks the published interface rather than judging answer quality.

Source: https://github.com/SarutobiSasuke8/mcp-eval-harness
Docs: https://github.com/SarutobiSasuke8/mcp-eval-harness#readme
Listing: https://mcprack.dev/servers/mcp-eval-harness/
Status: alpha
Transport: CLI (stdio or Streamable HTTP targets)
License: MIT
Review: not yet recorded
Package: Source only

## Highlights

- Checks exact tool, prompt and resource lists, plus advertised input schemas
- Golden comparisons with only_paths and ignore_paths for timestamps and request ids
- Error-path assertions for tool errors and JSON-RPC error codes, over stdio or Streamable HTTP
- Golden files change only with an explicit --update-goldens flag, never implicitly

## Current limitations

- Version 0.1.0 on main. Publish-ready, but not published to npm or tagged; install from git.
- Prompt and resource listing contracts are supported; content goldens for those surfaces are deferred.
- This is not an LLM judge or a hosted service.

## Setup

No install command recorded; consult the repository.

Check prerequisites, permissions and current versions. Inclusion is not an independent security audit.