---
description: Contribute from a source checkout and evaluate retrieval reproducibly without publishing private corpora.
---

# Evaluation and contributing

Improve a guide, a client integration or the memory engine through the repository's [CONTRIBUTING guide](https://github.com/r-spade/ormah/blob/c1fa7073f0f20e25701002a7a3071d28730787d2/CONTRIBUTING.md). Website Markdown is the canonical user-guide source; detailed source-development instructions remain in the Ormah repository.

## Source development

Use a source checkout and the repository's development installation (`make install` in an activated Python environment), then its tests. Eval modules are development tooling; a normal installed wheel may not contain the source `eval/` package.

For a docs change, update the relevant website article and its source evidence. For runtime changes, include a synthetic reproduction and focused tests. Report issues without sharing personal memories, credentials, recovery kits or transcripts.

## Retrieval evaluations

Ormah has separate **Whisper** and **recall** harnesses:

```bash
ormah eval whisper run --show-failures
ormah eval recall run
```

These commands require the source development environment and an appropriate local corpus. `make eval` runs the repository's quality-bar workflow. It is not a prerequisite for reading the docs or connecting a client.

| Harness | Measures / options |
| --- | --- |
| Whisper | Injection and suppression against labeled cases; category filtering, session simulation, JSON output and fail-below bars |
| Recall | Retrieval against the selected corpus; top-k, fail-below and regression checks |
| Labeling workflows | Mine/import/export provisional pairs, then human review before confirmed labels |

The CLI also contains live-log mining commands. They read sensitive operational data and are not part of a safe synthetic smoke test. Use deliberate, local data handling when building a private evaluation corpus.

## Honest evidence

Corpora can contain real memory and are intentionally local-only/gitignored. Do not commit them to make a run reproducible. Prefer synthetic, reviewable cases for public tests.

For a useful evaluation report record source revision, model/provider configuration, corpus/version or non-sensitive fixture description, labeled-case policy, command, metric definitions and failure cases. Separate provisional labels from confirmed ones, and simulation from observed client delivery.

Historical local scores without that provenance do not describe current product quality. This page publishes no floating benchmark snapshot and makes no numerical performance promise. The docs modernization did not run paid retrieval benchmarks.

## Documentation checks

This website keeps one route inventory for publishing, navigation, sitemap and search, checks emitted internal links/fragments, and records a pinned source contract. See [the website repository (access required)](https://github.com/r-spade/ormah-web) for its docs checks and contribution notes.

Every article provides its public Markdown source. Maintainers with website-repository access can use its Edit link; anyone can [suggest a docs correction](https://github.com/r-spade/ormah/issues/new?title=Docs%3A%20correction). Source review dates must advance only after a real review, never automatically at build time.