Evaluation and contributing#

Improve a guide, a client integration or the memory engine through the repository's CONTRIBUTING guide. Website Markdown is the canonical user-guide source; detailed source-development instructions remain in the Ormah repository.

Source development#

Use a source checkout and the repository's development installation (make install in an activated Python environment), then its tests. Eval modules are development tooling; a normal installed wheel may not contain the source eval/ package.

For a docs change, update the relevant website article and its source evidence. For runtime changes, include a synthetic reproduction and focused tests. Report issues without sharing personal memories, credentials, recovery kits or transcripts.

Retrieval evaluations#

Ormah has separate Whisper and recall harnesses:

ormah eval whisper run --show-failures
ormah eval recall run

These commands require the source development environment and an appropriate local corpus. make eval runs the repository's quality-bar workflow. It is not a prerequisite for reading the docs or connecting a client.

The CLI also contains live-log mining commands. They read sensitive operational data and are not part of a safe synthetic smoke test. Use deliberate, local data handling when building a private evaluation corpus.

Honest evidence#

Corpora can contain real memory and are intentionally local-only/gitignored. Do not commit them to make a run reproducible. Prefer synthetic, reviewable cases for public tests.

For a useful evaluation report record source revision, model/provider configuration, corpus/version or non-sensitive fixture description, labeled-case policy, command, metric definitions and failure cases. Separate provisional labels from confirmed ones, and simulation from observed client delivery.

Historical local scores without that provenance do not describe current product quality. This page publishes no floating benchmark snapshot and makes no numerical performance promise. The docs modernization did not run paid retrieval benchmarks.

Documentation checks#

This website keeps one route inventory for publishing, navigation, sitemap and search, checks emitted internal links/fragments, and records a pinned source contract. See the website repository (access required) for its docs checks and contribution notes.

Every article provides its public Markdown source. Maintainers with website-repository access can use its Edit link; anyone can suggest a docs correction. Source review dates must advance only after a real review, never automatically at build time.