Test documentation

The following catalogues separate template checks from algorithm acceptance checks. See also Shared-package tests for the runtime and contract suite.

Template test catalogue

These tests validate the generator and rendered repository structure. Shared runtime/contract tests have their own catalogue, rendered from the installed package on the documentation site's Shared-package tests page. Generated algorithm acceptance tests are documented separately in the next section of the site and in docs/TESTING.md inside each generated repository. Keep these three scopes distinct.

Running template tests

uv sync --group dev --group docs
uv run ruff check src tests scripts docs/hooks.py
uv run ruff format --check src tests scripts docs/hooks.py
uv run pytest -q
uv run pytest tests/test_profiles.py -q
uv run mkdocs build --strict

Tests use temporary renders, toy implementations and synthetic data. MEDS-DEV provisioning is mocked or exercised through capability fixtures in this suite; passing it does not mean a real clinical dataset was run. Parametrized functions produce several pytest cases. Helper fixtures are not separate acceptance tests.

Test inventory

test_generator.py

Source

Test	What it checks	Why it exists
test_generator_binds_installed_version_and_forwards_answers	Checks repository URL, installed-version tag, answers and Copier options passed by the launcher.	Generate from the version the user installed and preserve reproducible updates.
test_bad_answers_fail_before_copy	Parametrizes invalid answer-file contents and requires failure before generation.	Avoid partial projects from malformed user input.
test_version_does_not_generate	Runs the version option without invoking Copier.	Keep informational CLI commands free of generation side effects.

test_profiles.py

Source

Test	What it checks	Why it exists
test_algorithm_name_controls_module_classes_exports_and_command	Renders a named algorithm and checks its module, classes, exports and console entry point.	Make concrete algorithms meaningfully importable and executable.
test_invalid_or_colliding_algorithm_names_are_rejected	Parametrizes invalid slugs/class names and collisions with reserved packages.	Reject generated projects that cannot import or would shadow their dependencies.
test_all_profiles_coexist_and_selection_is_required	Renders all profiles and checks explicit selection when a repository is ambiguous.	Support algorithm families without silently choosing a lifecycle.
test_slurm_forwards_dataset_configuration_to_every_step	Checks propagation of shared dataset/predicate/config arguments through the launcher.	Prevent one pipeline step from using a different data or configuration binding.
test_descriptors_dispatch_only_their_profile_with_complete_arguments	Checks each MEDS-DEV descriptor against its profile and required step arguments.	Keep generated external command wiring aligned with the core contract.
test_registration_keeps_profile_descriptors_separate	Registers several profiles independently into MEDS-DEV.	Avoid overwriting one profile with another in a family repository.
test_generated_requests_reject_unknown_arguments	Dispatches generated CLI requests with unknown arguments.	Fail configuration typos instead of silently ignoring them.
test_slurm_selects_one_profile_and_isolates_default_outputs	Checks profile selection and default artifact/run destinations.	Prevent cross-profile artifact collisions during cluster execution.
test_generated_acceptance_tests_cover_every_profile	Collects generated acceptance tests and checks their profile parametrization.	Ensure an implemented family cannot leave a declared profile untested.
test_recorded_upstream_inputs_become_explicit_arguments	Resolves recorded upstream inputs and rejects nonexistent artifact-input roles.	Make reused artifacts supply the correct explicit downstream inputs.
test_rendered_spec_points_to_core_and_contract_is_not_copied	Checks shared imports and absence of generated copies of the contract package.	Keep fixes and normative specifications owned by the shared package.
test_rendered_report_gate_checks_completed_profile_with_other_stubs	Completes one profile while others remain stubs and checks the report gate.	Require evidence for completed profiles even in partly implemented families.

test_render.py

Source

Test	What it checks	Why it exists
test_renders_six_algorithm_profiles	Renders each profile, compiles Python and inspects its generated contract/operational files.	Catch generation failures across the full supported profile set.
test_copier_answers_record_portable_update_metadata	Checks the answers file retains update provenance.	Allow later Copier updates without reconstructing how the project was generated.
test_mimic_task_prepends_pinned_meds_dev_venv_only_for_task	Inspects environment arguments used for dataset and task preparation.	Use the intended task tool without globally changing unrelated command resolution.
test_example_signal_dataset_has_no_length_or_position_leak	Checks that labels match signal-code presence, with variable history lengths and both classes.	Sanity-check the example signal fixture; this is not a statistical proof that all shortcuts are absent.
test_primary_behavior_is_a_required_failing_placeholder	Checks the generated test requires real signal binding and a shuffled-label control.	Do not let an unimplemented scientific behavior test pass by default.
test_predicates_are_external_not_a_rendered_semantic_format	Renders without predicates and checks the descriptor and absence of a semantics file.	Avoid inventing an algorithm feature representation in the template.
test_implementation_prompt_belongs_to_template_readme_not_rendered_repo	Checks the copyable implementation prompt exists only in the template README.	Avoid shipping duplicate agent instructions into each generated project.
test_spec_has_one_template_source_and_renders_to_root	Checks the generated SPEC uses the single template source.	Prevent competing copies of generated specification guidance.
test_prediction_keys_are_compared_directly	Parametrizes reordered, missing, extra and duplicate prediction rows.	Judge cohort coverage by keys rather than row count or row order.
test_meds_dev_predicates_capability_is_explicit	Checks MEDS-DEV capability detection for predicate passthrough.	Fail unsupported integrations instead of silently dropping dataset semantics.
test_only_zero_shot_meds_dev_runs_require_task_criteria	Checks which profile adapters require task-criteria capability.	Require task support where needed without imposing it on other profiles.
test_meds_dev_ref_is_checked_out_in_clone	Inspects the configured revision used when provisioning MEDS-DEV.	Keep integration runs reproducible rather than following an implicit default branch.
test_measurement_manifest_is_plausible	Exercises measured publication and inspects recorded resources/provenance.	Catch missing or implausible run evidence at the integration boundary.
test_only_zero_shot_profiles_receive_the_task	Checks task arguments in generated zero-shot versus other profile wiring.	Keep task-independent preprocessing/inference independent of the evaluation task.
test_zero_shot_stub_rejects_a_missing_task_before_work	Invokes the generated CLI with missing/invalid task inputs.	Validate the boundary before reaching algorithm implementation.
test_task_spec_ledger_and_label_blind_index	Checks task/predicate digests, relaxation-ledger round trips, index projection and truth attachment.	Preserve task semantics and keep scoring inputs label-blind.
test_predictions_and_cli_lists_follow_meds_dev	Checks prediction layout and CLI list argument handling across profiles.	Keep generated commands consumable by MEDS-DEV.
test_meds_dev_evaluates_every_parquet_under_predictions	Checks handling of recursively stored prediction files.	Avoid omitting shards during evaluation.
test_artifacts_are_siblings_and_slurm_lists_only_profile_commands	Checks artifact layout and profile-specific launcher commands.	Avoid nested artifact overlap and accidental execution of unrelated steps.
test_contract_helpers_for_conformance_checks	Exercises shared conformance helpers through a rendered repository.	Ensure generated projects can use the installed core test utilities.
test_predict_only_reads_context_labels_never_evaluated_ones	Checks training-context access and forbids evaluated outcomes.	Protect predict-only/in-context scoring from target leakage.
test_label_leakage_check_strips_what_the_profile_may_not_read	Checks forbidden outcomes are removed according to each profile's permissions.	Make the adversarial rerun respect training/context permissions while withholding evaluation truth.
test_submit_stages_reuse_shared_task_independent_artifacts	Checks staged SLURM execution reuses preprocessing/pretraining as intended.	Avoid repeating shared work or confusing task-specific and independent artifacts.
test_submit_overrides_resources_args_and_sbatch_flags_from_the_environment	Supplies launcher environment overrides and checks the resulting command/resource flags.	Let documented run overrides take effect predictably.

test_typed_execution.py

Source

Test	What it checks	Why it exists
test_rendered_algorithm_runs_persisted_lifecycle_with_structured_config	Implements a tiny rendered algorithm, runs its lifecycle and reloads fitted state with structured configuration.	Demonstrate the generated API works beyond empty stubs and records resolved settings.

Release checks outside pytest

tests/wheel_smoke.py installs/executes the generator against a local tagged template mirror. It checks single-profile and all-profile generation, compilation, canonical source URL and recorded release tag. It detects packaging and version/source mistakes without requiring a real release tag to exist yet.

scripts/check_release.py requires matching exact PyPI pins for runtime, conformance and development uses of the shared package, then checks that the pinned version has downloadable files on PyPI. A Git pin used to review unreleased core changes deliberately fails this release gate. Strict MkDocs checks links/builds; Twine checks built distribution metadata. Neither proves scientific correctness.

Update a catalogue row whenever its test is added, renamed or changes purpose. A green template suite is evidence about the template, not evidence that a generated algorithm has been implemented.

Tests in an algorithm repository

This guide explains the acceptance tests generated with every algorithm repository. It is independent of the shared package's unit tests and of the template's rendering tests. Algorithms must implement their own scientific checks; generic contract checks cannot prove fidelity to a paper or clinical validity.

Running and interpreting tests

uv sync --extra test
uv run pytest -rs
uv run pytest -m meds_dev -rs
uv run pytest -m real_data -rs

The default run does not authorize or provision clinical data or cluster work. meds_dev explicitly opts in to the isolated MEDS-DEV integration; real_data opts in to the MIMIC-IV demo. Follow the approval gates in the implementation guide before the demo or a full-data run. Marked integration fixtures can skip if not explicitly selected. Every algorithm_stub skip identifies unfinished implementation, not success. The primary-behavior placeholder intentionally fails, including in a freshly generated repository.

Acceptance inventory

File and test	What it checks	Why it exists
test_contract.py::test_profile_implementations_match_contract	Compares each registered profile's commands with the core lifecycle.	Detect incorrect profile registration or dispatch wiring.
test_contract.py::test_algorithm_configuration_roundtrips	Saves and reloads the concrete config.	Preserve exact settings across independent processes; extend with algorithm-specific invalid-value tests.
test_end_to_end.py::test_end_to_end	Runs the algorithm-owned adapter for every profile and shared lifecycle checks, including fresh outputs and label-blind reruns.	Verify persisted artifacts, prediction coverage and permitted label use through the whole pipeline.
test_primary_behavior.py::test_primary_behavior	Starts as a required failure; replace it with real positive-signal execution and a shuffled-label negative control, proving that the signal reaches actual algorithm features.	Require evidence of the algorithm's intended scientific behavior rather than only valid output files.
test_report.py::test_completed_algorithm_has_a_completed_report	Validates implementation-report evidence for completed profiles.	Keep completion claims tied to source-fidelity and validation evidence.
test_operations.py::test_algorithm_yaml_is_a_meds_dev_descriptor	Checks each profile descriptor's required structure.	Catch packaging errors before launching an integration run.
test_operations.py::test_slurm_dry_run_is_non_mutating	Runs each profile's launcher in dry-run mode and checks files and listed commands.	Make execution plans inspectable without submitting a job or modifying the repository.
test_meds_dev_e2e.py::test_meds_dev_runs_algorithm_yaml	Registers each completed profile, executes it through MEDS-DEV, and checks output schema, keys, manifest and memory evidence.	Prove the declared descriptor works in the isolated integration environment. Requires explicit meds_dev selection.
test_mimic_demo_e2e.py::test_full_dag_on_mimic_iv_demo	Executes each completed profile on the prepared MIMIC-IV demo task and checks prediction schema/keys, artifacts and recorded evidence.	Exercise the complete real-data boundary before a full dataset. Requires explicit real_data selection and the implementation-guide approval gate.

Tests with a profile parameter run for every declared profile. tests/e2e.py supplies algorithm-owned adapters, not additional pytest tests: it defines how to run the implementation and inspect fitted features. Implement these adapters instead of modifying shared checks to accommodate an error.

Algorithm-specific tests to add

Add source-backed tests for numerical behavior, temporal cutoffs, feature selection, frozen pretrained weights where applicable, payload compatibility, configuration effects and fresh-process reconstruction. Neural configuration tests should prove that capacity, learning rate, batch size, weight decay and dropout control the intended code. The generated config round-trip test alone does not prove that.

Keep a short what/why entry here for each added scientific test. Report exact pass/skip counts and explain remaining skips. Completing another profile must not mask missing evidence in an existing one.