Application Evolution Notes

This page records small, concrete changes in LabFrog behavior as the app evolves. The goal is to make future maintenance easier by showing what changed, why it changed, and which workflow it protects.

2026-07-21: Form Runtime Binding Extraction

The remaining practical forms.py thinning is complete without splitting the large FlaskForm class definitions:

  • Campaign-specific option and field-alias binding now lives in labfrog/form_factory.py, alongside the other runtime-bound form utilities.

  • Cached diagnostics, custom-field, default-layout, and selected parameter-option queries now live in labfrog/helpers/form_lookups.py.

  • labfrog.forms.apply_parameter_options and labfrog.forms.apply_field_aliases remain import-compatible re-exports.

  • InputForm, DynamicForm, and SearchForm remain in labfrog/forms.py; further class-body splitting would add indirection without isolating a clear responsibility.

This is a non-behavioral ownership change intended to make future form work land in the module that owns either runtime binding or data lookup.

The former doc/code_structure_roadmap.md is now retired: it has no remaining scheduled work, and its completed changes are recorded on this page. Future structural work follows these maintenance rules:

  • Verify refactors with focused tests and the repository gate appropriate to their risk.

  • Keep compatibility re-exports only where callers still need them; new callers should import from the canonical module.

  • Patch and monkeypatch the module that owns the implementation, not a re-export shim.

  • Add no new logic to helper_functions.py; it exists for compatibility.

  • Keep one concern per helper module, but do not force an extraction that creates circular imports or empty indirection.

  • Keep structural changes local-first, without external calls or secrets.

2026-07-20: Read the Docs Navigation Revamp

The documentation was reorganized into nine concise reader paths: Home, Getting Started, Using LabFrog, Configuring Forms, LabFrog to DAMNIT, Deployment and Administration, Data and Integrations, Development, and Project Information. Redundant public pages became short migration stubs so old URLs still resolve; the home page and task guides now own routing instead of separate overview, quickstart, site-map, navigation, core-concept, and current-build narratives. Detailed architecture, runtime-performance, benchmark, compatibility, and evolution material remains available as secondary maintainer reference rather than crowding primary navigation.

The synchronized DAMNIT overview remains first-class documentation. DAMNIT owns the canonical source, uv run poe sync-damnit-docs refreshes LabFrog’s rendered copy, and uv run poe sync-damnit-docs-check is the freshness gate. The application’s /damnit-workflow page remains its visual companion. The existing authenticated Read the Docs dependency flow and generated benchmark/screenshot assets were deliberately retained.

The completed revamp enabled warning-as-error Sphinx builds, Mermaid rendering, focused synchronization tests, corrected version/citation/command metadata, and strict documentation validation. Its completion pass reported 519 passed, 1 skipped, and 9 deselected tests, plus successful lint, formatting, license, coverage-docs, benchmark-JSON, strict Sphinx, link-check, and documentation-build gates. Live browser review and the sibling-checkout freshness command were not available in that sandbox; generated output and synchronization behavior were checked locally instead.

2026-07-03: MongoDB Maintenance and Layout Audits

Database-maintenance entrypoints were consolidated so operational scripts are easier to find and safer to run:

  • scripts/db/README.md is now the canonical index for maintenance scripts that write to MongoDB.

  • scripts/db/audit_layout_fields.py audits saved layout references for stale fields, dangling references, and missing layout names. Repair flags dry-run by default, and --write is required before changes are persisted.

  • scripts/backup_mongodb_data/ now documents backup, restore, collection comparison, and orphan diagnostic cleanup workflows.

This protects old layouts and production-like databases by making cleanup steps explicit before any write happens.

2026-07-03: Entry Form and Search UI Polish

The entry and search views were tightened without changing the stored record model:

  • Add/Edit Entry toolbar controls were compacted for the two-column layout.

  • Shared base and shot-form styles were adjusted to keep toolbar/status content from overlapping and to make sections visually calmer.

  • Search Records rendering received smaller usability fixes while preserving the existing query and projection behavior.

The intent is to make live operator flows easier to scan while keeping the server-side data model stable.

2026-07-02: Wiki Target, Beamtime, and DAMNIT Metadata Bridge

LabFrog now persists more export-friendly identity metadata from selectable campaign and target sources:

  • MediaWiki-backed target selections can carry fields such as target.wiki_page, target.wiki_ref, target.type, target.status, and curated material/thickness/provider details.

  • Campaign selections persist experiment_id plus beamtime identity fields such as beamtime_id, beamtime_name, beamtime_wiki_page, and beamtime_wiki_ref.

  • The sibling SQLite export and DAMNIT reconciler use those captured fields to populate HZDRShot.metadata.target.* and related provenance.

This keeps LabFrog records useful for downstream matching without requiring operators to type DAMNIT-specific metadata by hand.

2026-07-02: Live Shot-Day Reliability and Day-Scope Navigation

The live Add/Edit workflow was hardened for interrupted operator sessions:

  • Validation failures and demo-limit saves re-render the submitted form values instead of dropping operator input.

  • Browser-side draft storage restores unsaved blank fields after navigation away and back, while leaving server-prefilled values authoritative.

  • Shot-day/session filtering is centralized in labfrog/helpers/navigation.py:docs_for_day_scope() so Add/Edit navigation, goto filters, and Search Records use the same day-scope interpretation.

  • Explicit shot-day values take precedence; otherwise an eight-hour gap rule prevents short pauses, including midnight-spanning pauses, from creating a new session. The SQLite exporter mirrors this rule per campaign.

The protected workflow is real shot-day data entry: a blocked save or brief navigation detour should not silently lose work.

2026-06-24: Testing, Coverage, and Benchmark Docs Automation

Documentation freshness checks became part of the normal local workflow:

  • uv run poe coverage-docs refreshes the generated test coverage map from the current coverage JSON.

  • uv run poe coverage-docs-check validates that the committed coverage map is up to date.

  • Benchmark documentation is refreshed from stored benchmark JSONs instead of hand-edited summaries.

  • uv run poe prepush-fast now includes the lightweight docs synchronization checks; uv run poe prepush-full adds the Sphinx HTML build and browser smoke tests.

This keeps status tables and benchmark summaries tied to generated artifacts instead of stale prose.

2026-06-24: HZDR Pilot and DAMNIT Workflow Hints

LabFrog gained lightweight operator/developer hints for the HZDR pilot path:

  • Kafka event wording in the developer guidance now points at the shared hzdr-event-v1 envelope and the experiment_id + shot_id join key.

  • DAMNIT workflow screenshots and a local workflow page are vendored through the docs sync helper.

  • Deployment notes distinguish production shotsheet, demo server, and follow-up service-management work.

This documentation is intentionally descriptive. It does not claim live broker validation unless a later verification pass records it.

2026-06-19: Code Structure Improvements

This change set is non-behavioral β€” no user-visible features changed β€” but it significantly reduces the navigation surface for future development.

helper_functions.py decomposition: labfrog/helper_functions.py was a 2,676-line catch-all imported by nearly every route module. Its pure-logic functions were extracted into focused submodules under labfrog/helpers/:

New submodule

Contents

labfrog/helpers/query.py

Query parsing, sanitization, serialization, datetime range normalization

labfrog/helpers/target.py

Target field normalization and JSON-safe serialization

labfrog/helpers/demo.py

Demo-mode filtering, document counting, limit enforcement, default application

labfrog/helpers/aliases.py

Field alias lookup with request-level and app-level TTL caching

labfrog/helpers/layout.py

Layout selection, field identity, campaign header lookup, CRUD helpers, available-campaign enumeration (~1,460 lines)

labfrog/helpers/form_lookups.py

Session/header helpers extracted from forms.py: _get_header_choice, _set_header_choice, _header_session_key, _find_selected_header_doc, _request_cache_bucket

helper_functions.py now re-exports all extracted names for backward compatibility and sits at ~810 lines (mostly re-exports plus retrieve_data, create_new_version, and field comparison helpers).

Legacy code removal:

  • labfrog/startup_state.py deleted (compat shim for request_state.py; no callers)

  • labfrog/legacy/ directory deleted (9 tab-shim files, all callers already gone)

  • Removed tab modules map to the feature modules that replaced them: add_entry, edit_entry, diagnostic_options, diagnostic_definitions, field_selection, search_records, organize_sections, and parameter_options.

CSS extraction:

  • ~510 lines of inline <style> removed from base_page.html and _flash_styles.html

  • All base page and flash notification styles moved to labfrog/static/labfrog-base.css

Template cleanup:

  • _datatables_assets.html and _datatables_management_styles.html deleted (one-line wrappers around their _shared_* counterparts; no direct includers)

  • _auth_card_styles.html deleted (stale CSS draft, never included anywhere)

All 456 tests pass before and after every commit in this set.

The completed structure context and current maintenance rules are consolidated in the 2026-07-21 entry above.

2026-06-19: Template Macro Extraction

Non-behavioral. Three repeated HTML blocks shared between add_entry.html and edit_entry.html were extracted into labfrog/templates/_entry_form_macros.html:

  • toolbar_left(form, campaign) β€” Mode selector and Campaign selector with optional Add Campaign button. Identical in both templates.

  • goto_navigate_panel(mode, campaign_days, shot_day_label) β€” Navigate/Go panel (day-select + shot/set number input + Go button, with set-mode variant). Identical in both templates.

  • diagnostics_card(collapse_id, toggle_id, dynamic_field_names, form, diff_fields=none) β€” The collapsible Diagnostics card. Parameterized on collapse/toggle IDs and optional diff_fields (used only by edit_entry to highlight changed values).

Both templates now use {% import '_entry_form_macros.html' as entry_macros %} and call the macros. Future divergence between Add Entry and Edit Entry is now immediately visible as a diff in the macro parameters rather than buried in duplicate blocks.

All 456 tests pass.

2026-06-19: Route Import Migration and Template Partial Documentation

Follow-up to the code structure improvements above. Non-behavioral.

Route import migration (roadmap item B): All 19 route and utility modules that previously imported from labfrog.helper_functions have been updated to import directly from the canonical labfrog/helpers/ submodule. The re-export shim in helper_functions.py remains for the small number of functions not yet extracted (retrieve_data, create_new_version, compare_field_diffs, update_names_collection, build_campaign_day_summary). This completes the caller migration β€” new logic should go into helpers/ and callers should import it there directly.

Management partial documentation (roadmap item D): Each of the five _shared_management_*.html partials now has a one-line comment listing which management pages include it (action_bar, callout, card_styles, note, panel_header).

All 456 tests pass after these changes.

2026-04-28: Calculations, Target Details, and Search Plot Display

  • Adjust Parameters now supports calculated and calculated_prev_shot field types. Current-value formulas use x; previous-shot formulas use x and prev_x. Previous-shot calculations are limited to shot mode.

  • Calculated fields can optionally round results to a configured number of decimal places from 0 through 12.

  • Target OTHER values now store material, thickness, and notes as nested target details while Add/Edit Entry still loads them into normal form fields.

  • Search Records flattens target objects before table and plot rendering. Tables keep OTHER details on one line; plots split details into connected lines for readability.

  • Shot day and shot group text inputs remain hidden until their tag checkbox is enabled, reducing disabled-field clutter without changing form semantics.

  • Search Records keeps boolean plot fields as booleans, keeps comments as text, and caches the latest shareable search URL for return navigation.

2026-04-25: Form Layouts and Parameter Options

  • Data-entry navigation now preserves saved select values even when a value is no longer active in the current option set. This keeps older entries editable and makes target cards repopulate after Go, prev, and next navigation.

  • Parameter Options now has a Deactivate all action for fields such as Target, Campaign, and custom select parameters. Option rows are built with DOM nodes so option text cannot accidentally become more checkboxes or other controls.

  • Organize Form saves are mirrored back into the linked Select Fields layout. Renamed sections, active fields, inactive fields, and matching custom-field section assignments stay aligned across the form builder and parameter setup.

Baseline Workflow

The current layout workflow has two linked documents:

  • Select Fields stores the field layout: which parameters and diagnostics are active for a campaign or reusable layout.

  • Organize Form stores the presentation layout: section names, order, color, and inactive staging areas for the active field layout.

When the organize form changes section names or moves fields inactive, the linked field layout should change at the same time. That keeps later edits in Select Fields, Parameter Options, and Add/Edit Entry aligned with the form users see.