munch-ease-backend/refactor-project-strategy.md

Doof Backend Refactor Strategy

This document outlines a phased, low-risk plan to improve structure, readability, and Pythonic design. It also serves as a living progress log. Update checkboxes and the Progress Log as steps complete.

Goals

• Improve module boundaries and readability by splitting monolithic files and clarifying responsibilities.
• Strengthen API design (consistent response models, error handling, security, health checks).
• Reduce DB round-trips and add indexes where needed for performance.
• Centralize configuration and lifecycle concerns.
• Improve tests with fixtures and reduce duplication.

Guiding principles

• Small, incremental PR-sized steps; keep tests green.
• Preserve public API paths for v1 unless explicitly stated.
• Avoid new dependencies unless they clearly justify themselves.
• Add types, docstrings, and examples where they clarify intent.

---

Phase 0 — Scaffolding and safety rails

• Create this strategy document and track progress
• Add central settings module (env-based, no new deps)
• Add a tiny /healthz endpoint plan (defer code to Phase 1)
• Decide on router file layout and migration approach

Acceptance criteria

• A settings module exists and can be imported without side effects
• Document tracks status and provides an actionable, sequenced plan

---

Phase 1 — API structure and lifecycle

• Extract routers by feature
  • api/recipes.py
  • api/meals.py
  • api/persons.py
  • api/shopping.py
  • api/auth.py
• Wire routers in main with minimal app code
• Move dev reverse proxy setup into a lifespan handler and ensure httpx client is closed
• Add /healthz endpoint (simple JSON: {"status": "ok"})
• Create api/deps module for get_db, cookie_person, and error_response
• Use settings.py (DOOF_DB) for DB path in main and deps

Acceptance criteria

• main.py primarily wires app, routers, settings, and lifespan
• Reverse proxy cleaned up; no stray global clients
• Health check available and excluded from auth/security

Status: Complete

Notes

• Lifespan handler initializes a dev httpx.AsyncClient using settings.frontend_dev_url and stores it on app.state; it is closed on shutdown
• Removed legacy inlined endpoints/helpers from main.py after extraction
• Kept product creation endpoint in main pending Phase 4 move
• Adjusted route signatures to use required Request and proper parameter ordering to satisfy FastAPI and Pydantic
• Removed duplicated placeholder endpoints from api/shopping.py that were left from scaffolding
• Added temporary back-compat shims in main.py (get_duplicates, validate_meal) forwarding to api.meals to keep tests passing; plan to remove in Phase 4 when tests are updated to import from feature modules

---

Phase 2 — API design and OpenAPI

• Ensure response_model is set on all endpoints that return models
• Normalize inconsistent routes (remove trailing slash on POST /shopping/) — N/A in code; no stray trailing slash routes
• Replace MealIdWrapper body with path param for requesting meals: POST /shopping/current/meals/{mealId}
  • Decision: Kept existing v1 route POST /shopping/current/meals/me with body wrapper to avoid breaking tests; path-param variant deferred to v2
• Add Location header on create endpoints (recipes, meals, persons, products)
  • Decision: Retained 200 response codes for v1 compatibility (tests expect 200); 201 can be adopted in v2
• Define cookie-based security scheme in OpenAPI and apply to secured routes (by operationId)
• Keep reusable ProblemDetails responses; ensure 4xx schemas reference it consistently

Acceptance criteria

• openapi.json shows correct schemas, cookieAuth security for protected operations, and references to reusable ProblemDetails responses — Met
• Clients can generate SDKs without manual fixes — Improved (consistent models and responses)

---

Phase 3 — Data access and performance

• Centralize transaction scoping per-request (dependency) and remove scattered conn.commit() in handlers
  • Implemented in api/deps.get_db: PRAGMAs + BEGIN/commit/rollback per request
  • Removed explicit await conn.commit() calls from handlers and product DB helpers
• Add DB PRAGMAs on connect (WAL, foreign_keys=ON, synchronous=NORMAL)
• Batch-load related data to avoid N+1 (recipes/ingredients, meals/participants)
  • Batch-load meal participants (fetch IDs once, bulk load persons)
  • Batch-load recipe ingredients across a page in api/recipes.list_recipes
• Add indexes for common filters/joins
  • ingredients.recipe_id
  • ingredients.meal_id
  • meal_participants.meal_id, role
  • meal_recipes.meal_id
  • recipes.date_hidden (+ name, id composite for pagination)
  • persons.name (for LIKE queries)

Acceptance criteria

• Noticeable reduction in query count for list/detail endpoints (verified by logging/sqlite trace/inspection of code paths)
• No functional regressions; tests still green

Status: Complete

Notes

• PRAGMAs applied on connect and write transactions now scoped to each HTTP request
• Indexes added to improve common lookups and pagination
• Batch-loading of recipe ingredients implemented; meal participants batching implemented via meals.bulk_load_participants and used by api/meals.get_upcoming_meals.

---

Phase 4 — Modeling and validation

• Move inline API request/response models out of main.py into feature modules
  • ProductUrl (moved to api/products.py)
  • CurrentShoppingList (in api/shopping.py)
  • PurchasedShoppingList (in api/shopping.py)
  • LoginBody (in api/auth.py)
• Extract validation (e.g., validate_meal) into a service layer for reuse
• Add enums/constants for participant roles

Acceptance criteria

• Cleaner main.py; feature modules own their request/response contracts
• Validation logic reusable outside HTTP layer

---

Testing and fixtures

• Introduce pytest fixtures for common setup (DB, test data, auth)
• Remove duplication in tests and centralize helpers
• Add tests for new health endpoint and 201 Location headers

Acceptance criteria

• Test suite readability improved; repeated setup minimized
• Coverage for new behaviors added

---

Stretch items

• Structured logging with request IDs
• A background job or cron-friendly script for cleanup tasks
• Optional migration to pydantic-settings if desired later

---

Concrete file plan (proposed)

• main.py (app wiring only)
• settings.py (centralized configuration)
• api/
  • init.py
  • recipes.py, meals.py, persons.py, shopping.py, auth.py (routers)
• domain/
  • recipes/models.py, repo.py, service.py
  • meals/models.py, repo.py, service.py
  • ingredients/models.py, repo.py, service.py
  • persons/models.py, repo.py, service.py
  • shopping/models.py, repo.py, service.py

Note: We can adopt this structure gradually without moving DB code immediately; start with routers, then iterate.

---

Risks and mitigations

• Router extraction could break imports
  • Mitigation: introduce api package and re-export symbols as needed during transition
• OpenAPI changes can affect clients
  • Mitigation: keep v1 paths stable; document changes like 201 + Location
• N+1 fixes may alter behavior subtly
  • Mitigation: add dedicated tests for list/detail payloads

---

Progress log

• 2025-10-18: Created strategy document and added settings.py (not yet wired)
• 2025-10-18: Finalized router layout and health endpoint plan — Phase 0 complete
• 2025-10-18: Created api package and scaffolded routers (recipes, meals, persons, shopping, auth) with placeholders
• 2025-10-18: Extracted recipes routes to api/recipes.py and wired router; added /healthz
• 2025-10-18: Extracted persons routes to api/persons.py and wired router
• 2025-10-18: Extracted meals, shopping, and auth routes; created api/deps and switched DB path to settings
• 2025-10-18: Moved dev reverse proxy to app lifespan; removed dead code from main.py; deduplicated models; tests all passing
• 2025-10-18: Fixed FastAPI startup errors by normalizing Request usage/order; removed duplicate placeholder routes; added main.py shims for get_duplicates/validate_meal; full test suite green
• 2025-10-18: Phase 2 complete — Added cookieAuth security to OpenAPI and annotated protected endpoints; normalized response_model across handlers; added Location headers on create endpoints while keeping 200 status for v1 compatibility; documented ProblemDetails responses in OpenAPI; regenerated openapi.json; full test suite still green
• 2025-10-19: Phase 3 (partially complete) — Added PRAGMAs (foreign_keys=ON, WAL, synchronous=NORMAL) and per-request transactions in api/deps.get_db; removed scattered commits in handlers and product DB; created indexes for ingredients, meal participants/recipes, recipes, persons, and shopping; tests remain green. Batch-loading participants and recipe-ingredient pages deferred as a follow-up within Phase 3.
• 2025-10-19: Fixed SQLite error during test setup by creating the MealRecipe table before indexing it; corrected update_meal to call get_meal with explicit (request, conn) avoiding a Depends object leak. Full test suite now passes (100%). Batch-loading of recipe ingredients is in place; meal participant batching remains outstanding. -.
• 2025-10-19: Implemented participant batch-loading (meals.bulk_load_participants) and updated api/meals.get_upcoming_meals to use it; re-ran the test suite (green). Phase 3 marked complete; Phase 4-5 next.
  • 2025-10-19: Phase 4 (partial) — Extracted ProductUrl and product creation endpoint to api/products.py; wired new router; extracted get_duplicates to meals/service.py and re-exported via meals.__init__; tests still green.
  • 2025-10-19: Phase 4 — Extracted validate_meal into meals/service.py, re-exported via meals.__init__, and updated api/meals.validate_meal wrapper to map to ProblemDetails. Full test suite remains green.
  • 2025-10-19: Phase 4 — Added centralized role constants in meals/roles.py and replaced string literals in meals/db.py; re-exported constants via meals.__init__. Removed back-compat shims from main.py and updated tests to import helpers directly from modules. Tests remain green.

---

Next actions

• Phase 3: Implement batch-loading to eliminate N+1
  • Batch-load meal participants and persons
  • Batch-load recipe ingredients for list pages
  • Optionally add lightweight query logging to validate reductions
• Phase 4: Move remaining request/response models and helpers (ProductUrl, CurrentShoppingList, PurchasedShoppingList, LoginBody, validate_meal/get_duplicates) fully into feature modules and update tests to import from there; then remove back-compat shims from main.py
• Testing and fixtures: Add fixtures for DB/auth and tests for health + Location headers; consider adding perf checks

Follow-ups (v2 candidates)

• Adopt 201 Created for create endpoints and adjust tests/clients
• Switch POST /shopping/current/meals/me (body) to POST /shopping/current/meals/{mealId} (path param)

Health endpoint plan (Phase 1 target)

• Path: GET /healthz
• Response: {"status": "ok"}
• No DB access; fast and safe

Router extraction migration notes

• Step 1: Create api package and empty routers with existing path/operation_id/signatures (no logic change)
• Step 2: Move handlers from main.py into respective files, fix imports
• Step 3: In main.py, include_router for each; keep response_model and decorator metadata intact
• Step 4: Run tests and adjust import paths