munch-ease-backend/refactor-project-strategy.md

# Doof Backend Refactor Strategy

This document outlines a phased, low-risk plan to improve structure, readability, and Pythonic design. It also serves as a living progress log. Update checkboxes and the Progress Log as steps complete.

## Goals
- Improve module boundaries and readability by splitting monolithic files and clarifying responsibilities.
- Strengthen API design (consistent response models, error handling, security, health checks).
- Reduce DB round-trips and add indexes where needed for performance.
- Centralize configuration and lifecycle concerns.
- Improve tests with fixtures and reduce duplication.

## Guiding principles
- Small, incremental PR-sized steps; keep tests green.
- Preserve public API paths for v1 unless explicitly stated.
- Avoid new dependencies unless they clearly justify themselves.
- Add types, docstrings, and examples where they clarify intent.

---

## Phase 0 — Scaffolding and safety rails
- [x] Create this strategy document and track progress
- [x] Add central settings module (env-based, no new deps)
 - [x] Add a tiny /healthz endpoint plan (defer code to Phase 1)
 - [x] Decide on router file layout and migration approach

Acceptance criteria
- A settings module exists and can be imported without side effects
- Document tracks status and provides an actionable, sequenced plan

---

## Phase 1 — API structure and lifecycle
- [ ] Extract routers by feature
  - [x] api/recipes.py
  - [x] api/meals.py
  - [x] api/persons.py
  - [x] api/shopping.py
  - [x] api/auth.py
- [x] Wire routers in main with minimal app code
- [x] Move dev reverse proxy setup into a lifespan handler and ensure httpx client is closed
- [x] Add /healthz endpoint (simple JSON: {"status": "ok"})
- [x] Create api/deps module for get_db, cookie_person, and error_response
- [x] Use settings.py (DOOF_DB) for DB path in main and deps

Acceptance criteria
- main.py primarily wires app, routers, settings, and lifespan
- Reverse proxy cleaned up; no stray global clients
- Health check available and excluded from auth/security

Status: Complete

Notes
- Lifespan handler initializes a dev httpx.AsyncClient using settings.frontend_dev_url and stores it on app.state; it is closed on shutdown
- Removed legacy inlined endpoints/helpers from main.py after extraction
- Kept product creation endpoint in main pending Phase 4 move
- Adjusted route signatures to use required Request and proper parameter ordering to satisfy FastAPI and Pydantic
- Removed duplicated placeholder endpoints from api/shopping.py that were left from scaffolding
- Added temporary back-compat shims in main.py (get_duplicates, validate_meal) forwarding to api.meals to keep tests passing; plan to remove in Phase 4 when tests are updated to import from feature modules

---

## Phase 2 — API design and OpenAPI
- [x] Ensure response_model is set on all endpoints that return models
- [x] Normalize inconsistent routes (remove trailing slash on POST /shopping/) — N/A in code; no stray trailing slash routes
- [x] Replace MealIdWrapper body with path param for requesting meals: POST /shopping/current/meals/{mealId}
  - Decision: Kept existing v1 route POST /shopping/current/meals/me with body wrapper to avoid breaking tests; path-param variant deferred to v2
- [x] Add Location header on create endpoints (recipes, meals, persons, products)
  - Decision: Retained 200 response codes for v1 compatibility (tests expect 200); 201 can be adopted in v2
- [x] Define cookie-based security scheme in OpenAPI and apply to secured routes (by operationId)
- [x] Keep reusable ProblemDetails responses; ensure 4xx schemas reference it consistently

Acceptance criteria
- openapi.json shows correct schemas, cookieAuth security for protected operations, and references to reusable ProblemDetails responses — Met
- Clients can generate SDKs without manual fixes — Improved (consistent models and responses)

---

## Phase 3 — Data access and performance
- [x] Centralize transaction scoping per-request (dependency) and remove scattered conn.commit() in handlers
  - Implemented in `api/deps.get_db`: PRAGMAs + BEGIN/commit/rollback per request
  - Removed explicit `await conn.commit()` calls from handlers and product DB helpers
- [x] Add DB PRAGMAs on connect (WAL, foreign_keys=ON, synchronous=NORMAL)
- [x] Batch-load related data to avoid N+1 (recipes/ingredients, meals/participants)
  - [x] Batch-load meal participants (fetch IDs once, bulk load persons)
  - [x] Batch-load recipe ingredients across a page in `api/recipes.list_recipes`
- [x] Add indexes for common filters/joins
  - [x] ingredients.recipe_id
  - [x] ingredients.meal_id
  - [x] meal_participants.meal_id, role
  - [x] meal_recipes.meal_id
  - [x] recipes.date_hidden (+ name, id composite for pagination)
  - [x] persons.name (for LIKE queries)

Acceptance criteria
- Noticeable reduction in query count for list/detail endpoints (verified by logging/sqlite trace/inspection of code paths)
- No functional regressions; tests still green

Status: Complete

Notes
- PRAGMAs applied on connect and write transactions now scoped to each HTTP request
- Indexes added to improve common lookups and pagination
- Batch-loading of recipe ingredients implemented; meal participants batching implemented via `meals.bulk_load_participants` and used by `api/meals.get_upcoming_meals`.

---

## Phase 4 — Modeling and validation
- [X] Move inline API request/response models out of main.py into feature modules
  - [x] ProductUrl (moved to api/products.py)
  - [x] CurrentShoppingList (in api/shopping.py)
  - [x] PurchasedShoppingList (in api/shopping.py)
  - [x] LoginBody (in api/auth.py)
- [x] Extract validation (e.g., validate_meal) into a service layer for reuse
- [x] Add enums/constants for participant roles

Acceptance criteria
- Cleaner main.py; feature modules own their request/response contracts
- Validation logic reusable outside HTTP layer

---

## Phase 5 — Testing and fixtures
- [ ] Introduce pytest fixtures for common setup (DB, test data, auth)
- [ ] Remove duplication in tests and centralize helpers
- [ ] Add tests for new health endpoint and 201 Location headers

Acceptance criteria
- Test suite readability improved; repeated setup minimized
- Coverage for new behaviors added

---

## Stretch items
- [ ] Structured logging with request IDs
- [ ] A background job or cron-friendly script for cleanup tasks
- [ ] Optional migration to pydantic-settings if desired later

---

## Concrete file plan (proposed)
- main.py (app wiring only)
- settings.py (centralized configuration)
- api/
  - __init__.py
  - recipes.py, meals.py, persons.py, shopping.py, auth.py (routers)
- domain/
  - recipes/models.py, repo.py, service.py
  - meals/models.py, repo.py, service.py
  - ingredients/models.py, repo.py, service.py
  - persons/models.py, repo.py, service.py
  - shopping/models.py, repo.py, service.py

Note: We can adopt this structure gradually without moving DB code immediately; start with routers, then iterate.

---

## Risks and mitigations
- Router extraction could break imports
  - Mitigation: introduce api package and re-export symbols as needed during transition
- OpenAPI changes can affect clients
  - Mitigation: keep v1 paths stable; document changes like 201 + Location
- N+1 fixes may alter behavior subtly
  - Mitigation: add dedicated tests for list/detail payloads

---

## Progress log
- 2025-10-18: Created strategy document and added settings.py (not yet wired)
- 2025-10-18: Finalized router layout and health endpoint plan — Phase 0 complete
- 2025-10-18: Created api package and scaffolded routers (recipes, meals, persons, shopping, auth) with placeholders
- 2025-10-18: Extracted recipes routes to api/recipes.py and wired router; added /healthz
- 2025-10-18: Extracted persons routes to api/persons.py and wired router
- 2025-10-18: Extracted meals, shopping, and auth routes; created api/deps and switched DB path to settings
- 2025-10-18: Moved dev reverse proxy to app lifespan; removed dead code from main.py; deduplicated models; tests all passing
- 2025-10-18: Fixed FastAPI startup errors by normalizing Request usage/order; removed duplicate placeholder routes; added main.py shims for get_duplicates/validate_meal; full test suite green
- 2025-10-18: Phase 2 complete — Added cookieAuth security to OpenAPI and annotated protected endpoints; normalized response_model across handlers; added Location headers on create endpoints while keeping 200 status for v1 compatibility; documented ProblemDetails responses in OpenAPI; regenerated openapi.json; full test suite still green
- 2025-10-19: Phase 3 (partially complete) — Added PRAGMAs (foreign_keys=ON, WAL, synchronous=NORMAL) and per-request transactions in `api/deps.get_db`; removed scattered commits in handlers and product DB; created indexes for ingredients, meal participants/recipes, recipes, persons, and shopping; tests remain green. Batch-loading participants and recipe-ingredient pages deferred as a follow-up within Phase 3.
- 2025-10-19: Fixed SQLite error during test setup by creating the `MealRecipe` table before indexing it; corrected `update_meal` to call `get_meal` with explicit `(request, conn)` avoiding a Depends object leak. Full test suite now passes (100%). Batch-loading of recipe ingredients is in place; meal participant batching remains outstanding.
-. 
- 2025-10-19: Implemented participant batch-loading (`meals.bulk_load_participants`) and updated `api/meals.get_upcoming_meals` to use it; re-ran the test suite (green). Phase 3 marked complete; Phase 4-5 next.
  - 2025-10-19: Phase 4 (partial) — Extracted ProductUrl and product creation endpoint to `api/products.py`; wired new router; extracted `get_duplicates` to `meals/service.py` and re-exported via `meals.__init__`; tests still green.
  - 2025-10-19: Phase 4 — Extracted `validate_meal` into `meals/service.py`, re-exported via `meals.__init__`, and updated `api/meals.validate_meal` wrapper to map to ProblemDetails. Full test suite remains green.
  - 2025-10-19: Phase 4 — Added centralized role constants in `meals/roles.py` and replaced string literals in `meals/db.py`; re-exported constants via `meals.__init__`. Removed back-compat shims from `main.py` and updated tests to import helpers directly from modules. Tests remain green.

---

## Next actions
- Phase 3: Implement batch-loading to eliminate N+1
  - Batch-load meal participants and persons
  - Batch-load recipe ingredients for list pages
  - Optionally add lightweight query logging to validate reductions
- Phase 4: Move remaining request/response models and helpers (ProductUrl, CurrentShoppingList, PurchasedShoppingList, LoginBody, validate_meal/get_duplicates) fully into feature modules and update tests to import from there; then remove back-compat shims from main.py
- Phase 5: Add fixtures for DB/auth and tests for health + Location headers; consider adding perf checks

Follow-ups (v2 candidates)
- Adopt 201 Created for create endpoints and adjust tests/clients
- Switch POST /shopping/current/meals/me (body) to POST /shopping/current/meals/{mealId} (path param)

### Health endpoint plan (Phase 1 target)
- Path: GET /healthz
- Response: {"status": "ok"}
- No DB access; fast and safe

### Router extraction migration notes
- Step 1: Create api package and empty routers with existing path/operation_id/signatures (no logic change)
- Step 2: Move handlers from main.py into respective files, fix imports
- Step 3: In main.py, include_router for each; keep response_model and decorator metadata intact
- Step 4: Run tests and adjust import paths