From 50a1fcaeeee2099d61d7223f4e4bb024e967810e Mon Sep 17 00:00:00 2001 From: jableader Date: Sun, 19 Oct 2025 15:20:36 +1100 Subject: [PATCH] Openapi refactor complete --- refactor-project-strategy.md | 208 ----------------------------------- 1 file changed, 208 deletions(-) delete mode 100644 refactor-project-strategy.md diff --git a/refactor-project-strategy.md b/refactor-project-strategy.md deleted file mode 100644 index 81dcdb3..0000000 --- a/refactor-project-strategy.md +++ /dev/null @@ -1,208 +0,0 @@ -# Doof Backend Refactor Strategy - -This document outlines a phased, low-risk plan to improve structure, readability, and Pythonic design. It also serves as a living progress log. Update checkboxes and the Progress Log as steps complete. - -## Goals -- Improve module boundaries and readability by splitting monolithic files and clarifying responsibilities. -- Strengthen API design (consistent response models, error handling, security, health checks). -- Reduce DB round-trips and add indexes where needed for performance. -- Centralize configuration and lifecycle concerns. -- Improve tests with fixtures and reduce duplication. - -## Guiding principles -- Small, incremental PR-sized steps; keep tests green. -- Preserve public API paths for v1 unless explicitly stated. -- Avoid new dependencies unless they clearly justify themselves. -- Add types, docstrings, and examples where they clarify intent. - ---- - -## Phase 0 — Scaffolding and safety rails -- [x] Create this strategy document and track progress -- [x] Add central settings module (env-based, no new deps) - - [x] Add a tiny /healthz endpoint plan (defer code to Phase 1) - - [x] Decide on router file layout and migration approach - -Acceptance criteria -- A settings module exists and can be imported without side effects -- Document tracks status and provides an actionable, sequenced plan - ---- - -## Phase 1 — API structure and lifecycle -- [ ] Extract routers by feature - - [x] api/recipes.py - - [x] api/meals.py - - [x] api/persons.py - - [x] api/shopping.py - - [x] api/auth.py -- [x] Wire routers in main with minimal app code -- [x] Move dev reverse proxy setup into a lifespan handler and ensure httpx client is closed -- [x] Add /healthz endpoint (simple JSON: {"status": "ok"}) -- [x] Create api/deps module for get_db, cookie_person, and error_response -- [x] Use settings.py (DOOF_DB) for DB path in main and deps - -Acceptance criteria -- main.py primarily wires app, routers, settings, and lifespan -- Reverse proxy cleaned up; no stray global clients -- Health check available and excluded from auth/security - -Status: Complete - -Notes -- Lifespan handler initializes a dev httpx.AsyncClient using settings.frontend_dev_url and stores it on app.state; it is closed on shutdown -- Removed legacy inlined endpoints/helpers from main.py after extraction -- Kept product creation endpoint in main pending Phase 4 move -- Adjusted route signatures to use required Request and proper parameter ordering to satisfy FastAPI and Pydantic -- Removed duplicated placeholder endpoints from api/shopping.py that were left from scaffolding -- Added temporary back-compat shims in main.py (get_duplicates, validate_meal) forwarding to api.meals to keep tests passing; plan to remove in Phase 4 when tests are updated to import from feature modules - ---- - -## Phase 2 — API design and OpenAPI -- [x] Ensure response_model is set on all endpoints that return models -- [x] Normalize inconsistent routes (remove trailing slash on POST /shopping/) — N/A in code; no stray trailing slash routes -- [x] Replace MealIdWrapper body with path param for requesting meals: POST /shopping/current/meals/{mealId} - - Decision: Kept existing v1 route POST /shopping/current/meals/me with body wrapper to avoid breaking tests; path-param variant deferred to v2 -- [x] Add Location header on create endpoints (recipes, meals, persons, products) - - Decision: Retained 200 response codes for v1 compatibility (tests expect 200); 201 can be adopted in v2 -- [x] Define cookie-based security scheme in OpenAPI and apply to secured routes (by operationId) -- [x] Keep reusable ProblemDetails responses; ensure 4xx schemas reference it consistently - -Acceptance criteria -- openapi.json shows correct schemas, cookieAuth security for protected operations, and references to reusable ProblemDetails responses — Met -- Clients can generate SDKs without manual fixes — Improved (consistent models and responses) - ---- - -## Phase 3 — Data access and performance -- [x] Centralize transaction scoping per-request (dependency) and remove scattered conn.commit() in handlers - - Implemented in `api/deps.get_db`: PRAGMAs + BEGIN/commit/rollback per request - - Removed explicit `await conn.commit()` calls from handlers and product DB helpers -- [x] Add DB PRAGMAs on connect (WAL, foreign_keys=ON, synchronous=NORMAL) -- [x] Batch-load related data to avoid N+1 (recipes/ingredients, meals/participants) - - [x] Batch-load meal participants (fetch IDs once, bulk load persons) - - [x] Batch-load recipe ingredients across a page in `api/recipes.list_recipes` -- [x] Add indexes for common filters/joins - - [x] ingredients.recipe_id - - [x] ingredients.meal_id - - [x] meal_participants.meal_id, role - - [x] meal_recipes.meal_id - - [x] recipes.date_hidden (+ name, id composite for pagination) - - [x] persons.name (for LIKE queries) - -Acceptance criteria -- Noticeable reduction in query count for list/detail endpoints (verified by logging/sqlite trace/inspection of code paths) -- No functional regressions; tests still green - -Status: Complete - -Notes -- PRAGMAs applied on connect and write transactions now scoped to each HTTP request -- Indexes added to improve common lookups and pagination -- Batch-loading of recipe ingredients implemented; meal participants batching implemented via `meals.bulk_load_participants` and used by `api/meals.get_upcoming_meals`. - ---- - -## Phase 4 — Modeling and validation -- [X] Move inline API request/response models out of main.py into feature modules - - [x] ProductUrl (moved to api/products.py) - - [x] CurrentShoppingList (in api/shopping.py) - - [x] PurchasedShoppingList (in api/shopping.py) - - [x] LoginBody (in api/auth.py) -- [x] Extract validation (e.g., validate_meal) into a service layer for reuse -- [x] Add enums/constants for participant roles - -Acceptance criteria -- Cleaner main.py; feature modules own their request/response contracts -- Validation logic reusable outside HTTP layer - ---- - -## Testing and fixtures -- [ ] Introduce pytest fixtures for common setup (DB, test data, auth) -- [ ] Remove duplication in tests and centralize helpers -- [x] Add tests for new health endpoint and 201 Location headers - -Acceptance criteria -- Test suite readability improved; repeated setup minimized -- Coverage for new behaviors added - ---- - -## Stretch items -- [ ] Structured logging with request IDs -- [ ] A background job or cron-friendly script for cleanup tasks -- [ ] Optional migration to pydantic-settings if desired later - ---- - -## Concrete file plan (proposed) -- main.py (app wiring only) -- settings.py (centralized configuration) -- api/ - - __init__.py - - recipes.py, meals.py, persons.py, shopping.py, auth.py (routers) -- domain/ - - recipes/models.py, repo.py, service.py - - meals/models.py, repo.py, service.py - - ingredients/models.py, repo.py, service.py - - persons/models.py, repo.py, service.py - - shopping/models.py, repo.py, service.py - -Note: We can adopt this structure gradually without moving DB code immediately; start with routers, then iterate. - ---- - -## Risks and mitigations -- Router extraction could break imports - - Mitigation: introduce api package and re-export symbols as needed during transition -- OpenAPI changes can affect clients - - Mitigation: keep v1 paths stable; document changes like 201 + Location -- N+1 fixes may alter behavior subtly - - Mitigation: add dedicated tests for list/detail payloads - ---- - -## Progress log -- 2025-10-18: Created strategy document and added settings.py (not yet wired) -- 2025-10-18: Finalized router layout and health endpoint plan — Phase 0 complete -- 2025-10-18: Created api package and scaffolded routers (recipes, meals, persons, shopping, auth) with placeholders -- 2025-10-18: Extracted recipes routes to api/recipes.py and wired router; added /healthz -- 2025-10-18: Extracted persons routes to api/persons.py and wired router -- 2025-10-18: Extracted meals, shopping, and auth routes; created api/deps and switched DB path to settings -- 2025-10-18: Moved dev reverse proxy to app lifespan; removed dead code from main.py; deduplicated models; tests all passing -- 2025-10-18: Fixed FastAPI startup errors by normalizing Request usage/order; removed duplicate placeholder routes; added main.py shims for get_duplicates/validate_meal; full test suite green -- 2025-10-18: Phase 2 complete — Added cookieAuth security to OpenAPI and annotated protected endpoints; normalized response_model across handlers; added Location headers on create endpoints while keeping 200 status for v1 compatibility; documented ProblemDetails responses in OpenAPI; regenerated openapi.json; full test suite still green -- 2025-10-19: Phase 3 (partially complete) — Added PRAGMAs (foreign_keys=ON, WAL, synchronous=NORMAL) and per-request transactions in `api/deps.get_db`; removed scattered commits in handlers and product DB; created indexes for ingredients, meal participants/recipes, recipes, persons, and shopping; tests remain green. Batch-loading participants and recipe-ingredient pages deferred as a follow-up within Phase 3. -- 2025-10-19: Fixed SQLite error during test setup by creating the `MealRecipe` table before indexing it; corrected `update_meal` to call `get_meal` with explicit `(request, conn)` avoiding a Depends object leak. Full test suite now passes (100%). Batch-loading of recipe ingredients is in place; meal participant batching remains outstanding. --. -- 2025-10-19: Implemented participant batch-loading (`meals.bulk_load_participants`) and updated `api/meals.get_upcoming_meals` to use it; re-ran the test suite (green). Phase 3 marked complete; Phase 4-5 next. - - 2025-10-19: Phase 4 (partial) — Extracted ProductUrl and product creation endpoint to `api/products.py`; wired new router; extracted `get_duplicates` to `meals/service.py` and re-exported via `meals.__init__`; tests still green. - - 2025-10-19: Phase 4 — Extracted `validate_meal` into `meals/service.py`, re-exported via `meals.__init__`, and updated `api/meals.validate_meal` wrapper to map to ProblemDetails. Full test suite remains green. - - 2025-10-19: Phase 4 — Added centralized role constants in `meals/roles.py` and replaced string literals in `meals/db.py`; re-exported constants via `meals.__init__`. Removed back-compat shims from `main.py` and updated tests to import helpers directly from modules. Tests remain green. - ---- - -## Next actions -- Phase 3: Implement batch-loading to eliminate N+1 - - Batch-load meal participants and persons - - Batch-load recipe ingredients for list pages - - Optionally add lightweight query logging to validate reductions -- Phase 4: Move remaining request/response models and helpers (ProductUrl, CurrentShoppingList, PurchasedShoppingList, LoginBody, validate_meal/get_duplicates) fully into feature modules and update tests to import from there; then remove back-compat shims from main.py -- Testing and fixtures: Add fixtures for DB/auth and tests for health + Location headers; consider adding perf checks - -Follow-ups (v2 candidates) -- Adopt 201 Created for create endpoints and adjust tests/clients -- Switch POST /shopping/current/meals/me (body) to POST /shopping/current/meals/{mealId} (path param) - -### Health endpoint plan (Phase 1 target) -- Path: GET /healthz -- Response: {"status": "ok"} -- No DB access; fast and safe - -### Router extraction migration notes -- Step 1: Create api package and empty routers with existing path/operation_id/signatures (no logic change) -- Step 2: Move handlers from main.py into respective files, fix imports -- Step 3: In main.py, include_router for each; keep response_model and decorator metadata intact -- Step 4: Run tests and adjust import paths \ No newline at end of file