205 lines
No EOL
10 KiB
Markdown
205 lines
No EOL
10 KiB
Markdown
# Doof Backend Refactor Strategy
|
|
|
|
This document outlines a phased, low-risk plan to improve structure, readability, and Pythonic design. It also serves as a living progress log. Update checkboxes and the Progress Log as steps complete.
|
|
|
|
## Goals
|
|
- Improve module boundaries and readability by splitting monolithic files and clarifying responsibilities.
|
|
- Strengthen API design (consistent response models, error handling, security, health checks).
|
|
- Reduce DB round-trips and add indexes where needed for performance.
|
|
- Centralize configuration and lifecycle concerns.
|
|
- Improve tests with fixtures and reduce duplication.
|
|
|
|
## Guiding principles
|
|
- Small, incremental PR-sized steps; keep tests green.
|
|
- Preserve public API paths for v1 unless explicitly stated.
|
|
- Avoid new dependencies unless they clearly justify themselves.
|
|
- Add types, docstrings, and examples where they clarify intent.
|
|
|
|
---
|
|
|
|
## Phase 0 — Scaffolding and safety rails
|
|
- [x] Create this strategy document and track progress
|
|
- [x] Add central settings module (env-based, no new deps)
|
|
- [x] Add a tiny /healthz endpoint plan (defer code to Phase 1)
|
|
- [x] Decide on router file layout and migration approach
|
|
|
|
Acceptance criteria
|
|
- A settings module exists and can be imported without side effects
|
|
- Document tracks status and provides an actionable, sequenced plan
|
|
|
|
---
|
|
|
|
## Phase 1 — API structure and lifecycle
|
|
- [ ] Extract routers by feature
|
|
- [x] api/recipes.py
|
|
- [x] api/meals.py
|
|
- [x] api/persons.py
|
|
- [x] api/shopping.py
|
|
- [x] api/auth.py
|
|
- [x] Wire routers in main with minimal app code
|
|
- [x] Move dev reverse proxy setup into a lifespan handler and ensure httpx client is closed
|
|
- [x] Add /healthz endpoint (simple JSON: {"status": "ok"})
|
|
- [x] Create api/deps module for get_db, cookie_person, and error_response
|
|
- [x] Use settings.py (DOOF_DB) for DB path in main and deps
|
|
|
|
Acceptance criteria
|
|
- main.py primarily wires app, routers, settings, and lifespan
|
|
- Reverse proxy cleaned up; no stray global clients
|
|
- Health check available and excluded from auth/security
|
|
|
|
Status: Complete
|
|
|
|
Notes
|
|
- Lifespan handler initializes a dev httpx.AsyncClient using settings.frontend_dev_url and stores it on app.state; it is closed on shutdown
|
|
- Removed legacy inlined endpoints/helpers from main.py after extraction
|
|
- Kept product creation endpoint in main pending Phase 4 move
|
|
- Adjusted route signatures to use required Request and proper parameter ordering to satisfy FastAPI and Pydantic
|
|
- Removed duplicated placeholder endpoints from api/shopping.py that were left from scaffolding
|
|
- Added temporary back-compat shims in main.py (get_duplicates, validate_meal) forwarding to api.meals to keep tests passing; plan to remove in Phase 4 when tests are updated to import from feature modules
|
|
|
|
---
|
|
|
|
## Phase 2 — API design and OpenAPI
|
|
- [x] Ensure response_model is set on all endpoints that return models
|
|
- [x] Normalize inconsistent routes (remove trailing slash on POST /shopping/) — N/A in code; no stray trailing slash routes
|
|
- [x] Replace MealIdWrapper body with path param for requesting meals: POST /shopping/current/meals/{mealId}
|
|
- Decision: Kept existing v1 route POST /shopping/current/meals/me with body wrapper to avoid breaking tests; path-param variant deferred to v2
|
|
- [x] Add Location header on create endpoints (recipes, meals, persons, products)
|
|
- Decision: Retained 200 response codes for v1 compatibility (tests expect 200); 201 can be adopted in v2
|
|
- [x] Define cookie-based security scheme in OpenAPI and apply to secured routes (by operationId)
|
|
- [x] Keep reusable ProblemDetails responses; ensure 4xx schemas reference it consistently
|
|
|
|
Acceptance criteria
|
|
- openapi.json shows correct schemas, cookieAuth security for protected operations, and references to reusable ProblemDetails responses — Met
|
|
- Clients can generate SDKs without manual fixes — Improved (consistent models and responses)
|
|
|
|
---
|
|
|
|
## Phase 3 — Data access and performance
|
|
- [x] Centralize transaction scoping per-request (dependency) and remove scattered conn.commit() in handlers
|
|
- Implemented in `api/deps.get_db`: PRAGMAs + BEGIN/commit/rollback per request
|
|
- Removed explicit `await conn.commit()` calls from handlers and product DB helpers
|
|
- [x] Add DB PRAGMAs on connect (WAL, foreign_keys=ON, synchronous=NORMAL)
|
|
- [x] Batch-load related data to avoid N+1 (recipes/ingredients, meals/participants)
|
|
- [x] Batch-load meal participants (fetch IDs once, bulk load persons)
|
|
- [x] Batch-load recipe ingredients across a page in `api/recipes.list_recipes`
|
|
- [x] Add indexes for common filters/joins
|
|
- [x] ingredients.recipe_id
|
|
- [x] ingredients.meal_id
|
|
- [x] meal_participants.meal_id, role
|
|
- [x] meal_recipes.meal_id
|
|
- [x] recipes.date_hidden (+ name, id composite for pagination)
|
|
- [x] persons.name (for LIKE queries)
|
|
|
|
Acceptance criteria
|
|
- Noticeable reduction in query count for list/detail endpoints (verified by logging/sqlite trace/inspection of code paths)
|
|
- No functional regressions; tests still green
|
|
|
|
Status: Complete
|
|
|
|
Notes
|
|
- PRAGMAs applied on connect and write transactions now scoped to each HTTP request
|
|
- Indexes added to improve common lookups and pagination
|
|
- Batch-loading of recipe ingredients implemented; meal participants batching implemented via `meals.bulk_load_participants` and used by `api/meals.get_upcoming_meals`.
|
|
|
|
---
|
|
|
|
## Phase 4 — Modeling and validation
|
|
- [ ] Move inline API request/response models out of main.py into feature modules
|
|
- [ ] ProductUrl
|
|
- [ ] CurrentShoppingList
|
|
- [ ] PurchasedShoppingList
|
|
- [ ] LoginBody
|
|
- [ ] Extract validation (e.g., validate_meal) into a service layer for reuse
|
|
- [ ] Add enums/constants for participant roles
|
|
|
|
Acceptance criteria
|
|
- Cleaner main.py; feature modules own their request/response contracts
|
|
- Validation logic reusable outside HTTP layer
|
|
|
|
---
|
|
|
|
## Phase 5 — Testing and fixtures
|
|
- [ ] Introduce pytest fixtures for common setup (DB, test data, auth)
|
|
- [ ] Remove duplication in tests and centralize helpers
|
|
- [ ] Add tests for new health endpoint and 201 Location headers
|
|
|
|
Acceptance criteria
|
|
- Test suite readability improved; repeated setup minimized
|
|
- Coverage for new behaviors added
|
|
|
|
---
|
|
|
|
## Stretch items
|
|
- [ ] Structured logging with request IDs
|
|
- [ ] A background job or cron-friendly script for cleanup tasks
|
|
- [ ] Optional migration to pydantic-settings if desired later
|
|
|
|
---
|
|
|
|
## Concrete file plan (proposed)
|
|
- main.py (app wiring only)
|
|
- settings.py (centralized configuration)
|
|
- api/
|
|
- __init__.py
|
|
- recipes.py, meals.py, persons.py, shopping.py, auth.py (routers)
|
|
- domain/
|
|
- recipes/models.py, repo.py, service.py
|
|
- meals/models.py, repo.py, service.py
|
|
- ingredients/models.py, repo.py, service.py
|
|
- persons/models.py, repo.py, service.py
|
|
- shopping/models.py, repo.py, service.py
|
|
|
|
Note: We can adopt this structure gradually without moving DB code immediately; start with routers, then iterate.
|
|
|
|
---
|
|
|
|
## Risks and mitigations
|
|
- Router extraction could break imports
|
|
- Mitigation: introduce api package and re-export symbols as needed during transition
|
|
- OpenAPI changes can affect clients
|
|
- Mitigation: keep v1 paths stable; document changes like 201 + Location
|
|
- N+1 fixes may alter behavior subtly
|
|
- Mitigation: add dedicated tests for list/detail payloads
|
|
|
|
---
|
|
|
|
## Progress log
|
|
- 2025-10-18: Created strategy document and added settings.py (not yet wired)
|
|
- 2025-10-18: Finalized router layout and health endpoint plan — Phase 0 complete
|
|
- 2025-10-18: Created api package and scaffolded routers (recipes, meals, persons, shopping, auth) with placeholders
|
|
- 2025-10-18: Extracted recipes routes to api/recipes.py and wired router; added /healthz
|
|
- 2025-10-18: Extracted persons routes to api/persons.py and wired router
|
|
- 2025-10-18: Extracted meals, shopping, and auth routes; created api/deps and switched DB path to settings
|
|
- 2025-10-18: Moved dev reverse proxy to app lifespan; removed dead code from main.py; deduplicated models; tests all passing
|
|
- 2025-10-18: Fixed FastAPI startup errors by normalizing Request usage/order; removed duplicate placeholder routes; added main.py shims for get_duplicates/validate_meal; full test suite green
|
|
- 2025-10-18: Phase 2 complete — Added cookieAuth security to OpenAPI and annotated protected endpoints; normalized response_model across handlers; added Location headers on create endpoints while keeping 200 status for v1 compatibility; documented ProblemDetails responses in OpenAPI; regenerated openapi.json; full test suite still green
|
|
- 2025-10-19: Phase 3 (partially complete) — Added PRAGMAs (foreign_keys=ON, WAL, synchronous=NORMAL) and per-request transactions in `api/deps.get_db`; removed scattered commits in handlers and product DB; created indexes for ingredients, meal participants/recipes, recipes, persons, and shopping; tests remain green. Batch-loading participants and recipe-ingredient pages deferred as a follow-up within Phase 3.
|
|
- 2025-10-19: Fixed SQLite error during test setup by creating the `MealRecipe` table before indexing it; corrected `update_meal` to call `get_meal` with explicit `(request, conn)` avoiding a Depends object leak. Full test suite now passes (100%). Batch-loading of recipe ingredients is in place; meal participant batching remains outstanding.
|
|
-.
|
|
- 2025-10-19: Implemented participant batch-loading (`meals.bulk_load_participants`) and updated `api/meals.get_upcoming_meals` to use it; re-ran the test suite (green). Phase 3 marked complete; Phase 4-5 next.
|
|
|
|
---
|
|
|
|
## Next actions
|
|
- Phase 3: Implement batch-loading to eliminate N+1
|
|
- Batch-load meal participants and persons
|
|
- Batch-load recipe ingredients for list pages
|
|
- Optionally add lightweight query logging to validate reductions
|
|
- Phase 4: Move remaining request/response models and helpers (ProductUrl, CurrentShoppingList, PurchasedShoppingList, LoginBody, validate_meal/get_duplicates) fully into feature modules and update tests to import from there; then remove back-compat shims from main.py
|
|
- Phase 5: Add fixtures for DB/auth and tests for health + Location headers; consider adding perf checks
|
|
|
|
Follow-ups (v2 candidates)
|
|
- Adopt 201 Created for create endpoints and adjust tests/clients
|
|
- Switch POST /shopping/current/meals/me (body) to POST /shopping/current/meals/{mealId} (path param)
|
|
|
|
### Health endpoint plan (Phase 1 target)
|
|
- Path: GET /healthz
|
|
- Response: {"status": "ok"}
|
|
- No DB access; fast and safe
|
|
|
|
### Router extraction migration notes
|
|
- Step 1: Create api package and empty routers with existing path/operation_id/signatures (no logic change)
|
|
- Step 2: Move handlers from main.py into respective files, fix imports
|
|
- Step 3: In main.py, include_router for each; keep response_model and decorator metadata intact
|
|
- Step 4: Run tests and adjust import paths |