jableader 65d655851d feat: MemberRef adoption for recipes.createdBy and shopping.purchasedBy; add TDD; update spec; keep suite green

2025-11-01 17:53:07 +11:00

25 KiB

Raw Blame History

0.5 Validated v2 household behaviors (tests snapshot)

GET /api/v1/households/{householdSlug}/shopping/{listId} returning purchased list + lookups scoped to household; cross-household returns 404.
POST /api/v1/households/{householdSlug}/meals/{mealId}/consumed marks a meal consumed within the household; validates timezone on provided consumedDate; clears outstanding meal requests only within that household.
POST /api/v1/households/{householdSlug}/shopping/current/meals/me requests a meal under the household scope; visible only within that household in GET current.
DELETE /api/v1/households/{householdSlug}/shopping/current/meals/{mealId} unrequests the meal (scoped) and returns { ok: true }.
Comprehensive v2 coverage exists for scoping, purchases, request/unrequest, meals CRUD/consumed, and OpenAPI security. PASS.

Backend Specification: Household Multi-Tenancy (v2)

This document has been validated against the current codebase (v1) to ensure the plan captures all required changes. It starts with a concise baseline of what exists today, then details the v2 multi-tenancy/auth refactor with concrete, file-scoped steps and acceptance criteria.

Date reviewed: 2025-11-01 (post-cutover: JWT + households live, Argon2 enabled, members endpoint added, v2 routers inlined; all checks green; OpenAPI exported)

Repo modules checked: main.py, api/* (v2-only; *_v2 modules removed), users/*, households/*, meals/*, ingredients/*, recipes/*, products/*, shopping/*, common.py, db.py, settings.py, tests in tests/*. persons/* remains for migration compatibility but has no routes.

Key conventions in v1:

Response shape uses camelCase aliases (via ApiModel in common.py).
RFC7807 Problem Details are returned for 400/404/422 with application/problem+json (handlers in main.py).
Pagination uses a Page<T> envelope: { items: T[], nextCursor?: string, prevCursor?: string, total: number }.
Some endpoints require a user_id cookie; missing cookie generally yields 422 (validation), except one special-case mapping to 401 (see below).

0. Current v1 Baseline (retired)

0.1 Auth (prototype)

Mechanism: user_id cookie containing a Person ID.
Endpoints (api/auth.py):
- POST /api/v1/auth/login body { username: string } → sets user_id cookie if person exists; 404 ProblemDetails if not.
- POST /api/v1/auth/refresh → returns the current Person (requires cookie).
Dependencies (api/deps.py):
- cookie_person: requires cookie, loads Person by id; 401 if unknown id, 422 if cookie missing (FastAPI validation).
- error_response: builds RFC7807 responses.

Special-case 401: Removed. v1 cookie-based auth and routes have been retired in favor of JWT-only v2.

0.2 API surface (historical)

Persons (api/persons.py)
- GET /api/v1/persons → Page<Person> with optional name filter q, cursor pagination.
- POST /api/v1/persons → create Person; sets Location header.
Recipes (api/recipes.py)
- GET /api/v1/recipes → Page<Recipe>; loads ingredients per page.
- GET /api/v1/recipes/{id} → full recipe (ingredients + createdBy).
- GET /api/v1/recipes/parse?url=... (auth required) → scrape/parse a recipe; 400 if not found.
- GET /api/v1/recipes/ingredients/parse?ingredients=...&ingredients=... → parse raw ingredient lines (no auth); matches existing products.
- POST /api/v1/recipes (auth required) → validate (≥1 ingredient), perform versioning (hide base if id≥0), set createdById, insert ingredients; sets Location header.
- DELETE /api/v1/recipes/{id} (auth required) → soft-delete (hide) recipe.
Meals (api/meals.py)
- GET /api/v1/meals/upcoming?from&to → upcoming unconsumed meals in range; bulk loads participants, recipes, extra ingredients.
- GET /api/v1/meals/{id} → meal with participants/recipes/extra ingredients.
- POST /api/v1/meals → validate meal, insert participants, recipes, extra ingredients; sets Location header.
- PUT /api/v1/meals/{id} → validate/update and return current state.
- POST /api/v1/meals/{id}/consumed (auth required) → marks consumed; requires timezone in consumed_date if provided; removes any shopping requests for the meal.
- DELETE /api/v1/meals/{id} (auth required) → soft-delete meal via shopping cleanup then update.
Shopping (api/shopping.py)
- GET /api/v1/shopping/current → aggregate of outstanding ingredient requests (no auth), requested meals, purchased items, plus lookup maps (meals, recipes, ingredients, and any referenced purchased lists). Outward storeName enum uses "home" for internal empty string.
- GET /api/v1/shopping/{listId} → purchased shopping list + lookups; 404 if not found.
- POST /api/v1/shopping (auth required) → purchase list with items; validates invariants in repository; returns full list + lookups. Missing/invalid cookie maps to 401 (special-case).
- GET /api/v1/shopping/current/me/ingredients (auth required) → outstanding ingredient requests for me (personId from cookie).
- POST /api/v1/shopping/current/me/ingredients (auth required) → sync my ingredient requests (add missing, remove extra). Matches existing entries by id or line.
- POST /api/v1/shopping/current/meals/me (auth required) → request a meal; 404 if meal not found; prevents duplicates.
- DELETE /api/v1/shopping/current/meals/{mealId} (auth required) → unrequest a meal; 404 if meal not found.

0.3 Data model (SQLite, created by `db.create()`)

Person(id PK, name UNIQUE)
Recipe(id PK, name, link, serves, image_urls TEXT JSON, based_on_recipe FK, date_created, created_by_id FK NOT NULL, date_hidden, hidden_by_id FK)
Ingredient(id PK, name, line, preparation, unit, quantity REAL, product_id FK, recipe_id FK, meal_id FK)
Product(id PK, product_id UNIQUE, shop_code, link, name, quantity, unit, img_small, img_large, raw_data TEXT) + ProductTag(food_item_id, tag)
Meal(id PK, suggested_date, consumed_date NULL, deleted_date NULL, purchase_date NULL)
- MealParticipant(meal_id, person_id, role)
- MealRecipe(meal_id, recipe_id, servings)
ShoppingList(id PK, created_date, store_name, purchased_by_id FK)
- ShoppingListItem(id PK, ingredient_id FK, list_id FK NULL for requests, person_id FK, meal_id FK, recipe_id FK, created_date)

0.4 Validated behaviors and invariants (carried forward into v2 where applicable)

ProblemDetails content-type returned for 400/404/422.
Pagination: cursor is treated leniently (invalid → start). prevCursor is computed by a DB helper; total is non-null integer.
Recipe creation requires ≥1 ingredient; soft-deletes older version when updating.
Meals validation (api.meals.validate_meal):
- Must have ≥1 chef, ≥1 cleanup, ≥1 consumer.
- Must have either at least one recipe or at least one extra ingredient.
- No duplicate participants by role.
- Each MealRecipe.servings must be > 0.
- POST {id} and body.id must match for update.
Shopping invariants (shopping.repository):
- Purchase requires purchased_by_id and ≥1 item.
- Requests: exactly one of (ingredient|meal); person required; ingredient requests must have valid ingredient id (or be inserted when syncing); meal request must not already exist; purchased meal auto-updates purchase_date and clears requests when all meal ingredients are covered by list items.
- Outward storeName uses home|woolworths|coles (maps internal empty string to home).
Auth cookie missing:
- Most protected endpoints → 422 from FastAPI validation.
- Special case: POST /api/v1/shopping → 401 via custom handler.

1. Objective (status)

v2 household-scoped API is complete and v1 has been removed from the app. Tests have been migrated to v2 equivalents or disabled when purely legacy. OpenAPI reflects JWT bearer and household scoping.

Refactor the backend from a single-tenant architecture to a robust, multi-tenant system based on "Households". This requires evolving the data model to enforce data isolation, overhauling the authentication system to support standard credential types, and introducing an invitation mechanism for household management. This plan is adapted to the existing codebase.

2. Core Concepts & Data Model

Data Isolation: All primary resources (recipes, meals, shopping_lists, etc.) MUST be strictly scoped to a household_id. No API endpoint should ever return data from a household the authenticated user does not belong to.
User vs. Person: The existing persons table and Person model will be replaced by a users table and a HouseholdMember link table.
Authentication: Replace the prototype user_id cookie with a standard JWT-based system.

New & Modified Data Models

households (New Table):
- id: Primary Key
- name: TEXT NOT NULL
- slug: TEXT NOT NULL UNIQUE
users (New Table, replaces persons):
- id: Primary Key
- email: TEXT NOT NULL UNIQUE
- display_name: TEXT NOT NULL
- profile_photo_url: TEXT
household_members (New Table):
- user_id: FK to users.id
- household_id: FK to households.id
- role: TEXT NOT NULL (e.g., 'admin', 'member')
- PRIMARY KEY (user_id, household_id)
local_credentials (New Table):
- user_id: PK, FK to users.id
- hashed_password: TEXT NOT NULL
oauth_credentials (New Table):
- user_id: FK to users.id
- provider: TEXT NOT NULL (e.g., 'google')
- provider_user_id: TEXT NOT NULL
- PRIMARY KEY (provider, provider_user_id)
household_invitations (New Table):
- id: Primary Key
- household_id: FK to households.id
- email: TEXT NOT NULL
- invited_by_user_id: FK to users.id
- token: TEXT NOT NULL UNIQUE
- expires_at: DATETIME NOT NULL
- status: TEXT NOT NULL ('pending', 'accepted', 'expired')

3. API & Logic Changes

3.1. Authentication API (current)

api/auth.py implements JWT-based authentication with Argon2 hashing for new passwords (PBKDF2 verification fallback for legacy hashes).
- POST /api/v1/auth/register: { email, password, displayName } → creates User + LocalCredentials; returns access token; sets HttpOnly refresh cookie.
- POST /api/v1/auth/login: { email, password } → validates; returns access token; sets HttpOnly refresh cookie.
- POST /api/v1/auth/refresh: returns a fresh access token (reads refresh cookie).
- POST /api/v1/auth/logout: clears refresh cookie.

Notes:

get_current_user (JWT) is used by protected endpoints (households, shopping purchase, etc.). Legacy cookie_person remains only for historical v1 module references and will be removed with full v2 completion.
RFC7807 error semantics and the existing OpenAPI augmentation are preserved. The special-case 401 mapping for POST shopping is obsolete under JWT.

3.2. Household & Tenancy API (current)

api/households.py:
- GET /api/v1/users/me/households: (JWT) lists the user's households.
- POST /api/v1/households: (JWT) creates a household and adds the creator as admin.
- Scoped router /api/v1/households/{householdSlug} with GET /whoami and GET /members (returns { id, displayName, role }).

Household-scoped routes (implemented):

api/recipes_v2.py: /api/v1/households/{householdSlug}/recipes list/get/create/delete.
api/meals.py: /api/v1/households/{householdSlug}/meals upcoming/get/create/update/consumed/delete (inlined from v2).
api/shopping.py: /api/v1/households/{householdSlug}/shopping current/list-by-id/purchase/request/unrequest (inlined from v2, with shared DTOs in api/shopping_models.py).

Repositories accept household_id and filter by it across meals, recipes, and shopping domains.

3.3. Invitation API

Add to api/households.py:
- POST /api/v1/households/{householdSlug}/invitations: (Auth: JWT, household membership). Body { email }. Creates HouseholdInvitation, sends email.
Add to api/auth.py:
- POST /api/v1/invitations/accept: (Auth: JWT). Body { token }. Validates token, adds user to household.

4. Actionable Implementation Steps

[~] Database Schema and Migration:
- ✅ Schema Definition: Added new packages and tables:
  - users with tables User, LocalCredentials, OAuthCredentials (see users/repository.py).
  - households with tables Household, HouseholdMember, HouseholdInvitation (see households/repository.py).
  - db.create() now initializes these tables alongside existing v1 tables.
- ✅ Migration Logic: Implemented scripts/migration_to_households.py which:
  - Connect to the database (reusing logic from db.py).
  - Calls create() for new repos to ensure tables exist.
  - Adds a household_id column to tenant tables: Recipe, Ingredient, Meal, MealParticipant, MealRecipe, ShoppingList, ShoppingListItem (idempotent).
  - Creates indices idx_<table>_household_id for all above tables.
  - Creates a default household { name: "My Household", slug: "default" } and backfills household_id with its ID for existing rows.
  - Ports Person rows to User (email derived as <name>@example.com) and creates HouseholdMember links (role admin).
- Bootstrap Update: Modify db.py so that a fresh database bootstrap (db.create_schema) calls the create() functions for the new repositories and not the old persons repository.

✅ Indices/Constraints: Enforced unique Household.slug; added idx_*_household_id indices; foreign keys added with ON DELETE CASCADE where applicable in new tables.
✅ Acceptance (initial): Added tests/test_migration_households.py covering: new tables exist, household_id columns exist, default household created, and data porting from Person to User and HouseholdMember. Full test suite passes.
Pending follow-ups for this step:
- Add composite indices (household_id, id) where high-cardinality pagination will benefit.
- Extend migration to add FK constraints from tenant tables to Household(id) where safe.
- Plan and implement data backfill for cross-table references once users replace persons in code.

[✅] Implement New Authentication System:
- Implemented v2 JWT auth while keeping v1 cookie auth intact during transition:
  - api/auth_v2.py now issues HS256 JWT access tokens and sets an HttpOnly refresh cookie.
    - Endpoints: POST /api/v1/auth/register, POST /api/v1/auth/login, POST /api/v1/auth/refresh, POST /api/v1/auth/logout.
  - api/deps.get_current_user verifies JWT access tokens and loads the User from DB.
  - security.py provides a minimal JWT utility with configurable issuer/audience, secrets, and TTLs.
  - settings.py extended with JWT config and secrets via env.
  - Tests updated: tests/test_auth_and_households_v2.py now expects JWT-shaped tokens and verifies refresh flow.
  - OpenAPI augmentation updated to include refreshV2 in protected ops and to mark /users/me/* and /households/* with bearerAuth + 403.
- Notes:
  - Password hashing remains SHA-256 placeholder; to be upgraded to bcrypt/argon2 in a follow-up.
  - v1 cookie auth remains operational until all routes are migrated under households and updated.
- Acceptance: Unauthenticated requests return 401; household membership failures continue to return 403; tests pass.
[✅] Implement Household Scoping:
- ✅ Created households/ package with models.py and repository.py.
- ✅ Added initial api/households.py router:
  - GET /api/v1/users/me/households (requires bearer token) → lists memberships.
  - POST /api/v1/households (requires bearer token) → creates household and adds current user as admin.
- ✅ Implemented get_household_from_slug in api/deps.py.
- ✅ Refactor main.py:
  - Create a new APIRouter for household-scoped routes, e.g., household_router = APIRouter(prefix="/api/v1/households/{householdSlug}").
  - Mounted a scoped helper endpoint and a new recipes v2 router under this prefix.

✅ Recipes scoping:
- Added api/recipes_v2.py providing /api/v1/households/{householdSlug}/recipes with list/get/create/delete.
  - Added scoped repo helpers in recipes/repository.py and exported via recipes/__init__.py.
  - Tests in tests/test_recipes_household_v2.py validate isolation across households (PASS).
- ✅ Meals: Added api/meals_v2.py with:
  - /api/v1/households/{householdSlug}/meals/upcoming filtering by household_id.
  - /api/v1/households/{householdSlug}/meals/{id} returns 404 across households.
POST /api/v1/households/{householdSlug}/meals/{id}/consumed marks consumed with optional consumedDate (requires timezone if provided); removes outstanding meal requests; all operations scoped to household.
POST /api/v1/households/{householdSlug}/meals creates a meal (scoped) with Location header; reuses v1 validation (≥1 chef, ≥1 cleanup, ≥1 consumer; ≥1 recipe or ≥1 extra ingredient).
PUT /api/v1/households/{householdSlug}/meals/{id} updates a meal (scoped), enforcing URL/body ID match and validation; returns updated state.
DELETE /api/v1/households/{householdSlug}/meals/{id} soft-deletes a meal (scoped) after removing outstanding requests in that household.
- Repository functions find_upcoming_meals_by_date_range_scoped and find_meal_by_id_scoped added. Tests verify isolation and consumed behavior.
✅ Shopping:
- Added api/shopping_v2.py with:
  - GET /api/v1/households/{householdSlug}/shopping/current (scoped aggregate).
  - GET /api/v1/households/{householdSlug}/shopping/{listId} returning purchased list + lookups scoped to household; cross-household returns 404.
  - POST /api/v1/households/{householdSlug}/shopping to purchase list items scoped to household; validates invariants and updates outstanding requests.
- Scoped helpers in shopping/repository.py and shopping/__init__.py filter by household_id (find items, load list, purchased ingredients, and purchase_scoped).
Tests: tests/test_shopping_household_v2.py (current isolation), tests/test_shopping_list_by_id_v2.py (list-by-id scoping), tests/test_shopping_purchase_v2.py (scoped purchase), tests/test_shopping_request_meal_v2.py (request/unrequest). PASS.
Notes: Normalized Ingredient.preparation to allow NULLs from DB (treated as empty string) to avoid 422 in v2 responses; set purchased_by_id in scoped purchase from JWT user.
Tests: tests/test_meals_write_v2.py verifies create/update/delete flows under household scope; all v2 scoping tests PASS.
Acceptance: The same queries as v1, when run under different household slugs and users, return isolated data sets; cross-household access yields 403. Achieved.
Notes:
- Migration added household_id columns and indices, enabling next step to filter by household without additional schema changes.

[~] Implement Household & Invitation Logic:
- ✅ Households router implemented for listing and creating households.
- ✅ Invitations API:
  - POST /api/v1/households/{householdSlug}/invitations (JWT + membership): creates a pending invitation and returns a token.
  - POST /api/v1/invitations/accept (JWT): validates token, adds user as member, marks invitation as accepted.
  - Tests: tests/test_invitations_v2.py cover create + accept flow and membership visibility.
- ⏳ Email Delivery: Stub only. Pending adding an email sender utility/service and persistence of delivery state.

Acceptance: Admins can invite by email; invite accept adds user to household; listing households shows membership with roles. (Met for API behavior; email sending pending.)

[✅] Update OpenAPI Specification:
- ✅ Augmentation updated in api/openapi.py:
  - Adds bearerAuth security scheme (cookieAuth removed).
  - Marks household routes and /users/me/* with bearer security and adds 403 Problem response.
  - Preserves RFC7807 Problem responses and shopping storeName outward enum normalization.

✅ New tests tests/test_openapi_security.py verify presence of bearerAuth and 403s.
✅ Export script writes updated openapi.json; re-run after adding meals v2 write endpoints to include them in the schema.

[✅] Refactor and Test:

Tests updated to v2 household-scoped endpoints with JWT setup. Legacy v1 API tests are currently skipped (tests/test_main.py, tests/test_v1.py).
Full suite green under make all-checks. OpenAPI export successful.
Next cleanup: remove persons/ package and remaining references in domain internals once users fully replace persons in models.

5. Gaps vs Current Codebase (summary)

Status summary:

Implemented: JWT auth v2 with refresh cookie; household domains and membership; invitations; full household scoping across recipes/meals/shopping; OpenAPI augmentation with bearerAuth and 403; RFC7807 preserved; tests green.
DTO alignment: Meals use MemberRef { id, displayName } (no Person in outward schema). MemberRef consolidated in api/dtos.py. Shopping DTOs/mappers consolidated in api/shopping_models.py.
DTO alignment (v2): Meals use MemberRef; Recipes include createdById and createdBy (MemberRef); Shopping purchasedBy is now a MemberRef on outward lists.
Security: Password hashing now prefers Argon2 for new accounts with PBKDF2 verification fallback.
New: Household members listing GET /api/v1/households/{householdSlug}/members returning [{ id, displayName, role }].
Preserved: camelCase responses, Page<T> semantics, Location headers on create, shopping storeName normalization ("home").
Codebase cleanup: v2 routers are inlined as canonical modules; legacy *_v2.py files removed. v1 cookie auth and routers are not mounted.
Current v2 recipes shape: createdById and createdBy (MemberRef) are included in outward Recipe responses. hiddenBy will be added when hide flows are wired to users.

Remaining work (prioritized cleanup to final state):

Remove the persons/ package and all code references across domains (recipes, meals, shopping). Replace Person with users/HouseholdMember everywhere:

Code hotspots today: recipes/models.py (imports Person), recipes/repository.py (FKs, hide_recipe signature), meals/models.py (participants as List[Person]), api/shopping_models.py and shopping/models.py (ShoppingList.purchased_by typed as Person), and shopping/repository.py (FKs to Person).
Replace internal usages with user_id/MemberRef where outward, and update repository DDL to stop referencing Person.
Remove api.deps.cookie_person once no tests or code depend on it.

Recipes outward schema: DONE for createdById/createdBy (MemberRef). Remaining: add hiddenBy (MemberRef) when hide flows are wired to users.
Shopping outward DTOs: DONE — ShoppingListOut.purchasedBy is a MemberRef; mapping added. Ensure clients shift to displayName.
Delete or port legacy v1 test modules that are currently skipped (tests/test_main.py, tests/test_v1.py). Either migrate assertions to v2 routes or remove them so the full test run has no skips. Then remove the last vestiges of v1-only helpers.
Database polish:

Add composite indices like (household_id, id) where pagination benefits (e.g., Recipe, Meal, ShoppingListItem).
Add explicit FK constraints from tenant tables to Household(id) where safe.
Ensure fresh bootstraps include household_id in table DDL (e.g., recipes.repository.create), not just via migration.

6. Edge Cases and Error Semantics

Unauthenticated → 401 (JWT), not 422 (replace the v1 cookie-missing 422/401 hybrid behavior).
Not a member of household → 403 with ProblemDetails.
Validation stays 400 with ProblemDetails (e.g., meal validation failures; recipe without ingredients; shopping invariants).
422 reserved for request model validation errors.

7. Rollout & Migration Plan (updated)

Phase A: Introduced users alongside persons; added JWT while cookie auth remained briefly for transition.
Phase B: Added households and backfilled default household; migrated routes under /households/{householdSlug}.
Phase C (now): Removed cookie auth from app surface and inlined v2 routers; final step is to remove persons entirely and ensure no tests are skipped.

Data migration acceptance:

All existing data appears under a default household and is accessible to the migrated user accounts.

8. Acceptance Criteria (v2)

Authentication: JWT bearer with register/login; refresh via HttpOnly cookie; get_current_user used in protected routes.
Tenancy: All read/write queries filter by household_id; cross-household access returns 403; invitation flow works.
API parity: v1 behavior preserved aside from auth/paths; ProblemDetails and pagination semantics intact; Location headers set on creates.
OpenAPI: Security scheme is JWT; household path param present; arrays required and enums normalized; 400/404/422 standardized; 403 added on household routes.

Final-state definition (what “done” looks like):

No legacy v1 endpoints mounted; no *_v2.py files in repo (done).
No persons package in codebase or responses; all tests ported from v1 and no tests are skipped.
OpenAPI reflects only JWT-secured, household-scoped endpoints.

25 KiB Raw Blame History