Refactor household models and repository queries to use dynamic column selection; remove email from invitation schema in OpenAPI

README.md

@@ -1,132 +1,146 @@
-Meal planner backend
+## Doof Backend (aka Munch Ease) 🍽️
+
+FastAPI backend for collaborative meal planning, recipe wrangling, and grocery shopping. Household-scoped, JWT-secured, SQLite-fast. Bring your recipes, we’ll do the rest.
+
+## Why you’ll love it
+
+- 🚀 Fast and modern API with FastAPI + Pydantic v2
+- 🏠 Household-scoped everything (recipes, meals, shopping) for clean multi-user isolation
+- 🔐 JWT access tokens + HttpOnly refresh cookie (Argon2 password hashing)
+- 🧪 100% test-friendly: ephemeral SQLite, deterministic APIs, RFC7807 errors
+- 🧠 Ingredient NLP parsing with product matching
+- 🛒 One-click shopping lists: request meals or individual ingredients, dedup done for you
+- 🧾 OpenAPI on tap for your frontend and SDKs
+- 🔗 Built-in recipe parsing from URLs (with BeautifulSoup + httpx)
+
+## Stack
+
+- Runtime: Python 3.11+
+- Web: FastAPI, Starlette
+- Data models: Pydantic v2 (camelCase JSON via custom `ApiModel`)
+- Database: SQLite (aiosqlite), schema bootstrapped in each `repository.py`
+- Auth: Custom HMAC-SHA256 JWTs + Argon2 password hashing
+- Parsing/Scraping: `ingredient-parser-nlp`, `beautifulsoup4`, `httpx`
+- Tooling: ruff (lint+format), mypy (typecheck), pytest, pre-commit, uvicorn
+
+## Features at a glance
+
+- 🔑 Auth: Register, login, refresh, logout; access bearer token + HttpOnly refresh cookie
+- 🏡 Households: Create, list, invite members; accept invitations via shareable links
+- 🧾 Recipes: Create, list, paginate, delete (hide) with actor attribution; parse-from-url helper
+- 🧪 Ingredients: NLP parse one or many lines; product matching baked in
+- 🍽️ Meals: Plan, get, update, delete, mark consumed; validate participants and recipes
+- 🛍️ Shopping: Request meals and ingredients, view current list, purchase to a list, fetch past lists
+- 🩺 Health: `GET /healthz` returns a tiny “ok” model for probes
+- 📜 Errors: RFC7807 Problem Details everywhere, with tidy camelCase payloads
+
+## Project structure
+
+- `main.py` — FastAPI app factory, routers, exception handlers, health, frontend proxy/static
+- `settings.py` — Environment-driven runtime settings (no external deps)
+- `security.py` — Minimal JWT utilities (HS256) + helpers
+- `db.py` — aiosqlite connect + `create()` bootstraps all domain tables
+- `api/` — HTTP surface (versioned under `/api/v1`) - STRICTLY NO SQL AT THIS LAYER!
+	- `auth.py` — register, login, refresh, logout
+	- `households.py` — create/list, members, invitations, scoped routes
+	- `ingredients.py` — household-scoped NLP parsing
+	- `recipes.py` — household-scoped list/get/create/delete, public utilities
+	- `meals.py` — household-scoped CRUD + mark consumed
+	- `shopping.py` — household-scoped current list, purchase, request/unrequest
+	- `openapi.py` — OpenAPI augmentation (cookie auth, problem+json)
+	- `deps.py` — DB/session, auth, household scoping, error helpers
+- Domain packages (models + repository + sql mutation + helpers):
+	- `persons/`
+	- `users/`, `households/`, `ingredients/`, `recipes/` (incl. `scraping.py`), `meals/`, `products/` (Coles/Woolworths helpers), `shopping/`
+	- Strive to be useful as a fairly portal package independant of the http layer
+- `scripts/` — Utility scripts
+  - `export_openapi.py` — writes `openapi.json` from the live app
+  - `manual_parse_recipes.py` — Test parsing against a variety of sources
+- `tests/` — API and domain tests with fixtures and sample files
 
 ## Quickstart
 
 First time setup:
+
 ```bash
 make install
 ```
 
-Run the development server:
+Run the dev server:
+
 ```bash
 make dev
 ```
 
 Run tests:
+
 ```bash
 make test
 ```
 
-Run all quality checks (lint, typecheck, test, format check, OpenAPI export):
+Run all checks (lint, typecheck, tests, format check, OpenAPI export):
+
 ```bash
 make all-checks
 ```
 
-## Structure
+## Environment
 
-- `main.py`: FastAPI app with all HTTP endpoints.
-- `db.py`: aiosqlite connection + schema bootstrap across subpackages (calls each feature's `repository.create`).
-- Domain packages with models and persistence:
-	- `products/` (models.py, repository.py, scrapers for Woolworths/Coles)
-	- `ingredients/` (models.py, repository.py)
-	- `recipes/` (models.py, repository.py, scraping.py)
-	- `meals/` (models.py, repository.py, service.py)
-	- `persons/` (models.py, repository.py)
-	- `shopping/` (models.py, repository.py)
-- `tests/`: unit and API tests with sample HTTP fixtures.
+These are read from the environment (see `settings.py`):
 
-## Getting started
+- `DOOF_DB` — SQLite file path (default `./data/doof.sqlite`)
+- `DOOF_PROD` — `true/false` controls frontend proxy vs. static serving (default `false`)
+- `FRONTEND_DEV_URL` — dev server to reverse-proxy in non-prod (default `http://localhost:8000/`)
+- `DOOF_JWT_ISSUER`, `DOOF_JWT_AUDIENCE` — JWT claims
+- `DOOF_JWT_ACCESS_TTL`, `DOOF_JWT_REFRESH_TTL` — TTLs in seconds (default 900/2592000)
+- `DOOF_JWT_ACCESS_SECRET_B64`, `DOOF_JWT_REFRESH_SECRET_B64` — base64 secrets (use in prod!)
 
-### Manual setup (alternative to make install)
+Dev convenience: if secrets aren’t provided, deterministic dev secrets are used. Don’t ship those.
+
+## API surface (v1)
+
+- Base: `/api/v1`
+- Auth: `/auth/register`, `/auth/login`, `/auth/refresh`, `/auth/logout`
+- Households: `/households`, `/users/me/households`, `/households/{slug}/members`, `/households/{slug}/whoami`, invitations create/accept
+- Ingredients: `/households/{slug}/ingredients/parse` (single or batch parsing)
+- Recipes: `/households/{slug}/recipes` (list/paged, create), `/{id}` (get/delete), `/parse-from-url`
+- Meals: `/households/{slug}/meals` (create/update/delete/get/upcoming/mark-consumed)
+- Shopping: `/households/{slug}/shopping/current`, `/{listId}`, request/unrequest meals and ingredients, purchase lists
+- Health: `/healthz`
+
+Errors are consistent Problem Details (`application/problem+json`). Models serialize in camelCase.
+
+## Make targets
+
+- `make install` — Create venv and install dependencies
+- `make dev` — Run dev server (`uvicorn main:app --reload`)
+- `make test` — Run tests (pytest -q)
+- `make format` — Format with ruff
+- `make lint` — Lint with ruff
+- `make typecheck` — Type check with mypy
+- `make openapi` — Export OpenAPI to `openapi.json`
+- `make all-checks` — Lint + typecheck + tests + format check + OpenAPI export
+- `make clean` — Remove venv and caches
+
+## Development notes
+
+- Schema bootstrap: `db.create(conn)` calls each feature’s `repository.create` to make tables
+- Frontend integration:
+	- Dev: requests for non-`/api/*` are reverse-proxied to `FRONTEND_DEV_URL`
+	- Prod: static files served from `./front-dist`
+- Security: Argon2 password hashing; HS256 JWTs signed with your secrets; refresh token stored as HttpOnly cookie
+- DX niceties: camelCase JSON by default, strict validation, helpful error messages
+
+## OpenAPI
+
+Generate the spec file used by the frontend and CI:
 
-Create and activate virtual environment:
 ```bash
-python3 -m venv .venv
-source .venv/bin/activate  # On Windows: .venv\Scripts\activate
+make openapi
 ```
 
-Install packages:
-```bash
-pip install -r ./requirements.txt
-pip install -r ./dev-requirements.txt
-```
+This writes `openapi.json` at the repo root.
 
-Install pre-commit hooks:
-```bash
-pip install pre-commit
-pre-commit install
-```
+---
 
-### Running the application
-
-Run API (dev):
-```bash
-make dev
-# or: uvicorn main:app --reload
-```
-
-Run tests:
-```bash
-make test
-# or: pytest -q
-```
-
-### Available Make targets
-
-- `make install` - Create venv and install all dependencies
-- `make dev` - Run development server
-- `make test` - Run tests
-- `make format` - Format code with ruff
-- `make lint` - Lint code with ruff
-- `make typecheck` - Type check with mypy
-- `make openapi` - Export OpenAPI schema
-- `make all-checks` - Run all quality checks
-- `make clean` - Remove venv and caches
-
-## Tooling
-
-This repo includes baseline configs in `pyproject.toml`:
-- ruff (format and lint)
-- mypy (type check)
-
-Pre-commit hooks are configured to run:
-- ruff (lint + format)
-- mypy
-- trailing whitespace fixer
-- end-of-file fixer
-
-Quality checks (run locally):
-```bash
-make format      # Format code
-make lint        # Lint code
-make typecheck   # Type check
-make all-checks  # Run all checks
-```
-
-## Environment variables
-
-Copy `.env.example` to `.env` and customize as needed:
-
-- DOOF_DB: Path to sqlite database (default: `./data/doof.sqlite`)
-- DOOF_PROD: Set to `true` in production (default: `false`)
-- FRONTEND_DEV_URL: Frontend dev server URL for reverse proxy (default: `http://localhost:8080/`)
-- DOOF_PORT: Port the server listens on when containerized; align Dockerfile `EXPOSE` accordingly.
-
-## OpenAPI schema
-
-- Generate the schema artifact used by the frontend and CI checks:
-```
-python scripts/export_openapi.py
-```
-
-This writes `openapi.json` to the repo root. Versioned endpoints live under `/api/v1`, legacy under `/api` (deprecated with `Deprecation` header).
-
-## Schema lint/diff (manual)
-
-Optionally, lint and compare schemas locally using Node tools:
-```
-npx -y @stoplight/spectral-cli lint openapi.json
-npx -y openapi-diff --fail-on-changed --fail-on-incompatible path/to/baseline.json openapi.json
-```
-
-Keep a `baseline.json` on release branches to detect breaking changes.
+Built with love and leftovers. Hungry for issues and PRs. 🧑‍🍳

api/__init__.py

@@ -1,2 +1,2 @@
 # API package for FastAPI routers.
-# Routers will be split by feature: recipes, meals, persons, shopping, auth.
+# Routers are split by feature: recipes, meals, shopping, auth, households.

api/households.py

@@ -7,7 +7,9 @@ import aiosqlite
 from fastapi import APIRouter, Depends, Request
 
 from api.deps import get_current_user, get_db, error_response, get_household_from_slug
+from pydantic import Field
 from common import ApiModel
+from households import repository as households_repo
 from users.models import User
 
 router = APIRouter(tags=["households"])
@@ -34,19 +36,8 @@ async def list_my_households(
     user: User = Depends(get_current_user),
     conn: aiosqlite.Connection = Depends(get_db),
 ):
-    results: list[HouseholdResponse] = []
-    async with conn.execute(
-        """
-        SELECT h.id, h.name, h.slug FROM Household h
-        JOIN HouseholdMember m ON m.household_id = h.id
-        WHERE m.user_id = ?
-        ORDER BY h.id
-        """,
-        (user.id,),
-    ) as c:
-        async for row in c:
-            results.append(HouseholdResponse(id=int(row[0]), name=row[1], slug=row[2]))
-    return results
+    results = await households_repo.list_for_user(conn, user.id)
+    return [HouseholdResponse.model_validate(h) for h in results]
 
 
 @router.post("/households", response_model=HouseholdResponse)
@@ -57,21 +48,10 @@ async def create_household(
     conn: aiosqlite.Connection = Depends(get_db),
 ):
     slug = slugify(body.name)
-    try:
-        async with conn.execute(
-            "INSERT INTO Household (name, slug) VALUES (?, ?)", (body.name, slug)
-        ) as cur:
-            lrid = cur.lastrowid
-            if lrid is None:
-                return error_response(request, 400, "Unable to create household")
-            hid = int(lrid)
-        await conn.execute(
-            "INSERT INTO HouseholdMember (user_id, household_id, role) VALUES (?, ?, ?)",
-            (user.id, hid, "admin"),
-        )
-        return HouseholdResponse(id=hid, name=body.name, slug=slug)
-    except Exception:
+    household = await households_repo.create_for_user(conn, body.name, slug, user.id)
+    if household is None:
         return error_response(request, 400, "Unable to create household")
+    return HouseholdResponse.model_validate(household)
 
 
 # Household-scoped router and endpoint to validate scoping mechanics
@@ -100,57 +80,51 @@ async def list_members(
     household=Depends(get_household_from_slug),
     conn: aiosqlite.Connection = Depends(get_db),
 ):
-    members: list[HouseholdMember] = []
-    async with conn.execute(
-        """
-        SELECT u.id, u.display_name, m.role
-        FROM HouseholdMember m
-        JOIN User u ON u.id = m.user_id
-        WHERE m.household_id = ?
-        ORDER BY lower(u.display_name), u.id
-        """,
-        (household["id"],),
-    ) as c:
-        async for row in c:
-            members.append(HouseholdMember(id=int(row[0]), display_name=row[1], role=row[2]))
-    return members
+    member_data = await households_repo.list_members(conn, household["id"])
+    return [HouseholdMember.model_validate(m) for m in member_data]
 
 
 # Invitations
-class CreateInvitationBody(ApiModel):
-    email: str
-
-
 class InvitationResponse(ApiModel):
     token: str
     status: str = "pending"
 
 
-@scoped.post("/invitations", response_model=InvitationResponse)
+class InviteLinkResponse(ApiModel):
+    # Force snake_case in JSON output to match spec and tests
+    invite_link: str = Field(serialization_alias="invite_link")
+
+
+@scoped.post("/invitations", response_model=InviteLinkResponse)
 async def create_invitation(
     request: Request,
-    body: CreateInvitationBody,
     user: User = Depends(get_current_user),
     household=Depends(get_household_from_slug),
     conn: aiosqlite.Connection = Depends(get_db),
 ):
     import secrets
     from datetime import datetime, timedelta
+    from urllib.parse import urljoin, urlencode
+    from settings import settings
 
     token = secrets.token_urlsafe(24)
     expires_at = (datetime.utcnow() + timedelta(days=14)).isoformat() + "Z"
-    try:
-        await conn.execute(
-            """
-            INSERT INTO HouseholdInvitation (household_id, email, invited_by_user_id, token, expires_at, status)
-            VALUES (?, ?, ?, ?, ?, ?)
-            """,
-            (household["id"], body.email, user.id, token, expires_at, "pending"),
+
+    success = await households_repo.create_invitation(
+        conn, household["id"], user.id, token, expires_at
     )
-        return InvitationResponse(token=token)
-    except Exception:
+
+    if not success:
         return error_response(request, 400, "Unable to create invitation")
 
+    base = settings.frontend_dev_url
+    # Ensure base ends with a slash for urljoin
+    if not base.endswith("/"):
+        base = base + "/"
+    path_with_query = f"invitations/accept?{urlencode({'token': token})}"
+    invite_link = urljoin(base, path_with_query)
+    return InviteLinkResponse(invite_link=invite_link)
+
 
 class AcceptInvitationBody(ApiModel):
     token: str
@@ -169,39 +143,25 @@ async def accept_invitation(
     user: User = Depends(get_current_user),
     conn: aiosqlite.Connection = Depends(get_db),
 ):
-    token = body.token
     # Lookup invitation
-    async with conn.execute(
-        "SELECT id, household_id, status FROM HouseholdInvitation WHERE token = ?",
-        (token,),
-    ) as c:
-        row = await c.fetchone()
-    if not row:
+    invitation = await households_repo.get_invitation_by_token(conn, body.token)
+    if not invitation:
         return error_response(request, 404, "Invitation not found")
-    inv_id = int(row[0])
-    hid = int(row[1])
-    status = row[2]
-    if status != "pending":
+
+    if invitation.status != "pending":
         return error_response(request, 400, "Invitation not pending")
-    # Add membership if not exists
-    await conn.execute(
-        "INSERT OR IGNORE INTO HouseholdMember (user_id, household_id, role) VALUES (?, ?, ?)",
-        (user.id, hid, "member"),
-    )
-    # Mark invitation accepted
-    await conn.execute(
-        "UPDATE HouseholdInvitation SET status = 'accepted' WHERE id = ?",
-        (inv_id,),
+
+    # Add membership and mark invitation accepted
+    await households_repo.accept_invitation(
+        conn, invitation.id, user.id, invitation.household_id
     )
+
     # Load household details for response
-    async with conn.execute(
-        "SELECT id, name, slug FROM Household WHERE id = ?",
-        (hid,),
-    ) as c:
-        hrow = await c.fetchone()
-    if not hrow:
+    household = await households_repo.get_household_by_id(conn, invitation.household_id)
+    if not household:
         return error_response(request, 404, "Household not found")
+
     return AcceptInvitationResponse(
         status="accepted",
-        household=HouseholdResponse(id=int(hrow[0]), name=hrow[1], slug=hrow[2]),
+        household=HouseholdResponse.model_validate(household),
     )

api/meals.py

@@ -440,13 +440,9 @@ async def delete_meal_scoped(
     if not meal:
         return error_response(request, 404, "Meal not found")
     # Remove outstanding requests for this meal in current household
-    try:
     from shopping.repository import remove_meal_request_scoped
 
     await remove_meal_request_scoped(conn, meal_id, hid)
-    except Exception:
-        # Fallback: remove regardless of household (legacy cleanup)
-        await shopping.remove_request(conn, person=None, meal=meal)
     await meals.delete_meal(conn, meal.id)
 
     return MealOut(

api/recipes.py

@@ -252,7 +252,8 @@ async def parse_from_url(
     # Build an unsaved Recipe object using NLP parsing and product matching; do not insert
     r = await recipes.parse_recipe(conn, user, body.url)
     if not r:
-        return error_response(None, 404, "Recipe data not found at URL")
+        # Parsing failed: treat as unprocessable rather than not-found
+        return error_response(None, 422, "Unable to parse recipe from URL")
     # Return the same shape a client would POST to create
     return RecipeCreate(
         name=r.name,

backend-spec.md

@@ -1,372 +0,0 @@
-## 0.2 API surface (historical)
-- Persons API removed.
-
-  - Recipes (v1 historical paths):
-    - Added `api/shopping.py` with:
-      - GET `/api/v1/households/{householdSlug}/shopping/current` (scoped aggregate).
-      - GET `/api/v1/households/{householdSlug}/shopping/{listId}` returning purchased list + lookups scoped to household; cross-household returns 404.
- Implementation policy updates (2025-11-01):
- - Routers must use repository helpers for all persistence; direct `conn.execute(...)` calls in routers are prohibited, except for controlled PRAGMA/transaction management in `api/deps.py`.
- - Recipes create now accepts a lean body (`RecipeCreate`) without internal IDs and sets `createdById` from the JWT user; delete uses a repository helper to set `date_hidden` and `hidden_by_id` atomically.
-## 0.5 Validated v2 household behaviors (tests snapshot)
-  - GET `/api/v1/households/{householdSlug}/shopping/{listId}` returning purchased list + lookups scoped to household; cross-household returns 404.
-  - POST `/api/v1/households/{householdSlug}/meals/{mealId}/consumed` marks a meal consumed within the household; validates timezone on provided `consumedDate`; clears outstanding meal requests only within that household.
-  - POST `/api/v1/households/{householdSlug}/shopping/current/meals/me` requests a meal under the household scope; visible only within that household in GET current.
-  - DELETE `/api/v1/households/{householdSlug}/shopping/current/meals/{mealId}` unrequests the meal (scoped) and returns `{ ok: true }`.
-  - POST `/api/v1/households/{householdSlug}/shopping/current/ingredients` requests an ad‑hoc ingredient scoped to household+user; duplicates deduped per user per household.
-  - DELETE `/api/v1/households/{householdSlug}/shopping/current/ingredients` removes an ad‑hoc ingredient request for the current user in this household; idempotent.
-  - POST `/api/v1/households/{householdSlug}/recipes/parse-from-url` returns a RecipeCreate payload parsed from a URL (stateless; household auth enforced). Shape matches the body accepted by `POST /recipes`.
-  - GET `/api/v1/households/{householdSlug}/ingredients/parse` parses ingredients (scoped; JWT + membership). Supports either `?line=...` (returns a single `Ingredient`) or repeated `?lines=...` query params (returns `Ingredient[]`). Attempts best-effort product matching.
-  - Comprehensive v2 coverage exists for scoping, purchases, request/unrequest, meals CRUD/consumed, and OpenAPI security. PASS.
-# Backend Specification: Household Multi-Tenancy (v2)
-
-This document has been validated against the current codebase (v1) to ensure the plan captures all required changes. It starts with a concise baseline of what exists today, then details the v2 multi-tenancy/auth refactor with concrete, file-scoped steps and acceptance criteria.
-
-Date reviewed: 2025-11-02 (post-cutover: JWT + households live, Argon2-only, members endpoint added, v2 routers consolidated; all checks green; OpenAPI exported; persons fully removed)
-
-Repo modules checked: `main.py`, `api/*` (v2-only; no `*_v2.py` files remain), `users/*`, `households/*`, `meals/*`, `ingredients/*`, `recipes/*`, `products/*`, `shopping/*`, `common.py`, `db.py`, `settings.py`, tests in `tests/*`. The legacy `persons/*` package has been deleted.
-
-Key conventions in v1:
-- Response shape uses camelCase aliases (via `ApiModel` in `common.py`).
-- RFC7807 Problem Details are returned for 400/404/422 with `application/problem+json` (handlers in `main.py`).
-- Pagination uses a `Page<T>` envelope: `{ items: T[], nextCursor?: string, prevCursor?: string, total: number }`.
-- Some endpoints require a `user_id` cookie; missing cookie generally yields 422 (validation), except one special-case mapping to 401 (see below).
-
----
-
-## 0. Current v1 Baseline (retired)
-
-### 0.1 Auth (prototype)
-- Mechanism: `user_id` cookie containing a Person ID.
-- Endpoints (`api/auth.py`):
-  - POST `/api/v1/auth/login` body `{ username: string }` → sets `user_id` cookie if person exists; 404 ProblemDetails if not.
-  - POST `/api/v1/auth/refresh` → returns the current Person (requires cookie).
-- Dependencies (`api/deps.py`):
-  - `cookie_person`: requires cookie, loads Person by id; 401 if unknown id, 422 if cookie missing (FastAPI validation).
-  - `error_response`: builds RFC7807 responses.
-
-Special-case 401: Removed. v1 cookie-based auth and routes have been retired in favor of JWT-only v2.
-
-### 0.2 API surface (historical)
-- Persons API removed.
-
-  - Recipes (`api/recipes.py`)
-  - GET `/api/v1/recipes` → `Page<Recipe>`; loads ingredients per page.
-  - GET `/api/v1/recipes/{id}` → full recipe (ingredients + createdBy).
-  - POST `/api/v1/households/{householdSlug}/recipes/parse-from-url` (auth required) → scrape/parse a recipe URL; returns `RecipeCreate` (no id/createdBy); 404 if not found.
-  - GET `/api/v1/households/{householdSlug}/ingredients/parse` → parse ingredients, supporting `line` (single) or repeated `lines` params (batch). JWT + membership; matches existing products.
-  - POST `/api/v1/recipes` (auth required) → validate (≥1 ingredient), perform versioning (hide base if id≥0), set createdById, insert ingredients; sets `Location` header.
-  - DELETE `/api/v1/recipes/{id}` (auth required) → soft-delete (hide) recipe.
-
-- Meals (`api/meals.py`)
-  - GET `/api/v1/meals/upcoming?from&to` → upcoming unconsumed meals in range; bulk loads participants, recipes, extra ingredients.
-  - GET `/api/v1/meals/{id}` → meal with participants/recipes/extra ingredients.
-  - POST `/api/v1/meals` → validate meal, insert participants, recipes, extra ingredients; sets `Location` header.
-  - PUT `/api/v1/meals/{id}` → validate/update and return current state.
-  - POST `/api/v1/meals/{id}/consumed` (auth required) → marks consumed; requires timezone in `consumed_date` if provided; removes any shopping requests for the meal.
-  - DELETE `/api/v1/meals/{id}` (auth required) → soft-delete meal via shopping cleanup then update.
-
-- Shopping (`api/shopping.py`)
-  - GET `/api/v1/shopping/current` → aggregate of outstanding ingredient requests (no auth), requested meals, purchased items, plus lookup maps (`meals`, `recipes`, `ingredients`, and any referenced purchased lists). Outward storeName enum uses `"home"` for internal empty string.
-  - GET `/api/v1/shopping/{listId}` → purchased shopping list + lookups; 404 if not found.
-  - POST `/api/v1/shopping` (auth required) → purchase list with items; validates invariants in repository; returns full list + lookups. Missing/invalid cookie maps to 401 (special-case).
-  - GET `/api/v1/shopping/current/me/ingredients` (auth required) → outstanding ingredient requests for me (personId from cookie).
-  - POST `/api/v1/shopping/current/me/ingredients` (auth required) → sync my ingredient requests (add missing, remove extra). Matches existing entries by `id` or `line`.
-  - POST `/api/v1/shopping/current/meals/me` (auth required) → request a meal; 404 if meal not found; prevents duplicates.
-  - DELETE `/api/v1/shopping/current/meals/{mealId}` (auth required) → unrequest a meal; 404 if meal not found.
-
-### 0.3 Data model (SQLite, created by `db.create()`)
-  (legacy Person table removed; users table is canonical)
-- Recipe(id PK, name, link, serves, image_urls TEXT JSON, based_on_recipe FK, date_created, created_by_id FK NOT NULL → User.id, date_hidden, hidden_by_id FK → User.id)
-- Ingredient(id PK, name, line, preparation, unit, quantity REAL, product_id FK, recipe_id FK, meal_id FK)
-- Product(id PK, product_id UNIQUE, shop_code, link, name, quantity, unit, img_small, img_large, raw_data TEXT) + ProductTag(food_item_id, tag)
-- Meal(id PK, suggested_date, consumed_date NULL, deleted_date NULL, purchase_date NULL)
-  - MealParticipant(meal_id, person_id, role)
-  - MealRecipe(meal_id, recipe_id, servings)
-- ShoppingList(id PK, created_date, store_name, purchased_by_id FK → User.id)
-  - ShoppingListItem(id PK, ingredient_id FK, list_id FK NULL for requests, person_id FK → User.id, meal_id FK, recipe_id FK, created_date)
-
-### 0.4 Validated behaviors and invariants (carried forward into v2 where applicable)
-- ProblemDetails content-type returned for 400/404/422.
-- Pagination: `cursor` is treated leniently (invalid → start). `prevCursor` is computed by a DB helper; `total` is non-null integer.
-- Recipe creation requires ≥1 ingredient; soft-deletes older version when updating.
-- Meals validation (`api.meals.validate_meal`):
-  - Must have ≥1 chef, ≥1 cleanup, ≥1 consumer.
-  - Must have either at least one recipe or at least one extra ingredient.
-  - No duplicate participants by role.
-  - Each MealRecipe.servings must be > 0.
-  - POST `{id}` and body.id must match for update.
-- Shopping invariants (`shopping.repository`):
-  - Purchase requires `purchased_by_id` and ≥1 item.
-  - Requests: exactly one of (ingredient|meal); person required; ingredient requests must have valid ingredient id (or be inserted when syncing); meal request must not already exist; purchased meal auto-updates `purchase_date` and clears requests when all meal ingredients are covered by list items.
-  - Outward `storeName` uses `home|woolworths|coles` (maps internal empty string to `home`).
-- Auth cookie missing:
-  - Most protected endpoints → 422 from FastAPI validation.
-  - Special case: POST `/api/v1/shopping` → 401 via custom handler.
-
----
-
-## 1. Objective (status)
-v2 household-scoped API is complete and v1 has been removed from the app. The `persons` package and routes are deleted. Tests have been migrated to v2 equivalents or disabled when purely legacy. OpenAPI reflects JWT bearer and household scoping.
-
-Refactor the backend from a single-tenant architecture to a robust, multi-tenant system based on "Households". This requires evolving the data model to enforce data isolation, overhauling the authentication system to support standard credential types, and introducing an invitation mechanism for household management. This plan is adapted to the existing codebase.
-
-## 2. Core Concepts & Data Model
-
-- **Data Isolation**: All primary resources (`recipes`, `meals`, `shopping_lists`, etc.) MUST be strictly scoped to a `household_id`. No API endpoint should ever return data from a household the authenticated user does not belong to.
-- **User vs. Person**: The existing `persons` table and `Person` model will be replaced by a `users` table and a `HouseholdMember` link table.
-- **Authentication**: Replace the prototype `user_id` cookie with a standard JWT-based system.
-
-### New & Modified Data Models
-
-- **`households`** (New Table):
-  - `id`: Primary Key
-  - `name`: `TEXT NOT NULL`
-  - `slug`: `TEXT NOT NULL UNIQUE`
-- **`users`** (New Table, replaces `persons`):
-  - `id`: Primary Key
-  - `email`: `TEXT NOT NULL UNIQUE`
-  - `display_name`: `TEXT NOT NULL`
-  - `profile_photo_url`: `TEXT`
-- **`household_members`** (New Table):
-  - `user_id`: FK to `users.id`
-  - `household_id`: FK to `households.id`
-  - `role`: `TEXT NOT NULL` (e.g., 'admin', 'member')
-  - `PRIMARY KEY (user_id, household_id)`
-- **`local_credentials`** (New Table):
-  - `user_id`: PK, FK to `users.id`
-  - `hashed_password`: `TEXT NOT NULL`
-- **`oauth_credentials`** (New Table):
-  - `user_id`: FK to `users.id`
-  - `provider`: `TEXT NOT NULL` (e.g., 'google')
-  - `provider_user_id`: `TEXT NOT NULL`
-  - `PRIMARY KEY (provider, provider_user_id)`
-- **`household_invitations`** (New Table):
-  - `id`: Primary Key
-  - `household_id`: FK to `households.id`
-  - `email`: `TEXT NOT NULL`
-  - `invited_by_user_id`: FK to `users.id`
-  - `token`: `TEXT NOT NULL UNIQUE`
-  - `expires_at`: `DATETIME NOT NULL`
-  - `status`: `TEXT NOT NULL` ('pending', 'accepted', 'expired')
-
-## 3. API & Logic Changes
-
-### 3.1. Authentication API (current)
-
-- `api/auth.py` implements JWT-based authentication with Argon2 hashing for new passwords (PBKDF2 verification fallback for legacy hashes).
-  - `POST /api/v1/auth/register`: `{ email, password, displayName }` → creates User + LocalCredentials; returns access token; sets HttpOnly refresh cookie.
-  - `POST /api/v1/auth/login`: `{ email, password }` → validates; returns access token; sets HttpOnly refresh cookie.
-  - `POST /api/v1/auth/refresh`: returns a fresh access token (reads refresh cookie).
-  - `POST /api/v1/auth/logout`: clears refresh cookie. After logout, subsequent `POST /api/v1/auth/refresh` returns 401.
-
-Notes:
-- `get_current_user` (JWT) is used by protected endpoints (households, shopping purchase, etc.). Legacy cookie-based helpers have been removed from the app surface.
-- RFC7807 error semantics and the existing OpenAPI augmentation are preserved. The special-case 401 mapping for POST shopping is obsolete under JWT.
-
-### 3.2. Household & Tenancy API (current)
-
-- `api/households.py`:
-  - `GET /api/v1/users/me/households`: (JWT) lists the user's households.
-  - `POST /api/v1/households`: (JWT) creates a household and adds the creator as admin.
-  - Scoped router `/api/v1/households/{householdSlug}` with `GET /whoami` and `GET /members` (returns `{ id, displayName, role }`).
-
-Household-scoped routes (implemented):
-- `api/recipes.py`: `/api/v1/households/{householdSlug}/recipes` list/get/create/delete.
-- `api/meals.py`: `/api/v1/households/{householdSlug}/meals` upcoming/get/create/update/consumed/delete (inlined from v2).
-- `api/shopping.py`: `/api/v1/households/{householdSlug}/shopping` current/list-by-id/purchase/request/unrequest (inlined from v2, with shared DTOs in `api/shopping_models.py`).
-
-Shopping requests parity (preserved in v2):
-- Request meal: `POST /api/v1/households/{householdSlug}/shopping/current/meals/me` (scoped) and unrequest `DELETE /current/meals/{mealId}`.
-- Request individual ingredient: `POST /api/v1/households/{householdSlug}/shopping/current/ingredients` (scoped). Response is `ListIngredientItem`; item appears in `GET /current` under `outstandingItems`. Household isolation enforced. Duplicate requests for the same ingredient by the same user within the same household return the existing request (no duplicate rows). Covered by `tests/test_shopping_request_ingredient_dedupe_v2.py`.
- - Unrequest individual ingredient: `DELETE /api/v1/households/{householdSlug}/shopping/current/ingredients` (scoped). Body `{ ingredientId }`. Returns `{ ok: true }` even if nothing was deleted.
-
-Repositories accept `household_id` and filter by it across `meals`, `recipes`, and `shopping` domains.
-
-Route surface lockdown:
-- Legacy unscoped endpoints (e.g., `/api/v1/meals/*`, `/api/v1/shopping/*`) are not exposed; all data routes are under `/api/v1/households/{householdSlug}/...`.
-- Protected household routes return 401 without Authorization; with Authorization, non-existent households yield 404, and non-membership yields 403.
-- Tests: `tests/test_route_surface_lockdown.py` asserts these behaviors.
-
-### 3.3. Invitation API
-
-- **Add to `api/households.py`**:
-  - **`POST /api/v1/households/{householdSlug}/invitations`**: (Auth: JWT, household membership). Body `{ email }`. Creates `HouseholdInvitation`, sends email.
-- **Add to `api/auth.py`**:
-  - **`POST /api/v1/invitations/accept`**: (Auth: JWT). Body `{ token }`. Validates token, adds user to household, and returns `{ status: "accepted", household: { id, name, slug } }`.
-
-## 4. Actionable Implementation Steps
-
-1.  **[~] Database Schema**:
-    - ✅ **Schema Definition**: Added new packages and tables:
-      - `users` with tables `User`, `LocalCredentials`, `OAuthCredentials` (see `users/repository.py`).
-      - `households` with tables `Household`, `HouseholdMember`, `HouseholdInvitation` (see `households/repository.py`).
-      - `db.create()` now initializes these tables alongside existing v1 tables.
-  - ✅ **Bootstrap**: `db.create()` initializes all v2 tables, including `users` and `households`, and tenant tables already include `household_id` columns and indices by default. The legacy ad-hoc migration script has been removed; tests and dev use fresh DDL via `db.create()`.
-
-### Feature parity checklist (OpenAPI diffs vs master)
-
-Completed:
-- Auth endpoints (`/api/v1/auth/*`) migrated to JWT with tokens/refresh cookie.
-- Household-scoped recipes/meals/shopping endpoints in place.
-- `POST /api/v1/households/{householdSlug}/recipes/parse-from-url` implemented (returns `RecipeCreate`).
-- `GET /api/v1/households/{householdSlug}/ingredients/parse` implemented with dual modes: `?line=...` (single `Ingredient`) or repeated `?lines=...` (returns `Ingredient[]`). Scoped.
-
-Outstanding (tracked):
-- None identified blocking parity for shopping/recipes needed by the frontend as of 2025-11-01. Re-check if any v1 product scrape/create endpoint needs re-exposure; current frontend uses household flows and parsing utilities.
-    - **Bootstrap Update**: Modify `db.py` so that a fresh database bootstrap (`db.create_schema`) calls the `create()` functions for the new repositories and *not* the old `persons` repository.
-  - ✅ **Indices/Constraints**: Enforced unique `Household.slug`; added `idx_*_household_id` indices; foreign keys added with `ON DELETE CASCADE` where applicable in new tables.
-  - ✅ **Acceptance (initial)**: Household tables and `household_id` columns are covered by repository DDL and exercised by v2 tests. Full test suite passes.
-
-  - Pending follow-ups for this step:
-    - (Done) Add composite indices `(household_id, id)` where high-cardinality pagination will benefit (Recipe, Meal).
-    - Extend migration to add FK constraints from tenant tables to `Household(id)` where safe.
-    - Plan and implement data backfill for cross-table references once `users` replace `persons` in code.
-    - Ensure fresh bootstraps include `household_id` in all tenant table DDL (now updated for Ingredient, Recipe, Meal, ShoppingList, ShoppingListItem).
-
-2.  **[✅] Implement New Authentication System**:
-    - Implemented JWT auth (Argon2 password hashing) and removed v1 cookie auth:
-      - `api/auth.py` issues HS256 JWT access tokens and sets an HttpOnly refresh cookie.
-        - Endpoints: `POST /api/v1/auth/register`, `POST /api/v1/auth/login`, `POST /api/v1/auth/refresh`, `POST /api/v1/auth/logout`.
-      - `api/deps.get_current_user` verifies JWT access tokens and loads the `User` from DB.
-      - `security.py` provides a minimal JWT utility with configurable issuer/audience, secrets, and TTLs.
-      - `settings.py` extended with JWT config and secrets via env.
-      - Tests updated to expect JWT-shaped tokens and verify refresh flow.
-      - OpenAPI augmentation marks `/users/me/*` and `/households/*` with `bearerAuth` + `403`.
-    - Notes:
-      - Password hashing upgraded to Argon2 via `argon2-cffi`.
-      - v1 cookie auth is removed.
-    - Acceptance: Unauthenticated requests return 401; household membership failures continue to return 403; tests pass.
-
-3.  **[✅] Implement Household Scoping**:
-    - ✅ Created `households/` package with `models.py` and `repository.py`.
-    - ✅ Added initial `api/households.py` router:
-      - `GET /api/v1/users/me/households` (requires bearer token) → lists memberships.
-      - `POST /api/v1/households` (requires bearer token) → creates household and adds current user as admin.
-    - ✅ Implemented `get_household_from_slug` in `api/deps.py`.
-    - ✅ Refactor `main.py`:
-      - Household-scoped routers consolidated under `api/households.py` and mounted at `/api/v1/households/{householdSlug}`.
-      - Recipes, meals, and shopping routers are v2-only and live in `api/recipes.py`, `api/meals.py`, and `api/shopping.py` respectively.
-  - ✅ Recipes scoping:
-    - Consolidated into `api/recipes.py` providing `/api/v1/households/{householdSlug}/recipes` with list/get/create/delete.
-      - Scoped repo helpers in `recipes/repository.py` and exported via `recipes/__init__.py`.
-      - Tests validate isolation across households (PASS).
-    - ✅ Meals: Consolidated into `api/meals.py` with:
-      - `/api/v1/households/{householdSlug}/meals/upcoming` filtering by `household_id`.
-      - `/api/v1/households/{householdSlug}/meals/{id}` returns 404 across households.
-  - `POST /api/v1/households/{householdSlug}/meals/{id}/consumed` marks consumed with optional `consumedDate` (requires timezone if provided); removes outstanding meal requests; all operations scoped to household.
-  - `POST /api/v1/households/{householdSlug}/meals` creates a meal (scoped) with Location header; reuses v1 validation (≥1 chef, ≥1 cleanup, ≥1 consumer; ≥1 recipe or ≥1 extra ingredient).
-  - `PUT /api/v1/households/{householdSlug}/meals/{id}` updates a meal (scoped), enforcing URL/body ID match and validation; returns updated state.
-  - `DELETE /api/v1/households/{householdSlug}/meals/{id}` soft-deletes a meal (scoped) after removing outstanding requests in that household.
-      - Repository functions `find_upcoming_meals_by_date_range_scoped` and `find_meal_by_id_scoped` added. Tests verify isolation and consumed behavior.
-  - ✅ Shopping:
-    - Added `api/shopping_v2.py` with:
-      - GET `/api/v1/households/{householdSlug}/shopping/current` (scoped aggregate).
-      - GET `/api/v1/households/{householdSlug}/shopping/{listId}` returning purchased list + lookups scoped to household; cross-household returns 404.
-      - POST `/api/v1/households/{householdSlug}/shopping` to purchase list items scoped to household; validates invariants and updates outstanding requests.
-    - Scoped helpers in `shopping/repository.py` and `shopping/__init__.py` filter by `household_id` (find items, load list, purchased ingredients, and purchase_scoped).
-  - Tests: `tests/test_shopping_household_v2.py` (current isolation), `tests/test_shopping_list_by_id_v2.py` (list-by-id scoping), `tests/test_shopping_purchase_v2.py` (scoped purchase), `tests/test_shopping_request_meal_v2.py` (request/unrequest). PASS.
-  - Notes: Normalized Ingredient.preparation to allow NULLs from DB (treated as empty string) to avoid 422 in v2 responses; set purchased_by_id in scoped purchase from JWT user.
-  - Tests: `tests/test_meals_write_v2.py` verifies create/update/delete flows under household scope; all v2 scoping tests PASS.
-  - **Acceptance**: The same queries as v1, when run under different household slugs and users, return isolated data sets; cross-household access yields 403. Achieved.
-
-  - Notes:
-    - Migration added `household_id` columns and indices, enabling next step to filter by household without additional schema changes.
-    - Data access policy: API layers must delegate persistence to repository modules; no direct SQL in routers. Current status: recipes, meals, and shopping routers call into their repositories for reads/writes. Legacy `*_v2.py` files have been removed; canonical routers are `api/recipes.py`, `api/meals.py`, and `api/shopping.py`.
-
-4.  **[~] Implement Household & Invitation Logic**:
-    - ✅ Households router implemented for listing and creating households.
-    - ✅ Invitations API:
-      - `POST /api/v1/households/{householdSlug}/invitations` (JWT + membership): creates a pending invitation and returns a token.
-      - `POST /api/v1/invitations/accept` (JWT): validates token, adds user as member, marks invitation as accepted.
-      - Tests: `tests/test_invitations_v2.py` cover create + accept flow and membership visibility.
-    - ⏳ Email Delivery: Stub only. Pending adding an email sender utility/service and persistence of delivery state.
-  - **Acceptance**: Admins can invite by email; invite accept adds user to household; listing households shows membership with roles. (Met for API behavior; email sending pending.)
-
-5.  **[✅] Update OpenAPI Specification**:
-    - ✅ Augmentation updated in `api/openapi.py`:
-      - Adds `bearerAuth` security scheme (cookieAuth removed).
-      - Marks household routes and `/users/me/*` with bearer security and adds `403` Problem response.
-      - Preserves RFC7807 Problem responses and shopping storeName outward enum normalization.
-  - ✅ New tests `tests/test_openapi_security.py` verify presence of bearerAuth and 403s.
-  - ✅ Export script writes updated `openapi.json`; re-run after adding meals v2 write endpoints to include them in the schema.
-
-6.  **[✅] Refactor and Test**:
-  - Tests updated to v2 household-scoped endpoints with JWT setup. Legacy v1 API tests are skipped (`tests/test_v1.py`), and the legacy `tests/test_main.py` has been removed.
-  - Full suite green under `make all-checks`. OpenAPI export successful.
-  - Next cleanup: remove `persons/` package and remaining references in domain internals once users fully replace persons in models.
-
----
-
-## 5. Gaps vs Current Codebase (summary)
-
-Status summary:
-- Implemented: JWT auth v2 with refresh cookie; household domains and membership; invitations; full household scoping across recipes/meals/shopping; OpenAPI augmentation with bearerAuth and 403; RFC7807 preserved; tests green.
-- DTO alignment: Meals use MemberRef { id, displayName } (no Person in outward schema). MemberRef consolidated in `api/dtos.py`. Shopping DTOs/mappers consolidated in `api/shopping_models.py`.
- - DTO alignment (v2): Meals use MemberRef; Recipes include `createdById` and `createdBy` (MemberRef); Shopping `purchasedBy` is now a MemberRef on outward lists.
-- Security: Password hashing now uses Argon2 exclusively (argon2-cffi). No PBKDF2 fallback remains.
-- New: Household members listing `GET /api/v1/households/{householdSlug}/members` returning [{ id, displayName, role }].
-- Preserved: camelCase responses, `Page<T>` semantics, `Location` headers on create, shopping storeName normalization ("home").
- - Codebase cleanup: v2 routers are inlined as canonical modules; legacy `*_v2.py` files removed. v1 cookie auth and routers are not mounted.
- - Current v2 recipes shape: `createdById` and `createdBy` (MemberRef) are included. Delete now returns `hiddenById` and `hiddenBy` (MemberRef) for the acting user.
-
-Remaining work (prioritized cleanup to final state):
-1. Remove any remaining references to legacy `Person` semantics in comments or deep internals; ensure all code paths exclusively use Users and HouseholdMember.
-  - Verify repositories and models have no lingering Person FKs or types; outward schemas consistently use `MemberRef`.
-  - Ensure no API module performs direct SQL; all persistence must flow through repositories (enforced during cleanup).
-2. Recipes outward schema: DONE for `createdById`/`createdBy` and `hiddenById`/`hiddenBy` (MemberRef on delete). If/when we expose hide actor on other flows, keep MemberRef.
-3. Shopping outward DTOs: DONE — `ShoppingListOut.purchasedBy` is a MemberRef; mapping added. Ensure clients shift to `displayName`.
-4. Delete or port any remaining legacy v1 test modules that are currently skipped (e.g., `tests/test_v1.py`). Either migrate assertions to v2 routes or remove them so the full test run has no skips. Remove the last vestiges of v1-only helpers.
-5. Database polish:
-  - Add composite indices like `(household_id, id)` where pagination benefits (e.g., Recipe, Meal, ShoppingListItem).
-  - Add explicit FK constraints from tenant tables to `Household(id)` where safe.
-  - Ensure fresh bootstraps include `household_id` in table DDL (e.g., `recipes.repository.create`), not just via migration.
-
----
-
-## 6. Edge Cases and Error Semantics
-- Unauthenticated → 401 (JWT), not 422 (replace the v1 cookie-missing 422/401 hybrid behavior).
-- Not a member of household → 403 with ProblemDetails.
-- Validation stays 400 with ProblemDetails (e.g., meal validation failures; recipe without ingredients; shopping invariants).
-- `422` reserved for request model validation errors.
-
----
-
-## 7. Rollout & Migration Plan (updated)
-- Phase A: Introduced `users` alongside `persons`; added JWT while cookie auth remained briefly for transition.
-- Phase B: Added households and backfilled default household; migrated routes under `/households/{householdSlug}`.
-- Phase C (now): Removed cookie auth from app surface and inlined v2 routers; final step is to remove `persons` entirely and ensure no tests are skipped.
-
-Data migration acceptance:
-- All existing data appears under a default household and is accessible to the migrated user accounts.
-
----
-
-## 8. Acceptance Criteria (v2)
-- Authentication: JWT bearer with register/login; refresh via HttpOnly cookie; `get_current_user` used in protected routes.
-- Tenancy: All read/write queries filter by `household_id`; cross-household access returns 403; invitation flow works.
-- API parity: v1 behavior preserved aside from auth/paths; ProblemDetails and pagination semantics intact; Location headers set on creates.
-- OpenAPI: Security scheme is JWT; household path param present; arrays required and enums normalized; 400/404/422 standardized; 403 added on household routes.
-
-Final-state definition (what “done” looks like):
-- No legacy v1 endpoints mounted; no `*_v2.py` files in repo (done).
-- No `persons` package in codebase or responses; all tests ported from v1 and no tests are skipped.
-- OpenAPI reflects only JWT-secured, household-scoped endpoints.
-
----
-
-Implementation policy updates (2025-11-01):
-
-- Routers must use repository helpers for all persistence; direct `conn.execute(...)` calls in routers are prohibited, except for controlled PRAGMA/transaction management in `api/deps.py`.
-- Recipes create now accepts a lean body (`RecipeCreate`) without internal IDs and sets `createdById` from the JWT user; delete uses a repository helper to set `date_hidden` and `hidden_by_id` atomically.
-- Canonical API files are:
-  - `api/recipes.py` (household-scoped recipes)
-  - `api/meals.py` (household-scoped meals)
-  - `api/shopping.py` (household-scoped shopping)
-  - `api/households.py` (memberships and scoped helpers)
-  - `api/auth.py` (JWT register/login/refresh/logout)
-
-Outstanding cleanup:
-- Remove remaining `Person` fallbacks in domain repositories and delete the `persons/` package once tests are migrated. Ensure no code paths depend on `cookie_person`.
-- Consider adding versioning endpoints for recipes explicitly rather than overloading POST.
- - Add a temporary test helper to seed `User`/`HouseholdMember` rows from legacy `Person` when needed (now available as `tests/user_fixtures.py`). Use this to migrate tests off `persons` before deleting the package.

households/models.py

@@ -1,3 +1,4 @@
+from __future__ import annotations
 from typing import ClassVar
 
 from common import ApiModel
@@ -5,17 +6,15 @@ from common import ApiModel
 
 class Household(ApiModel):
     KEYS: ClassVar[list[str]] = ["id", "name", "slug"]
-
-    id: int = -1
+    id: int
     name: str
     slug: str
 
 
 class HouseholdMember(ApiModel):
-    KEYS: ClassVar[list[str]] = ["user_id", "household_id", "role"]
-
-    user_id: int
-    household_id: int
+    KEYS: ClassVar[list[str]] = ["id", "display_name", "role"]
+    id: int
+    display_name: str
     role: str
 
 
@@ -23,16 +22,13 @@ class HouseholdInvitation(ApiModel):
     KEYS: ClassVar[list[str]] = [
         "id",
         "household_id",
-        "email",
         "invited_by_user_id",
         "token",
         "expires_at",
         "status",
     ]
-
-    id: int = -1
+    id: int
     household_id: int
-    email: str
     invited_by_user_id: int
     token: str
     expires_at: str

households/repository.py

@@ -1,3 +1,8 @@
+from __future__ import annotations
+from typing import List
+from households.models import Household, HouseholdMember, HouseholdInvitation
+
+
 async def create(conn):
     # Households table
     await conn.execute(
@@ -30,7 +35,6 @@ async def create(conn):
         CREATE TABLE IF NOT EXISTS HouseholdInvitation (
             id INTEGER PRIMARY KEY,
             household_id INTEGER NOT NULL,
-            email TEXT NOT NULL,
             invited_by_user_id INTEGER NOT NULL,
             token TEXT NOT NULL UNIQUE,
             expires_at DATETIME NOT NULL,
@@ -75,3 +79,109 @@ async def are_members(conn, household_id: int, user_ids: list[int]) -> bool:
         return True
     found = await member_ids_in_household(conn, household_id, user_ids)
     return found == set(user_ids)
+
+
+async def list_for_user(conn, user_id: int) -> List[Household]:
+    results: List[Household] = []
+    cols = ", ".join([f"h.{k}" for k in Household.KEYS])
+    async with conn.execute(
+        f"""
+        SELECT {cols} FROM Household h
+        JOIN HouseholdMember m ON m.household_id = h.id
+        WHERE m.user_id = ?
+        ORDER BY h.id
+        """,
+        (user_id,),
+    ) as c:
+        async for row in c:
+            results.append(Household(**{k: v for k, v in zip(Household.KEYS, row)}))
+    return results
+
+
+async def create_for_user(conn, name: str, slug: str, user_id: int) -> Household | None:
+    try:
+        async with conn.execute(
+            "INSERT INTO Household (name, slug) VALUES (?, ?)", (name, slug)
+        ) as cur:
+            lrid = cur.lastrowid
+            if lrid is None:
+                return None
+            hid = int(lrid)
+        await conn.execute(
+            "INSERT INTO HouseholdMember (user_id, household_id, role) VALUES (?, ?, ?)",
+            (user_id, hid, "admin"),
+        )
+        return Household(id=hid, name=name, slug=slug)
+    except Exception:
+        return None
+
+
+async def list_members(conn, household_id: int) -> List[HouseholdMember]:
+    members: List[HouseholdMember] = []
+    cols = ", ".join([f"u.{k}" for k in ["id", "display_name"]] + ["m.role"])
+    async with conn.execute(
+        f"""
+        SELECT {cols}
+        FROM HouseholdMember m
+        JOIN User u ON u.id = m.user_id
+        WHERE m.household_id = ?
+        ORDER BY lower(u.display_name), u.id
+        """,
+        (household_id,),
+    ) as c:
+        async for row in c:
+            members.append(
+                HouseholdMember(**{k: v for k, v in zip(HouseholdMember.KEYS, row)})
+            )
+    return members
+
+
+async def create_invitation(
+    conn, household_id: int, invited_by_user_id: int, token: str, expires_at: str
+) -> bool:
+    try:
+        await conn.execute(
+            """
+            INSERT INTO HouseholdInvitation (household_id, invited_by_user_id, token, expires_at, status)
+            VALUES (?, ?, ?, ?, ?)
+            """,
+            (household_id, invited_by_user_id, token, expires_at, "pending"),
+        )
+        return True
+    except Exception:
+        return False
+
+
+async def get_invitation_by_token(conn, token: str) -> HouseholdInvitation | None:
+    cols = ", ".join(HouseholdInvitation.KEYS)
+    async with conn.execute(
+        f"SELECT {cols} FROM HouseholdInvitation WHERE token = ?",
+        (token,),
+    ) as c:
+        row = await c.fetchone()
+    if not row:
+        return None
+    return HouseholdInvitation(**{k: v for k, v in zip(HouseholdInvitation.KEYS, row)})
+
+
+async def accept_invitation(conn, invitation_id: int, user_id: int, household_id: int):
+    await conn.execute(
+        "INSERT OR IGNORE INTO HouseholdMember (user_id, household_id, role) VALUES (?, ?, ?)",
+        (user_id, household_id, "member"),
+    )
+    await conn.execute(
+        "UPDATE HouseholdInvitation SET status = 'accepted' WHERE id = ?",
+        (invitation_id,),
+    )
+
+
+async def get_household_by_id(conn, household_id: int) -> Household | None:
+    cols = ", ".join(Household.KEYS)
+    async with conn.execute(
+        f"SELECT {cols} FROM Household WHERE id = ?",
+        (household_id,),
+    ) as c:
+        row = await c.fetchone()
+    if not row:
+        return None
+    return Household(**{k: v for k, v in zip(Household.KEYS, row)})

main.py

@@ -89,7 +89,7 @@ async def request_validation_exc_handler(request: Request, exc: Exception):
     for e in exc.errors():
         loc = ".".join([str(p) for p in e.get("loc", [])])
         errors.setdefault(loc, []).append(e.get("msg"))
-    # Legacy cookie-based auth behavior removed; standard 422 for validation errors
+    # Standard 422 for request validation errors
 
     body = ProblemDetails(
         title="Validation Error",

meals/repository.py

@@ -284,7 +284,7 @@ async def bulk_load_participants(conn, meals: List[Meal]) -> None:
     """Populate participants for many meals in one query to avoid N+1.
 
     For each meal, fills meal.chefs, meal.cleanup, meal.consumers using a bulk
-    lookup of MealParticipant rows and a single persons.get_by_ids fetch.
+    lookup of MealParticipant rows and a single user lookup.
     """
     if not meals:
         return

openapi.json

@@ -381,23 +381,13 @@
             }
           }
         ],
-        "requestBody": {
-          "required": true,
-          "content": {
-            "application/json": {
-              "schema": {
-                "$ref": "#/components/schemas/CreateInvitationBody"
-              }
-            }
-          }
-        },
         "responses": {
           "200": {
             "description": "Successful Response",
             "content": {
               "application/json": {
                 "schema": {
-                  "$ref": "#/components/schemas/InvitationResponse"
+                  "$ref": "#/components/schemas/InviteLinkResponse"
                 }
               }
             }
@@ -1764,19 +1754,6 @@
         ],
         "title": "CreateHouseholdBody"
       },
-      "CreateInvitationBody": {
-        "properties": {
-          "email": {
-            "type": "string",
-            "title": "Email"
-          }
-        },
-        "type": "object",
-        "required": [
-          "email"
-        ],
-        "title": "CreateInvitationBody"
-      },
       "CurrentShoppingList": {
         "properties": {
           "outstandingItems": {
@@ -2078,23 +2055,18 @@
         ],
         "title": "IngredientPurchaseItemIn"
       },
-      "InvitationResponse": {
+      "InviteLinkResponse": {
         "properties": {
-          "token": {
+          "invite_link": {
             "type": "string",
-            "title": "Token"
-          },
-          "status": {
-            "type": "string",
-            "title": "Status",
-            "default": "pending"
+            "title": "Invite Link"
           }
         },
         "type": "object",
         "required": [
-          "token"
+          "invite_link"
         ],
-        "title": "InvitationResponse"
+        "title": "InviteLinkResponse"
       },
       "ListIngredientItem": {
         "properties": {

recipes/__init__.py

@@ -23,16 +23,37 @@ from recipes.repository import (
     load_recipe_ingredients as load_recipe_ingredients,
     row_to_recipe as row_to_recipe,
 )
-from recipes.scraping import scrape_recipe_ldata as _scrape_recipe_ldata
+from recipes.scraping import (
+    scrape_recipe_ldata as _scrape_recipe_ldata,
+    scrape_recipe_ldata_from_html as _scrape_recipe_ldata_from_html,
+)
 
 
-async def parse_recipe(conn, created_by, url: str) -> Optional[Recipe]:
-    ldata = await _scrape_recipe_ldata(url)
+async def parse_recipe(conn, created_by, url: str, log=None, dump_dir: str | None = None) -> Optional[Recipe]:
+    """Parse a recipe from a URL. Returns None if parsing fails.
+
+    Accepts an optional log callable taking a single string argument; when provided,
+    the scraper will emit diagnostic messages useful for manual testing.
+    """
+    ldata = await _scrape_recipe_ldata(url, log=log, dump_dir=dump_dir)
+
     if ldata:
         return await _get_recipe_from_ldata(conn, url, ldata, created_by)
     return None
 
 
+async def parse_recipe_from_html(conn, created_by, base_url: str, html: str, log=None) -> Optional[Recipe]:
+    """Parse a recipe from raw HTML (offline). Returns None if parsing fails.
+
+    This mirrors parse_recipe() but uses already-downloaded HTML via the offline
+    scraper entrypoint. Useful for tests/assertions against saved snapshots.
+    """
+    ldata = _scrape_recipe_ldata_from_html(html, base_url, log=log)
+    if ldata:
+        return await _get_recipe_from_ldata(conn, base_url, ldata, created_by)
+    return None
+
+
 def find_yield(recipe_ldata: dict) -> int:
     if "recipeYield" in recipe_ldata:
         yield_vals = recipe_ldata["recipeYield"]

recipes/scraping.py

@@ -1,37 +1,275 @@
 import json
-from typing import Optional, Iterable
+from typing import Optional, Iterable, List, Tuple, Dict
 
 import httpx
 from bs4 import BeautifulSoup
+import html as _html
 
-# A realistic browser header profile improves success rates against some CDNs/bot protections.
+# POLICY: Strict ld+json-only recipe extraction
+# ---------------------------------------------
+# This scraper MUST NOT perform complex HTML-based ingredient parsing.
+# Only two things are allowed when parsing HTML:
+#   1) Discover additional, more-friendly variants of the same page (e.g., amp/print)
+#   2) Locate and parse <script type="application/ld+json"> blocks that contain a Recipe
+# Do NOT extract from application/json, __NEXT_DATA__, plugin markup, microdata, or generic
+# DOM heuristics. This keeps behavior predictable and aligned with sites' structured data.
+
+# Determine brotli support to avoid unreadable responses when the runtime lacks a decoder.
+_HAS_BROTLI = False
+try:  # brotli or brotlicffi
+    import brotli as _brotli  # type: ignore
+    _HAS_BROTLI = True
+except Exception:
+    try:
+        import brotlicffi as _brotlicffi  # type: ignore
+        _HAS_BROTLI = True
+    except Exception:
+        _HAS_BROTLI = False
+
+# Base headers shared across profiles; specific profiles add UA and optional fetch headers.
 DEFAULT_HEADERS = {
-    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
-    "Accept-Language": "en-US,en;q=0.9",
-    "Accept-Encoding": "gzip, deflate, br",
+    "Accept": "text/html,application/xhtml+xml;q=0.9,*/*;q=0.8",
+    "Accept-Language": "en-US,en;q=0.8",
+    # Only advertise br when we can decode it, otherwise prefer gzip/deflate for reliability
+    "Accept-Encoding": "gzip, deflate, br" if _HAS_BROTLI else "gzip, deflate",
     "Connection": "keep-alive",
+}
+
+# Header profiles: start with an automation-forward identity, then fall back to browser-like
+HEADER_PROFILES = [
+    {
+        "name": "automation",
+        "headers": {
+            **DEFAULT_HEADERS,
+            "User-Agent": "MunchEaseRecipeBot/1.0 (+https://example.com/bot)",
+            "From": "bot@example.com",
+        },
+        "add_referer": False,
+    },
+    {
+        "name": "chrome-desktop",
+        "headers": {
+            **DEFAULT_HEADERS,
             "Upgrade-Insecure-Requests": "1",
             "Sec-Fetch-Dest": "document",
             "Sec-Fetch-Mode": "navigate",
             "Sec-Fetch-Site": "none",
             "Sec-Fetch-User": "?1",
-    # A modern desktop Chrome UA with platform tokens; not tied to any user data.
+            # Client hints
+            "sec-ch-ua": '"Chromium";v="127", "Not=A?Brand";v="24", "Google Chrome";v="127"',
+            "sec-ch-ua-mobile": "?0",
+            "sec-ch-ua-platform": '"Linux"',
             "User-Agent": (
                 "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) "
                 "Chrome/127.0.0.0 Safari/537.36"
             ),
+        },
+        "add_referer": True,
+    },
+    {
+        "name": "chrome-mobile",
+        "headers": {
+            **DEFAULT_HEADERS,
+            "Upgrade-Insecure-Requests": "1",
+            "Sec-Fetch-Dest": "document",
+            "Sec-Fetch-Mode": "navigate",
+            "Sec-Fetch-Site": "none",
+            "Sec-Fetch-User": "?1",
+            "sec-ch-ua": '"Chromium";v="127", "Not=A?Brand";v="24", "Google Chrome";v="127"',
+            "sec-ch-ua-mobile": "?1",
+            "sec-ch-ua-platform": '"Android"',
+            "User-Agent": (
+                "Mozilla/5.0 (Linux; Android 12; Pixel 5) AppleWebKit/537.36 (KHTML, like Gecko) "
+                "Chrome/127.0.0.0 Mobile Safari/537.36"
+            ),
+        },
+        "add_referer": True,
+    },
+    {
+        "name": "chrome-full",
+        "headers": {
+            **DEFAULT_HEADERS,
+            "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
+            "Upgrade-Insecure-Requests": "1",
+            "Sec-Fetch-Dest": "document",
+            "Sec-Fetch-Mode": "navigate",
+            "Sec-Fetch-Site": "none",
+            "Sec-Fetch-User": "?1",
+            "sec-ch-ua": '"Chromium";v="127", "Google Chrome";v="127", ";Not A Brand";v="99"',
+            "sec-ch-ua-mobile": "?0",
+            "sec-ch-ua-platform": '"Linux"',
+            "sec-ch-ua-platform-version": '"6.8.0"',
+            "DNT": "1",
+            "Priority": "u=0, i",
+            "User-Agent": (
+                "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) "
+                "Chrome/127.0.0.0 Safari/537.36"
+            ),
+        },
+        "add_referer": True,
+    },
+    {
+        "name": "safari-mac",
+        "headers": {
+            **DEFAULT_HEADERS,
+            "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
+            "Upgrade-Insecure-Requests": "1",
+            "User-Agent": (
+                "Mozilla/5.0 (Macintosh; Intel Mac OS X 14_0) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.0 Safari/605.1.15"
+            ),
+        },
+        "add_referer": True,
+    },
+    {
+        "name": "firefox-desktop",
+        "headers": {
+            **DEFAULT_HEADERS,
+            "User-Agent": (
+                "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:128.0) Gecko/20100101 Firefox/128.0"
+            ),
+        },
+        "add_referer": True,
+    },
+    {
+        "name": "safari-ios",
+        "headers": {
+            **DEFAULT_HEADERS,
+            "User-Agent": (
+                "Mozilla/5.0 (iPhone; CPU iPhone OS 17_0 like Mac OS X) "
+                "AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.0 Mobile/15E148 Safari/604.1"
+            ),
+        },
+        "add_referer": True,
+    },
+    {
+        "name": "edge-desktop",
+        "headers": {
+            **DEFAULT_HEADERS,
+            "Upgrade-Insecure-Requests": "1",
+            "Sec-Fetch-Dest": "document",
+            "Sec-Fetch-Mode": "navigate",
+            "Sec-Fetch-Site": "none",
+            "Sec-Fetch-User": "?1",
+            "sec-ch-ua": '"Chromium";v="127", "Not=A?Brand";v="24", "Microsoft Edge";v="127"',
+            "sec-ch-ua-mobile": "?0",
+            "sec-ch-ua-platform": '"Linux"',
+            "User-Agent": (
+                "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) "
+                "Chrome/127.0.0.0 Safari/537.36 Edg/127.0.0.0"
+            ),
+        },
+        "add_referer": True,
+    },
+]
+
+
+def looks_like_jsonld(html: str) -> bool:
+    """Lightweight check for presence of JSON-LD Recipe data in an HTML string.
+
+    We only check for ld+json tags or @context+schema.org signatures.
+    This is an optimization hint and does not parse JSON.
+    """
+    if not html:
+        return False
+    low = html.lower()
+    if "application/ld+json" in low:
+        return True
+    if "@context" in low and "schema.org" in low:
+        return True
+    return False
+
+
+def _force_safe_accept_encoding(headers: Dict[str, str]) -> Dict[str, str]:
+    """Return a copy of headers where Accept-Encoding excludes brotli.
+
+    Use this in contexts where brotli support may be missing to avoid unreadable payloads.
+    """
+    h = dict(headers)
+    h["Accept-Encoding"] = "gzip, deflate"
+    return h
+
+
+async def fetch_first_2xx_html(
+    client: httpx.AsyncClient,
+    url: str,
+    log=None,
+    safe_accept_encoding: bool = True,
+) -> Tuple[Optional[str], Dict[str, Optional[str]]]:
+    """Try header profiles against a URL and return the first 2xx HTML and metadata.
+
+    Returns (html, meta) where meta includes:
+      - status, final_url, content_type, content_encoding
+      - profile (name), request_headers (effective request headers)
+    If no profile returns 2xx, returns (None, meta_with_last_error_or_status).
+    """
+    def _log(msg: str) -> None:
+        if log:
+            try:
+                log(msg)
+            except Exception:
+                pass
+
+    from urllib.parse import urlparse
+    parsed = urlparse(url)
+    origin = f"{parsed.scheme}://{parsed.netloc}/"
+    last_meta: Dict[str, Optional[str]] = {
+        "status": None,
+        "final_url": None,
+        "content_type": None,
+        "content_encoding": None,
+        "profile": None,
+        "error": None,
+        "request_headers": None,
     }
+    last_html: Optional[str] = None
+    for prof in HEADER_PROFILES:
+        headers = dict(prof["headers"])  # copy
+        if prof.get("add_referer"):
+            headers.setdefault("Referer", origin)
+        if safe_accept_encoding:
+            headers = _force_safe_accept_encoding(headers)
+        try:
+            resp = await client.get(url, headers=headers, follow_redirects=True)
+        except Exception as e:
+            last_meta.update({"error": f"{type(e).__name__}: {e}", "profile": prof["name"]})
+            _log(f"GET[{prof['name']}] {url} -> EXC {type(e).__name__}: {e}")
+            continue
+        _log(f"GET[{prof['name']}] {url} -> {resp.status_code}")
+        meta = {
+            "status": str(resp.status_code),
+            "final_url": str(resp.url),
+            "content_type": resp.headers.get("content-type"),
+            "content_encoding": resp.headers.get("content-encoding"),
+            "profile": prof["name"],
+            "error": None,
+            "request_headers": json.dumps(headers),
+        }
+        last_meta = meta
+        if 200 <= resp.status_code < 300:
+            html = resp.text
+            last_html = html
+            # Prefer early return if JSON-LD signature seems present
+            if looks_like_jsonld(html):
+                return html, meta
+            # else keep searching other profiles for a more suitable variant
+            continue
+    return last_html, last_meta
 
 
 def _is_recipe_ldata(ldata_node) -> bool:
-    if "@type" in ldata_node:
+    """Return True when a JSON-LD node represents a Recipe.
+
+    Handles cases where @type is a string or a list (order-insensitive).
+    """
+    if "@type" not in ldata_node:
+        return False
     typ = ldata_node["@type"]
+    if isinstance(typ, str):
+        return typ.lower() == "recipe"
     if isinstance(typ, list):
-            typ = typ[0]
-
-        if isinstance(typ, str) and typ.lower() == "recipe":
+        for t in typ:
+            if isinstance(t, str) and t.lower() == "recipe":
                 return True
-
     return False
 
 
@@ -43,6 +281,7 @@ def _fallback_urls(url: str) -> Iterable[str]:
     - add `?output=amp` if no existing query
     - add `&output=amp` if query exists
     - try `/amp` path suffix if not already present
+    - try `?amp=1` and bare `?amp` which some sites honor as AMP toggles
     """
     yield url
     try:
@@ -62,6 +301,42 @@ def _fallback_urls(url: str) -> Iterable[str]:
             amp2 = urlunparse(parsed._replace(path=amp_path))
             if amp2 != url:
                 yield amp2
+        # Additional common AMP toggles
+        q_amp = dict(parse_qsl(parsed.query, keep_blank_values=True))
+        if q_amp.get("amp") != "1":
+            q_amp["amp"] = "1"
+            amp1 = urlunparse(parsed._replace(query=urlencode(q_amp, doseq=True)))
+            if amp1 != url:
+                yield amp1
+        if "amp" not in (q_amp or {}):
+            bare = urlunparse(parsed._replace(query=(parsed.query + ("&" if parsed.query else "") + "amp")))
+            if bare != url:
+                yield bare
+
+        # HTTP scheme fallbacks when original is HTTPS (helps sites with misconfigured TLS)
+        if parsed.scheme == "https":
+            http_base = urlunparse(parsed._replace(scheme="http"))
+            if http_base != url:
+                yield http_base
+            # http + output=amp
+            q2 = dict(parse_qsl(parsed.query, keep_blank_values=True))
+            if q2.get("output") != "amp":
+                q2["output"] = "amp"
+            http_amp = urlunparse(parsed._replace(scheme="http", query=urlencode(q2, doseq=True)))
+            if http_amp != url:
+                yield http_amp
+            # http + amp=1
+            q3 = dict(parse_qsl(parsed.query, keep_blank_values=True))
+            if q3.get("amp") != "1":
+                q3["amp"] = "1"
+            http_amp1 = urlunparse(parsed._replace(scheme="http", query=urlencode(q3, doseq=True)))
+            if http_amp1 != url:
+                yield http_amp1
+            # http path amp
+            if not parsed.path.endswith("/amp"):
+                http_amp_path = urlunparse(parsed._replace(scheme="http", path=parsed.path.rstrip("/") + "/amp"))
+                if http_amp_path != url:
+                    yield http_amp_path
     except Exception:
         # Be conservative if URL parsing fails
         pass
@@ -70,67 +345,415 @@ def _fallback_urls(url: str) -> Iterable[str]:
 BLOCK_STATUSES = {403, 406, 429, 460}
 
 
-async def scrape_recipe_ldata(url: str) -> Optional[dict]:
-    # Try the URL with browser-like headers and fallback strategies when blocked.
-    async with httpx.AsyncClient() as client:
-        for candidate in _fallback_urls(url):
-            # Some CDNs prefer a referer; provide same-origin referer as a harmless hint.
-            headers = dict(DEFAULT_HEADERS)
-            headers.setdefault("Referer", candidate)
-            response = await client.get(candidate, headers=headers, follow_redirects=True)
-            if response.status_code in BLOCK_STATUSES:
-                # Try next fallback
+async def scrape_recipe_ldata(url: str, log=None, dump_dir: Optional[str] = None) -> Optional[dict]:
+    """Return best-effort recipe JSON-LD (or heuristic dict) for the URL.
+
+    If 'log' is provided (callable taking a string), diagnostic messages are emitted
+    during scraping. No environment toggles are used; behavior matches production.
+    """
+
+    def _log(msg: str) -> None:
+        if log:
+            try:
+                log(msg)
+            except Exception:
+                pass
+
+    try:
+        async with httpx.AsyncClient(http2=True, timeout=20.0) as client:
+            visited: set[str] = set()
+
+            # Queue of URLs to try, seeded with simple fallbacks; we'll append amp/print variants we discover.
+            to_try: List[str] = list(_fallback_urls(url))
+            while to_try:
+                candidate = to_try.pop(0)
+                if candidate in visited:
                     continue
-            if response.status_code >= 300:
-                # Try next fallback on non-2xx
+                visited.add(candidate)
+                # Try each header profile and attempt extraction per profile
+                any_2xx = False
+                from urllib.parse import urlparse, urlunparse
+                parsed = urlparse(candidate)
+                origin = f"{parsed.scheme}://{parsed.netloc}/"
+                for prof in HEADER_PROFILES:
+                    prof_name = prof["name"]
+                    headers = dict(prof["headers"])  # copy
+                    if prof.get("add_referer"):
+                        headers.setdefault("Referer", origin)
+                    try:
+                        resp = await client.get(candidate, headers=headers, follow_redirects=True)
+                    except Exception as e:
+                        _log(f"GET[{prof_name}] {candidate} -> EXC {type(e).__name__}: {e}")
+                        continue
+                    _log(f"GET[{prof_name}] {candidate} -> {resp.status_code}")
+                    if resp.status_code in BLOCK_STATUSES or resp.status_code >= 300:
+                        continue
+                    any_2xx = True
+                    html = resp.text
+                    soup = BeautifulSoup(html, "html.parser")
+                    # Optional debug dump
+                    if dump_dir:
+                        try:
+                            _dump_response(dump_dir, candidate, prof_name, resp)
+                        except Exception as e:
+                            _log(f"dump error: {type(e).__name__}: {e}")
+                    # Detect common bot protection pages early
+                    if (soup.title and "just a moment" in _extract_text(soup.title).lower()) or "challenge-platform" in html:
+                        _log("Detected Cloudflare-like challenge; trying next profile")
+                        continue
+                    # FRIENDLY DISCOVERY: follow declared amphtml and obvious print links
+                    try:
+                        # rel="amphtml"
+                        amp_link = None
+                        for link in soup.find_all("link"):
+                            rel = link.get("rel")
+                            href = link.get("href")
+                            if not href:
+                                continue
+                            if isinstance(rel, list) and any(r.lower() == "amphtml" for r in rel):
+                                amp_link = href
+                                break
+                            if isinstance(rel, str) and rel.lower() == "amphtml":
+                                amp_link = href
+                                break
+                        if amp_link:
+                            from urllib.parse import urljoin
+                            amp_abs = urljoin(candidate, amp_link)
+                            if amp_abs not in visited and amp_abs not in to_try:
+                                to_try.append(amp_abs)
+                                _log(f"discovered amphtml link: {amp_abs}")
+
+                        # print links (plugin or generic)
+                        from urllib.parse import urljoin, parse_qsl, urlencode
+                        for a in soup.find_all("a", href=True):
+                            href = a["href"]
+                            low = href.lower()
+                            if any(k in low for k in ["print", "wprm-print", "tasty-recipes-print", "/print/"]):
+                                absu = urljoin(candidate, href)
+                                if absu not in visited and absu not in to_try:
+                                    to_try.append(absu)
+                        # linked JSON-LD files
+                        for link in soup.find_all("link"):
+                            rel = link.get("rel")
+                            typ = (link.get("type") or link.get("as") or "").lower()
+                            href = link.get("href")
+                            if not href:
+                                continue
+                            if (isinstance(rel, list) and any(r.lower() == "alternate" for r in rel)) or (
+                                isinstance(rel, str) and rel.lower() == "alternate"
+                            ):
+                                if "ld+json" in typ:
+                                    from urllib.parse import urljoin
+                                    absu = urljoin(candidate, href)
+                                    if absu not in visited and absu not in to_try:
+                                        to_try.append(absu)
+                                        _log(f"discovered linked ld+json: {absu}")
+                        # query param prints: add print=1 once if missing
+                        parsed_url = parsed
+                        q_items = list(parse_qsl(parsed_url.query, keep_blank_values=True))
+                        has_print = any(k == "print" for k, _ in q_items)
+                        if not has_print:
+                            q_items.append(("print", "1"))
+                            print_url = urlunparse(parsed_url._replace(query=urlencode(q_items, doseq=True)))
+                            if print_url not in visited and print_url not in to_try:
+                                to_try.append(print_url)
+                                _log(f"queued print variant: {print_url}")
+                    except Exception as e:
+                        _log(f"discovery error: {type(e).__name__}: {e}")
+
+                    found = _extract_ldata_from_soup(soup, candidate, _log)
+                    if found:
+                        return found
+
+                if not any_2xx:
+                    _log("All header profiles blocked or non-2xx; trying next fallback")
                     continue
 
-            # Extract the recipe ld+json data from this candidate
-            soup = BeautifulSoup(response.text, "html.parser")
-            for ld in soup.find_all("script", type="application/ld+json"):
+            _log("No recipe data found after all fallbacks")
+            return None
+    except Exception as e:
+        _log(f"EXC during scraping: {type(e).__name__}: {e}")
+        return None
+
+
+def _extract_ldata_from_soup(soup: BeautifulSoup, candidate: str, log=None) -> Optional[dict]:
+    def _log(msg: str) -> None:
+        if log:
             try:
-                    data = json.loads(ld.text)
-                    # _dump_json_data_to_log(data)
-                    if _is_recipe_ldata(data):
+                log(msg)
+            except Exception:
+                pass
+
+    import re as _re
+
+    def _strip_to_json_candidate(text: str) -> str:
+        # Remove HTML/JS comment wrappers and CDATA markers
+        t = text.strip()
+        t = _re.sub(r"<!--|-->", "", t)
+        t = _re.sub(r"/\*.*?\*/", "", t, flags=_re.S)
+        t = _re.sub(r"(^|\s)//.*$", "", t, flags=_re.M)
+        # Trim to outermost JSON-like braces/brackets
+        first_brace = t.find("{")
+        first_brack = t.find("[")
+        starts = [i for i in [first_brace, first_brack] if i != -1]
+        if not starts:
+            return t
+        start = min(starts)
+        last_brace = t.rfind("}")
+        last_brack = t.rfind("]")
+        ends = [i for i in [last_brace, last_brack] if i != -1]
+        if not ends:
+            return t
+        end = max(ends)
+        return t[start : end + 1]
+
+    def _remove_trailing_commas(s: str) -> str:
+        prev = None
+        curr = s
+        # Iteratively remove trailing commas before } or ]
+        for _ in range(3):  # limit passes
+            prev = curr
+            curr = _re.sub(r",\s*([}\]])", r"\1", curr)
+            if curr == prev:
+                break
+        return curr
+
+    def _parse_json_lenient(text: str) -> Optional[object]:
+        # Try strict first
+        try:
+            return json.loads(text)
+        except Exception:
+            pass
+        # Clean wrappers and comments
+        t = _strip_to_json_candidate(text)
+        # Sometimes multiple roots are concatenated; try to form a list conservatively
+        if t.count("{") > 1 and "}\n{" in t:
+            parts = [p for p in t.split("\n") if p.strip()]
+            maybe = "[" + ",".join(parts) + "]"
+            try:
+                return json.loads(maybe)
+            except Exception:
+                pass
+        # Remove trailing commas
+        t2 = _remove_trailing_commas(t)
+        try:
+            return json.loads(t2)
+        except Exception:
+            pass
+        # As a last resort, if no double quotes exist but single quotes do, try naive conversion
+        if '"' not in t2 and "'" in t2:
+            t3 = t2.replace("'", '"')
+            try:
+                return json.loads(t3)
+            except Exception:
+                pass
+        return None
+    # 1) JSON-LD scripts (strict mode: only source of truth)
+    ld_nodes = soup.find_all("script", attrs={"type": _re.compile(r"ld\+json", _re.I)})
+    _log(f"Found {len(ld_nodes)} ld+json scripts")
+    for idx, ld in enumerate(ld_nodes):
+        try:
+            text = ld.text.strip()
+            data = None
+            try:
+                data = json.loads(text)
+            except json.decoder.JSONDecodeError:
+                # Try lenient parsing strategies
+                data = _parse_json_lenient(text)
+                if data is not None:
+                    _log(f"ld[{idx}]: parsed with lenient JSON repair")
+                else:
+                    raise
+
+            if isinstance(data, dict) and _is_recipe_ldata(data):
+                _log(f"ld[{idx}]: direct @type Recipe found")
                 return data
 
-                    if "@graph" in data:
-                        for item in data["@graph"]:
+            if isinstance(data, dict) and "@graph" in data:
+                for gidx, item in enumerate(data["@graph"]):
                     if _is_recipe_ldata(item):
+                        _log(f"ld[{idx}]: @graph item {gidx} is Recipe")
                         return item
 
             if isinstance(data, list):
-                        for item in data:
+                for lidx, item in enumerate(data):
                     if _is_recipe_ldata(item):
+                        _log(f"ld[{idx}]: list item {lidx} is Recipe")
                         return item
 
-                except (json.decoder.JSONDecodeError, KeyError):
-                    pass
+        except (json.decoder.JSONDecodeError, KeyError) as e:
+            _log(f"ld[{idx}]: JSON parse/Key error: {type(e).__name__}: {e}")
 
+    # 1b) JSON-LD within <noscript> blocks
+    try:
+        ns_count = 0
+        for ns in soup.find_all("noscript"):
+            payload = ns.get_text(strip=False) or ns.decode_contents(formatter="html") or ""
+            if not payload:
+                continue
+            nsoup = BeautifulSoup(payload, "html.parser")
+            nodes = nsoup.find_all("script", attrs={"type": _re.compile(r"ld\+json", _re.I)})
+            ns_count += len(nodes)
+            for nidx, node in enumerate(nodes):
+                try:
+                    text = (node.text or node.get_text() or "").strip()
+                    if not text:
+                        continue
+                    data = _parse_json_lenient(text) or {}
+                    if isinstance(data, dict) and _is_recipe_ldata(data):
+                        _log(f"noscript.ld[{nidx}]: direct Recipe found")
+                        return data
+                    if isinstance(data, dict) and "@graph" in data:
+                        for gidx, item in enumerate(data.get("@graph") or []):
+                            if _is_recipe_ldata(item):
+                                _log(f"noscript.ld[{nidx}]: @graph item {gidx} is Recipe")
+                                return item
+                    if isinstance(data, list):
+                        for lidx, item in enumerate(data):
+                            if _is_recipe_ldata(item):
+                                _log(f"noscript.ld[{nidx}]: list item {lidx} is Recipe")
+                                return item
+                except Exception as e:
+                    _log(f"noscript.ld[{nidx}]: parse error: {type(e).__name__}: {e}")
+        if ns_count:
+            _log(f"noscript: scanned {ns_count} ld+json scripts")
+    except Exception as e:
+        _log(f"noscript scan error: {type(e).__name__}: {e}")
+
+    # 1c) JSON-LD by content signature in any script or HTML when type is missing
+    def _extract_json_objects_around(text: str) -> List[str]:
+        out: List[str] = []
+        if not text:
+            return out
+        t = _html.unescape(text)
+        key = "@context"
+        pos = 0
+        while True:
+            at = t.find(key, pos)
+            if at == -1:
+                break
+            # find preceding '{'
+            lb = t.rfind("{", 0, at)
+            if lb == -1:
+                pos = at + len(key)
+                continue
+            i = lb
+            depth = 0
+            in_str = False
+            esc = False
+            end = -1
+            while i < len(t):
+                ch = t[i]
+                if in_str:
+                    if esc:
+                        esc = False
+                    elif ch == "\\":
+                        esc = True
+                    elif ch == '"':
+                        in_str = False
+                else:
+                    if ch == '"':
+                        in_str = True
+                    elif ch == '{':
+                        depth += 1
+                    elif ch == '}':
+                        depth -= 1
+                        if depth == 0:
+                            end = i
+                            break
+                i += 1
+            if end != -1:
+                out.append(t[lb:end+1])
+                pos = end + 1
+            else:
+                pos = at + len(key)
+        return out
+
+    def _try_parse_jsonld_candidates(cands: List[str], origin: str) -> Optional[dict]:
+        for cidx, cand in enumerate(cands):
+            data = _parse_json_lenient(cand)
+            if data is None:
+                continue
+            if isinstance(data, dict) and _is_recipe_ldata(data):
+                _log(f"{origin}[{cidx}]: Recipe found")
+                return data
+            if isinstance(data, dict) and "@graph" in data:
+                for gidx, item in enumerate(data.get("@graph") or []):
+                    if _is_recipe_ldata(item):
+                        _log(f"{origin}[{cidx}]: @graph item {gidx} is Recipe")
+                        return item
+            if isinstance(data, list):
+                for lidx, item in enumerate(data):
+                    if _is_recipe_ldata(item):
+                        _log(f"{origin}[{cidx}]: list item {lidx} is Recipe")
+                        return item
         return None
 
-    # Fallback return to satisfy static analysis
+    # Scan all <script> tags for JSON-LD signature when nothing found
+    try:
+        texts: List[str] = []
+        for sc in soup.find_all("script"):
+            txt = (sc.string or sc.get_text() or "").strip()
+            if not txt:
+                continue
+            if "@context" not in txt or "schema.org" not in txt:
+                continue
+            texts.append(txt)
+        if texts:
+            cands: List[str] = []
+            for t in texts:
+                cands.extend(_extract_json_objects_around(t))
+            found = _try_parse_jsonld_candidates(cands, origin="script-scan")
+            if found:
+                return found
+    except Exception as e:
+        _log(f"script content scan error: {type(e).__name__}: {e}")
+
+    # Finally, scan the full HTML string
+    try:
+        html_str = str(soup)
+        if "@context" in html_str and "schema.org" in html_str:
+            cands = _extract_json_objects_around(html_str)
+            found = _try_parse_jsonld_candidates(cands, origin="html-scan")
+            if found:
+                return found
+    except Exception as e:
+        _log(f"html scan error: {type(e).__name__}: {e}")
     return None
 
 
-def _dump_json_data_to_log(data: dict) -> str:
+def _dump_response(dump_dir: str, url: str, prof: str, resp: httpx.Response) -> None:
     import os
-    import re
+    from urllib.parse import urlparse
+    os.makedirs(dump_dir, exist_ok=True)
+    p = urlparse(url)
+    base = f"{p.netloc}{p.path}"
+    if not base or base.endswith("/"):
+        base += "index"
+    safe = base.replace("/", "_").replace("?", "_").replace("&", "_")
+    fname = f"{safe}__{prof}__{resp.status_code}.html"
+    meta = f"{safe}__{prof}__{resp.status_code}.meta"
+    fpath = os.path.join(dump_dir, fname)
+    mpath = os.path.join(dump_dir, meta)
+    with open(fpath, "w", encoding=resp.encoding or "utf-8", errors="ignore") as f:
+        f.write(resp.text)
+    with open(mpath, "w", encoding="utf-8") as f:
+        f.write(f"URL: {url}\n")
+        f.write(f"Profile: {prof}\n")
+        f.write(f"Status: {resp.status_code}\n")
+        f.write(f"Content-Type: {resp.headers.get('content-type','')}\n")
 
-    dir = "./data/dump"
-    if not os.path.exists(dir):
-        os.makedirs(dir)
 
-    prefix = "ldata_"
-    suffix = ".json"
-    file_ids = [
-        int(re.findall(r"\d+", f)[0])
-        for f in os.listdir(dir)
-        if re.match(prefix + r"\d+" + suffix, f)
-    ]
-    id = max(file_ids) + 1 if file_ids else 0
-    filename = f"{prefix}{id}{suffix}"
-    full_path = os.path.join(dir, filename)
-    with open(full_path, "w") as f:
-        json.dump(data, f, indent=4)
-    return full_path
+def scrape_recipe_ldata_from_html(html: str, base_url: str, log=None) -> Optional[dict]:
+    """Extract recipe JSON-LD from raw HTML (strict ld+json-only).
+
+    base_url is used for relative URL resolution and as a fallback name/link context.
+    """
+    soup = BeautifulSoup(html, "html.parser")
+    return _extract_ldata_from_soup(soup, base_url, log)
+
+
+
+def _extract_text(el) -> str:
+    return " ".join(el.get_text(" ", strip=True).split()) if el else ""
+

scripts/generate_expected_from_snapshots.py

@@ -0,0 +1,64 @@
+"""
+Generate expected Recipe models from saved HTML snapshots (offline).
+
+Reads HTML files from tests/sample_files/recipes and writes expected
+Recipe JSONs into tests/sample_files/recipes/expected with matching slugs.
+
+Run:
+    python -m scripts.generate_expected_from_snapshots
+"""
+
+import json
+from pathlib import Path
+from typing import Optional
+from types import SimpleNamespace
+
+from db import connect, create
+from recipes import parse_recipe_from_html
+
+
+SNAP_DIR = Path("tests/sample_files/recipes")
+
+
+def _write_expected(slug: str, recipe) -> None:
+    # Write <slug>.recipe.json at SNAP_DIR root (canonical)
+    new_out = SNAP_DIR / f"{slug}.recipe.json"
+    if recipe is None:
+        new_out.write_text("null")
+    else:
+        new_out.write_text(recipe.model_dump_json(by_alias=True, indent=2))
+
+
+async def _run() -> None:
+    html_files = sorted(p for p in SNAP_DIR.glob("*.html"))
+    if not html_files:
+        print("no HTML snapshots found; run scripts.save_recipe_pages first")
+        return
+
+    # Minimal created_by object the parser expects (id, display_name)
+    created_by = SimpleNamespace(id=1, display_name="Snapshot Generator")
+    conn = await connect()
+    await create(conn)
+    try:
+        for p in html_files:
+            slug = p.stem
+            url = None
+            meta_path = SNAP_DIR / f"{slug}.meta.json"
+            if meta_path.exists():
+                try:
+                    meta = json.loads(meta_path.read_text())
+                    url = meta.get("final_url") or meta.get("url")
+                except Exception:
+                    pass
+            base_url = url or slug
+            html = p.read_text()
+            recipe = await parse_recipe_from_html(conn, created_by, base_url, html)
+            _write_expected(slug, recipe)
+            print(f"wrote expected for {slug}: {'ok' if recipe else 'none'}")
+    finally:
+        await conn.close()
+
+
+if __name__ == "__main__":
+    import asyncio
+    asyncio.run(_run())

scripts/manual_parse_recipes.py

@@ -0,0 +1,133 @@
+"""
+Manual harness to exercise recipes.parse_recipe against specific public URLs.
+
+This WILL NOT run in CI or with `make test` — run it explicitly:
+
+    make install  # first time
+    python -m scripts.manual_parse_recipes
+
+Notes:
+- Creates/bootstraps a local SQLite DB (./data/doof.sqlite) for lookups.
+- Does not persist recipes; it just prints parsed results.
+- Network access required to fetch pages.
+"""
+
+import asyncio
+from types import SimpleNamespace
+from typing import Any, Dict, List
+
+from db import connect, create
+from recipes import parse_recipe
+from scripts.recipe_urls import URLS
+import os
+
+# This harness streams results as they complete and keeps per-URL logs grouped.
+
+
+def _fmt(v) -> str:
+    if v is None:
+        return "-"
+    return str(v)
+
+
+def _print_recipe(r) -> None:
+    print("\n=== Parsed Recipe ===")
+    print(f"name:        {_fmt(getattr(r, 'name', None))}")
+    print(f"serves:      {_fmt(getattr(r, 'serves', None))}")
+    print(f"link:        {_fmt(getattr(r, 'link', None))}")
+    imgs = getattr(r, "image_urls", []) or []
+    if imgs:
+        print(f"images[0]:   {imgs[0]}")
+    ings = getattr(r, "ingredients", []) or []
+    print(f"ingredients: {len(ings)}")
+    for ing in ings[:10]:
+        # Each is an Ingredient model with line/name/quantity/unit
+        n = getattr(ing, "name", "")
+        q = getattr(ing, "quantity", "")
+        u = getattr(ing, "unit", "")
+        line = getattr(ing, "line", "")
+        print(f"  - {n} ({q} {u}) :: {line}")
+
+
+async def _parse_one(conn, url: str, dump_dir: str | None) -> Dict[str, Any]:
+    # Minimal created_by object the parser expects (id, display_name)
+    created_by = SimpleNamespace(id=1, display_name="Manual Tester")
+    logs: List[str] = []
+    def _log(msg: str):
+        # Collect per-URL logs; we'll print them later in a grouped section
+        logs.append(msg)
+    try:
+        r = await parse_recipe(conn, created_by, url, log=_log, dump_dir=dump_dir)
+    except Exception as e:
+        return {"url": url, "recipe": None, "logs": logs, "error": str(e)}
+    return {"url": url, "recipe": r, "logs": logs, "error": None}
+
+
+async def main() -> None:
+    conn = await connect()
+    # Ensure schema exists for product matching in ingredient parsing
+    await create(conn)
+    try:
+        dump_dir = os.environ.get("SCRAPER_DUMP_DIR") or None
+        concurrency = min(8, len(URLS))
+        sem = asyncio.Semaphore(concurrency)
+        # Consume all available slots and stagger release initially to avoid bursts
+        for _ in range(concurrency):
+            await sem.acquire()
+        # Release all slots with slight delays to avoid thundering herd
+        for i in range(concurrency):
+            asyncio.get_event_loop().call_later(i * 0.5, sem.release)
+
+        async def _worker(u: str):
+            async with sem:
+                return await _parse_one(conn, u, dump_dir)
+
+        start_time = asyncio.get_event_loop().time()
+        tasks = [asyncio.create_task(_worker(url)) for url in URLS]
+
+
+        # Stream results as they arrive; print per-URL blocks to avoid jumbled output
+        totals = {"total": 0, "parsed": 0, "errors": 0, "no_recipe": 0}
+        for task in asyncio.as_completed(tasks):
+            res = await task
+            url = res["url"]
+            totals["total"] += 1
+            print("\n==============================")
+            print(f"Result: {url}")
+            print("==============================")
+            if res.get("error"):
+                totals["errors"] += 1
+                print(f"ERROR: failed to parse {url}:")
+                print(f"  {res['error']}")
+            r = res.get("recipe")
+            if r is None and not res.get("error"):
+                totals["no_recipe"] += 1
+                print(f"WARN: no recipe data found at {url}")
+            if r is not None:
+                totals["parsed"] += 1
+                _print_recipe(r)
+
+            # Logs
+            logs: List[str] = res.get("logs") or []
+            # Print logs for failures only to reduce noise; clip to a reasonable limit
+            if (r is None or res.get("error")) and logs:
+                clip = logs[:200]
+                print("-- logs --")
+                for line in clip:
+                    print(f"  dbg: {line}")
+                if len(logs) > len(clip):
+                    print(f"  .. ({len(logs) - len(clip)} more lines clipped) ..")
+
+            print("\n=== Summary ===")
+            print(f"Total URLs processed: {totals['total']}")
+            print(f"Successfully parsed:  {totals['parsed']}")
+            print(f"Errors:               {totals['errors']}")
+            print(f"No recipe found:      {totals['no_recipe']}")
+            print(f"Time elapsed: {asyncio.get_event_loop().time() - start_time:.2f} seconds")
+            print(f"Remaining URLs:       {len(URLS) - totals['total']}")
+    finally:
+        await conn.close()
+
+
+if __name__ == "__main__":
+    asyncio.run(main())

scripts/recipe_urls.py

@@ -0,0 +1,37 @@
+"""
+Central list of recipe URLs used by manual harnesses and snapshot tools.
+
+Keep this list focused on stable, public pages across a variety of sites.
+"""
+
+URLS = [
+    # ld+json: YES (Recipe present under @graph)
+    "https://cookieandkate.com/favorite-fried-eggs-recipe/",
+    # ld+json: UNKNOWN (blocked/SSL issues in our environment)
+    "https://www.taste.com.au/recipes/classic-chewy-brownie/f0960a83-1fa9-4e3e-b616-a995182613e0",
+    # ld+json: PRESENT but MALFORMED (first ld+json is Recipe but not valid JSON)
+    "https://www.inspiredtaste.net/24412/cocoa-brownies-recipe/",
+    # ld+json: YES (Recipe present)
+    "https://www.simplyrecipes.com/recipes/french_toast/",
+    # Additional diverse sources
+    # ld+json: YES (Recipe present)
+    "https://pinchofyum.com/lemon-chicken-piccata-with-grilled-bread",
+    # ld+json: YES (Recipe present)
+    "https://www.bbcgoodfood.com/recipes/chicken-tikka-masala",
+    # ld+json: YES (Recipe present)
+    "https://www.recipetineats.com/beef-stroganoff/",
+    # ld+json: YES (Recipe present)
+    "https://sallysbakingaddiction.com/triple-chocolate-layer-cake/",
+    # ld+json: YES (Recipe present)
+    "https://downshiftology.com/recipes/shakshuka/",
+    # ld+json: UNKNOWN (403 blocked by CDN)
+    "https://damndelicious.net/2025/08/01/corn-salsa/",
+    # ld+json: UNKNOWN (403 blocked by CDN)
+    "https://www.thekitchn.com/chicken-tamale-pie-23752740",
+    # ld+json: YES (Recipe present)
+    "https://www.simplyrecipes.com/recipes/perfect_guacamole/",
+    # ld+json: YES (Recipe present)
+    "https://www.kingarthurbaking.com/recipes/banana-bread-recipe",
+    # ld+json: YES (Recipe present; multiple ld+json, only the first has Recipe)
+    "https://tasty.co/recipe/one-pot-garlic-parmesan-pasta",
+]

scripts/save_recipe_pages.py

@@ -0,0 +1,122 @@
+"""
+Snapshot recipe pages locally for offline inspection and future parsing tests.
+
+Run explicitly (not part of CI by default):
+
+    python -m scripts.save_recipe_pages
+
+This will download each URL in scripts/recipe_urls.py and save into:
+    tests/sample_files/recipes/
+
+For each URL, it writes:
+  - <slug>.html      The raw page HTML
+  - <slug>.meta.json JSON metadata (url, fetched_at, status, final_url, error)
+
+Notes:
+  - Uses the same browser-like headers as our scraper.
+  - Respects redirects; stores final_url.
+  - Skips writing HTML on hard errors but still writes a .meta.json with the error.
+  - Creates parent folders as needed.
+"""
+
+import asyncio
+import json
+import os
+import re
+from datetime import datetime, timezone
+from pathlib import Path
+from typing import Dict, Tuple
+
+import httpx
+
+from scripts.recipe_urls import URLS
+from recipes.scraping import DEFAULT_HEADERS, HEADER_PROFILES, _fallback_urls, fetch_first_2xx_html, looks_like_jsonld
+
+
+OUT_DIR = Path("tests/sample_files/recipes")
+
+
+def _safe_slug_from_url(url: str) -> str:
+    # Keep domain and last path segment(s) for readability; replace non-word with '-'
+    from urllib.parse import urlparse
+    p = urlparse(url)
+    host = p.netloc.replace(":", "-")
+    path = p.path.rstrip("/")
+    if not path:
+        seg = "index"
+    else:
+        # use last two segments if available to avoid collisions like /recipe/ vs /recipe-2/
+        parts = [s for s in path.split("/") if s]
+        seg = "-".join(parts[-2:]) if len(parts) >= 2 else parts[-1]
+    raw = f"{host}-{seg}".lower()
+    slug = re.sub(r"[^a-z0-9._-]+", "-", raw).strip("-")
+    return slug or "page"
+
+
+async def _fetch_best_html(client: httpx.AsyncClient, url: str) -> Tuple[str, Dict]:
+    meta: Dict = {
+        "url": url,
+        "fetched_at": datetime.now(timezone.utc).isoformat(),
+        "status": None,
+        "final_url": None,
+        "error": None,
+        "profile": None,
+        "content_type": None,
+        "content_encoding": None,
+        "candidate": None,
+    }
+    visited: set[str] = set()
+    candidates = list(_fallback_urls(url))
+    for cand in candidates:
+        if cand in visited:
+            continue
+        visited.add(cand)
+        html, prof_meta = await fetch_first_2xx_html(client, cand, safe_accept_encoding=True)
+        # Merge selected metadata
+        meta.update({
+            "status": int(prof_meta.get("status") or 0) or None,
+            "final_url": prof_meta.get("final_url"),
+            "content_type": prof_meta.get("content_type"),
+            "content_encoding": prof_meta.get("content_encoding"),
+            "profile": prof_meta.get("profile"),
+            "request_headers": prof_meta.get("request_headers"),
+            "candidate": cand,
+            "error": prof_meta.get("error"),
+        })
+        if html and looks_like_jsonld(html):
+            return html, meta
+        # If html is present, keep as a fallback in case later candidates fail
+        if html:
+            fallback_html = html
+            fallback_meta = dict(meta)
+            continue
+    # Nothing succeeded; return empty payload with last meta
+    return "", meta
+
+
+async def main() -> None:
+    OUT_DIR.mkdir(parents=True, exist_ok=True)
+    async with httpx.AsyncClient(http2=True) as client:
+        sem = asyncio.Semaphore(6)
+
+        async def worker(u: str):
+            async with sem:
+                html, meta = await _fetch_best_html(client, u)
+                slug = _safe_slug_from_url(u)
+                html_path = OUT_DIR / f"{slug}.html"
+                meta_path = OUT_DIR / f"{slug}.meta.json"
+                try:
+                    # Always write metadata
+                    meta_path.write_text(json.dumps(meta, indent=2))
+                    # Only persist HTML on successful fetch
+                    if meta.get("status") and 200 <= int(meta["status"]) < 300 and html:
+                        html_path.write_text(html)
+                    print(f"saved: {u} -> {html_path.name} (status={meta.get('status')}, error={meta.get('error')})")
+                except Exception as e:
+                    print(f"failed to save for {u}: {type(e).__name__}: {e}")
+
+        await asyncio.gather(*(worker(u) for u in URLS))
+
+
+if __name__ == "__main__":
+    asyncio.run(main())

settings.py

@@ -21,7 +21,7 @@ class Settings:
     prod: bool = os.environ.get("DOOF_PROD", "false").lower() in {"1", "true", "yes"}
 
     # Frontend dev server for reverse proxy in non-prod
-    frontend_dev_url: str = os.environ.get("FRONTEND_DEV_URL", "http://localhost:8080/")
+    frontend_dev_url: str = os.environ.get("FRONTEND_DEV_URL", "http://localhost:8000/")
 
     # JWT settings
     jwt_issuer: str = os.environ.get("DOOF_JWT_ISSUER", "doof-backend")

tests/sample_files/recipes/README.md

@@ -0,0 +1,17 @@
+# Recipe page snapshots
+
+This folder contains raw HTML snapshots and metadata for public recipe pages, saved for offline inspection and to support unit tests without hitting third-party sites.
+
+Generated by:
+- `python -m scripts.save_recipe_pages` (downloads pages listed in `scripts/recipe_urls.py`)
+- `python -m scripts.generate_expected_from_snapshots` (parses HTML into structured recipe JSON)
+
+File layout per URL:
+- `<slug>.html` — raw page HTML (only for successful 2xx responses)
+- `<slug>.meta.json` — metadata with original URL, final URL, status code, fetch time, and any error
+- `<slug>.recipe.json` — expected parsed Recipe model (Pydantic-serialized, by_alias=True). This is the canonical expectation file used by offline tests.
+
+Notes:
+- Some domains may block requests (e.g., 403) or fail TLS verification. In those cases, only the `.meta.json` is written.
+- Keep `scripts/recipe_urls.py` curated to stable, public URLs to minimize churn.
+

tests/sample_files/recipes/cheese-omelette.recipe.json

@@ -0,0 +1,91 @@
+{
+  "id": -1,
+  "name": "Cheese Omelette",
+  "link": "cheese-omelette",
+  "serves": 1,
+  "imageUrls": [
+    "https://www.allrecipes.com/thmb/JS43mD2rA6_cCs9eTlXNGRHT5oQ=/1500x0/filters:no_upscale():max_bytes(150000):strip_icc()/5143634-ac5ad80b28f44c53bd0fd6d570f61f0d.jpg"
+  ],
+  "ingredients": [
+    {
+      "id": -1,
+      "name": "eggs",
+      "line": "3 large eggs",
+      "unit": "Items",
+      "quantity": 3.0,
+      "preparation": "",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "milk",
+      "line": "1 tablespoon milk, or as needed",
+      "unit": "Tablespoon",
+      "quantity": 1.0,
+      "preparation": "",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "salt and white pepper",
+      "line": "salt and freshly ground white pepper to taste",
+      "unit": "Items",
+      "quantity": 1.0,
+      "preparation": "freshly ground",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "butter",
+      "line": "2 tablespoons butter",
+      "unit": "Tablespoon",
+      "quantity": 2.0,
+      "preparation": "",
+      "productId": 4,
+      "recipeId": null,
+      "mealId": null,
+      "product": {
+        "id": 4,
+        "productId": "712251",
+        "shopCode": "woolworths",
+        "link": "https://www.woolworths.com.au/shop/productdetails/712251/western-star-unsalted-butter-chef-s-choice",
+        "name": "Western Star Unsalted Butter Chef's Choice",
+        "quantity": 500,
+        "unit": "g",
+        "imgSmall": "https://cdn0.woolworths.media/content/wowproductimages/small/712251.jpg",
+        "imgLarge": "https://cdn0.woolworths.media/content/wowproductimages/large/712251.jpg"
+      }
+    },
+    {
+      "id": -1,
+      "name": "Emmentaler cheese",
+      "line": "0.25 cup shredded Emmentaler cheese",
+      "unit": "Cup",
+      "quantity": 0.25,
+      "preparation": "shredded",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    }
+  ],
+  "basedOnRecipe": null,
+  "dateCreated": "2025-11-04T18:24:29.300361+11:00",
+  "createdById": 1,
+  "createdBy": {
+    "id": 1,
+    "displayName": "Snapshot Generator"
+  },
+  "dateHidden": null,
+  "hiddenById": null,
+  "hiddenBy": null
+}

tests/sample_files/recipes/cookieandkate.com-favorite-fried-eggs-recipe.meta.json

@@ -0,0 +1,12 @@
+{
+  "url": "https://cookieandkate.com/favorite-fried-eggs-recipe/",
+  "fetched_at": "2025-11-04T07:23:56.579020+00:00",
+  "status": 200,
+  "final_url": "https://cookieandkate.com/favorite-fried-eggs-recipe/",
+  "error": null,
+  "profile": "automation",
+  "content_type": "text/html; charset=UTF-8",
+  "content_encoding": "gzip",
+  "candidate": "https://cookieandkate.com/favorite-fried-eggs-recipe/",
+  "request_headers": "{\"Accept\": \"text/html,application/xhtml+xml;q=0.9,*/*;q=0.8\", \"Accept-Language\": \"en-US,en;q=0.8\", \"Accept-Encoding\": \"gzip, deflate\", \"Connection\": \"keep-alive\", \"User-Agent\": \"MunchEaseRecipeBot/1.0 (+https://example.com/bot)\", \"From\": \"bot@example.com\"}"
+}

tests/sample_files/recipes/cookieandkate.com-favorite-fried-eggs-recipe.recipe.json

@@ -0,0 +1,48 @@
+{
+  "id": -1,
+  "name": "Favorite Fried Eggs",
+  "link": "https://cookieandkate.com/favorite-fried-eggs-recipe/",
+  "serves": 1,
+  "imageUrls": [
+    "https://cookieandkate.com/images/2018/09/crispy-fried-egg-recipe-225x225.jpg",
+    "https://cookieandkate.com/images/2018/09/crispy-fried-egg-recipe-260x195.jpg",
+    "https://cookieandkate.com/images/2018/09/crispy-fried-egg-recipe-320x180.jpg",
+    "https://cookieandkate.com/images/2018/09/crispy-fried-egg-recipe.jpg"
+  ],
+  "ingredients": [
+    {
+      "id": -1,
+      "name": "extra-virgin olive oil",
+      "line": "1 tablespoon extra-virgin olive oil",
+      "unit": "Tablespoon",
+      "quantity": 1.0,
+      "preparation": "",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "egg",
+      "line": "1 egg",
+      "unit": "Items",
+      "quantity": 1.0,
+      "preparation": "",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    }
+  ],
+  "basedOnRecipe": null,
+  "dateCreated": "2025-11-04T18:24:29.367540+11:00",
+  "createdById": 1,
+  "createdBy": {
+    "id": 1,
+    "displayName": "Snapshot Generator"
+  },
+  "dateHidden": null,
+  "hiddenById": null,
+  "hiddenBy": null
+}

tests/sample_files/recipes/damndelicious.net-01-corn-salsa.meta.json

@@ -0,0 +1,12 @@
+{
+  "url": "https://damndelicious.net/2025/08/01/corn-salsa/",
+  "fetched_at": "2025-11-04T07:23:56.680123+00:00",
+  "status": 403,
+  "final_url": "https://damndelicious.net/2025/08/01/corn-salsa/amp",
+  "error": null,
+  "profile": "edge-desktop",
+  "content_type": "text/html; charset=UTF-8",
+  "content_encoding": "gzip",
+  "candidate": "http://damndelicious.net/2025/08/01/corn-salsa/amp",
+  "request_headers": "{\"Accept\": \"text/html,application/xhtml+xml;q=0.9,*/*;q=0.8\", \"Accept-Language\": \"en-US,en;q=0.8\", \"Accept-Encoding\": \"gzip, deflate\", \"Connection\": \"keep-alive\", \"Upgrade-Insecure-Requests\": \"1\", \"Sec-Fetch-Dest\": \"document\", \"Sec-Fetch-Mode\": \"navigate\", \"Sec-Fetch-Site\": \"none\", \"Sec-Fetch-User\": \"?1\", \"sec-ch-ua\": \"\\\"Chromium\\\";v=\\\"127\\\", \\\"Not=A?Brand\\\";v=\\\"24\\\", \\\"Microsoft Edge\\\";v=\\\"127\\\"\", \"sec-ch-ua-mobile\": \"?0\", \"sec-ch-ua-platform\": \"\\\"Linux\\\"\", \"User-Agent\": \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36 Edg/127.0.0.0\", \"Referer\": \"http://damndelicious.net/\"}"
+}

tests/sample_files/recipes/downshiftology.com-recipes-shakshuka.meta.json

@@ -0,0 +1,12 @@
+{
+  "url": "https://downshiftology.com/recipes/shakshuka/",
+  "fetched_at": "2025-11-04T07:23:56.673481+00:00",
+  "status": 200,
+  "final_url": "https://downshiftology.com/recipes/shakshuka/",
+  "error": null,
+  "profile": "automation",
+  "content_type": "text/html; charset=UTF-8",
+  "content_encoding": "gzip",
+  "candidate": "https://downshiftology.com/recipes/shakshuka/",
+  "request_headers": "{\"Accept\": \"text/html,application/xhtml+xml;q=0.9,*/*;q=0.8\", \"Accept-Language\": \"en-US,en;q=0.8\", \"Accept-Encoding\": \"gzip, deflate\", \"Connection\": \"keep-alive\", \"User-Agent\": \"MunchEaseRecipeBot/1.0 (+https://example.com/bot)\", \"From\": \"bot@example.com\"}"
+}

tests/sample_files/recipes/downshiftology.com-recipes-shakshuka.recipe.json

@@ -0,0 +1,178 @@
+{
+  "id": -1,
+  "name": "Shakshuka Recipe (Easy & Traditional)",
+  "link": "https://downshiftology.com/recipes/shakshuka/",
+  "serves": 6,
+  "imageUrls": [
+    "https://i2.wp.com/www.downshiftology.com/wp-content/uploads/2023/12/Shakshuka-main-1.jpg",
+    "https://downshiftology.com/wp-content/uploads/2023/12/Shakshuka-main-1-500x500.jpg",
+    "https://downshiftology.com/wp-content/uploads/2023/12/Shakshuka-main-1-500x375.jpg",
+    "https://downshiftology.com/wp-content/uploads/2023/12/Shakshuka-main-1-480x270.jpg"
+  ],
+  "ingredients": [
+    {
+      "id": -1,
+      "name": "olive oil",
+      "line": "2 tablespoons olive oil",
+      "unit": "Tablespoon",
+      "quantity": 2.0,
+      "preparation": "",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "onion",
+      "line": "1  medium onion (diced)",
+      "unit": "Items",
+      "quantity": 1.0,
+      "preparation": "(diced)",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "red bell pepper",
+      "line": "1  red bell pepper (seeded and diced)",
+      "unit": "Items",
+      "quantity": 1.0,
+      "preparation": "(seeded and diced)",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "garlic",
+      "line": "4  garlic cloves (finely chopped)",
+      "unit": "Items",
+      "quantity": 4.0,
+      "preparation": "(finely chopped)",
+      "productId": 2,
+      "recipeId": null,
+      "mealId": null,
+      "product": {
+        "id": 2,
+        "productId": "294517",
+        "shopCode": "woolworths",
+        "link": "https://www.woolworths.com.au/shop/productdetails/294517/la-famiglia-garlic-bread",
+        "name": "La Famiglia Garlic Bread",
+        "quantity": 1,
+        "unit": "Loaf",
+        "imgSmall": "https://cdn0.woolworths.media/content/wowproductimages/small/294517.jpg",
+        "imgLarge": "https://cdn0.woolworths.media/content/wowproductimages/large/294517.jpg"
+      }
+    },
+    {
+      "id": -1,
+      "name": "paprika",
+      "line": "2 teaspoon paprika",
+      "unit": "Teaspoon",
+      "quantity": 2.0,
+      "preparation": "",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "cumin",
+      "line": "1 teaspoon cumin",
+      "unit": "Teaspoon",
+      "quantity": 1.0,
+      "preparation": "",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "chili powder",
+      "line": "¼ teaspoon chili powder",
+      "unit": "Teaspoon",
+      "quantity": 0.25,
+      "preparation": "",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "whole peeled tomatoes",
+      "line": "1 (28-ounce can) whole peeled tomatoes",
+      "unit": "Ounce",
+      "quantity": 1.0,
+      "preparation": "",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "eggs",
+      "line": "6  large eggs",
+      "unit": "Items",
+      "quantity": 6.0,
+      "preparation": "",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "salt and pepper",
+      "line": "salt and pepper (to taste)",
+      "unit": "Items",
+      "quantity": 1.0,
+      "preparation": "",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "fresh cilantro",
+      "line": "1  small bunch fresh cilantro (chopped)",
+      "unit": "Items",
+      "quantity": 1.0,
+      "preparation": "(chopped)",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "fresh parsley",
+      "line": "1  small bunch fresh parsley (chopped)",
+      "unit": "Items",
+      "quantity": 1.0,
+      "preparation": "(chopped)",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    }
+  ],
+  "basedOnRecipe": null,
+  "dateCreated": "2025-11-04T18:24:29.453255+11:00",
+  "createdById": 1,
+  "createdBy": {
+    "id": 1,
+    "displayName": "Snapshot Generator"
+  },
+  "dateHidden": null,
+  "hiddenById": null,
+  "hiddenBy": null
+}

tests/sample_files/recipes/pinchofyum.com-lemon-chicken-piccata-with-grilled-bread.meta.json

@@ -0,0 +1,12 @@
+{
+  "url": "https://pinchofyum.com/lemon-chicken-piccata-with-grilled-bread",
+  "fetched_at": "2025-11-04T07:23:56.582663+00:00",
+  "status": 200,
+  "final_url": "https://pinchofyum.com/lemon-chicken-piccata-with-grilled-bread",
+  "error": null,
+  "profile": "automation",
+  "content_type": "text/html; charset=UTF-8",
+  "content_encoding": "gzip",
+  "candidate": "https://pinchofyum.com/lemon-chicken-piccata-with-grilled-bread",
+  "request_headers": "{\"Accept\": \"text/html,application/xhtml+xml;q=0.9,*/*;q=0.8\", \"Accept-Language\": \"en-US,en;q=0.8\", \"Accept-Encoding\": \"gzip, deflate\", \"Connection\": \"keep-alive\", \"User-Agent\": \"MunchEaseRecipeBot/1.0 (+https://example.com/bot)\", \"From\": \"bot@example.com\"}"
+}

tests/sample_files/recipes/pinchofyum.com-lemon-chicken-piccata-with-grilled-bread.recipe.json

@@ -0,0 +1,144 @@
+{
+  "id": -1,
+  "name": "Lemon Chicken Piccata with Grilled Bread",
+  "link": "https://pinchofyum.com/lemon-chicken-piccata-with-grilled-bread",
+  "serves": 4,
+  "imageUrls": [
+    "https://pinchofyum.com/wp-content/uploads/Chicken-Piccata-Recipe-225x225.jpg",
+    "https://pinchofyum.com/wp-content/uploads/Chicken-Piccata-Recipe-260x195.jpg",
+    "https://pinchofyum.com/wp-content/uploads/Chicken-Piccata-Recipe-320x180.jpg",
+    "https://pinchofyum.com/wp-content/uploads/Chicken-Piccata-Recipe.jpg"
+  ],
+  "ingredients": [
+    {
+      "id": -1,
+      "name": "boneless skinless chicken breasts",
+      "line": "1 pound boneless skinless chicken breasts",
+      "unit": "Pound",
+      "quantity": 1.0,
+      "preparation": "",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "flour",
+      "line": "1/2 cup flour",
+      "unit": "Cup",
+      "quantity": 0.5,
+      "preparation": "",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "salt and pepper",
+      "line": "salt and pepper",
+      "unit": "Items",
+      "quantity": 1.0,
+      "preparation": "",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "LAND O LAKES® European Style Super Premium Salted Butter",
+      "line": "4 tablespoons LAND O LAKES® European Style Super Premium Salted Butter",
+      "unit": "Tablespoon",
+      "quantity": 4.0,
+      "preparation": "",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "olive oil",
+      "line": "2 tablespoons olive oil",
+      "unit": "Tablespoon",
+      "quantity": 2.0,
+      "preparation": "",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "white wine",
+      "line": "1/2 cup white wine",
+      "unit": "Cup",
+      "quantity": 0.5,
+      "preparation": "",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "chicken broth",
+      "line": "1 1/2 cups chicken broth",
+      "unit": "Cup",
+      "quantity": 1.5,
+      "preparation": "",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "lemon",
+      "line": "1 large lemon, sliced thinly (leave about 1/4 of the lemon intact for the juice)",
+      "unit": "Items",
+      "quantity": 1.0,
+      "preparation": "sliced thinly",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "jarred capers",
+      "line": "1/4 cup jarred capers",
+      "unit": "Cup",
+      "quantity": 0.25,
+      "preparation": "",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "fresh parsley",
+      "line": "fresh parsley",
+      "unit": "Items",
+      "quantity": 1.0,
+      "preparation": "",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    }
+  ],
+  "basedOnRecipe": null,
+  "dateCreated": "2025-11-04T18:24:29.545344+11:00",
+  "createdById": 1,
+  "createdBy": {
+    "id": 1,
+    "displayName": "Snapshot Generator"
+  },
+  "dateHidden": null,
+  "hiddenById": null,
+  "hiddenBy": null
+}

tests/sample_files/recipes/sallysbakingaddiction.com-triple-chocolate-layer-cake.meta.json

@@ -0,0 +1,12 @@
+{
+  "url": "https://sallysbakingaddiction.com/triple-chocolate-layer-cake/",
+  "fetched_at": "2025-11-04T07:23:56.660551+00:00",
+  "status": 200,
+  "final_url": "https://sallysbakingaddiction.com/triple-chocolate-layer-cake/",
+  "error": null,
+  "profile": "automation",
+  "content_type": "text/html; charset=UTF-8",
+  "content_encoding": "gzip",
+  "candidate": "https://sallysbakingaddiction.com/triple-chocolate-layer-cake/",
+  "request_headers": "{\"Accept\": \"text/html,application/xhtml+xml;q=0.9,*/*;q=0.8\", \"Accept-Language\": \"en-US,en;q=0.8\", \"Accept-Encoding\": \"gzip, deflate\", \"Connection\": \"keep-alive\", \"User-Agent\": \"MunchEaseRecipeBot/1.0 (+https://example.com/bot)\", \"From\": \"bot@example.com\"}"
+}

tests/sample_files/recipes/sallysbakingaddiction.com-triple-chocolate-layer-cake.recipe.json

@@ -0,0 +1,282 @@
+{
+  "id": -1,
+  "name": "Deliciously Moist Chocolate Layer Cake",
+  "link": "https://sallysbakingaddiction.com/triple-chocolate-layer-cake/",
+  "serves": 12,
+  "imageUrls": [
+    "https://sallysbakingaddiction.com/wp-content/uploads/2013/04/triple-chocolate-cake-4-225x225.jpg",
+    "https://sallysbakingaddiction.com/wp-content/uploads/2013/04/triple-chocolate-cake-4-260x195.jpg",
+    "https://sallysbakingaddiction.com/wp-content/uploads/2013/04/triple-chocolate-cake-4-320x180.jpg",
+    "https://sallysbakingaddiction.com/wp-content/uploads/2013/04/triple-chocolate-cake-4.jpg"
+  ],
+  "ingredients": [
+    {
+      "id": -1,
+      "name": "all-purpose flour",
+      "line": "1 and 3/4 cups (219g) all-purpose flour (spooned & leveled)",
+      "unit": "Cup",
+      "quantity": 1.75,
+      "preparation": "spooned &",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "unsweetened natural cocoa powder",
+      "line": "3/4 cup (62g) unsweetened natural cocoa powder",
+      "unit": "Cup",
+      "quantity": 0.75,
+      "preparation": "",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "granulated sugar",
+      "line": "1 and 3/4 cups (350g) granulated sugar",
+      "unit": "Cup",
+      "quantity": 1.75,
+      "preparation": "",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "baking soda",
+      "line": "2 teaspoons baking soda",
+      "unit": "Teaspoon",
+      "quantity": 2.0,
+      "preparation": "",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "baking powder",
+      "line": "1 teaspoon baking powder",
+      "unit": "Teaspoon",
+      "quantity": 1.0,
+      "preparation": "",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "salt",
+      "line": "1 teaspoon salt",
+      "unit": "Teaspoon",
+      "quantity": 1.0,
+      "preparation": "",
+      "productId": 5,
+      "recipeId": null,
+      "mealId": null,
+      "product": {
+        "id": 5,
+        "productId": "33245",
+        "shopCode": "woolworths",
+        "link": "https://www.woolworths.com.au/shop/productdetails/33245/saxa-iodised-table-salt-shaker",
+        "name": "Saxa Iodised Table Salt Shaker",
+        "quantity": 750,
+        "unit": "g",
+        "imgSmall": "https://cdn0.woolworths.media/content/wowproductimages/small/033245.jpg",
+        "imgLarge": "https://cdn0.woolworths.media/content/wowproductimages/large/033245.jpg"
+      }
+    },
+    {
+      "id": -1,
+      "name": "espresso powder",
+      "line": "2 teaspoons espresso powder (optional)",
+      "unit": "Teaspoon",
+      "quantity": 2.0,
+      "preparation": "",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "vegetable oil",
+      "line": "1/2 cup (120ml) vegetable oil (or canola oil or melted coconut oil)",
+      "unit": "Cup",
+      "quantity": 0.5,
+      "preparation": "",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "eggs",
+      "line": "2 large eggs, at room temperature",
+      "unit": "Items",
+      "quantity": 2.0,
+      "preparation": "at room temperature",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "pure vanilla extract",
+      "line": "2 teaspoons pure vanilla extract",
+      "unit": "Teaspoon",
+      "quantity": 2.0,
+      "preparation": "",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "buttermilk",
+      "line": "1 cup (240ml) buttermilk, at room temperature",
+      "unit": "Cup",
+      "quantity": 1.0,
+      "preparation": "at room temperature",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "strong hot coffee",
+      "line": "1 cup (240ml) freshly brewed strong hot coffee (regular or decaf)",
+      "unit": "Cup",
+      "quantity": 1.0,
+      "preparation": "freshly brewed",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "unsalted butter",
+      "line": "1 and 1/4 cups (282g) unsalted butter, softened to room temperature",
+      "unit": "Cup",
+      "quantity": 1.25,
+      "preparation": "softened to room temperature",
+      "productId": 4,
+      "recipeId": null,
+      "mealId": null,
+      "product": {
+        "id": 4,
+        "productId": "712251",
+        "shopCode": "woolworths",
+        "link": "https://www.woolworths.com.au/shop/productdetails/712251/western-star-unsalted-butter-chef-s-choice",
+        "name": "Western Star Unsalted Butter Chef's Choice",
+        "quantity": 500,
+        "unit": "g",
+        "imgSmall": "https://cdn0.woolworths.media/content/wowproductimages/small/712251.jpg",
+        "imgLarge": "https://cdn0.woolworths.media/content/wowproductimages/large/712251.jpg"
+      }
+    },
+    {
+      "id": -1,
+      "name": "confectioners’ sugar",
+      "line": "3 and 1/2 cups (420g) confectioners’ sugar",
+      "unit": "Cup",
+      "quantity": 3.5,
+      "preparation": "",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "unsweetened cocoa powder",
+      "line": "3/4 cup (62g) unsweetened cocoa powder (natural or dutch process)",
+      "unit": "Cup",
+      "quantity": 0.75,
+      "preparation": "",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "heavy cream",
+      "line": "3-5 Tablespoons (45-75ml) heavy cream (or half-and-half or milk), at room temperature",
+      "unit": "Tablespoon",
+      "quantity": 3.0,
+      "preparation": "at room temperature",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "salt",
+      "line": "1/4 teaspoon salt",
+      "unit": "Teaspoon",
+      "quantity": 0.25,
+      "preparation": "",
+      "productId": 5,
+      "recipeId": null,
+      "mealId": null,
+      "product": {
+        "id": 5,
+        "productId": "33245",
+        "shopCode": "woolworths",
+        "link": "https://www.woolworths.com.au/shop/productdetails/33245/saxa-iodised-table-salt-shaker",
+        "name": "Saxa Iodised Table Salt Shaker",
+        "quantity": 750,
+        "unit": "g",
+        "imgSmall": "https://cdn0.woolworths.media/content/wowproductimages/small/033245.jpg",
+        "imgLarge": "https://cdn0.woolworths.media/content/wowproductimages/large/033245.jpg"
+      }
+    },
+    {
+      "id": -1,
+      "name": "pure vanilla extract",
+      "line": "1 teaspoon pure vanilla extract",
+      "unit": "Teaspoon",
+      "quantity": 1.0,
+      "preparation": "",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "semi-sweet chocolate chips",
+      "line": "optional for decoration: semi-sweet chocolate chips",
+      "unit": "Items",
+      "quantity": 1.0,
+      "preparation": "",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    }
+  ],
+  "basedOnRecipe": null,
+  "dateCreated": "2025-11-04T18:24:29.632664+11:00",
+  "createdById": 1,
+  "createdBy": {
+    "id": 1,
+    "displayName": "Snapshot Generator"
+  },
+  "dateHidden": null,
+  "hiddenById": null,
+  "hiddenBy": null
+}

tests/sample_files/recipes/tasty.co-recipe-one-pot-garlic-parmesan-pasta.meta.json

@@ -0,0 +1,12 @@
+{
+  "url": "https://tasty.co/recipe/one-pot-garlic-parmesan-pasta",
+  "fetched_at": "2025-11-04T07:23:56.855490+00:00",
+  "status": 200,
+  "final_url": "https://tasty.co/recipe/one-pot-garlic-parmesan-pasta",
+  "error": null,
+  "profile": "automation",
+  "content_type": "text/html; charset=utf-8",
+  "content_encoding": "gzip",
+  "candidate": "https://tasty.co/recipe/one-pot-garlic-parmesan-pasta",
+  "request_headers": "{\"Accept\": \"text/html,application/xhtml+xml;q=0.9,*/*;q=0.8\", \"Accept-Language\": \"en-US,en;q=0.8\", \"Accept-Encoding\": \"gzip, deflate\", \"Connection\": \"keep-alive\", \"User-Agent\": \"MunchEaseRecipeBot/1.0 (+https://example.com/bot)\", \"From\": \"bot@example.com\"}"
+}

tests/sample_files/recipes/tasty.co-recipe-one-pot-garlic-parmesan-pasta.recipe.json

@@ -0,0 +1,169 @@
+{
+  "id": -1,
+  "name": "One-Pot Garlic Parmesan Pasta Recipe by Tasty",
+  "link": "https://tasty.co/recipe/one-pot-garlic-parmesan-pasta",
+  "serves": 4,
+  "imageUrls": [
+    "https://img.buzzfeed.com/thumbnailer-prod-us-east-1/f69a7f4192b94d8395757b365ac6d866/GarlicParmPasta.jpg?resize=1200:*"
+  ],
+  "ingredients": [
+    {
+      "id": -1,
+      "name": "unsalted butter",
+      "line": "2 tablespoons unsalted butter",
+      "unit": "Tablespoon",
+      "quantity": 2.0,
+      "preparation": "",
+      "productId": 4,
+      "recipeId": null,
+      "mealId": null,
+      "product": {
+        "id": 4,
+        "productId": "712251",
+        "shopCode": "woolworths",
+        "link": "https://www.woolworths.com.au/shop/productdetails/712251/western-star-unsalted-butter-chef-s-choice",
+        "name": "Western Star Unsalted Butter Chef's Choice",
+        "quantity": 500,
+        "unit": "g",
+        "imgSmall": "https://cdn0.woolworths.media/content/wowproductimages/small/712251.jpg",
+        "imgLarge": "https://cdn0.woolworths.media/content/wowproductimages/large/712251.jpg"
+      }
+    },
+    {
+      "id": -1,
+      "name": "garlic",
+      "line": "4 cloves garlic, minced",
+      "unit": "Items",
+      "quantity": 4.0,
+      "preparation": "minced",
+      "productId": 2,
+      "recipeId": null,
+      "mealId": null,
+      "product": {
+        "id": 2,
+        "productId": "294517",
+        "shopCode": "woolworths",
+        "link": "https://www.woolworths.com.au/shop/productdetails/294517/la-famiglia-garlic-bread",
+        "name": "La Famiglia Garlic Bread",
+        "quantity": 1,
+        "unit": "Loaf",
+        "imgSmall": "https://cdn0.woolworths.media/content/wowproductimages/small/294517.jpg",
+        "imgLarge": "https://cdn0.woolworths.media/content/wowproductimages/large/294517.jpg"
+      }
+    },
+    {
+      "id": -1,
+      "name": "chicken broth",
+      "line": "2 cups chicken broth",
+      "unit": "Cup",
+      "quantity": 2.0,
+      "preparation": "",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "milk",
+      "line": "1 cup milk",
+      "unit": "Cup",
+      "quantity": 1.0,
+      "preparation": "",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "fettuccine",
+      "line": "8 oz fettuccine",
+      "unit": "Ounce",
+      "quantity": 8.0,
+      "preparation": "",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "salt",
+      "line": "salt, to taste",
+      "unit": "Items",
+      "quantity": 1.0,
+      "preparation": "",
+      "productId": 5,
+      "recipeId": null,
+      "mealId": null,
+      "product": {
+        "id": 5,
+        "productId": "33245",
+        "shopCode": "woolworths",
+        "link": "https://www.woolworths.com.au/shop/productdetails/33245/saxa-iodised-table-salt-shaker",
+        "name": "Saxa Iodised Table Salt Shaker",
+        "quantity": 750,
+        "unit": "g",
+        "imgSmall": "https://cdn0.woolworths.media/content/wowproductimages/small/033245.jpg",
+        "imgLarge": "https://cdn0.woolworths.media/content/wowproductimages/large/033245.jpg"
+      }
+    },
+    {
+      "id": -1,
+      "name": "pepper",
+      "line": "pepper, to taste",
+      "unit": "Items",
+      "quantity": 1.0,
+      "preparation": "",
+      "productId": 6,
+      "recipeId": null,
+      "mealId": null,
+      "product": {
+        "id": 6,
+        "productId": "75194",
+        "shopCode": "woolworths",
+        "link": "https://www.woolworths.com.au/shop/productdetails/75194/mckenzie-s-pepper-black-ground",
+        "name": "Mckenzie's Pepper Black Ground",
+        "quantity": 100,
+        "unit": "g",
+        "imgSmall": "https://cdn0.woolworths.media/content/wowproductimages/small/075194.jpg",
+        "imgLarge": "https://cdn0.woolworths.media/content/wowproductimages/large/075194.jpg"
+      }
+    },
+    {
+      "id": -1,
+      "name": "parmesan cheese",
+      "line": "¼ cup grated parmesan cheese",
+      "unit": "Cup",
+      "quantity": 0.25,
+      "preparation": "grated",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "fresh parsley",
+      "line": "2 tablespoons fresh parsley, chopped",
+      "unit": "Tablespoon",
+      "quantity": 2.0,
+      "preparation": "chopped",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    }
+  ],
+  "basedOnRecipe": null,
+  "dateCreated": "2025-11-04T18:24:29.662183+11:00",
+  "createdById": 1,
+  "createdBy": {
+    "id": 1,
+    "displayName": "Snapshot Generator"
+  },
+  "dateHidden": null,
+  "hiddenById": null,
+  "hiddenBy": null
+}

tests/sample_files/recipes/www.bbcgoodfood.com-recipes-chicken-tikka-masala.meta.json

@@ -0,0 +1,12 @@
+{
+  "url": "https://www.bbcgoodfood.com/recipes/chicken-tikka-masala",
+  "fetched_at": "2025-11-04T07:23:56.582856+00:00",
+  "status": 200,
+  "final_url": "https://www.bbcgoodfood.com/recipes/chicken-tikka-masala",
+  "error": null,
+  "profile": "automation",
+  "content_type": "text/html; charset=utf-8",
+  "content_encoding": "gzip",
+  "candidate": "https://www.bbcgoodfood.com/recipes/chicken-tikka-masala",
+  "request_headers": "{\"Accept\": \"text/html,application/xhtml+xml;q=0.9,*/*;q=0.8\", \"Accept-Language\": \"en-US,en;q=0.8\", \"Accept-Encoding\": \"gzip, deflate\", \"Connection\": \"keep-alive\", \"User-Agent\": \"MunchEaseRecipeBot/1.0 (+https://example.com/bot)\", \"From\": \"bot@example.com\"}"
+}

tests/sample_files/recipes/www.bbcgoodfood.com-recipes-chicken-tikka-masala.recipe.json

@@ -0,0 +1,175 @@
+{
+  "id": -1,
+  "name": "Chicken tikka masala",
+  "link": "https://www.bbcgoodfood.com/recipes/chicken-tikka-masala",
+  "serves": 10,
+  "imageUrls": [
+    "https://images.immediate.co.uk/production/volatile/sites/30/2020/08/recipe-image-legacy-id-202451_12-50a0c95.jpg?resize=440,400"
+  ],
+  "ingredients": [
+    {
+      "id": -1,
+      "name": "vegetable oil",
+      "line": "4 tbsp vegetable oil",
+      "unit": "Tablespoon",
+      "quantity": 4.0,
+      "preparation": "",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "butter",
+      "line": "25g butter",
+      "unit": "Gram",
+      "quantity": 25.0,
+      "preparation": "",
+      "productId": 4,
+      "recipeId": null,
+      "mealId": null,
+      "product": {
+        "id": 4,
+        "productId": "712251",
+        "shopCode": "woolworths",
+        "link": "https://www.woolworths.com.au/shop/productdetails/712251/western-star-unsalted-butter-chef-s-choice",
+        "name": "Western Star Unsalted Butter Chef's Choice",
+        "quantity": 500,
+        "unit": "g",
+        "imgSmall": "https://cdn0.woolworths.media/content/wowproductimages/small/712251.jpg",
+        "imgLarge": "https://cdn0.woolworths.media/content/wowproductimages/large/712251.jpg"
+      }
+    },
+    {
+      "id": -1,
+      "name": "onions",
+      "line": "4 onions roughly chopped",
+      "unit": "Items",
+      "quantity": 4.0,
+      "preparation": "roughly chopped",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "chicken tikka masala paste",
+      "line": "6 tbsp chicken tikka masala paste (use shop-bought or make your own – see recipe, below)",
+      "unit": "Tablespoon",
+      "quantity": 6.0,
+      "preparation": "",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "red peppers",
+      "line": "2 red peppers deseeded and cut into chunks",
+      "unit": "Items",
+      "quantity": 2.0,
+      "preparation": "deseeded and cut into chunks",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "boneless, skinless chicken breasts",
+      "line": "8 boneless, skinless chicken breasts cut into 2.5cm cubes",
+      "unit": "Items",
+      "quantity": 8.0,
+      "preparation": "cut into 2.5 cm cubes",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "chopped tomatoes",
+      "line": "2 x 400g cans  chopped tomatoes",
+      "unit": "Gram",
+      "quantity": 2.0,
+      "preparation": "",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "tomato purée",
+      "line": "4 tbsp tomato purée",
+      "unit": "Tablespoon",
+      "quantity": 4.0,
+      "preparation": "",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "mango chutney",
+      "line": "2-3 tbsp  mango chutney",
+      "unit": "Tablespoon",
+      "quantity": 2.0,
+      "preparation": "",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "double cream",
+      "line": "150ml double cream",
+      "unit": "Items",
+      "quantity": 150.0,
+      "preparation": "",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "natural yogurt",
+      "line": "150ml natural  yogurt",
+      "unit": "Items",
+      "quantity": 150.0,
+      "preparation": "",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "coriander leaves",
+      "line": "chopped  coriander leaves, to serve",
+      "unit": "Items",
+      "quantity": 1.0,
+      "preparation": "chopped",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    }
+  ],
+  "basedOnRecipe": null,
+  "dateCreated": "2025-11-04T18:24:29.711728+11:00",
+  "createdById": 1,
+  "createdBy": {
+    "id": 1,
+    "displayName": "Snapshot Generator"
+  },
+  "dateHidden": null,
+  "hiddenById": null,
+  "hiddenBy": null
+}

tests/sample_files/recipes/www.inspiredtaste.net-24412-cocoa-brownies-recipe.meta.json

@@ -0,0 +1,12 @@
+{
+  "url": "https://www.inspiredtaste.net/24412/cocoa-brownies-recipe/",
+  "fetched_at": "2025-11-04T07:23:56.582248+00:00",
+  "status": 200,
+  "final_url": "https://www.inspiredtaste.net/24412/cocoa-brownies-recipe/",
+  "error": null,
+  "profile": "automation",
+  "content_type": "text/html; charset=UTF-8",
+  "content_encoding": "gzip",
+  "candidate": "https://www.inspiredtaste.net/24412/cocoa-brownies-recipe/",
+  "request_headers": "{\"Accept\": \"text/html,application/xhtml+xml;q=0.9,*/*;q=0.8\", \"Accept-Language\": \"en-US,en;q=0.8\", \"Accept-Encoding\": \"gzip, deflate\", \"Connection\": \"keep-alive\", \"User-Agent\": \"MunchEaseRecipeBot/1.0 (+https://example.com/bot)\", \"From\": \"bot@example.com\"}"
+}

tests/sample_files/recipes/www.inspiredtaste.net-24412-cocoa-brownies-recipe.recipe.json

@@ -0,0 +1 @@
+null

tests/sample_files/recipes/www.kingarthurbaking.com-recipes-banana-bread-recipe.meta.json

@@ -0,0 +1,12 @@
+{
+  "url": "https://www.kingarthurbaking.com/recipes/banana-bread-recipe",
+  "fetched_at": "2025-11-04T07:23:56.767749+00:00",
+  "status": 200,
+  "final_url": "https://www.kingarthurbaking.com/recipes/banana-bread-recipe",
+  "error": null,
+  "profile": "automation",
+  "content_type": "text/html; charset=UTF-8",
+  "content_encoding": "gzip",
+  "candidate": "https://www.kingarthurbaking.com/recipes/banana-bread-recipe",
+  "request_headers": "{\"Accept\": \"text/html,application/xhtml+xml;q=0.9,*/*;q=0.8\", \"Accept-Language\": \"en-US,en;q=0.8\", \"Accept-Encoding\": \"gzip, deflate\", \"Connection\": \"keep-alive\", \"User-Agent\": \"MunchEaseRecipeBot/1.0 (+https://example.com/bot)\", \"From\": \"bot@example.com\"}"
+}

tests/sample_files/recipes/www.kingarthurbaking.com-recipes-banana-bread-recipe.recipe.json

@@ -0,0 +1,199 @@
+{
+  "id": -1,
+  "name": "Banana Bread",
+  "link": "https://www.kingarthurbaking.com/recipes/banana-bread-recipe",
+  "serves": 18,
+  "imageUrls": [
+    "https://www.kingarthurbaking.com/sites/default/files/recipe_legacy/5-3-large.jpg"
+  ],
+  "ingredients": [
+    {
+      "id": -1,
+      "name": "unsalted butter",
+      "line": "8 tablespoons (113g) unsalted butter, at cool room temperature",
+      "unit": "Tablespoon",
+      "quantity": 8.0,
+      "preparation": "at cool room temperature",
+      "productId": 4,
+      "recipeId": null,
+      "mealId": null,
+      "product": {
+        "id": 4,
+        "productId": "712251",
+        "shopCode": "woolworths",
+        "link": "https://www.woolworths.com.au/shop/productdetails/712251/western-star-unsalted-butter-chef-s-choice",
+        "name": "Western Star Unsalted Butter Chef's Choice",
+        "quantity": 500,
+        "unit": "g",
+        "imgSmall": "https://cdn0.woolworths.media/content/wowproductimages/small/712251.jpg",
+        "imgLarge": "https://cdn0.woolworths.media/content/wowproductimages/large/712251.jpg"
+      }
+    },
+    {
+      "id": -1,
+      "name": "light brown sugar or dark brown sugar",
+      "line": "2/3 cup (142g) light brown sugar or dark brown sugar, packed",
+      "unit": "Cup",
+      "quantity": 0.667,
+      "preparation": "",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "King Arthur Pure Vanilla Extract",
+      "line": "1 teaspoon King Arthur Pure Vanilla Extract",
+      "unit": "Teaspoon",
+      "quantity": 1.0,
+      "preparation": "",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "cinnamon",
+      "line": "1 teaspoon cinnamon",
+      "unit": "Teaspoon",
+      "quantity": 1.0,
+      "preparation": "",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "nutmeg",
+      "line": "1/4 teaspoon nutmeg",
+      "unit": "Teaspoon",
+      "quantity": 0.25,
+      "preparation": "",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "baking soda",
+      "line": "1 teaspoon baking soda",
+      "unit": "Teaspoon",
+      "quantity": 1.0,
+      "preparation": "",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "baking powder",
+      "line": "1 teaspoon baking powder",
+      "unit": "Teaspoon",
+      "quantity": 1.0,
+      "preparation": "",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "table salt",
+      "line": "1 teaspoon table salt",
+      "unit": "Teaspoon",
+      "quantity": 1.0,
+      "preparation": "",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "bananas",
+      "line": "1 1/2 cups (340g) bananas, mashed",
+      "unit": "Cup",
+      "quantity": 1.5,
+      "preparation": "mashed",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "apricot jam or orange marmalade",
+      "line": "3 tablespoons (64g) apricot jam or orange marmalade, optional but tasty",
+      "unit": "Tablespoon",
+      "quantity": 3.0,
+      "preparation": "",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "honey",
+      "line": "1/4 cup (85g) honey",
+      "unit": "Cup",
+      "quantity": 0.25,
+      "preparation": "",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "eggs",
+      "line": "2 large eggs",
+      "unit": "Items",
+      "quantity": 2.0,
+      "preparation": "",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "King Arthur Unbleached All-Purpose Flour",
+      "line": "2 1/4 cups (270g) King Arthur Unbleached All-Purpose Flour",
+      "unit": "Cup",
+      "quantity": 2.25,
+      "preparation": "",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "walnuts",
+      "line": "1/2 cup (57g) chopped walnuts, optional",
+      "unit": "Cup",
+      "quantity": 0.5,
+      "preparation": "chopped",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    }
+  ],
+  "basedOnRecipe": null,
+  "dateCreated": "2025-11-04T18:24:29.834444+11:00",
+  "createdById": 1,
+  "createdBy": {
+    "id": 1,
+    "displayName": "Snapshot Generator"
+  },
+  "dateHidden": null,
+  "hiddenById": null,
+  "hiddenBy": null
+}

tests/sample_files/recipes/www.recipetineats.com-beef-stroganoff.meta.json

@@ -0,0 +1,12 @@
+{
+  "url": "https://www.recipetineats.com/beef-stroganoff/",
+  "fetched_at": "2025-11-04T07:23:56.628726+00:00",
+  "status": 200,
+  "final_url": "https://www.recipetineats.com/beef-stroganoff/",
+  "error": null,
+  "profile": "automation",
+  "content_type": "text/html; charset=UTF-8",
+  "content_encoding": "gzip",
+  "candidate": "https://www.recipetineats.com/beef-stroganoff/",
+  "request_headers": "{\"Accept\": \"text/html,application/xhtml+xml;q=0.9,*/*;q=0.8\", \"Accept-Language\": \"en-US,en;q=0.8\", \"Accept-Encoding\": \"gzip, deflate\", \"Connection\": \"keep-alive\", \"User-Agent\": \"MunchEaseRecipeBot/1.0 (+https://example.com/bot)\", \"From\": \"bot@example.com\"}"
+}

tests/sample_files/recipes/www.recipetineats.com-beef-stroganoff.recipe.json

@@ -0,0 +1,178 @@
+{
+  "id": -1,
+  "name": "Beef Stroganoff",
+  "link": "https://www.recipetineats.com/beef-stroganoff/",
+  "serves": 4,
+  "imageUrls": [
+    "https://www.recipetineats.com/tachyon/2018/01/Beef-Stroganoff_2-1-1.jpg",
+    "https://www.recipetineats.com/tachyon/2018/01/Beef-Stroganoff_2-1-1.jpg?resize=500%2C500",
+    "https://www.recipetineats.com/tachyon/2018/01/Beef-Stroganoff_2-1-1.jpg?resize=500%2C375",
+    "https://www.recipetineats.com/tachyon/2018/01/Beef-Stroganoff_2-1-1.jpg?resize=480%2C270"
+  ],
+  "ingredients": [
+    {
+      "id": -1,
+      "name": "scotch fillet steak / boneless rib eye",
+      "line": "600 g / 1.2 lb   scotch fillet steak / boneless rib eye ((Note 1))",
+      "unit": "Gram",
+      "quantity": 600.0,
+      "preparation": "",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "vegetable oil",
+      "line": "2 tbsp vegetable oil (, divided)",
+      "unit": "Tablespoon",
+      "quantity": 2.0,
+      "preparation": "(, divided)",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "onion",
+      "line": "1  large onion ((or 2 small onions), sliced)",
+      "unit": "Items",
+      "quantity": 1.0,
+      "preparation": "sliced",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "mushrooms",
+      "line": "300 g / 10 oz   mushrooms (, sliced (not too thin))",
+      "unit": "Gram",
+      "quantity": 300.0,
+      "preparation": ", sliced",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "butter",
+      "line": "40 g / 3 tbsp   butter",
+      "unit": "Gram",
+      "quantity": 40.0,
+      "preparation": "",
+      "productId": 4,
+      "recipeId": null,
+      "mealId": null,
+      "product": {
+        "id": 4,
+        "productId": "712251",
+        "shopCode": "woolworths",
+        "link": "https://www.woolworths.com.au/shop/productdetails/712251/western-star-unsalted-butter-chef-s-choice",
+        "name": "Western Star Unsalted Butter Chef's Choice",
+        "quantity": 500,
+        "unit": "g",
+        "imgSmall": "https://cdn0.woolworths.media/content/wowproductimages/small/712251.jpg",
+        "imgLarge": "https://cdn0.woolworths.media/content/wowproductimages/large/712251.jpg"
+      }
+    },
+    {
+      "id": -1,
+      "name": "flour",
+      "line": "2 tbsp flour ((Note 2))",
+      "unit": "Tablespoon",
+      "quantity": 2.0,
+      "preparation": "",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "beef broth",
+      "line": "2 cups / 500 ml   beef broth (, preferably salt reduced)",
+      "unit": "Cup",
+      "quantity": 2.0,
+      "preparation": "",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "Dijon mustard",
+      "line": "1 tbsp Dijon mustard",
+      "unit": "Tablespoon",
+      "quantity": 1.0,
+      "preparation": "",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "sour cream",
+      "line": "150 ml / 2/3 cup   sour cream",
+      "unit": "Cup",
+      "quantity": 150.0,
+      "preparation": "",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "Salt and pepper",
+      "line": "Salt and pepper",
+      "unit": "Items",
+      "quantity": 1.0,
+      "preparation": "",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "pasta or egg noodles of choice",
+      "line": "250 - 300 g / 8 - 10 oz   pasta or egg noodles of choice ((Note 3))",
+      "unit": "Gram",
+      "quantity": 250.0,
+      "preparation": "",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "chives",
+      "line": "Chopped chives (, for garnish (optional))",
+      "unit": "Items",
+      "quantity": 1.0,
+      "preparation": "Chopped",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    }
+  ],
+  "basedOnRecipe": null,
+  "dateCreated": "2025-11-04T18:24:29.900537+11:00",
+  "createdById": 1,
+  "createdBy": {
+    "id": 1,
+    "displayName": "Snapshot Generator"
+  },
+  "dateHidden": null,
+  "hiddenById": null,
+  "hiddenBy": null
+}

tests/sample_files/recipes/www.simplyrecipes.com-recipes-french_toast.meta.json

@@ -0,0 +1,12 @@
+{
+  "url": "https://www.simplyrecipes.com/recipes/french_toast/",
+  "fetched_at": "2025-11-04T07:23:56.582449+00:00",
+  "status": 200,
+  "final_url": "https://www.simplyrecipes.com/recipes/french_toast/",
+  "error": null,
+  "profile": "automation",
+  "content_type": "text/html;charset=utf-8",
+  "content_encoding": "gzip",
+  "candidate": "https://www.simplyrecipes.com/recipes/french_toast/",
+  "request_headers": "{\"Accept\": \"text/html,application/xhtml+xml;q=0.9,*/*;q=0.8\", \"Accept-Language\": \"en-US,en;q=0.8\", \"Accept-Encoding\": \"gzip, deflate\", \"Connection\": \"keep-alive\", \"User-Agent\": \"MunchEaseRecipeBot/1.0 (+https://example.com/bot)\", \"From\": \"bot@example.com\"}"
+}

tests/sample_files/recipes/www.simplyrecipes.com-recipes-french_toast.recipe.json

@@ -0,0 +1,139 @@
+{
+  "id": -1,
+  "name": "The Best French Toast",
+  "link": "https://www.simplyrecipes.com/recipes/french_toast/",
+  "serves": 4,
+  "imageUrls": [
+    "https://www.simplyrecipes.com/thmb/34kTh59L8NjsXOB8nqw3hdYIKbs=/1500x0/filters:no_upscale():max_bytes(150000):strip_icc()/Simply-Recipes-Best-French-Toast-LEAD-4-ce3d4ce3d69b4c79a7bb3d9b83c8c3fc.jpg"
+  ],
+  "ingredients": [
+    {
+      "id": -1,
+      "name": "eggs",
+      "line": "4 eggs",
+      "unit": "Items",
+      "quantity": 4.0,
+      "preparation": "",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "milk",
+      "line": "2/3 cup milk",
+      "unit": "Cup",
+      "quantity": 0.667,
+      "preparation": "",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "cinnamon",
+      "line": "2 teaspoons cinnamon",
+      "unit": "Teaspoon",
+      "quantity": 2.0,
+      "preparation": "",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "2-day-old bread",
+      "line": "8 thick slices 2-day-old bread (better if slightly stale)",
+      "unit": "Items",
+      "quantity": 8.0,
+      "preparation": "",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "Butter",
+      "line": "Butter (can sub vegetable oil)",
+      "unit": "Items",
+      "quantity": 1.0,
+      "preparation": "",
+      "productId": 4,
+      "recipeId": null,
+      "mealId": null,
+      "product": {
+        "id": 4,
+        "productId": "712251",
+        "shopCode": "woolworths",
+        "link": "https://www.woolworths.com.au/shop/productdetails/712251/western-star-unsalted-butter-chef-s-choice",
+        "name": "Western Star Unsalted Butter Chef's Choice",
+        "quantity": 500,
+        "unit": "g",
+        "imgSmall": "https://cdn0.woolworths.media/content/wowproductimages/small/712251.jpg",
+        "imgLarge": "https://cdn0.woolworths.media/content/wowproductimages/large/712251.jpg"
+      }
+    },
+    {
+      "id": -1,
+      "name": "Maple syrup",
+      "line": "Maple syrup",
+      "unit": "Items",
+      "quantity": 1.0,
+      "preparation": "",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "orange zest",
+      "line": "2 teaspoons freshly grated orange zest",
+      "unit": "Teaspoon",
+      "quantity": 2.0,
+      "preparation": "freshly grated",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "Triple Sec",
+      "line": "1/4 cup Triple Sec",
+      "unit": "Cup",
+      "quantity": 0.25,
+      "preparation": "",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "Fresh berries",
+      "line": "Fresh berries",
+      "unit": "Items",
+      "quantity": 1.0,
+      "preparation": "",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    }
+  ],
+  "basedOnRecipe": null,
+  "dateCreated": "2025-11-04T18:24:29.962968+11:00",
+  "createdById": 1,
+  "createdBy": {
+    "id": 1,
+    "displayName": "Snapshot Generator"
+  },
+  "dateHidden": null,
+  "hiddenById": null,
+  "hiddenBy": null
+}

tests/sample_files/recipes/www.simplyrecipes.com-recipes-perfect_guacamole.meta.json

@@ -0,0 +1,12 @@
+{
+  "url": "https://www.simplyrecipes.com/recipes/perfect_guacamole/",
+  "fetched_at": "2025-11-04T07:23:56.744212+00:00",
+  "status": 200,
+  "final_url": "https://www.simplyrecipes.com/recipes/perfect_guacamole/",
+  "error": null,
+  "profile": "automation",
+  "content_type": "text/html;charset=utf-8",
+  "content_encoding": "gzip",
+  "candidate": "https://www.simplyrecipes.com/recipes/perfect_guacamole/",
+  "request_headers": "{\"Accept\": \"text/html,application/xhtml+xml;q=0.9,*/*;q=0.8\", \"Accept-Language\": \"en-US,en;q=0.8\", \"Accept-Encoding\": \"gzip, deflate\", \"Connection\": \"keep-alive\", \"User-Agent\": \"MunchEaseRecipeBot/1.0 (+https://example.com/bot)\", \"From\": \"bot@example.com\"}"
+}

tests/sample_files/recipes/www.simplyrecipes.com-recipes-perfect_guacamole.recipe.json

@@ -0,0 +1,161 @@
+{
+  "id": -1,
+  "name": "How to Make the Best Guacamole",
+  "link": "https://www.simplyrecipes.com/recipes/perfect_guacamole/",
+  "serves": 4,
+  "imageUrls": [
+    "https://www.simplyrecipes.com/thmb/J4kA2m6jKMgkQwZhG-RYpjZBeFQ=/1500x0/filters:no_upscale():max_bytes(150000):strip_icc()/Guacamole-LEAD-6-2-64cfcca253c8421dad4e3fad830219f6.jpg"
+  ],
+  "ingredients": [
+    {
+      "id": -1,
+      "name": "ripe avocados",
+      "line": "2 ripe avocados",
+      "unit": "Items",
+      "quantity": 2.0,
+      "preparation": "",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "salt",
+      "line": "1/4 teaspoon salt, plus more to taste",
+      "unit": "Teaspoon",
+      "quantity": 0.25,
+      "preparation": "",
+      "productId": 5,
+      "recipeId": null,
+      "mealId": null,
+      "product": {
+        "id": 5,
+        "productId": "33245",
+        "shopCode": "woolworths",
+        "link": "https://www.woolworths.com.au/shop/productdetails/33245/saxa-iodised-table-salt-shaker",
+        "name": "Saxa Iodised Table Salt Shaker",
+        "quantity": 750,
+        "unit": "g",
+        "imgSmall": "https://cdn0.woolworths.media/content/wowproductimages/small/033245.jpg",
+        "imgLarge": "https://cdn0.woolworths.media/content/wowproductimages/large/033245.jpg"
+      }
+    },
+    {
+      "id": -1,
+      "name": "fresh lime or lemon juice",
+      "line": "1 tablespoon fresh lime or lemon juice",
+      "unit": "Tablespoon",
+      "quantity": 1.0,
+      "preparation": "",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "red onion or thinly sliced green onion",
+      "line": "2-4 tablespoons minced red onion or thinly sliced green onion",
+      "unit": "Tablespoon",
+      "quantity": 2.0,
+      "preparation": "minced",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "serrano (or jalapeño) chiles",
+      "line": "1-2 serrano (or jalapeño) chiles, stems and seeds removed, minced",
+      "unit": "Items",
+      "quantity": 1.0,
+      "preparation": "stems and seeds removed, minced",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "cilantro",
+      "line": "2 tablespoons cilantro (leaves and tender stems), finely chopped",
+      "unit": "Tablespoon",
+      "quantity": 2.0,
+      "preparation": "finely chopped",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "black pepper",
+      "line": "Pinch freshly ground black pepper",
+      "unit": "Items",
+      "quantity": 1.0,
+      "preparation": "freshly ground",
+      "productId": 6,
+      "recipeId": null,
+      "mealId": null,
+      "product": {
+        "id": 6,
+        "productId": "75194",
+        "shopCode": "woolworths",
+        "link": "https://www.woolworths.com.au/shop/productdetails/75194/mckenzie-s-pepper-black-ground",
+        "name": "Mckenzie's Pepper Black Ground",
+        "quantity": 100,
+        "unit": "g",
+        "imgSmall": "https://cdn0.woolworths.media/content/wowproductimages/small/075194.jpg",
+        "imgLarge": "https://cdn0.woolworths.media/content/wowproductimages/large/075194.jpg"
+      }
+    },
+    {
+      "id": -1,
+      "name": "ripe tomato",
+      "line": "1/2 ripe tomato, chopped (optional)",
+      "unit": "Items",
+      "quantity": 0.5,
+      "preparation": "chopped",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "Red radish or jicama slices",
+      "line": "Red radish or jicama slices for garnish (optional)",
+      "unit": "Items",
+      "quantity": 1.0,
+      "preparation": "",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "Tortilla chips",
+      "line": "Tortilla chips , to serve",
+      "unit": "Items",
+      "quantity": 1.0,
+      "preparation": "",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    }
+  ],
+  "basedOnRecipe": null,
+  "dateCreated": "2025-11-04T18:24:30.070234+11:00",
+  "createdById": 1,
+  "createdBy": {
+    "id": 1,
+    "displayName": "Snapshot Generator"
+  },
+  "dateHidden": null,
+  "hiddenById": null,
+  "hiddenBy": null
+}

tests/sample_files/recipes/www.taste.com.au-classic-chewy-brownie-f0960a83-1fa9-4e3e-b616-a995182613e0.meta.json

@@ -0,0 +1,12 @@
+{
+  "url": "https://www.taste.com.au/recipes/classic-chewy-brownie/f0960a83-1fa9-4e3e-b616-a995182613e0",
+  "fetched_at": "2025-11-04T07:23:56.581964+00:00",
+  "status": 403,
+  "final_url": "http://www.taste.com.au/recipes/classic-chewy-brownie/f0960a83-1fa9-4e3e-b616-a995182613e0/amp",
+  "error": "SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:1016)",
+  "profile": "edge-desktop",
+  "content_type": "text/html",
+  "content_encoding": null,
+  "candidate": "http://www.taste.com.au/recipes/classic-chewy-brownie/f0960a83-1fa9-4e3e-b616-a995182613e0/amp",
+  "request_headers": "{\"Accept\": \"text/html,application/xhtml+xml;q=0.9,*/*;q=0.8\", \"Accept-Language\": \"en-US,en;q=0.8\", \"Accept-Encoding\": \"gzip, deflate\", \"Connection\": \"keep-alive\", \"User-Agent\": \"MunchEaseRecipeBot/1.0 (+https://example.com/bot)\", \"From\": \"bot@example.com\"}"
+}

tests/sample_files/recipes/www.thekitchn.com-chicken-tamale-pie-23752740.meta.json

@@ -0,0 +1,12 @@
+{
+  "url": "https://www.thekitchn.com/chicken-tamale-pie-23752740",
+  "fetched_at": "2025-11-04T07:23:56.686401+00:00",
+  "status": 200,
+  "final_url": "https://www.thekitchn.com/chicken-tamale-pie-23752740",
+  "error": null,
+  "profile": "automation",
+  "content_type": "text/html; charset=utf-8",
+  "content_encoding": "gzip",
+  "candidate": "https://www.thekitchn.com/chicken-tamale-pie-23752740",
+  "request_headers": "{\"Accept\": \"text/html,application/xhtml+xml;q=0.9,*/*;q=0.8\", \"Accept-Language\": \"en-US,en;q=0.8\", \"Accept-Encoding\": \"gzip, deflate\", \"Connection\": \"keep-alive\", \"User-Agent\": \"MunchEaseRecipeBot/1.0 (+https://example.com/bot)\", \"From\": \"bot@example.com\"}"
+}

tests/sample_files/recipes/www.thekitchn.com-chicken-tamale-pie-23752740.recipe.json

@@ -0,0 +1,211 @@
+{
+  "id": -1,
+  "name": "Cheesy Chicken Tamale Pie",
+  "link": "https://www.thekitchn.com/chicken-tamale-pie-23752740",
+  "serves": 6,
+  "imageUrls": [
+    "https://cdn.apartmenttherapy.info/image/upload/f_jpg,q_auto:eco,c_fill,g_auto,w_1500,ar_16:9/tk%2Fphoto%2F2025%2F10-2025%2F2025-10-chicken-tamale-pie%2Fchicken-tamale-pie-0",
+    "https://cdn.apartmenttherapy.info/image/upload/f_jpg,q_auto:eco,c_fill,g_auto,w_1500,ar_4:3/tk%2Fphoto%2F2025%2F10-2025%2F2025-10-chicken-tamale-pie%2Fchicken-tamale-pie-0",
+    "https://cdn.apartmenttherapy.info/image/upload/f_jpg,q_auto:eco,c_fill,g_auto,w_1500,ar_1:1/tk%2Fphoto%2F2025%2F10-2025%2F2025-10-chicken-tamale-pie%2Fchicken-tamale-pie-0"
+  ],
+  "ingredients": [
+    {
+      "id": -1,
+      "name": "unsalted butter",
+      "line": "3 tablespoons unsalted butter, divided",
+      "unit": "Tablespoon",
+      "quantity": 3.0,
+      "preparation": "divided",
+      "productId": 4,
+      "recipeId": null,
+      "mealId": null,
+      "product": {
+        "id": 4,
+        "productId": "712251",
+        "shopCode": "woolworths",
+        "link": "https://www.woolworths.com.au/shop/productdetails/712251/western-star-unsalted-butter-chef-s-choice",
+        "name": "Western Star Unsalted Butter Chef's Choice",
+        "quantity": 500,
+        "unit": "g",
+        "imgSmall": "https://cdn0.woolworths.media/content/wowproductimages/small/712251.jpg",
+        "imgLarge": "https://cdn0.woolworths.media/content/wowproductimages/large/712251.jpg"
+      }
+    },
+    {
+      "id": -1,
+      "name": "corn muffin mix",
+      "line": "1 (8.5-ounce) box corn muffin mix, such as Jiffy",
+      "unit": "Ounce",
+      "quantity": 1.0,
+      "preparation": "",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "egg",
+      "line": "1  large egg",
+      "unit": "Items",
+      "quantity": 1.0,
+      "preparation": "",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "sour cream",
+      "line": "1/2 cup sour cream, plus more for serving",
+      "unit": "Cup",
+      "quantity": 0.5,
+      "preparation": "",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "scallions",
+      "line": "4  medium scallions, thinly sliced (about 1/2 cup), plus more for garnish",
+      "unit": "Cup",
+      "quantity": 4.0,
+      "preparation": "thinly sliced",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "garlic",
+      "line": "2 cloves garlic, minced",
+      "unit": "Items",
+      "quantity": 2.0,
+      "preparation": "minced",
+      "productId": 2,
+      "recipeId": null,
+      "mealId": null,
+      "product": {
+        "id": 2,
+        "productId": "294517",
+        "shopCode": "woolworths",
+        "link": "https://www.woolworths.com.au/shop/productdetails/294517/la-famiglia-garlic-bread",
+        "name": "La Famiglia Garlic Bread",
+        "quantity": 1,
+        "unit": "Loaf",
+        "imgSmall": "https://cdn0.woolworths.media/content/wowproductimages/small/294517.jpg",
+        "imgLarge": "https://cdn0.woolworths.media/content/wowproductimages/large/294517.jpg"
+      }
+    },
+    {
+      "id": -1,
+      "name": "chili powder",
+      "line": "2 teaspoons chili powder",
+      "unit": "Teaspoon",
+      "quantity": 2.0,
+      "preparation": "",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "ground cumin",
+      "line": "1/2 teaspoon ground cumin",
+      "unit": "Teaspoon",
+      "quantity": 0.5,
+      "preparation": "",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "chicken",
+      "line": "3 cups shredded, cooked chicken (from 1/2 rotisserie chicken, about 10 ounces)",
+      "unit": "Cup",
+      "quantity": 3.0,
+      "preparation": "shredded, cooked",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "red enchilada sauce",
+      "line": "1 (10-ounce) can red enchilada sauce, or 1 1/4 cups homemade enchilada sauce",
+      "unit": "Ounce",
+      "quantity": 1.0,
+      "preparation": "",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "frozen corn kernels",
+      "line": "1 cup frozen corn kernels, preferably fire-roasted (do not thaw)",
+      "unit": "Cup",
+      "quantity": 1.0,
+      "preparation": "",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "pickled jalapeños",
+      "line": "1/2 cup drained sliced pickled jalapeños, coarsely chopped, plus more for serving",
+      "unit": "Cup",
+      "quantity": 0.5,
+      "preparation": "drained sliced, coarsely chopped",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "sharp cheddar cheese",
+      "line": "4 ounces sharp cheddar cheese, shredded (about 1 cup)",
+      "unit": "Ounce",
+      "quantity": 4.0,
+      "preparation": "shredded",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    },
+    {
+      "id": -1,
+      "name": "Monterey Jack cheese",
+      "line": "4 ounces shredded Monterey Jack cheese, shredded (about 1 cup)",
+      "unit": "Ounce",
+      "quantity": 4.0,
+      "preparation": "shredded",
+      "productId": -1,
+      "recipeId": null,
+      "mealId": null,
+      "product": null
+    }
+  ],
+  "basedOnRecipe": null,
+  "dateCreated": "2025-11-04T18:24:30.117818+11:00",
+  "createdById": 1,
+  "createdBy": {
+    "id": 1,
+    "displayName": "Snapshot Generator"
+  },
+  "dateHidden": null,
+  "hiddenById": null,
+  "hiddenBy": null
+}

tests/test_invitations_v2.py

@@ -46,7 +46,17 @@ class TestInvitationsV2(unittest.IsolatedAsyncioTestCase):
             json={"email": "invitee@test.com"},
         )
         assert r.status_code == 200, r.text
-        token = r.json()["token"]
+        body = r.json()
+        invite_link = body.get("invite_link")
+        assert isinstance(invite_link, str) and "/invitations/accept?token=" in invite_link
+
+        # Extract token from invite_link
+        from urllib.parse import urlparse, parse_qs
+
+        qs = parse_qs(urlparse(invite_link).query)
+        token_list = qs.get("token", [])
+        assert token_list and isinstance(token_list[0], str)
+        token = token_list[0]
         assert isinstance(token, str) and len(token) >= 16
 
         # Register invitee

tests/test_recipes_offline_expected.py

@@ -0,0 +1,96 @@
+import asyncio
+import json
+from pathlib import Path
+from types import SimpleNamespace
+
+import pytest
+
+from db import connect, create
+from recipes import parse_recipe_from_html
+
+
+SNAP_DIR = Path("tests/sample_files/recipes")
+
+
+def _iter_expected_files():
+    return sorted(SNAP_DIR.glob("*.recipe.json"))
+
+
+def _load_expected(path: Path):
+    # New-style file contains the model JSON directly or null
+    return json.loads(path.read_text())
+
+
+def _expected_subset(expected: dict):
+    # expected is serialized with by_alias=True, so image_urls is imageUrls
+    name = expected.get("name")
+    link = expected.get("link")
+    serves = expected.get("serves")
+    image_urls = expected.get("imageUrls") or []
+    ingredients = expected.get("ingredients") or []
+    ing_lines = [ing.get("line") for ing in ingredients]
+    return {
+        "name": name,
+        "link": link,
+        "serves": serves,
+        "image_urls": image_urls,
+        "ingredient_lines": ing_lines,
+    }
+
+
+def _actual_subset(model):
+    # Stable, assertion-friendly projection
+    return {
+        "name": model.name,
+        "link": model.link,
+        "serves": model.serves,
+        "image_urls": list(model.image_urls or []),
+        "ingredient_lines": [ing.line for ing in (model.ingredients or [])],
+    }
+
+
+@pytest.mark.parametrize("exp_path", _iter_expected_files())
+def test_offline_parse_matches_expected(exp_path: Path):
+    async def _run():
+        # exp_path is <slug>.recipe.json; Path.stem removes only the last suffix (".json"),
+        # leaving ".recipe" in the stem. Strip the trailing ".recipe" to get the HTML slug.
+        slug = exp_path.stem
+        if slug.endswith(".recipe"):
+            slug = slug[: -len(".recipe")]
+        expected = _load_expected(exp_path)
+
+        html_path = SNAP_DIR / f"{slug}.html"
+        assert html_path.exists(), f"snapshot HTML missing for {slug}"
+        html = html_path.read_text()
+
+        # Discover base URL from meta if available
+        meta_path = SNAP_DIR / f"{slug}.meta.json"
+        base_url = slug
+        if meta_path.exists():
+            try:
+                meta = json.loads(meta_path.read_text())
+                base_url = meta.get("final_url") or meta.get("url") or slug
+            except Exception:
+                pass
+
+        created_by = SimpleNamespace(id=1, display_name="Snapshot Generator")
+        conn = await connect()
+        await create(conn)
+        try:
+            model = await parse_recipe_from_html(conn, created_by, base_url, html)
+        finally:
+            await conn.close()
+
+        if expected is None:
+            assert model is None, f"expected no parse for {slug}, but got a model"
+            return
+
+        assert model is not None, f"expected a model for {slug}, got None"
+
+        exp_sub = _expected_subset(expected)
+        act_sub = _actual_subset(model)
+
+        # Compare key fields; ingredients compare by line only for stability
+        assert act_sub == exp_sub
+
+    asyncio.run(_run())

tests/test_recipes_parse_from_url_integration_v2.py

@@ -50,6 +50,8 @@ class TestRecipesParseFromUrlIntegrationV2(unittest.IsolatedAsyncioTestCase):
                 self.text = text
 
         class DummyClient:
+            def __init__(self, *args, **kwargs):
+                pass
             async def __aenter__(self):
                 return self
 
@@ -90,6 +92,8 @@ class TestRecipesParseFromUrlIntegrationV2(unittest.IsolatedAsyncioTestCase):
                 self.text = text
 
         class DummyClient:
+            def __init__(self, *args, **kwargs):
+                pass
             async def __aenter__(self):
                 return self
 
@@ -107,9 +111,9 @@ class TestRecipesParseFromUrlIntegrationV2(unittest.IsolatedAsyncioTestCase):
                 headers=self.headers,
                 json={"url": SAMPLE_URL},
             )
-            assert r.status_code == 404, r.text
+            assert r.status_code == 422, r.text
             body = r.json()
-            assert body.get("status") == 404
+            assert body.get("status") == 422
         finally:
             scraping.httpx.AsyncClient = orig_client
 
@@ -127,6 +131,8 @@ class TestRecipesParseFromUrlIntegrationV2(unittest.IsolatedAsyncioTestCase):
                 self.text = text
 
         class DummyClient:
+            def __init__(self, *args, **kwargs):
+                pass
             async def __aenter__(self):
                 return self
 

tests/test_recipes_parse_from_url_v2.py

@@ -40,11 +40,11 @@ class TestRecipesParseFromUrlV2(unittest.IsolatedAsyncioTestCase):
         # Monkeypatch scraper to avoid network
         import recipes.scraping as scraping
 
-        async def fake_scrape(url: str):
+        async def fake_scrape(url: str, log=None, dump_dir=None):
             assert url == "https://example.com/recipe"
             return {"@type": "Recipe", "name": "Example", "recipeIngredient": ["2 eggs"]}
 
-        async def fake_scrape_none(url: str):
+        async def fake_scrape_none(url: str, log=None, dump_dir=None):
             return None
 
         # Success case
@@ -67,7 +67,7 @@ class TestRecipesParseFromUrlV2(unittest.IsolatedAsyncioTestCase):
         finally:
             recipes_pkg._scrape_recipe_ldata = orig
 
-        # Not found case
+        # Unprocessable (parse failure) case
         recipes_pkg._scrape_recipe_ldata = fake_scrape_none
         try:
             r2 = self.client.post(
@@ -75,8 +75,8 @@ class TestRecipesParseFromUrlV2(unittest.IsolatedAsyncioTestCase):
                 headers=self.headers,
                 json={"url": "https://example.com/missing"},
             )
-            assert r2.status_code == 404, r2.text
+            assert r2.status_code == 422, r2.text
             pb = r2.json()
-            assert pb.get("status") == 404
+            assert pb.get("status") == 422
         finally:
             recipes_pkg._scrape_recipe_ldata = orig

tighten-api-spec.md

@@ -1,113 +0,0 @@
-# Tighten Public API Nullability
-
-Make the external API more consistent and predictable by eliminating unnecessary nulls (nullable fields) in models and responses. This plan lists concrete, low-risk changes, their rationale, and exact files/lines to modify. Each task is checkable and includes verification steps.
-
-Date: 2025-10-21
-
-## Principles
-
-- Prefer non-nullable types where domain requires a value (DB constraints, logic always sets it).
-- Keep optional only when a field truly may be absent by design (e.g., hiddenBy, prevCursor when first page).
-- Preserve backward compatibility where feasible. When changing response shapes, update tests and OpenAPI examples.
-- Pydantic already excludes None on serialization in some places; we still tighten model types to improve OpenAPI and client SDKs.
-
-## Quick wins (low risk)
-
-- [x] Page.total is non-nullable with default 0
-  - Why: Pagination always returns a number. Current `Optional[int]` leads to `null` in schema and potential nulls in responses.
-  - Change: in `common.py`, change `total: Optional[int]` to `total: int = Field(default=0, description="Total count")`.
-  - Verify:
-    - [ ] mypy/pyright/ruff pass.
-    - [ ] Tests for persons/recipes list remain green.
-    - [ ] OpenAPI shows `total` as `integer` (no anyOf null).
-
-- [x] Recipe.created_by_id non-nullable
-  - Why: DB enforces NOT NULL and creation flow always sets it.
-  - Change: in `recipes/models.py` set `created_by_id: int` (remove Optional). Keep `created_by: Optional[Person]` (hydrated field).
-  - Knock-on: `api/recipes.load_full_recipe` can drop the `if r.created_by_id is not None` guard.
-  - Verify:
-    - [ ] All recipe-related tests green.
-    - [ ] OpenAPI for Recipe shows `createdById` required.
-
-- [x] Product.raw_data excluded from public schema
-  - Why: Internal/testing helper currently typed as `Optional[dict]` -> visible as nullable in OpenAPI.
-  - Change: in `products/models.py` use a `PrivateAttr` (with a `raw_data` property) so it stays out of the schema without creating Input/Output variants.
-  - Verify:
-    - [ ] Product schema in OpenAPI does not include `rawData`.
-    - [ ] Tests referencing raw_data still pass (field remains available in code, excluded from schema/response).
-
-## Shopping models and endpoints
-
-`ShoppingListItem` currently represents two cases (ingredient request vs meal request), so several linking fields are nullable. We can reduce nulls in the public API by introducing outward-facing variants while keeping the DB model as-is.
-
-- [ ] Optional: Introduce discriminated union for API returns (medium change)
-  - Rationale: Return `oneOf` in OpenAPI with variant-specific required fields; eliminates irrelevant nullable properties for each variant.
-  - Approach (sketch):
-    - Define `ListIngredientItem` and `RequestedMealItem` pydantic models with a `kind` discriminator.
-    - Update `api/shopping.py` response models (CurrentShoppingList and PurchasedShoppingList) to use `Union[ListIngredientItem, RequestedMealItem]` for item arrays.
-    - Conversion helpers in `shopping` module to map from `ShoppingListItem` DB model to the outward union.
-  - Verify:
-  - [ ] Update v2 tests to accept the new shape while preserving field meanings.
-    - [ ] OpenAPI shows `oneOf` for shopping list items.
-
-- [ ] Tighten invariants without breaking shape (keep for now)
-  - Keep model but document invariants (only one of ingredient_id/meal_id required; recipe_id optional when meal request). Repository already validates; consider pydantic validators later.
-
-## Persons and Recipes listings
-
-- [ ] Ensure total is populated for Person and Recipe lists
-  - Already implemented in `api/persons.py` and `api/recipes.py` using repository `count_*` helpers. After making `Page.total` non-nullable, nothing else required.
-
-## Authentication dependency
-
-- [x] Provide strict non-null person dependency for protected endpoints
-  - Why: Many endpoints assume an authenticated user; typing as non-null simplifies signatures and docs.
-  - Change:
-    - Consolidated on a single dependency `cookie_person` (strict): `Cookie(..., alias="user_id")` and raises 401 if missing/unknown.
-    - Removed `require_cookie_person` and switched usages to `cookie_person`.
-  - Verify:
-    - [x] Endpoint signatures updated.
-    - [x] Unauthorized behavior covered by handlers; overall tests still pass.
-
-## File-by-file checklist (edits)
-
-- [x] `common.py`
-  - [x] Page.total -> `int = Field(default=0, ...)`
-
-- [x] `recipes/models.py`
-  - [x] `created_by_id: int`
-
-- [x] `api/recipes.py`
-  - [x] In `load_full_recipe`, set `r.created_by = await persons.get_by_id(conn, r.created_by_id)` unconditionally.
-
-- [x] `products/models.py`
-  - [x] `raw_data` moved to `PrivateAttr` with property; kept out of schema.
-
-- [x] `api/deps.py`
-  - [x] Added `require_cookie_person(...) -> persons.Person` that raises 401.
-  - [x] Updated protected endpoints to depend on `require_cookie_person`.
-
-- [ ] (Optional) Shopping API union types
-  - [x] Add outward-facing union models and mapping helpers.
-  - [x] Update `api/shopping.py` response models to use union.
-
-## Tests and validation
-
-- [x] Run format/lint/typecheck
-  - make format && make lint && make typecheck
-- [x] Run tests
-  - make test
-- [x] Export OpenAPI and inspect schema
-  - make openapi (confirm: Recipe.createdById required; Product has single schema; Page.total non-nullable; cookie param required on protected ops.)
-
-## Rollout notes
-
-- API change in OpenAPI (non-null total; createdById required). Client SDKs generated from the spec may need re-gen. Runtime remains compatible because server fills these fields.
-- Even without the optional union refactor, the quick wins remove several unnecessary nulls.
-
-## Acceptance criteria
-
-- OpenAPI no longer marks Page.total and Recipe.createdById as nullable.
-- Product.rawData does not appear in the public schema.
-- All tests pass; no runtime regressions.
-- Optional: Shopping list items use `oneOf` variants.