Dev sync: new pages appear while I work

A new code page should show up in goatboat (and so in /tests, feeds and the tree) as soon as I save the file, with no commit or push. This doc is the design. It's a plan to review, not built yet.

The problem

Goatboat knows about code pages (src/app/**/page.tsx in the site) only through the routes sync. Today that runs in a GitHub Action on pushes to main. So the loop for a new page is: write it, commit it (the hook stamps its id), push, wait for the Action, then look. For a page I'm only trying out, that's far too slow.

Running the existing sync by hand from a dev checkout isn't safe. The sync does more than add pages, and the database is shared and live:

What a full sync doesWhy it's unsafe from a working tree
Creates drafts for new routes (and placeholder drafts for missing parents)Fine
Adopts an existing row by idFine for drafts, not for live rows
Moves a row when its folder moved, and leaves redirectsA rename in a half-finished branch redirects live URLs before the code deploys
Refreshes file and headings on existing rowsLive pages would get headings the deployed code doesn't have yet
Flags every code page not in my tree as missingMissing pages are hidden everywhere the public can look. A stale branch hides live pages

The idea

A second, much smaller sync for development: additive only.

The dev sync can create rows, and it can only touch rows that are still drafts.

Everything else (moves, redirects, flagging missing, anything on a published page) stays with the full sync in the GitHub Action on main. The Action doesn't change at all.

That one rule is what keeps the two from conflicting. The dev sync never writes anything the Action relies on for live pages, so neither can undo or fight the other on a published page.

What stays the same

  • The GitHub Action, its workflow file and its trigger (push to main touching src/app/**).
  • The full sync's behaviour, and the /api/sync endpoint for callers who don't ask for the new mode.
  • The // goatboat: <uuid> first line, and the pre-commit hook that stamps it.
  • Goatboat owns everything about a page except its existence and its place in the code. A sync never overwrites a title, description or any other metadata.

Goatboat: an additive mode on /api/sync

Request

The body gets an optional mode. Without it, the request is a full sync exactly as today.

json
{ "mode": "additive", "approutes": [ { "id": "…", "path": "/tests/x", "file": "src/app/tests/x/page.tsx", "toc": [] } ] }

mode is "full" (the default) or "additive". The route list has the same shape and the same validation in both modes.

What additive does, route by route

Routes are handled shallowest first, as the full sync does.

SituationAdditive result
No row with this id, and nothing at this pathCreate a draft, with its headings. Create placeholder drafts for any missing parents. Reported as created
A row with this id exists and is a draft code pageRefresh file and headings, and clear a missing flag. Reported as updated or restored
A row with this id exists and is a draft db row (made in goatboat by hand)Adopt it as a code page. Reported as adopted
A row with this id exists and is published (any source)Leave it alone. Reported as skipped
A row with this id exists at a different path (the folder moved)Leave it alone. Reported as wouldMove; the full sync does the move
Another row already holds this path under a different idLeave it alone. Reported as conflicts. The rest of the list still goes through
A code page in goatboat isn't in the listNothing. Additive never flags missing

Two points about conflicts:

  • The full sync fails the whole run on a conflict. Additive reports it and carries on, so one clash doesn't stop other new pages from appearing. The Action still fails loudly on the same clash at push time, which is where it should block.
  • Placeholders count as creates and are drafts, so they follow the same rule.

Response

The same JSON as today, plus the new groups. A full sync never returns skipped, wouldMove or conflicts. Every group is a list of paths, and empty groups are fine, as now.

Safety properties

  • All changes in one run happen in one transaction, as now.
  • Nothing is ever deleted.
  • A published row is never written to. This is checked in the code that writes, not just by the route's logic, so a future bug can't quietly widen it.
  • The same token and the same Authorization check as the full sync. The mode is something my own script asks for, so a separate token would add no real protection for a single-owner setup.

Tests (in tests/, one thing per test)

Each has the usual // Checks: and // Expected: comment.

  • Additive creates a draft for a new route.
  • Additive creates placeholder drafts for missing parents.
  • Additive doesn't flag a code page as missing when it isn't listed.
  • Additive refreshes headings on a draft code page.
  • Additive clears the missing flag on a draft code page that is listed again.
  • Additive leaves a published code page untouched, even when its headings differ.
  • Additive doesn't move a row whose folder changed, and reports wouldMove.
  • Additive reports a path conflict and still creates the other routes.
  • Additive adopts a draft db row that has the same id.
  • A full sync still behaves as before (the existing sync tests keep passing unchanged).

The site: one command, then automatic

npm run routes

One command that does both steps:

  1. Stamp any page that has no id (the same script the pre-commit hook uses, without staging).
  2. Send the route list to goatboat with mode: "additive" and print the result.

It uses the same Bitwarden token lookup as routes:sync. It's safe to run any time, as often as I like. Running it twice in a row changes nothing the second time.

routes:sync stays as it is, for a full sync by hand when I really want one.

Automatic while npm run dev is running

A small watcher runs alongside the dev server:

  • On start, it does one routes run, so goatboat is current when I sit down.
  • It watches src/app for page.tsx files being created, deleted, renamed or edited (headings change the list), waits half a second after the last event so editors can finish writing, then runs the same stamp-and-send.
  • It remembers a fingerprint of the last list it sent and does nothing when the list hasn't changed. This also stops it from reacting to its own stamping.
  • Runs never overlap: an event during a run queues one more run after it.
  • If goatboat is down, or the request fails, it prints one short line and keeps going. A sync failure must never stop the dev server.
  • It needs the sync token, which npm run dev fetches from Bitwarden once at start, the same way it fetches the read token. The token is never printed.

Why stamping from a watcher is OK

Writing an id into a file I might have open could clash with an unsaved buffer. It doesn't in practice: only pages with no id are stamped, and every existing page has one. A new file either starts empty (nothing typed yet) or arrives with content already saved, so the editor reloads it cleanly. I'd still wait the half second before stamping.

How it works with the GitHub Action

The Action runs a full sync on every push to main from a clean checkout. The two syncs touch different things:

Page stateDev syncThe Action
New page, not committedCreates the draftDoesn't know it yet
Same page, committed and pushedDoes nothing newAdopts the row by id (the stamped id is the same one the dev sync used), refreshes it, and it's tracked from here on
Published pageNever touches itDoes everything, as now
Folder renamedReports wouldMoveMoves it, with redirects
File deletedDoes nothingFlags it missing

The stamped id is what ties them together: the dev sync uses the id in the file, and the commit includes that file, so the Action finds the same row.

Edge cases

  • A page I never commit. Its row stays a hidden draft. The next full sync flags it missing (it isn't in main). The next dev run clears that flag again if the file is still in my tree. The flapping only ever involves drafts, and nobody but me sees them. It disappears when I commit the page or delete the row in goatboat.
  • Rename before the first commit. The row keeps its old path until the commit, then the full sync moves it. In the meantime wouldMove shows in the output, and the new path doesn't appear in /tests until then. This is the one place where "pops up instantly" doesn't hold. I'd rather accept that than let the dev sync create redirects.
  • A copied page. The stamp script already gives copies a new id, so a copy shows up as its own new row.
  • A path already taken in goatboat (for example a page I made by hand in goatboat, with a different id). It's reported as a conflict and left alone. The full sync would also refuse it, so the fix is the same either way: delete the goatboat row, or use its id.
  • Two machines. Both send additive requests, which only create or touch drafts, so they can't corrupt each other.

What I'm deliberately not doing

  • No moves, redirects or missing flags from dev. That's the Action's job.
  • No new auth or token scope.
  • No change to the Action, the hook, the stamp format or the database schema. There's no migration.
  • No sync from the production site or from Vercel builds.
  • No syncing of metadata. Goatboat stays the owner of titles, descriptions and the rest.

Build order

Each step works on its own and ends with something I can use.

  1. Goatboat additive mode. Schema change to the request, the mode in syncAppRoutes, the tests above, and the sync endpoint's description on goatboat's home page. Done when a request with mode: "additive" behaves as the table says and the old sync tests still pass.
  2. npm run routes and --additive on the sync script. Done when I add a page, run one command, and see it in /tests.
  3. The dev watcher. Done when saving a new page.tsx makes it appear in /tests within a couple of seconds, with no command.

Questions I'd still like answered

  • Is half a second the right wait, or do I want it faster even if it syncs mid-typing now and then? (Syncing early is harmless: the list is re-sent on the next change and nothing duplicates.)
  • Should the watcher log every run, or only when it created or changed something? My pick: only when something happened, plus errors.