Feature/statement parsing #1

Merged

TehRiehlDeal merged 8 commits from feature/statement-parsing into main 2026-05-27 15:14:07 -07:00

Conversation 0 Commits 8 Files Changed 47

+5133 -12

TehRiehlDeal commented 2026-05-27 15:05:02 -07:00

Owner

Copy Link

Copy Source

No description provided.

TehRiehlDeal added 8 commits 2026-05-27 15:05:02 -07:00

Add bulk statement-import feature for transactions ... a8a47e38c1

Lets a user upload a CSV, OFX/QFX, or PDF bank statement, parses the
transactions, flags duplicates and possible transfers against existing
records, and bulk-creates the accepted rows after a final read-only
confirmation step.

Backend
- New `statements` module with CSV (papaparse), OFX/QFX (node-ofx-parser),
  and PDF (pdf-parse) parsers behind a `StatementParser` strategy
  interface; format detected by content sniffing + extension.
- `DuplicateDetectorService` checks FITID/externalId exact matches first,
  then a date+amount+description Jaro-Winkler heuristic, then cross-account
  transfer pairing.
- New `POST /statements/parse` (multipart, in-memory, 10MB cap) returns the
  parsed preview without writing, including per-row `status` (`new`,
  `duplicate`, `needs_review`, `possible_transfer`) and any `needsMapping`
  payload when CSV headers are unrecognized.
- `POST /transactions/bulk` accepts up to 500 rows, chunks them 50 at a
  time inside `prisma.$transaction`, applies balance deltas, and writes a
  single `ActivityLog` row per chunk instead of one per transaction.
- Schema: nullable `external_id` column on `Transaction` plus composite
  indexes on `(account_id, external_id)` and `(account_id, date)` for fast
  dedupe-window queries. Not encrypted — it's an opaque bank ID used as a
  lookup key.

Frontend
- `ImportStatementDialog` runs a 4-step wizard: Upload → Column Mapping (if
  needed) → Review (editable table with duplicate/transfer badges) →
  Confirm (read-only summary with projected per-account balance impact and
  count-bearing primary button). The Confirm step gates the actual write,
  and Back to Review preserves all edit/checkbox state.
- New `bulkCreateTransactions` action on the transactions store.
- "Import Statement" button added next to Export on the Transactions page,
  with a success toast and a refresh of the transactions + accounts stores.

Tests
- 306 backend tests (29 suites), 195 frontend tests (31 suites), all green.
- Fixtures under `test/fixtures/statements/` cover three CSV sign
  conventions (signed-amount, debit/credit, credit-card), OFX 1.x SGML,
  OFX 2.x XML, and a credit-card QFX with the CCSTMTRS branch.

Versions bumped to 0.4.0 on both packages per the lockstep rule.

NOTE: the Prisma migration in
`prisma/migrations/20260527203542_add_transaction_external_id/` still
needs to be applied to the live database with `prisma migrate deploy` —
the DB at 10.0.3.82 wasn't reachable from the dev environment.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Move pnpm onlyBuiltDependencies into pnpm-workspace.yaml ... cf7e8a5d3f

pnpm 10.x no longer reads the "pnpm" key in package.json and now hard-fails
the install with ERR_PNPM_IGNORED_BUILDS when build scripts are ignored.
Migrate the existing allowlist to pnpm-workspace.yaml and add msw (pulled
in transitively by vitest 4.x).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Keep onlyBuiltDependencies in both package.json and pnpm-workspace.yaml ... a551ec06b4

The previous commit relied on pnpm-workspace.yaml alone, but that field is
a pnpm 10+ feature. CI and the Dockerfiles both pin pnpm 9 (via corepack /
pnpm/action-setup), and pnpm 9 only reads onlyBuiltDependencies from the
package.json "pnpm" field. Without it, ERR_PNPM_IGNORED_BUILDS blocked the
install. Keep both definitions in sync: pnpm 10 reads the workspace file
(and emits a benign warning about the package.json field), pnpm 9 reads
package.json.

Also includes msw, a new transitive of vitest 4.x that now needs the
explicit allow.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Move onlyBuiltDependencies into .npmrc so CI's pnpm 10 honors it ... ca371d93c3

pnpm-workspace.yaml's onlyBuiltDependencies (pnpm 10+) wasn't being read
in CI even though that's where pnpm 10 docs say it should live — the
install still bailed with ERR_PNPM_IGNORED_BUILDS. .npmrc is the long-
established, version-agnostic location pnpm honors regardless of the
installed major. Add the allowlist there.

The duplicate entries in pnpm-workspace.yaml and package.json#pnpm stay
in place — they're harmless and serve as documentation for anyone running
older pnpm.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Pin pnpm 10.33.0 in CI so onlyBuiltDependencies is honored ... aefa5abf95

pnpm/action-setup@v4 with `version: 9` was actually resolving to pnpm
10.x in CI (confirmed by the pnpm-10-only WARN about the package.json
"pnpm" field and by the "Verifying lockfile against supply-chain
policies" step). pnpm 10 reads onlyBuiltDependencies from
pnpm-workspace.yaml — our config has been correct there since the first
fix — but whichever 10.x the action picked apparently didn't, so every
install failed with ERR_PNPM_IGNORED_BUILDS.

Pin to 10.33.0 explicitly. That's the version where I verified locally
that pnpm-workspace.yaml's onlyBuiltDependencies is read correctly and
the install completes cleanly.

Dockerfiles still pin pnpm@9 via corepack, which reads the legacy
package.json#pnpm.onlyBuiltDependencies (still in place), so production
image builds remain unaffected.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Pin CI to pnpm 9.14.4 to dodge the strict ERR_PNPM_IGNORED_BUILDS gate ... a79ee5f479

The Install dependencies step has been failing with
ERR_PNPM_IGNORED_BUILDS no matter where I put the onlyBuiltDependencies
allowlist (package.json#pnpm, pnpm-workspace.yaml, project .npmrc) and
no matter which pnpm 10.x is installed. The strict build-script gate was
introduced in pnpm 9.15 / 10.0; pnpm 9.14.4 predates it and just runs
postinstall scripts the way pnpm has for years — matching what the
Dockerfiles already do via corepack `pnpm@9`.

Also reverts the short-lived `--ignore-scripts` install workaround,
which skipped @prisma/client's postinstall and left runtime files
missing so `prisma generate` couldn't complete.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Replace pnpm/action-setup with corepack to actually pin the pnpm version ... 6a6d629bcf

pnpm/action-setup@v4 was ignoring its `version` input on this runner and
installing pnpm 10.x no matter what value we passed. That's why every
attempt to land an onlyBuiltDependencies allowlist failed —
ERR_PNPM_IGNORED_BUILDS kept blocking the install.

Switch to corepack, which Node 22 ships with, and `corepack prepare
pnpm@9.14.4 --activate`. Same mechanism the Dockerfiles use. Adds an
explicit `pnpm --version` line so future CI runs make the actual
installed version visible.

Dropped `cache: pnpm` from actions/setup-node@v4 since pnpm isn't on
PATH yet at that step — the pnpm store cache wasn't doing much for us
on first runs anyway.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Fix backend lint errors in the statement-parser code ... 9f0af6bfb8

The new statements module was hitting type-safety errors from
typescript-eslint's recommendedTypeChecked config:

- parse-statement.dto.ts: tighten the Transform decorator's signature so
  the JSON.parse path returns a typed object or undefined, not `any`.
- duplicate-detector.service.ts: drop the unused
  WEAK_DESCRIPTION_SIMILARITY constant left over from earlier logic.
- csv.parser.ts and ofx.parser.ts: the parse() methods were `async`
  without any `await` (require-await). Convert them to non-async
  functions that return a Promise — wrap parseSync() in a try/catch so
  thrown errors still surface as rejected promises for spec callers
  that use `.rejects.toThrow()`.
- ofx.parser.ts: replace `require('node-ofx-parser')` with a typed
  `import * as ofxLib`, backed by a hand-written declaration file at
  src/types/node-ofx-parser.d.ts that captures the bank + credit-card
  transaction shapes we consume.
- pdf.parser.ts: import the typed `PDFParse` class from pdf-parse
  directly instead of lazy-requiring it as `any`. Keep the test seam
  but back it with a typed PdfTextExtractor function instead of the
  ad-hoc `any` shape.

Also pulls in the prettier reformat that `eslint --fix` produced across
the touched files and their specs.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

TehRiehlDeal merged commit 0284923d5e into main 2026-05-27 15:14:07 -07:00

TehRiehlDeal deleted branch feature/statement-parsing 2026-05-27 15:14:07 -07:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

TehRiehlDeal (Kevin Riehl)

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: TehRiehlDeal/TehRiehlBudget#1