Every carrier invoice was checked by hand. Only the variances needed a person.

Manual reconciliation was slow, and the alternative kept failing

Invoices arrived from multiple carriers, each with its own layout, terminology and structure: multi-page, table-heavy, and carrying image-based logos that defeat plain text extraction. Someone validated billing amounts circuit by circuit against the resell reports and investigated every variance over $100 before approving.

The reason it had not already been automated is that invoice extraction fails in a dozen unglamorous ways, most of which only appear at volume:

• No shared structure across vendors. Template extraction works until a carrier redesigns their invoice, and then it silently returns the wrong field rather than failing loudly.
• Page breaks slice table rows. A circuit's ID lands on one page and its charge on the next, which naive row parsing reads as two incomplete records.
• The matching key has no fixed shape. Circuit IDs appear in wildly different alphanumeric formats. A pattern finds every candidate without knowing which is correct, and there are usually several.
• The carrier name is often only in the logo. These are hybrid documents (branding rendered as an image, line items as text), and vendor context drives everything downstream.
• OCR misreads look like data. Slashes read as digits, cells merge. These produce plausible values rather than obvious errors, which is what makes them dangerous.
• A model asked to read an unusual invoice may infer rather than admit uncertainty. A confidently wrong amount is worse than a blank one.

Neither patterns nor the model can do this alone

The tempting design is one or the other. Regex is precise and brittle: it finds every possible Circuit ID and cannot tell which is right. A language model reads context but misses rare alphanumeric formats it has no reason to expect, and will guess when the document is ambiguous.

So the two are given the jobs they are actually good at. The system scans the document for pattern matches, collects every candidate, and hands those candidates, with the raw text and the layout signals, to the model, which decides. The regex supplies recall; the model supplies the judgement.

Full automation was never the target either. Around 90% automated, with human review reserved for low-confidence cases, is the honest number, and it is the one that holds data quality.

Extraction as a service, reconciliation as a workflow

The pipeline splits in two, deliberately. A dedicated extraction service does the PDF work and returns structured JSON. The workflow engine picks it up from there, joining against the resell data pulled from Synapse, matching on normalised Circuit ID, calculating variance per circuit and evaluating it against a single threshold.

Within ±$100 the record approves automatically. Beyond it, the record is flagged and routed to a reviewer with the specific mismatch surfaced. The output is an exception list rather than a spreadsheet to re-check.

Every manual correction is treated as a signal that the prompt is insufficient: the layout is inspected to find why extraction failed, and the prompt and the carrier pattern store are updated. Each update has to stay backward compatible, so regression checking is part of the loop rather than an afterthought.