Key figures
- Differently formatted supplier quotes
- Thousands
- LLM spend in the re‑architected pipeline
- Capped
- Recovery sweep in the re‑architected pipeline
- 5 min
The brief
The challenge
An AI pricing product for construction contractors, locked into a no-code WeWeb front, with thousands of differently formatted supplier quotes to turn into structured prices, and supplier consultations still tracked by hand.
What we built
The extraction platform that turns customers' supplier quotes into validated, structured prices; the foundation of the FastAPI backend-for-frontend that let Bulq.ai move off WeWeb customer by customer; and S2C, an automated supplier-consultation backend, all designed for Bulq.ai's move to its own Google Cloud.
Outcome
An owned, tested platform: a re-architected extraction pipeline built to recover from failures on its own and keep LLM spend bounded, with human review only where judgement is needed, plus a tested backend for automated supplier consultations.
Related services: Full-stack developmentData collection and analysisCloud infrastructure
Expertise
What it took.
Data engineering
- PostgreSQL as the single source of truth for each document's lifecycle: every file registered once by content hash and claimed by compare-and-set, so a duplicate or stale task writes nothing.
- LLM cost ceilings per document and per day, stored in the data, plus leases and a five-minute sweeper that recover lost work.
- Entity resolution for suppliers and jobsites by trigram similarity, and a review grid kept in sync with Google Sheets.
Software engineering
- A tested Flask extraction API and a React app for human validation in up to three steps, with quotes whose lines already balance validated automatically.
- The foundation of a FastAPI backend-for-frontend: httpOnly-cookie auth and enterprise SSO (Bulq.ai's team has since moved login to Zitadel), role-and-ownership authorization, signed direct-to-storage uploads, and a repository layer built so the data backend could move from Supabase to direct PostgreSQL, the setup the Cloud Run deployments now use.
- S2C: a tested backend with explicit state machines, optimistic concurrency, a transactional email outbox and a supplier-portal API behind hashed personal links.
Cloud & infrastructure
- Extraction and collection workloads integrated into the staging environment of Bulq.ai's OpenTofu-managed GCP platform, with IAM-authenticated database access through proxy sidecars and least-privilege bindings.
- A Cloud Tasks queue paced to its consumer, chosen over Pub/Sub in an architecture decision record, and a collection job sized from 30 days of production measurements.
- An interactive infrastructure map, kept in sync with the OpenTofu code by a CI drift check and set up to be served behind Identity-Aware Proxy.
Project 01
Extraction platform
Flask · React · Cloud Run · Cloud Tasks · LLMs
The operations app behind Bulq.ai's data. It connects to each customer's SharePoint, OneDrive, Google Drive or on-premise shares, runs syncs on a schedule or on demand, follows every quote through LLM extraction and puts it through human review, in up to three steps, before it reaches search. DataGem wrote nearly all of it, then re-architected the pipeline (V3, shown here, built and tested ahead of its rollout) around PostgreSQL as the single source of truth.
- Goal
- Industrialise the intake of thousands of differently formatted supplier quotes, with only verified data reaching search.
- Highlights
- One Cloud Task per document, compare-and-set claims and up to three passes per file
- Per-document and daily LLM cost ceilings
- Duplicate suppliers and jobsites suggested by trigram similarity
- Line-by-line validation next to the source document
- Stack
- Python
- Flask
- React 19
- PostgreSQL + RLS
- Cloud Tasks
- Cloud Run Jobs
- Google Sheets API
Project 02
Main app: off WeWeb, onto FastAPI
FastAPI backend-for-frontend (Next.js front by Bulq.ai's team)
DataGem designed and built the foundation of the backend-for-frontend behind Bulq.ai's new React app: an auth proxy that kept tokens in httpOnly cookies, with enterprise SAML SSO (Bulq.ai's team has since moved login to Zitadel); role-and-ownership authorization; the first business APIs (price statistics, jobsites, quotes, companies); signed direct-to-storage uploads; and the per-company switch that let customers leave the WeWeb app one at a time. The Next.js interface is the work of Bulq.ai's in-house team.
- Goal
- Replace the no-code WeWeb front with an owned, tested stack, without a big-bang migration.
- Highlights
- Repository layer that moves the data backend from Supabase to direct PostgreSQL by configuration
- Price statistics with median and MAD-based outlier detection
- OIDC service-to-service calls to the search microservice
- Stack
- Python
- FastAPI
- PostgreSQL
- Supabase
- Cloud Storage
Project 03
S2C: automated supplier consultation
FastAPI · PostgreSQL RLS · Cloud Tasks · Brevo
A backend built to automate what buyers still track by hand: versioned tender documents, AI lot-splitting, supplier-recommendation and conformity runs (executed by Bulq.ai's AI service, orchestrated by this backend), templated invitations with personal links, a supplier-portal API for questions and offers per lot, offer comparison and award. The buyer and supplier screens come next.
- Goal
- Take supplier consultations from a manual kanban to an auditable, automated pipeline from tender documents to award.
- Highlights
- Row-level security that scopes data to each tenant, with supplier access only through dedicated SQL routines
- AI runs dispatched through Cloud Tasks with leases, progress and recovery sweeps
- Email outbox with a shared daily quota and delivery tracking
- Supplier links stored only as SHA-256 hashes
- Stack
- FastAPI
- PostgreSQL + RLS
- Cloud Tasks
- Cloud Scheduler
- Brevo
- Cloud Storage
Project 04
Cloud platform integration
OpenTofu · Google Cloud
Bulq.ai's infrastructure team rebuilt its Google Cloud estate as OpenTofu code. DataGem integrated the extraction and data-collection workloads into its staging environment to the same keyless, least-privilege standard, and built an interactive infrastructure map of the platform that a CI check keeps in step with the code.
- Goal
- Run DataGem's workloads on the client's hardened platform without weakening it.
- Highlights
- Cloud Tasks queue paced to its consumer; collection job sized from production measurements
- IAM-authenticated database access through proxy sidecars
- Identity-Aware Proxy option added to the shared entry-point module
- Infrastructure map checked against the code in CI, packaged to deploy by image digest behind IAP
- Stack
- OpenTofu
- Cloud Run
- Cloud SQL
- Cloud Tasks
- IAP
- GitHub Actions
