Delivering a multimodal document-processing PoC

The problem

The document-processing PoC handled both images and vector and text information embedded in PDFs. Making it usable in a customer environment still required organizational identity, per-user workspaces, secret management, model connectivity, and service operations to work as one delivery unit.

Scope I owned

• Took over the existing image-capable implementation and owned the application layer needed for customer delivery
• Introduced OIDC through an organizational identity provider, bound the existing per-user workspace isolation to organizational identity, and implemented managed-identity secret retrieval and service startup configuration
• Restricted LLM API traffic to an approved resource and made model-name resolution safe across environments
• Isolated certificate, reverse-proxy, and model-connectivity failures by layer, then worked across customer and platform boundaries to open the path

Validation with real data

• Detected a customer-visible output regression even though the unit suite was green, then used controlled comparisons across code and data revisions to identify PDF text segmentation as the cause
• Added the fix and regression coverage, redeployed it, and rechecked the structured output against real data
• Identified a mismatch between a large LLM request and the assigned quota, then reran the browser workflow end to end after quota adjustment

Performance and requirements in practice

• Wait times: Three users running the same input were queued serially, with waits of up to 75 minutes. I split the cause into serialized processing and per-user reference caches, first added parallel execution (three at a time), then proposed to the research lead a cache shared across users and keyed by a hash of the content. We agreed and I implemented it. With the shared cache, the same input went from about an hour to about four minutes, with zero LLM calls
• Confirming the purpose before building: For a “masking” request, I asked in the first meeting whether the goal was to keep content away from the LLM or away from people. Only after confirming it was about what people see did I build the UI and API, avoiding a rebuild
• Keeping fixes minimal: After shipping provisional acceptance criteria and a working screen early, two people said they could not find the results. Instead of redesigning the navigation, a minimal change that put results first resolved it

Approach

• Connected an existing implementation using a VLM API for image input and primarily vector and text analysis for PDFs to a customer-operable delivery environment
• Wired organizational identity, managed-identity secret retrieval, and the LLM API endpoint and model configuration to fail closed
• Traced a post-test output regression to PDF text segmentation through controlled real-data comparisons, then added the fix and regression coverage
• Identified a mismatch between LLM request size and quota, then revalidated the end-to-end browser workflow after quota adjustment
• When several users ran the same input, serialized waits reached 75 minutes. Separated the causes into serialized processing and per-user caches, added parallelism, and proposed — and agreed with the research lead — a shared cache