Taking a research PoC into the product — knowledge graph feature shipped to production

The problem

This is not code only I will read — it means changing an existing product’s own codebase. Work has to pass review, mesh with what other engineers are building, and still reach release. On top of that, graph features tend to fall into a state where “it renders, but you cannot read it” as data grows. Laying out nodes and edges alone does not tell you what is there.

Design decisions

• Methods whose effectiveness had been confirmed in the research phase were ported into a form that holds up against the product’s data volume and response time requirements. There is a gap between working in a verification environment and working on a screen customers use daily
• Visualization uses force-directed layout so analysis results can be read alongside the structure
• Graph computation is heavy, and written naively it blocks the event loop. Reworked the async handling and secured responsiveness including cache capacity design
• Measured performance with real data loaded, identified why processing failed against memory limits, and reflected that in the design
• Data model changes required redesigning join tables, so migrations were applied in stages, each one reversible
• Ran each feature through front end, back end, and infrastructure alone, removing the wait time between layers

Approach

• Ported analysis methods validated in the research phase (community detection via Louvain, structural hole analysis, dimensionality reduction via UMAP, density maps) to meet the product's data volume and response time requirements
• Visualization places nodes by a force-directed model, overlaid with analysis results so both can be read together
• Resolved graph computation blocking the event loop, and secured responsiveness including cache capacity design
• Measured performance with real data loaded, identified the cause of failures traced to memory limits, and fed that back into the design
• Enabled streaming responses, including permission configuration and parameter management on the platform side
• Data model changes required redesigning join tables, so migrations were applied in stages
• Ran each feature through every layer alone, removing the wait time between layers