#078 - LightRAG | From Document Retrieval to Knowledge Synthesis

This article explores a promising and powerful open source tool to build knowledge bases from engineering documents.

Greeting folks.

I’ve been on a good run lately digging into some very interesting tools in the RAG world. This article will explore one such tool.

Imagine, if you will indulge me, you’re tasked with a forensic analysis of a design decision made five years ago. The project folder is a labyrinth of PDFs, excel sheets, DWGs, and reports, maybe a pile of emails. The information is technically ‘all there’, but it's disconnected.

On a somewhat related tangent, my sister-in-law recently dropped off a massive box of busted toys for our kids. Some stuff had missing cables, broken pieces, no remote etc. It was a complete pile of shit to be honest, you get the picture.

This is the same thing, where do you start? How do you make sense of it? What good is it?

The senior engineer who held the "master model" in their head, the intricate web of dependencies, trade-offs, and judgment calls, left the company two years ago. Now, the job is not only to find the documents; it's to reconstruct the logic that connects them. You must resurrect a dead knowledge system.

This problem of "context-free" information plagues our workflows, both digital and analog.

It’s the fundamental limitation of many AI tools currently being pushed into the engineering world, particularly those built on a simple Retrieval-Augmented Generation (RAG) framework.

This article explores some of the fundamentals of a RAG system and how they can be used to support professional engineering work. This is only the beginning for RAG, this rabbit hole is deep and will progressively take over many of the knowledge hunting/curation tasks that engineers typically encounter.

A knowledge graph I am developing in LightRAG