What is GraphRAG – and When Is It Worth Using?

Chatbots existed long before ChatGPT came onto the scene. But back then, talking to them quickly revealed their limitations: they mostly relied on rigid rules, could only understand specific questions – or didn’t allow free-form questions at all, offering only predefined options to choose from. My own experience was frustrating: almost every attempt ended with a phone call to customer service. The chatbot was more of a barrier than a help.

Then came ChatGPT – and with it, a new era. Suddenly, you felt understood. The hype around generative AI took off. But as fascinating as these new language models are, one key question remains: Can they reliably answer our questions?

Maybe a comparison helps: We, the users, are like lead detectives. We have a hunch, we formulate a question, and the AI is like a patrol officer who was at the scene – but only after the crime had already occurred. It sounds confident. It sounds smart. But is it really capable of answering all our questions?

Language Models: Brilliant, But With Gaps in Knowledge and Memory

As impressive as modern language models may seem, they have critical weaknesses. They hallucinate – meaning they sometimes provide information that is made up, incorrect, or simply illogical – and they do so with great linguistic confidence.

On top of that, their knowledge is limited. A language model only knows what it was trained on – and that’s mostly a view into the past. A model trained in 2023 won’t be able to give accurate answers about events in 2024 – unless it is specifically updated with new information.

Back to our detective analogy: the patrol officer in the interrogation room has a good memory – but wasn’t there when the crime happened. Without new clues, they’re left to speculate.

RAG – The Solution?

To tackle these problems – limited knowledge and hallucinated answers – a new approach was developed in 2020: Retrieval-Augmented Generation, or RAG for short.

The idea is simple: If the language model can’t deliver a useful answer in the prompt, it’s often because it simply doesn’t know enough – its training data may be outdated or incomplete. RAG brings in additional information from external sources – like an investigator laying relevant documents on the table for a witness.

Imagine you’re at police headquarters. In front of you sits your colleague – the language model. You ask a specific question: “What items were found at the crime scene?” The officer looks at you, hesitates. He wasn’t there. His information is based on old records – and they’re not enough.

Now you hand him a binder full of relevant documents and forensic findings from the scene. Suddenly, he starts talking. More precisely. More confidently. He now has the context he needs to answer your question properly. That binder – carefully selected and tailored to the question – is what RAG provides. Not just vague knowledge, but statements based on concrete sources.

In a simple RAG setup, a document is split into smaller text sections – like the individual pages of our binder. The system then searches for the passages most similar to the question – measured by mathematical proximity in a so-called vector space (the technique behind this is called embedding, i.e., turning text into numbers). The more similar a passage is to the question, the more likely it contains the right answer.

So far, so good. But even this method has its pitfalls. What if a “page” contains information about a different crime scene? That’s exactly what happens when RAG selects the wrong context or when the right context is too large for the language model to handle.

A Real-Life Example: ESG Reports
Let’s take ESG reports from large companies, which document goals and measures related to environment, social responsibility, and corporate governance. These reports often span over 100 pages. If you ask a question like “What has your company done to save water?”, a single paragraph usually isn’t enough. The answer is spread across many pages.

A basic RAG system – also called Baseline-RAG – reaches its limits here: it can’t load enough context into the prompt to answer the question in a well-founded way. RAG is a major step forward – but in its simple, naive setup, it’s not a catch-all solution. Complex cases require more.

GraphRAG – The Detective With a Relationship Map

In 2024, Microsoft Research introduced a new extension of the RAG principle: GraphRAG. Its goal: to answer questions that refer not just to individual sections, but to entire – often very long – documents.

Here’s the clever part: GraphRAG doesn’t work directly with text chunks anymore. Instead, it first creates a knowledge graph – a network of facts, entities (people, things, organizations), and their relationships.

Imagine yourself as a detective pinning clues to a wall: photos of suspects, places, times – all connected by red strings. That’s exactly what GraphRAG does automatically. It builds a network from the document, systematically organizing relevant information. This reveals connections that remain hidden in regular text.

An illustrative example:
Let’s take QualityMinds as an organization. At the top level of the knowledge graph, you have the entity “Company QualityMinds.” Below that are the “Teams” within the company. And on the third level, the individual “People” assigned to those teams. Three levels, three “communities,” as GraphRAG calls them. Each level can contain information, and the structure allows more targeted answers.

The advantage: By reorganizing textual knowledge, thematically grouped units are formed where statements about one entity are close together. That means if a question targets a specific team, the AI doesn’t need to search the whole document – it can look directly in the relevant section of the network.

Global Search and Local Search – Two Strategies, Two Perspectives

GraphRAG’s first version includes two search strategies:

• Global Search looks at the big picture. It analyzes whole communities – like all statements at the team or company level – and is well suited for questions like: “What is the central theme of the document?”
• Local Search follows specific clues. It identifies individual entities in the graph that match the question semantically, then tracks their relationships: What statements, figures, or details are connected to these nodes?

Our experience shows: Local Search delivers particularly strong results when precise information or concrete facts are required.

A Practical Comparison in the ESG Context

How effective are these methods? Let’s look at a practical example: The ESG report of a company from 2022, working in water, waste, and energy services. The question:“What share of purchased energy came from renewable sources?”

We see that Local Search responded with many numbers, and the correct answer is among them. Global Search was unable to provide an answer, likely because no relevant data was available at the community level. And Baseline RAG apparently did not identify the relevant text passage that contains the correct answer.

Question: What is the share of purchased energy that comes from renewable sources?

• Local Search: ## Share of renewable energy sources in purchased energy
  • Energy
    In 2022, the share of renewable energy sources in the purchased energy of the company in Germany amounted to 85%. This means that of the total 225,000 MWh consumed by the company itself, 85% came from green energy sources. Of these 225,000 MWh, 85,000 MWh were self-produced, while 140,000 MWh were purchased. [Data: Sources (49); Entities (290, 293); Relationships (230)]. It should also be noted that the share of renewable sources in the total energy production (electricity and heat) for distribution and self-consumption was around 26% in 2022. This shows that the company is actively working to increase the share of renewable energies in its overall mix in order to achieve its sustainability goals. [Data: Sources (49); Entities (290, 293); Relationships (230)].
    
  • Conclusion
    Overall, the data shows that the company in Germany has a significant share of renewable energy sources in its energy purchases, which is important both for the environment and for the company’s strategy. The continuous increase in the share of renewable energy is an important step toward a more sustainable energy future.
    
• Global Search: Sorry, I cannot answer this question based on the provided data.
• Baseline RAG: Sorry, I cannot answer this question based on the provided data.

When the Detective Digs Through Too Many Files – The Limits of GraphRAG

As powerful as GraphRAG is, it has limitations worth knowing. First challenge: traceability. In classic RAG setups, it’s easy to verify the model’s answer – the text chunks come directly from the original document. You can point to the exact source.

In GraphRAG, that’s harder. Answers are based on extracted entities, relationships, and summaries from the entire document. A single entity might combine info from many pages – making it harder to trace the exact origin of a statement.

For us as lead detectives, that means: The patrol officer names the binder as the source – but the trail leads to a general “reading” of the whole thing, not a single page.

Second issue: cost. GraphRAG uses large language models (LLMs) to build the knowledge graph. The larger the document, the more compute – and cost – is required. The good news: the graph only needs to be created once, and can then be used for many queries. The actual question-answering is relatively inexpensive. The patrol officer only needs to read the binder once.

GraphRAG or Baseline-RAG? – The Right Tool for the Right Case

GraphRAG isn’t always the best choice. If a document is long, complex, or contains scattered information – GraphRAG can be very helpful. But if it’s short, well-structured, and compact – like a FAQ or internal info sheet – then simple RAG is usually sufficient.

Our practical tip: Start with Baseline-RAG. Watch how accurate the answers are, and note any weaknesses: Are important connections missing? Are there hallucinations? Then decide whether it’s worth switching to GraphRAG – like a detective who knows when a quick question is enough and when it’s time to open the entire archive and pin the clues to the wall.

Ready to do your own investigating? We’ll support you.

Whether you’re just experimenting with RAG or already working on concrete use cases – we at QualityMinds are happy to support you.
In our AI Lab, we’ll show you real GraphRAG applications and help you find the right solution for your context.

Reach out to us – we’ll help you ask the right questions and get the best answers: ai@qualityminds.de