Real-life test

Real-life test on the messy OJ extraction corpus — duplicate article ids, mid-paragraph titles, truncated bodies, and amending-provision junk. Side-by-side Graph vs Jev tables for the six sample queries, baked offline (Graph offline; Jev scored once via OpenRouter). This page makes no API calls. Live chips on /jev still spend quota; reading here does not. See also Early Jev — same queries after cheap title↔body validation while building the corpus.

Graph

Jev

BM25

Same BM25 shortlist — Graph expand vs Jev rerank.

Run: 2026-09-24T06:18:44.205Z · Model: typesafe/jev-1.13 · OpenRouter cost: ~$0.0031 · Corpus: 171 docs / 224 nodes / 387 edges

Graph: MiniSearch top-12 + 1-hop cites/covers expand (keyword hits first). Jev: MiniSearch top-15 → OpenRouter Decisions typesafe/jev-1.13 → sort by jevScore. Gold: curator-selected DORA passages for each rank slot.

Columns: Graph = MiniSearch top-12 + 1-hop expand; Jev = MiniSearch top-15 reranked by score; Gold = curator pick for that rank slot (honest when the gold id never entered either shortlist).

Real-life test: retrieval on the messy extract

The clearest lesson from this bake-off is not “Jev vs Graph” in the abstract — it is that neither retrieval trick repairs bad corpus prep. Where titles and article boundaries are wrong, both systems look clever and still miss. Where the text is clean, the methods diverge for understandable reasons.

Jev won

Q4 (third-party risk): keyword recall favoured narrower hits (Art 29/31); Jev promoted art-28 General principles to #1 — closer to the gold “how should entities manage ICT TPRM?” framing.

Q1 (partial): with Art 5/6 locked out of the shortlist, Jev still lifted protection/recovery articles (9, 11) that Graph left buried under title-heavy noise.

Graph RAG won

Q2 (incident reporting): Graph landed the right cluster early (Art 19/17/18). Jev over-promoted a mis-titled Art 4 fragment that talks about “notification”, which looks relevant to a scorer and wrong to a lawyer.

Q3 (TLPT): Graph put operative art-26 first; Jev preferred recitals over the binding article.

Both lost — bad prep work

Extraction from the OJ PDF left duplicate article ids, titles taken from mid-paragraph cross-references, truncated bodies (e.g. Art 19 ≈ one sentence), and amending-provision junk labelled as Art 45. Roughly a quarter of article records look suspicious after dedupe-by-longest-text.

Q1: true Art 6’s winning duplicate has title , point 2, of Directive…, so MiniSearch never treats it as “ICT risk management framework”; Art 16 wins by title match instead.

Q5: Art 45 in this corpus is CSDR amendment text, not information-sharing — no reranker can invent the missing article.

Q6: Scope is the clean counter-example — Art 2 title/text are intact, and both pipelines agree with gold.

What are the ICT risk management framework requirements?

Ranking	What simple graphRAG answered	What JevRank answered	What supposed to be perfect answer	What this row tells us
1	`art-16`Simplified ICT risk management framework	`art-16`Simplified ICT risk management framework2.90 · Directly answers	`art-6`ICT risk management framework (corpus title corrupted)title corrupted in corpusabsent from Graph shortlist; absent from Jev shortlist	Keyword bias: Art 16 wins both lists because its title literally contains “ICT risk management framework”, while true Art 6 never enters the MiniSearch shortlist.
2	`art-27`Requirements for testers for the carrying out of TLPT	`art-9`Protection and prevention2.88 · Directly answers	`art-5`Governance and organisationabsent from Graph shortlist; absent from Jev shortlist	Art 5 (governance) is absent from both top-15 pools — graph expand and Jev cannot recover a doc the keyword stage never saw.
3	`art-32`Structure of the Oversight Framework	`art-28`General principles2.69 · Directly answers	`art-15`Further harmonisation of ICT risk management tools, methods, processes…	Jev usefully promotes Art 9 (protection) from deeper keyword ranks; Graph stays stuck on title-heavy hits like Art 27/32.
4	`art-24`General requirements for the performance of digital operational resilie…	`art-33`Tasks of the Lead Overseer2.67 · Directly answers	`art-7`ICT systems, protocols and toolsabsent from Graph shortlist; absent from Jev shortlist	Art 6’s corpus title is corrupted (“, point 2, of Directive…”) so even if retrieved it would look like junk — piping issue, not a model failure.
5	`art-17`ICT-related incident management process	`art-11`Response and recovery2.61 · Directly answers	`art-16`Simplified ICT risk management framework (small/micro entities only)	Art 16 as rank-5 gold is only the simplified framework for small entities; ranking it #1 overstates its generality.

When must financial entities report major ICT-related incidents?

Ranking	What simple graphRAG answered	What JevRank answered	What supposed to be perfect answer	What this row tells us
1	`art-21`Centralisation of reporting of major ICT-related incidents	`art-4`of Directive 2013/36/EU, which shall immediately transmit the notificat…2.83 · Directly answers	`art-19`Reporting of major ICT-related incidents and voluntary notification of…	Graph finds the right cluster (Art 21/19/18/17); Jev oddly elevates corrupted Art 4 notification text above the actual reporting article.
2	`art-19`Reporting of major ICT-related incidents and voluntary notification of…	`art-17`ICT-related incident management process2.21 · Useful	`art-17`ICT-related incident management process	Art 17 (process) outranks Art 19 (reporting) under Jev — useful context, but not the timing/reporting obligation itself.
3	`art-18`Classification of ICT-related incidents and cyber threats	`rec-53`Recital (53)1.76 · Useful	`art-18`Classification of ICT-related incidents and cyber threats	Classification (Art 18) is correctly near the top on Graph; Jev keeps it but behind process and a weak recital.
4	`art-17`ICT-related incident management process	`art-19`Reporting of major ICT-related incidents and voluntary notification of…1.68 · Useful	`art-21`Centralisation of reporting of major ICT-related incidents	Art 19 drops to Jev #4 despite being the direct answer — score gap vs Art 4 shows label noise when titles/text are misaligned.
5	`rec-100`Recital (100)	`art-18`Classification of ICT-related incidents and cyber threats1.52 · Useful	`art-20`Harmonisation of reporting content/timelines (corpus title corrupted)	Neither system surfaces precise hour/day timelines cleanly; Art 19’s body in this corpus is truncated to one sentence.

What is TLPT and who must perform threat-led penetration testing?

Ranking	What simple graphRAG answered	What JevRank answered	What supposed to be perfect answer	What this row tells us
1	`art-26`Advanced testing of ICT tools, systems and processes based on TLPT	`rec-56`Recital (56)2.23 · Useful	`art-26`Advanced testing of ICT tools, systems and processes based on TLPT	Graph nails Art 26 at #1; Jev prefers explanatory recitals (56, 18) over the operative TLPT article.
2	`rec-18`Recital (18)	`rec-18`Recital (18)2.15 · Useful	`art-27`Requirements for testers for the carrying out of TLPT	Art 27 (who may test) sits #3 on Graph and #5 on Jev — both see it, but recitals crowd the Jev top.
3	`art-27`Requirements for testers for the carrying out of TLPT	`art-26`Advanced testing of ICT tools, systems and processes based on TLPT2.07 · Useful	`art-24`General requirements for the performance of digital operational resilie…	Art 26 remains in Jev’s top 5 with a middling “Useful” score rather than “Directly answers”.
4	`rec-56`Recital (56)	`rec-61`Recital (61)2.06 · Useful	`art-25`Testing of ICT tools and systems	Graph also pulls corrupted Art 6 into the TLPT shortlist via keyword bleed — a false neighbour.
5	`art-6`, point 2, of Directive (EU) 2022/2555;	`art-27`Requirements for testers for the carrying out of TLPT2.03 · Useful	`rec-56`Recital (56) — TLPT rationale	Gold Art 24/25 (general testing) are weaker matches than Art 26/27; systems that over-weight “testing” may dilute TLPT specificity.

How should financial entities manage ICT third-party risk?

Ranking	What simple graphRAG answered	What JevRank answered	What supposed to be perfect answer	What this row tells us
1	`art-29`Preliminary assessment of ICT concentration risk at entity level	`art-28`General principles2.99 · Directly answers	`art-28`General principles (ICT third-party risk)	Clear Jev win: Art 28 (TPRM general principles) jumps to #1; Graph had it only around #7 behind concentration/criticality docs.
2	`art-31`Designation of critical ICT third-party service providers	`rec-71`Recital (71)2.96 · Directly answers	`art-29`Preliminary assessment of ICT concentration risk at entity level	Graph leads with Art 29 concentration risk — related but narrower than the overarching third-party principles.
3	`art-16`Simplified ICT risk management framework	`rec-65`Recital (65)2.93 · Directly answers	`art-30`Key contractual provisionsabsent from Graph shortlist; absent from Jev shortlist	Art 30 (key contractual provisions) is absent from both shortlists — another keyword-pool miss for a core TPRM article.
4	`rec-92`Recital (92)	`art-29`Preliminary assessment of ICT concentration risk at entity level2.87 · Directly answers	`art-31`Designation of critical ICT third-party service providers	Jev mixes strong TPRM hits (28, 29, recitals) with an off-topic Art 26 TLPT promotion at #5.
5	`rec-63`Recital (63)	`art-26`Advanced testing of ICT tools, systems and processes based on TLPT2.56 · Directly answers	`rec-65`Recital (65) — third-party risk contextabsent from Graph shortlist	Designation of critical providers (Art 31) ranks high on Graph but is oversight machinery, not entity-level TPRM how-to.

What information must be shared on cyber threats?

Ranking	What simple graphRAG answered	What JevRank answered	What supposed to be perfect answer	What this row tells us
1	`art-18`Classification of ICT-related incidents and cyber threats	`rec-52`Recital (52)2.81 · Directly answers	`art-45`Information-sharing arrangements (real DORA Art 45; corpus misaligned)corpus title/text are amendments, not Art 45absent from Graph shortlist; absent from Jev shortlist	True Art 45 (information-sharing arrangements) is missing/mislabelled in the corpus; neither system can retrieve the real article.
2	`art-37`Request for information	`art-4`of Directive 2013/36/EU, which shall immediately transmit the notificat…2.70 · Directly answers	`rec-34`Recital (34) — encourage cyber threat information sharing	Graph surfaces Rec 34 (encourage sharing) at #5; Jev ranks classification and unrelated notification text higher.
3	`rec-52`Recital (52)	`art-1`Subject matter2.31 · Useful	`rec-33`Recital (33) — limited/fragmented information sharingabsent from Graph shortlist; absent from Jev shortlist	Rec 33 (fragmented sharing) never appears — gold context lost to keyword stage.
4	`art-20`and submit them to the competent authority. In the event that a technic…	`rec-32`Recital (32)2.15 · Useful	`art-18`Classification of ICT-related incidents and cyber threats (related, not…	Art 18 answers “what is a cyber threat” more than “what must be shared” — keyword collision on “cyber threats”.
5	`rec-34`Recital (34)	`art-18`Classification of ICT-related incidents and cyber threats2.13 · Useful	`rec-52`Recital (52) — threat intelligence context if present	Honest takeaway: this question is broken by corpus alignment; fix Art 45 text before trusting either pipeline.

Which entities are in scope of DORA?

Ranking	What simple graphRAG answered	What JevRank answered	What supposed to be perfect answer	What this row tells us
1	`art-2`Scope	`art-2`Scope3.00 · Directly answers	`art-2`Scope	Both systems correctly put Art 2 (Scope) at #1 — the rare clean agreement with gold.
2	`art-4`of Directive 2013/36/EU, which shall immediately transmit the notificat…	`rec-42`Recital (42)2.38 · Useful	`art-3`Definitionsabsent from Jev shortlist	Art 3 (Definitions) only appears late on Graph (#14) and never on Jev’s scored 15 — entity lists partly live there.
3	`art-26`Advanced testing of ICT tools, systems and processes based on TLPT	`art-26`Advanced testing of ICT tools, systems and processes based on TLPT2.20 · Useful	`art-1`Subject matterabsent from Jev shortlist	Jev demotes noisy Art 4 (corrupted title) that Graph still ranks #2 — good semantic filter.
4	`art-38`General investigations	`rec-41`Recital (41)2.00 · Useful	`art-16`Simplified ICT risk management framework (proportionality for certain e…	Art 26 TLPT still leaks into “scope” results for both paths via shared vocabulary (entities/must/perform).
5	`art-50`Administrative penalties and remedial measures	`art-16`Simplified ICT risk management framework1.84 · Useful	`rec-42`Recital (42) — entity coverage context	Art 1 (subject matter) is only a Graph-expand neighbour; Jev never scores it because MiniSearch top-15 skipped it.