Early Jev catches the bug

Yes — Jev can cheaply validate the corpus while you build it. Score title↔body fit on extracted nodes, surgically fix or flag bad ones, then run the same six sample queries through the same Graph vs Jev pipelines. Use it for extraction coherence, not full legal QA.

Graph

query Jev

BM25

Jev (prep)

Jev under the extract holds the shortlist; Graph and query-time Jev still compete.

Validate run: 65 articles · ~$0.0019 · Compare run: 2026-09-24T06:42:29.653Z · typesafe/jev-1.13 · ~$0.0031 · Repairs: 19 nodes · Corpus: 171 docs / 224 nodes / 387 edges

Combined OpenRouter spend (validate + compare) ≈ $0.0050 — well under a nickel.

Cleaned corpus (smart title-aware dedupe + official DORA short titles + OJ plaintext body recovery for truncated articles). Same pipelines as Real-life test: Graph MiniSearch top-12 + 1-hop expand; Jev MiniSearch top-15 → typesafe/jev-1.13. Gold unchanged.

When to use Early Jev

Title/body coherence gates during OJ / PDF extraction — not as a substitute for a lawyer reading the regulation.
Prefer scoring all articles once if the model is cheap (here: 65 arts ≈ $0.0019); otherwise heuristic-flagged + gold-critical ids first.
Pair with a curated short-title map and longest-coherent-text dedupe — Jev flags; you fix surgically.

Pipeline

Validation gate

Messy → fixed

What changed after Early Jev

Unblocked gold

Q1: Art 6 now #1 for Graph and Jev (was locked out by a corrupted title). Art 5 enters the Graph expanded shortlist via cites.

Q5: Art 45 is information-sharing again — Graph #1 / Jev #2 (was CSDR amendment text under longest-wins dedupe).

Stopped the polluter

Q2: Art 4 keeps "Proportionality principle"; the notification fragment no longer hijacks report/notify queries. Art 19 body recovered from local OJ plaintext (~119 → full article).

Still method, not magic

TLPT recital bias and Graph vs Jev taste differences remain. Cleaning the extract removes the false "both systems failed" story — it does not make the two pipelines identical.

Contrast with the Real-life test on the messy extract — same queries, broken prep, different lesson.

After the fix: still use Jev at retrieval?

On this cleaned corpus, Graph RAG wins more often at rank 1 (5/6 gold hits) than Jev (3/6). Jev is no longer the hero of the demo — Early Jev already did its best work in prep. Keep Jev in retrieval as an optional disambiguator, not as the default always-on path.

Graph better at #1

Q2, Q3, Q5: Graph lands the operative article first (Art 19, 26, 45). Jev prefers a neighbouring process article or a recital that also "directly answers" in prose terms — useful context, wrong rank-1 for a lawyer looking up the rule.

Q1, Q6: both match gold once titles are fixed (Art 6, Art 2) — reranking adds little.

Jev still wins one hard case

Q4 (TPRM): Graph latches onto Art 6 (risk framework wording) via keywords + graph; Jev promotes art-28 General principles to #1 — the actual third-party chapter opener. That is the residual retrieval value: semantic choice inside a misleading shortlist.

Practical recommendation

Must-have: Early Jev (or equivalent) during corpus build — ~$0.002 here, large quality jump.

Nice-to-have: Jev at query time when the question is principle-level / easy to keyword-mismatch (like Q4), or when you want calibrated scores to drop junk.

Skip by default: always-on Jev for every chip click on a clean, title-faithful legal corpus — Graph (or even BM25 alone) is cheaper and often sharper at rank 1.

What are the ICT risk management framework requirements?

Ranking	What simple graphRAG answered	What JevRank answered	What supposed to be perfect answer	What this row tells us
1	`art-6`ICT risk management framework	`art-6`ICT risk management framework3.00 · Directly answers	`art-6`ICT risk management framework	Early Jev + title repair unblocked gold: Art 6 now tops both shortlists (was absent on messy corpus because its winning duplicate had a mid-sentence title).
2	`art-16`Simplified ICT risk management framework	`art-16`Simplified ICT risk management framework2.94 · Directly answers	`art-5`Governance and organisationabsent from Jev shortlist	Art 16 remains strong by literal title match — still useful for small entities, but no longer crowds out the real framework article.
3	`art-27`Requirements for testers for the carrying out of TLPT	`art-9`Protection and prevention2.90 · Directly answers	`art-15`Further harmonisation of ICT risk management tools, methods, processes and policiesabsent from Graph shortlist	Jev lifts protection (Art 9) over graph’s TLPT/oversight noise once Art 6 is correctly labelled.
4	`art-32`Structure of the Oversight Framework	`art-33`Tasks of the Lead Overseer2.63 · Directly answers	`art-7`ICT systems, protocols and toolsabsent from Graph shortlist; absent from Jev shortlist	Art 5 (governance) enters the Graph expanded list via cites→art-6 but still sits outside keyword top-12 — expand helps, BM25 alone still under-ranks governance wording.
5	`art-24`General requirements for the performance of digital operational resilience testing	`rec-21`Recital (21)2.45 · Useful	`art-16`Simplified ICT risk management framework (small/micro entities)	Remaining gap is query wording vs Art 5/7/15 titles — corpus prep fixed the hard miss; ranking nuance remains.

When must financial entities report major ICT-related incidents?

Ranking	What simple graphRAG answered	What JevRank answered	What supposed to be perfect answer	What this row tells us
1	`art-19`Reporting of major ICT-related incidents and voluntary notification of significant cyber threats	`art-17`ICT-related incident management process2.31 · Useful	`art-19`Reporting of major ICT-related incidents and voluntary notification of significant cyber threats	Art 19 recovered from truncated OJ stub (~119 chars → full article) and now leads Graph; Jev ranks it #2 behind process Art 17.
2	`art-21`Centralisation of reporting of major ICT-related incidents	`art-19`Reporting of major ICT-related incidents and voluntary notification of significant cyber threats2.29 · Useful	`art-17`ICT-related incident management process	Art 4 no longer pollutes: smart dedupe + official title keep Proportionality, so the notification fragment cannot hijack “report/notify” queries.
3	`art-18`Classification of ICT-related incidents and cyber threats	`rec-53`Recital (53)1.76 · Useful	`art-18`Classification of ICT-related incidents and cyber threats	Incident cluster (19/21/18/17) is clean for Graph — early validation’s body recovery matters as much as title fixes here.
4	`art-17`ICT-related incident management process	`rec-51`Recital (51)1.53 · Useful	`art-21`Centralisation of reporting of major ICT-related incidents	Jev still prefers management-process language (Art 17) slightly over the operative reporting article — calibrated, not broken.
5	`rec-100`Recital (100)	`art-18`Classification of ICT-related incidents and cyber threats1.49 · Tangential	`art-20`Harmonisation of reporting content and templatesabsent from Graph shortlist; absent from Jev shortlist	Gold Art 20/21 remain reachable; messy-corpus Art 4 false positive is gone.

What is TLPT and who must perform threat-led penetration testing?

Ranking	What simple graphRAG answered	What JevRank answered	What supposed to be perfect answer	What this row tells us
1	`art-26`Advanced testing of ICT tools, systems and processes based on TLPT	`rec-56`Recital (56)2.12 · Useful	`art-26`Advanced testing of ICT tools, systems and processes based on TLPT	Graph still correctly puts operative Art 26 first — unchanged win vs recital-heavy Jev preferences.
2	`rec-18`Recital (18)	`rec-18`Recital (18)2.10 · Useful	`art-27`Requirements for testers for the carrying out of TLPT	Jev again favours Rec 56/18 rationale over binding Art 26 — early corpus cleanup does not change scorer taste for recitals.
3	`art-3`Definitions	`art-26`Advanced testing of ICT tools, systems and processes based on TLPT2.06 · Useful	`art-24`General requirements for the performance of digital operational resilience testingabsent from Graph shortlist	Art 27 testers requirements surface in both top-5 — good agreement on the “who” half of the question.
4	`art-27`Requirements for testers for the carrying out of TLPT	`art-27`Requirements for testers for the carrying out of TLPT1.98 · Useful	`art-25`Testing of ICT tools and systems	Definitions (Art 3) appears in Graph via expand/keyword after title repair — less pollution than before.
5	`rec-56`Recital (56)	`rec-44`Recital (44)1.57 · Useful	`rec-56`Recital (56) — TLPT rationale	TLPT story is mostly stable; the Early Jev win is elsewhere (Q1/Q2/Q5).

How should financial entities manage ICT third-party risk?

Ranking	What simple graphRAG answered	What JevRank answered	What supposed to be perfect answer	What this row tells us
1	`art-6`ICT risk management framework	`art-28`General principles2.99 · Directly answers	`art-28`General principles (ICT third-party risk)	Jev still wins: promotes Art 28 General principles to #1 while Graph’s keyword stage overweights Art 6 once that title is fixed.
2	`art-29`Preliminary assessment of ICT concentration risk at entity level	`rec-71`Recital (71)2.97 · Directly answers	`art-29`Preliminary assessment of ICT concentration risk at entity level	Side effect of cleaning Art 6: Graph Q4 now leads with the framework article — honest trade-off, not a regression of TPRM gold.
3	`art-31`Designation of critical ICT third-party service providers	`art-29`Preliminary assessment of ICT concentration risk at entity level2.84 · Directly answers	`art-30`Key contractual provisionsabsent from Graph shortlist; absent from Jev shortlist	Art 29 concentration risk is high for both — clean title helped keep it visible.
4	`art-16`Simplified ICT risk management framework	`art-26`Advanced testing of ICT tools, systems and processes based on TLPT2.50 · Directly answers	`art-31`Designation of critical ICT third-party service providers	Art 28 enters Graph shortlist (expanded) even when not keyword #1 — structure helps.
5	`rec-92`Recital (92)	`rec-18`Recital (18)2.35 · Useful	`rec-65`Recital (65) — third-party risk contextabsent from Graph shortlist; absent from Jev shortlist	Early validation fixed framework labels; third-party ranking differences remain method, not corpus, driven.

What information must be shared on cyber threats?

Ranking	What simple graphRAG answered	What JevRank answered	What supposed to be perfect answer	What this row tells us
1	`art-45`Information-sharing arrangements on cyber threat information and intelligence	`rec-52`Recital (52)2.86 · Directly answers	`art-45`Information-sharing arrangements on cyber threat information and intelligence	Headline Early Jev win: Art 45 is real information-sharing again (was CSDR amendment junk under longest-text dedupe) — Graph #1, Jev #2.
2	`art-18`Classification of ICT-related incidents and cyber threats	`art-45`Information-sharing arrangements on cyber threat information and intelligence2.76 · Directly answers	`rec-34`Recital (34) — encourage cyber threat information sharing	Jev slightly prefers Rec 52 threat-intel context over Art 45 — still lands the gold article in the top-2.
3	`art-19`Reporting of major ICT-related incidents and voluntary notification of significant cyber threats	`art-1`Subject matter2.25 · Useful	`rec-33`Recital (33) — limited/fragmented information sharingabsent from Graph shortlist; absent from Jev shortlist	Art 19 (now full text) correctly appears as related incident reporting, not as a false substitute for sharing arrangements.
4	`art-37`Request for information	`art-19`Reporting of major ICT-related incidents and voluntary notification of significant cyber threats2.21 · Useful	`art-18`Classification of ICT-related incidents and cyber threats	Rec 34/33 gold context still harder for keyword recall — expected; article-level fix was the blocker.
5	`rec-52`Recital (52)	`art-18`Classification of ICT-related incidents and cyber threats2.16 · Useful	`rec-52`Recital (52) — threat intelligence context	No reranker required to invent missing text anymore — the article exists with the right title.

Which entities are in scope of DORA?

Ranking	What simple graphRAG answered	What JevRank answered	What supposed to be perfect answer	What this row tells us
1	`art-2`Scope	`art-2`Scope3.00 · Directly answers	`art-2`Scope	Still the clean agreement case: Art 2 Scope #1 for both Graph and Jev.
2	`art-26`Advanced testing of ICT tools, systems and processes based on TLPT	`rec-42`Recital (42)2.27 · Useful	`art-3`Definitions	Art 3 Definitions keeps a correct title after smart dedupe — Graph includes it early.
3	`art-19`Reporting of major ICT-related incidents and voluntary notification of significant cyber threats	`art-26`Advanced testing of ICT tools, systems and processes based on TLPT2.04 · Useful	`art-1`Subject matterabsent from Jev shortlist	Cleaning other articles adds some incidental hits (Art 19/26) into Graph top-5 — noisier than messy run but gold Art 2 remains unchallenged.
4	`art-38`General investigations	`art-19`Reporting of major ICT-related incidents and voluntary notification of significant cyber threats1.58 · Useful	`art-16`Simplified ICT risk management framework (proportionality)absent from Jev shortlist	Jev keeps Rec 42 entity-coverage context near the top — useful complement to Art 2.
5	`art-3`Definitions	`art-45`Information-sharing arrangements on cyber threat information and intelligence1.26 · Tangential	`rec-42`Recital (42) — entity coverage contextabsent from Graph shortlist	Scope was never the broken path; Early Jev’s value shows on Q1/Q5/Q2 instead.