GitLaw Legal Research

​

The Problem with AI Legal Research

Before explaining what GitLaw does, you need to understand why traditional AI approaches fail for legal research - and why this matters for your practice.

​

The Citation Hallucination Problem

Standard AI models (like ChatGPT, Claude, or Gemini) have a serious problem: they make up citations. They’ll confidently cite cases that don’t exist, misquote holdings, or attribute statements to the wrong court. This isn’t a bug - it’s fundamental to how these systems work. Real examples of AI hallucination:

• Citing “Smith v. Jones, 542 F.3d 1234 (9th Cir. 2018)” - a case that doesn’t exist
• Quoting a statute with incorrect section numbers
• Attributing a holding to the Supreme Court when it came from a district court
• Inventing law review articles with plausible-sounding authors

​

Why This Happens: How AI Actually Works

To understand the problem, you need to understand two concepts: 1. Training Data Cutoff AI models are trained on data up to a certain date. After that, they know nothing. A model trained through 2023 has no knowledge of:

• Cases decided in 2024
• Statutory amendments
• New regulations
• Recent court interpretations

When asked about recent developments, the AI either says “I don’t know” or - more dangerously - makes something up that sounds plausible. 2. The Single-Source Problem (Hapax Legomenon) AI models learn from their training data. When a topic appears only once or a few times in that data, the AI treats that source as authoritative - even if it’s wrong. This is catastrophic for legal research:

• OCR errors in scanned documents - A case citation scanned with an error (e.g., “542 F.3d” instead of “524 F.3d”) becomes “truth” to the AI
• Transcription mistakes - A single court opinion with a typo in a party name propagates as fact
• Outdated sources - If the only training source for a niche legal topic is from 2015, the AI presents 2015 law as current
• Regional law - For smaller jurisdictions or specialized courts, there may be only one or two training sources, and any errors in them become gospel

This problem is dramatically worse for non-English jurisdictions. English legal content dominates AI training data. For German, French, Portuguese, or Japanese law, training sources are scarcer. If the one Portuguese case discussing a particular doctrine was OCR’d with errors, the AI will confidently cite that error forever. The AI doesn’t know it’s wrong. It presents these errors with the same confidence as well-established legal principles. 3. Weak Agentic Design (AI That Doesn’t Actually Search) Many “AI research” tools have a fundamental design flaw: the AI can choose whether to search or just answer from memory. Without proper safeguards, the AI often:

• Skips the search entirely - It “thinks” it knows the answer and responds from training data
• Searches superficially - Does one quick search, finds nothing useful, and falls back to hallucination
• Mixes sources - Combines real search results with made-up information to fill gaps

This is why you see AI tools with “web search” enabled still producing fake citations. The AI decided it already knew the answer, or the search didn’t return what it expected, so it fabricated the rest. You have no way to know when this happens. The AI presents hallucinated content with the same confidence as researched content. There’s no indicator saying “I made this up because my search failed.” 4. RAG (Retrieval-Augmented Generation) To fix the knowledge cutoff problem, many AI systems use “RAG” - they search a database, retrieve relevant documents, and feed them to the AI as context. This sounds good but has serious limitations:

​

The Multi-Jurisdictional Nightmare

These problems compound dramatically for international or multi-jurisdictional research: Non-English Jurisdictions

• Most AI training data is English-language
• Case law from Germany, France, Brazil, Japan is underrepresented
• Legal terminology doesn’t translate directly
• Citation formats vary by country
• AI hallucinates non-English citations at even higher rates

Complex Comparative Questions

• “Compare GDPR enforcement with CCPA” requires deep knowledge of both regimes
• Surface-level search returns overview articles, not authoritative sources
• AI may conflate different legal systems
• Nuances between common law and civil law traditions get lost

---

​

How GitLaw Solves These Problems

GitLaw is Arbiter’s solution to AI legal research failures. It works fundamentally differently from RAG-based systems.

​

Real-Time Web Search, Not Static Databases

Instead of searching a pre-built database (which is always out of date), GitLaw searches the live internet for legal content when you ask a question. This means:

• No knowledge cutoff - Finds cases decided yesterday
• Current regulations - Gets the latest version of statutes
• Recent commentary - Finds new law review articles and analysis

​

Contextual Retrieval, Not Keyword Matching

GitLaw Premium doesn’t just match keywords. It understands what you’re asking and retrieves content contextually:

​

Native Language Legal Search

For non-English jurisdictions, GitLaw searches in the native legal language:

• German: Searches “BGB § 823 Schadensersatzpflicht” not just “German tort law”
• French: Searches “responsabilité délictuelle Code civil” not “French liability”
• Spanish: Searches “incumplimiento contractual Código Civil” not “Spanish breach of contract”
• Portuguese: Searches “Lei Geral de Proteção de Dados” not “Brazilian privacy law”

This dramatically improves results for international research.

​

Citation Verification

Unlike standard AI that generates citations from its imagination, GitLaw:

1. Searches for actual sources
2. Extracts real citations from those sources
3. Provides links to verify
4. Indicates confidence level for each citation

You can see exactly where each citation came from and click through to verify.

---

​

Research Modes

Arbiter offers two GitLaw modes:

---

​

Using GitLaw

​

Enabling Web Research

​

What Happens Behind the Scenes

When GitLaw runs:

1. Query Generation - AI formulates optimal search terms (including native language terms for international questions)
2. Multi-Source Search - Searches across legal databases, court websites, regulatory sites, and authoritative sources
3. Result Analysis - AI reads and analyzes each result for relevance
4. Citation Extraction - Identifies and extracts actual citations from sources
5. Synthesis - Composes answer with proper attribution to sources

​

Research Results Card

When research completes, you’ll see an expandable card showing:

• Sources Found - How many sources were retrieved
• Queries Used - What search terms were used
• Citations - All legal citations extracted with links
• Confidence Levels - How reliable each citation is

---

​

Citation Quality Indicators

GitLaw indicates how confident it is in each citation:

---

​

Best Questions for GitLaw

---

​

When GitLaw Runs Automatically

GitLaw triggers automatically in certain situations:

---

​

Saving Your Preferences

Set your default research mode:

1. Go to Settings → Research Preferences
2. Select default: GitLaw Premium or Fast Search
3. Your preference applies to all new research sessions

You can always override per-query using the dropdown.

---

​

Troubleshooting

---

​

Best Practices

---

​

Next Steps