AI Document Intelligence - AI Agents
Company: AECC Global | Role: PM - AI & Growth | Year: 2024
An Agents-powered document verification and extraction system that replaced a manual, error-prone process for validating student application documents.
The Problem
International student applications require validating 8–15 documents per student: transcripts, English test scores, financial statements, passports, visa history, etc. The status quo:
- A dedicated team of document review specialists manually checked each document
- Average review time: 45–60 minutes per student application
- The baseline error rate was not recorded in this case study.
- Bottleneck: document review was holding up the downstream counsellor team Scale of the problem: AECC processes thousands of applications monthly. Manual review was unsustainable and becoming the #1 bottleneck to growth.
Discovery Process
- Process mapping session with document review team - mapped every step, decision point, and error type in the current workflow
- Interviews with downstream stakeholders (counsellors, admissions team) to understand what “good” document data meant to them
What Discovery Revealed
- 80% of errors came from 4 document types: transcripts, bank statements, IELTS/TOEFL certificates, visa pages
- The review team had an informal mental checklist per document type - this was the core IP to digitise
- LLMs were a fit because the task required reasoning under ambiguity, not just OCR extraction
- A hybrid approach (OCR → LLM extraction → rule validation → human review queue) would reduce manual load while keeping humans in the loop for edge cases.
Solution Design
Architecture (simplified)
Document upload
↓
OCR layer (extract raw text)
↓
LLM extraction layer (structured JSON output per doc type)
↓
Rule validation engine (flag anomalies: expired docs, mismatched names, etc.)
↓
Confidence scoring → Auto-approve (high confidence) | Human review queue (low confidence)
My PRD Decisions
Decision 1: LLM for extraction, rules for validation Rationale: LLMs are excellent at structured extraction from messy layouts. Rules are more reliable and auditable for binary validation logic (is the date in range? does the name match?). Separating concerns made the system more explainable and debuggable. Decision 2: Human-in-the-loop for anything below 85% confidence Rationale: We cannot afford false positives in compliance-sensitive document processing. The goal was to reduce manual work, not eliminate human judgment. I set the threshold after running the model on 500 historical documents and measuring where errors clustered. Decision 3: Document type-specific prompt templates Rationale: A generic “extract information from this document” prompt performed poorly. I worked with the document review team to write detailed extraction schemas per document type - essentially digitising their tribal knowledge into prompt engineering.
Delivery
Timeline: 16 weeks
| Phase | Key Output |
|---|---|
| Discovery & mapping | Process map, error taxonomy, prompt design brief |
| Prototype (4 doc types) | Proof of concept with accuracy benchmarks |
| Integration with CRM | Documents auto-tagged and reviewed in-workflow |
| Pilot (50 applications) | Precision/recall measurement, threshold tuning |
| Full rollout | Team training, SLA updates, monitoring dashboard |
Results
- Manual review time reduced by ~65% (45 min → ~16 min per application)
- Counsellor Adoption was very low, 90% of documents were still upload via the non Agentic flow
What I Learned
- Confidence thresholds are a business decision, not a technical one, getting alignment with compliance and ops on acceptable error rates before building saved us from post-launch conflict
- The human review queue needs UX love too, we initially underinvested in the review interface We are fixing in v2.