2026–PresentPrivate active developmentCreatedEducation Data Infrastructure

Uniscout

A data pipeline that remembers where every answer came from

A private Mongolian university-data pipeline that discovers official sources, extracts eleven categories, preserves raw evidence, and tracks how canonical records change over time.

Role: Creator and system designer
Period: 2026–Present
Status: Private active development
Ownership: Created

Overview

A career advisor cannot give a trustworthy university recommendation from a paragraph copied once from a website. Programs, admission rules, tuition, scholarships, exchange options, and accreditation change. The same institution may spread those facts across a main site, an admission subdomain, PDF notices, and a JavaScript application.

Uniscout is the private data pipeline behind the current Intelligo career direction. It discovers Mongolian university sources, verifies which sites are official, collects raw evidence, extracts bilingual structured records, and preserves enough provenance to explain where an answer came from.

The product is not “an AI that crawls the web.” The product is a controlled path from a changing public source to a reviewable record.

My role

I designed and implemented the discovery, crawling, extraction, validation, truth-store, archive, and review boundaries. I also defined the eleven-category education model and how the pipeline hands usable data to Intelligo.

The source repository and collected dataset are private. The implementation counts below describe the inspected local repository; they are not claims of nationwide coverage or a public launch.

The source problem comes before the model

The obvious starting point was a list of known university domains. That was not enough.

some institutions use several official subdomains;
ministry records and live websites can disagree;
admission portals may be SPAs or block ordinary crawlers;
the useful fact may live in a PDF or a dated news item;
the same program can appear under Mongolian and English names;
a correct value becomes misleading when its academic year is unknown.

Uniscout therefore treats source discovery, official-site verification, freshness, and change detection as first-class parts of the system—not cleanup around extraction.

Discovery and trust

The discovery pipeline combines several signals instead of trusting one directory:

known seed institutions;
.edu.mn domains observed through certificate data;
ministry registry candidates;
common university subdomain probes;
a five-check official-site verifier;
URL deduplication and source trust scoring.

Discovery produces candidates. Verification decides whether a candidate belongs in the institution registry. This separation prevents “the crawler found a page” from silently becoming “this is an official source.”

03 · System view

Architecture

The pipeline moves through explicit evidence boundaries:

seeds + discovery sources
        → candidate aggregation
        → official-site verification
        → prioritized crawl strategy
        → raw HTML and Markdown archive
        → category extraction
        → Pydantic validation
        → canonical truth store
        → archive, diff, and review queue
        → Intelligo product data

Each university runs in an isolated process. The crawl strategy checks useful subdomains and sitemaps before following general links. Admission, program, tuition, and enrollment surfaces receive deterministic priority instead of leaving the entire crawl order to a model.

The agent can choose the next useful action, but only through bounded tools for crawling, sitemap inspection, API discovery, extraction, and persistence.

Eleven categories, not one university blob

The schema separates university data into eleven areas:

university identity and contact information;
schools, programs, degrees, curricula, and EESH requirements;
admissions and deadlines;
tuition, fees, and payment plans;
scholarships;
exchange programs;
student life and facilities;
career services and placement information;
faculty and research;
events and announcements;
accreditation and regulatory records.

This boundary matters because categories have different sources and freshness. A university address may remain valid for years while an admission deadline can expire in weeks.

Provenance is part of the value

Important values can carry the source URL, supporting quote, confidence, freshness, and verification status. Raw HTML and normalized Markdown are saved so extraction can be replayed without immediately hitting the source again.

The canonical truth store does not overwrite history blindly. A changed record can archive the previous version, calculate a diff, and enter a review queue. That makes “what changed?” answerable without comparing two opaque AI responses.

Failure is a stored state

Extraction is validated with typed Pydantic schemas and can self-correct a bounded number of times. If a category still fails, the pipeline can preserve partial output and validation errors instead of presenting an incomplete record as complete.

SPA detection creates another explicit branch. When a Vue, React, Next.js, or Nuxt surface does not expose useful server-rendered content, the pipeline can inspect JavaScript bundles for API patterns. API output still passes through the same source, validation, and freshness rules.

07 · By the numbers

Implementation numbers

Local commits: 151
Python modules under `src`: 98
Python test files: 74
Test functions: 227
Schema modules: 33
Structured data categories: 11

08 · Decision record

Trade-offs

The pipeline deliberately stores more evidence than a simple scraper. Raw pages, category files, crawl logs, canonical records, and archives increase disk usage and operational complexity.

That cost buys replay, review, and correction. For education advice, a smaller opaque dataset would be easier to ship and harder to trust.

The agentic layer also remains bounded. Giving an agent unrestricted crawling freedom could discover more pages, but it would make cost, reproducibility, and failure diagnosis less predictable.

09 · Outcome

Current state

The discovery, verification, prioritized crawling, bilingual schemas, validation, truth records, archive, diff, and review foundations are implemented. The remaining work includes hardening the orchestrator loop, completing freshness behavior, and running the full pipeline repeatedly across the institution registry.

Uniscout is therefore substantial private infrastructure—not a public national dataset, a finished search product, or proof that the current Intelligo recommendations are ready.

10 · Reflection

What I learned

AI can help decide where to look and how to interpret a difficult page. It cannot remove the need to model source authority, time, missing data, and review.

For this product, the most valuable answer is not merely plausible. It is an answer that can show its source, its age, its confidence, and what changed since the last time the system looked.

Next case study

dev.mn

↗