Most government activity isn't on Google

There's a comfortable assumption underneath most modern research habits: that if something public exists, a search engine has seen it, and an AI chatbot can therefore tell you about it. For most of the web, that's roughly true. For policy, it's false — and the gap between those two facts is where careers get ruined at 10:42 on a Tuesday, when leadership forwards you an article and asks why they're reading it first.

We build monitoring infrastructure for a living, so we spend our days in the places policy actually publishes. Here is what that landscape really looks like.

Where policy is born

Policy does not publish to the web the way news does. It publishes to official journals and gazettes — the Bundesanzeiger, the BOE, the Gazzetta Ufficiale, the Journal Officiel, and their hundreds of regional and municipal cousins. It publishes to ministry and regulator sites built by the lowest bidder a decade ago. It publishes to committee pages where the agenda is a PDF inside a page inside a session-based URL. It publishes to consultation portals, comitology registers, procurement bulletins, and — increasingly decisive — to hearing video, where the positions that will decide a file are stated out loud and never written down at all.

Each of these is public. Each is official. And each is, in its own way, invisible to the tools you'd instinctively reach for.

Why the index never sees it

Search engines index what the web links to and what crawlers can economically reach. Policy sources fail those tests in five recurring ways:

1. No links point in. A gazette notice is born with an audience of nobody. Nothing links to it on day one — and links are how crawlers find things and decide they matter. By the time anything links to it, it's Day 5, and it's news.
2. PDF-first publishing. Much of the world's regulation is published as scanned or generated PDFs inside archive pages. Even where these are technically indexable, they rank nowhere, surface for nothing, and are unreadable to a chatbot answering from memory.
3. Hostile architecture — by accident. Session-based URLs, search-form-only access, no sitemaps, robots exclusions applied by IT departments that never considered anyone would want to crawl the planning committee's minutes. None of it is secrecy. All of it is invisibility.
4. Language. The consultation that reprices your market publishes in Spanish, or Polish, or Korean. English-language indexing and English-language habit — newsletters, alerts, the wires — will meet it days later, if ever.
5. It's spoken, not written. The rapporteur's real position, the regulator's tell, the amendment that will die in committee — increasingly these exist first as three sentences in hour two of a four-hour hearing video. No transcript, no index, no search.

Add these up and you get the uncomfortable arithmetic of policy risk: the more decisive a document is for you specifically, the less likely the indexed web has it. The things everyone can find are, by definition, the things that give no one an edge.

"But I'd have read about it"

You might — on Day 5, when it clears a newsroom's newsworthiness bar. The specialist press is excellent, and you should keep it. But a journalist publishes what's a story for thousands of readers, not what moved your file: the transposition in one member state, the annex, the local decree, the witness list. The item that blindsides you is usually not news to anyone but you. That's not a flaw in the press; it's the definition of the press.

And a chatbot? A chatbot answers from the indexed web — the exact part of the web policy skips. Ask it about yesterday's gazette and you'll get a confident summary of everything except yesterday's gazette. The failure mode isn't ignorance; it's fluent ignorance.

What reading the unindexed web actually takes

We're obviously not neutral here — this is the problem we build for. But the requirements hold whoever builds it:

1. Crawl the sources directly. Not the index — the gazette, the portal, the committee page itself, added source by source, maintained when they break.
2. Read every language, deliver in yours. The original kept, the working translation instant.
3. Remember who's who. A publication only means something against the committees, rapporteurs and companies behind it — tracked over months, not per query.
4. Judge relevance against your files, not newsworthiness. The whole point is catching what will never be news.

Do that, and Day 0 stops being the day you find out about later.

Related reading: Every blindside was public first — the anatomy of a blindside →