Applied Agentic Systems

Civitas has to deliver information at scale. We had a few paths we could take to deliver information to our end users at scale; quickly. We could have build an agentic system you'd chat with to get the information you need, so an LLM would browse the web for you and find the information you need. We could farm this out to any web browser agent we fancied. This would be incredibly easy to implement but fails because the process of getting the information is seldom reproducible, and we must deal with hallucinated information. As a primary source information aggregator, we cannot deal in hallucinations, so for even this reason we moved on from agent in the loop strategies.

We instead deployed an agentic system internally, that could, at scale, test, code and repair a series of clients for each state/territory in the United States. We then build another agentic system that translated the modeled responses into our Political Data Schema (LDS).

This succeeds because LLMs are generally very good at mapping clusters of data to like data. As such, as we model the responses, we can examine these data in the context of the website and map our client returns to the LDS. We have connected this system to a spider and in the span of about an hour (and at zero cost) we have modeled every endpoint on a given website, categorized the responses to requests from that endpoint, mapped those models to our internal data schema, and created a library for making requests to each states different endpoints.

How can you trust LLMs to write code? TLDR: We Trust, but Verify.

We are not asking for our agents to write a kernel architecture, we are asking for a very mundane and very common operation, that is model an API response and make a request to an API. Regardless these are not one-shot API responses and modelings. We are testing each of these until we have a functional model and response. We orchestrate, for each client, in each language, a sand-boxed environment with the required packages installed. We use these environments to ensure that we can (1.) make valid requests to the server and (2.) check the returned response against our model. Putting testing in the development loop allows us to give feedback automatically to coding agents.

We are now able to efficiently scrape information from these sites, reproducible, and without straining the systems we are scraping.

Why make this?

For Civitas, approximately 16% of states have a publicly accessible API endpoint that is documented. For those states we can build native clients and authorize using their preferred systems. To reach the other 84% of states, or a large number of them, we have to build systems like this. In the past other services have built their systems by hand. We instead have opted for this approach.

We want an information system that can address any hosted piece of government information. In the near future, we will be expanding this to all government websites and APIs. This will give us a structured flow of information outside of just the legislature. We can keep tabs on ancillary body, agencies, and much much more.

You know what the coolest part of all of this is? Google Gemini flash and pro 2.0 are free within a daily set of rate limits. We can basically run this code virtually without error on a managed sub vCPU core instance (~$3.50 a month) and get web-scrapers that can repair themselves 24/7.

Self-healing Web-scrapers

This workflow is currently being deployed asynchronously to our other systems (no locally sourced LLMs it is just a python script that makes api calls). It can push changes to a GitHub repository of all of our web-scrapers when a web-scraper fails, it will pull the afflicted code and render a fix. Because more often the response has changed slightly, it is a quick adjustment to our model. We can deliver these fixes in near real time. Our senior dev just have to push any changed models upstream. What does this mean? Outages are rare, and response times are better than our competition.

A quick aside, this technology generalizes well outside of the domain of politics, in fact I see google, or exa.ai, doing something similar in search in the near future. Allowing for the population of entire sheets worth of structured data in a single query. I guess a man can dream.

It is becoming more and more likely that those who posses and comprehend data are then ones who shape political outcomes and drive changes. We want to get that information to you at a low or minimal cost and deliver high quality data and insights at scale. To do this we must think differently about how we build and maintain information networks over time.

I have MIT licensed our non-Civitas specialized version of this webscraper, it is a little neutered because of that. JSA-AWARE - Jackson's Super Awesome Agentic Webscraper & API Reverse Engineer