Online: Auditing Police AI in Public - Lessons from the ShotSpotter Debate for the AI Era in Public Policy #691

Chicago’s fight over ShotSpotter, the gunshot-detection system the city decommissioned, has become a valuable case for rethinking how cities buy, defend, and evaluate technology. ShotSpotter alert data, it turned out, belonged to the vendor rather than the city of Chicago. Some of the most damning studies of the technology were the product of a FOIA mistake. How can Chicago learn from these lessons to strengthen its civic-tech infrastructure for the looming AI era of public policy?

This talk introduces “Public Interest AI Auditing” as an open-source, independent approach to evaluating AI technologies in public policy. Using the case of AI assisted police report writing tools, the talk introduces an AI tool called PARIS (the Police AI Report & Incident Simulator) as a tool for examining how these systems may hallucinate. I’ll walk through real cases, the simulation method behind the tool, and what it means for broader efforts to evaluate AI tools in public policy insulated from independent evaluation by trade secrets exemptions.

The state of Illinois’ pending AI regulation (SB 315) is poised to mandate new third-party AI-audits but with no public capacity to perform independent audits, which makes it more likely that Chicago will see more ShotSpotter-like debates in the future. Chicago’s civic-tech community, however, is well-positioned to fill that need for independence, and I will conclude the talk by describing how it can accomplish this through open-sourced AI tools and public audits.