For immediate assistance with a network intrusion, ransomware
attack, or BEC, please contact: IrongateResponse@irongatesecurity.com  

I recently attended the ClaimsXchange Education X Napa event. Surrounded by professionals from across the property and casualty ecosystem—brokers, underwriters, claims professionals, coverage counsel, and risk managers—there was certainly no shortage of conversations about insurance. I met people from a range of adjacent industries as well, including medical records analysis, commercial railroading, and property and casualty services.

Despite the variety of disciplines represented, one topic consistently dominated every conversation: AI.

The event itself featured two educational tracks, one dedicated entirely to AI-related topics. But the discussion didn’t stop when the sessions ended. In fact, it often intensified during networking, social gatherings, and dinners.

Opinions among attendees varied widely. Some participants were aggressively exploring adoption and practical use cases, while others were far more skeptical—arguing that AI is still too immature to meaningfully impact the industry due to issues like model hallucinations and bias. Even the panel I participated in reflected this divide. Some panelists were pioneering emerging AI use cases, while others expressed little trust in any AI-generated output.

Our panel focused on the ethical use of AI. While we held differing views on its day-to-day applicability, we agreed on one central point: AI will eventually take center stage. The key lies in learning how to harness its capabilities, understanding its limitations, and developing stronger professional disciplines around its use.

During the discussion, two primary use cases emerged that helped frame the conversation and expand the audience’s understanding of AI.

1. Individual productivity and brainstorming

Tools like Claude, ChatGPT, and Gemini make it remarkably easy to generate images, outline ideas, draft summaries, or help structure initial thinking. These models are trained on enormous datasets with billions of parameters, allowing them to provide useful contextual assistance.

When used responsibly, these tools can significantly enhance productivity and creative output without necessarily impacting headcount. They function best as accelerators for thinking rather than replacements for it.

2. Enterprise AI models trained on proprietary data

The second use case involves organizations developing internal AI models tailored to their specific workflows and trained on proprietary datasets. In the insurance context, this could include models that help predict claim outcomes, monitor reserve adequacy, or assist employees in evaluating risk using insights derived from historical company data.

In many ways, this approach mirrors technologies the legal and insurance industries have already adopted—such as predictive analytics, machine learning, and technology-assisted review in the eDiscovery process.

Courts have long permitted technology-assisted review systems to perform initial document screening, identifying materials that should be excluded from further review or flagged for deeper analysis. In those cases, the discovery hosting provider must attest to the methodology, demonstrate accuracy rates, and validate the reliability of the model.

That framework offers a useful parallel to how we should think about using AI tools today.

Just as in eDiscovery, AI outputs should not be blindly trusted. Human oversight remains essential. The user must validate results, verify facts, and ensure the conclusions are sound. In other words, the human must remain firmly in the loop.

There have already been several widely documented examples of model hallucination—instances where an AI system, when asked a question it cannot answer, generates fabricated information rather than admitting uncertainty.

One of the most notable examples involved attorneys who used an LLM to prepare legal briefs submitted to a court. When the judge reviewed the filings, the cited cases simply did not exist. Upon investigation, the attorneys admitted they had relied on AI-generated research without independently verifying the citations.

Cases like this provide a powerful reminder that AI output should never be accepted without scrutiny.

At the same time, it’s important to remember that errors exist in every system humans rely on. No technology—or process—is foolproof. What matters is understanding the limitations of the tools we use. When we do that, we can set realistic expectations, improve reliability, and develop a clearer sense of what trustworthy output should look like. 

Contact us today to learn more about our Active Defense services.

Steve Ramey has spent the past two decades helping clients protect, investigate, and respond to events involving their digital interests. He has led hundreds of data breach investigations, assessed incident response and security programs, and successfully advised organizations through extortion negotiations.