Local newsrooms are building AI chatbots fast and cheap

Over the past year, major newsrooms across the country have rolled out generative AI chatbots. The Hearst-owned San Francisco Chronicle came out with its Chowbot for foodies and a Kamala Harris “news assistant” for voters. The Washington Post launched Climate Answers, its first-ever chatbot fueled by reporting from the climate desk. And chatbots that help readers navigate article archives have gone live at both Forbes and the Financial Times.

For many local newsrooms, though, the work of building, refining, and maintaining an experimental chatbot may seem daunting, or an unnecessary resource or budget drain. That’s especially true of small local publishers that have less than a dozen staffers, and no product developer or dedicated engineering staff.

A new report from the UNC Center for Innovation & Sustainability in Local Media (CISLM) documents what building a chatbot looks like for these newsrooms.

CISLM gave four small local newsrooms across the Southeastern U.S product development support and financing to launch their own chatbot. The program, called Local NewsBot Studio, launched four products back in May and ran them on each publisher’s website for 45 days, collecting feedback from the newsrooms throughout the process.

“One of the purposes of this report is to show that there are ways to experiment with chatbots that might not be perfect, or the most refined product, but that it is a possibility,” said Yanan Sun, a researcher at CISLM and author of the report.

CISLM’s development process was fast and scrappy. Altogether, it took less than a month from initial conversations with each newsroom to the chatbots going live. In most cases, initial demos were up and running in just one week. For all the pilots, CISLM used Zapier, a platform that allows users to build simple apps and chatbots without coding. The cost to build and run each chatbot came out to about $40 per month.

These were not copy-and-paste products, though. Each newsroom had its own needs it brought to the CISLM team. “There’s a lot of pressure from the industry in terms of funders and journalism support organizations, who are encouraging local newsrooms to use AI without creating a robust, thoughtful, useful program that they can customize,” said Sarah Vassello, a product manager at CISLM. While some newsrooms were looking for customer service tools, others asked for “content focused” chatbots.

The News Reporter, a weekly newspaper in Columbus County, North Carolina, wanted a chatbot that could handle basic reader questions and subscription inquiries, especially outside of business hours. Henrico Citizen, a digital outlet serving greater Richmond, Virginia, asked for a mix of both customer service and content. Their bot could answer questions about how to advertise and also allowed readers to navigate its story archive.

Chapelboro, the site for a radio station in Chapel Hill, North Carolina, similarly wanted to juice its on-site search functionality with a chatbot that could pull up archived reporting and answer general newsroom questions. And Atlanta Civic Circle (ACC), a nonprofit publication focused on housing and politics in metro Atlanta, honed in on its mission and built a chatbot to answer questions about local elections and other civic issues.

Even with customization, though, only one of the newsrooms is walking away committed to running a chatbot beyond the program.

“One of the things we wanted to illustrate was that there’s an option out here to experiment and play with AI, and we’ll help you with that. But if it’s not meeting your needs, don’t feel that you must adopt this,” said Vassello. “Maybe they don’t have to use AI, maybe they don’t have to use a chatbot.”

Customer service vs. on-site search

Content-focused chatbots were by and large the hardest to build in CISLM’s program.

Chapelboro and Henrico Citizen both told CISLM they saw a chatbot as an opportunity to level up their on-site search. Since neither had a standalone archive, it was difficult to locate and fold in stories that were not actively featured on the homepage or topic pages. To get the chatbot demo ready, CISLM manually collected and sometimes scraped their sites to create the chatbot’s knowledge base.

Another issue was that a lot of content in the archive was time-bound. To keep the chatbot from outputting outdated information, the knowledge base had to be updated regularly with new stories. Henrico Citizen found that updating its chatbot “Henry” could pull focus from reporting, especially with a team of just three full-time and three part-time journalists.

Customer service chatbots were far easier to build. Newsrooms submitted a guide to CISLM with some of their most frequently asked and answered questions. These reference sheets were fed into the chatbot as a knowledge base.

The News Reporter was primed for this type of customer service chatbot. “They were dealing with a really small team and maybe an older audience base, who are really used to picking up the phone and calling or just coming to the front desk,” said Vassello. “It was taking away from their reporting time.”

“The News Reporter Help Desk” appeared as a pop-up on the bottom right-hand side of the site’s homepage.

More than 90% of the questions asked by readers, the Help Desk could give actionable answers to, like how to manage a subscription, how to submit a tip, or how to submit an obituary. All this information was largely static, so it rarely required updates.

To keep the chatbots on topic and to minimize errors, none of the chatbots were enabled to search the web. They could only pull from the knowledge base. One downstream effect of this guardrail was that often the chatbots simply declined to answer user questions, responding instead with variations on “I don’t know.” Altogether, 32% of conversations across the four chatbots included at least one question that triggered that response. Even simple questions like “who is the current president of the United States?” provoked an “I don’t know,” when there was no reporting involving Donald Trump in the knowledge base.

Another challenge for The News Reporter was getting readers to use the tool. From May 1 through June 15, The News Reporter Help Desk received just 11 inquiries.

“Most of our readers access our site via mobile devices, where screen space is limited and needs to be devoted to news content and advertising,” Rachel Smith, director of operations at The News Reporter, told me. Smith said that while there was a homepage bubble to alert readers about the chatbot, on mobile it was mainly accessible through the site’s support page. “Given this limited engagement, we chose not to invest further in the project.”

Across the board, the CISLM study documented low uptake. Each of the publications had different approaches to marketing and publicizing their tools, but during the 45 days the four tools received a combined 185 inquiries. Sun said there are too many variables to draw any direct conclusion about these low usage rates and that CISLM hopes to do more audience-focused research in the future.

There is reason to believe that some readers simply don’t like engaging with chatbots or are not looking for that experience on news sites. A recent survey study by Poynter and the University of Minnesota found that 49% of Americans have “no interest” in using an AI chatbot to get information from news organizations.

A “mistake-free” chatbot

In the case of Chapelboro and its chatbot “Chappy” — made in the style of Microsoft’s Clippy — a few readers did message the team directly asking to take it down. Ultimately, Chapelboro decided not to continue operating the chatbot after the program. The newsroom’s problem was less with adoption and reader feedback, and more because of the chance of responses containing hallucinations and outdated information.

“Accuracy is their highest value and that ultimately clashed with what they’re trying to do,” said Jessica Mahone, the interim director of CISLM. “It made too many mistakes to the point where it was not aligned with their mission.”

Chapelboro told CISLM they would pursue a chatbot in the future, only “if it one day could be mistake-free.”

Some hallucinations in other chatbots were resolved before launch. The News Reporter chatbot, for example, made up the newsroom office location, the days of the week the paper publishes, and the price of a paper. CISLM was able to make changes to the knowledge base during the demo phase and no further hallucinations were reported.

“To minimize hurt to audience trust, all the newsrooms mentioned in their promotion of the chatbot that it was an experimental program,” said Sun. Still, no chatbot that is powered by LLMs currently on the market is guaranteed to be “mistake-free.”

“Even with a higher budget or more resources, at the current stage I won’t make the promise that [a chatbot] will be mistake-free,” Sun said, encouraging links back to articles so users can find vetted information to verify the chatbot’s response. “We attempted to decrease the chance of mistakes, but there’s always this possibility.”

Atlanta Civic Circle had the most success with their chatbot, “Ask ACC,” and plans to continue operating an updated version of the tool this fall.

“Our audience wants quick, reliable access to information, especially around complex public policy and electoral politics,” said Saba Long, the executive director of ACC. With its policy explainer chatbot, it became the first newsroom in Metro Atlanta to roll out a major reader-facing AI tool. Now, some local nonprofits outside of journalism have reached out to explore whether a chatbot would work for their own engagement needs.

Since the CISLM program ended, ACC contracted Sun to make changes to the chatbot and roll it out again for Atlanta’s local election cycle this fall. That includes adding a feature that will auto-update the chatbot with new articles as they’re published on ACC’s election hub in order to keep its responses timely and accurate.

Part of ACC’s success was that their content-focused chatbot was bounded by a specific beat, which made it easier to build the knowledge base. From day one, “Ask ACC” also aligned with the publication’s mission.

“It feels to me less about searching and finding reporting and more about giving citizens information for civic engagement purposes. They as an organization see it as something that can extend their mission,” said Mahone. “All of our participants were really game to try this out and then learn how it aligned with what they do — or not.”