Spanish fact-checking organisation, Fundación Maldita.es, processes hundreds of fact-checking requests daily through its community-driven chatbot system. As the organisation has grown since 2018, so has the complexity of managing vast amounts of data - from articles and expert databases to multimedia submissions requiring verification. The grant allowed them to tackle these scaling challenges by introducing AI into specific workflow points, focusing on modular integration that supports journalists rather than replacing them.

The problem: Understanding the user's needs

The inspiration for Maldita's AI assistant emerged from recognising a fundamental gap between the organisation's technological capabilities and those of disinformation spreaders. "We realised that our languages and our codes were a bit behind in comparison to those who spread disinformation," explains Pablo Pérez Benavente, the project's lead engineer.

The team discovered multimodal models capable of processing text, images, and voice messages - exactly the type of content flooding Maldita's fact-checking pipelines. But the primary motivation remained centred on supporting their journalists.

As knowledge became fragmented across the organisation, when writing a new article, fact-checkers often had to search for the previously published articles of Maldita in Google, using a not very efficient in house search engine or rely on colleagues who wrote the articles, which is challenging, especially to newcomers. At the same time, Maldita’s methodology requires that every article is supported by experts as sources. Maldita has a database that has been improved over the years but it was hard for fact-checkers to look for the exact experts they need when they are investigating a content. The AI assistant aims to streamline this retrieval process both for the articles and the experts, and eventually serve the broader fact-checking community through Maldita's technology stack.

Building the solution: From user research to prototype

The development followed a systematic approach across multiple work packages, each addressing different aspects of the fact-checking workflow to identify specific pain points where AI could provide meaningful assistance. The methodology began with extensive user research, including one-on-one interviews and focus groups.

The team then collaborated with Maldita's design team to visualise feasible solutions, creating both minimum viable scenarios and ideal development targets.

The roadmap involved iterative testing with end users throughout the development process. "We would do some testing usually with the end users, and then once the testing is done, either iteration listing out next steps or ideally deployment," Pérez Benavente explains. Insights from impact coordinators and social media specialists, not just fact-checkers, underscored the value of broad research.

Technical architecture and tools

The team used Coder for collaborative development, allowing engineers to work independently whilst sharing APIs. Hugging Face provided access to open-source models, whilst OpenAI's API offered additional capabilities.

The first work package developed a system for matching images, videos, texts, and voice messages. This “simple” task proved complex, with interconnected modules affecting each other’s performance. The second work package built an AI-powered search engine for Maldita's article database and experts database, a cornerstone for other Maldita’s tools like grounded chatbots and narrative analysis. The second work package included a new “Analyze with AI” function that, when implemented, will suggest to the fact-checkers both published articles of Maldita and possible experts to be contacted as sources. 

Assembling the right expertise

The project required skills that differed significantly from Maldita's previous AI implementations, as this project demanded more quantifiable, metrics-driven approaches requiring "core data science type of process" skills. The team needed expertise in planning different tests, defining test conditions, and identifying data variability combinations to ensure comprehensive evaluation.

Key roles included:

• An archivist with deep knowledge of Maldita's tagging schemes, labels, and data registration processes 

• An AI engineer focused on model development and integration 

• A backend engineer handling software development for API implementation

Having a global perspective on technical development proved vital to reduce misunderstandings across roles.

Navigating implementation challenges

Open-source model integration brought obstacles. Although Spanish is widely represented, frontier models often exist only in English, with performance in Spanish “clearly subpar,” notes Pérez Benavente.

Beyond language issues, the clean, structured data used for model training differed dramatically from "the collection of highly varied, messy and unbalanced datasets", like the screenshots and extremely long forwarded texts that Maldita processes daily. This forced "a modular system using different models for different kinds of data, affecting both usage and efficiency."

Performance evaluation was another hurdle. Full dataset testing proved unfeasible, and inconsistent documentation of test conditions made it hard to distinguish dataset issues from model shortcomings.

The opportunities and future directions

The AI-powered search engine serves as a foundation for future developments, including chatbots grounded in Maldita's article database and systems that combine articles with narrative analysis to identify debunked information.

The modular matching system also offers potential for expansion, now that its strengths and weaknesses are better understood.

The project also revealed internal coordination opportunities: multiple teams had unknowingly worked on related database improvements, showing the value of cross-team communication. Time constraints imposed by the grant also proved beneficial, pushing the team to prioritise effectively.

Lessons for newsrooms

• Start with systematic user research across departments: Maldita's approach of conducting extensive interviews and focus groups with various newsroom roles proved essential for identifying real needs rather than assumed problems.

• Plan for the complexity of modular AI systems: What appears simple often involves intricate, interconnected components where failure in one module affects the entire system. Careful conceptualisation at the beginning, identifying potential weak points, and understanding module dependencies can prevent significant delays and performance issues.

• Document everything, especially testing conditions: Without proper documentation, it becomes impossible to distinguish between dataset variations and actual model performance.