Nawaat, an independent Tunisian newsroom, has been in existence for over two decades. During this time, it has witnessed many historic political events and movements, including a dictatorship and the 2011 Tunisian revolution. The team responded to the call of bringing a long-held dream to life - to make the organisation’s vast historical archive across the English, French, and Arabic languages accessible to all, with the support of AI. 

The problem: Providing utility to its archival data

While the idea to implement this project with AI is relatively more recent, Nawaat is no stranger to working at the cutting edge of technology. In fact the idea for Nawaat AI dates back to 2011, when they had decided to create a tool they called the ‘Time Machine’ to understand the links between different entities - people, persons, events etc. At the time, they used MySQL and Wordpress to create taxonomies since the overall AI technology landscape wasn’t mature during the period. They still use wordpress as the basic framework for nawaat.org and taxonomies to structure content. Things changed post 2020 with the rise of AI and they revisited the idea with renewed enthusiasm. 

“We started even before getting the grant to build certain capabilities on AWS and testing some LLMs. So we've seen that the idea that was discussed and dreamt of 13 years ago is now really taking shape and facilitated with the use of AI,” said Sami Ben Gharbia, Co-founder of Nawaat.

While some implementation was taking place, this grant helped them break resource constraints like being able to hire the right people and developing the core AI infrastructure, said Houssem Hajlaoui, DevOps Engineer at Nawaat. 

Building the solution: Enabling discovery and record-keeping

Nawaat AI taps into the newsroom's archives and works twofold - both internally and also externally for its readers. It helps the journalists in the team to easily discover and access articles related to what they’re researching or writing even if they have only just joined the organisation. It also helps its readers engage with historical coverage of an issue. 

The overall tool allows readers to navigate a certain topic chronologically through history, or through the evolution of the theme itself. For example, this could be about how women’s rights evolved in Tunisia in the last 20 years and important milestones.

“It helps in content discovery and it resolves the issue of staff needing to know what Nawaat covered during the last 20 plus years. The other part is for our users, our readers, to understand the evolution of our coverage around a certain topic or certain person or certain region,” said Ben Gharbia. 

It includes an advanced search feature with a possibility to ask follow up questions, a Time Machine feature, and a timeline. In the chat feature, users can ask a question and receive a well-structured answer with references to Nawaat’s previous coverage. The tool also allows them to convert this into a timeline for quick and easy comprehension. The third, Time Machine feature, summarises a particular period that the user selects to reflect the crucial news moments during that period.

A greater and more crucial need for Nawaat AI’s Time Machine feature lies in the context of press freedom and freedom of speech in Tunisia, which is leaning towards a dictatorship.

“Nawaat AI is literally your primary source and this is not only a credit to Nawaat and the work that's been done for 20 plus years, but this is also because it’s in a dictatorship. If I am to research, for instance, women’s rights or feminism in the United States, I have 10 billion sources, because to the extent it's a democracy. But in the context of a country that was a dictatorship all the way to 2011 and lived through a democratic transition for 10 years, it's not that obvious. Now it's unfortunately relapsing back. Nawaat is literally the only place where you basically have the history of the country,” explains Hajlaoui.
 
To the team, it’s not just archival content they need to organise, but serves a higher mission of recording the history and struggles that their country has been undergoing over time. It is a way to overcome historical amnesia, and educate those “who didn’t live through it all”, said Hajlaoui.

The opportunities: Breaking siloes by working in interdisciplinary teams

The overall strength of staff at Nawaat stands at about 16. However, the team building out the tool itself was small and interdisciplinary covering development (infrastructure, design, frontend), user engagement, and editorial input. The technical team comprised external professionals including for frontend design, backend assistance, and AI/infrastructure development. Meanwhile, internal newsroom staff provided essential feedback and testing.

One challenge that the team kept in mind while developing the tool was the accuracy and relevance of articles that the system fetches. They also have to ensure that all content produced is in line with Nawaat’s editorial charter and principles.

Extensive research and optimisation, including providing the entire Nawaat archive as input and the editorial charter as guidelines, led to a very high rate of accuracy, said Hajlaoui. Nawaat AI aims for 95 - 98% accuracy. Still, human validation is required for content published on the main website, added Hajlaoui.

Other challenges the team faced included deciding which AI model to choose, what system to use while retrieving content, and, to avoid being too ambitious with their goals.

Lessons for newsrooms

• Plan for long-term financial sustainability: Develop a strategic plan to optimise and cover significant infrastructure costs (servers, storage, maintenance, AI processing) that will arise after the initial grant period to ensure the project's long-term survival.

• Explore product monetisation: Consider selling the product or service to other smaller, independent newsrooms in the region as a potential strategy to generate revenue and support ongoing operations.

• Focus on innovation over competition: Prioritise innovation as a survival strategy to navigate difficult operational periods or industry transitions, rather than innovating solely to gain a competitive edge in the market.