Building LakFeed's Category System: Harder Than It Looks

When you visit LakFeed and tap "Sports" or "Business", you're seeing the result of a categorisation system that classifies thousands of articles per day across three languages. That system is simple on the surface and moderately complex underneath — and it taught us a few things about the gap between "this should be easy" and "this actually works."

The Naive Approach

Our first implementation was keyword matching. We built a list of terms for each category in each language, and if an article's title or body contained enough of them, it got tagged.

This worked well for Sports — "cricket", "ක්‍රිකට්", "கிரிக்கெட்" are unambiguous. It worked less well for Politics, where terms like "minister" and "parliament" appear in business news, crime reporting, and sports (politicians attend sports events, it turns out). And it fell apart for "Technology", where the Sinhala and Tamil vocabulary is a mix of loanwords, calques, and transliterations that don't map cleanly to a fixed term list.

The Problem with Sinhala NLP

For English, there are well-established NLP libraries, pre-trained classification models, and large labelled datasets you can fine-tune against. For Sinhala, the tooling is thinner. There is active research — the University of Moratuwa and others have published work on Sinhala NLP — but production-grade libraries that you can drop into a pipeline are rare.

Tamil has better tooling than Sinhala, partly because the Tamil-speaking developer community is larger globally. But even for Tamil, the available models are mostly trained on formal written Tamil from Tamil Nadu, which differs enough from Sri Lankan Tamil usage that off-the-shelf classifiers make mistakes on local news content.

What We Actually Built

Our current system uses a hybrid approach:

Source-level priors. Some sources are heavily category-skewed: a sports broadcaster publishes almost entirely sports content, a business daily publishes almost entirely business content. We weight these priors heavily. An article from a known sports source gets a Sports prior even before we look at the content.

Title keyword matching with fuzzy normalisation. Rather than exact keyword matching, we normalise Sinhala text through Unicode canonical composition (NFC), strip diacritics for matching purposes, and apply a small set of morphological rules to handle common suffixes. It's not full morphological analysis, but it handles the most common variance.

Manual category overrides. For edge cases and systematic errors, we have an internal tool that lets us mark a classification as wrong and feed it back into the system. We use this sparingly, but it's been essential for catching patterns we hadn't anticipated.

Fallback to "General". If the system isn't confident, it doesn't guess. The article goes into General, which is better than a confident wrong answer in Sports.

What We're Improving

The current system struggles with:

• Opinion pieces that reference multiple topics without being "about" any of them
• Breaking news that arrives without enough context in the title or early paragraphs
• Transliterated proper nouns that look like category keywords but aren't (a politician named "Sportsminister" is a problem we've actually had to handle)

We're experimenting with a lightweight embedding-based classifier trained on our own labelled data. It's slower per article but substantially more accurate on the edge cases above. We expect to roll it into production before the end of the year.

The Lesson

Categorisation feels like a solved problem until you try to solve it in a low-resource multilingual setting. The right level of ambition is: build something that works well for the common cases, handles the edge cases gracefully, and is honest about uncertainty. The wrong level of ambition is: build something that guesses confidently when it shouldn't.

We're still calibrating. But the categories on LakFeed today are substantially more accurate than they were at launch, and they'll keep improving.