AI evaluation · Arabic NLP

How to Evaluate an Arabic Chatbot Before Launch

Nada Shaban · October 2026 · 6 min read

An AI chatbot can look great in a demo and still fail real users on day one. With Arabic, the risks multiply: dialects, mixed Arabic–English messages and users who switch language mid-conversation. This is the practical process I use to decide whether a bot is ready to launch.

1. Build a golden dataset first

A golden dataset is a fixed set of test questions with the answer or behavior you expect. It turns "it seems fine" into something you can measure and repeat after every change.

Tag every question so you can see where the bot fails, not only how often:

Tag	Examples
Intent	pricing, account help, cancellation, product question
Language & register	Egyptian Arabic, Gulf Arabic, MSA, English, mixed "Franco" Arabic
Test type	happy path, edge case, out of scope, adversarial
Difficulty	easy, medium, hard

Tip: write the same question in several dialects. "عايز ألغي الاشتراك" and "أبغى ألغي الاشتراك" mean the same thing, and the bot must handle both.

2. Automate the run

Manual testing does not scale and is hard to repeat. A small Python script can send every question to the bot, save each answer with a timestamp, and write everything to a spreadsheet. Rerun it after every prompt or model change and compare the results.

3. Check language match on every answer

One of the most common failures in bilingual bots: the user writes in Arabic and the bot answers in English, or the other way around. It is easy to miss in a demo and very visible to real users. Check it automatically on every answer: detect the script of the question and of the answer, and flag any mismatch.

4. Score with a simple, clear rubric

Correct: is the information right and complete?
Language: same language and a natural register for the user?
Safe: no invented facts, and a polite refusal when out of scope?
Tone: clear, friendly and on brand?

Keep it to a few criteria with clear definitions, so two reviewers would give the same score.

5. Report by segment, then decide

One overall score hides the real problems. Break results down by tag: a bot can pass 95% of English questions and fail half of the dialect ones. Then write a short report for decision makers:

What was tested and how
Results by language, intent and difficulty
The top issues, each with a real example
A clear recommendation: go, go with fixes or no-go

Key takeaway: test the way real users will talk to the bot, in every dialect and language they use, and make the launch decision from evidence, not from a demo.

I built a small open-source version of this workflow with synthetic data: arabic-chatbot-eval on GitHub.