Indic-language AI safety evaluation

Read Oeran’s research brief on comparing AI safety and helpfulness across Indic languages, with proposed evaluation methods and related work.

What are we trying to measure?

Oeran’s first research effort asks whether an AI system’s safety and helpfulness change when a person uses an Indic language. A model should respond appropriately to risky requests while still helping people with ordinary requests on sensitive topics.

This brief describes a proposed study. Its scope is an initial set of five languages, with English as a point of comparison.

Languages we want to explore
  • తెలుగు Telugu
  • हिन्दी Hindi
  • தமிழ் Tamil
  • ಕನ್ನಡ Kannada
  • മലയാളം Malayalam

How could we compare languages?

  1. Compare the same intent across languages. Ask fluent reviewers whether scenarios carry the same meaning and tone before using them as pairs.
  2. Include locally written scenarios. Some situations need regional context, everyday phrasing, or code-switching to reflect how people actually communicate.
  3. Measure safety and helpfulness separately. Track both unsafe responses and unnecessary refusals. Record ambiguous cases and reviewer disagreement so readers can interpret results.

Why go beyond translation?

Translation helps isolate the effect of language when the underlying request stays the same. It can also flatten context, miss idioms, or sound different from how a fluent speaker would phrase the request. Locally written cases answer a different question about how a model behaves in realistic interactions.

  • IndicSafe

    Studies how model safety judgments vary across 12 Indic languages using scenarios grounded in South Asian contexts.

  • SurakshaEval

    Uses human-written scenarios in ten Indian languages and English to study safety in both shared and locally specific contexts.

Discuss the evaluation with Oeran