State of the art where the audio gets hard.

The first question every speech system asks

Before a machine can transcribe, translate or reply, it has to answer a question most of us never notice ourselves asking: which language is this? That is spoken language identification, or LID. Given a stretch of audio, the system names the language being spoken from a fixed list of candidates.

Humans are astonishingly good at it. Four-day-old infants already tell their mother’s language apart from an unfamiliar one, long before they understand a single word (Mehler et al., 1988). They are listening to rhythm and melody, not vocabulary. A machine must learn the same trick from the raw signal: sixteen thousand air-pressure samples a second, with no spaces, no letters and no script.

A useful picture is the front desk of a large multilingual hospital. The receptionist does not treat anyone. They decide which specialist you see. If they get that wrong, the best specialist in the building cannot help you, because they are not hearing your problem in your language.

In a modern voice pipeline, LID sits just after voice-activity detection, and its one label steers four things at once (Li, Ma & Lee, 2013):

• which speech recogniser, or which language setting, transcribes the audio;
• which direction a translation system runs;
• which language the assistant’s language model replies in, and which voice speaks the reply;
• which human team a call is handed to.

In India this is daily infrastructure, not a thought experiment. Kisan Call Centres answer farmers in 22 languages (Department of Agriculture & Farmers Welfare). The Tele-MANAS mental-health helpline routes callers to state cells by language (PIB, 2026). Bhashini offers automatic language detection across its national voice services (PIB, 2026). As our paper puts it, errors at this stage propagate: “a Maithili call sent to a Hindi recogniser degrades every later component.”

A short history: from phonotactics to self-supervised encoders

LID is one of the oldest problems in speech technology, and its history is a story of steadily richer representations of the same signal (Li, Ma & Lee, 2013).

Sound sequences (1990s). Languages differ in which sounds they allow and in what order. Early systems exploited that directly: recognise a string of phones, then score it with a small n-gram model per language. Zissman’s comparison of four such approaches on telephone speech set the template for a decade (Zissman, 1996), and NIST’s Language Recognition Evaluations turned the problem into a shared benchmark (Sadjadi et al., 2018).

Utterance embeddings (2010s). The field then learned to summarise a whole utterance as a single vector. i-vectors compressed an utterance’s acoustic statistics into a few hundred numbers (Dehak et al., 2011). Deep networks trained directly on language labels overtook them, especially on short clips (Lopez-Moreno et al., 2014). x-vectors pooled frame-level features into a learned embedding (Snyder et al., 2018); attentive statistics pooling let the network decide which frames mattered (Okabe et al., 2018); and ECAPA-TDNN refined the recipe into today’s workhorse (Desplanques et al., 2020). VoxLingua107 made 107-language training data widely available (Valk & Alumäe, 2021), and compact models such as NVIDIA’s AmberNet followed (Jia et al., 2023).

Self-supervised encoders (2020 onward). wav2vec 2.0 showed that a network could learn speech structure from unlabelled audio (Baevski et al., 2020). XLSR-53 and XLS-R scaled that across dozens, then 128, languages (Conneau et al., 2021; Babu et al., 2022), and w2v-BERT added masked prediction (Chung et al., 2021). In this era one pre-trained encoder sprouts several small heads: a CTC head for transcription (Graves et al., 2006) and a classification head for language. Meta’s MMS is the clearest example: 1,107-language ASR built from per-language adapters and CTC heads, shipped alongside separate LID models covering up to 4,017 languages (Pratap et al., 2024). NVIDIA found that multilingual self-supervised Conformers are “accidental learners” of language identity (Bartley et al., 2023). For Indian languages, IndicWav2Vec, IndicSUPERB and the 279k-hour MahaDhwani encoder brought the same approach home (Javed et al., 2022; Javed et al., 2023; Bhogale et al., 2025).

A parallel thread: the language as a token. Alongside this, researchers began folding LID into the recogniser. Watanabe, Hori and Hershey made the language a symbol the end-to-end model emits before the transcript (2017). Google trained single models on nine Indian languages, with and without a language-ID input (Toshniwal et al., 2018; Kannan et al., 2019). Later work added auxiliary LID objectives inside the encoder (Chen et al., 2023). That thread leads straight to the models that dominate today.

One model for everything: the text-generation turn

By 2020, natural-language processing had found an intoxicating idea. GPT-2 had framed language models as “unsupervised multitask learners” (Radford et al., 2019). T5 recast every NLP task as text-to-text (Raffel et al., 2020). Then GPT-3 showed that one next-token predictor with 175 billion parameters could translate, answer questions and do arithmetic from a prompt and a few examples, with no task-specific head at all (Brown et al., 2020).

The lesson that crossed into speech was simple and powerful. With enough data and compute, stop building a head per task. Turn every task into open-ended text generation, and let one model learn them all.

Whisper is where that lesson landed. An encoder reads 30 seconds of log-Mel spectrogram; an autoregressive decoder writes text. The paper is explicit about the design: “We use a simple format to specify all tasks and conditioning information as a sequence of input tokens to the decoder.” The first of those tokens is the language: “we predict the language being spoken which is represented by a unique token for each language in our training set (99 total)” (Radford et al., 2023). A Hindi transcript therefore begins:

<|startoftranscript|> <|hi|> <|transcribe|> <|notimestamps|> नमस्ते, आप कैसे हैं? <|endoftext|>

LID became a side effect of generation. The model’s belief about the language is simply whichever language token wins the first decoding step.

The label grew a script. Whisper’s tokens are two-letter ISO 639-1-style codes, with no notion of writing system. Later systems widened the code. Meta’s Omnilingual ASR, covering more than 1,600 languages, conditions on {language}_{script} pairs such as hin_Deva and urd_Arab: an ISO 639-3 language code joined to an ISO 15924 script code (Omnilingual ASR team, 2025; repo). Google’s Chirp 3 uses BCP-47 locale tags such as pa-Guru-IN (docs). Qwen3-ASR writes the language as a plain-text prefix before the transcript (Qwen Team, 2026).

The family tree kept growing. OWSM reproduced Whisper-style training on public data (Peng et al., 2023). Qwen-Audio unified more than 30 audio tasks under hierarchical tags (Chu et al., 2023). Mistral’s Voxtral (July 2025) joins a Whisper large-v3 encoder to Mistral language decoders, in 3B and 24B variants, adds a dedicated transcribe-mode token, and advertises “automatic language detection” (Mistral AI, 2025; Voxtral, 2025). Qwen3-Omni and Qwen3.5-Omni fold recognition into omni-modal models (Qwen Team, 2025; 2026). At the frontier, Gemini 3.5 Flash and OpenAI’s GPT-audio treat “which language is this?” as one more question to answer in text.

It is worth saying plainly: this was a real advance. One training run, one model, remarkable robustness, and language identification apparently for free. Qwen3-ASR reports 97.9% average LID accuracy across four mainstream benchmarks. The question this post asks is narrower. For the first decision in a voice pipeline, is “for free” good enough?

Our position: decouple the first decision

Despite all of this progress, we believe the language decision belongs in a model of its own. It can be optional for a pipeline that already knows its language. For any voice agent that does not, it is the first component you build.

The field is quietly converging on the same view from the other side. The newest recognisers increasingly ask for the language rather than detect it. Omnilingual ASR takes an optional language_script code, and conditioning on it “largely resolves the wrong-script problem” of its CTC models. NVIDIA’s Canary requires source_lang. AI4Bharat’s IndicConformer is called as model(wav, "hi", "ctc"). Meta’s MMS loads a per-language adapter. And vendors advise the same thing in their docs: Sarvam says to “always specify the language code when you know the language of the audio. This improves accuracy and reduces processing time” (Sarvam docs).

So the real question is not whether a voice system needs a language decision. It is who makes it, how well, and whether you can see it. A dedicated model gives you six things a language token cannot:

1. Weights tuned for one job. Every gradient step is spent separating languages, not predicting the next word.
2. A calibrated confidence. A probability you can threshold, so the system can abstain, ask, or fall back.
3. Its own time resolution. A decision per turn or per window, rather than one per 30-second chunk or per file.
4. Its own label space. Languages your recogniser has never seen, and an honest “none of these”.
5. Its own life cycle. Benchmarked, versioned and rolled back independently of the recogniser.
6. A small footprint. Small enough to run on a phone or a CPU, in front of any recogniser you choose.

The rest of this post defends that position in three passes: first the theory of why a dedicated objective learns language better, then the linguistic and data realities that make the problem hard, and finally the systems case.

Why a dedicated objective learns language better

This is the technical heart of the argument. It rests on three observations: the evidence for language lives in the acoustics, not the text; an autoregressive recogniser spends almost none of its training signal on that evidence; and a dedicated classifier can use a toolkit that a language token cannot.

The signal is far richer than the text

Written Hindi is a few Devanagari characters per word, three bytes each in UTF-8. Across 17 languages, spoken language carries linguistic information at a strikingly constant rate of about 39 bits per second (Coupé et al., 2019). The waveform that carries it runs at 256,000 bits per second (16 kHz, 16-bit), and even a compact 80-band log-Mel front end produces 8,000 numbers a second. That surplus is not noise. It holds the speaker, the room, the channel, the emotion, and, crucially, cues to the language that never reach the page.

Spoken-language-recognition research has long organised those cues into a hierarchy (Li, Ma & Lee, 2013): acoustic-phonetic (which sounds exist), phonotactic (which sequences are allowed), prosodic (rhythm, stress, tone) and lexical. Rhythm alone separates language classes measurably (Ramus, Nespor & Mehler, 1999). Our paper catalogues the cues that matter for India: the degree of retroflexion (stronger in Tamil than Hindi, weaker in Nepali), Hindi’s four-way stop series and Gujarati’s breathy vowels, vowel length that is contrastive in Dravidian languages but not in Bengali or Assamese, lexical tone in Punjabi, Dogri, Bodo and Meitei, and rare segments such as Sindhi implosives and Santali checked finals.

And one thing is missing from speech entirely: script. Devanagari alone serves at least ten of our 23 classes, and Hindi and Urdu are famously “one language, two scripts” (King, 1994). A model trained to emit text must guess a writing system from sound, a variable the audio simply does not contain.

LID is a problem about the acoustic signal. An objective defined on text tokens sees it only indirectly.

What each objective actually optimises

A dedicated LID model minimises a loss on a softmax over languages. Every parameter update follows the gradient of one question: which language?

Here x is the audio, ℓ the true language, and γ the focal parameter (γ = 0 recovers ordinary cross-entropy). Trained this way, deep classifiers drift toward neural collapse: features of each class tighten around their mean, and the class means spread into a maximally separated, equiangular configuration (Papyan, Han & Donoho, 2020). That is precisely the geometry you want when Hindi, Urdu and Maithili sit a few phonetic cues apart.

An autoregressive recogniser minimises something else: the average surprise of every token in the transcript, with the language token as just the first.

The language term carries a weight of 1/T. In Whisper’s tokenizer we measured about 1.1 tokens per English word and 5.7 per Hindi word, so a 30-second segment yields roughly 85 target tokens in English and 430 in Hindi. The language decision therefore receives about 1% of the supervision in English and about 0.2% in Hindi. The languages that most need a good LID get the smallest share of it. (This is our estimate, not a published measurement.)

Two further details weaken that signal. Whisper’s language targets were not human labels at all: “these language targets are sourced from the aforementioned VoxLingua107 model”, a separate LID classifier (Radford et al., 2023). And in the standard fine-tuning recipe the language token is fixed per dataset, so the “LID” target becomes a constant (Hugging Face, 2022).

Shared weights, competing objectives. When one network serves several objectives, their gradients can conflict, and the dominant objective wins the shared parameters (Yu et al., 2020; Sener & Koltun, 2018; Kendall et al., 2018). Multilingual models show the same pattern as negative interference and the “curse of multilinguality” (Wang, Lipton & Tsvetkov, 2020; Conneau et al., 2020). Inside an autoregressive recogniser, language identification is the minority objective, sharing every weight with next-token prediction.

Reading the wrong layer. A decoder cross-attends to the final encoder layer. But language identity does not peak there. Layer-wise analyses of self-supervised speech models show information shifting across depth (Pasad et al., 2021; Pasad et al., 2023). In our encoder, linear-probe macro-F1 for language rises to 0.56 at blocks 9–11 of the pre-trained model and collapses to 0.13 at block 17. After fine-tuning it peaks at blocks 11–12 (0.86).

The toolkit a language token cannot use

A dedicated model can be shaped around the problem. Each tool below has a literature behind it, and our staged study measured what it is worth.

• Pooling. Statistics and attentive statistics pooling compress a variable-length utterance into one vector, attending to the few frames that carry a retroflex burst or a tonal contour (Snyder et al., 2018; Okabe et al., 2018). In our study, pooling heads differed by only about 1 point: the head matters less than you would think.
• Layer aggregation. A learned softmax-weighted sum over every encoder block, as in SUPERB and ELMo’s scalar mix, lets the classifier read language where it lives (Yang et al., 2021; Peters et al., 2018). It added 0.9 points in-domain and lifted every acoustic family: telephony +2.8, noise +1.9, confusable pairs +1.4.
• The loss decides calibration. Focal loss down-weights easy examples by (1 − p)^γ, concentrating gradient on hard, confusable clips (Lin et al., 2017), and improves calibration (Mukhoti et al., 2020). Cosine-margin heads such as AM-Softmax and ArcFace sharpen embeddings but produce badly calibrated scores (Wang et al., 2018; Deng et al., 2019). In our study they reached an expected calibration error of 0.75–0.79; focal loss reached 0.023 with the best macro-F1. Post-hoc temperature scaling (Guo et al., 2017) then cut our all-23 test ECE from 0.054 to 0.008.
• Sampling for a skewed world. Temperature sampling, p_ℓ ∝ n_ℓ^1/T, trades head for tail (Arivazhagan et al., 2019). At T = 2 the two languages under 500 hours that the in-domain test covers, Dogri and Konkani, rose from 0.742 to 0.930 F1, and Dogri alone from 0.50 to 0.89. Stacking a second rebalancer (Menon et al., 2021) counted the prior twice and cost 3.5 points in-domain; the two largest classes collapsed, English from 0.906 to 0.699 F1 and Hindi from 0.713 to 0.344.
• Augmentation aimed at deployment. Duration cropping, MUSAN noise and babble, room impulses, 8 kHz codecs and SpecAugment (Ko et al., 2015; Snyder et al., 2015; Ko et al., 2017; Park et al., 2019). Cropping alone added 9.5 points at 0.5 s; multi-talker babble took our babble conditions from 0.804 to 0.920.
• Curriculum and weight averaging. A competence curriculum raised the weight of the confusable Indo-Aryan cluster as training progressed (Platanios et al., 2019), and a uniform “soup” of seven checkpoints ranked first of 31 candidates (Wortsman et al., 2022).

None of these levers is available to a language token without retraining the whole recogniser around it. A dedicated model can pull all of them at once.

When the language token is wrong, the whole inference breaks

In an autoregressive recogniser the language token is the first domino. Get it right and it is invisible. Get it wrong and every token after it inherits the mistake.

The mechanism: a wrong first token becomes a wrong task

An autoregressive decoder conditions each step on its own earlier outputs. In training it sees the true prefix; at inference it sees its own guesses. That mismatch, known as exposure bias, means an early error is carried forward rather than corrected (Bengio et al., 2015; Ranzato et al., 2016). The language token is the earliest error there is.

Whisper’s training data explains what the model does next. When a clip’s audio language and transcript language disagreed, OpenAI dropped it from recognition training, but “if the transcript language is English” they kept the pair “as X→en speech translation training examples instead” (Radford et al., 2023, §2.1). A mismatch between the language token and the audio is therefore something the model learned to treat as translation. English audio decoded with --language fr comes back as fluent French (openai/whisper #2285). Peng et al. even observed Whisper translating code-switched speech into Mandarin, a direction it was never trained on (Peng et al., 2023).

This is what makes the failure dangerous. The output is fluent, so it looks like success. It belongs with the failure modes the hallucination literature has documented in neural recognisers: output that is “fluent and coherent” but untethered from what was said (Frieske & Shi, 2024); whole invented phrases in about 1% of Whisper transcripts, 38% of which contained explicit harms (Koenecke et al., 2024); and, in Whisper’s own words, “complete hallucination where the model will output a transcript entirely unrelated to actual audio.”

The measurements

When researchers split results by whether the language was identified correctly, the gap is not a few points. It is the difference between a usable transcript and a useless one.

Adversaries know this too. By fooling only Whisper’s language detector, Olivier and Raj pushed WER above 100% for Armenian and Lithuanian and to 75% or more for Czech, Danish and Indonesian; outputs became “a mostly nonsensical mix of the true and target language” (Olivier & Raj, 2023).

The same conclusion appears across very different architectures. “Language-agnostic ASR performance highly relies on the accuracy of LID as wrong LID predictions may lead to transcription in the wrong language” (Kwok, Yip & Chng, 2024). “Performance can deteriorate dramatically when LID is absent”, because of “cascaded errors inherent in the implicit language identification and monolingual recognition pipeline” (Li et al., 2025). “A key to understanding their imbalanced performance across different languages is to examine if the model actually knows which language it should transcribe” (Chen et al., 2023). And for CTC models, which have no token at all, the failure shows up as script: they “often fail due to script misprediction… the decoded characters belong to another language altogether” (Omnilingual ASR team, 2025).

In the wild

• Speech-LLMs. A Google DeepMind study reports that “models often misidentify the output language, compromising transcription fidelity and downstream application quality”; Gemini Flash produced wrong-language output on 0.6–2.2% of clips with no language hint (Kim et al., 2026). At a million calls a day, that is 6,000 to 22,000 wrong-language transcripts.
• Hindi and Urdu. Whisper users report Hindi detected as Urdu, and Urdu or Persian script even when language='hi' is set (#1662). A Hindi speaker’s audio in a realtime voice agent came back “in japanese, urdu, and some other scripts” (openai-realtime-agents #46).
• Model upgrades. Whisper large-v3 transcribed an English clip as Indonesian where large-v2 did not; a Hugging Face maintainer called it “a model regression from large-v2 -> v3” (transformers #27937).
• One decision per file. openai-whisper prints “Detecting language using up to the first 30 seconds” and applies the answer to the entire recording (transcribe.py). Forty seconds of leading silence led whisper.cpp to decide the audio was Catalan (#1104). Whisper’s maintainer is candid: “It’s intended for monolingual audio inputs” (#1160).

Catastrophic WER, then hallucination

Wrong LID not only causes catastrophic WER but also triggers hallucination in autoregressive speech models. The two failures are easy to run together, and the arithmetic of the error rate separates them. Word error rate counts the substitutions, deletions and insertions needed to turn a transcript into what was said, and divides by the number of words said: (S + D + I) / N. A recogniser that writes every word in the wrong script, or translates every word, substitutes all of them and scores 100%. That is catastrophic, but it is bounded: the model is still following the audio, one wrong word for each word spoken, as in Figure 7. Substitutions and deletions can never outnumber the words spoken, so an error rate above 100% has only one source: insertions, words nobody said. Past 100%, a recogniser has invented more words than it recognised.

Error rates past that line turn up wherever the language decision fails. The attack on Whisper’s language detector described above pushed them past it for Armenian and Lithuanian (Olivier & Raj, 2023). On Pashto, where no Whisper model wrote Pashto script in more than 0.8% of utterances, zero-shot word error rates ran from 90% to 297%, and the medium model reached 461% on Common Voice; as the author notes, “a model generating Arabic-script output on Pashto audio has not achieved ASR in any interpretable sense” (Rahman, 2026).

Why the model keeps writing. The training objective explains the second failure. A recogniser trained only to predict the next token has learned that fluent text is rewarded whether or not the audio supports it. Studying LLM-based recognisers, Shivakumar and colleagues describe hallucinations as outputs with “high insertion rates” that “arise when the model deviate from input stimuli (speech signal), prioritizing distributional patterns resulting in semantically and phonetically unrelated outputs”. On meeting audio it was not trained on, their 8-billion-parameter model reached word error rates of 100.26% on headset microphones and 227.98% on distant ones, and on the distant microphones insertions alone came to 195.4 words for every hundred spoken. In the authors’ words, “the WER explodes to greater than 100% which is suggestive of increased hallucinations”. A reinforcement-learning stage that rewards low error rates directly cut those insertions to 9.0 and the error rate to 39.69% (Shivakumar et al., 2025; Figure 7c). The lesson for the language token is direct. It is the strongest “distributional pattern” a decoder is ever handed: it sets the prior for every step that follows. Hand it the wrong one and the model is being asked, from its first step, to prefer its prior to the audio.

The shape of the failure. What follows a wrong token has a recognisable form. The decoder starts in the wrong script or the wrong language, meets words that its prior cannot place, fills them with words that were never spoken, and often falls into a loop, because once a phrase has repeated a few times the repetition is itself the most likely continuation. With 30% of the audio masked, Whisper-small produced 51,797 insertions, 86% of them in such loops, “catastrophic autoregressive cycling where the decoder repeats phrases hundreds of times”; on Indian-accented speech, Whisper-large-v3’s insertion rate reached 9.62% (Ginjala et al., 2026). Long-form decoding “often exhibit[s] hallucinations, repetition loops, and content omissions” (Ahn et al., 2026), and non-speech segments alone are enough to trigger a recurring set of invented phrases (Barański et al., 2025). Figure 7b draws the pattern out word by word for a Hindi, an Urdu and a Maithili caller. Handed the wrong token, the recogniser loops on one phrase twenty times and writes about three times as many words as were spoken, so its error rate climbs to about 300%; handed the right code by a dedicated identifier, it returns the sentence as spoken.

Data and linguistic constraints: where the problem gets hard

On clean, long, read speech in well-resourced languages, LID looks solved. Deployed audio is none of those things. The two largest multilingual speech efforts of the decade open with the same admission. MMS: “current speech technology is restricted to about one hundred languages which is a small fraction of the over 7,000 languages spoken around the world” (Pratap et al., 2024). Omnilingual ASR: “most of the world’s 7,000+ languages remain unsupported, leaving thousands of long-tail languages behind. Expanding ASR coverage has been costly and limited by architectures that restrict language support” (Omnilingual ASR team, 2025).

LID inherits both halves of that problem. Few languages have data, and the ones that sound alike crowd each other. India has both at once.

Twenty-two languages by constitutional design

India’s Eighth Schedule began with 14 languages. Sindhi was added in 1967; Konkani, Manipuri and Nepali in 1992; Bodo, Dogri, Maithili and Santali in 2004 (Ministry of Home Affairs, 2017). English is an official language of the Union by statute but is not scheduled (Official Languages Act, 1963).

The 2011 Census rationalised 19,569 raw mother-tongue returns into 1,369 mother tongues and grouped them into 121 languages; 96.71% of the population returned a scheduled language. Speaker numbers span four orders of magnitude, from Hindi (528M, 43.6%) to Bodo (1.5M) and Sanskrit (24,821) (Office of the Registrar General, 2018). The labels are administrative, not uniform linguistic units. The Census category “Hindi” groups 56 mother tongues, including Bhojpuri (50.6M), Rajasthani (25.8M), Chhattisgarhi (16.2M) and Magahi (12.7M), and Maithili itself was counted under Hindi until 2001. The government notes that no fixed criterion separates a language from a dialect. An LID system must learn class boundaries that are partly conventional.

Four families, one linguistic area

Fifteen scheduled languages are Indo-Aryan, four Dravidian, two Tibeto-Burman and one Austroasiatic (Munda). Indo-Aryan speakers make up 78.1% of the population and Dravidian speakers 19.6% (Glottolog 5.3; Census 2011).

Genealogy understates how alike these languages sound. Emeneau (1956) described India as a linguistic area: across families, most languages have retroflex consonants contrasting with dentals, verb-final clauses with postpositions, chains of conjunctive participles and echo words. Masica (1976) extended the analysis. Retroflex systems are distributed geographically rather than genetically (Arsenault, 2017), and Assamese is a notable exception, having merged dentals and retroflexes into alveolars (Mahanta, 2012). Indian LID research has accordingly noted heavily overlapping phone sets (Dey, Sahidullah & Saha, 2022). On FLEURS, the South Asian group has the lowest LID accuracy of any language group (Conneau et al., 2023).

Close relatives and dialect continua

Several scheduled languages are sisters, or former “dialects”, of one another:

• Hindi and Urdu are sister languoids under Hindustani and share one colloquial form (Hammarström et al., 2026; Kachru, 2008). Urdu→Hindi is the single most common error in VoxLingua107 (Valk & Alumäe, 2021).
• Punjabi and Dogri share 93% of basic vocabulary (Kogan, 2016), and Grierson (1916) described Dogri as a dialect of Punjabi.
• Konkani and Marathi share 80–82% of basic vocabulary; Grierson (1905) treated Konkani as a Marathi dialect.
• Assamese, Bengali and Odia share 86–88% of basic vocabulary and one Glottolog node, and share numeral-classifier morphology with Maithili (Emeneau, 1956).
• Maithili speakers from the eastern dialect area commonly shift to Hindi when talking to speakers from other regions (Central Institute of Indian Languages).

Lexicostatistics measures vocabulary, not acoustics. But these are exactly the pairs where LID systems err, ours included: Hindi→Maithili (4.7% of Hindi clips), Urdu→Hindi (4.0%), Odia→Assamese (3.3%), Konkani→Marathi (2.3%). Some confusions cross family lines but not geography: Assamese→Bodo (3.0%) and Manipuri→Bodo (2.7%) reflect centuries of contact in north-eastern India, an areal similarity that no family tree records (Thomason, 2000). Of all our residual errors, 61% stay within the reference language’s family.

Scarce and skewed data

Speech resources do not track population. Urdu has more than 50 million first-language speakers but less audio in our pool than Maithili or Nepali, while English is abundant because most of its 128.5 million Indian speakers use it as a second or third language. Our 54,643-hour training pool holds 13,039 hours of English and 369 of Kashmiri, a 35-fold imbalance. Most openly available LID models cover only 15–21 of the 23 classes.

Low-resource languages are left with scarce training data. Figure 9 sets every class’s hours in our pool against its first-language speakers. The head is heavy: eight classes have more than 2,000 hours each, led by English (13,039) and Hindi (8,121), and together they hold 76% of the pool. The tail is thin: Kashmiri (369 hours), Konkani (375), Sindhi (390), Santali (394) and Dogri (478) have under 500 hours each, though each is the first language of 2.3 to 7.4 million people, and Manipuri (539) and Bodo (590) are barely above them. Between them the five scarcest hold 2,006 hours, under 4% of the pool and less than a sixth of what English has on its own. Per speaker the spread is wider still: about 15 hours per million speakers for Hindi and 14 for Urdu, but nearly 400 for Nepali and Bodo, so a language’s size does not predict how much of it a model will hear.

This is a pool assembled for a dedicated identifier from 18 corpora, and it is still thin enough to constrain the research: the balanced 1,000-hour subset of our first experiments could hold only 20 of the 23 classes, because Kashmiri, Santali and Sindhi had too little audio at that scale. In our staged study, trained on the natural distribution, the two languages under 500 hours that the in-domain test covers, Dogri and Konkani, reached 0.742 F1 against 0.949 for the eight largest; temperature sampling, described above, closed most of that gap.

General-purpose ASR is skewed further still. Of Whisper’s 680,000 training hours, 65% are English; 98 other languages share 117,000 hours, and “WER halves for every 16× increase in training data” (model card; Radford et al., 2023). Whisper has no language token at all for 8 of the 22 scheduled languages: Odia, Maithili, Konkani, Dogri, Bodo, Santali, Manipuri and Kashmiri (tokenizer.py). Omnilingual ASR trains on 63,592 hours of English but 236.7 of Hindi, 7.6 of Manipuri (character error rate 51.1%) and 7.3 of Kashmiri (165.8%) (results table).

Here a dedicated LID has a structural advantage that is easy to miss. A recogniser needs transcripts; a language identifier needs one label per clip. The IISc Vaani corpus holds 205.5 hours of Manipuri audio but 0.46 hours transcribed; 465.5 hours of Maithili against 20.8; 31.5 hours of Kashmiri against 0.06 (Vaani). IndicVoices collected 7,348 hours and transcribed 1,639 (Javed et al., 2024). A standalone LID can learn from all of that audio. A joint recogniser can learn language only from the transcribed slice. For the hardest, lowest-resource languages, that is a 20- to 500-fold difference in usable data.

The speaker: multilingual, code-mixing, accented

In 2011, 26.0% of Indians reported a second language and 7.1% a third (Census 2011, Table C-17). About 17% of Hindi–English social-media posts mix languages (Bali et al., 2014), and in code-switched speech more than half of the embedded segments last only 200–400 ms (Barras et al., 2020): shorter than many single words. The language changes faster than any 30-second window can follow. Language diarization, locating where each language is spoken, has become its own benchmark on Indian conversations, and it is far from solved: the best team at DISPLACE 2024 still had a 25.05% language-diarization error rate (Baghel et al., 2023; Kalluri et al., 2024).

“India is not a contest over who can post the best accuracy number. It is a man on a four-thousand-rupee phone, a ceiling fan whirring overhead, who begins a sentence in Hindi, finishes it in Bengali, and drops an Indian-accented English word somewhere in the middle.”

That English word is its own challenge. English has only 0.26 million first-language speakers in India but 128.5 million speakers in total. Indian English is an L1-influenced variety (Kachru, 1983), with apical, often retroflex /t d/, a merged /v/–/w/ and syllable-based timing (Masica, 1972; Wiltshire & Harnsberger, 2006). Acoustically, it sits closer to the other 22 classes than native English does. General-purpose LID struggles here: in one study, per-speaker accuracy on English ranged “from 35% for a speaker with a distinctive Indian accent to 100%” (Kukk & Alumäe, 2022). Mother-tongue influence runs the other way too: our own model confuses English mostly with Hindi (2.7%), Santali (2.0%) and Marathi (1.7%), exactly the L1-conditioned pattern the phonetics literature predicts.

The channel: short, noisy and narrowband

Voice queries last one or two seconds. A large share of rural access is 8 kHz telephony: G.711 caps bandwidth at 4 kHz (ITU-T, 1988), and 41% of India’s 1.35 billion telephone subscriptions are rural (TRAI, 2026). Many of the phonetic cues above are sparse, and a one-second clip may contain none of them. Systems that look strong in-corpus fall apart across corpora: Indian LID has been measured at 11.8% equal error rate in-corpus against 43.3% cross-corpus (Dey, Saha & Sahidullah, 2021), and IndicSUPERB accuracy falls from 98.2% on seen speakers to 90.8% on unseen ones (Javed et al., 2023).

Not only India

Confusable clusters are the rule wherever languages share a history, which is why the field’s flagship evaluations are built around them.

• NIST’s 2017 Language Recognition Evaluation focused on “differentiating closely related languages (14 in total) drawn from 5 language clusters, namely Arabic, Chinese, English, Iberian, and Slavic” (Sadjadi et al., 2018).
• Its 2022 successor found the greatest confusability among Southern African languages (Ndebele, Tsonga, Venda, Xhosa, Zulu), between South African English and Indian-accented South African English, and among Tunisian, Algerian and Libyan Arabic (Lee et al., 2023).
• Arabic is a dialect continuum across seventeen countries (Shon et al., 2020).
• In Europe, bilingual LID reaches only 80.0% on Finnish–Estonian, and misidentified Danish–Swedish clips carry 59.7% WER (Amooie et al., 2026).
• In Southeast Asia, Qwen3-ASR’s remaining FLEURS errors “stem from confusion between Malay (ms) and Indonesian (id)” (Qwen Team, 2026).
• In China, Whisper large-v3 labels Cantonese as Mandarin zh rather than yue (#2419).

India is the extreme case: every one of these properties at once, across 1.4 billion people.

Specialization erodes what little LID there is

General-purpose recognisers are trained on the internet’s idea of speech: podcasts, lectures, videos. Many of the places where speech technology matters most sound nothing like that.

The domains that need their own recognisers

• Medicine. Clinical documentation lives on drug names, dosages and anatomy spoken fast by tired people. On real clinical interviews in Indian hospitals, Whisper large-v3 scored 46.8% WER in English, 71.7% in Hindi and 98.6% in Kannada (Kumar et al., 2025). The first multilingual medical ASR dataset appeared only recently (Le-Duc et al., 2025), which tells you medical ASR has been built one language at a time.
• Law. Adalat AI transcribes proceedings in roughly a quarter of India’s courtrooms. Its engineers found Indic transcripts stopping mid-sentence because Whisper’s token cap is too small for Indic scripts, and had to shrink their windows to six or seven seconds (Adalat AI).
• Finance. Earnings calls mix jargon, numbers and speakers from every accent; Earnings-22 was built precisely because general models struggle on them (Del Rio et al., 2022).
• Aviation. Air-traffic-control speech is fast, formulaic, noisy and accented, and open models are fine-tuned specifically for it (van Doorn et al., 2024).
• Children. Child speech differs acoustically from adult speech, and Whisper must be fine-tuned to close the gap (Attia et al., 2024).
• Public services. Kisan Call Centre agronomy, Tele-MANAS mental-health calls and Bhashini’s citizen services each carry their own vocabulary, in 22 languages.

The standard remedy is to fine-tune a general model on domain data. Prompt-conditioned fine-tuning of Whisper cut WER by up to 33% on unseen domains, including medical conversation, air-traffic control and financial meetings (Liao et al., 2023). It works. But look at what it does to language identification.

What fine-tuning does to the language decision

It forgets other languages. Fully fine-tuning Whisper on four new languages raised the average WER on its four original languages from 11.68% to 32.29%; Chinese went from 14.37% to 68.93% (Song et al., 2024). Fine-tuning Whisper large-v3 on 12 hours of Irish took Welsh from 27.5% to 96.8% WER and Occitan from 64.2% to 79.8% (Qian et al., 2024). Notably, the Irish was trained under the English language token, because adding a new token performed slightly worse. The language slot had become a convenience, not an identity.

It confuses old languages with new ones. After Whisper was extended to ten new languages, most language-agnostic errors came from English audio being labelled as Esperanto, one of the additions (Kwok, Yip & Chng, 2024).

It trains the language slot to a constant. The standard fine-tuning recipe fixes the language for the dataset, so every training example carries the same language target (Hugging Face, 2022). A medical Hindi model is taught to say “Hindi” every time, whatever it hears.

There is an honest nuance. Probing a Whisper encoder after English and multilingual medical fine-tuning, Kahali et al. found language identity still almost perfectly linearly separable (F1 ≥ 0.99 from the first layer on) (Kahali et al., 2026). The knowledge survives in the encoder. What degrades is the decoder’s one-token read-out of it, which is exactly what a dedicated classifier replaces.

Why this makes an external LID mandatory

Every standard fix for forgetting moves the language decision outside the model. Language-specific LoRA modules must be selected by language (Song et al., 2024). MMS adapters are loaded by language. A fleet of domain models (medical Hindi, legal Tamil, agricultural Marathi) is routed by language. Specialization does not merely weaken built-in LID. It makes an external LID a requirement, and one well-built LID can then serve the entire fleet.