Amharic Speech-to-Text: Acoustic Architectures and Challenges

1. Acoustic and Morpho-Syntactic Obstacles

Proclitics and Word Spacing

Prepositions, conjunctions, relative pronouns, and negation markers attach directly to Amharic nouns and verbs as proclitics and enclitics:

Full Orthographic Word:  ስለማይመለከተው (səlemayəmmäläkätäw)
Decomposition:           ስለ- (because) + -ማይ- (which does not) + -መለከት- (concern) + -ው (him)
English Translation:     "Because it does not concern him"

Because Ge’ez historically used the two-dot word divider (፡), modern written Amharic has inconsistent whitespace. If a model transcribes ስለ ማይመለከተው with an inserted space, Word Error Rate (WER) scores two word errors for a nearly identical character sequence.

Subword SentencePiece Byte-Pair Encoding (BPE) tokenization trained on normalized text reduces this mismatch.

Code-Switching

Urban and diaspora speakers often switch between Amharic and English within a sentence:

Speaker Audio: "በጣም sorry፣ meeting-ኡን postpone ማድረግ አለብን።"
ASR Output:    "በጣም ሶሪ ሚቲንጉን ፖስትፖን ማድረግ አለብን።" (Phonetic Fidel)
Alternative:   "በጣም sorry meeting-ውን postpone ማድረግ አለብን።" (Mixed Script)

Monolingual models can hallucinate Ethiopic syllables on English phoneme sequences. Handling this takes bilingual pre-training and a multi-script vocabulary.

---

2. CTC vs. Autoregressive Encoder-Decoder

flowchart TD
    subgraph CTC ["CTC Acoustic Modeling (Wav2Vec 2.0 / Conformer-CTC)"]
        direction TB
        A1["Acoustic Audio (16kHz)"] --> B1["Convolutional Feature Encoder"]
        B1 --> C1["Conformer / Transformer Blocks"]
        C1 --> D1["Linear CTC Projection Layer"]
        D1 --> E1["Fidel Syllable Greedy / Beam Search"]
    end

    subgraph EncDec ["Autoregressive Encoder-Decoder (Whisper)"]
        direction TB
        A2["Log-Mel Spectrogram"] --> B2["Audio Encoder (Transformer)"]
        B2 --> C2["Cross-Attention Mechanism"]
        C2 --> D2["Autoregressive Text Decoder"]
        D2 --> E2["Conditioned Text Tokens (with LM Prior)"]
    end

Architectural Dimension	CTC Models (Wav2Vec2-XLSR-53 / Conformer)	Sequence-to-Sequence (Whisper-large-v3)
Decoding	Non-autoregressive; suits real-time streaming pipelines.	Autoregressive token-by-token generation; slower.
Language Model Prior	Outputs acoustic predictions. Requires external n-gram/neural LM beam rescoring.	Internal language model resolves grammar, spacing, and punctuation.
Failure Mode	Aligns acoustic frames directly to character emissions.	Repetition loops and hallucinations during background noise or silence.

For real-time voice (telephony, live captioning), use Conformer-CTC or fine-tuned Wav2Vec2. For offline transcription of podcasts and legal depositions, Whisper-large-v3 gives better punctuation and formatting.