Microsoft Speech Recognition: Alternate results with confidence score?

Question

I'm new to working with the Microsoft.Speech recognizer (using Microsoft Speech Platform SDK Version 11) and I'm trying to have it output the n-best recognition matches from a simple grammar, along with the confidence score for each.

According to the documentation (and as mentioned in the answer to this question), one should be able to use e.Result.Alternates to access the recognized words other than the top-scoring one. However, even after resetting the confidence rejection threshold to 0 (which should mean nothing is rejected), I still only get one result, and no alternates (although the SpeechHypothesized events indicate that at least one of the other words does seem to be recognized with non-zero confidence at some point).

My question: Can anyone explain to me why I only get one recognized word, even when the confidence rejection threshold is set to zero? How can I get the other possible matches and their confidence scores? What am I missing here?

Below is my code. Thanks in advance to anyone who can help :)

---

In the sample below, the recognizer is sent a wav file of the word "news", and has to select from similar words ("noose", "newts"). I want to extract a list of the recognizer's confidence score for EACH word (they should all be non-zero), even though it will only return the best one ("news") as the result.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using Microsoft.Speech.Recognition;

namespace SimpleRecognizer
{
    class Program
    {
        static readonly string[] settings = new string[] {
            "CFGConfidenceRejectionThreshold",
            "HighConfidenceThreshold", 
            "NormalConfidenceThreshold",
            "LowConfidenceThreshold"};

        static void Main(string[] args)
        {
            // Create a new SpeechRecognitionEngine instance.
            SpeechRecognitionEngine sre = new SpeechRecognitionEngine(); //en-US SRE

            // Configure the input to the recognizer.
            sre.SetInputToWaveFile(@"C:\Users\Anjana\Documents\news.wav");

            // Display Recognizer Settings (Confidence Thresholds)
            ListSettings(sre);

            // Set Confidence Threshold to Zero (nothing should be rejected)
            sre.UpdateRecognizerSetting("CFGConfidenceRejectionThreshold", 0);
            sre.UpdateRecognizerSetting("HighConfidenceThreshold", 0);
            sre.UpdateRecognizerSetting("NormalConfidenceThreshold", 0);
            sre.UpdateRecognizerSetting("LowConfidenceThreshold", 0);

            // Display New Recognizer Settings
            ListSettings(sre);

            // Build a simple Grammar with three choices
            Choices topics = new Choices();
            topics.Add(new string[] { "news", "newts", "noose" });
            GrammarBuilder gb = new GrammarBuilder();
            gb.Append(topics);
            Grammar g = new Grammar(gb);
            g.Name = "g";

            // Load the Grammar
            sre.LoadGrammar(g);

            // Register handlers for Grammar's SpeechRecognized Events
            g.SpeechRecognized += new EventHandler<SpeechRecognizedEventArgs>(gram_SpeechRecognized);

            // Register a handler for the recognizer's SpeechRecognized event.
            sre.SpeechRecognized += new EventHandler<SpeechRecognizedEventArgs>(sre_SpeechRecognized);

            // Register Handler for SpeechHypothesized
            sre.SpeechHypothesized += new EventHandler<SpeechHypothesizedEventArgs>(sre_SpeechHypothesized);

            // Start recognition.
            sre.Recognize();

            Console.ReadKey(); //wait to close

        }
        static void gram_SpeechRecognized(object sender, SpeechRecognizedEventArgs e)
        {
            Console.WriteLine("\nNumber of Alternates from Grammar {1}: {0}", e.Result.Alternates.Count.ToString(), e.Result.Grammar.Name);
            foreach (RecognizedPhrase phrase in e.Result.Alternates)
            {
                Console.WriteLine(phrase.Text + ", " + phrase.Confidence);
            }
        }
        static void sre_SpeechRecognized(object sender, SpeechRecognizedEventArgs e)
        {
            Console.WriteLine("\nSpeech recognized: " + e.Result.Text + ", " + e.Result.Confidence);
            Console.WriteLine("Number of Alternates from Recognizer: {0}", e.Result.Alternates.Count.ToString());
            foreach (RecognizedPhrase phrase in e.Result.Alternates)
            {
                Console.WriteLine(phrase.Text + ", " + phrase.Confidence);
            }
        }
        static void sre_SpeechHypothesized(object sender, SpeechHypothesizedEventArgs e)
        {
            Console.WriteLine("Speech from grammar {0} hypothesized: {1}, {2}", e.Result.Grammar.Name, e.Result.Text, e.Result.Confidence);
        }
        private static void ListSettings(SpeechRecognitionEngine recognizer)
        {
            foreach (string setting in settings)
            {
                try
                {
                    object value = recognizer.QueryRecognizerSetting(setting);
                    Console.WriteLine("  {0,-30} = {1}", setting, value);
                }
                catch
                {
                    Console.WriteLine("  {0,-30} is not supported by this recognizer.",
                      setting);
                }
            }
            Console.WriteLine();
        }
    }
}

This gives the following output:

Original recognizer settings:
  CFGConfidenceRejectionThreshold = 20
  HighConfidenceThreshold        = 80
  NormalConfidenceThreshold      = 50
  LowConfidenceThreshold         = 20

Updated recognizer settings:
  CFGConfidenceRejectionThreshold = 0
  HighConfidenceThreshold        = 0
  NormalConfidenceThreshold      = 0
  LowConfidenceThreshold         = 0

Speech from grammar g hypothesized: noose, 0.2214646
Speech from grammar g hypothesized: news, 0.640804

Number of Alternates from Grammar g: 1
news, 0.9208503

Speech recognized: news, 0.9208503
Number of Alternates from Recognizer: 1
news, 0.9208503

I also tried implementing this with a separate phrase for each word (instead of one phrase with three choices), and even with a separate grammar for each word/phrase. The results are basically the same: only one "alternate".

Answer 1

I believe this is another place where SAPI lets you ask for things that the SR engine doesn't really support.

Both Microsoft.Speech.Recognition and System.Speech.Recognition use the underlying SAPI interfaces to do their work; the only difference is which SR engine gets used. (Microsoft.Speech.Recognition uses the Server engine; System.Speech.Recognition uses the Desktop engine.)

Alternates are primarily designed for dictation, not context-free grammars. You can always get one alternate for a CFG, but the alternate generation code looks like it won't expand the alternates for CFGs.

Unfortunately, the Microsoft.Speech.Recognition engine doesn't support dictation. (It does, however, work with much lower quality audio, and it doesn't need training.)