Speech Recognition in Xamarin.iOS - Xamarin (2023)

This article introduces the new Speech API and shows how to implement it in a Xamarin.iOS app to support continuous speech recognition and transcribe speech (from live or recorded audio streams) to text.

New in iOS 10, Apple released the Speech Recognition API, which allows an iOS app to support continuous speech recognition and transcribe speech (from live or recorded audio streams) into text.

According to Apple, the Speech Recognition API has the following features and benefits:

• Very accurate
• state of the art
• Easy to use
• Fast
• Supports multiple languages
• Respects users' privacy

How speech recognition works

Speech recognition is implemented in an iOS app by capturing either live or pre-recorded audio (in any of the spoken languages ​​supported by the API) and passing it to a speech recognizer, which returns a plain-text transcription of the spoken words.

Keyboard dictation

When most users think of speech recognition on an iOS device, they think of the built-in Siri voice assistant that was released with the iPhone 4S along with keyboard dictation in iOS 5.

Keyboard dictation is supported by any interface element that supports TextKit (egUITextFieldorUITextArea) and is activated by the user clicking the dictation button (just to the left of the spacebar) on the iOS virtual keyboard.

Apple has released the following keyboard dictation statistics (collected since 2011):

• Keyboard dictation has been widely used since its release in iOS 5.
• Around 65,000 apps use it every day.
• About a third of all iOS dictations are performed in a third-party app.

Keyboard Dictation is extremely easy to use as it requires no effort on the part of the developer other than using a TextKit UI element in the app's UI design. Keyboard dictation also has the benefit of not requiring any special permission requests from the app before it can be used.

Apps using the new Speech Recognition APIs must grant special permissions from the user, since Speech Recognition requires the transmission and temporary storage of data on Apple's servers. Please see oursSecurity and privacy improvementsdocumentation for details.

Although keyboard dictation is easy to implement, it comes with some limitations and disadvantages:

• It requires the use of a text entry field and the display of a keyboard.
• It only works with live audio input and the app has no control over the audio recording process.
• It offers no control over the language used to interpret the user's speech.
• The app has no way of knowing if the dictation key is even available to the user.
• The app cannot customize the audio recording process.
• It provides a very flat set of results that lack information such as timing and confidence.

Speech Recognition API

New in iOS 10, Apple released the Speech Recognition API, which provides a more powerful way for an iOS app to implement speech recognition. This API is the same one Apple uses to power both Siri and keyboard dictation, and it's capable of providing fast transcription with cutting-edge accuracy.

The results provided by the Speech Recognition API are transparently tailored to each user without the app having to collect or access private user data.

Speech Recognition API returns results to the calling app in near real-time while the user is speaking, providing more information about the translation results than just text. These include:

• Multiple interpretations of what the user said.
• Confidence levels for each translation.
• timing information.

As mentioned above, audio for translation can be provided from either a live feed or from a pre-recorded source and in any of the 50+ languages ​​and dialects supported by iOS 10.

The Speech Recognition API can be used on any iOS device running iOS 10 and in most cases requires an active internet connection as most of the translations take place on Apple's servers. However, some newer iOS devices support continuous on-device translation of certain languages.

Apple has integrated an availability API to determine if a specific language is currently available for translation. The app should use this API instead of directly testing for internet connection itself.

As noted in the Keyboard Dictation section above, speech recognition requires the transmission and temporary storage of data on Apple's servers over the Internet, and therefore the appmustRequest the user's permission to perform the detection by including theNSSpeechRecognitionUsageDescriptionkey in hisInfo.plistFile and calling theSFSpeechRecognizer.RequestAuthorizationMethod.

Based on the audio source used for speech recognition, other changes to the appInfo.plistFile may be required. Please see oursSecurity and privacy improvementsdocumentation for details.

Introducing speech recognition in an app

There are four key steps the developer needs to take to adopt speech recognition in an iOS app:

• Provide a usage description in the appInfo.plistfile with theNSSpeechRecognitionUsageDescriptionButton. For example, a camera app might contain the following description:"This way you can take a picture just by saying the word 'cheese'."
• Request authorization by callingSFSpeechRecognizer.RequestAuthorizationMethod to present an explanation (provided in theNSSpeechRecognitionUsageDescriptionKey above), why the app wants the user to access speech recognition, in a dialog box, and allow them to accept or decline.
• Create a speech recognition request:
  • For pre-recorded audio on disk, use theSFSpeechURLRecognitionRequestClass.
  • For live audio (or audio from memory), use theSFSPeechAudioBufferRecognitionRequestClass.
• Pass the speech recognition request to a speech recognizer (SFSpeechRecognizer) to start detection. The app can optionally capture the returned dataSFSpeechRecognitionTaskto monitor and track detection results.

These steps are detailed below.

Providing a Description of Use

To provide what is requiredNSSpeechRecognitionUsageDescriptionkey onInfo.plistfile, proceed as follows:

Request Authorization

To request the necessary user authorization that allows the app to access speech recognition, edit the main view controller class and add the following code:

using System;using UIKit;using Speech;namespace MonkeyTalk{ public subclass ViewController : UIViewController { protected ViewController (IntPtr handle) : base (handle) { // Note: this .ctor file should not contain any initialization logic. } public override void ViewDidLoad () { base.ViewDidLoad (); // Request user authorization SFSpeechRecognizer.RequestAuthorization ((SFSpeechRecognizerAuthorizationStatus status) => { // Take action based on status change (status) { case SFSpeechRecognizerAuthorizationStatus.Authorized: // User authorized speech recognition ... break; case SFSpeechRecognizerAuthorizationStatus.Denied: // User rejected speech recognition... break; case SFSpeechRecognizerAuthorizationStatus.NotDetermined: // Waiting for approval... break; case SFSpeechRecognizerAuthorizationStatus.Restricted: // The device is not authorized... break; } }); } }}

TheRequest Authorizationmethod ofSFSpeechRecognizerThe class requests permission from the user to access speech recognition using the reason given by the developer in theNSSpeechRecognitionUsageDescriptionkey theInfo.plistFile.

ASFSpeechRecognizerAuthorizationStatusResult is returned toRequest AuthorizationMethod callback routine that can be used to take action based on the user's permission.

Recognize recorded speech

If the app wants to recognize speech from a pre-recorded WAV or MP3 file, it can use the following code:

Using System;Using UIKit;Using Speech;Using Foundation;...public void RecognizeFile(NSUrl url){ // Access new recognizer var Recognizer = new SFSpeechRecognizer(); // Is the default language supported? if (recognizer == null) { // no, back to the caller return; } // Is detection available? if (!recognizer.Available) { // no, back to the caller return; } // Create recognition task and start recognition var request = new SFSpeechUrlRecognitionRequest (url); detectr.GetRecognitionTask (request, (SFSpeechRecognitionResult result, NSError err) => { // Was there an error? if (err != null) { // Handle error... } else { // Is this the final translation? if (result.Final) { Console.WriteLine("You said \"{0}\".", result.BestTranscription.FormattedString); } } });}

If you look at this code in detail, it first tries to create a speech recognizer (SFSpeechRecognizer). If the default language for speech recognition is not supported,Nullreturned and the function exited.

If speech recognition is available for the default language, the app will check if it is currently available for recognition usingAccessibleProperty. For example, detection may not be available if the device does not have an active internet connection.

ASFSpeechUrlRecognitionRequestarises from theNSUrlLocation of the recorded file on the iOS device and is passed to the speech recognition to process it with a callback routine.

When the callback is called when theNS erroris notNullAn error has occurred that needs to be handled. Because speech recognition is incremental, the callback routine can be called more than onceSFSpeechRecognitionResult.Finalproperty is tested to see if the translation is complete and the best version of the translation is written out (Best Transcription).

Detect Live Speech

If the app wants to recognize live speech, the process is very similar to recognizing recorded speech. For example:

using System;using UIKit;using Speech;using Foundation;using AVFoundation;...#region Private Variablesprivate AVAudioEngine AudioEngine = new AVAudioEngine ();private SFSpeechRecognizer SpeechRecognizer = new SFSpeechRecognizer ();private SFSpeechAudioBufferRecognitionRequest LiveSpeechRequest = new SFSpeechAudioBufferRecognitionRequest ();private SFSpeechRecognitionTask RecognitionTask;#endregion...public void StartRecording (){ // Establish audio session var node = AudioEngine.InputNode; var recordingFormat = node.GetBusOutputFormat(0); node.InstallTapOnBus(0, 1024, recordingFormat, (AVAudioPcmBuffer buffer, AVAudioTime when) => { // Append buffer to recognition request LiveSpeechRequest.Append (buffer); }); // Start recording AudioEngine.Prepare(); NSError error; AudioEngine.StartAndReturnError(Output Error); // Has recording started? if (error != null) { // handle error and return ... return; } // Start recognition RecognitionTask = SpeechRecognizer.GetRecognitionTask (LiveSpeechRequest, (SFSpeechRecognitionResult result, NSError err) => { // Was there an error? if (err != null) { // Handle error ... } else { // If (result.Final) { Console.WriteLine ("You said \"{0}\".", result.BestTranscription.FormattedString); } } });}public void StopRecording (){ AudioEngine .To stop (); LiveSpeechRequest.EndAudio();}public void CancelRecording(){ AudioEngine.Stop(); RecognitionTask.Cancel();}

Looking at this code in detail, several private variables are created to handle the detection process:

private AVAudioEngine AudioEngine = new AVAudioEngine ();private SFSpeechRecognizer SpeechRecognizer = new SFSpeechRecognizer ();private SFSpeechAudioBufferRecognitionRequest LiveSpeechRequest = new SFSpeechAudioBufferRecognitionRequest ();private SFSpeechRecognitionTask RecognitionTask;

It uses AV Foundation to record audio that is routed to aSFSpeechAudioBufferRecognitionRequestto process the application for recognition:

var node = AudioEngine.InputNode;var recordingFormat = node.GetBusOutputFormat (0);node.InstallTapOnBus (0, 1024, recordingFormat, (AVAudioPcmBuffer buffer, AVAudioTime when) => { // Append buffer to recognition request LiveSpeechRequest.Append (buffer) ; });

The app will try to start recording and all errors will be handled if the recording fails to start:

AudioEngine.Prepare();NSError error;AudioEngine.StartAndReturnError (out error); // Has recording started? if (error != null) { // Handle error and return... return;}

The discovery task is started and a handle is kept to the discovery task (SFSpeechRecognitionTask):

RecognitionTask = SpeechRecognizer.GetRecognitionTask (LiveSpeechRequest, (SFSpeechRecognitionResult result, NSError err) => { ...});

The callback is used in a manner similar to that for pre-recorded speech above.

If the user stops the recording, both the audio engine and the speech recognition request are informed:

AudioEngine.Stop ();LiveSpeechRequest.EndAudio ();

If the user cancels the recognition, the audio engine and the recognition task are informed:

AudioEngine.Stop ();RecognitionTask.Cancel ();

Calling is importantRecognitionTask.Cancelwhen the user cancels the translation to free up both memory and the device's processor.

Limits of Speech Recognition

Apple imposes the following limitations when working with speech recognition in an iOS app:

• Speech recognition is free for all apps, but its use is not unlimited:
  • Individual iOS devices have a limited number of detections that can be performed per day.
  • Apps are throttled globally on a request-per-day basis.
• The app must be prepared to deal with speech recognition network connection and usage rate limiting errors.
• Speech recognition can cause high costs on the user's iOS device due to both battery drain and high network traffic. Because of this, Apple imposes a strict audio duration limit of about a minute or less of speech.

When an app routinely hits its rate throttling limits, Apple asks the developer to get in touch with them.

Privacy and usability considerations

Apple has the following proposal to be transparent and respect user privacy when integrating speech recognition into an iOS app:

• If you record the user's speech, ensure that the app's UI clearly states the recording. For example, the app could play a "recording" sound and display a recording indicator.
• Do not use speech recognition for sensitive user information such as passwords, health records, or financial information.
• View the recognition resultsBeforeaffect them. This not only gives feedback on what the app is doing, but allows the user to handle detection errors as they are made.

Summary

This article introduced the new Speech API and showed how to implement it in a Xamarin.iOS app to support continuous speech recognition and transcribe speech (from live or recorded audio streams) to text.

• SpeakToMe (example)