Streaming mode for short audio recognition
Data streaming mode allows you to simultaneously send audio for recognition and get recognition results over the same connection.
Unlike other recognition methods, you can get intermediate results while speech is in progress. After a pause, the service returns final results and starts recognizing the next utterance.
For example, as soon as the user starts talking to Yandex.Station, the speaker begins transmitting the speech to the server for recognition. The server processes the data and returns the intermediate and final results of each utterance recognition. The intermediate results are used for showing the user the progress of speech recognition. Once the final results are available, Yandex.Station performs the requested action, such as playing a movie.
To use the service, create an app that will perform speech recognition in data streaming mode, i.e., send audio fragments and process responses with recognition results.
Important
Streaming mode is designed for real-time audio recognition. If you need to send a recorded audio file, use a different method.
Using the service
Creating a client app
For speech recognition, the app should first send a message with recognition settings and then send messages with audio fragments.
While the audio fragments are sent, the service simultaneously returns recognized text fragments for processing (such as outputting them to the console).
To enable the app to access the service, you need to generate the client interface code for the programming language you use. Generate this code from the stt_service.proto file hosted in the Yandex.Cloud API repository.
See examples of client apps below. In addition, see the gRPC documentation for detailed instructions on how to generate interfaces and deploy client apps for various programming languages.
Authorization in the service
In each request, the application must transmit the ID of folder that you have been granted the editor
role or higher for. For more information, see Access management.
The application must also be authenticated for each request, such as with an IAM token. Learn more about service authentication.
Recognition result
During the recognition process, the speech is segmented into phrases (known as utterances). An utterance is a fragment of speech consisting of one or more words, followed by a period of silence. An utterance may contain multiple sentences if there is no pause between them.
A speech recognition result provides alternative versions of a single utterance. The response from the service may contain multiple utterances.
Final results of speech recognition are formed when the speech recognition system detects the end of an utterance. You can send multiple messages with an audio fragment and it will be a single utterance.
Intermediate results of speech recognition are formed during utterance recognition. Getting intermediate results allows you to quickly respond to the recognized speech without waiting for the end of the utterance to get the final result.
You can configure the service to return intermediate recognition results. In the message with recognition settings, specify config.specification.partial_results=true
. In the response, final=false
indicates the intermediate results and final=true
indicates the final results.
Limitations of a speech recognition session
After receiving the message with the recognition settings, the service starts a recognition session. The following limitations apply to each session:
-
You can't send audio fragments too often or too rarely. The time between messages to the service should be approximately the same as the duration of the audio fragments you send, but no more than 5 seconds.
For example, send 400 ms of audio for recognition every 400 ms.
-
Maximum duration of transmitted audio for the entire session: 5 minutes.
-
Maximum size of transmitted audio data: 10 MB.
If messages aren't sent to the service within 5 seconds or the data duration or size limit is reached, the session is terminated. To continue speech recognition, reconnect and send a new message with the speech recognition settings.
Service API
The service is located at: stt.api.cloud.yandex.net:443
Message with recognition settings
Parameter | Description |
---|---|
config | object Field with the recognition settings and folder ID. |
config .specification |
object Recognition settings. |
config .specification .languageCode |
string The language to use for recognition. Acceptable values:
|
config .specification .model |
string The language model to be used for recognition. The closer the model is matched, the better the recognition result. You can only specify one model per request. Acceptable values depend on the selected language. Default value: general . |
config .specification .profanityFilter |
boolean The profanity filter. Acceptable values:
|
config .specification .partialResults |
boolean The intermediate results filter. Acceptable values:
|
config .specification .audioEncoding |
string The format of the submitted audio. Acceptable values:
|
config .specification .sampleRateHertz |
integer (int64) The sampling frequency of the submitted audio. Required if format is set to LINEAR16_PCM . Acceptable values:
|
folderId | string ID of the folder that you have access to. Required for authorization with a user account (see the UserAccount resource). Don't specify this field if you make a request on behalf of a service account. Maximum string length: 50 characters. |
Audio message
Parameter | Description |
---|---|
audio_content |
An audio fragment represented as an array of bytes. The audio must match the format specified in the message with recognition settings. |
Response structure
If speech fragment recognition is successful, you will receive a message containing a list of recognition results (chunks[]
). Each result contains the following fields:
alternatives[]
: List of alternative recognition results. Each alternative contains the following fields:text
: Recognized text.confidence
: Recognition accuracy. Currently the service always returns1
, which is the same as 100%.
final
: Set totrue
if the result is final, and tofalse
if it is intermediate.
Error codes returned by the server
To see how gRPC statuses correspond to HTTP codes, see google.rpc.Code.
List of possible gRPC errors returned by the service:
Code | Status | Description |
---|---|---|
3 | INVALID_ARGUMENT |
Incorrect request parameters specified. Details are provided in the details field. |
16 | UNAUTHENTICATED |
The operation requires authentication. Check the IAM token and the ID of the folder that you passed |
13 | INTERNAL |
Internal server error. This error means that the operation cannot be performed due to a server-side technical problem. For example, due to insufficient computing resources. |
Examples
To try the examples in this section:
-
Clone the Yandex.Cloud API repository:
git clone https://github.com/yandex-cloud/cloudapi
-
Get the ID of the folder your account has been granted access to.
-
In the examples, an IAM token is used for authentication (other authentication methods). Get an IAM token:
- Instructions for a Yandex account.
- Instructions for a service account.
-
Download a sample audio file for recognition. The audio file is in LPCM format with a sampling rate of 8000.
Then proceed to creating a client app.
-
Install the grpcio-tools package using the pip package manager:
$ pip install grpcio-tools
-
Go to the directory hosting the Yandex.Cloud API repository, create an
output
directory, and generate the client interface code there:$ cd cloudapi $ mkdir output $ python -m grpc_tools.protoc -I . -I third_party/googleapis --python_out=output --grpc_python_out=output google/api/http.proto google/api/annotations.proto yandex/api/operation.proto google/rpc/status.proto yandex/cloud/operation/operation.proto yandex/cloud/ai/stt/v2/stt_service.proto
As a result, the
stt_service_pb2.py
andstt_service_pb2_grpc.py
client interface files as well as dependency files will be created in theoutput
directory. -
Create a file (for example,
test.py
) in the root of theoutput
directory and add the following code to it:#coding=utf8 import argparse import grpc import yandex.cloud.ai.stt.v2.stt_service_pb2 as stt_service_pb2 import yandex.cloud.ai.stt.v2.stt_service_pb2_grpc as stt_service_pb2_grpc CHUNK_SIZE = 4000 def gen(folder_id, audio_file_name): # Configure recognition settings. specification = stt_service_pb2.RecognitionSpec( language_code='ru-RU', profanity_filter=True, model='general', partial_results=True, audio_encoding='LINEAR16_PCM', sample_rate_hertz=8000 ) streaming_config = stt_service_pb2.RecognitionConfig(specification=specification, folder_id=folder_id) # Send a message with the recognition settings. yield stt_service_pb2.StreamingRecognitionRequest(config=streaming_config) # Read the audio file and send its contents in chunks. with open(audio_file_name, 'rb') as f: data = f.read(CHUNK_SIZE) while data != b'': yield stt_service_pb2.StreamingRecognitionRequest(audio_content=data) data = f.read(CHUNK_SIZE) def run(folder_id, iam_token, audio_file_name): # Establish a connection with the server. cred = grpc.ssl_channel_credentials() channel = grpc.secure_channel('stt.api.cloud.yandex.net:443', cred) stub = stt_service_pb2_grpc.SttServiceStub(channel) # Send data for recognition. it = stub.StreamingRecognize(gen(folder_id, audio_file_name), metadata=(('authorization', 'Bearer %s' % iam_token),)) # Process server responses and output the result to the console. try: for r in it: try: print('Start chunk: ') for alternative in r.chunks[0].alternatives: print('alternative: ', alternative.text) print('Is final: ', r.chunks[0].final) print('') except LookupError: print('Not available chunks') except grpc._channel._Rendezvous as err: print('Error code %s, message: %s' % (err._state.code, err._state.details)) if __name__ == '__main__': parser = argparse.ArgumentParser() parser.add_argument('--token', required=True, help='IAM token') parser.add_argument('--folder_id', required=True, help='folder ID') parser.add_argument('--path', required=True, help='audio file path') args = parser.parse_args() run(args.folder_id, args.token, args.path)
-
Execute the created file by passing arguments with the IAM token, folder ID, and path to the audio file to recognize:
$ export FOLDER_ID=b1gvmob95yysaplct532 $ export IAM_TOKEN=CggaATEVAgA... $ python test.py --token ${IAM_TOKEN} --folder_id ${FOLDER_ID} --path speech.pcm Start chunk: alternative: Hello Is final: False Start chunk: alternative: Hello world Is final: True
-
Go to the directory with the Yandex.Cloud API repository, create a direct named
src
, and generate a dependency file namedpackage.json
in it:$ cd cloudapi $ mkdir src $ cd src $ npm init
-
Install the necessary packages using npm:
$ npm install grpc @grpc/proto-loader google-proto-files --save
-
Download a gRPC public key certificate from the official repository and save it in the root of the
src
directory. -
Create a file, for example
index.js
, in the root of thesrc
directory and add the following code to it:const fs = require('fs'); const grpc = require('grpc'); const protoLoader = require('@grpc/proto-loader'); const CHUNK_SIZE = 4000; // Get the folder ID and IAM token from the environment variables. const folderId = process.env.FOLDER_ID; const iamToken = process.env.IAM_TOKEN; // Read the file specified in the arguments. const audio = fs.readFileSync(process.argv[2]); // Specify the recognition settings. const request = { config: { specification: { languageCode: 'ru-RU', profanityFilter: true, model: 'general', partialResults: true, audioEncoding: 'LINEAR16_PCM', sampleRateHertz: '8000' }, folderId: folderId } }; // How often audio is sent in milliseconds. // For LPCM format, the frequency can be calculated using the formula: CHUNK_SIZE * 1000 / ( 2 * sampleRateHertz). const FREQUENCY = 250; const serviceMetadata = new grpc.Metadata(); serviceMetadata.add('authorization', `Bearer ${iamToken}`); const packageDefinition = protoLoader.loadSync('../yandex/cloud/ai/stt/v2/stt_service.proto', { includeDirs: ['node_modules/google-proto-files', '..'] }); const packageObject = grpc.loadPackageDefinition(packageDefinition); // Establish a connection with the server. const serviceConstructor = packageObject.yandex.cloud.ai.stt.v2.SttService; const grpcCredentials = grpc.credentials.createSsl(fs.readFileSync('./roots.pem')); const service = new serviceConstructor('stt.api.cloud.yandex.net:443', grpcCredentials); const call = service['StreamingRecognize'](serviceMetadata); // Send a message with the recognition settings. call.write(request); // Read the audio file and send its contents in chunks. let i = 1; const interval = setInterval(() => { if (i * CHUNK_SIZE <= audio.length) { const chunk = new Uint16Array(audio.slice((i - 1) * CHUNK_SIZE, i * CHUNK_SIZE)); const chunkBuffer = Buffer.from(chunk); call.write({audioContent: chunkBuffer}); i++; } else { call.end(); clearInterval(interval); } }, FREQUENCY); // Process server responses and output the result to the console. call.on('data', (response) => { console.log('Start chunk: '); response.chunks[0].alternatives.forEach((alternative) => { console.log('alternative: ', alternative.text) }); console.log('Is final: ', Boolean(response.chunks[0].final)); console.log(''); }); call.on('error', (response) => { // Handle errors. console.log(response); });
-
Set the
FOLDER_ID
andIAM_TOKEN
variables used in the script and run the created file. Specify the path to the audio file in the arguments:$ export FOLDER_ID=b1gvmob95yysaplct532 $ export IAM_TOKEN=CggaATEVAgA... $ node index.js speech.pcm Start chunk: alternative: Hello world Is final: true