Get started
Log in
Yandex SpeechKit

Streaming mode for short audio recognition

Data streaming mode allows you to simultaneously send audio for recognition and get recognition results over the same connection.

Unlike other recognition methods, you can get intermediate results while speech is in progress. After a pause, the service returns final results and starts recognizing the next utterance.

For example, as soon as the user starts talking to Yandex.Station, the speaker begins transmitting the speech to the server for recognition. The server processes the data and returns the intermediate and final results of each utterance recognition. The intermediate results are used for showing the user the progress of speech recognition. Once the final results are available, Yandex.Station performs the requested action, such as playing a movie.

To use the service, create an app that will perform speech recognition in data streaming mode, i.e., send audio fragments and process responses with recognition results.

Important

Streaming mode is designed for real-time audio recognition. If you need to send a recorded audio file, use a different method.

Using the service

Creating a client app

For speech recognition, the app should first send a message with recognition settings and then send messages with audio fragments.

While the audio fragments are sent, the service simultaneously returns recognized text fragments for processing (such as outputting them to the console).

To enable the app to access the service, you need to generate the client interface code for the programming language you use. Generate this code from the stt_service.proto file hosted in the Yandex.Cloud API repository.

See examples of client apps below. In addition, see the gRPC documentation for detailed instructions on how to generate interfaces and deploy client apps for various programming languages.

Authorization in the service

In each request, the application must transmit the ID of folder that you have been granted the editor role or higher for. For more information, see Access management.

The application must also be authenticated for each request, such as with an IAM token. Learn more about service authentication.

Recognition result

During the recognition process, the speech is segmented into phrases (known as utterances). An utterance is a fragment of speech consisting of one or more words, followed by a period of silence. An utterance may contain multiple sentences if there is no pause between them.

A speech recognition result provides alternative versions of a single utterance. The response from the service may contain multiple utterances.

Final results of speech recognition are formed when the speech recognition system detects the end of an utterance. You can send multiple messages with an audio fragment and it will be a single utterance.

Intermediate results of speech recognition are formed during utterance recognition. Getting intermediate results allows you to quickly respond to the recognized speech without waiting for the end of the utterance to get the final result.

You can configure the service to return intermediate recognition results. In the message with recognition settings, specify config.specification.partial_results=true. In the response, final=false indicates the intermediate results and final=true indicates the final results.

Limitations of a speech recognition session

After receiving the message with the recognition settings, the service starts a recognition session. The following limitations apply to each session:

  • You can't send audio fragments too often or too rarely. The time between messages to the service should be approximately the same as the duration of the audio fragments you send, but no more than 5 seconds.

    For example, send 400 ms of audio for recognition every 400 ms.

  • Maximum duration of transmitted audio for the entire session: 5 minutes.

  • Maximum size of transmitted audio data: 10 MB.

If messages aren't sent to the service within 5 seconds or the data duration or size limit is reached, the session is terminated. To continue speech recognition, reconnect and send a new message with the speech recognition settings.

Service API

The service is located at: stt.api.cloud.yandex.net:443

Message with recognition settings

Parameter Description
config object
Field with the recognition settings and folder ID.
config
.specification		 object
Recognition settings.
config
.specification
.languageCode		 string
The language to use for recognition.
Acceptable values:
  • ru-RU (by default) — Russian.
  • en-US — English.
  • tr-TR — Turkish.
config
.specification
.model		 string
The language model to be used for recognition.
The closer the model is matched, the better the recognition result. You can only specify one model per request.
Acceptable values depend on the selected language. Default value: general.
config
.specification
.profanityFilter		 boolean
The profanity filter.
Acceptable values:
  • true — Exclude profanity from recognition results.
  • false (by default) — Do not exclude profanity from recognition results.
config
.specification
.partialResults		 boolean
The intermediate results filter.
Acceptable values:
  • true — Return intermediate results (part of the recognized utterance). For intermediate results, final is set to false.
  • false (default) — Return only the final results (the entire recognized utterance).
config
.specification
.audioEncoding		 string
The format of the submitted audio.
Acceptable values:
config
.specification
.sampleRateHertz		 integer (int64)
The sampling frequency of the submitted audio.
Required if format is set to LINEAR16_PCM. Acceptable values:
  • 48000 (default) — Sampling rate of 48 kHz.
  • 16000 — Sampling rate of 16 kHz.
  • 8000 — Sampling rate of 8 kHz.
folderId string

ID of the folder that you have access to. Required for authorization with a user account (see the UserAccount resource). Don't specify this field if you make a request on behalf of a service account.

Maximum string length: 50 characters.

Audio message

Parameter Description
audio_content An audio fragment represented as an array of bytes. The audio must match the format specified in the message with recognition settings.

Response structure

If speech fragment recognition is successful, you will receive a message containing a list of recognition results (chunks[]). Each result contains the following fields:

  • alternatives[]: List of alternative recognition results. Each alternative contains the following fields:
    • text: Recognized text.
    • confidence: Recognition accuracy. Currently the service always returns 1, which is the same as 100%.
  • final: Set to true if the result is final, and to false if it is intermediate.

Error codes returned by the server

To see how gRPC statuses correspond to HTTP codes, see google.rpc.Code.

List of possible gRPC errors returned by the service:

Code Status Description
3 INVALID_ARGUMENT Incorrect request parameters specified. Details are provided in the details field.
16 UNAUTHENTICATED The operation requires authentication. Check the IAM token and the ID of the folder that you passed
13 INTERNAL Internal server error. This error means that the operation cannot be performed due to a server-side technical problem. For example, due to insufficient computing resources.

Examples

To try the examples in this section:

  1. Clone the Yandex.Cloud API repository:

    git clone https://github.com/yandex-cloud/cloudapi

  2. Get the ID of the folder your account has been granted access to.

  3. In the examples, an IAM token is used for authentication (other authentication methods). Get an IAM token:

  4. Download a sample audio file for recognition. The audio file is in LPCM format with a sampling rate of 8000.

Then proceed to creating a client app.

  1. Install the grpcio-tools package using the pip package manager:

    $ pip install grpcio-tools

  2. Go to the directory hosting the Yandex.Cloud API repository, create an output directory, and generate the client interface code there:

    $ cd cloudapi
$ mkdir output
$ python -m grpc_tools.protoc -I . -I third_party/googleapis --python_out=output --grpc_python_out=output google/api/http.proto google/api/annotations.proto yandex/api/operation.proto google/rpc/status.proto yandex/cloud/operation/operation.proto yandex/cloud/ai/stt/v2/stt_service.proto

    As a result, the stt_service_pb2.py and stt_service_pb2_grpc.py client interface files as well as dependency files will be created in the output directory.

  3. Create a file (for example, test.py) in the root of the output directory and add the following code to it:

    #coding=utf8
import argparse

import grpc

import yandex.cloud.ai.stt.v2.stt_service_pb2 as stt_service_pb2
import yandex.cloud.ai.stt.v2.stt_service_pb2_grpc as stt_service_pb2_grpc


CHUNK_SIZE = 4000

def gen(folder_id, audio_file_name):
    # Configure recognition settings.
    specification = stt_service_pb2.RecognitionSpec(
        language_code='ru-RU',
        profanity_filter=True,
        model='general',
        partial_results=True,
        audio_encoding='LINEAR16_PCM',
        sample_rate_hertz=8000
    )
    streaming_config = stt_service_pb2.RecognitionConfig(specification=specification, folder_id=folder_id)

    # Send a message with the recognition settings.
    yield stt_service_pb2.StreamingRecognitionRequest(config=streaming_config)

    # Read the audio file and send its contents in chunks.
    with open(audio_file_name, 'rb') as f:
        data = f.read(CHUNK_SIZE)
        while data != b'':
            yield stt_service_pb2.StreamingRecognitionRequest(audio_content=data)
            data = f.read(CHUNK_SIZE)


def run(folder_id, iam_token, audio_file_name):
    # Establish a connection with the server.
    cred = grpc.ssl_channel_credentials()
    channel = grpc.secure_channel('stt.api.cloud.yandex.net:443', cred)
    stub = stt_service_pb2_grpc.SttServiceStub(channel)

    # Send data for recognition.
    it = stub.StreamingRecognize(gen(folder_id, audio_file_name), metadata=(('authorization', 'Bearer %s' % iam_token),))

    # Process server responses and output the result to the console.
    try:
        for r in it:
            try:
                print('Start chunk: ')
                for alternative in r.chunks[0].alternatives:
                    print('alternative: ', alternative.text)
                print('Is final: ', r.chunks[0].final)
                print('')
            except LookupError:
                print('Not available chunks')
    except grpc._channel._Rendezvous as err:
        print('Error code %s, message: %s' % (err._state.code, err._state.details))


if __name__ == '__main__':
    parser = argparse.ArgumentParser()
    parser.add_argument('--token', required=True, help='IAM token')
    parser.add_argument('--folder_id', required=True, help='folder ID')
    parser.add_argument('--path', required=True, help='audio file path')
    args = parser.parse_args()

    run(args.folder_id, args.token, args.path)

  4. Execute the created file by passing arguments with the IAM token, folder ID, and path to the audio file to recognize:

    $ export FOLDER_ID=b1gvmob95yysaplct532
$ export IAM_TOKEN=CggaATEVAgA...
$ python test.py --token ${IAM_TOKEN} --folder_id ${FOLDER_ID} --path speech.pcm
Start chunk:
alternative: Hello
Is final: False

Start chunk:
alternative: Hello world
Is final: True

  1. Go to the directory with the Yandex.Cloud API repository, create a direct named src, and generate a dependency file named package.json in it:

    $ cd cloudapi
$ mkdir src
$ cd src
$ npm init

  2. Install the necessary packages using npm:

    $ npm install grpc @grpc/proto-loader google-proto-files --save

  3. Download a gRPC public key certificate from the official repository and save it in the root of the src directory.

  4. Create a file, for example index.js, in the root of the src directory and add the following code to it:

    const fs = require('fs');
const grpc = require('grpc');
const protoLoader = require('@grpc/proto-loader');
const CHUNK_SIZE = 4000;

// Get the folder ID and IAM token from the environment variables.
const folderId = process.env.FOLDER_ID;
const iamToken = process.env.IAM_TOKEN;

// Read the file specified in the arguments.
const audio = fs.readFileSync(process.argv[2]);

// Specify the recognition settings.
const request = {
    config: {
        specification: {
            languageCode: 'ru-RU',
            profanityFilter: true,
            model: 'general',
            partialResults: true,
            audioEncoding: 'LINEAR16_PCM',
            sampleRateHertz: '8000'
        },
        folderId: folderId
    }
};

// How often audio is sent in milliseconds.
// For LPCM format, the frequency can be calculated using the formula: CHUNK_SIZE * 1000 / ( 2 * sampleRateHertz).
const FREQUENCY = 250;

const serviceMetadata = new grpc.Metadata();
serviceMetadata.add('authorization', `Bearer ${iamToken}`);

const packageDefinition = protoLoader.loadSync('../yandex/cloud/ai/stt/v2/stt_service.proto', {
    includeDirs: ['node_modules/google-proto-files', '..']
});
const packageObject = grpc.loadPackageDefinition(packageDefinition);

// Establish a connection with the server.
const serviceConstructor = packageObject.yandex.cloud.ai.stt.v2.SttService;
const grpcCredentials = grpc.credentials.createSsl(fs.readFileSync('./roots.pem'));
const service = new serviceConstructor('stt.api.cloud.yandex.net:443', grpcCredentials);
const call = service['StreamingRecognize'](serviceMetadata);

// Send a message with the recognition settings.
call.write(request);

// Read the audio file and send its contents in chunks.
let i = 1;
const interval = setInterval(() => {
    if (i * CHUNK_SIZE <= audio.length) {
        const chunk = new Uint16Array(audio.slice((i - 1) * CHUNK_SIZE, i * CHUNK_SIZE));
        const chunkBuffer = Buffer.from(chunk);
        call.write({audioContent: chunkBuffer});
        i++;
    } else {
        call.end();
        clearInterval(interval);
    }
}, FREQUENCY);

// Process server responses and output the result to the console.
call.on('data', (response) => {
    console.log('Start chunk: ');
    response.chunks[0].alternatives.forEach((alternative) => {
        console.log('alternative: ', alternative.text)
    });
    console.log('Is final: ', Boolean(response.chunks[0].final));
    console.log('');
});

call.on('error', (response) => {
    // Handle errors.
    console.log(response);
});

  5. Set the FOLDER_ID and IAM_TOKEN variables used in the script and run the created file. Specify the path to the audio file in the arguments:

    $ export FOLDER_ID=b1gvmob95yysaplct532
$ export IAM_TOKEN=CggaATEVAgA...
$  node index.js speech.pcm

Start chunk:
alternative: Hello world
Is final:  true