Skip to main content

Receive Live Captioning

This example goes over how you can use the Symbl Streaming API to do live captioning. This example uses both the Symbl's Javascript SDK, which is meant to be run using Node.js, and native Javascript which can be run in the browser:

Connect#

The first thing we do is connect to the Web Socket using the SDK. If you're using the SDK you can use the onMessageResponse and onSpeechDetected handlers after the connection is established, otherwise you'll have to parse the response in the onmessage callback for the WebSocket.

const {sdk} = require('symbl-node');
const uuid = require('uuid').v4;
(async () => {
try {
// Initialize the SDK
await sdk.init({
appId: appId,
appSecret: appSecret,
basePath: 'https://api.symbl.ai',
})
// Need unique Id
const id = uuid()
// Start Realtime Request (Uses Real-time WebSocket API behind the scenes)
const connection = await sdk.startRealtimeRequest({
id,
insightTypes: ['action_item', 'question'],
config: {
meetingTitle: 'My Test Meeting',
confidenceThreshold: 0.7,
timezoneOffset: 480, // Offset in minutes from UTC
languageCode: 'en-US',
sampleRateHertz: 44100,
},
speaker: {
// Optional, if not specified, will simply not send an email in the end.
userId: 'emailAddress', // Update with valid email
name: 'My name'
},
handlers: {
/**
* This will return live speech-to-text transcription of the call.
*/
onSpeechDetected: (data) => {
console.log(JSON.stringify(data))
if (data) {
const {punctuated} = data
console.log('Live: ', punctuated && punctuated.transcript)
}
},
/**
* When processed messages are available, this callback will be called.
*/
onMessageResponse: (data) => {
console.log('onMessageResponse', JSON.stringify(data, null, 2))
},
}
});
} catch (e) {
console.error(e);
}
})();

Testing#

Create a javascript file named app.js and copy this code into the file. Fill in the placeholder values with the proper values. Use npm to install the required libraries: npm install symbl-node uuid. Now in the terminal run

$ node app.js

If successful you should receive a response in the console.

Handlers Reference#

  • handlers: This object has the callback functions for different events

    • onSpeechDetected: To retrieve the real-time transcription results as soon as they are detected. You can use this callback to render live transcription which is specific to the speaker of this audio stream.

    onSpeechDetected JSON Response Example#

    {
    "type": "recognition_result",
    "isFinal": true,
    "payload": {
    "raw": {
    "alternatives": [{
    "words": [{
    "word": "Hello",
    "startTime": {
    "seconds": "3",
    "nanos": "800000000"
    },
    "endTime": {
    "seconds": "4",
    "nanos": "200000000"
    }
    }, {
    "word": "world.",
    "startTime": {
    "seconds": "4",
    "nanos": "200000000"
    },
    "endTime": {
    "seconds": "4",
    "nanos": "800000000"
    }
    }],
    "transcript": "Hello world.",
    "confidence": 0.9128385782241821
    }]
    }
    },
    "punctuated": {
    "transcript": "Hello world."
    },
    "user": {
    "userId": "emailAddress",
    "name": "John Doe",
    "id": "23681108-355b-4fc3-9d94-ed47dd39fa56"
    }
    }
    • onMessageResponse: This callback function contains the "finalized" transcription data for this speaker and if used with multiple streams with other speakers this callback would also provide their messages. The "finalized" messages mean that the automatic speech recognition has finalized the state of this part of transcription and has declared it "final". Therefore, this transcription will be more accurate than onSpeechDetected.

    onMessageResponse JSON Response Example#

    [{
    "from": {
    "id": "0a7a36b1-047d-4d8c-8958-910317ed9edc",
    "name": "John Doe",
    "userId": "emailAddress"
    },
    "payload": {
    "content": "Hello world.",
    "contentType": "text/plain"
    },
    "id": "59c224c2-54c5-4762-9582-961bf250b478",
    "channel": {
    "id": "realtime-api"
    },
    "metadata": {
    "disablePunctuation": true,
    "timezoneOffset": 480,
    "originalContent": "Hello world.",
    "words": "[{\"word\":\"Hello\",\"startTime\":\"2021-02-04T20:34:59.029Z\",\"endTime\":\"2021-02-04T20:34:59.429Z\"},{\"word\":\"world.\",\"startTime\":\"2021-02-04T20:34:59.429Z\",\"endTime\":\"2021-02-04T20:35:00.029Z\"}]",
    "originalMessageId": "59c224c2-54c5-4762-9582-961bf250b478"
    },
    "dismissed": false,
    "duration": {
    "startTime": "2021-02-04T20:34:59.029Z",
    "endTime": "2021-02-04T20:35:00.029Z"
    }
    }]