Streaming API

Get started with the Streaming API

The Symbl.ai Streaming API uses the WebSocket protocol to process audio and provide conversation intelligence in real time. Implement the Streaming API in your web app to provide your users with active support from Symbl.ai, including captions, sentiment analysis, and insights like discussed topics and questions, among other features.

Symbl.ai SDKs

The code samples provided for the Streaming API use the Symbl.ai Javascript, Python, and Web SDKs. For details about the SDKs and additional code samples, see:

Authentication

The following code samples ask you to include your App ID and App Secret, the credentials used to generate Symbl.ai API access tokens, for testing purposes only. For a production environment, Symbl.ai recommends that you do not store your Symbl.ai API credentials in plain text or in your source code.

For more information about your App ID and App Secret, see the Get Started Guide.

Stream with the Web SDK

This section describes how to use the Web SDK to open a connection, process an audio stream, and close a connection.

Before you begin

Before you try the code samples, ensure your developer environment meets the following requirements.

Create a new directory or folder for your Web SDK project.
In the new directory, create a file named index.html.

Try the Streaming API

To try the code sample, paste the following code into your index.html file and replace the placeholder values.

This code sample authorizes you with the Streaming API and starts capturing audio from your mic. The sample also returns a conversation ID, which can be used to generate conversation intelligence.

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <title>Symbl Web SDK example</title>
  <script src="https://sdk.symbl.ai/js/beta/symbl-web-sdk/latest/symbl.min.js"></script>
  <script>
      // START changing values here.
      
      const appId = '<APP_ID>';
      const appSecret = '<APP_SECRET>';

      // STOP changing values here.

      const start = async () => {

      try {

          // Symbl recommends replacing the App ID and App Secret with an Access Token for authentication in production applications.
          // For more information about authentication see https://docs.symbl.ai/docs/developer-tools/authentication/.
          const symbl = new Symbl({
              appId: appId,
              appSecret: appSecret,
              // accessToken: '<your Access Token>' // for production use
          });
          
          // Open a Streaming API WebSocket Connection and start processing audio from your input device.
          const connection = await symbl.createAndStartNewConnection();

          // Retrieve the conversation ID for the conversation.
          connection.on("conversation_created", (conversationData) => {
            const conversationId = conversationData.data.conversationId;
            console.log(`${conversationId}`);
            document.querySelector("#conversationId").innerHTML = `${conversationId}`;
            document.querySelector("#startButton").setAttribute("disabled", "");
          });

          // Retrieve real-time transcription from the conversation.
          connection.on("speech_recognition", (speechData) => {
            const name = speechData.user ? speechData.user.name : "User";
            const transcript = speechData.punctuated.transcript;
            console.log(`${name}: `, transcript);
            document.querySelector("#speechRecognition").innerHTML = `${name}: ${transcript}`;
          });
          
          // This is a helper method for testing purposes.
          // It waits 60 seconds before continuing to the next API call.
          await Symbl.wait(60000);
          
          // Stops processing audio, but keeps the WebSocket connection open.
          await connection.stopProcessing();
          
          // Closes the WebSocket connection.
          connection.disconnect();

          document.querySelector("#startButton").removeAttribute("disabled");
      } catch(e) {
          // Handle errors here.
      }
    }
  </script>
</head>

<body>

  <button id="startButton" onclick="javascript: start()">Start Processing</button>

  <p><b>Conversation ID:</b> <span id="conversationId">None</span></p>

  <p id="speechRecognition">Click <b>Start Processing</b> and begin speaking to see transcription. If prompted, allow access to your microphone. The page records for 60 seconds and then closes the connection.<br> <br> If nothing happens, check your <a href="https://platform.symbl.ai/#/home">Symbl App ID and App Secret</a> in this HTML file on lines 16 and 17 respectively.</p>

</body>

</html>

Where:

<APP_ID> and <APP_SECRET> are your App ID and Secret from your Symbl.ai Home page.

To test the sample, open index.html in your web browser.

Stream with the JS SDK

This section describes how to use the JS SDK to open a connection, process an audio stream, and close a connection. The JS SDK is designed to be used with Node.js.

Before you begin

Before you try the code samples, ensure your developer environment meets the following requirements.

Install Node.js
- Install Node.js with a package manager.
- Download and install Node.js manually.
To capture audio input from your microphone, install SoX (Windows/Mac) or ALSA Tools (Linux).
- Download and install SoX. (Windows/Mac)
- To install ALSA Tools on Debian or Ubuntu:
```
sudo apt-get update && sudo apt-get install alsa-base alsa-utils
```
- To install ALSA Tools on Arch:
```
sudo pacman -Syu && sudo pacman -S alsa-tools
```
Create a new directory or folder and then initialize a Node.js project.
```
npm init -y
```
In the new directory, create a file named stream.js.
Create a file named subscribe.js.
Install the mic package with npm.
```
npm install mic
```
Install the Symbl.ai JS SDK with npm.
```
npm install @symblai/symbl-js
```

Try the Streaming API

To try the code sample, paste the following code into your stream.js file and replace the placeholder values.

This code sample authorizes you with the Streaming API, starts capturing audio from your mic, and provides conversation intelligence insights based on what you speak. The sample also returns a conversation ID, which can be used to generate conversation intelligence.

const { sdk } = require('@symblai/symbl-js');
const uuid = require('uuid').v4;

const mic = require('mic')
const sampleRateHertz = 16000

// START changing values here.

const appId = '<APP_ID>';
const appSecret = '<APP_SECRET>';

const config = {
  meetingTitle: '<TITLE>',
  sampleRateHertz: sampleRateHertz
}

const speaker = {
  name: '<NAME>',
  userId: '<EMAIL>'
}

// STOP changing values here.

const insightTypes = [
  'question',
  'action_item',
  'follow_up',
  'topic'
]

const micInstance = mic({
  rate: sampleRateHertz,
  channels: '1',
  debug: false,
  exitOnSilence: 6,
});

// Need unique ID and best to use uuid in production
// const connectionId = uuid()
const connectionId = Buffer.from(appId).toString('base64'); // for testing

(async () => {
  try {
    // Initialize the SDK
    await sdk.init({
      appId: appId,
      appSecret: appSecret,
      basePath: 'https://api.symbl.ai',
    })

    // Start Real-time Request (Uses Real-time WebSocket API behind the scenes)
    const connection = await sdk.startRealtimeRequest({
      id: connectionId,
      speaker: speaker,
      insightTypes: insightTypes,
      config: config,
      handlers: {
        /**
         * This will return live speech-to-text transcription of the call.
         * There are other handlers that can be seen in the full example.
         */
        onSpeechDetected: (data) => {
          if (data) {
            const {
              punctuated
            } = data
            console.log('Live: ', punctuated && punctuated.transcript)
          }
        }
      }
    });

    // Logs conversationId which is used to access the conversation afterwards
    console.log('Successfully connected. Conversation ID: ', connection.conversationId);

    const micInputStream = micInstance.getAudioStream()
    /** Raw audio stream */
    micInputStream.on('data', (data) => {
      // Push audio from Microphone to websocket connection
      connection.sendAudio(data)
    })

    micInputStream.on('error', function (err) {
      console.log('Error in Input Stream: ' + err)
    })

    micInputStream.on('startComplete', function () {
      console.log('Started listening to Microphone.')
    })

    micInputStream.on('silence', function () {
      console.log('Got SIGNAL silence')
    })

    micInstance.start()

    setTimeout(async () => {
      // Stop listening to microphone
      micInstance.stop()
      console.log('Stopped listening to Microphone.')
      try {
        // Stop connection
        await connection.stop()
        console.log('Connection Stopped.')
      } catch (e) {
        console.error('Error while stopping the connection.', e)
      }
    }, 120 * 1000) // Stop connection after 2 minute i.e. 120 secs
  } catch (err) {
    console.error('Error: ', err)
  }
})();

Where:

<APP_ID> and <APP_SECRET> are your App ID and Secret from your Symbl.ai Home page.
<TITLE> is the name of the meeting or conversation. If no name is provided, the Streaming API sets the name to the conversation ID.
<NAME> is the name of the person speaking.
<EMAIL> is the email address of the person speaking. If provided, when the audio stream ends, the Streaming API sends an email with conversation intelligence to the given address.

To run the code sample:

On the command line, go to your Node.js project directory.
Run the code sample.
```
node stream.js
```
The first time you run the code sample, you might be prompted to allow access to your microphone. To try the sample, permit access to your device's mic.

Try subscribing to an audio stream

To open a read-only connection to an existing audio stream without initializing a mic or audio device, use the Subscribe feature of the Streaming API. To subscribe to an existing audio stream, you pass your code the connectionId for the target stream.

To try the code sample, paste the following code into your subscribe.js file and replace the placeholder values.

This code sample authorizes you with the Streaming API and connects to a running instance of the previous code sample.

const { sdk } = require('@symblai/symbl-js');

// START changing values here.

const appId = '<APP_ID>';
const appSecret = '<APP_SECRET>';

// STOP changing values here.

// Subscribe to connection using connectionId that was defined as `connectionId` in previous example.
// We'll use the same constant Base64 string as before
const connectionId = Buffer.from(appId).toString('base64'); // for testing

(async () => {
  try {
    // Initialize the SDK
    await sdk.init({
      appId: appId,
      appSecret: appSecret,
      basePath: 'https://api.symbl.ai',
    })

    sdk.subscribeToStream(connectionId, (data) => {
        const { type } = data;
        if (type === 'message_response') {

            const { messages } = data;

            // You get any messages here
            messages.forEach(message => {
              console.log(`Message: ${message.payload.content}`)
            });

        } else if (type === 'insight_response') {

            const { insights } = data;

            // You get any insights here
            insights.forEach(insight => {
                console.log(`Insight: ${insight.type} - ${insight.text}`);
            });

        } else if (type === 'topic_response') {
            const { topics } = data;
            
            // You get any topic phrases here
            topics.forEach(topic => {
                console.log(`Topic detected: ${topic.phrases}`)
            });

        } else if (type === 'message' && data.message.hasOwnProperty('punctuated')) {

            const { transcript } = data.message.punctuated;

            // Live punctuated full transcript as opposed to broken into messages
            console.log(`Live transcript: ${transcript}`)
        }

        // The raw data response
        console.log(`Response type: ${data.type}. Object: `, data);

    });
  } catch (err) {
    console.error('Error: ', err)
  }
})();

Where:

<APP_ID> and <APP_SECRET> are your App ID and Secret from your Symbl.ai Home page.

To run the code sample:

If it is not already running, run the previous code sample.
On a new instance of the command line, go to your Node.js project directory.
Run the code sample.

node subscribe.js

Stream with the Python SDK

This section describes how to use the Python SDK to open a connection, process an audio stream, and close a connection.

Before you begin

Before you try the code samples, ensure your developer environment meets the following requirements.

Download and install Python.
Create a new directory or folder for your Python project.
In the new directory, create a file named stream.py.
On the command line, use pip to install the Python SDK.
```
pip3 install --upgrade symbl
```

Try the Streaming API

To try the code sample, paste the following code into your stream.py file and replace the placeholder values.

import symbl

# START changing values here.

app_id = "<APP_ID>"
app_secret = "<APP_SECRET>"

config = {
  meetingTitle: "<TITLE>"
}

speaker = {
  "name": "<NAME>",
  "email": "<EMAIL>"
}

# STOP changing values here.

events = {
    "message_response": lambda response: print(
        "Final Messages -> ",
        [message["payload"]["content"] for message in response["messages"]],
    ),
    "message": lambda response: print(
        "live transcription : {}".format(
            response["message"]["punctuated"]["transcript"]
        )
    )
    if "punctuated" in response["message"]
    else print(response),
    "insight_response": lambda response: [
        print(
            "Insights Item of type {} detected -> {}".format(
                insight["type"], insight["payload"]["content"]
            )
        )
        for insight in response["insights"]
    ],
    "topic_response": lambda response: [
        print(
            "Topic detected -> {} with root words, {}".format(
                topic["phrases"], topic["rootWords"]
            )
        )
        for topic in response["topics"]
    ],
}

connection_object = symbl.Streaming.start_connection(
    credentials={"app_id": app_id, "app_secret": app_secret},
    insight_types=["question", "action_item", "follow_up", "topic"],
    speaker=speaker,
    config=config
)

connection_object.subscribe(events)
connection_object.send_audio_from_mic()

# connection_object.stop() # ends the connection.

Where:

<APP_ID> and <APP_SECRET> are your App ID and Secret from your Symbl.ai Home page.
<TITLE> is the name of the meeting or conversation. If no name is provided, the Streaming API sets the name to the conversation ID.
<NAME> is the name of the person speaking.
<EMAIL> is the email address of the person speaking. If provided, when the audio stream ends, the Streaming API sends an email with conversation intelligence to the given address.

To run the code sample:

On the command line, go to your Python project directory.
Run the code sample.
```
python3 stream.py
```
The first time you run the code sample, you might be prompted to allow access to your microphone. To try the sample, permit access to your device's mic.