Introduction

Base URL

https://api.symbl.ai/v1

Symbl is a comprehensive suite of APIs for analyzing natural human conversations - both for your team’s internal conversations and of course the conversations you are having with your customers. Built on our Contextual Conversation Intelligence (C2I) technology, the APIs enable you to rapidly incorporate human-level understanding that goes beyond simple natural language processing of voice and text conversations.

Get real-time analysis of free-flowing discussions like meetings, sales calls, support conversations, emails, chat, social conversations, transcripts, etc to automatically surface highly relevant summary topics, contextual insights, suggestive action items, follow-ups, decisions and questions - without the use of upfront training data, wake words or custom classifiers.

Getting Started

Postman

Here a quick Postman link for people who love to get their hands dirty.

Run in Postman

Ready to test? When you've built out your integration, review our testing tips to ensure that your integration works end to end

Explore Sample Apps

Use any of the sample integrations as a starting point to extend your own custom applications

If you're interested in what all you can do with Symbl, check out our sample applications on GitHub that demonstrate how Symbl can be used to connect to Twilio Media Streams, Salesforce Dashboard, Outlook Calendar and many more.

Symbl for Twilio Flex

Twilio Flex as an inbound adapter for streaming audio to through Symbl's Websocket API.

Symbl for Zoom

Our app that lets you invite Symbl to your zoom meeting by just pasting in the meeting invite.

Explore more

Browse our demo library and look at sample code on how to integrate voice intelligence into existing applications.

Authentication

If you don't already have your app id or app secret, log in to the platform to get your credentials.

To invoke any API call, you must have a valid Access Token generated using the valid application credentials.

To generate the token using the appId and appSecret, the HTTP POST request needs to be made with these details.

POST https://api.symbl.ai/oauth2/token:generate
{
  "type": "application",
  "appId": "your_appId",
  "appSecret": "your_appSecret"
}

curl -k -X POST "https://api.symbl.ai/oauth2/token:generate" \
     -H "accept: application/json" \
     -H "Content-Type: application/json" \
     -d "{ \"type\": \"application\", \"appId\": \"<appId>\", \"appSecret\": \"<appSecret>\"}"

 const request = require('request');

 const authOptions = {
   method: 'post',
   url: "https://api.symbl.ai/oauth2/token:generate",
   body: {
       type: "application",
       appId: "<appId>",
       appSecret: "<appSecret>"
   },
   json: true
 };

 request(authOptions, (err, res, body) => {
   if (err) {
     console.error('error posting json: ', err);
     throw err
   }

   console.log(JSON.stringify(body, null, 2));
 });

JavaScript code to generate the Access Token. The code should work with NodeJS 7+ and browsers. You will need to install request for this sample code.

$ npm i request

For a valid appId and appSecret combination, the success response will be returned like this.

 {
   "accessToken": "your_accessToken",
   "expiresIn": 3600
 }

accessToken - Token to be used for authorization in X-API-KEY header.
expiresIn - Duration in seconds after which the accessToken expires.

For any invalid appId and appSecret combination, HTTP 401 Unauthorized response code will be returned.

How Tos

We know the fastest deterrent to adopting new technology is friction. We’ve created a series of step by step ‘How tos’ to get you to value quickly and easily.

Get Live Transcription - Phone Call (Node.js) Telephony

Get the live transcription in your Node.js application by making a call to a valid phone number.

Making a phone call is also the quickest way to test Symbl’s Telephony API. It can make an outbound call to a phone number using a traditional public switched telephony network (PSTN), any SIP trunks, or SIP endpoints that can be accessed over the internet using a SIP URI.

Set up Symbl

To get started, you’ll need your account credentials and Node.js installed (> v8.x) on your machine.

We’ll use the Symbl module for Node.js in this guide. Make sure you have a Node project set up. If you don’t have one, you can set one up using npm init.

From the root directory of your project, run the following command to add symbl-node in your project dependencies.

$ npm i --save symbl-node

Retrieve your credentials

Your credentials include your appId and appSecret. You can find them on the home page of the platform. credentials page

Initialize SDK

const {sdk, SpeakerEvent} = require("symbl-node");

sdk.init({
    // Your appId and appSecret https://platform.symbl.ai
    appId: 'your_appId',
    appSecret: 'your_appSecret'
}).then(async () => {
    console.log('SDK initialized.');
    try {
      // You code goes here.
    } catch (e) {
        console.error(e);
    }
}).catch(err => console.error('Error in SDK initialization.', err));

In a new file called index.js in the project, add the following lines to import and initialize the SDK. Replace the appId and appSecret in the code with yours.

Make a phone call

The quickest way to test the Telephony API is to make a phone call to any valid phone number. The Telephony API only works with phone numbers in the U.S. and Canada.

tutorial phone integration

Your application uses the SDK to:
- make REST call to Telephony API with phone number details and then Symbl makes the call
- subscribe to live results over a WebSocket connection
Once the call is accepted on the phone, the Telephony API starts receiving audio from the call.
Live transcription is streamed back to your application using the subscribed WebSocket connection.

const connection = await sdk.startEndpoint({
    endpoint: {
      type: 'pstn', // when making a regular phone call
      // Replace this with a real phone number
      phoneNumber: '1XXXXXXXXXX' // include country code, example - 19998887777
    }
  });
const {connectionId} = connection;
console.log('Successfully connected. Connection Id: ', connectionId);

To make a phone call, call the startEndpoint with type set to ‘pstn’ and a valid U.S./Canadian phone number phoneNumber.

To get the live transcription, you need to subscribe to the connection to get the results. You need to call the subscribeToConnection method in the SDK and pass the connectionId and a callback method which will be called on for every new event including the (live) transcription.

// Subscribe to connection using connectionId.
sdk.subscribeToConnection(connectionId, (data) => {
  const {type} = data;
  if (type === 'transcript_response') {
      const {payload} = data;

      // You get live transcription here!!
      process.stdout.write('Live: ' + payload && payload.content + '\r');

  } else if (type === 'message_response') {
      const {messages} = data;

      // You get processed messages in the transcript here!!! Real-time but not live! :)
      messages.forEach(message => {
          process.stdout.write('Message: ' + message.payload.content + '\n');
      });
  } else if (type === 'insight_response') {
      const {insights} = data;
      // You get any insights here!!!
      insights.forEach(insight => {
          process.stdout.write(`Insight: ${insight.type} - ${insight.text} \n\n`);
      });
  }
});

When you use this API, you can use a callback function within the subscribeToConnection function. You pass an object and declare the type of response you want in the type field.

transcript_response type is used for low latency live transcription results
message_response type is used for processed transcription results in real-time. These are not the same as transcript_response which are low latency but typically generate a few seconds after the processing and contain messages split by sentence boundaries.
insight_response is used for any insights detected in real-time

End the Call

To end the call, you should make a stopEndpoint call. The following code stops the call after 60 seconds. Your business logic should determine when the call should end.

// Stop call after 60 seconds to automatically.
setTimeout(async () => {
  const connection = await sdk.stopEndpoint({ connectionId });
  console.log('Stopped the connection');
  console.log('Conversation ID:', connection.conversationId);
}, 60000); // Change the 60000 with higher value if you want this to continue for more time.

The stopEndpoint will return an updated connection object which will have the conversationId in the response. You can use conversationId to fetch the results even after the call using the Conversation API.

Code

You can find the complete code used in this guide here.

Test

To verify and check if the code is working:

Run your code:

$ node index.js

You should receive a phone call to the number you used in the startEndpoint call. Accept the call.
Start speaking in English (default language) and you should see the live transcription added to the console in real-time.
The call should automatically end after 60 seconds. If you end it sooner and don’t invoke stopEndpoint, you will not receive the conversationId. If you need to access the results generated in the call, you should invoke stopEndpoint even if it was ended without explicitly invoking stopEndpoint prior to this point.

Note

The example above invokes stopEndpoint after a fixed timeout of 60 seconds. This is for demonstrating the stop functionality and it is not the recommended method of implementation for your application. In a real implementation, you should invoke startEndpoint and stopEndpoint as needed by the business logic of your application, i.e when you would like Symbl to start processing and stop processing.

How to use Symbl REST telephony API to get live transcription

In this how to you'll learn to use Symbl REST telephony API to call on the phone or meeting URL and get live transcription using websockets

Get Started

Retrieve your credentials

Your credentials include your appId and appSecret. You can find them on the home page of the platform.

Appid

add credentials to .env file filling in APP_ID and APP_SECRET variables.

Authenticating

When using REST API, we would need to pass auth token in header. For that we've created component ProtectedPage. This component executes Symbl specific REST endpoint, to retrieve auth token and store it in context. Later on we can retrieve this token from the helper hook useAuth

This is how we would retrieve the token:

async function loginToSymbl() {
    const response = await fetch('https://api.symbl.ai/oauth2/token:generate', {
      method: 'POST',
      headers: {
        'Content-Type': 'application/json',
      },
      mode: 'cors',
      body: JSON.stringify({
        type: 'application',
        appId: process.env.APP_ID,
        appSecret: process.env.APP_SECRET,
      }),
    })
    const json = await response.json()
    console.log('Access Token is: ', json)
    setToken(json.accessToken)

Later on in any place in application we can use const { token } = useAuth() to get the token.

Calling phone or meeting using REST API

First of all what we need to do is to collect parameters from the user to send to Symbl Telephoni REST API, so Symbl will know how to initiate the call and how to process the conversation.

type - Symbl supports two meetings types. PSTN and SIP
phoneNumber - Naturally we need to have a phone number to call, but in case of SIP type we will need to call url. Something like sip:124@domain.com. For simplicity we will pass this data under phoneNumber parameter.
dtmf - This will be meeting code, which we need to pass in case meeting has such a code
summaryEmail - you can pass a list of emails which will get summary of the call email.
insightTypes - there are several insight types that we can pass to Symbl.

Collecting these parameters is basically a set of React controlled input fields. You can check component PhoneConfigurations.tsx if you are curious how it's implemented.

This component have two different behaviours. One is calling api/call route, that we've covered in (how-to-nextjs-node-sdk.md)[#how-to-access-symbl-node-sdk-through-nextjs-app-to-call-on-the-phone-or-meeting-url-and-get-live-transcription] guide. The other version of it is basically instead of calling api/call with user entered parameters, to call

https://api.symbl.ai/v1/endpoint:connect

This will look like this:

const phoneNumberOrUri =
  type === 'sip' ? { uri: _phoneNumber } : { phoneNumber: _phoneNumber }
const res = await fetch('https://api.symbl.ai/v1/endpoint:connect', {
  method: 'POST',
  headers: {
    'x-api-key': token as string,
    'Content-Type': 'application/json',
  },
  mode: 'cors',
  body: JSON.stringify({
    operation: 'start',
    endpoint: {
      type,
      ...phoneNumberOrUri,
      dtmf,
    },
    insightTypes,
    actions: [
      {
        invokeOn: 'stop',
        name: 'sendSummaryEmail',
        parameters: {
          emails: summaryEmails.split(','),
        },
      },
    ],
    data: {
      session: {
        name: 'Call from nextjs Phone(Client only)',
      },
    },
  }),
})

When we will execute fetch call to endpoint:connect endpoint, you will get a phone call either on your phone or sip url according to what you've chosen in UI.

What we also need to do is to actually get a response of api call and set our connectionId in the app. We will do that by calling connectionResponseHandler callback function that we pass to PhoneConfigurations.tsx component. There we will set our response as conversationData in our app.

Connecting via websockets to get live transcriptions

Now when we've got our phone call, we need to have a way to get realtime insights on the ongoing conversation. What we will do is monitor conversationData changes and if it changes, we will start our websocket.

useEffect(() => {
  let ws: any = null

  //Check if we have conversationData and if we haven't already started websocket connection
  if (conversationData && !ws) {
    ws = new WebSocket(
      `wss://api.symbl.ai/session/subscribe/${connectionId}?access_token=${token}`
    )
    ws.onmessage = (event: MessageEvent) => {
      const data = JSON.parse(event.data)
      if (data.type === 'message_response') {
        setMessages(data.messages)
      }
      if (data.type === 'transcript_response') {
        setLiveTranscript(data)
      }
      setRealTimeData(data)
    }
  }

  // cleanup method which will be called before next execution. in your case unmount.
  return () => {
    ws && ws.close
  }
}, [conversationData])

Here we will subscribe to Symbl websocket url wss://api.symbl.ai/session/subscribe/${connectionId}?access_token=${token} to get realtime insights on ongoing conversation.

We will get either transcript_response type or message_response type or insight_response type. For example message_response will look like this:

{
  "type": "message_response",
  "messages": [
    {
      "from": {},
      "payload": {
        "content": "Yeah, tell us sir.",
        "contentType": "text/plain"
      },
      "duration": {
        "startTime": "2020-10-22T15:32:14.500Z",
        "endTime": "2020-10-22T15:32:16.000Z"
      }
    }
  ]
}

Now what is left is to render UI based on what data we get.

How To get Live transcriptions and insights in a Telephone call using Symbl SDK

Realtime Output with PSTN Dialing using Voice SDK

In this example let's walk through how to get the live transcription and insights events in a Telephone call.

The code is available here:

Code Sandbox

Getting started

View on Github

Open .env file and add your APP_ID, APP_SECRET, SUMMARY_EMAIL. You can get APP_ID and APP_SECRET from
https://platform.symbl.ai

We will use symbl-node package:

require('dotenv').config()
const {sdk} = require('symbl-node')

Let's start by initialising symbl-node sdk:

await sdk.init({
  appId: process.env.APP_ID,
  appSecret: process.env.APP_SECRET,
  basePath: 'https://api.symbl.ai',
})

Now let's start our connection:

const connection = await sdk.startEndpoint(endpointConfig)

First of all let's provide phone number and endpoint type:

   endpoint: {
   type: 'pstn',
   phoneNumber: process.env.DEFAULT_PHONE_NUMBER,
   },

In case you want to use sip connection, you can use type: sip and provide SIP URI to dial-in to. This should be unique for an active call/meeting in your system. You can also provide dtmf code if you have one. You can find this code on meeting platform invite. You can leave it blank if not connecting to meeting platform

   dtmf: "<code>",
   type: 'sip',
   uri: 'sip:124@domain.com'

You can also pass custom audioConfig. If not provided, it uses PCMU with 800 sample rate. If you want to provide it, you can do it like so:

  audioConfig: {
  encoding: 'OPUS',
  sampleRate: 48000
},

Insights

Symbl provide various insights from the call. Main insights categories are question and action_item. In order to include insights in processing, you need to specify them in configuration like so:

   insightTypes: ['action_item', 'question'],

Actions

You can specify different actions to happen during the call. We will define just one:

   actions: [
    {
      invokeOn: 'stop',
      name: 'sendSummaryEmail',
      parameters: {
        emails: [process.env.SUMMARY_EMAIL], // Add valid email addresses to received email
      },
    },
  ],

When our connection will be stopped, Symbl will send summary mail provided in .env file

Let's also name our session by providing:

   data: {
    session: {
    name: 'My Test Meeting',
    },
  },

Getting connection Id.

For subscribing to the data, we will need to use connectionId unique to each active connection. to get it you can simply retrieve it from connection response:

   const connectionId = connection.connectionId

Sending speaker events.

We can send different speaker events to our connection indicating that different speakers started speaking. That will give us more personalised insights and get better meeting summary

In our example we will do it by calling helper getScheduleEvent function, that we will review in a bit. We pass SpeakerEvent type to it by using SpeakerEvent.types enum from symbl-node, passing user data and timestamp:

  const scheduleEvent = getScheduleEvent(sdk, connectionId)

  setTimeout(() => {
    // Schedule all the events to be sent.
    scheduleEvent(SpeakerEvent.types.startedSpeaking, users.john, 0)
    scheduleEvent(SpeakerEvent.types.stoppedSpeaking, users.john, 5)
  }, 1000)

We retrieve users just from the global array of users but in real world example that might be users data retrieved from database:

  const users = {
    john: {
      userId: 'john@example.com',
      name: 'John',
    },
    mary: {
      userId: 'mary@example.com',
      name: 'Mary',
    },
  }

In order to push event to our connection we will create an event like so:

  const speakerEvent = new SpeakerEvent({
    type: eventType,
    user,
  })

  speakerEvent.timestamp = new Date().toISOString()

And push it using pushEventOnConnection function provided by SDK:

  sdk.pushEventOnConnection(connectionId, speakerEvent.toJSON(), (err) => {
    if (err) {
      console.error('Error during push event.', err)
    } else {
      console.log('Event pushed!')
    }
  })

Realtime insights

In order to get real time data, we need to subscribe to the connection. We can do so by using:

  sdk.subscribeToConnection(connectionId, (data) => {})

data has type field, so we will check for transcript_response, insight_response and message_response

transcript_response are our live transcriptions that we want to preview in the console:

  if (type === 'transcript_response') {
    const {payload} = data
    process.stdout.write('Live: ' + payload && payload.content + '\r')
  }

Transcripts are changing all the time, but once they are processed into reasonable message and not just words, we will get message_response with messages array:

  if (type === 'message_response') {
    const {messages} = data
    messages.forEach((message) => {
      process.stdout.write('Message: ' + message.payload.content + '\n')
    })
  }

And finally if there will be any question or action item during conversation, you will get insight_response:

  if (type === 'insight_response') {
    const {insights} = data
    insights.forEach((insight) => {
      process.stdout.write(`Insight: ${insight.type} - ${insight.text} \n\n`)
    })
  }

As you can see with this data there is a wide array of implementations you can do.

How to access Symbl Node SDK through NextJS

In this how to you'll learn to access Symbl Node SDK through NextJS app to call on the phone or meeting url and get live transcription

Symbl comes with REST api, so same thing we can get by calling specific endpoint on the client side, but there are some use cases that you would want to use particularly server side.For example if you want to do some business logic decisions based on the data received from symbl.

Get Started

Retrieve your credentials

Your credentials include your appId and appSecret. You can find them on the home page of the platform.

Appid

add credentials to .env file filling in APP_ID and APP_SECRET variables.

Create NextJS API routes

Let's create /api/call.ts file which will be our api endpoint. We will use it both for calling to our phone or ending the call.

NextJS API endpoints will reside under pages/api and will have the following structure

export default (req: any, res: any) => {
  res.statusCode = 200
  res.setHeader('Content-Type', 'application/json')
  const params = JSON.parse(req.body)
  res.end(params)
}

we will get bunch of params that we will pass from the client and will pass to Symbl node sdk. One of such params will be start param.

export default (req: any, res: any) => {
  res.statusCode = 200
  res.setHeader('Content-Type', 'application/json')
  const {
    phoneNumber,
    dtmf,
    type,
    summaryEmails,
    start,
    connectionId,
    insightTypes,
  } = JSON.parse(req.body)

  if (start) {
    console.log('Calling ', phoneNumber)
    startEndpoint(
      { phoneNumber, dtmf, type, summaryEmails, insightTypes },
      onSpeechDetected
    ).then((connectionId: string) => {
      console.log('Connection Started', connectionId)
      res.end(JSON.stringify({ connectionId }))
    })
  } else {
    if (connectionId) {
      console.log('Connection Stopped', connectionId)
      stopEndpoint(connectionId).then((connection) => {
        res.end(JSON.stringify(connection))
      })
    }
  }
}

We will use start and connectionId parameters to determine whether we are starting or ending the call.

Basically what happens is every time we start the call, we will get back the connectionId from Symbl, that we will pass to the client. Then later if we want to end the call, we need to pass this connectionId along with start: false to end the call.

Using Symbl Node SDK

If you look at the code the most dominant part will be this one:

startEndpoint(
  { phoneNumber, dtmf, type, summaryEmails, insightTypes },
  onSpeechDetected
)

That is actually a function that we will write, which will pass parameters to symbl node sdk and trigger onSpeechDetected callback. This callback can be your custom business logic, but in our case we will simply log the response.

async function onSpeechDetected(data: any) {
  const { type } = data
  console.log('Response type:', type)
}

Symbl integration

Let's create a file called integrations/symbl/utils.ts and create startEndpoint and stopEndpoint functions, but before we need to initialize our sdk

Initializing sdk

const sdkInit = async () => {
  await sdk.init({
    appId: process.env.APP_ID,
    appSecret: process.env.APP_SECRET,
    basePath: apiBase,
  })
}

We are using .env file in this app, so we need to add our APP_ID and APP_SECRET there.

`startEndpoint`

startEndpoint will get several parameters from the client:

type - Symbl supports two meetings types. PSTN and SIP
phoneNumber - Naturally we need to have a phone number to call, but in case of SIP type we will need to call url. Something like sip:124@domain.com. For simplicity we will pass this data under phoneNumber parameter.
dtmf - This will be meeting code, which we need to pass in case meeting has such a code
summaryEmail - you can pass a list of emails which will get summary of the call email.
insightTypes - there are several insight types that we can pass to Symbl.

export const startEndpoint = async (
  {
    type,
    phoneNumber,
    dtmf,
    summaryEmails,
    insightTypes,
  }: {
    type: string
    phoneNumber: string
    dtmf: string
    summaryEmails: string
    insightTypes: string
  },
  callback: (data: any) => Promise<void>,
  endCallAfterInSeconds = 300
): Promise<any> => {
  //init our sdk
  await sdkInit()
}

Usyng Symbl Node SDK

Now let's import symbl-node import { sdk } from 'symbl-node'

And use startEndpoint method by passing all the parameters we've got from the client.

const phoneNumberOrUri = type === 'sip' ? { uri: phoneNumber } : { phoneNumber }
const connection = await sdk.startEndpoint(
  {
    endpoint: {
      type,
      ...phoneNumberOrUri,
      dtmf,
    },
    intents,
    insightTypes,
    actions: [
      {
        invokeOn: 'stop',
        name: 'sendSummaryEmail',
        parameters: {
          emails: summaryEmails.split(','),
        },
      },
    ],
    data: {
      session: {
        name: `Live Intent Detection NextJS demo - ${phoneNumber}`,
      },
    },
  },
  callback
)

You can see here in the code, that we either pass phoneNumber or uri, based on whether we have SIP or PSTN configuration

When executing this function we will initiate the call and tell Symbl to analyse it. What will be returned is connectionId in which we will be particalarly interested for both subscribing to events and stopping our call.

`stopEndpoint`

This function will get connectionId and call sdk.stopEndpoint function to stop the connection.

const stopEndpoint = async (connectionId: string) => {
  console.log('Stopping connection for ' + connectionId)

  try {
    const connection = await sdk.stopEndpoint({
      connectionId,
    })

    console.log('Summary Info:', connection.summaryInfo)
    console.log('Conversation ID:', connection.conversationId)

    return {
      summaryInfo: connection.summaryInfo,
      conversationId: connection.conversationId,
    }
  } catch (e) {
    console.error('Error while stopping the connection.', e)
    throw e
  }
}

Now we want to create an endpoint that we can call to subscribe for realtime events. For our example we will use socket.io library to use websockets and emit changes from the server to the client.

Let's create a new endpoint by creating a file /api/subscribeToPhoneEvents.ts

We will use Socket.io specific setup within our handler

if (!res.socket.server.io) {
  const io = new Server(res.socket.server)
  io.on('connection', (socket: any) => {
    socket.on('subscribeToEvents', (msg: any) => {
      subscribeToRealtimeEvents(msg.connectionId, (data) => {
        socket.emit('realtimeEvent', data)
      })
    })
    socket.on('endCall', (msg: any) => {
      stopEndpoint(msg.connectionId)
    })
  })

  res.socket.server.io = io
} else {
  console.log('socket.io already running')
}
res.end()

So what we are doing here?

Starting socket.io server if it doesn't exist const io = new Server(res.socket.server)
When client is connected, we initiate subscription:

socket.on('subscribeToEvents', (msg: any) => {
  subscribeToRealtimeEvents(msg.connectionId, (data) => {
    socket.emit('realtimeEvent', data)
  })
})

Here we basically pass connectionId that was created after our call to telephony API to helper function subscribeToRealtimeEvents. We also pass a callback to trigger realtimeEvent that will push the event and the data from the server to client via websockets.

Subscribing to Symbl realtime events

To subscribe to realtime events, we will create another helper function under integrations/symbl/utils.ts

export async function subscribeToRealtimeEvents(
  connectionId: string,
  handler: (data: any) => void
) {
  console.log(`Subscribe to events of connection: ${connectionId}`)
  await sdkInit()
  sdk.subscribeToConnection(connectionId, handler)
}

Here we will instantiate sdk and call subscribeToConnection method within NodeSDK.

Server side summary

At this point we both have /api/call api route to initiate the call and api/subscribeToPhoneEvents to subscribe to events using socket.io

Client side setup

In order to set up our client properly we need two parts.

Creating PhoneConfigurations component - we won't dive into this React component, but the main idea of it is to get user input for the following parameters and call /api/call endpoint with these params.

const params = {
  type,
  phoneNumber,
  dtmf,
  summaryEmails,
  insightTypes,
}

Subscribing based on connectionId. We've seen already that we are getting back connectionId from api/call endpoint, so in order to subscribe to real time events we will use this:

useEffect(() => {
  console.log('Connection Id:', connectionId)
  if (connectionId) {
    fetch('/api/subscribeToPhoneEvents', {
      method: 'POST',
    }).finally(() => {
      socket = io()
      socket.on('connect', () => {
        console.log('connect')
        socket.emit('subscribeToEvents', { connectionId })
      })

      socket.on('realtimeEvent', (data: any) => {
        console.log('Real time event', data)
        if (data.type === 'transcript_response') {
          setLiveTranscript(data.payload.content)
        }
        if (data.type === 'message_response') {
          const oldMsgs = messagesList.current
          setMessages([...oldMsgs, ...data.messages])
        }
        if (data.type === 'insight_response') {
          const oldInsights = insightsList.current
          setInsights([...oldInsights, ...data.insights])
        }
      })

      socket.on('disconnect', () => {
        console.log('disconnect')
      })
    })
  }
}, [connectionId])

As you can see in the code, we set different state variables based on different responses we are getting from socket.io realtimeEvent message.

Response that we will get will be either transcript_response type or message_response type or insight_response type. For example message_response will look like this:

{
  "type": "message_response",
  "messages": [
    {
      "from": {},
      "payload": {
        "content": "Yeah, tell us sir.",
        "contentType": "text/plain"
      },
      "duration": {
        "startTime": "2020-10-22T15:32:14.500Z",
        "endTime": "2020-10-22T15:32:16.000Z"
      }
    }
  ]
}

Now what is left is to render UI based on what data we get.

How to pass different audio codecs to Symbl endpoint

The code is available here:

Code Sandbox

Sometimes you need to pass different audio codecs when passing the audio. This example shows how to pass them. The ones currently supported are

OPUS:
- Supported Sample Rates -- 16000Hz, 24000Hz,48000Hz
- Both CBR (Constant Bitrate) and VBR (Variable Bitrate) are supported
- Support for in-band FEC

SPEEX:
- Supported Sample Rates -- 16000Hz
- VBR is not supported

LINEAR16
- Supported Sample Rates -- 44100Hz

MULAW
- Supported Sample Rates -- 8000Hz

Getting started

View on Github

Open .env file and add your APP_ID, APP_SECRET, SUMMARY_EMAIL. You can get APP_ID and APP_SECRET from
https://platform.symbl.ai

we will use symbl-node package.

require('dotenv').config()

const {sdk} = require('symbl-node')

Let's start by initialising symbl-node sdk

await sdk.init({
  appId: process.env.APP_ID,
  appSecret: process.env.APP_SECRET,
  basePath: 'https://api.symbl.ai',
})

Now after we initialised, we need to start the connection by running

const connection = await sdk.startEndpoint(endpointConfig)

So how do you pass custom codecs? It's as simple as passing custom audio config

          endpoint: {
            //*****************Custom Audio Config******************
            audioConfig: {
              encoding: 'OPUS',
              sampleRate: 16000,
            },
            //******************************************************
            type: 'pstn',
            phoneNumber: process.env.DEFAULT_PHONE_NUMBER,
          },

If you have a requirement to use a codec not included in the ones above or have any other queries, please drop an e-mail to support@symbl.ai

Getting started

View on Github

This example runs on node server, so we will use symbl-node package.

Open .env file and add your APP_ID, APP_SECRET, SUMMARY_EMAIL. You can get APP_ID and APP_SECRET from https://platform.symbl.ai

  require('dotenv').config()
  const {sdk} = require('symbl-node')

For demo purposes, we're using mic to simply get audio from microphone and pass it on to websocket connection

We use mic module so make sure you check that all requirements for the package are met. For instance we need sox package or ALSA tools to be installed.

mic installation instructions: https://www.npmjs.com/package/mic#installation

on Mac it can be as simple as running brew install sox

  const mic = require('mic')
  const sampleRateHertz = 16000
  const micInstance = mic({
    rate: sampleRateHertz,
    channels: '1',
    debug: false,
    exitOnSilence: 6,
  })

Initialize SDK

  (async () => {
    try {
      await sdk.init({
        appId: process.env.APP_ID,
        appSecret: process.env.APP_SECRET,
        basePath: 'https://api.symbl.ai',
      })
    } catch (e) {}
  })()

We will also need a unique id to associate to our symbl request. We will create this Id using uuid package

const id = uuid();

Now we can start the connection using sdk.startRealtimeRequest. We need to provide several important options to our realtime request.

insightTypes - we need to provide which insights will be detected. Supported ones are action_item and question.
config object with meeting title, confidence treshold and the sampleRate
speaker object to define who is the speaker. To distinguish between different speakers we also need to provide userId with valid email, so after meeting will end, we will receive summary email
handlers - these handlers are used to detect speach, messages and insights.
- onSpeechDetected - This will return live transcription of the call
- onMessageResponse - When processed messages are available, this callback will be called
- onInsightResponse - When Symbl detects an insight, this callback will be called.

So our complete code will look like this:

const connection = await sdk.startRealtimeRequest({
  id,
  insightTypes: ['action_item', 'question'],
  config: {
    meetingTitle: 'My Test Meeting',
    confidenceThreshold: 0.7,
    timezoneOffset: 480, // Offset in minutes from UTC
    languageCode: 'en-US',
    sampleRateHertz,
  },
  speaker: {
    // Optional, if not specified, will simply not send an email in the end.
    userId: 'valid@email.com', // Update with valid email
    name: 'My name',
  },
  handlers: {
    onSpeechDetected: (data) => {
      console.log(JSON.stringify(data))
      // For live transcription
      if (data) {
        const {punctuated} = data
        console.log('Live: ', punctuated && punctuated.transcript)
      }
    },
    onMessageResponse: (data) => {
      // When a processed message is available
      console.log('onMessageResponse', JSON.stringify(data, null, 2))
    },
    onInsightResponse: (data) => {
      // When an insight is detected
      console.log('onInsightResponse', JSON.stringify(data, null, 2))
    },
  },
})

How To use different languages and timezones with Symbl

Code Sandbox

This example shows how to use languages other than English and also how to pass in timezone in which the conversation is taking place. For languages: Currently the following languages are supported

['en-US', 'en-AU', 'en-GB', 'fr-CA', 'fr-FR', 'de-DE', 'it-IT', 'nl-NL', 'es-US', 'ja-JP']

The above are all BCP-47 standard language codes and currently ONLY 1 should be passed in the languages array as shown below. Support for detecting multiple languages in the same conversation will be added soon!

For timezone: Please refer to https://en.wikipedia.org/wiki/List_of_tz_database_time_zones for a list of timeZones

You can also use moment-timezone package to obtain a list of timeZones like the following const timeZones = moment.tz.names()

NOTE: If languages is NOT passed in the startEndpoint call the API will fallback to 'en-US'. If timezone is NOT passed the API will fall back to 'UTC'.

Getting started

View on Github

Open .env file and add your APP_ID, APP_SECRET, SUMMARY_EMAIL. You can get APP_ID and APP_SECRET from https://platform.symbl.ai

we will use symbl-node package.

require('dotenv').config()

const {sdk} = require('symbl-node')

Let's start by initialising symbl-node sdk

await sdk.init({
  appId: process.env.APP_ID,
  appSecret: process.env.APP_SECRET,
  basePath: 'https://api.symbl.ai',
})

Now start your endpoint and provide language and timezone properties:

const connection = await sdk.startEndpoint({
  ...config,
  languages: ['ja-JP'],
  timezone: 'Asia/Tokyo',
})

How To Dial-in using Symbl Voice SDK, push speaker events and get summary URL

Code Sandbox

PSTN Dial-in using Voice SDK

In this example, it establishes a connection using a phone number through PSTN, to send speaker events, generate insights, and display a summary URL with the output. You can see how to initialize the Voice SDK, connect to the endpoint, push speaker events and get the summary URL.

Getting started

View on Github

Open .env file and add your APP_ID, APP_SECRET, SUMMARY_EMAIL. You can get APP_ID and APP_SECRET from https://platform.symbl.ai

Let's start by inintialising symbl-node sdk

await sdk.init({
  appId: process.env.APP_ID,
  appSecret: process.env.APP_SECRET,
  basePath: 'https://api.symbl.ai',
})

Connecting

const connection = await sdk.startEndpoint(endpointConfig)

First of all let's provide phone number and endpoint type:

      endpoint: {
        type: 'pstn',
        phoneNumber: process.env.DEFAULT_PHONE_NUMBER,
      },

   dtmf: "<code>",
   type: 'sip',
   uri: 'sip:124@domain.com'

You can also pass custom audioConfig. If not provided, it uses PCMU with 800 sample rate. If you want to provide it, you can do it like so:

audioConfig: {
  encoding: 'OPUS',
  sampleRate: 48000
},

Getting connection Id.

To send speaker events we will need connectionId unique to each active connection. to get it you can simply retrieve it from connection response:

const connectionId = connection.connectionId

Sending speaker events.

We can send different speaker events to our connection indicating that different speakers started speaking. That will give us more personalised insights and get better meeting summary

const scheduleEvent = getScheduleEvent(sdk, connectionId)

setTimeout(() => {
  // Schedule all the events to be sent.
  scheduleEvent(SpeakerEvent.types.startedSpeaking, users.john, 0)
  scheduleEvent(SpeakerEvent.types.stoppedSpeaking, users.john, 5)
}, 1000)

We retrieve users just from the global array of users but in real world example that might be users data retrieved from database.

const users = {
  john: {
    userId: 'john@example.com',
    name: 'John',
  },
  mary: {
    userId: 'mary@example.com',
    name: 'Mary',
  },
}

In order to push event to our connection we will create an event like so

const speakerEvent = new SpeakerEvent({
  type: eventType,
  user,
})

speakerEvent.timestamp = new Date().toISOString()

And push it using pushEventOnConnection function provided by SDK

sdk.pushEventOnConnection(connectionId, speakerEvent.toJSON(), (err) => {
  if (err) {
    console.error('Error during push event.', err)
  } else {
    console.log('Event pushed!')
  }
})

Get real-time transcriptions with native WebSocket API

In this how to, you will learn how to get started with Symbl’s most accurate conversation analysis api, native WebSocket API.

const accessToken = await getAccessToken();
const symblEndpoint = 'wss://api.symbl.ai/v1/realtime/insights/' +
                        UNIQUE_MEETING_ID +
                        '?accessToken=' + SYMBL_ACCESS_TOKEN;

Create the WebSocket endpoint by tagging on a unique meeting identifier and a query param, accessToken to the url:

const uniqueMeetingId = btoa(EMAIL_ADDRESS)
const accessToken = await getAccessToken();
const symblEndpoint = 'wss://api.symbl.ai/v1/realtime/insights/' +
                        UNIQUE_MEETING_ID +
                        '?accessToken=' + accessToken;

Now that we have constructed the endpoint, let's create a new WebSocket!

const ws = new WebSocket(symblEndpoint);

Before we connect the WebSocket to the endpoint we first want to subscribe to its event listeners so we don’t miss any messages.

// Fired when a message is received from the WebSocket server
ws.onmessage = (event) => {
    console.log(event);
};

// Fired when the WebSocket closes unexpectedly due to an error or lost connetion
ws.onerror  = (err) => {
   console.error(err);
};

// Fired when the WebSocket connection has been closed
ws.onclose = (event) => {
  console.info('Connection to websocket closed');
};

Now we are ready to start the WebSocket connection.

const ws = new WebSocket(
ws = send(JSON.stringify(
  {
    type: 'start_reqest',
    meetingTitle: 'Websockets How-to', // Conversation name
    insightTypes: ['question', 'action_item'], // Will enable insight generation
    config: {
      confidenceThreshold: 0.5,
      languageCode: 'en-US',
      speechRecognition: {
        encoding: 'LINEAR16',
        sampleRateHertz: 44100,
    },
    speaker: {
      userId: 'example@symbl.ai',
      name: 'Example Sample',
    }
  });

To pass the audio stream to the WebSocket API, we first need to grant access to peripherals. We can do this with the Navigator API by accessing mediaDevices and calling getUserMedia.

const stream = await navigator.mediaDevices.getUserMedia({ audio: true, video: false });

Now that we have granted access to the microphone we can actually use the WebSocket to handle the the WebSocket the data streamed to Symbl’s cloud so transcripts and insights can be analyzed in real-time. We can do this by creating a new AudioContext and using the microphones stream we retrieved from the Promise resolution above to create a new source, and processor.

const handleSuccess = (stream) => {
    const AudioContext = window.AudioContext;
    const context = new AudioContext();
    const source = context.createMediaStreamSource(stream);
    const processor = context.createScriptProcessor(1024, 1, 1);
    const gainNode = context.createGain();
    source.connect(gainNode);
    this.gainNode.connect(processor);
    processor.connect(context.destination);
    processor.onaudioprocess = (e: any) => {
      // convert to 16-bit payload
      const inputData = e.inputBuffer.getChannelData(0) || new Float32Array(this.bufferSize);
      const targetBuffer = new Int16Array(inputData.length);
      for (let index = inputData.length; index > 0; index--) {
          targetBuffer[index] = 32767 * Math.min(1, inputData[index]);
      }
      // Send to websocket
      if (ws.readyState === WebSocket.OPEN) {
        ws.send(targetBuffer.buffer);
      }
    };
  };
  handleSuccess(stream);

Know more about Real-time-websocket.

How to do text processing

Get Started

Retrieve your credentials

Your credentials include your appId and appSecret. You can find them on the home page of the platform.

Appid

add credentials to .env file filling in APP_ID and APP_SECRET variables.

Authenticating

This is how we would retrieve the token:

async function loginToSymbl() {
    const response = await fetch('https://api.symbl.ai/oauth2/token:generate', {
      method: 'POST',
      headers: {
        'Content-Type': 'application/json',
      },
      mode: 'cors',
      body: JSON.stringify({
        type: 'application',
        appId: process.env.APP_ID,
        appSecret: process.env.APP_SECRET,
      }),
    })
    const json = await response.json()
    console.log('Access Token is: ', json)
    setToken(json.accessToken)

Later on in any place in application we can use const { token } = useAuth() to get the token.

Using Symbl Text Async API

The Async Text API allows you to process any text payload to get the transcription and conversational insights. It can be useful in any use case where you have access to the textual content of a type of conversation, and you want to extract the insightful items supported by the Conversation API. If you want to add more content to the same conversation, use PUT Async Text API.

Symbl Text async API will return jobId which we can monitor for completion by using polling mechanism. Once job is completed, we will get conversationId which we can use to retrieve data with Conversation API.

Text upload and processing

Text async API is built for particular format of messages list that you can upload and parse to get conversational insights. The format should look something like that:

[
  {
    "payload": {
      "content": "Hi Mike, Natalia here. Hope you don’t mind me reaching out. Who would be the best possible person to discuss internships and student recruitment at ABC Corp? Would you mind pointing me toward the right person and the best way to reach them? Thanks in advance for your help, I really appreciate it!"
    },
    "from": {
      "userId": "natalia@example.com",
      "name": "Natalia"
    },
    "duration": {
      "startTime": "2020-07-21T16:02:19.01Z",
      "endTime": "2020-07-21T16:04:19.99Z"
    }
  },
  {
    "payload": {
      "content": "Hey Natalia, thanks for reaching out. I am connecting you with Steve who handles recruitements for us."
    },
    "from": {
      "userId": "mike@abccorp.com",
      "name": "Mike"
    },
    "duration": {
      "startTime": "2020-07-21T16:04:19.99Z",
      "endTime": "2020-07-21T16:04:20.99Z"
    }
  }
]

So we will have a textarea on the page where you can paste you content. Then in order to process it, you need to click Submit for processing button. Once you do that, there are several things that happen

1. Get Relevant params for Video URL or file

You will see the following params defined within useEffect

const requestTextOptions = {
  method: 'POST',
  headers: {
    'x-api-key': token,
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({
    messages: JSON.parse(data),
  }),
}

const requestOptions = {
  method: 'GET',
  headers: {
    'x-api-key': token,
  },
}

2. Use Job API to poll for job status.

Previously we mentioned that there should be some kind of polling mechanism to check whether the job is finished or not. Symbl has Job API That we can use for that.

async function check(jobId: string) {
  const checkJob = await fetch(
    `https://api.symbl.ai/v1/job/${jobId}`,
    requestOptions
  )
  const checkJobJson = await checkJob.json()
  setJobStatus(checkJobJson)
  if (checkJobJson.status === 'in_progress') {
    check(jobId)
    return
  } else {
    setSentForProcessing(false)
  }
}

So the flow will be:

// Execute the request
const processingResponse = await fetch(
  'https://api.symbl.ai/v1/process/text',
  requestTextOptions
)
const processingResponseJson = await processingResponse.json()
// Check Job Status
check(processingResponseJson.jobId)

After Job is finished we can get data from Conversation API. We will do that by using <ConversationCard/>component.

On this page you will also see this component <Transcripts conversationId={conversationData.conversationId} />

This is prebuilt component from @symblai/react-elements package. As soon as you provide it with conversationId, It will nicely render conversation transcripts. There is also <Topics conversationId={conversationData.conversationId}/> component that will do the same but for topics.

How to do audio processing

Get Started

Retrieve your credentials

Your credentials include your appId and appSecret. You can find them on the home page of the platform.

Appid

add credentials to .env file filling in APP_ID and APP_SECRET variables.

Authenticating

This is how we would retrieve the token:

async function loginToSymbl() {
    const response = await fetch('https://api.symbl.ai/oauth2/token:generate', {
      method: 'POST',
      headers: {
        'Content-Type': 'application/json',
      },
      mode: 'cors',
      body: JSON.stringify({
        type: 'application',
        appId: process.env.APP_ID,
        appSecret: process.env.APP_SECRET,
      }),
    })
    const json = await response.json()
    console.log('Access Token is: ', json)
    setToken(json.accessToken)

Later on in any place in application we can use const { token } = useAuth() to get the token.

Using Symbl audio Async API

Symbl Audio async API is used to process audio file or url and return jobId which we can monitor for completion by using polling mechanism. Once job is completed, we will get conversationId which we can use to retrieve data with Conversation API.

File Upload and processing.

If you look at pages/audio/index.tsx file, you will notice that for file upload we are using

<FileOrUrlInput onSubmit={onAudioSubmit} />

We won't dive into how it's implemented but basically the idea is to toggle between URL or file input. Once you click on the button with caption Submit for processing, it will call onAudioSubmit function passing either URL or file.

const onAudioSubmit = async (data: FileOrUrlInputSubmitType) => {
  if (data.type === 'file') {
    setFile(data?.file as any)
  } else {
    setAudioUrl(data?.url as string)
  }
}

This will set our state. All the processing is done within useAudioAsyncAPI hook. This hook monitors the state change

const { jobStatus, sentForProcessing } = useAudioAsyncAPI(
  audioUrl !== '' ? audioUrl : file,
  asyncApiParams.query
)

and based on file/url change will start the following flow

1. Transcoding audio file

You will see the following function defined within useEffect

async function transcodeFile(_file: File) {
  // tslint:disable-next-line
  const transcoder = new (Transcoder as any)()
  const transcodedFile = await transcoder.load(_file)
  return transcodedFile
}

We will also create a new AbortController to abort transcoding and also to send controller.signal to Symbl API

const controller = new AbortController()

2. Get Request parameters for either file or url

We will also define a function to conditionally get different request parameters that we need to send in our REST call. After all there is a difference whether we send a file or URL

async function getFileOrUrlOptions() {
  if (isFile) {
    const file = data
    const transcodedFile: any = await transcodeFile(file as File)
    const requestOptionsAudio = {
      method: 'POST',
      headers: {
        'x-api-key': token,
        'Content-Type': MIME_TYPES['mp3'],
      },
      body: transcodedFile,
      signal: controller.signal,
    }
    return requestOptionsAudio
  } else {
    const url = data
    const requestOptionsAudio = await {
      method: 'POST',
      headers: {
        'x-api-key': token,
        'Content-Type': 'application/json',
      },
      body: JSON.stringify({
        url: url,
        confidenceThreshold: 0.6,
        timezoneOffset: 0,
      }),
    }
    return requestOptionsAudio
  }
}

3. Use Job API to poll for job status.

Previously we mentioned that there should be some kind of polling mechanism to check whether the job is finished or not. Symbl has Job API That we can use for that.

async function check(jobId: string) {
  const checkJob = await fetch(
    `https://api.symbl.ai/v1/job/${jobId}`,
    requestOptions
  )
  const checkJobJson = await checkJob.json()
  setJobStatus(checkJobJson)
  if (checkJobJson.status === 'in_progress') {
    check(jobId)
    return
  } else {
    setSentForProcessing(false)
  }
}

So the flow will be:

//Transcode and get Parameters
const requestOptionsAudio = await getFileOrUrlOptions()
// Execute the request
const processingResponse = await fetch(urlAudio, requestOptionsAudio)
const processingResponseJson = await processingResponse.json()
// Check Job Status
check(processingResponseJson.jobId)

After Job is finished we can get data from Conversation API. We will do that by using <ConversationCard/>component.

On this page you will also see this component <Transcripts conversationId={conversationData.conversationId} />

Passing additional Async API parameters to get more granular insights

How to do video processing

Get Started

Retrieve your credentials

Your credentials include your appId and appSecret. You can find them on the home page of the platform.

Credentials faf6f434

add credentials to .env file filling in APP_ID and APP_SECRET variables.

Authenticating

This is how we would retrieve the token:

async function loginToSymbl() {
    const response = await fetch('https://api.symbl.ai/oauth2/token:generate', {
      method: 'POST',
      headers: {
        'Content-Type': 'application/json',
      },
      mode: 'cors',
      body: JSON.stringify({
        type: 'application',
        appId: process.env.APP_ID,
        appSecret: process.env.APP_SECRET,
      }),
    })
    const json = await response.json()
    console.log('Access Token is: ', json)
    setToken(json.accessToken)

Later on in any place in application we can use const { token } = useAuth() to get the token.

Using Symbl Video Async API

Symbl Video async API is used to process video file or url and return jobId which we can monitor for completion by using polling mechanism. Once job is completed, we will get conversationId which we can use to retrieve data with Conversation API.

File Upload and processing.

If you look at pages/video/index.tsx file, you will notice that for file upload we are using

<FileOrUrlInput onSubmit={onVideoSubmit} />

const onVideoSubmit = async (data: FileOrUrlInputSubmitType) => {
  setConversationData(null)
  if (data.type === 'file') {
    setFile(data?.file as any)
  } else {
    setVideoUrl(data?.url as string)
  }
}

This will set our state. All the processing is done within useAsyncVideoApi hook. This hook monitors the state change

const { jobStatus, sentForProcessing } = useAsyncVideoApi(
  videoUrl !== '' ? videoUrl : file,
  asyncApiParams.query
)

and based on file/url change will start the following flow

1. Get Relevant params for Video URL or file

You will see the following function defined within useEffect

async function getFileOrUrlOptions() {
  if (isFile) {
    const file = data
    const requestOptionsVideo = {
      method: 'POST',
      headers: {
        'x-api-key': token,
        'Content-Type': 'video/mp4',
      },
      body: file,
      json: true,
    }
    return requestOptionsVideo
  } else {
    const url = data
    const requestOptionsVideo = await {
      method: 'POST',
      headers: {
        'x-api-key': token,
        'Content-Type': 'application/json',
      },
      body: JSON.stringify({
        url: url,
        confidenceThreshold: 0.6,
        timezoneOffset: 0,
      }),
    }
    return requestOptionsVideo
  }
}

Here we define a function to conditionally get different request parameters that we need to send in our REST call. After all there is a difference whether we send a file or URL

2. Use Job API to poll for job status.

Previously we mentioned that there should be some kind of polling mechanism to check whether the job is finished or not. Symbl has Job API That we can use for that.

async function check(jobId: string) {
  const checkJob = await fetch(
    `https://api.symbl.ai/v1/job/${jobId}`,
    requestOptions
  )
  const checkJobJson = await checkJob.json()
  setJobStatus(checkJobJson)
  if (checkJobJson.status === 'in_progress') {
    check(jobId)
    return
  } else {
    setSentForProcessing(false)
  }
}

So the flow will be:

//Transcode and get Parameters
const requestOptionsVideo = await getFileOrUrlOptions()
// Execute the request
const processingResponse = await fetch(urlAudio, requestOptionsVideo)
const processingResponseJson = await processingResponse.json()
// Check Job Status
check(processingResponseJson.jobId)

After Job is finished we can get data from Conversation API. We will do that by using <ConversationCard/>component.

On this page you will also see this component <Transcripts conversationId={conversationData.conversationId} />

This is prebuilt component from @symblai/react-elements package. As soon as you provide it with conversationId, It will nicely render conversation transcripts. There is also <Topics conversationId={conversationData.conversationId}/> component that will do the same but for topics.

Passing additional Async API parameters to get more granular insights

How to retrieve Conversation Data

Get Started

Retrieve your credentials

Your credentials include your appId and appSecret. You can find them on the home page of the platform.

Appid

add credentials to .env file filling in APP_ID and APP_SECRET variables.

Authenticating

This is how we would retrieve the token:

async function loginToSymbl() {
    const response = await fetch('https://api.symbl.ai/oauth2/token:generate', {
      method: 'POST',
      headers: {
        'Content-Type': 'application/json',
      },
      mode: 'cors',
      body: JSON.stringify({
        type: 'application',
        appId: process.env.APP_ID,
        appSecret: process.env.APP_SECRET,
      }),
    })
    const json = await response.json()
    console.log('Access Token is: ', json)
    setToken(json.accessToken)

Later on in any place in application we can use const { token } = useAuth() to get the token.

Getting data from Conversation API

In order to get data from conversation API we will need to get conversationId. We can either use a demo id: 6055920157065216 or we can initiate the phone call using either Symbl Node SDK or Symbl REST api. Both flows will update conversationId parameter in the app and we will be able to retrieve data that we want.

In this demo app we have <ConversationCard/> component that will accept the following parameters:

const ConversationCard = ({
  title,
  children,
  conversationId: cId,
  type,
}: {
  title: string
  conversationId: string
  children: JSX.Element
  type:
    | 'messages'
    | 'insights'
    | 'topics'
    | 'members'
    | 'questions'
    | 'follow-ups'
    | 'action-items'
}) => {}

This card component can render different data retrieved from Conversation API. The functionality is rather simple. Based on different types, card will render different data retrieved from Conversation API.

const fetchData = async () => {
  const res = await fetch(`${CONVERSATION_API}/${conversationId}/${endpoint}`, {
    method: 'GET',
    headers: {
      'x-api-key': token as string,
      'Content-Type': 'application/json',
    },
    mode: 'cors',
  })

  const json = await res.json()
  setResponseData(json)
}

You can read more about conversation API here

Get Speaker Separated Transcripts - Diarization with Async API

Enable the Speaker Diarization (Speaker Separation) for the Async Audio or Async Video APIs to get speaker separated transcripts and insights.

Enabling the Diarization

Enabling Speaker Separation in the Async Audio/Video API is as simple as adding the enableSpeakerDiarization=true and diarizationSpeakerCount=<NUMBER_OF_UNIQUE_SPEAKERS> query-parameters below:

$ curl --location --request POST 'https://api.symbl.ai/v1/process/video/url?enableSpeakerDiarization=true&diarizationSpeakerCount=2&webhookUrl=<WEBHOOK_URL>' --header 'Content-Type: application/json' --header 'x-api-key: <X-API-KEY> --data-raw '{
    "url": "https://storage.googleapis.com/demo-conversations/interview-prep.mp4"
}'

The above snippet shows a cURL command for consuming the Async Video URL based API which takes in the url for a publicly available URL of a Video File.

The <X-API-KEY> needs to be replaced with the token generated token:generate endpoint.
The <WEBHOOK_URL> can be replaced with a WebHook URL for receiving the status for the Job created after calling the API.
For accuracy, the <NUMBER_OF_UNIQUE_SPEAKERS> should match the number of unique speakers in the Audio/Video data.

The above URL has two query-parameters:

enableSpeakerDiarization=true which will enable the speaker separation for the Audio/Video data under consideration.
diarizationSpeakerCount=2 which sets the number of unique speakers in the Audio/Video data under consideration.

Getting the uniquely identified speakers (Members)

Invoking the members call in the Conversation API will return the uniquely identified speakers for this conversation when Speaker Diarization is enabled. View a sample output below:

{
    "members": [
        {
            "id": "9d6d34d9-5019-4694-9c9a-8ba7bfc8cfab",
            "name": "Speaker 1"
        },
        {
            "id": "2f69f1c8-bf0a-48ef-b47f-95ae5a4de325",
            "name": "Speaker 2"
        }
    ]
}

The name assigned to a uniquely identified speaker/member from a Diarized Audio/Video will follow the format Speaker <number> where <number> is arbitrary and does not necessarily reflect in what order someone spoke.

The id can be used to identify a speaker/member for that specific conversation and can be used to update the details for the specific member demonstrated below in the Updating Detected Members section.

Getting the Speaker Separated Results

Invoking the messages call in the Conversation API would return the speaker separated results. View a snippet for the above URL below:

{
    "messages": [
        {
            "id": "4591723946704896",
            "text": "You're hired two words, everybody loves to hear.",
            "from": {
                "id": "2f69f1c8-bf0a-48ef-b47f-95ae5a4de325",
                "name": "Speaker 2"
            },
            "startTime": "2020-08-04T07:18:17.573Z",
            "endTime": "2020-08-04T07:18:21.573Z",
            "conversationId": "5105430690791424"
        },
        {
            "id": "6328236401229824",
            "text": "But before we hear these words comes the interview today's video is part one in a series.",
            "from": {
                "id": "2f69f1c8-bf0a-48ef-b47f-95ae5a4de325",
                "name": "Speaker 2"
            },
            "startTime": "2020-08-04T07:18:21.973Z",
            "endTime": "2020-08-04T07:18:30.473Z",
            "conversationId": "5105430690791424"
        },
        ...
    ]
}

The above snippet shows the speaker in the from object with a unique-id. These are the uniquely identified members of this conversation.

Similarly, invoking the insights call in the Conversation API would also reflect the identified speakers in the detected insights. The response below demonstrates this:

{
    "insights": [
        {
            "id": "5501181057040384",
            "text": "We need to go over three more common interview questions.",
            "type": "action_item",
            "score": 1,
            "messageIds": [
                "5710067261243392"
            ],
            "entities": [],
            "phrases": [
                {
                    "type": "action_phrase",
                    "text": "go over three more common interview questions"
                }
            ],
            "from": {
                "id": "2f69f1c8-bf0a-48ef-b47f-95ae5a4de325",
                "name": "Speaker 2"
            },
            "definitive": true,
            "assignee": {
                "id": "2f69f1c8-bf0a-48ef-b47f-95ae5a4de325",
                "name": "Speaker 2"
            }
        },
        {
            "id": "5519156904460288",
            "text": "How did you hear about this position?",
            "type": "question",
            "score": 0.999988666660899,
            "messageIds": [
                "4616389407014912"
            ],
            "from": {
                "id": "2f69f1c8-bf0a-48ef-b47f-95ae5a4de325",
                "name": "Speaker 2"
            }
        },
        ...
    ]
}

Updating the Detected Members

The detected members (unique speakers) would have names like Speaker 1 as the ASR wouldn’t have any context to who this speaker is (name or other details of the speaker). Therefore, it is important to update the details of the detected speakers after the Job is complete.

The members call in the Conversation API returns the uniquely identified speakers as shown in the Getting the uniquely identified speakers (Members) section above when the Diarization is enabled.

Let’s consider the same set of members that can be retrieved by calling the GET members call in the Conversation API.

{
    "members": [
        {
            "id": "9d6d34d9-5019-4694-9c9a-8ba7bfc8cfab",
            "name": "Speaker 1"
        },
        {
            "id": "2f69f1c8-bf0a-48ef-b47f-95ae5a4de325",
            "name": "Speaker 2"
        }
    ]
}

We can now use the PUT members call to update the details of a specific member as shown below. This call would update the Speaker 2 as shown in the above section with the values in the cURL’s request-body

$ curl --location --request PUT 'https://api.symbl.ai/v1/conversations/<CONVERSATION_ID>/members/2f69f1c8-bf0a-48ef-b47f-95ae5a4de325' --header 'Content-Type: application/json' --header 'x-api-key: <X-API-KEY> --data-raw '{
    "id": "2f69f1c8-bf0a-48ef-b47f-95ae5a4de325",
    "email": "john@example.com",
    "name": "John Doe"
}'

The <CONVERSATION_ID> needs to be replaced with the actual conversationId
The <X-API-KEY> needs to be replaced with the token generated with the token:generate endpoint.

The URL in line 1 above has the id of the member we want to append to /members with the request body containing the updated nameof this member.

There is also the option to include the email of the member. The email will be used as an identifier for tracking those specific members uniquely in that conversation. (Refer the Appending to an existing conversation with Diarization section below for more details)

After the above call is successful, we will receive the following response:

{
    "message": "Member with id: 2f69f1c8-bf0a-48ef-b47f-95ae5a4de325 for conversationId: <CONVERSATION_ID> updated successfully! The update should be reflected in all messages and insights along with this conversation"
}

The message is self-explanatory and tells us that all the references to the member with the id of 2f69f1c8-bf0a-48ef-b47f-95ae5a4de325 in the conversation should now reflect the new values we updated this member with. That includes insights, messages and the conversation’s members as well.

So if we call the members API now, we would see the following result:

{
    "members": [
        {
            "id": "9d6d34d9-5019-4694-9c9a-8ba7bfc8cfab",
            "name": "Speaker 1"
        },
        {
            "id": "2f69f1c8-bf0a-48ef-b47f-95ae5a4de325",
            "email": "john@example.com",
            "name": "John Doe"
        }
    ]
}

And similarly, with the messages API call, we would see the updates reflected below as well:

{
    "messages": [
        {
            "id": "4591723946704896",
            "text": "You're hired two words, everybody loves to hear.",
            "from": {
                "id": "2f69f1c8-bf0a-48ef-b47f-95ae5a4de325",
                "email": "john@example.com",
                "name": "John Doe"
            },
            "startTime": "2020-08-04T07:18:17.573Z",
            "endTime": "2020-08-04T07:18:21.573Z",
            "conversationId": "5105430690791424"
        },
        {
            "id": "6328236401229824",
            "text": "But before we hear these words comes the interview today's video is part one in a series.",
            "from": {
                "id": "2f69f1c8-bf0a-48ef-b47f-95ae5a4de325",
                "email": "john@example.com",
                "name": "John Doe"
            },
            "startTime": "2020-08-04T07:18:21.973Z",
            "endTime": "2020-08-04T07:18:30.473Z",
            "conversationId": "5105430690791424"
        },
        ...
    ]
}

Curious about the insights API? It would reflect these updates as well!

{
    "insights": [
        {
            "id": "5501181057040384",
            "text": "We need to go over three more common interview questions.",
            "type": "action_item",
            "score": 1,
            "messageIds": [
                "5710067261243392"
            ],
            "entities": [],
            "phrases": [
                {
                    "type": "action_phrase",
                    "text": "go over three more common interview questions"
                }
            ],
            "from": {
                "id": "2f69f1c8-bf0a-48ef-b47f-95ae5a4de325",
                "email": "john@example.com",
                "name": "John Doe"
            },
            "definitive": true,
            "assignee": {
                "id": "2f69f1c8-bf0a-48ef-b47f-95ae5a4de325",
                "name": "Speaker 2"
            }
        },
        {
            "id": "5519156904460288",
            "text": "How did you hear about this position?",
            "type": "question",
            "score": 0.999988666660899,
            "messageIds": [
                "4616389407014912"
            ],
            "from": {
                "id": "2f69f1c8-bf0a-48ef-b47f-95ae5a4de325",
                "email": "john@example.com",
                "name": "John Doe"
            }
        },
        ...
    ]
}

Appending to an existing conversation with Diarization

Because conversations don’t neatly end at once and may resume later, our Async API allows you to update/append an existing conversation. You can read more about this capability here.

To enable Diarization with the append capability, the request structure is the same as shown above for creating a new Conversation. You would need to pass in enableSpeakerDiarization=true and diarizationSpeakerCount=<NUMBER_OF_UNIQUE_SPEAKERS> query-parameters.

However, there is one caveat in how the ASR works with Diarization. Consider the below:

An example scenario

We send a recorded conversation to the Async API with 2 speakers John and Alice with enableSpeakerDiarization=true. The diarization identifies them as Speaker 1 and Speaker 2 respectively. We then update the above speakers with their email as john@example.com and alice@example.com

Now we use the append call for appending another conversation with 2 speakers John and May with enableSpeakerDiarization=true. Let’s assume that the diarization would now identify these as Speaker 1 and Speaker 2 respectively. As discussed before, these numbers are arbitrary and have nothing to do with the order in which the speakers spoke in the conversation.

After this job is complete we will have 4 members in this conversation:

John
Alice
Speaker 1 (Which is John again)
Speaker 2 (Which is May)

Since John and Speaker 1 refer to the same speaker but are labeled as different speakers, their member references would be different for all messages and insights that they are a part of.

The `email` identifier

This is where the email identifier comes in. The PUT members call can uniquely identify and merge a member with the same email parameter and eliminate any duplicate references with a single reference across the entire conversation which would update all the references including the members, messages and insights.

If we were to execute a PUT members call with the below body where 74001a1d-4e9e-456a-84ed-81bbd363333a refers to the id of Speaker 1 from the above scenario, this would eliminate this member and would update all the references with member represented by 2f69f1c8-bf0a-48ef-b47f-95ae5a4de325 which we know is John Doe.

$ curl --location --request PUT 'https://api.symbl.ai/v1/conversations/<CONVERSATION_ID>/members/74001a1d-4e9e-456a-84ed-81bbd363333a' --header 'Content-Type: application/json' --header 'x-api-key: <X-API-KEY> --data-raw '{
    "id": "74001a1d-4e9e-456a-84ed-81bbd363333a",
    "email": "john@example.com",
    "name": "John Doe"
}'

This is possible because the email uniquely identifies that user.

Best Practices

The diarizationSpeakerCount should be equal to the number of unique speakers present in the conversation for best results as the Diarization model uses this number to probabilistically determine the speakers. If this number is different than the actual speakers, then it might introduce false-positives for some part of the transcriptions.
For the best experience, the Sample Rate of the data should be greater than or equal to 16000Hz.

That’s it!

You have the steps to start separating the speakers in any Audio/Video-based conversation.

You can visit docs.symbl.ai to learn more about the following APIs:

Symbl Adapter for Chime SDK

The Symbl Conversation AI Adapter for Chime SDK is the simplest way to get started with Symbl in your Amazon Chime video platform.

Currently, the Symbl Conversational AI Adapter has the following features:

Realtime closed-captioning.
Realtime conversational insights (Action Items, Questions, Follow-ups).
Transcription.

Prerequisites

You must have the following installed:

Set Up for Symbl

Install symbl-chime-adapter

To get started you will need to add the symbl-chime-adapter to your node dependencies.

$ npm install --save symbl-chime-adapter

Symbl Credentials

Create an account in the Symbl Console if you have not done so already.
After you login, you will find your appId and appSecret on the home page.
Create a .env file in demos/browser and demos/serverless/src that includes your appId and appSecret as shown below.

SYMBL_APP_ID=<xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx>
SYMBL_APP_SECRET=<xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx>

The App ID and App Secret are used to authenticate your session with Symbl by generating an access token.

Initialization

The Symbl adapter should be initialized after connecting to your Chime video conference.

To connect each Chime participant to Symbl, it is best practice to generate the client’s access token on your backend service.

After the token is generated it can be returned along with the chime meeting configuration as shown below. For a full example of this, please see this page of our chime demo application.

const res = await fetch(`https://api.symbl.ai/oauth2/token:generate`, {
      method: 'POST',
      mode: 'cors',
      cache: 'no-cache',
      credentials: 'same-origin',
      headers: {
        'Content-Type': 'application/json',
      },
      redirect: 'follow',
      referrerPolicy: 'no-referrer',
      body: JSON.stringify({
        type: 'application',
        appId: symblAppId,
        appSecret: symblAppSecret,
      }),
    });
    accessToken = await res.json();

    return response(200, 'application/json', JSON.stringify({
    JoinInfo: {
      Meeting: meeting,
      Attendee: attendee,
      Symbl: accessToken,
    },
  }, null, 2));
};

After the token is generated and returned by your server, the Symbl Access Token can be applied by setting the static property ACCESS_TOKEN of the Symbl class.

Symbl.ACCESS_TOKEN = joinInfo.Symbl.accessToken;

After the access token has been set and your client has joined the Chime meeting, you can instantiate the Symbl class.

The Symbl constructor takes two parameters:

Chime

Field	Type	Required	Description
attendeeId	string	true	ID for video conference participant
userName	string	false	Human readable name for video conference participant
meetingId	string	true	Unique identifier for the meeting
meeting	string	true	Name of the meeting

Options

Field	type	Required	Supported Values	Default Value	Description
confidenceThreshold	double	false	0.5-1.0	0.5	Your access token retrieved through authentication with
languageCode	boolean	false	en-US, en-GB, en-AU, it-IT, nl-NL, fr-FR, fr-CA, de-DE, es-US, ja-JP	en-US	The language code as per the BCP 47 specification
insightsEnabled	boolean	false		true	Whether or not to generate insights.

Example Config

/**
    @param {object} chime - chime configuration
        {
            @property {string} attendeeId -- Client attendee id
            @property {string} userName - User name
            @property {string} meetingId -- UUID of the meeting
            @property {string} meeting meeting name
        },
        @param {object} config - Symbl Configuration
        {
            @property {number} confidenceThreshold  optional | default: 0.5 | 0.0 - 1.0 minimum confidence value produce valid insight
            @property {string} languageCode         optional - default: 'en-US' | The language code as per the BCP 47 specification
            @property {boolean} insightsEnabled     optional - default: true -- false if language code is not english.
        }
*/
  this.symbl = new Symbl(
      {
          attendeeId: this.configuration.credentials.attendeeId,
          userName: this.configuration.credentials.externalUserId.split('#').pop(),
          meetingId: this.configuration.credentials.meetingId,
          meeting: this.meeting,
      }, {
          confidenceThreshold: 0.5,
          languageCode: 'en-US',
          insightsEnabled: true,
      }
  );

Learn how to subscribe to Symbl’s conversational insights and transcription.

There are three different event types that you can subscribe through with Symbl’s Conversational AI Adapter for Chime.

Closed Captions
Transcripts
Insights

Each subscriber function, subscribeToCaptioningEvents

Closed Captions

Real-time closed captioning is enabled by default and provides realtime transcription of your audio content. Closed captioning tracks can be added to any video element through the callback handler to your Symbl instance.

Handler

Closed caption handler has 3 callback functions:

onCaptioningToggled - Will be called whenever closed captioning is toggled on or off.
oncCaptionCreated - Event that occurs after speech is first detecting, having the caption object created after as a parameter.
onCaptionUpdated - Event that occurs after speech is no longer detected, indicating the end of transcriipton.

The handler can be added using the subscribeToCaptioningEvents function of the Symbl instance.

const captioningHandler = {
    onCaptioningToggled: (ccEnabled: boolean) => {
        // Implement
    },
    onCaptionCreated: (caption: Caption) => {
        console.warn('Caption created', caption);
        // Retrieve the video element that you wish to add the caption tracks to.
        const activeVideoElement = getActiveVideoElement() as HTMLVideoElement;
        if (activeVideoElement) {
            subtitle.setVideoElement(activeVideoElement);
        }
    },
    onCaptionUpdated: (caption: Caption) => {
        const activeVideoElement = getActiveVideoElement() as HTMLVideoElement;

        subtitle.setVideoElement(activeVideoElement);
    },
};
symbl.subscribeToCaptioningEvents(captioningHandler);

Setting the video element that subtitles will be superimposed over should be done by calling the setVideoElement function on the Caption class.

If your video chat application has alternating primary video tiles, this can be used to change which element is active.

Realtime Insights

Realtime insights are generated as Symbl processes the conversation in your video chat platform.

When an Insight is detected by Symbl and the onInsightCreated event is emitted, an Insight object is passed to the callback function provided in the Insight handler.

The Insight class holds data about the insight generated.

Insight Class Attributes

Field	Type	Description
type	string	Insight type. Can beaction_item, question, or follow_up.
text	string	Insight content
id	string	ID of the Insight
confidence	number

Assignee

field	type	description
name	string	Insight assignee name
userId	string	Insight assignee email address
id	string	Insight assignee ID

Hints (follow-up's only)

field	type	description
key	string	Insight assignee name
value	string	Insight assignee email address
id	string	Insight assignee ID

Field	Type	Description
key	string
value

Handler

The Symbl adapter exposes a handler function, subscribeToInsightEvents, that has a callback function onInsightCreated.

new Symbl(chimeConfiguration, {insightsEnabled: true});

Subscribing to the Insight publisher is achieved by passing a handler to the subscribeToInsightEvents function of your Symbl instance.

Example

When an insight event is emitted from the onInsightCreated handler, you can use the Insight object returned and either use the createElement function to create a default element or you can use the data included in the Insight object returned in the handlers callback to create your own element, capture the data and store as a metric, etc.

const insightHandler = {
    onInsightCreated: (insight: Insight) => {

        // Creates a predesigned insight element
        const element = insight.createElement();

        // Customize any styling
        element.classList.add('mx-auto');
        element.style.width = '98%';

        /** OR create your own element
        insight.element = document.createElement('div');
        insight.element.innerHTML = `<div style="width: auto; height: 400px">
                                 <h3 style="font-width: 400;">
                                     ${insight.type}
                                 </h3>
                                 <br>
                                 <h2>
                                     ${insight.text}
                                 </h2>
                              </div>`;
        **/

        // Retrieve container you wish to add insights to.
        const insightContainer = document.getElementById('receive-insight');

        // Call add on the insight object to add it to DIV
        insight.add(insightContainer);
    }
};

// Subscribe to realtime insight events using the handler created above
this.symbl.subscribeToInsightEvents(insightHandler);

Default Insights Element

Realtime Transcripts

Realtime transcripts differ slightly from realtime closed-captioning. A Transcript object is created when after speech is no longer detected.

Attributes

Field	Type	Description
message	string	Speaker content
userName	string	Speaker name
userId	string	Speaker ID
timeStamp	Date	Time of the transcript event

Handler

The Transcript handler has only one event, onTranscriptCreated. Each time a speaker finishes speaking a Transcript object is generated.

Subscribing to the transcript publisher is achieved by passing a handler to the subscribeToTranscriptEvents function of your Symbl instance.

const transcriptHandler = {
    onTranscriptCreated: (transcript: Transcript) => {
      // Handle transcript item
    }
};
this.symbl.subscribeToTranscriptEvents(transcriptHandler);

Generating Meeting Summary URL

To generate a meeting summary URL, you need only to call the getSummaryUrlfunction of your Symbl instance.

const meetingSummaryUrl = await symbl.getSummaryUrl();

Symbl SDK (Node.js)

The Programmable Symbl SDK allows you to add Conversational Intelligence directly into your web applications and meeting platforms. With the Symbl SDK , you can generate intelligent insights such as action items, topics and questions.

This section demonstrates how to use the SDK to add voice integration to your existing application.

Symbl SDK (Node.js)

Initialize the Symbl SDK
Connect to Endpoints
Active Speaker Events
Subscribe to Real-time Events
Language and Timezone
Complete Example
Send Summary Email
Tuning your Summary Page
Streaming Audio in Real-time

System requirement: NodeJS 7+

Initialize the SDK

After installing, initialize the SDK

 sdk.init({
    appId: 'yourAppId',
    appSecret: 'yourAppSecret',
    basePath: 'https://api.symbl.ai'
  })
  .then(() => console.log('SDK Initialized.'))
  .catch(err => console.error('Error in initialization.', err));

1. After getting your appId and appSecret, use the command below to install the SDK and add it to your npm project's package.json.

$ npm install --save symbl-node

2. Reference the SDK in either the ES5 or ES6 way.

The ES5 way

var sdk = require('symbl-node').sdk;

The ES6 way

import { sdk } from 'symbl-node';

Connect to Endpoints

The code snippet below dials in using PSTN and hangs up after 60 seconds.

const {
  sdk
} = require('symbl-node');

sdk.init({
  appId: 'yourAppId',
  appSecret: 'yourAppSecret',
  basePath: 'https://api.symbl.ai'
}).then(() => {
  sdk.startEndpoint({
    endpoint: {
      type: 'pstn', // This can be pstn or sip
      phoneNumber: '<number_to_call>', // include country code
      dtmf: '<meeting_id>' // if password protected, use "dtmf": "<meeting_id>#,#<password>#"
    }
  }).then(connection => {
    console.log('Successfully connected.', connection.connectionId);

    // Scheduling stop endpoint call after 60 seconds for demonstration purposes
    // In real adoption, sdk.stopEndpoint() should be called when the meeting or call actually ends

    setTimeout(() => {
      sdk.stopEndpoint({
        connectionId: connection.connectionId
      }).then(() => {
        console.log('Stopped the connection');
        console.log('Summary Info:', connection.summaryInfo);
        console.log('Conversation ID:', connection.conversationId);
      }).catch(err => console.error('Error while stopping the connection', err));
    }, 60000);
  }).catch(err => console.error('Error while starting the connection', err));

}).catch(err => console.error('Error in SDK initialization.', err));

We recommend using SIP whenever possible instead of PSTN as it provides higher audio quality options as compared to PSTN. SIP endpoint provides an optional audio configuration as well. Contact us for your specific requirements.

This SDK supports dialing through a simple phone number - PSTN or a Voice Over IP system - SIP endpoint. If you don't have your own voice over IP system, you will want to use a phone number to make the connection.

PSTN (Public Switched Telephone Networks)

The Publicly Switched Telephone Network (PSTN) is the network that carries your calls when you dial in from a landline or cell phone. It refers to the worldwide network of voice-carrying telephone infrastructure, including privately-owned and government-owned infrastructure.

endpoint: {
  type: 'pstn',
  phoneNumber: '14083380682', // Phone number to dial in
  dtmf: '6155774313#' // Joining code
}

SIP (Session Initiation Protocol)

Session Initiation Protocol (SIP) is a standardized communications protocol that has been widely adopted for managing multimedia communication sessions for voice and video calls. SIP may be used to establish connectivity between your communications infrastructures and Symbl's communications platform.

endpoint: {
  type: 'sip',
  uri: 'sip:555@your_sip_domain', // SIP URI to dial in
  audioConfig: { // Optionally any audio configuration
    sampleRate: 16000,
    encoding: 'PCMU',
    sampleSize: '16'
  }
}

Active Speaker Events

The example on the right shows how to connect to a PSTN endpoint, create a speakerEvent instance and push events on connection

const {
  sdk,
  SpeakerEvent
} = require('symbl-node');

sdk.init({
  appId: 'yourAppId',
  appSecret: 'yourAppSecret',
  basePath: 'https://api.symbl.ai'
}).then(() => {
  sdk.startEndpoint({
    endpoint: {
      type: 'pstn',
      phoneNumber: '<number_to_call>', // include country code
      dtmf: '<meeting_id>' // if password protected, use "dtmf": "<meeting_id>#,#<password>#"
    }
  }).then(connection => {
    const connectionId = connection.connectionId;
    console.log('Successfully connected.', connectionId);

    const speakerEvent = new SpeakerEvent();
    speakerEvent.type = SpeakerEvent.types.startedSpeaking;
    speakerEvent.user = {
      userId: 'john@example.com',
      name: 'John'
    };
    speakerEvent.timestamp = new Date().toISOString();

    sdk.pushEventOnConnection(
      connectionId,
      speakerEvent.toJSON(),
      (err) => {
        if (err) {
          console.error('Error during push event.', err);
        } else {
          console.log('Event pushed!');
        }
      }
    );

    // Scheduling stop endpoint call after 60 seconds for demonstration purposes
    // In real adoption, sdk.stopEndpoint() should be called when the meeting or call actually ends

    setTimeout(() => {
      sdk.stopEndpoint({
        connectionId: connection.connectionId,
      }).then(() => {
        console.log('Stopped the connection');
        console.log('Summary Info:', connection.summaryInfo);
        console.log('Conversation ID:', connection.conversationId);
      }).catch(err => console.error('Error while stopping the connection.', err));
    }, 60000);
  }).catch(err => console.error('Error while starting the connection', err));

}).catch(err => console.error('Error in SDK initialization.', err));

NOTE: Setting the timestamp for speakerEvent is optional but it's recommended to provide accurate timestamps in the events when they occurred to get more precision.

Events can be pushed to an on-going connection to have them processed. The code snippet to the right shows a simple example.

Every event must have a type to define the purpose of the event at a more granular level, usually to indicate different activities associated with the event resource. For example - A "speaker" event can have type as started_speaking. An event may have additional fields specific to the event.

Currently, Symbl only supports the speaker event which is described below.

Speaker Event

The speaker event is associated with different individual attendees in the meeting or session. An example of a speaker event is shown below.

In the code example the user needs to have userId field to uniquely identify the user.

Speaker Event has the following types:

started_speaking

This event contains the details of the user who started speaking with the timestamp in ISO 8601 format when he started speaking.

const speakerEvent = new SpeakerEvent({
  type: SpeakerEvent.types.startedSpeaking,
  timestamp: new Date().toISOString(),
  user: {
    userId: 'john@example.com',
    name: 'John'
  }
});

stopped_speaking

This event contains the details of the user who stopped speaking with the timestamp in ISO 8601 format when he stopped speaking.

const speakerEvent = new SpeakerEvent({
  type: SpeakerEvent.types.stoppedSpeaking,
  timestamp: new Date().toISOString(),
  user: {
    userId: 'john@example.com',
    name: 'John'
  }
});

As shown in the above examples, it's ok to reuse the same speakerEvent instance per user, by changing the event's type to optimize by reducing the number of instances for SpeakerEvent.

A startedSpeaking event is pushed on the on-going connection. You can use pushEventOnConnection() method from the SDK to push the events.

Symbl SDK also lets you subscribe to real-time events when you connect to one of the Endpoints specified in the above sections. These include

Real-Time Transcription
Real-Time Insights
Real-Time Messages
Real-Time Intents

The below example shows how to achieve this.

Initialize SDK

const {sdk, SpeakerEvent} = require("symbl-node");

sdk.init({
    // Your appId and appSecret https://platform.symbl.ai
    appId: 'your_appId',
    appSecret: 'your_appSecret'
}).then(async () => {
    console.log('SDK initialized.');
    try {
      // You code goes here.
    } catch (e) {
        console.error(e);
    }
}).catch(err => console.error('Error in SDK initialization.', err));

Add the above lines to import and initialize the SDK. Replace the appId and appSecret in the code. You can find the them by signing up on Symbl Developer Platform

Make a phone call

const connection = await sdk.startEndpoint({
    endpoint: {
        type: 'pstn', // when making a regular phone call
        // Replace this with a real phone number
        phoneNumber: '1XXXXXXXXXX' // include country code
    }
});
const {connectionId} = connection;
console.log('Successfully connected. Connection Id: ', connectionId);

The above snippet makes a phone call, by calling the startEndpoint with type set to pstn and a valid US/Canada Phone Number phoneNumber. You can also call in via type sip as well and the steps below will still remain the same.

// Subscribe to connection using connectionId.
sdk.subscribeToConnection(connectionId, (data) => {
  const {type} = data;
  if (type === 'transcript_response') {
      const {payload} = data;

      // You get live transcription here!!
      process.stdout.write('Live: ' + payload && payload.content + '\r');

  } else if (type === 'message_response') {
      const {messages} = data;

      // You get processed messages in the transcript here!!! Real-time but not live! :)
      messages.forEach(message => {
          process.stdout.write('Message: ' + message.payload.content + '\n');
      });
  } else if (type === 'insight_response') {
      const {insights} = data;
      // See <link here> for more details on Insights
      // You get any insights here!!!
      insights.forEach(insight => {
          process.stdout.write(`Insight: ${insight.type} - ${insight.text} \n\n`);
      });
  }
});

The above snippet calls the subscribeToConnection which requires the connectionId of the call and a callback function to be passed as the second argument which will be invoked when any of the above events are available to be consumed. The data received will contain type of the event. It can be one of transcript_response, message_response, insight_response. Lets go over them one by one.

transcript_response : This contains the real-time transcription data which is availabe as soon as its detected.
message_response : This will contain the array of the transcripts of all the speakers which will be logically separated by punctuations or the speakers if Active Speaker Events are pushed.
insight_response : This will contain the array of all the insights detected in real-time. These can be action-items or questions.

There is also a 4th type of event which is intent_response covered in a separate example.

End the call

// Stop call after 60 seconds to automatically.
setTimeout(async () => {
  const connection = await sdk.stopEndpoint({ connectionId });
  console.log('Stopped the connection');
  console.log('Conversation ID:', connection.conversationId);
}, 60000); // Change the 60000 with higher value if you want this to continue for more time.

To end the call gracefully, we call the stopEndpoint call to stop the call. The code snippet above simply stops the call after 60 seconds.

And we're done! That's how you can consume real-time events using Symbl SDK !

The complete code for the example above can be found here

Language and TimeZone

You can also specify languages other than 'English' from the ones supported to be used for calls made via 'PSTN' or 'SIP'. With this since the call can take place in a different timeZone you can also pass in the timeZone which will be used to render the Summary UI specific to that language and timeZone. The below sub-sections specify how to use these capabilities with Symbl SDK with a complete example.

Utilising other languages

Symbl SDK allows you to work with audio from multiple different languages. The currently supported languages are given below with the regions they are spoken in:

English (United States)
English (United Kingdom)
English (Australia)
French (Canada)
German (Germany)
Italic (Italy)
Dutch (Netherlands)
Japanese (Japan)
Spanish (Latin America)
French (France)

You can pass in the languages array in the startEndpoint as shown in the example linked below. Please note that currently only one language can be specified per call and support for detecting multiple languages in the same call will be added soon.

Specifying Time Zone

With calls taking place in different regions around the world it can get important to capture that information and utilise it. Symbl SDK allows you to pass in the timeZone in the startEndpoint call which will render the Summary UI in that specific Time Zone. A list of all the Time Zones is available here

Passing Language and TimeZone

A complete example showcasing this capability can be found here

Complete Example

const {
  sdk,
  SpeakerEvent
} = require('symbl-node');

sdk.init({
  appId: 'yourAppId',
  appSecret: 'yourAppSecret',
  basePath: 'https://api.symbl.ai'
}).then(() => {

  console.log('SDK Initialized');
  sdk.startEndpoint({
    endpoint: {
      type: 'pstn',
      phoneNumber: '14087407256',
      dtmf: '6327668#'
    }
  }).then(connection => {

    const connectionId = connection.connectionId;
    console.log('Successfully connected.', connectionId);
    const speakerEvent = new SpeakerEvent({
      type: SpeakerEvent.types.startedSpeaking,
      user: {
        userId: 'john@example.com',
        name: 'John'
      }
    });

    setTimeout(() => {
      speakerEvent.timestamp = new Date().toISOString();
      sdk.pushEventOnConnection(
        connectionId,
        speakerEvent.toJSON(),
        (err) => {
          if (err) {
            console.error('Error during push event.', err);
          } else {
            console.log('Event pushed!');
          }
        }
      );
    }, 2000);

    setTimeout(() => {
      speakerEvent.type = SpeakerEvent.types.stoppedSpeaking;
      speakerEvent.timestamp = new Date().toISOString();

      sdk.pushEventOnConnection(
        connectionId,
        speakerEvent.toJSON(),
        (err) => {
          if (err) {
            console.error('Error during push event.', err);
          } else {
            console.log('Event pushed!');
          }
        }
      );
    }, 12000);

    // Scheduling stop endpoint call after 60 seconds
    setTimeout(() => {
      sdk.stopEndpoint({
        connectionId: connection.connectionId
      }).then(() => {
        console.log('Stopped the connection');
        console.log('Summary Info:', connection.summaryInfo);
        console.log('Conversation ID:', connection.conversationId);
      }).catch(err => console.error('Error while stopping the connection.', err));
    }, 90000);

  }).catch(err => console.error('Error while starting the connection', err));

}).catch(err => console.error('Error in SDK initialization.', err));

Below is a quick simulated speaker event example that

Initializes the SDK
Initiates a connection with an endpoint
Sends a speaker event of type startedSpeaking for user John
Sends a speaker event of type stoppedSpeaking for user John
Ends the connection with the endpoint

Strictly for illustration and understanding purposes, the code to the right pushes events by simply using setTimeout() method periodically, but in real usage, they should be pushed as they occur.

const {
  sdk,
  SpeakerEvent
} = require('symbl-node');

sdk.init({
  appId: 'yourAppId',
  appSecret: 'yourAppSecret',
  basePath: 'https://api.symbl.ai'
}).then(() => {
  console.log('SDK Initialized');
  sdk.startEndpoint({
    endpoint: {
      type: 'pstn',
      phoneNumber: '14087407256',
      dtmf: '6327668#'
    },
    actions: [{
      "invokeOn": "stop",
      "name": "sendSummaryEmail",
      "parameters": {
        "emails": [
          "john@exmaple.com",
          "mary@example.com",
          "jennifer@example.com"
        ]
      }
    }],
    data: {
      session: {
        name: 'My Meeting Name' // Title of the Meeting
      },
      users: [{
          user: {
            name: "John",
            userId: "john@example.com",
            role: "organizer"
          }
        },
        {
          user: {
            name: "Mary",
            userId: "mary@example.com"
          }
        },
        {
          user: {
            name: "John",
            userId: "jennifer@example.com"
          }
        }
      ]
    }
  }).then((connection) => {
    console.log('Successfully connected.');

    // Events pushed in between
    setTimeout(() => {
      // After successful stop endpoint, an email with summary will be sent to "john@example.com" and "jane@example.com"
      sdk.stopEndpoint({
        connectionId: connection.connectionId
      }).then(() => {
        console.log('Stopped the connection');
        console.log('Summary Info:', connection.summaryInfo);
        console.log('Conversation ID:', connection.conversationId);
      }).catch(err => console.error('Error while stopping the connection.', err));
    }, 30000);

  }).catch(err => console.error('Error while starting the connection', err));

}).catch(err => console.error('Error in SDK initialization.', err));

Send Summary Email

This is an example of the summary page you can expect to receive at the end of your call

Summary Page

Take a look at the Sample Summary UI which is generated after a meeting is concluded.

Tuning your Summary Page

You can choose to tune your Summary Page with the help of query parameters to play with different configurations and see how the results look.

Query Parameters

You can configure the summary page by passing in the configuration through query parameters in the summary page URL that gets generated at the end of your meeting. See the end of the URL in this example:

https://meetinginsights.symbl.ai/meeting/#/eyJ1...I0Nz?insights.minScore=0.95&topics.orderBy=position

Query Parameter	Default Value	Supported Values	Description
`insights.minScore`	0.8	0.5 to 1.0	Minimum score that the summary page should use to render the insights
`insights.enableAssignee`	false	[true, false]	Enable to disable rending of the assignee and due date of the insight
`insights.enableAddToCalendarSuggestion`	true	[true, false]	Enable to disable add to calendar suggestion when applicable on insights
`insights.enableInsightTitle`	true	[true, false]	Enable or disable the title of an insight. The title indicates the originating person of the insight and if assignee of the insight.
`topics.enabled`	true	[true, false]	Enable or disable the summary topics in the summary page
`topics.orderBy`	'score'	['score', 'position']	Ordering of the topics. score - order topics by the topic importance score. position - order the topics by the position in the transcript they surfaced for the first time

Streaming Audio in Real Time

This section talks about streaming the audio in real-time using the Symbl SDK . We can use this API to pass in audio via a single stream and it also supports sending multiple isolated streams of audio, each of which can contain one or more speaker's audio data.

You can also consume the processed results in real-time, which include

Real Time Transcription
Real Time Insights (Action Items and Questions)
When using multiple audio streams (Each stream for 1 speaker) you also get access to speaker-separated data (including transcription and messages)

Example with Single Stream

The example below utilises mic package to stream audio in real-time. This will be a single stream of audio obtained through mic which may have one or more than one speaker's audio. The link to the complete example below can be found here

Import required packages

const {sdk} = require('symbl-node');
const uuid = require('uuid').v4;
// For demo purposes, we're using mic to simply get audio from microphone and pass it on to websocket connection
const mic = require('mic');

In the above snippet we import the sdk, uuid and mic npm packages. The uuid package is used for generating a unique ID to represent this stream and it's strongly recommended to use it. The mic package is used to obtain the audio stream in real-time to pass to the SDK.

Initialise an instance of mic

const sampleRateHertz = 16000;
const micInstance = mic({
    rate: sampleRateHertz,
    channels: '1',
    debug: false,
    exitOnSilence: 6
});

We now declare the sampleRateHertz variable to specify the Sample Rate of the audio obtained from the mic. It is imperative to use the same Sample Rate used for initialising the mic package and for passing in to the startRealtimeRequest of Symbl SDK as we will see below. Otherwise the transcription will be completely in-accurate.

We also initialise mic with channels: '1' (mono channel) audio as currently only mono channel audio data is supported.

Initialise the Symbl SDK

// Initialize the SDK
await sdk.init({
    appId: '__appId__',
    appSecret: '__appSecret__',
    basePath: 'https://api.symbl.ai'
});
// Need unique Id
const id = uuid();

Next we initialise a helper function to execute our code in the async/await style. The following code snippets (including the one just above) will be a part of the same function. We now initialise the Symbl SDK with the init call, passing in appId and appSecret which you can be obtain by signing up on Symbl Developer Platform We also initialise variable id with uuid function for the unique ID required for this stream as was also mentioned above in the import section snippet.

Call the startRealtimeRequest

// Start Real-time Request (Uses Realtime WebSocket API behind the scenes)
const connection = await sdk.startRealtimeRequest({
    id,
    insightTypes: ["action_item", "question"],
    config: {
        meetingTitle: 'My Test Meeting',
        confidenceThreshold: 0.7,
        timezoneOffset: 480, // Offset in minutes from UTC
        languageCode: "en-US",
        sampleRateHertz
    },
    speaker: { // Optional, if not specified, will simply not send an email in the end.
        userId: 'john.doe@example.com', // Update with valid email
        name: 'John'
    },
    handlers: {
        'onSpeechDetected': (data) => {
            console.log(JSON.stringify(data));
            // For live transcription
            if (data) {
                const {punctuated} = data;
                console.log('Live: ', punctuated && punctuated.transcript);
            }
        },
        'onMessageResponse': (data) => {
            // When a processed message is available
            console.log('onMessageResponse', JSON.stringify(data));
        },
        'onInsightResponse': (data) => {
            // When an insight is detected
            console.log('onInsightResponse', JSON.stringify(data));
        }
    }
});

The next call is made to startRealtimeRequest of the Symbl SDK and includes various parameters passed in. Lets breakdown the configuration and take a look at them one by one.

id: The unique ID that represents this stream. (This needs to be unique, which is why we are using uuid)
insightTypes: This array represents the type of insights that are to be detected. Today the supported ones are action_item and question.
config: This configuration object encapsulates the properties which directly relate to the conversation generated by the audio being passed.

a. meetingTitle : This optional parameter specifies the name of the conversation generated. You can get more info on conversations here

b. confidenceThreshold : This optional parameter specifies the confidence threshold for detecting the insights. Only the insights that have confidenceScore more than this value will be returned.

c. timezoneOffset : This specifies the actual timezoneOffset used for detecting the time/date related entities.

d. languageCode : It specifies the language to be used for transcribing the audio in BCP-47 format. (Needs to be same as the language in which audio is spoken)

e. sampleRateHertz : It specifies the sampleRate for this audio stream.
speaker: Optionally specify the details of the speaker whose data is being passed in the stream. This enables an e-mail with the Summary UI URL to be sent after the end of the stream.
handlers: This object has the callback functions for different events a. onSpeechDetected: To retrieve the real-time transcription results as soon as they are detected. We can use this callback to render live transcription which is specific to the speaker of this audio stream.

b. onMessageResponse: This callback function contains the "finalized" transcription data for this speaker and if used with multiple streams with other speakers this callback would also provide their messages. The "finalized" messages mean that the ASR has finalised the state of this part of transcription and has declared it "final".

c. onInsightResponse: This callback would provide with any of the detected insights in real-time as they are detected. As with the onMessageCallback above this would also return every speaker's insights in case of multiple streams.

Retrieve audio data from mic

console.log('Successfully connected.');

const micInputStream = micInstance.getAudioStream();
micInputStream.on('data', (data) => {
    // Push audio from Microphone to websocket connection
    connection.sendAudio(data);
});

console.log('Started listening to Microphone.');

After the startRealtimeRequest returns successfully, it signifies that the connection has been established successfully with the passed configuration. In the above snippet we now obtain the audio data from the micInputStream and as it's received we relay it to the active connection instance we now have with Symbl SDK .

Stop the stream

setTimeout(async () => {
    // Stop listening to microphone
    micInstance.stop();
    console.log('Stopped listening to Microphone.');
    try {
        // Stop connection
        await conversationData = connection.stop();
        console.log('Conversation ID: ' + conversationData.conversationId);
        console.log('Connection Stopped.');
    } catch (e) {
        console.error('Error while stopping the connection.', e);
    }
}, 60 * 1000); // Stop connection after 1 minute i.e. 60 secs

For the purpose of demoing a continuous audio stream we now simulate a stop on the above stream after 60 seconds. The connection.stop() would close the active connection and will trigger the optional email if the speaker config is included. Here the conversationData variable includes the conversationId you can use with the Conversation API to retrieve this conversation's data.

And that's it! This marks the completion of streaming audio in real-time (Single Audio Stream) with Symbl SDK . The complete code for the example explained above can be found here

With Multiple Streams

The same example explained above can be deployed on multiple machines, each with one speaker to simulate the multiple streams use-case. The only thing common needs to be the unique ID created in the above example which is used to initialize startRealtimeRequest request.

Having this unique ID in common across all different ensures that the audio streams of all the speakers are bound the context of a single conversation. This conversation can be retrieved by the conversationId via the Conversation API which will include the data of all the speakers connecting using the same common ID.

Real Time Telephony API

The Voice API provides the REST interface for adding Symbl to your call and generating actionable insights from your conversations. Telephony API only allows phone numbers in the USA and Canada

POST Telephony

Example API Call

curl -k -X POST "https://api.symbl.ai/v1/endpoint:connect" \
     -H "accept: application/json" \
     -H "Content-Type: application/json" \
     -H "x-api-key: <your_auth_token>" \
     -d @location_of_fileName_with_request_payload

const request = require('request');

const payload = {
  "operation": "start",
  "endpoint": {
    "type" : "pstn",
    "phoneNumber": "<number_to_call>", // include country code
    "dtmf": "<meeting_id>" // if password protected, use "dtmf": "<meeting_id>#,#<password>#"
  },
  "actions": [{
    "invokeOn": "stop",
    "name": "sendSummaryEmail",
    "parameters": {
      "emails": [
        "joe.symbl@example.com"
      ]
    }
  }],
  "data" : {
      "session": {
          "name" : "My Meeting"
      }
  }
}

const your_auth_token = "<your_auth_token>";

request.post({
    url: 'https://api.symbl.ai/v1/endpoint:connect',
    headers: {'x-api-key': your_auth_token},
    body: payload,
    json: true
}, (err, response, body) => {
  console.log(body);
});

The above command returns an object structured like this:

{
    "eventUrl": "https://api.symbl.ai/v1/event/771a8757-eff8-4b6c-97cd-64132a7bfc6e",
    "resultWebSocketUrl": "wss://api.symbl.ai/events/771a8757-eff8-4b6c-97cd-64132a7bfc6e",
    "connectionId": "771a8757-eff8-4b6c-97cd-64132a7bfc6e",
    "conversationId": "51356232423"
}

The Telephony Voice API allows you to easily use Symbl's Language Insights capabilities.

It exposes the functionality of Symbl to dial-in to the conference. Supported endpoints are given below. Additionally, events can be passed for further processing. The supported types of events are discussed in detail in the section below.

Symbl will connect to your SIP or PSTN exposed endpoints. The Audio stream will be fetched from the SIP or PSTN connection once connected.

HTTP REQUEST

POST https://api.symbl.ai/v1/endpoint:connect

Request Parameters

Parameter	Type	Description
`operation`	string	enum([start, stop]) - Start or Stop connection
`endpoint`	object	Object containing `Type` of the session - either `pstn or sip`, `phoneNumber` which is the meeting number symbl should call with country code prepended and `dtmf` which is the conference passcode.
`actions`	list	actions that should be performed while this connection is active. Currently only one action is supported - `sendSummaryEmail`
`data`	object	Object containing a `session` object which has a field `name` corresponding to the name of the meeting

Response Object

Field	Description
`eventUrl`	REST API to push speaker events as the conversation is in progress, to add additional speaker context in the conversation. Example - In an on-going meeting, you can push speaker events
`resultWebSocketUrl`	Same as eventUrl but over WebSocket. The latency of events is lower with a dedicated WebSocket connection.
`connectionId`	Ephemeral connection identifier of the request, to uniquely identify the telephony connection. Once the connection is stopped using “stop” operation, or is closed due to some other reason, the connectionId is no longer valid
`conversationId`	Represents the conversation - this is the ID that needs to be used in conversation api to access the conversation

To play around with a few examples, we recommend a REST client called Postman. Simply tap the button below to import a pre-made collection of examples.

Run in Postman

Try it out

When you have started the connection through the API, try speaking the following sentences and view the summary email that gets generated:

"Hey, it was nice meeting you yesterday. Let's catch up again next week over coffee sometime in the evening. I would love to discuss the next steps in our strategic roadmap with you."
"I will set up a meeting with Roger to discuss our budget plan for the next quarter and then plan out how much we can set aside for marketing efforts. I also need to sit down with Jess to talk about the status of the current project. I'll set up a meeting with her probably tomorrow before our standup."

Specifying Language and Timezone

Language

The Localized Summary UI provides users a translated meeting summary page chosen based on one of eight currently supported languages chosen when a session is initiated.

Language is specified by passing an array of language codes to the lanaguage parameter when making a call to the startEndpoint. Although the languages parameter takes an array of language codes, only the first language code in the array is used to specify the session’s language.

Supported Languages

Currently 8 language codes are supported stemming from 6 language locales.

Language	Language Code
English(US)	en-US
English(UK)	en-UK
English(Australia)	en-AU
French(Canada)	fr-CA
French(France)	fr-FR
Spanish(US)	es-us
Italian(Italy)	it-IT
Jaspanese(Japan)	ja-JP
Dutch(Netherlands)	nl-NL

Example

Below is an example of specifying that the meeting’s language should be in Spanish.

sdk.startEndpoint({
    endpoint: {
        audioConfig: {
            encoding: 'OPUS',
            sampleRate: 16000
        },
        type: 'sip',
        uri: 'sip:1_firebird_1367062@am-fs-lo-05.anymeeting.com'
    },
    languages: ['es-US']
});
}

Timezone

Specifying a timezone when initiating a session will result in the Summary UI displaying the meeting start time for that given region.

curl --location --request POST 'https://api.symbl.ai/v1/endpoint:connect' \
--header 'x-api-key: <access_token>' \
--header 'Content-Type: application/json' \
--data-raw '{
    "operation": "start",
    "endpoint": {
      "type" : "pstn",
      "phoneNumber": "__phone_number__",
      "dtmf": "__dtmf_sequence__"
    },
    "actions": [{
      "invokeOn": "stop",
      "name": "sendSummaryEmail",
      "parameters": {
        "emails": [
          "__email_address__"
        ]
      }
    }],
    "data" : {
        "session": {
            "name" : "__name_of_this_call__"
        }
    },
    "languages": ["it-IT"]
}'

Example

Below is an example of setting the timezone to the Pacific Timezone by passing the TZ database name to the timezone parameter while initiating a session:

sdk.startEndpoint({
    endpoint: {
        audioConfig: {
            encoding: 'OPUS',
            sampleRate: 16000
        },
        type: 'sip',
        uri: 'sip:1_firebird_1367062@am-fs-lo-05.anymeeting.com'
    },
    timezone: ' US/Pacific'
});

Full Code Example

To specify a language and timezone while initiating a meeting, two parameters required - languages and timezone.

The following code snippet would initiate a session with Japanese as the language and JST as the timezone.

sdk.startEndpoint({
    endpoint: {
        audioConfig: {
            encoding: 'OPUS',
            sampleRate: 16000
        },
        type: 'sip',
        uri: 'sip:1_firebird_1367062@am-fs-lo-05.anymeeting.com'
    },
    languages: ['ja-JP'],
    timezone: 'Asia/Tokyo'
});

Real Time WebSocket API

In the example below, we've used the websocket npm package for WebSocket Client, and mic for getting the raw audio from microphone.

$ npm i websocket mic

For this example, we are using your mic to stream audio data. You will most likely want to use other inbound sources for this

const WebSocketClient = require('websocket').client;

const mic = require('mic');

const micInstance = mic({
  rate: '44100',
  channels: '1',
  debug: false,
  exitOnSilence: 6
});

// Get input stream from the microphone
const micInputStream = micInstance.getAudioStream();
let connection = undefined;

Create a websocket client instance

const ws = new WebSocketClient();

ws.on('connectFailed', (err) => {
  console.error('Connection Failed.', err);
});

ws.on('connect', (connection) => {

  // Start the microphone
  micInstance.start();

  connection.on('close', () => {
    console.log('WebSocket closed.')
  });

  connection.on('error', (err) => {
    console.log('WebSocket error.', err)
  });

  connection.on('message', (data) => {
    if (data.type === 'utf8') {
      const {
        utf8Data
      } = data;
    console.log(utf8Data);  // Print the data for illustration purposes
    }
  });

  console.log('Connection established.');

  connection.send(JSON.stringify({
    "type": "start_request",
    "insightTypes": ["question", "action_item"],
    "config": {
      "confidenceThreshold": 0.9,
      // "timezoneOffset": 480, // Your timezone offset from UTC in minutes
      "languageCode": "en-US",
      "speechRecognition": {
        "encoding": "LINEAR16",
        "sampleRateHertz": 44100 // Make sure the correct sample rate is provided for best results
      },
      "meetingTitle": "Client Meeting"
    },
    "speaker": {
      "userId": "jane.doe@example.com",
      "name": "Jane"
    }
  }));

  micInputStream.on('data', (data) => {
    connection.send(data);
  });
});

For this example, we timeout our call after 2 minutes but you would most likely want to make the stop_request call when your websocket connection ends

  // Schedule the stop of the client after 2 minutes (120 sec)

  setTimeout(() => {
    micInstance.stop();
    // Send stop request
    connection.sendUTF(JSON.stringify({
      "type": "stop_request"
    }));
    connection.close();
  }, 120000);

Generate the token and replace it in the placeholder <accessToken>. Once the code is running, start speaking and you should see the message_response and insight_response messages getting printed on the console.

ws.connect(
  'wss://api.symbl.ai/v1/realtime/insights/1',
  null,
  null,
  { 'X-API-KEY': '<accessToken>' }
);

In the example below, we've used a Websocket that is compatible with most browsers and doesn't need any additional npm packages. Generate the token and replace it in the placeholder <accessToken>. Then, create the Websocket.

let url = `wss://api.symbl.ai/v1/realtime/insights/1?access_token=${'<accessToken>'}`
let ws = new WebSocket(url);

ws.onerror = (err) => {
  console.error('Connection Failed.', err);
};
ws.onopen = () => {
  console.log('Websocket open.')

  ws.onmessage = (event) => {
    if (event.type === 'message') {
      console.log(event.data);  // Print the data for illustration purposes
    }
  };

  ws.onclose = () => {
    console.log('WebSocket closed.')
  };

  ws.onerror = (err) => {
    console.log('WebSocket error.', err)
  };

  console.log('Connection established.');

  ws.send(JSON.stringify({
    "type": "start_request",
    "insightTypes": ["question", "action_item"],
    "config": {
      "confidenceThreshold": 0.9,
      // "timezoneOffset": 480, // Your timezone offset from UTC in minutes
      "languageCode": "en-US",
      "speechRecognition": {
        "encoding": "LINEAR16",
        "sampleRateHertz": 44100 // Make sure the correct sample rate is provided for best results
      },
      "meetingTitle": "Client Meeting"
    },
    "speaker": {
      "userId": "jane.doe@example.com",
      "name": "Jane"
    }
  }));
}

To get direct access to the mic, we're going to use an API in the WebRTC specification called getUserMedia().

Once the code is running, start speaking and you should see the message_response and insight_response messages getting printed on the console.

  const handleSuccess = function(stream) {
    const context = new AudioContext();
    const source = context.createMediaStreamSource(stream);
    const processor = context.createScriptProcessor(1024, 1, 1);
    source.connect(processor);
    processor.connect(context.destination);
    processor.onaudioprocess = function(e) {
      // convert to 16-bit payload
      const inputData = e.inputBuffer.getChannelData(0) || new Float32Array(this.options.bufferSize);
      const targetBuffer = new Int16Array(inputData.length);
      for (let index = inputData.length; index > 0; index--)
          targetBuffer[index] = 32767 * Math.min(1, inputData[index]);
      // Send to websocket
      if(ws.readyState === WebSocket.OPEN){
          ws.send(targetBuffer.buffer);
      }
    };
  };

  navigator.mediaDevices.getUserMedia({ audio: true, video: false })
    .then(handleSuccess);

  // Schedule the stop of the client after 2 minutes (120 sec)
  setTimeout(() => {
    // Send stop request
    ws.send(JSON.stringify({
      "type": "stop_request"
    }));
    ws.close();
  }, 120000);

Introduction

The WebSocket based real-time API by Symbl provides the direct, fastest and most accurate of all other interfaces to push the audio stream in real-time, and get the results back as soon as they're available.

Learn more about how to get real-time transcriptions with native WebSocket API.

Connection Establishment

This is a WebSocket endpoint, and hence it starts as an HTTP request that contains HTTP headers that indicate the client's desire to upgrade the connection to a WebSocket instead of using HTTP semantics. The server indicates its willingness to participate in the WebSocket connection by returning an HTTP 101 Switching Protocols response. After the exchange of this handshake, both client and service keep the socket open and begin using a message-based protocol to send and receive information. Please refer to WebSocket Specification RFC 6455 for the more in-depth understanding of the Handshake process.

Message Formats

Client and Server both can send messages after the connection is established. According to RFC 6455, WebSocket messages can have either a text or a binary encoding. The two encodings use different on-the-wire formats. Each format is optimized for efficient encoding, transmission, and decoding of the message payload.

Text Message

Text message over WebSocket must use UTF-8 encoding. Text Message is the serialized JSON message. Every text message has a type field to specify the type or the purpose of the message.

Binary Message

Binary WebSocket messages carry a binary payload. For the Real-time API, audio is transmitted to the service by using binary messages. All other messages are the Text messages.

Client Messages

This section describes the messages that originate from the client and are sent to service. The types of messages sent by the client are start_request, stop_request and binary messages containing audio.

Configuration

Main Message Body

Field	Required	Supported Values	Description
`type`	true	start_request, stop_request	Type of message
`insightTypes`	false	action_item, question	Types of insights to return. If not provided, no insights will be returned.
`config`	false		Configuration for this request. See the config section below for more details.
`speaker`	false		Speaker identity to use for audio in this WebSocket connection. If omitted, no speaker identification will be used for processing. See below.

config

Field	Required	Supported Values	Default Value	Description
`confidenceThreshold`	false	0.0 - 1.0	0.5	Minimum Confidence score that should be met for API to consider it as valid insight, if not provided defaults to 0.5 i.e. 50% or more
`languageCode`	false		en-US	The language code as per the BCP 47 specification
`speechRecognition`	false			Speaker identity to use for audio in this WebSocket connection. If omitted, no speaker identification will be used for processing. See below.

speechRecognition

Field	Required	Supported Values	Default Value	Description
`encoding`	false	LINEAR16, FLAC, MULAW	LINEAR16	Audio Encoding in which the audio will be sent over the WebSocket.
`sampleRateHertz`	false		16000	The rate of the incoming audio stream.

speaker

Field	Required	Description
`userId`	false	Any user identifier for the user.
`name`	false	Display name of the user.

Messages

Start Request

{
  "type": "start_request",
  "insightTypes": ["question", "action_item"],
  "config": {
    "confidenceThreshold": 0.9,
    "languageCode": "en-US",
    "speechRecognition": {
      "encoding": "LINEAR16",
      "sampleRateHertz": 16000
    }
  },
  "speaker": {
    "userId": "jane.doe@example.com",
    "name": "Jane"
  }
}

This is a request to start the processing after the connection is established. Right after this message has been sent, the audio should be streamed, any binary audio streamed before the receipt of this message will be ignored.

Stop Request

{
  "type": "stop_request"
}

This is a request to stop the processing. After the receipt of this message, the service will stop any processing and close the WebSocket connection.

Example of the message_response object

{
  "type": "message_response",
  "messages": [
    {
      "from": {
        "name": "Jane",
        "userId": "jane.doe@example.com"
      },
      "payload": {
        "content": "I was very impressed by your profile, and I am excited to know more about you.",
        "contentType": "text/plain"
      }
    },
    {
      "from": {
        "name": "Jane",
        "userId": "jane.doe@example.com"
      },
      "payload": {
        "content": "So tell me, what is the most important quality that you acquired over all of your professional career?",
        "contentType": "text/plain"
      }
    }
  ]
}

Sending Binary Messages with Audio

The client needs to send the audio to Service by converting the audio stream into a series of audio chunks. Each chunk of audio carries a segment of audio that needs to be processed. The maximum size of a single audio chunk is 8,192 bytes.

Service Messages

This section describes the messages that originate in Service and are sent to the client.

Service sends mainly two types of messages (message_response, insight_response) to the client as soon as they're available.

Message Response

The message_response contains the processed messages as soon as they're ready and available, in the processing of continuous audio stream. This message does not contain any insights.

Insight Response

Example of the insight_response object

{
  "type": "insight_response",
  "insights": [
    {
      "type": "question",
      "text": "So tell me, what is the most important quality that you acquired over all of your professional career?",
      "confidence": 0.9997962117195129,
      "hints": [],
      "tags": []
    },
    {
      "type": "action_item",
      "text": "Jane will look into the requirements on the hiring for coming financial year.",
      "confidence": 0.9972074778643447,
      "hints": [],
      "tags": [
        {
          "type": "person",
          "text": "Jane",
          "beginOffset": 0,
          "value": {
            "value": {
              "name": "Jane",
              "alias": "Jane",
              "userId": "jane.doe@symbl.ai"
            }
          }
        }
      ]
    }
  ]
}

The insight_response contains the insights from the ongoing conversation as soon as they are available. This message does not contain any messages.

Async API

The Async API provides a REST interface to allow you to run a job asynchronously in order to process insights out of audio and text files.

Text API

POST Async Text API

Example API call

curl --location --request POST 'https://api.symbl.ai/v1/process/text' \
--header 'x-api-key: <generated_valid_token>' \
--header 'Content-Type: application/json' \
--data-raw '{
  "name": "Your meeting name",
  "metadata" : {  "conversationType": ["sales"] },
  "entities": [{"customType": "Hiring Process", "text": "internships"}],
  "detectActionPhraseForMessages":true,
  "messages": [
    {
      "payload": {
        "content": "Hi Mike, Natalia here. Hope you don’t mind me reaching out. Who would be the best possible person to discuss internships and student recruitment at ABC Corp? Would you mind pointing me toward the right person and the best way to reach them? Thanks in advance for your help, I really appreciate it!"
      },
      "from": {
        "userId": "natalia@example.com",
        "name": "Natalia"
      },
        "duration":{
            "startTime":"2020-07-21T16:02:19.01Z",
            "endTime":"2020-07-21T16:04:19.99Z"
      }
    },
    {
      "payload": {
        "content": "Hey Natalia, thanks for reaching out. I am connecting you with Steve who handles recruitements for us."
      },
      "from": {
        "userId": "mike@abccorp.com",
        "name": "Mike"
      },
        "duration":{
            "startTime":"2020-07-21T16:04:19.99Z",
            "endTime":"2020-07-21T16:04:20.99Z"
      }
    }
  ]
}'

const request = require('request');

const options = {
  'method': 'POST',
  'url': 'https://api.symbl.ai/v1/process/text',
  'headers': {
    'Content-Type': 'application/json',
    'x-api-key': '<your_auth_token>'
  },
  body: JSON.stringify({
    "name": "Your meeting name",
    "metadata" : {  "conversationType": ["sales"] },
    "entities": [{"customType": "Hiring Process", "text": "internships"}],
    "detectActionPhraseForMessages":true,
    "messages": [
      {
        "payload": {
          "content": "Hi Mike, Natalia here. Hope you don’t mind me reaching out. Who would be the best possible person to discuss internships and student recruitment at ABC Corp? Would you mind pointing me toward the right person and the best way to reach them? Thanks in advance for your help, I really appreciate it!"
        },
        "from": {
          "userId": "natalia@example.com",
          "name": "Natalia"
        },
          "duration":{
              "startTime":"2020-07-21T16:02:19.01Z",
              "endTime":"2020-07-21T16:04:19.99Z"
        }
      },
      {
        "payload": {
          "content": "Hey Natalia, thanks for reaching out. I am connecting you with Steve who handles recruitements for us."
        },
        "from": {
          "userId": "mike@abccorp.com",
          "name": "Mike"
        },
          "duration":{
              "startTime":"2020-07-21T16:04:19.99Z",
              "endTime":"2020-07-21T16:04:20.99Z"
        }
      }
    ],
    "confidenceThreshold": 0.5
  })
};

request(options, function (error, response) {
  if (error) throw new Error(error);
  console.log(response.body);
});

The above request returns a response structured like this:

{
  "conversationId": "5815170693595136",
  "jobId": "9b1deb4d-3b7d-4bad-9bdd-2b0d7b3dcb6d"
}

HTTP REQUEST

POST https://api.symbl.ai/v1/process/text

Request Headers

Header Name	Required	Value
`x-api-key`	Yes	your_auth_token
`Content-Type`	Yes	`application/json`

Request Body

Field	Required	Type	Default	Description
`name`	No	string	`conversationId`	Your meeting name.
`messages`	Yes	list		Input Messages to look for insights. See the messages section below for more details.
`confidenceThreshold`	No	double	0.5	Minimum required confidence for the insight to be recognized. Range is between 0.0 to 1.0
`detectActionPhraseForMessages`	No	boolean	false	When `true` is passed it shows Actionable Phrases in each sentence of conversation. These sentences can be found in the Conversation's Messages API.
`entities`	No	list		Input custom entities which can be detected in conversation using Entities' API. For examples, check the sample code on the right.
`metadata`	No	JSON		By providing details about your conversation in `metadata` our algorithm optimizes data discovery. Currently, `metadata` supports `conversationType` which optimizes action-items, for more details see the metadata section below.

messages

Field	Required	Type	Description
`payload`	Yes	object	Input Messages to look for insights. See the messages section below for more details.
`from`	No	object	Information about the User information produced the content of this message.
`duration`	No	object	Duration object containing startTime and endTime for the transcript.

payload

Field	Required	Type	Default	Description
`contentType`	No	string (MIME type)	text/plain	To indicate the type and/or format of the content. Please see RFC 6838 for more details. Currently only text/plain is supported.
`from`	No	object		The content of the message in the specified MIME type in the contentType field.

from (user)

Field	Required	Type	Description
`name`	No	string	Name of the user.
`userId`	No	string	A unique identifier of the user. E-mail ID is usually a preferred identifier for the user.

duration

Field	Required	Type	Description
`startTime`	No	DateTime	The start time for the particular text content.
`endTime`	No	DateTime	The start time for the particular text content.

metadata

Field	Required	Type	Description
`conversationType`	No	list	Currently, we only support `['sales']`, this optimizes Symbl algorithm to detect sales specific action-items in a meeting. To know more when to use this conversationType, click here.

Query Params

Parameter	Required	Value
`webhookUrl`	No	Webhook url on which job updates to be sent. (This should be post API)

WebhookUrl will be used to send the status of job created for uploaded file. Every time the status of the job changes it will be notified on the WebhookUrl

Response Object

Field	Description
`conversationId`	ID to be used with Conversation API
`jobId`	ID to be used with Job API

Response on reaching limit

Field	Description
`Payload`	`{ "message" : "This API has a limit of maximum of 5 number of concurrent jobs per account. If you are looking to scale, and need more concurrent jobs than this limit, please contact us at support@symbl.ai" }`
`Header`	`{ "statusCode" : 429 }`

Webhook Payload

Field	Description
`jobId`	ID to be used with Job API
`status`	Current status of the job. (Valid statuses - [ `scheduled`, `in_progress`, `completed` ])

PUT Async Text API

The Async Text API allows you to process any text payload to append the transcription of the previous conversation and get updated conversational insights. It can be useful in any use case where you have access to the textual content of a type of conversation, and you want to extract the insightful items supported by the Conversation API.

Use the POST API to upload your content and generate a Conversation ID. If you want to append additional content to the same Conversation ID, use the PUT API.

Example API call

curl --location --request POST 'https://api.symbl.ai/v1/process/text/:conversationId' \
--header 'x-api-key: <generated_valid_token>' \
--header 'Content-Type: application/json' \
--data-raw '{
  "name": "Your meeting name",
  "metadata" : {  "conversationType": ["sales"] },
  "entities": [{"customType": "Hiring Process", "text": "internships"}],
  "detectActionPhraseForMessages":true,
  "messages": [
    {
      "payload": {
        "content": "Hi Mike, Natalia here. Hope you don’t mind me reaching out. Who would be the best possible person to discuss internships and student recruitment at ABC Corp? Would you mind pointing me toward the right person and the best way to reach them? Thanks in advance for your help, I really appreciate it!"
      },
      "from": {
        "userId": "natalia@example.com",
        "name": "Natalia"
      },
        "duration":{
            "startTime":"2020-07-21T16:02:19.01Z",
            "endTime":"2020-07-21T16:04:19.99Z"
      }
    },
    {
      "payload": {
        "content": "Hey Natalia, thanks for reaching out. I am connecting you with Steve who handles recruitements for us."
      },
      "from": {
        "userId": "mike@abccorp.com",
        "name": "Mike"
      },
        "duration":{
            "startTime":"2020-07-21T16:04:19.99Z",
            "endTime":"2020-07-21T16:04:20.99Z"
      }
    }
  ]
}'

const request = require('request');

const options = {
  'method': 'PUT',
  'url': 'https://api.symbl.ai/v1/process/text/' + your_conversationId,
  'headers': {
    'Content-Type': 'application/json',
    'x-api-key': '<your_auth_token>'
  },
  'body': JSON.stringify({
    "name": "Your meeting name",
    "metadata" : {  "conversationType": ["sales"] },
    "entities": [{"customType": "Hiring Process", "text": "internships"}],
    "detectActionPhraseForMessages":true,
    "messages": [
      {
        "payload": {
          "content": "Hi Mike, Natalia here. Hope you don’t mind me reaching out. Who would be the best possible person to discuss internships and student recruitment at ABC Corp? Would you mind pointing me toward the right person and the best way to reach them? Thanks in advance for your help, I really appreciate it!"
        },
        "from": {
          "userId": "natalia@example.com",
          "name": "Natalia"
        },
          "duration":{
              "startTime":"2020-07-21T16:02:19.01Z",
              "endTime":"2020-07-21T16:04:19.99Z"
        }
      },
      {
        "payload": {
          "content": "Hey Natalia, thanks for reaching out. I am connecting you with Steve who handles recruitements for us."
        },
        "from": {
          "userId": "mike@abccorp.com",
          "name": "Mike"
        },
          "duration":{
              "startTime":"2020-07-21T16:04:19.99Z",
              "endTime":"2020-07-21T16:04:20.99Z"
        }
      }
    ],
    "confidenceThreshold": 0.5
  })
};

request(options, function (error, response) {
  if (error) throw new Error(error);
  console.log(response.body);
});

The above request returns a response structured like this:

{
  "conversationId": "5815170693595136",
  "jobId": "9b1deb4d-3b7d-4bad-9bdd-2b0d7b3dcb6d"
}

HTTP REQUEST

PUT https://api.symbl.ai/v1/process/text/:conversationId

Request Headers

Header Name	Required	Value
`x-api-key`	Yes	your_auth_token
`Content-Type`	Yes	`application/json`

Path Params

Parameter	Value
`conversationId`	ConversationId got from the POST Async API for the text content

Request Body

Field	Required	Type	Default	Description
`name`	No	string	`conversationId`	Your meeting name.
`messages`	Yes	list		Input Messages to look for insights. See the messages section below for more details.
`confidenceThreshold`	No	double	0.5	Minimum required confidence for the insight to be recognized. Range is from 0.0 to 1.0.
`detectActionPhraseForMessages`	No	boolean	false	It shows Actionable Phrases in each sentence of conversation. Values supported are `true` and `false`. These sentences can be found in the Conversation's Messages API.
`entities`	No	list		Input custom entities which can be detected in your conversation using Entities' API. For example, check the sample code on the right.
`metadata`	No	JSON		By providing details about your conversation in `metadata` our algorithm optimizes data discovery. Currently, `metadata` supports `conversationType` which optimizes action-items, for more details see the metadata section below.

messages

Field	Required	Type	Description
`payload`	Yes	object	Input Messages to look for insights. See the messages section below for more details.
`from`	No	object	Information about the User information produced the content of this message.
`duration`	No	object	Duration object containing startTime and endTime for the transcript.

payload

Field	Required	Type	Default	Description
`contentType`	No	string (MIME type)	text/plain	To indicate the type and/or format of the content. Please see RFC 6838 for more details. Currently only text/plain is supported.
`from`	No	object		The content of the message in the specified MIME type in the contentType field.

from (user)

Field	Required	Type	Description
`name`	No	string	Name of the user.
`userId`	No	string	A unique identifier of the user. E-mail ID is usually a preferred identifier for the user.

duration

Field	Required	Type	Description
`startTime`	No	DateTime	The start time for the particular text content.
`endTime`	No	DateTime	The start time for the particular text content.

metadata

Field	Required	Type	Description
`conversationType`	No	list	Currently, we only support `['sales']`, this optimizes Symbl algorithm to detect specific action-items present in sales meetings. To know more when to use this conversationType, click here.

Query Params

Parameter	Required	Value
`webhookUrl`	No	Webhook url on which job updates to be sent. (This should be post API)

WebhookUrl will be used to send the status of job created for uploaded file. Every time the status of the job changes it will be notified on the WebhookUrl

Response Object

Field	Description
`conversationId`	ID to be used with Conversation API
`jobId`	ID to be used with Job API

Webhook Payload

Field	Description
`jobId`	ID to be used with Job API
`status`	Current status of the job. (Valid statuses - [ `scheduled`, `in_progress`, `completed`, `failed` ])

Audio API

Table of content: Async Audio API

POST Async Audio API
PUT Async Audio API
POST Async Audio URL API
PUT Async Audio URL API
Speaker Diarization

System requirement: NodeJS 7+

POST Async Audio API

The Async Audio API allows you to process an audio file and return the full text transcript along with conversational insights. It can be utilized for any use case where you have access to recorded audio and want to extract insights and other conversational attributes supported by Symbl's Conversation API.

Use the POST API to upload your file and generate a Conversation ID. If you want to append additional audio information to the same Conversation ID, use the PUT API.

Example API call - The sample request accepts just the raw audio file from the data with the MIME typeset in the Content-Type Header. The audio file should only have Mono Channel.

# Wave file
curl --location --request POST 'https://api.symbl.ai/v1/process/audio?name="Your meeting name"&webhookUrl=<your_webhook_url>&entities= [{"customType": "Custom_Entity_Type", "text": "Custom Entity to be searched in transcript"}]' \
--header 'Content-Type: audio/wav' \
--header 'x-api-key: <generated_valid_token>' \
--data-binary '@/file/location/audio.wav'

# MP3 File
curl --location --request POST 'https://api.symbl.ai/v1/process/audio?name="Your meeting name"&webhookUrl=<your_webhook_url>&entities= [{"customType": "Custom_Entity_Type", "text": "Custom Entity to be searched in transcript"}]' \
--header 'Content-Type: audio/mpeg' \
--header 'x-api-key: <generated_valid_token>' \
--data-binary '@/file/location/audio.mp3'

const request = require('request');
const fs = require('fs');
const accessToken = "<your_auth_token>";
const audioFileStream = fs.createReadStream('/file/location/audio.wav');

const audioOption = {
  url: 'https://api.symbl.ai/v1/process/audio',
  headers: {
    'x-api-key': accessToken,
    'Content-Type': 'audio/wav'
   },
  qs: {
    'name':'Your meeting name',
    'webhookUrl': 'https://your_webhook_url',
    'entitie's: [{'customType': 'Custom_Entity_Type', 'text': 'Custom Entity to be searched in transcript'}]
  },
  json: true,
};

audioFileStream.pipe(request.post(audioOption, (err, response, body) => {
  console.log(err, body);
}));

The above request returns a response structured like this:

{
  "conversationId": "5815170693595136",
  "jobId": "9b1deb4d-3b7d-4bad-9bdd-2b0d7b3dcb6d"
}

HTTP REQUEST

POST https://api.symbl.ai/v1/process/audio

Request Headers

Header Name	Required	Value
`x-api-key`	Yes	your_auth_token
`Content-Type`	No	Describes the format and codec of the provided audio data. Accepted values are `audio/wav`, `audio/mpeg`, `audio/mp3` and `audio/wave`.

** Content-Type: This field is optional.
If you're not sure about audio format, you can omit it since the API will automatically detect the content type.
But when it's mentioned, audio format is validated.

Query Params

Parameter	Required	Value
`name`	No	Your meeting name. (When not specified, it will default to the `conversationId`.)
`webhookUrl`	No	Webhook url on which job updates to be sent. (This should be post API)
`customVocabulary`	No	Contains a list of words and phrases that provide hints to the speech recognition task.
`entities`	No	Input custom entities which can be detected in your conversation using Entities' API. For example, check the sample code on right.
`detectActionPhraseForMessages`	No	Accepted values are `true` & `false`. It shows Actionable Phrases in each sentence of conversation. These sentences can be found in the Conversation's Messages API.
`conversationType`	No	Currently, we only support `['sales']`, this optimizes Symbl algorithm to detect sales specific action-items present in a meeting. To know more when to use this conversationType, click here.

WebhookUrl will be used to send the status of job created for uploaded audio. Every time the status of the job changes it will be notified on the WebhookUrl

Response Object on Success

Field	Description
`conversationId`	ID to be used with Conversation API
`jobId`	ID to be used with Job API

Response on reaching limit

Field	Description
`Payload`	`{ "message" : "This API has a limit of maximum of 5 number of concurrent jobs per account. If you are looking to scale, and need more concurrent jobs than this limit, please contact us at support@symbl.ai" }`
`Header`	`{ "statusCode" : 429 }`

Webhook Payload

Field	Description
`jobId`	ID to be used with Job API
`status`	Current status of the job. (Valid statuses - [ `scheduled`, `in_progress`, `completed`, `failed` ])

PUT Async Audio API

The Async Audio API allows you to process an additional audio file to the previous conversation, append the transcription and get conversational insights for updated conversation. It can be useful in any use case where you have access to multiple audio files of any type of conversation, and you want to extract the insightful items supported by the Conversation API.

Use the POST API to upload your file and generate a Conversation ID. If you want to append additional audio information to the same Conversation ID, use the PUT API.

Example API call - The sample request accepts just the raw audio file from the data with the MIME typeset in the Content-Type Header. The audio file should only have Mono Channel.

# Wave file
curl --location --request PUT 'https://api.symbl.ai/v1/process/audio/:conversationId?name="Your meeting name"&webhookUrl=<your_webhook_url>&entities= [{"customType": "Custom_Entity_Type", "text": "Custom Entity to be searched in transcript"}]' \
--header 'Content-Type: audio/wav' \
--header 'x-api-key: <generated_valid_token>' \
--data-binary '@/file/location/audio.wav'

# MP3 File
curl --location --request PUT 'https://api.symbl.ai/v1/process/audio/:conversationId?name="Your meeting name"&webhookUrl=<your_webhook_url>&entities= [{"customType": "Custom_Entity_Type", "text": "Custom Entity to be searched in transcript"}]' \
--header 'Content-Type: audio/mpeg' \
--header 'x-api-key: <generated_valid_token>' \
--data-binary '@/file/location/audio.mp3'

const request = require('request');
const fs = require('fs');

const accessToken = "<your_auth_token>";
const audioFileStream = fs.createReadStream('/file/location/audio.wav');

const audioOption = {
  url: 'https://api.symbl.ai/v1/process/audio' + your_conversationId,
  headers: {
    'x-api-key': accessToken,
    'Content-Type': 'audio/wav'
   },
  qs: {
    'name':'Your meeting name',
    'webhookUrl': 'https://your_webhook_url',
    'entities': [{'customType': 'Custom_Entity_Type', 'text': 'Custom Entity to be searched in transcript'}]
  },
  json: true,
};

audioFileStream.pipe(request.put(audioOption, (err, response, body) => {
  console.log(err, body);
}));

The above request returns a response structured like this:

{
  "conversationId": "5815170693595136",
  "jobId": "9b1deb4d-3b7d-4bad-9bdd-2b0d7b3dcb6d"
}

HTTP REQUEST

PUT https://api.symbl.ai/v1/process/audio/:conversationId

Request Headers

Header Name	Required	Value
`x-api-key`	Yes	your_auth_token
`Content-Type`	No	Describes the format and codec of the provided audio data. Accepted values are `audio/wav`, `audio/mpeg`, `audio/mp3` and `audio/wave`.

Path Params

Parameter	value
`conversationId`	conversationId which is provided by the first request submitted using POST async audio API

Query Params

Parameter	Required	Value
`name`	No	Your meeting name. (When not specified, it will default to the `conversationId`.)
`webhookUrl`	No	Webhook url on which job updates to be sent. (This should be post API)
`customVocabulary`	No	Contains a list of words and phrases that provide hints to the speech recognition task.
`detectActionPhraseForMessages`	No	Accepted values are `true` & `false`. It shows Actionable Phrases in each sentence of conversation. These sentences can be found in the Conversation's Messages API.
`entities`	No	Input custom entities which can be detected in your conversation using Entities' API. For example, check the sample code on right.
`conversationType`	No	Currently, we only support `['sales']`, this optimizes Symbl algorithm to detect sales specific action-items present in a meeting. To know more when to use this conversationType, click here.

WebhookUrl will be used to send the status of job created for uploaded audio. Every time the status of the job changes it will be notified on the WebhookUrl

Response Object

Field	Description
`conversationId`	ID to be used with Conversation API
`jobId`	ID to be used with Job API

Webhook Payload

Field	Description
`jobId`	ID to be used with Job API
`status`	Current status of the job. (Valid statuses - [ `scheduled`, `in_progress`, `completed`, `failed` ])

POST Async Audio URL API

The Async Audio URL API takes in a url link of your audio and returns the full text transcript along with conversational insights. It can be utilized for any use case where you have access to recorded audio stored publicly as URL and want to extract insights and other conversational attributes supported by Symbl's Conversation API.

Use the POST API to upload your file and generate a Conversation ID. If you want to append additional audio information to the same Conversation ID, use the PUT API.

Example API call

curl --location --request POST 'https://api.symbl.ai/v1/process/audio/url?name="Your meeting name"&webhookUrl=<your_webhook_url>&entities= [{"customType": "Custom_Entity_Type", "text": "Custom Entity to be searched in transcript"}]' \
--header 'Content-Type: application/json' \
--header 'x-api-key: <generated_valid_token>' \
--data-raw '{
  "url": "https://symbltestdata.s3.us-east-2.amazonaws.com/sample_audio_file.wav",
  "confidenceThreshold": 0.6,
  "timezoneOffset": 0
}'

const request = require('request');
const fs = require('fs');

const accessToken = "<your_auth_token>";

const audioOption = {
  url: 'https://api.symbl.ai/v1/process/audio/url',
  headers: {
    'x-api-key': accessToken,
    'Content-Type': 'application/json'
   },
  qs: {
    'name':'Your meeting name',
    'webhookUrl': 'https://your_webhook_url',
    'entities': [{'customType': 'Custom_Entity_Type', 'text': 'Custom Entity to be searched in transcript'}]
  },
  body: JSON.stringify({
    "url": "https://symbltestdata.s3.us-east-2.amazonaws.com/sample_audio_file.wav",
    "confidenceThreshold": 0.6,
    "timezoneOffset": 0
  })
};

request.post(audioOption, (err, response, body) => {
  console.log(err, body);
});

The above request returns a response structured like this:

{
  "conversationId": "5815170693595136",
  "jobId": "9b1deb4d-3b7d-4bad-9bdd-2b0d7b3dcb6d"
}

HTTP REQUEST

POST https://api.symbl.ai/v1/process/audio/url

Request Headers

Header Name	Required	Value
`x-api-key`	Yes	your_auth_token
`Content-Type`	Yes	Accepted value `application/json`

Request Body

Field	Required	Type	Supported Values	Default	Description
`url`	Yes	String		[]	A valid url string. The URL must be a publicly accessible url.
`customVocabulary`	No	list		[]	Contains a list of words and phrases that provide hints to the speech recognition task.
`confidenceThreshold`	No	double	0.5 to 1.0	0.5	Minimum required confidence for the insight to be recognized.

Query Params

Parameter	Required	Value
`name`	No	Your meeting name. (When not specified, it will default to the `conversationId`.)
`webhookUrl`	No	Webhook url on which job updates to be sent. (This should be post API)
`detectActionPhraseForMessages`	No	Accepted values are `true` & `false`. It shows Actionable Phrases in each sentence of conversation. These sentences can be found in the Conversation's Messages API.
`entities`	No	Input custom entities which can be detected in your conversation using Entities' API. For example, check the sample code on right.
`conversationType`	No	Currently, we only support `['sales']`, this optimizes Symbl algorithm to detect sales specific action-items present in a meeting. To know more when to use this conversationType, click here.

WebhookUrl will be used to send the status of job created for uploaded audio url. Every time the status of the job changes it will be notified on the WebhookUrl

Response Object on Success

Field	Description
`conversationId`	ID to be used with Conversation API
`jobId`	ID to be used with Job API

Response on reaching limit

Field	Description
`Payload`	`{ "message" : "This API has a limit of maximum of 5 number of concurrent jobs per account. If you are looking to scale, and need more concurrent jobs than this limit, please contact us at support@symbl.ai" }`
`Header`	`{ "statusCode" : 429 }`

Webhook Payload

Field	Description
`jobId`	ID to be used with Job API
`status`	Current status of the job. (Valid statuses - [ `scheduled`, `in_progress`, `completed`, `failed` ])

PUT Async Audio URL API

The Async Audio URL API allows you to append an additional audio url to the previous conversation, append the transcription and get conversational insights for updated conversation. It can be useful in any use case where you have access to multiple recorded audio stored publicly as URL of any type of conversation, and you want to extract the insightful items supported by the Conversation API.

Use the POST API to add your URL and generate a Conversation ID. If you want to append additional audio information to the same Conversation ID, use the PUT API.

Example API call - The sample request accepts just the raw audio file from the data with the MIME typeset in the Content-Type Header. The audio file should only have Mono Channel.

curl --location --request PUT 'https://api.symbl.ai/v1/process/audio/url/:conversationId?name="Your meeting name"&webhookUrl=<your_webhook_url>' \
--header 'Content-Type: application/json' \
--header 'x-api-key: <generated_valid_token>' \
--data-raw '{
  "url": "https://symbltestdata.s3.us-east-2.amazonaws.com/sample_audio_file.wav",
  "confidenceThreshold": 0.6,
  "timezoneOffset": 0
}'

const request = require('request');
const fs = require('fs');

const accessToken = "<your_auth_token>";

const audioOption = {
  url: 'https://api.symbl.ai/v1/process/audio/url/' + your_conversationId,
  headers: {
    'x-api-key': accessToken,
    'Content-Type': 'application/json'
   },
  qs: {
    'name':'Your meeting name',
    'webhookUrl': 'https://your_webhook_url',
  },
  body: JSON.stringify({
    "url": "https://symbltestdata.s3.us-east-2.amazonaws.com/sample_audio_file.wav",
    "confidenceThreshold": 0.6,
    "timezoneOffset": 0
  })
};

request.put(audioOption, (err, response, body) => {
  console.log(err, body);
});

The above request returns a response structured like this:

{
  "conversationId": "5815170693595136",
  "jobId": "9b1deb4d-3b7d-4bad-9bdd-2b0d7b3dcb6d"
}

HTTP REQUEST

PUT https://api.symbl.ai/v1/process/audio/url/:conversationId

Request Headers

Header Name	Required	Value
`x-api-key`	Yes	your_auth_token
`Content-Type`	Yes	Accepted values is `application/json`

Request Body

Field	Required	Type	Supported Values	Default	Description
`url`	Yes	String		[]	A valid url string. The URL must be a publicly accessible url.
`customVocabulary`	No	list		[]	Contains a list of words and phrases that provide hints to the speech recognition task.
`confidenceThreshold`	No	double	0.5 to 1.0	0.5	Minimum required confidence for the insight to be recognized.

Path Params

Parameter	value
`conversationId`	conversationId which is provided by the first request submitted using POST async audio API

Query Params

Parameter	Required	Value
`name`	No	Your meeting name. (When not specified, it will default to the `conversationId`.)
`webhookUrl`	No	Webhook url on which job updates to be sent. (This should be post API)
`detectActionPhraseForMessages`	No	Accepted values are `true` & `false`. It shows Actionable Phrases in each sentence of conversation. These sentences can be found in the Conversation's Messages API.
`entities`	No	Input custom entities which can be detected in your conversation using Entities' API. For example, check the sample code on right.
`conversationType`	No	Currently, we only support `['sales']`, this optimizes Symbl algorithm to detect sales specific action-items present in a meeting. To know more when to use this conversationType, click here.

WebhookUrl will be used to send the status of job created for uploaded audio. Every time the status of the job changes it will be notified on the WebhookUrl

Response Object

Field	Description
`conversationId`	ID to be used with Conversation API
`jobId`	ID to be used with Job API

Webhook Payload

Field	Description
`jobId`	ID to be used with Job API
`status`	Current status of the job. (Valid statuses - [ `scheduled`, `in_progress`, `completed`, `failed` ])

Speaker Diarization

The Async Audio APIs can detect and separate unique speakers in a single stream of audio without need of separate speaker events.

To enable this capability with either of the APIs the enableSpeakerDiarization and diarizationSpeakerCount query-params need to be passed with the request.

The diarizationSpeakerCount should be equal to the number of unique speakers in the conversation. If the number varies then this might introduce false positives in the diarized results.

Refer to the How-To Get Speaker Separated Transcripts - Diarization with Async API to get started with this capability!

If you’re looking for similar capability in Real-Time APIs, please refer to Speaker Events and Speaker Separation in WebSocket API sections.

Query Params

Parameter	Required	Value
`enableSpeakerDiarization`	Yes	Whether the diarization should be enabled for this conversation. Pass this as `true` to enable this capability.
`diarizationSpeakerCount`	Yes	The number of unique speakers in this conversation.

Video API

Table of content: Async Video API

POST Async Video API
PUT Async Video API
POST Async Video URL API
PUT Async Video URL API
Speaker Diarization

System requirement: NodeJS 7+

POST Async Video API

The Async Video API takes your video and returns the full text transcript along with conversational insights. It can be useful in any use case where you have access to the video file of any type of conversation, and you want to extract the insightful items supported by the Conversation API.

Example API call - The sample request accepts a raw video file from the data with the MIME typeset in the Content-Type Header.

curl --location --request POST 'https://api.symbl.ai/v1/process/video?name="Your meeting name"&webhookUrl=<your_webhook_url>&entities= [{"customType": "Custom_Entity_Type", "text": "Custom Entity to be searched in transcript"}]' \
--header 'Content-Type: video/mp4' \
--header 'x-api-key: <generated_valid_token>' \
--data-binary '@/file/location/your_video.mp4'

const request = require('request');
const fs = require('fs');

const accessToken = "<your_auth_token>";
const videoFileStream = fs.createReadStream('/file/location/video.mp4');

const videoOption = {
  url: 'https://api.symbl.ai/v1/process/video',
  headers: {
    'x-api-key': accessToken,
    'Content-Type': 'video/mp4'
   },
  qs: {
    'name':'Your meeting name',
    'webhookUrl': 'https://your_webhook_url',
    'entities': [{'customType': 'Custom_Entity_Type', 'text': 'Custom Entity to be searched in transcript'}]
  },
  json: true,
};

videoFileStream.pipe(request.post(videoOption, (err, response, body) => {
  console.log(err, body);
}));

The above request returns a response structured like this:

{
  "conversationId": "5815170693595136",
  "jobId": "9b1deb4d-3b7d-4bad-9bdd-2b0d7b3dcb6d"
}

HTTP REQUEST

POST https://api.symbl.ai/v1/process/video

Request Headers

Header Name	Required	Value
`x-api-key`	Yes	your_auth_token
`Content-Type`	Yes	Describes the format and codec of the provided video. Accepted value `video/mp4`

Query Params

Parameter	Required	Value
`name`	No	Your meeting name. (When not specified, it will default to the `conversationId`.)
`webhookUrl`	No	Webhook url on which job updates to be sent. (This should be post API)
`customVocabulary`	No	Contains a list of words and phrases that provide hints to the speech recognition task.
`detectActionPhraseForMessages`	No	Accepted values are `true` & `false`. It shows Actionable Phrases in each sentence of conversation. These sentences can be found in the Conversation's Messages API.
`entities`	No	Input custom entities which can be detected in your conversation using Entities' API. For example, check the sample code on right.
`conversationType`	No	Currently, we only support `['sales']`, this optimizes Symbl algorithm to detect sales specific action-items present in a meeting. To know more when to use this conversationType, click here.

WebhookUrl will be used to send the status of job created for uploaded video. Every time the status of the job changes it will be notified on the WebhookUrl

Response Object on Success

Field	Description
`conversationId`	ID to be used with Conversation API
`jobId`	ID to be used with Job API

Response on reaching limit

Field	Description
`Payload`	`{ "message" : "Too Many Requests" }`
`Header`	`{ "statusCode" : 429 }`

Webhook Payload

Field	Description
`jobId`	ID to be used with Job API
`status`	Current status of the job. (Valid statuses - [ `scheduled`, `in_progress`, `completed`, `failed` ])

PUT Async Video API

The Async Audio API allows you to append an additional audio to the previous conversation, append the transcription and get conversational insights for updated conversation. It can be useful in any use case where you have access to multiple video files of any type of conversation, and you want to extract the insightful items supported by the Conversation API.

Example API call - The sample request accepts just the raw video file from the data with the MIME typeset in the Content-Type Header.

# MP4 File
curl --location --request PUT 'https://api.symbl.ai/v1/process/video/:conversationId?name="Your meeting name"&webhookUrl=<your_webhook_url>&entities= [{"customType": "Custom_Entity_Type", "text": "Custom Entity to be searched in transcript"}]' \
--header 'Content-Type: video/mp4' \
--header 'x-api-key: <generated_valid_token>' \
--data-binary '@/file/location/your_video.mp4'

const request = require('request');
const fs = require('fs');

const accessToken = "<your_auth_token>";
const videoFileStream = fs.createReadStream('/file/location/video.mp4');

const videoOption = {
  url: 'https://api.symbl.ai/v1/process/video/' + your_conversationId,
  headers: {
    'x-api-key': accessToken,
    'Content-Type': 'video/mp4'
   },
  qs: {
    'name':'Your meeting name',
    'webhookUrl': 'https://your_webhook_url',
    'entities': [{'customType': 'Custom_Entity_Type', 'text': 'Custom Entity to be searched in transcript'}]
  },
  json: true,
};

videoFileStream.pipe(request.put(videoOption, (err, response, body) => {
  console.log(err, body);
}));

The above request returns a response structured like this:

{
  "conversationId": "5815170693595136",
  "jobId": "9b1deb4d-3b7d-4bad-9bdd-2b0d7b3dcb6d"
}

HTTP REQUEST

PUT https://api.symbl.ai/v1/process/video/:conversationId

Request Headers

Header Name	Required	Value
`x-api-key`	Yes	your_auth_token
`Content-Type`	Yes	Describes the format and codec of the provided video data. Accepted value `video/mp4`

Path Params

Parameter	value
`conversationId`	conversationId which is provided by the first request submitted using POST async video API

Query Params

Parameter	Required	Value
`name`	No	Your meeting name. (When not specified, it will default to the `conversationId`.)
`webhookUrl`	No	Webhook url on which job updates to be sent. (This should be post API)
`customVocabulary`	No	Contains a list of words and phrases that provide hints to the speech recognition task.
`detectActionPhraseForMessages`	No	Accepted values are `true` & `false`. It shows Actionable Phrases in each sentence of conversation. These sentences can be found in the Conversation's Messages API.
`entities`	No	Input custom entities which can be detected in your conversation using Entities' API. For example, check the sample code on right.
`conversationType`	No	Currently, we only support `['sales']`, this optimizes Symbl algorithm to detect sales specific action-items present in a meeting. To know more when to use this conversationType, click here.

WebhookUrl will be used to send the status of job created for uploaded video. Every time the status of the job changes it will be notified on the WebhookUrl

Response Object

Field	Description
`conversationId`	ID to be used with Conversation API
`jobId`	ID to be used with Job API

Webhook Payload

Field	Description
`jobId`	ID to be used with Job API
`status`	Current status of the job. (Valid statuses - [ `scheduled`, `in_progress`, `completed`, `failed` ])

POST Async Video URL API

The Async Video URL API allows you to process a mp4 video and return the full text transcript along with conversational insights. It can be utilized for any use case where you have access to recorded video stored publicly as URL and want to extract insights and other conversational attributes supported by Symbl's Conversation API.

Use the POST API to upload your file and generate a Conversation ID. If you want to append additional video information to the same Conversation ID, use the PUT API.

Example API call

curl --location --request POST 'https://api.symbl.ai/v1/process/video/url?name="Your meeting name"&webhookUrl=<your_webhook_url>&entities= [{"customType": "Custom_Entity_Type", "text": "Custom Entity to be searched in transcript"}]' \
--header 'Content-Type: application/json' \
--header 'x-api-key: <generated_valid_token>' \
--data-raw '{
  "url": "https://symbltestdata.s3.us-east-2.amazonaws.com/sample_video_file.mp4",
  "confidenceThreshold": 0.6,
  "timezoneOffset": 0
}'

const request = require('request');
const fs = require('fs');

const accessToken = "<your_auth_token>";

const videoOption = {
  url: 'https://api.symbl.ai/v1/process/video/url',
  headers: {
    'x-api-key': accessToken,
    'Content-Type': 'application/json'
   },
  qs: {
    'name':'Your meeting name',
    'webhookUrl': 'https://your_webhook_url',
    'entities': [{'customType': 'Custom_Entity_Type', 'text': 'Custom Entity to be searched in transcript'}]
  },
  body: JSON.stringify({
    "url": "https://symbltestdata.s3.us-east-2.amazonaws.com/sample_video_file.mp4",
    "confidenceThreshold": 0.6,
    "timezoneOffset": 0
  })
};

request.post(videoOption, (err, response, body) => {
  console.log(err, body);
});

The above request returns a response structured like this:

{
  "conversationId": "5815170693595136",
  "jobId": "9b1deb4d-3b7d-4bad-9bdd-2b0d7b3dcb6d"
}

HTTP REQUEST

POST https://api.symbl.ai/v1/process/video/url

Request Headers

Header Name	Required	Value
`x-api-key`	Yes	your_auth_token
`Content-Type`	Yes	Accepted value `application/json`

Request Body

Field	Required	Type	Supported Values	Default	Description
`url`	Yes	String		[]	A valid url string. The URL must be a publicly accessible url.
`customVocabulary`	No	list		[]	Contains a list of words and phrases that provide hints to the speech recognition task.

Query Params

Parameter	Required	Value
`name`	No	Your meeting name. (When not specified, it will default to the `conversationId`.)
`webhookUrl`	No	Webhook url on which job updates to be sent. (This should be post API)
`detectActionPhraseForMessages`	No	Accepted values are `true` & `false`. It shows Actionable Phrases in each sentence of conversation. These sentences can be found in the Conversation's Messages API.
`entities`	No	Input custom entities which can be detected in your conversation using Entities' API. For example, check the sample code on right.
`conversationType`	No	Currently, we only support `['sales']`, this optimizes Symbl algorithm to detect sales specific action-items present in a meeting. To know more when to use this conversationType, click here.

WebhookUrl will be used to send the status of job created for uploaded video url. Every time the status of the job changes it will be notified on the WebhookUrl

Response Object on Success

Field	Description
`conversationId`	ID to be used with Conversation API
`jobId`	ID to be used with Job API

Response on reaching limit

Field	Description
`Payload`	`{ "message" : "This API has a limit of maximum of 5 number of concurrent jobs per account. If you are looking to scale, and need more concurrent jobs than this limit, please contact us at support@symbl.ai" }`
`Header`	`{ "statusCode" : 429 }`

Webhook Payload

Field	Description
`jobId`	ID to be used with Job API
`status`	Current status of the job. (Valid statuses - [ `scheduled`, `in_progress`, `completed`, `failed` ])

PUT Async Video URL API

The Async Video URL API allows you to process an additional video file to the previous conversation, append the transcription and get conversational insights for updated conversation.

It can be useful in any use case where you have access to multiple recorded video stored publicly as URL of any type of conversation, and you want to extract the insightful items supported by the Conversation API.

Use the POST API to add your URL and generate a Conversation ID. If you want to append additional video information to the same Conversation ID, use the PUT API.

Example API call

curl --location --request PUT 'https://api.symbl.ai/v1/process/video/url/:conversationId?name="Your meeting name"&webhookUrl=<your_webhook_url>&entities= [{"customType": "Custom_Entity_Type", "text": "Custom Entity to be searched in transcript"}]' \
--header 'Content-Type: application/json' \
--header 'x-api-key: <generated_valid_token>' \
--data-raw '{
  "url": "https://symbltestdata.s3.us-east-2.amazonaws.com/sample_video_file.mp4",
  "confidenceThreshold": 0.6,
  "timezoneOffset": 0
}'

const request = require('request');
const fs = require('fs');

const accessToken = "<your_auth_token>";

const videoOption = {
  url: 'https://api.symbl.ai/v1/process/video/url/' + your_conversationId,
  headers: {
    'x-api-key': accessToken,
    'Content-Type': 'application/json'
   },
  qs: {
    'name':'Your meeting name',
    'webhookUrl': 'https://your_webhook_url',
    'entities': [{'customType': 'Custom_Entity_Type', 'text': 'Custom Entity to be searched in transcript'}]
  },
  body: JSON.stringify({
    "url": "https://symbltestdata.s3.us-east-2.amazonaws.com/sample_video_file.mp4",
    "confidenceThreshold": 0.6,
    "timezoneOffset": 0
  })
};

request.put(videoOption, (err, response, body) => {
  console.log(err, body);
});

The above request returns a response structured like this:

{
  "conversationId": "5815170693595136",
  "jobId": "9b1deb4d-3b7d-4bad-9bdd-2b0d7b3dcb6d"
}

HTTP REQUEST

PUT https://api.symbl.ai/v1/process/video/url/:conversationId

Request Headers

Header Name	Required	Value
`x-api-key`	Yes	your_auth_token
`Content-Type`	Yes	Accepted values is `application/json`

Request Body

Field	Required	Type	Supported Values	Default	Description
`url`	Yes	String		[]	A valid url string. The URL must be a publicly accessible url.
`customVocabulary`	No	list		[]	Contains a list of words and phrases that provide hints to the speech recognition task.
`confidenceThreshold`	No	double	0.0 to 1.0	0.5	Minimum required confidence for the insight to be recognized.

Path Params

Parameter	value
`conversationId`	conversationId which is provided by the first request submitted using POST async video API

Query Params

Parameter	Required	Value
`name`	No	Your meeting name. (When not specified, it will default to the `conversationId`.)
`webhookUrl`	No	Webhook url on which job updates to be sent. (This should be post API)
`detectActionPhraseForMessages`	No	Accepted values are `true` & `false`. It shows Actionable Phrases in each sentence of conversation. These sentences can be found in the Conversation's Messages API.
`entities`	No	Input custom entities which can be detected in your conversation using Entities' API. For example, check the sample code on right.
`conversationType`	No	Currently, we only support `['sales']`, this optimizes Symbl algorithm to detect sales specific action-items present in a meeting. To know more when to use this conversationType, click here.

WebhookUrl will be used to send the status of job created for uploaded video url. Every time the status of the job changes it will be notified on the WebhookUrl

Response Object

Field	Description
`conversationId`	ID to be used with Conversation API
`jobId`	ID to be used with Job API

Webhook Payload

Field	Description
`jobId`	ID to be used with Job API
`status`	Current status of the job. (Valid statuses - [ `scheduled`, `in_progress`, `completed`, `failed` ])

Speaker Diarization Async

The Async Video APIs can detect and separate unique speakers in a single stream of video without need of separate speaker events.

To enable this capability with either of the APIs the enableSpeakerDiarization and diarizationSpeakerCount query-params need to be passed with the request.

The diarizationSpeakerCount should be equal to the number of unique speakers in the conversation. If the number varies then this might introduce false positives in the diarized results.

Refer to the How-To Get Speaker Separated Transcripts - Diarization with Async API to get started with this capability!

If you’re looking for similar capability in Real-Time APIs, please refer to Speaker Events and Speaker Separation in WebSocket API sections.

Query Params

Parameter	Required	Value
`enableSpeakerDiarization`	Yes	Whether the diarization should be enabled for this conversation. Pass this as `true` to enable this capability.
`diarizationSpeakerCount`	Yes	The number of unique speakers in this conversation.

Other Languages

The Async Audio and Async Video APIs can work with languages other than English. The following list of languages (with their BCP-47 language-codes) are currently supported:

English (United States) – en-US
English (United Kingdom) – en-GB
English (Australia) – en-AU
English (Ireland) - en-IE
English (India) - en-IN
Chinese (China) - zh-CN
Russian (Russian Federation) - ru-RU
French (Canada) - fr-CA
French (France) - fr-FR
German (Germany) - de-DE
Italian (Italy) – it-IT
Dutch (Netherlands) – nl-NL
Japanese (Japan) – ja-JP
Spanish (United States) – es-US
Spanish (Spain) - es-ES
Arabic (Saudi Arabia) - ar-SA
Hindi (India) - hi-IN
Portuguese (Brazil) - pt-BR
Portuguese (Portugal) - pt-PT

To use one of the supported languages use the query-parameter languageCode with the language-codes specified above.

curl --location --request POST 'https://api.symbl.ai/v1/process/video/url?languageCode=en-US&webhookUrl=<your_webhook_url>' \
--header 'Content-Type: application/json' \
--header 'x-api-key: <generated_valid_token>' \
--data-raw '{
  "url": "https://symbltestdata.s3.us-east-2.amazonaws.com/sample_video_file.mp4",
  "confidenceThreshold": 0.6,
  "timezoneOffset": 0
}'

If the query-parameter languageCode is not specified then en-US is used as the default language.

Currently only the messages endpoint of Conversation API will return the transcribed data and insights will be return an empty array.

Query Params

Parameter	Required	Value
`languageCode`	Yes	The BCP-47 language-code of the language. example: `fr-CA`

Job API

The Job Status API is used to retrieve the status of an ongoing Async audio and Async video request. You can use the jobId received in the successful response of the Async API.

API Endpoint

https://api.symbl.ai/v1/job/{jobId}

Example API call

curl --location --request GET 'https://api.symbl.ai/v1/job/{jobId}' \
--header 'Content-Type: application/json' \
--header 'x-api-key: <generated_valid_token>'

const request = require('request');
const your_auth_token = '<your_auth_token>';

request.get({
    url: 'https://api.symbl.ai/v1/job/{jobId}',
    headers: { 'x-api-key': your_auth_token },
    json: true
}, (err, response, body) => {
  console.log(body);
});

The above request returns a response structured like this:

{
  "id": "9b1deb4d-3b7d-4bad-9bdd-2b0d7b3dcb6d",
  "status": "in_progress"
}

HTTP REQUEST

GET https://api.symbl.ai/v1/job/{jobId}

Response Parameters

Parameter	Description
`id`	The ID of the Job
`status`	Is one of type `scheduled`, `in_progress`, `completed`, `failed`

Conversation API

The Conversation API provides the REST API interface for the management and processing of your conversations

GET Conversation

This API returns the conversation meta-data like meeting name, member name and email, start and end time of the meeting, meeting type and meeting id.

API Endpoint

https://api.symbl.ai/v1/conversations/{conversationId}

Example API call

curl "https://api.symbl.ai/v1/conversations/{conversationId}" \
    -H "x-api-key: <api_token>"

const request = require('request');
const your_auth_token = '<your_auth_token>';

request.get({
    url: 'https://api.symbl.ai/v1/conversations/{conversationId}',
    headers: { 'x-api-key': your_auth_token },
    json: true
}, (err, response, body) => {
    console.log(body);
});

The above request returns a response structured like this:

{
    "id": "5179649407582208",
    "type": "meeting",
    "name": "Project Meeting #2",
    "startTime": "2020-02-12T11:32:08.000Z",
    "endTime": "2020-02-12T11:37:31.134Z",
    "members": [
        {
            "name": "John",
            "email": "John@example.com"
        },
        {
            "name": "Mary",
            "email": "Mary@example.com"
        },
        {
            "name": "Roger",
            "email": "Roger@example.com"
        }
    ]
}

HTTP REQUEST

GET https://api.symbl.ai/v1/conversations/{conversationId}

Response Object

Field	Description
`id`	unique conversation identifier
`type`	conversation type. default is `meeting`
`name`	name of the conversation
`startTime`	DateTime value
`endTime`	DateTime value
`members`	list of member objects containing name and email if detected

DELETE Conversation

This API permanently deletes conversation and all related entities such as messages, insights, topics etc. associated with the conversationId.

NOTE: Once delete operation is successful, all information associated with the conversationId is permanently lost and cannot be recovered later.

curl --location --request DELETE "https://api.symbl.ai/v1/conversations/{conversationId}" \
    -H "x-api-key: <api_token>"

const request = require('request');
const your_auth_token = '<your_auth_token>';

request.delete({
    url: 'https://api.symbl.ai/v1/conversations/{conversationId}',
    headers: { 'x-api-key': your_auth_token },
    json: true
}, (err, response, body) => {
    console.log(body);
});

The above request returns a response structured like this:

When conversationId is right.

{
    "message": "successfully deleted the conversation"
}

When conversationId is wrong.

{
    "message": "The conversationId is either invalid or does not exist."
}

HTTP REQUEST

DELETE https://api.symbl.ai/v1/conversations/{conversationId}

GET Messages/Transcript

This API returns a list of all the messages in a conversation. You can use this for providing transcription for video conference, meeting or telephone call.

Sentiment Analysis in messages Beta

In Messages API you can enable sentiment analysis over each message which is being spoken in the conversation.
All you need to do is pass sentiment=true as a query parameter. Read more about it here.

API Endpoint

https://api.symbl.ai/v1/conversations/{conversationId}/messages

Example API call

curl "https://api.symbl.ai/v1/conversations/{conversationId}/messages" \
    -H "x-api-key: <api_token>"

const request = require('request');
const your_auth_token = '<your_auth_token>';

request.get({
    url: 'https://api.symbl.ai/v1/conversations/{conversationId}/messages',
    headers: { 'x-api-key': your_auth_token },
    json: true
}, (err, response, body) => {
    console.log(body);
});

The above request returns a response structured like this:

{
    "messages": [
         {
             "id": "6412283618000896",
             "text": "Best package for you is $69.99 per month.",
             "from": {
                 "name": "Roger",
                 "email": "Roger@example.com"
             },
             "startTime": "2020-07-10T11:16:21.024Z",
             "endTime": "2020-07-10T11:16:26.724Z",
             "conversationId": "6749556955938816",
             "phrases": [
                {
                    "type": "action_phrase",
                    "text": "$69.99 per month"
                }
             ],
             "sentiment": {
                "polarity": {
                    "score": 0.6
                },
                "suggested": "positive"
              },
              "words": [
                 {
                     "word": "Best",
                     "startTime": "2020-08-18T11:10:14.536Z",
                     "endTime": "2020-08-18T11:10:15.536Z",
                 },
                 {
                     "word": "package",
                     "startTime": "2020-08-18T11:10:16.536Z",
                     "endTime": "2020-08-18T11:10:17.536Z",
                 },
                 {
                     "word": "for",
                     "startTime": "2020-08-18T11:10:18.536Z",
                     "endTime": "2020-08-18T11:10:19.536Z",
                 },
                 {
                     "word": "you",
                     "startTime": "2020-08-18T11:10:20.536Z",
                     "endTime": "2020-08-18T11:10:22.536Z",
                 },
                 {
                     "word": "is",
                     "startTime": "2020-08-18T11:10:22.536Z",
                     "endTime": "2020-08-18T11:10:25.536Z",
                 },
                 {
                     "word": "$69.99",
                     "startTime": "2020-08-18T11:10:25.536Z",
                     "endTime": "2020-08-18T11:10:27.536Z",
                 },
                 {
                     "word": "per",
                     "startTime": "2020-08-18T11:10:27.536Z",
                     "endTime": "2020-08-18T11:10:29.536Z",
                 },
                 {
                     "word": "month.",
                     "startTime": "2020-08-18T11:10:30.536Z",
                     "endTime": "2020-08-18T11:10:32.536Z",
                 }]
          },
         {
             "id": "5661493169225728",
             "text": "Okay, Where is the file?",
             "from": {
                 "name": "John",
                 "email": "John@example.com"
             }
             "startTime": "2020-08-18T11:11:14.536Z",
             "endTime": "2020-08-18T11:11:18.536Z",
             "conversationId": "5139780136337408",
             "phrases": [],
             "sentiment": {
                    "polarity": {
                        "score": 0.2
                    },
                    "suggested": "neutral"
              },
             "words": [
                 {
                     "word": "Okay,",
                     "startTime": "2020-08-18T11:11:14.536Z",
                     "endTime": "2020-08-18T11:11:14.936Z"
                 },
                 {
                     "word": "Where",
                     "startTime": "2020-08-18T11:11:14.936Z",
                     "endTime": "2020-08-18T11:11:15.436Z"
                 },
                 {
                     "word": "is",
                     "startTime": "2020-08-18T11:11:16.236Z",
                     "endTime": "2020-08-18T11:11:16.536Z"
                 },
                 {
                     "word": "the",
                     "startTime": "2020-08-18T11:11:16.536Z",
                     "endTime": "2020-08-18T11:11:16.936Z"
                 },
                 {
                     "word": "file?",
                     "startTime": "2020-08-18T11:11:16.936Z",
                     "endTime": "2020-08-18T11:11:17.236Z"
                 }
    ]
}

HTTP REQUEST

GET https://api.symbl.ai/v1/conversations/{conversationId}/messages

Query Params

Parameter	Required	Value	Description
`verbose`	No	true	Gives you word level timestamps of each sentence.
`sentiment`	No	true	Give you sentiment analysis on each message.

Response Object

Field	Description
`id`	unique message identifier
`text`	message text
`from`	user object with name and email
`startTime`	DateTime value
`endTime`	DateTime value
`conversationId`	unique conversation identifier
`words`	words object with word, startTime and endTime.
`phrases`	It shows the most important action phrases in each sentence. It's enabled when you pass `detectActionPhraseForMessages=true` during submiting the request in Async and Websocket API.
`sentiment`	Shows the sentiment polarity(intensity of negativity or positivity of a sentene) and suggested sentiment type (positive, negative and neutral)

GET Members

Returns a list of all the members in a conversation. A Member is referred to a participant in the conversation that is uniquely identified as a speaker. Identifying different participants in the meetings can be done by implementing speaker separation.

For more details on identifying members by Speaker Events or Active Talker events in Real-time using Voice SDK - here.

For more details on identifying members by independent audio stream integration using Websocket - here.

API Endpoint

https://api.symbl.ai/v1/conversations/{conversationId}/members

Example API call

curl "https://api.symbl.ai/v1/conversations/{conversationId}/members" \
    -H "x-api-key: <api_token>"

const request = require('request');
const your_auth_token = '<your_auth_token>';

request.get({
    url: 'https://api.symbl.ai/v1/conversations/{conversationId}/members',
    headers: { 'x-api-key': your_auth_token },
    json: true
}, (err, response, body) => {
    console.log(body);
});

The above request returns a response structured like this:

{
    "members": [
        {
            "id": "fc9b35cd-361f-41c6-9029-0944d21c7150",
            "name": "John",
            "email": "John@example.com"
        },
        {
            "id": "382362a2-eeec-46a3-8891-d50508293851",
            "name": "Mary",
            "email": "Mary@example.com"
        },
        {
            "id": "b7de3a33-a16c-4926-9d4d-a904c88271c2",
            "name": "Roger",
            "email": "Roger@example.com"
        }
    ]
}

HTTP REQUEST

GET https://api.symbl.ai/v1/conversations/{conversationId}/members

Response Object

Field	Description
`id`	member's unique identifier
`name`	member's name
`email`	member's email

PUT Update Members

Update an existing member in an conversation. This API can be used for updating the unique speakers detected as members from diarization as well.

To diarize/separate speakers in a single audio/video stream refer to the How-To Get Speaker Separated Transcripts - Diarization with Async API

For more details on identifying members by Speaker Events or Active Talker events in Real-time using Voice SDK - here.

For more details on identifying members by independent audio stream integration using Websocket - here.

API Endpoint

https://api.symbl.ai/v1/conversations/{conversationId}/members/{id}

Example API call

curl --location --request PUT 'https://api.symbl.ai/v1/conversations/{conversationId}/members/fc9b35cd-361f-41c6-9029-0944d21c7150' --header 'Content-Type: application/json' --header 'x-api-key: <valid-generated-token' --data-raw '{
    "id": "fc9b35cd-361f-41c6-9029-0944d21c7150",
    "email": "john@example.com",
    "name": "John"
}'

const request = require('request');
const your_auth_token = '<your_auth_token>';

request.put({
    url: 'https://api.symbl.ai/v1/conversations/{conversationId}/members/fc9b35cd-361f-41c6-9029-0944d21c7150',
    headers: { 'x-api-key': your_auth_token },
    body: {
        id: 'fc9b35cd-361f-41c6-9029-0944d21c7150',
        name: 'John',
        email: 'john@example.com'
    },
    json: true
}, (err, response, body) => {
    console.log(body);
});

The above request returns a response structured like this:

{
    "message": "Member with id: fc9b35cd-361f-41c6-9029-0944d21c7150 for conversationId: <conversationId> updated successfully! The update should be reflected in all messages and insights along with this conversation"
}

HTTP REQUEST

PUT https://api.symbl.ai/v1/conversations/{conversationId}/members/{id}

Request Body

Field	Required	Type	Description
`id`	Yes	string	The unique identifier of the member for this conversation. This can be retrieved from the members endpoint.
`name`	Yes	string	The name of the member.
`email`	No	string	The email-id of the member. If specified this can be used to correctly identify and merge the existing user in case the conversation is appended with a new diarized conversation which has one or more same speakers as the conversation it's being appended to.

Response Object

Field	Description
`message`	A description of the update. This message indicates that the member details have now been updated across the conversation for all the messages and action-items, follow-ups and questions. You can also get the updated member from the members endpoint.

GET Topics

The most relevant topics of discussion from the conversation that are generated based on the combination of the overall scope of the discussion.

This API returns all the topics generated from a conversation.

Sentiment Analysis in TopicsBeta

You can enable sentiment analysis over each topics which is being discussed in the conversation.
All you need to do is pass sentiment=true as a query parameter. Read more about it here.

Topic HierarchyBeta

You can enable topic hierarchy in Topics API by passing parentRefs=true. Topic Hierarchy breaks conversation in parent and child topics which helps outline the entire conversation faster. Read more about it here

API Endpoint

https://api.symbl.ai/v1/conversations/{conversationId}/topics

Example API call

curl "https://api.symbl.ai/v1/{conversationId}/topics" \
    -H "x-api-key: <api_token>"

const request = require('request');
const your_auth_token = '<your_auth_token>';

request.get({
    url: 'https://api.symbl.ai/v1/{conversationId}/topics',
    headers: { 'x-api-key': your_auth_token },
    json: true
}, (err, response, body) => {
    console.log(body);
});

Topic Hierarchy Sample Response (parentRefs=true):

{
    "topics": [
        {
            "id": "5907389282779136",
            "text": "interns",
            "type": "topic",
            "score": 0.7178597920690242,
            "messageIds": [
                "4600982711304192",
                "5487363432120320",
                "6109794119188480"
            ],
            "parentRefs": [
                {
                    "type": "topic",
                    "text": "company-wise hiring"
                }
            ]
        },
        {
            "id": "5776859730018304",
            "text": "company-wise hiring",
            "type": "topic",
            "score": 0.788856914361565,
            "messageIds": [
                "6298570346987520",
                "6330577953226752"
            ],
            "parentRefs": []
        },
        {
            "id": "6697188878974976",
            "text": "new regulations",
            "type": "topic",
            "score": 0.6968750176932417,
            "messageIds": [
                "5356560840654848",
                "5663440783802368",
                "5263998490509312",
                "6082396449406976",
                "4925138187321344",
            ],
            "parentRefs": [
                {
                    "type": "topic",
                    "text": "company-wise hiring"
                }
            ]
        }
    ]
}

Sentiment Sample Response (sentiment=true):

{
    "topics": [
        {
            "id": "5907389282779136",
            "text": "interns",
            "type": "topic",
            "score": 0.7178597920690242,
            "messageIds": [
                "4600982711304192",
                "5487363432120320",
                "6109794119188480"
            ],
            "sentiment": {
                "polarity": {
                    "score": 0.389
                },
                "suggested": "positive"
            },
            "parentRefs": []
        },
        {
            "id": "5776859730018304",
            "text": "company-wise hiring",
            "type": "topic",
            "score": 0.788856914361565,
            "messageIds": [
                "6298570346987520",
                "6330577953226752"
            ],
            "sentiment": {
                "polarity": {
                    "score": 0.012
                },
                "suggested": "neutral"
            },
            "parentRefs": []
        },
        {
            "id": "6697188878974976",
            "text": "new regulations",
            "type": "topic",
            "score": 0.6968750176932417,
            "messageIds": [
                "5356560840654848",
                "5663440783802368",
                "5263998490509312",
                "6082396449406976",
                "4925138187321344",
            ],
            "sentiment": {
                "polarity": {
                    "score": -0.809
                },
                "suggested": "negative"
            },
            "parentRefs": []
        }
    ]
}

HTTP REQUEST

GET https://api.symbl.ai/v1/conversations/{conversationId}/topics

Query Params

Parameter	Required	Value	Description
`sentiment`	No	true	Give you sentiment analysis on each topic in conversation.
`parentRefs`	No	true	Gives you topic hierarchy.

Response Object

Field	Description
`id`	unique conversation identifier
`text`	conversation text
`type`	response type. default is topics
`score`	confidence score of the generated topic. value from 0 - 1
`messageIds`	unique message identifiers of the corresponding messages
`entities`	list of detected entity objects in the insight with `type` - entity type and `text` - corresponding text
`parentRefs`	This is enabled when `parentRefs` is set to true in request. For more information read Topic Hierarchy.
`sentiment`	Shows the sentiment polarity(intensity of negativity or positivity of a sentene) and suggested sentiment type (positive, negative and neutral)

GET Questions

Any explicit question or request for information that comes up during the conversation, whether answered or not, is recognized as a question.

This API returns all the questions generated from the conversation.

API Endpoint

https://api.symbl.ai/v1/conversations/{conversationId}/questions

Example API call

curl "https://api.symbl.ai/v1/conversations/{conversationId}/questions" \
    -H "x-api-key: <api_token>"

const request = require('request');
const your_auth_token = '<your_auth_token>';

request.get({
    url: 'https://api.symbl.ai/v1/conversations/{conversationId}/questions',
    headers: { 'x-api-key': your_auth_token },
    json: true
}, (err, response, body) => {
    console.log(body);
});

The above request returns a response structured like this:

{
    "questions": [
        {
            "id": "5179649407582208",
            "text": "Push them for the two weeks delivery, right?",
            "type": "question",
            "score": 0.9730208796076476,
            "messageIds": [
                "5019269922291712"
            ],
            "entities": []
        },
        {
            "id": "5642466493464576",
            "text": "I think what is the Bahamas?",
            "type": "question",
            "score": 0.9119608386876195,
            "messageIds": [
                "5019269922291712"
            ],
            "entities": []
        },
        {
            "id": "5756718797553664",
            "text": "Okay need be detained, or we can go there in person and support them?",
            "type": "question",
            "score": 0.893303149769215,
            "messageIds": [
                "5019269922291712"
            ],
            "entities": []
        },
        {
            "id": "6235991715086336",
            "text": "Why is that holiday in US from 17?",
            "type": "question",
            "score": 0.9998053310511206,
            "messageIds": [
                "5019269922291712"
            ],
            "entities": []
        }
    ]
}

HTTP REQUEST

GET https://api.symbl.ai/v1/conversations/{conversationId}/questions

Response Object

Field	Description
`id`	unique conversation identifier
`text`	conversation text
`type`	response type. default is question
`score`	confidence score of the generated topic. value from 0 - 1
`messageIds`	unique message identifiers of the corresponding messages
`entities`	list of detected entity objects in the insight with `type` - entity type and `text` - corresponding text

GET Action-items

An action item is a specific outcome recognized in the conversation that requires one or more people in the conversation to act in the future. These actions can be definitive in nature and owned with a commitment to working on a presentation, sharing a file, completing a task, etc. Or they can be non-definitive like an idea, suggestion or an opinion that could be worked upon.

All action items are generated with action phrases, assignees and due dates so that you can build workflow automation with your own tools.

This API returns list of all the action items generated from the conversation.

API Endpoint

https://api.symbl.ai/v1/conversations/{conversationId}/action-items

Example API call

curl "https://api.symbl.ai/v1/conversations/{conversationId}/action-items" \
    -H "x-api-key: <api_token>"

const request = require('request');
const your_auth_token = '<your_auth_token>';

request.get({
    url: 'https://api.symbl.ai/v1/conversations/{conversationId}/action-items',
    headers: { 'x-api-key': your_auth_token },
    json: true
}, (err, response, body) => {
    console.log(body);
});

The above request returns a response structured like this:

{
    "actionItems": [
        {
            "id": "5633940379402240",
            "text": "Mary thinks we need to go ahead with the TV in Bangalore.",
            "type": "action_item",
            "score": 0.8659442937321238,
            "messageIds": [
                "4972726972317696"
            ],
            "phrases": [
                {
                    "type": "action_phrase",
                    "text": "create small file of like mp44 testing purpose"
                }
            ],
            "definitive": false,
            "entities": [],
            "assignee": {
                "name": "Mary",
                "email": "Mary@example.com"
            }
        },
        {
            "id": "5668855401676800",
            "text": "Call and Stephanie also brought up something to check against what Ison is given as so there's one more test that we want to do.",
            "type": "action_item",
            "score": 0.8660254037845785,
            "messageIds": [
                "6531517035577342"
            ],
            "phrases": [],
            "definitive": true,
            "entities": [],
            "assignee": {
                "name": "John",
                "email": "John@example.com"
            }
        },
        {
            "id": "5690029162627072",
            "text": "Checking the nodes with Eisner to make sure we covered everything so that will be x.",
            "type": "action_item",
            "score": 0.8657734634985154,
            "messageIds": [
                "6531517035577244"
            ],
            "phrases": [
                {
                    "type": "action_phrase",
                    "text": "Checking the nodes with Eisner to make sure we covered everything"
                }
            ],
            "definitive": true,
            "entities": [],
            "assignee": {
                "name": "John",
                "email": "John@example.com"
            }
        },
        {
            "id": "5707174000984064",
            "text": "Roger is going to work with the TV lab and make sure that test is also included, so we are checking to make sure not only with our complaints.",
            "type": "action_item",
            "score": 0.9999962500210938,
            "messageIds": [
                "6531517035527344"
            ],
            "phrases": [
                {
                    "type":"action_phrase",
                    "text":"Roger is going to work with the TV lab"
                }
            ],
            "definitive": true,
            "entities": [],
            "assignee": {
                "name": "Roger",
                "email": "Roger@example.com"
            }
        },
        {
            "id": "5757280188366848",
            "text": "Mary thinks it really needs to kick start this week which means the call with UV team and our us team needs to happen the next couple of days.",
            "type": "action_item",
            "score": 0.9999992500008438,
            "messageIds": [
                "6521517035577344"
            ],
            "phrases": [],
            "definitive": false,
            "entities": [],
            "assignee": {
                "name": "Mary",
                "email": "Mary@example.com"
            },
            "dueBy": "2020-02-10T07:00:00.000Z"
        }
    ]
}

HTTP REQUEST

GET https://api.symbl.ai/v1/conversations/{conversationId}/action-items

Response Object

Field	Description
`id`	unique conversation identifier
`text`	conversation text
`type`	response type. default is action_item
`score`	confidence score of the generated topic. value from 0 - 1
`messageIds`	unique message identifiers of the corresponding messages
`entities`	list of detected entity objects in the insight with `type` - entity type and `text` - corresponding text
`definitive`	Boolean indicating if the action-item is definitive or not. More about definitive here.
`phrases`	list of detected phrases with `type` - phrase type and `text` - corresponding text. The `action_phrase` type represents the actionable part of an insight.
`assignee`	this field contains the name and email of the person assigned to the action item

GET Follow-ups

This is a category of action items with a connotation to follow-up a request or a task like sending an email or making a phone call or booking an appointment or setting up a meeting.

This API returns list of all the follow ups generated from the conversation.

API Endpoint

https://api.symbl.ai/v1/conversations/{conversationId}/follow-ups

Example API call

curl "https://api.symbl.ai/v1/conversations/{conversationId}/follow-ups" \
    -H "x-api-key: <api_token>"

const request = require('request');
const your_auth_token = '<your_auth_token>';

request.get({
    url: 'https://api.symbl.ai/v1/conversations/{conversationId}/follow-ups',
    headers: { 'x-api-key': your_auth_token },
    json: true
}, (err, response, body) => {
    console.log(body);
});

The above request returns a response structured like this:

{
    "followUps": [
        {
            "id": "4526427164639111",
            "text": "We need to have the meeting today, and we're going to talk about how to run a product strategy Workshop is by Richard Holmes.",
            "type": "follow_up",
            "score": 0.8660254037851491,
            "messageIds": [
                "4675554024357888"
            ],
            "entities": [
                {
                    "type": "date",
                    "text": "today",
                    "offset": 28,
                    "value": "2020-06-22"
                },
                {
                    "type": "person",
                    "text": "Richard Holmes",
                    "offset": 110,
                    "value": {
                        "name": "Richard Holmes"
                    }
                }
            ],
            "phrases": [
                {
                    "text": "need to have the meeting today",
                    "type": "action_phrase"
                },
                {
                    "text": "talk about how to run a product strategy Workshop is by Richard Holmes",
                    "type": "action_phrase"
                }
            ],
            "from": {},
            "assignee": {},
            "dueBy": "2020-06-22T07:00:00.000Z"
        }
    ]
}

HTTP REQUEST

GET https://api.symbl.ai/v1/conversations/{conversationId}/follow-ups

Response Object

Field	Description
`id`	unique conversation identifier
`text`	conversation text
`type`	response type. default is follow_up
`score`	confidence score of the generated topic. value from 0 - 1
`messageIds`	unique message identifiers of the corresponding messages
`entities`	list of detected entity objects in the insight with `type` - entity type and `text` - corresponding text
`from`	user object with name and email
`assignee`	this field contains the name and email of the person assigned to the follow up
`phrases`	list of detected phrases with `type` - phrase type and `text` - corresponding text. The `action_phrase` type represents the actionable part of an insight

GET AnalyticsBeta

Analytics API provides you with functionality like finding speaker ratio, talk time, silence, pace and overlap in a conversation.

For each conversation it returns:

Speaker Ratio - Speaker’s talk and listen ratio and time.
Talk Time - Overall duration of the conversation.
Silence - Overall duration of silence.
Pace - Words per minute spoken in the conversation.
Overlap - When more than 1 speaker are speaking the same time, then conversation has overlap.

API Endpoint

https://api.symbl.ai/v1/conversations/{conversationId}/analytics

Example API call

curl "https://api.symbl.ai/v1/conversations/{conversationId}/analytics" \
    -H "x-api-key: <api_token>"

const request = require('request');
const your_auth_token = '<your_auth_token>';

request.get({
    url: 'https://api.symbl.ai/v1/conversations/{conversationId}/analytics',
    headers: { 'x-api-key': your_auth_token },
    json: true
}, (err, response, body) => {
    console.log(body);
});

The above request returns a response structured like this:

{
    "metrics": [
        {
            "type": "total_silence",
            "percent": 29.061,
            "seconds": 23.432
        },
        {
            "type": "total_talk_time",
            "percent": 70.939,
            "seconds": 57.199
        },
        {
            "type": "total_overlap",
            "percent": 55.071,
            "seconds": 31.5
        }
    ],
    "members": [
        {
            "id": "acc63fbe-c3cd-4daa-8ab0-b088142e5a0f",
            "name": "Speaker 1",
            "pace": {
                "wpm": 68
            },
            "talkTime": {
                "percentage": 40.912,
                "seconds": 23.401
            },
            "listenTime": {
                "percentage": 59.088,
                "seconds": 33.798
            },
            "overlap": {
                "overlapDuration": 31.5,
                "overlappingMembers": [
                    {
                        "id": "a52def45-be6e-484f-908b-9ac66eaecabb",
                        "name": "Speaker 2",
                        "percent": 61.58,
                        "seconds": 24.94
                    },
                    {
                        "id": "a52def45-be6e-484f-908b-9ac66eaecacb",
                        "name": "Speaker 3",
                        "percent": 7.51,
                        "seconds": 1.9
                    },
                    {
                        "id": "a52def45-be6e-484f-908b-9ac56eaecabb",
                        "name": "Speaker 4",
                        "percent": 12.199,
                        "seconds": 4.66
                    }
                ]
            }
        },
        {
            "id": "a52def45-be6e-484f-908b-9ac66eaecabb",
            "name": "Speaker 2",
            "pace": {
                "wpm": 132
            },
            "talkTime": {
                "percentage": 29.894,
                "seconds": 17.099
            },
            "listenTime": {
                "percentage": 70.106,
                "seconds": 40.1
            },
            "overlap": {
                "overlapDuration": 24.94,
                "overlappingMembers": [
                    {
                        "id": "acc63fbe-c3cd-4daa-8ab0-b088142e5a0f",
                        "name": "Speaker 1",
                        "percent": 61.58,
                        "seconds": 24.94
                    }
                ]
            }
        },
        {
            "id": "a52def45-be6e-484f-908b-9ac66eaecacb",
            "name": "Speaker 3",
            "pace": {
                "wpm": 189
            },
            "talkTime": {
                "percentage": 3.322,
                "seconds": 1.9
            },
            "listenTime": {
                "percentage": 96.678,
                "seconds": 55.299
            },
            "overlap": {
                "overlapDuration": 1.9,
                "overlappingMembers": [
                    {
                        "id": "acc63fbe-c3cd-4daa-8ab0-b088142e5a0f",
                        "name": "Speaker 1",
                        "percent": 7.51,
                        "seconds": 1.9
                    }
                ]
            }
        },
        {
            "id": "a52def45-be6e-484f-908b-9ac56eaecabb",
            "name": "Speaker 4",
            "pace": {
                "wpm": 152
            },
            "talkTime": {
                "percentage": 25.873,
                "seconds": 14.799
            },
            "listenTime": {
                "percentage": 74.127,
                "seconds": 42.4
            },
            "overlap": {
                "overlapDuration": 4.66,
                "overlappingMembers": [
                    {
                        "id": "acc63fbe-c3cd-4daa-8ab0-b088142e5a0f",
                        "name": "Speaker 1",
                        "percent": 12.199,
                        "seconds": 4.66
                    }
                ]
            }
        }
    ]
}

HTTP REQUEST

GET https://api.symbl.ai/v1/conversations/{conversationId}/analytics

Response Object

Fields	Description
type	metrics sub-type
id	unique user identifier
name	user name
email	user email
wpm	words per minutes for each speaker
talkTime	individual speaker’s total talk time
listenTime	individual speaker’s total listen time
overlap	member wise talk overlap details

GET Entities Beta

Entities API provides you with a functionality to extract entities(custom, location, person, date, number, organization,datetime,daterange, etc ) from the conversation.

API Endpoint

https://api.symbl.ai/v1/conversations/{conversationId}/entities

Example API call

curl "https://api.symbl.ai/v1/conversations/{conversationId}/entities" \
    -H "x-api-key: <api_token>"

const request = require('request');
const your_auth_token = '<your_auth_token>';

request.get({
    url: 'https://api.symbl.ai/v1/conversations/{conversationId}/entities',
    headers: { 'x-api-key': your_auth_token },
    json: true
}, (err, response, body) => {
    console.log(body);
});

The above request returns a response structured like this:

Custom Entity

{
            "type": "custom",
            "customType": "Company Executives",
            "value": "marketing director",
            "text": "marketing director",
            "messageRefs": [
                {
                    "id": "5118221462011904",
                    "text": "The marketing director is out-of-town due to some work.",
                    "offset": 4
                }
            ]
}

Person Entity

{
  "type": "person",
  "value": "jamie smith",
  "text": "jamie smith",
  "messageRefs": [
{
"id": "5979280332816384",
"text": "The name is Jamie Smith.", "offset": 12
} ]
}

Organization Entity

{
  "type": "organization",
  "value": "Vodafone",
  "text": "Vodafone",
  "messageRefs": [
{
"id": "6141473464516608",
"text": "Hello, this is Peter from Vodafone, I help you today.", "offset": 26
} ]
}

Date Entity:

{
"type": "date",
"value": "2020-07-15",
"text": "today",
"messageRefs": [
{
"id": "6141473464516608",
"text": "Hello, this is Peter from Vodafone, I help you today.", "offset": 47
},
{
"id": "4603163403354112",
"text": "Being a loyal customer at the three types of plan options that I can offer you today.",
"offset": 79
},
{
"id": "5936512994639872",
"text": "Is there anything else I may assist you with today?",
"offset": 45
}
]
}

Number Entity

{
            "type": "number",
            "value": "two",
            "text": "two",
            "messageRefs": [
                {
                    "id": "6137109238775808",
                    "text": "What do you two think of that?",
                    "offset": 12
                }
            ]
}

HTTP REQUEST

GET https://api.symbl.ai/v1/conversations/{conversationId}/entities

Note:

In Async Text custom entities needs to be passed in API body.
In Async Audio/Async Video API needs to be passed in Query Parameter.

Response Object

Field	Description
type	Defines the type of entity present.
value	The value of entity.
text	The text string matched by the algorithm
messageRefs	Contains message references like id, text and offset of entity in message.
customType	Optional. Show the custom entity type which was defined.

Experience API

POST Create Experience

This API returns the URL of the different Summary UIs depending upon the user request body.

Example API Call


# Video Summary
curl --location --request POST 'https://api.symbl.ai/v1/conversations/{conversationId}/experiences' \
--header 'Content-Type: application/json' \
--header 'x-api-key: <generated_valid_token>' \
--data-raw '{
    "name": "video-summary",
    "videoUrl": "https://storage.googleapis.com/rammer-transcription-bucket/small.mp4",
    "logo": "https://symblsanitydataset.s3.us-east-2.amazonaws.com/googleImage.webp"

}'

# Verbose text summary
curl --location --request POST 'https://api.symbl.ai/v1/conversations/{conversationId}/experiences' \
--header 'Content-Type: application/json' \
--header 'x-api-key: <generated_valid_token>' \
--data-raw '{
    "name": "verbose-text-summary",
    "logo": "https://symblsanitydataset.s3.us-east-2.amazonaws.com/googleImage.webp"
}'

> Video Summary
const request = require('request');
const your_auth_token = '<your_auth_token>';

request.post({
    url: 'https://api.symbl.ai/v1/conversations/{conversationId}/experiences',
    headers: {
      'x-api-key': your_auth_token,
      'Content-type': 'application/json',
    },
    body: JSON.stringify({
      "name": "video-summary",
      "videoUrl": "https://storage.googleapis.com/rammer-transcription-bucket/small.mp4",
      "logo": "https://symblsanitydataset.s3.us-east-2.amazonaws.com/googleImage.webp"
    }),
}, (err, response, body) => {
    console.log(body);
});

The above request returns a response structured like this:

Response for verbose-text-summary

{
    "name": "verbose-text-summary",
    "url": "https://meetinginsights.symbl.ai/meeting/#/eyJzZXNzaW9uSWQiOiI0NTMyNTY2NDc2NDU1OTM2In0="
}

Response for video-summary

{
    "name": "video-summary",
    "url": "https://meetinginsights.symbl.ai/meeting/#/eyJzZXNzaW9uSWQiOiI1ODU5NjczMDg1MzEzMDI0IiwidmlkZW9VcmwiOiJodHRwczovL3N0b3JhZ2UuZ29vZ2xlYXBpcy5jb20vcmFtbWVyLXRyYW5zY3JpcHRpb24tYnVja2V0L3NtYWxsLm1wNCJ9?showVideoSummary=true"
}

HTTP REQUEST

POST https://api.symbl.ai/v1/conversations/{conversationId}/experiences

Request Headers

Method	Path	Value
`x-api-key`	true	"Valid Token"
`Content-Type`	true	application/json

Request Body

Field	Required	Type	Supported Values	Description
`name`	true	string	verbose-text-summary, video-summary	For Summary UI use 'verbose-text-summary' and for Video Summary UI use 'video-summary'.
`videoUrl`	false	string		This field is only required when the field 'name' is set to 'video-summary'.
`logo`	false	string		This field accepts public URL for setting custom logo in Video Summary UI(video-summary) or Summary UI(verbose-text-summary).
`favicon`	false	string		This field accepts public URL for setting custom favicon in Video Summary UI(video-summary) or Summary UI(verbose-text-summary).

Video Summary UI

The Video Summary UI provides users the ability to interact with the Symbl elements(transcripts section, Insights, Filters) from a audio and video. It surfaces a screen where users can select key elements like topics, transcripts, and insights and the interface will surface the timestamp where this occurred and begin playback from there.

Features currently supported:

1. Auto scroll: The transcript auto scrolls while the video plays:

Video Summary UI

2. Video Playback: You can begin playback from a particular timestamp in the transcript by clicking on it:

Video Summary UI

3. Transcript Navigation: Clicking on an Insight takes you to the location related to the insight in the transcript and begins autoplay of the video from there: Video Summary UI

4.Topic Highlights: Selecting topics highlights the topics in the Transcript and generated Insights. In the search bar you will be able to toggle to other Topics along with the ability to autoplay the video from those points as well:

Video Summary UI

5. Audio file : This is supported by providing an audio file’s url to the videoUrl param. When an audio file is used, you will be provided a slightly different user experience -- while playing, the current speaker’s name, if available, is displayed where the video would otherwise be. Current audio formats supported are: mp3 and wav. Video Summary UI

How to Enable the Video Summary UI:

You need to add a query-parameter to the existing summary URL:

&showVideoSummary=true

What if there is no video in my url ?

Video Summary UI

Localized Summary UI

The Localized Summary UI provides users a translated meeting summary page chosen based on one of eight currently supported languages chosen when a session is initiated.

A timezone may also be specified that will change how the start time of the session is rendered in the summary UI.

Summary UI Comparison

German Summary UI

Localized Summary UI

Japanese Summary UI(no insights or topics)

Localized Summary UI

The language code specified will also change the language of the follow-up email after a session has ended.

Testing

Now that you've built out an integration using either the Voice SDK or Voice API, let's test to make sure your integration is working as expected.

If you are dialed in with your phone number, try speaking the following sentences to see the generated output

"Hey, it was nice meeting you yesterday. Let's catch up again next week over coffee sometime in the evening. I would love to discuss the next steps in our strategic roadmap with you."
"I will set up a meeting with Roger to discuss our budget plan for the next quarter and then plan out how much we can set aside for marketing efforts. I also need to sit down with Jess to talk about the status of the current project. I'll set up a meeting with her probably tomorrow before our standup."

2. If you are dialed into a meeting, try running any of the following videos with your meeting platform open and view the summary email that gets generated:

3. Try tuning your summary page with query parameters to customize your output.

Errors

// example auth token is incorrect
{
    "message": "Token validation failed for provided token."
}

Symbl uses the following HTTP codes:

Error Code	Meaning
200	OK -- Success.
201	Accepted -- Your request is successfully accepted.
400	Bad Request -- Your request is invalid.
401	Unauthorized -- Your API key is invalid.
403	Forbidden
404	Not Found -- The specified resource does not exist.
405	Method Not Allowed -- You tried to access an api with an invalid method.
413	Request Entity Too Large.
429	Too Many Requests -- Too many requests hit the API too quickly.
500	Internal Server Error -- We had a problem with our server. Try again later.

Resources

Async Audio Conversion

The below snippet shows how you can convert a file from .mp4 to .mp3 using the fluent-ffmpeg node module

const ffmpeg = require('fluent-ffmpeg');

ffmpeg('/my/path/to/original/file.mp4')
    .format('mp3')
    .on('end', () => {
        console.log('end');
    })
    .on('error', (err) => {
        // error
    })
    .save('/path/to/output/file.mp3');

Async Audio API supports files of either .wav or .mp3 and the file must have mono-channel audio only. Any other file formats can be converted using the code snippet from FFmpeg

$ npm install --save fluent-ffmpeg

Automatic Speech Recognition (ASR) Evaluation

This is a simple utility to perform a quick evaluation on the results generated by any Speech to text (STT) or Automatic Speech Recognition (ASR) System.

This utility can calculate following metrics:

Word Error Rate (WER), which is a most common metric of measuring the performance of a Speech Recognition or Machine translation system
Levenshtein Distance calculated at word level.
Number of Word level insertions, deletions and mismatches between the original file and the generated file.
Number of Phrase level insertions, deletions and mismatches between the original file and the generated file.
Color Highlighted text Comparison to visualize the differences.
General Statistics about the original and generated files (bytes, characters, words, new lines etc.)

Installation

$ npm install -g speech-recognition-evaluation

Usage

Simplest way to run your first evaluation is by simply passing original and generated options to asr-eval command. Where, original is a plain text file containing original transcript to be used as reference; usually this is generated by human beings. And generated is a plain text file containing generated transcript by the STT/ASR system.

$ asr-eval --original ./original-file.txt --generated ./generated-file.txt

For more information please visit this.

Media convertor

You can quickly transcode an audio file using transcodeMediaFile method.

const {transcodeMediaFile} = require('symbl-media');
(async () => {
    try {
        const result = await transcodeMediaFile('./my-input-file.wav', 'my-output-file.mp3', 'mp3');
        console.log('Successfully transcoded to: ', result.outPath);
    } catch (e) {
        console.error(e);
    }
})();

Currently this utility only supports one feature:

Transcode Audio file

This utility can be used as a library in your NodeJS code. You can simply install it in your local project.

$ npm install symbl-media --save

Use the transcode command to transcode the file.

$ media transcode -i ./my-input-file.wav -o ./my-output-file.mp3 -f mp3

For more information please visit this.

Overview

Symbl is an API platform for developers and businesses to rapidly deploy conversational intelligence at scale – on any channel of communication. Our comprehensive suite of APIs unlock proprietary machine learning algorithms that can ingest any form of conversation data to identify actionable insights across domains, timelines, and channels (voice, email, chat, social) contextually – without the need for any upfront training data, wake words, or custom classifiers.

What is conversational intelligence?

In its pure form, conversation intelligence refers to the ability to communicate in ways that create a shared concept of reality. It begins with trust and transparency to remove biases in decisions, enable participants, such as knowledge workers, to be more effective at their core function, eradicate mundane and repetitive parts of their work and empower participants both at work and beyond.

Here at Symbl.ai, we are using hybird approaches of machine learning and deep learning to augment human capability by analyzing conversations and surface knowledge and actions that matter.

How is Symbl.ai different from chatbot platforms?

In short: chatbots are intent-based, rule-based, often launched by ‘wake words’, and enable short conversations between humans and machines.

Symbl.ai is a developer platform and service capable of understanding context and meaning in natural conversations between humans. It can surface the things that matter in real-time, e.g. questions, action items, insights, contextual topics, signals, etc.

Additionally:

Chatbots or virtual assistants are commonly command-driven and often referred to as conversation AI systems. They add value to direct human-machine interaction via auditory or textual methods, and attempt to convincingly simulate how a human would behave in a conversation.

You can build chatbots by using existing intent-based systems like RASA, DialogFlow, Watson, Lex, etc. These systems identify intent based on the training data you provide, and these systems enable you to create rule-based conversation workflows between humans and machines.

We are building a platform that can contextually analyze natural conversations between two or more humans based on the meaning as opposed to keywords or wake words. We are also building it using models that require no training, so you can analyze conversations on both audio or text channels to get recommendations of outcomes without needing to train a custom engine for every new intent.

Next: explore supported use cases

Use Cases

Using Symbl you can build use cases for support, sales, collaboration apps and for workflow automation for single or multiple conversations to identify real-time growth opportunities, create indexed knowledge from conversations and drive productivity.

Meetings & UCaaS

Applying primarily to unified communication and collaboration platforms (UCaaS), you can add real-time recommendations of action items and next steps as part of your existing workflow. This would meaningfully improve meeting productivity by surfacing the things that matter, as the meeting occurs. Beyond real-time action items, take advantage of automated meetings summaries delivered to your preferred channel, like email, chat, Slack, calendar, etc.

Use real-time contextual recommendations to enable participants to drive efficiencies in their note-taking, save time and focus more on the meeting itself. Action items are surfaced contextually and in real-time and can be automated to trigger your existing workflows.

Post-meeting summaries are helpful for users that like to get more involved in the conversation as it happens, and prefer re-visiting information and action items post-meeting.

Benefits:

Humans tend to be slightly subjective when taking notes. Remove bias from notes with an objective conversation intelligence service that contextually surfaces what matters.
Humans are actually not that great at multitasking. Increase participation and engagement by adding a highly accurate note-taking AI service to the meeting.
Humans share… most of it. Access and search through complete meeting transcripts, meeting notes, action items, summaries, insights, contextual topics, questions, signals, etc.
Humans create culture. Understand patterns and trends in your organization’s meeting culture - sentiment, talk ratios, most productive conversations, etc.

Customer Care & CCaaS

As we understand it, customer care performance can be measured by 3 proxy metrics: customer satisfaction, time spent on call, and the number of calls serviced.

What if the introduction of a real-time passive conversation intelligence service into each call was to improve all 3 metrics at once? Real-time contextual understanding leads to suggested actions that a customer care agent can act upon during the call, enabling the agent to:

Focus on the human connection with the customer.
Come to a swifter resolution thanks to task automation
Serve more customers with elevated experience during a shift.

Further, the Symbl.ai platform is also capable of automating post-call data collection. This enables analysis of support conversations over time, agents, shifts,and groups, which leads to a better understanding of pain-points, topics of customer support conversation, etc.

Benefits: Support Organization

Improved customer experience thanks to more engaged support conversations.
Reduce average call handling time thanks to automated real-time actions.
Better data for coaching and benchmarking support staff.
High-level understanding of topics and summaries of support conversation.
Emotional analysis of conversational data.

Sales Enablement & CRM

Digital communication platforms used for sales engagements and customer interactions need to capture conversational data for benchmarking performance, improve net sales, and for identifying and replicating the best-performing sales scripts.

Use Symbl.ai to identify top-performing pitches by leveraging real-time insights. Accelerate the sales cycle by automating suggested action items in real-time, such as scheduling tasks and follow-ups via outbound work tool integrations. Keep your CRM up to date by automating the post-call entry with useful summaries.

Benefits: Sales Agent

Real-time suggested actions.
Real-time analysis & insights from the conversation.
Auto-scheduling tasks and follow-ups through outbound work tool integrations.
Full, searchable transcript of sales conversations.
Automate the post-call entry into your CRM.

Benefits: Sales Enablement / VP of Sales

A high-level performance view of the sales function.
Customizable dashboard to view calls in different filters.
Understand what works best in a sales call: topics, questions, competitor mentions, etc.
Replicate best performing scripts to train and coach your whole team to success.

Customers interact a lot with Brands on social media and other digital channels. These interactions include feedback, reviews, complaints, and a lot of other mentions. This is valuable data if used properly to derive insights for the business.

Symbl's APIs can be used along with social listening tools to extract, categorize all of this into actionable insights. For example, topics can be very helpful in abstracting data from product reviews, threads of conversation, and social media comments. Questions and requests from social interactions and forums can be identified to build a knowledge base and direct the customer conversations to the right resources.

With the right integrations to CRM tools and Knowledgebase, insights from social conversations can lead to better understanding customer sentiment towards the brand and more efficient customer service on social channels.

Benefits for Brands

Extract topics from reviews based on different levels of ratings and identify what leads to good/bad ratings.
Evaluate influencers, affiliates to work with the brand and ensure right messaging throughout the campaign.
Understand customer voice from comments and live interactions on Facebook, youtube channels.
Identify and document questions and requests from specific customers on product forums, comments, and replies to social media posts.
Guide customers to relevant knowledge base articles or support streams based on their complaints and queries on social media.
Enrich customer data on CRM based on insights identified from customer-specific social interactions.

Next: Learn more about the capabilities of the platform

Capabilities

Transcript

The platform provides a searchable transcript with word level timestamps and speaker information. The transcript is a refined output of the speech-to-text conversion.

The transcript is one of the easiest ways to navigate through the entire conversation. It can be sorted using speaker-specific or topic-specific filters. Additionally, each insight or action item can also lead to related parts of the transcript.
Transcripts can generated in both real-time and asynchronous manner for voice and video conversations. They can also be accessed through the pre-built post-conversation summary UI.

The pre-built UI enables editing, copying and sharing of transcripts from the conversation and can be enhanced to support the needs of desired user experience.

Summary Topics

Summary topics provide a quick overview of the key things that were talked about in the conversation.

Topic Hierarchy Beta

Topic hierarchy understands customer conversation and returns the parent (global) topic and it's child topics.

Parent Topic: Identify the highest level abstraction of a meeting. These are the key points on which the speakers of the meeting expanded and discussed at length.
Child topic: These are the subtopics that aggregate or are originated from the parent topic itself. Child Topics are the topics which are linked to the parent topic as they form the time chunks of the parent topics in a certain way.

Request object

Query Params

Parameter	Required	Value	Description
`parentRefs`	No	true	Gives you topic hierarchy when passed in Topic API.

Response object

Field	Description
`parentRefs`	For each topic in the response, it has a parentRefs field which point to its parent topic. If parentRefs is NULL it signifies that the topic is a parent.

Sample Response

{
    "topics": [
        {   //Child Topic
            "id": "5907389282779136",
            "text": "salesforce",
            "type": "topic",
            "score": 0.7178597920690242,
            "messageIds": [
                "4600982711304192",
                "5487363432120320",
                "6109794119188480"
            ],
            "parentRefs": [
                {
                    "type": "topic",
                    "text": "sales conversation"
                }
            ]
        },
        {   //Parent Topic
            "id": "6697188878974976",
            "text": "sales conversation",
            "type": "topic",
            "score": 0.6968750176932417,
            "messageIds": [
                "5356560840654848",
                "5663440783802368",
                "5263998490509312",
                "6082396449406976",
                "4925138187321344",
            ],
            "parentRefs": []
        }
    ]
}

Topic Hierarchy on a Conversation:

In sample diagram below that we created for your understanding. This shows how one conversation can have multiple parent (global) topics and each parent topic can have multiple child-topics in it.

For example, if in a meeting “Sales Conversation” was talked about and after it “Activity Call logs”, “Salesforce”, “Draft”, “Custom Integration” and “Jira” were talked.
So, the topic hierarchy algorithm will find a pattern in the conversation and make Sales Conversation as the parent topics and the rest of the topics as the child topics under it.

Topic Hierarchy

Sentiment AnalysisBeta

Sentiment analysis is the interpretation of the general thought, feeling, or sense of an object or a situation.

Sentiment Analysis works over sentence and topic(aspect) level. For topic level, the sentiment is a calculate over the topic messages scope i.e. it factors in the sentiment of messages where the topic was talked about.

Response object

Field	Description
`polarity`	show the intensity of the sentiment. It ranges from -1.0 to 1.0, where -1.0 is the most negative sentiment and 1.0 is the most positive sentiment.
`suggested`	display suggested sentiment type (negative, neutral and positive).

Suggested sentiment

polarity	Suggested Sentiment
-1.0 => x > -0.3	negative
-0.3 => x <= 0.3	neutral
0.3 < x <= 1.0	positive

Action Items

An action item is a specific outcome recognized in the conversation that requires one or more people in the conversation to take a specific action, e.g. set up a meeting, share a file, complete a task, etc.

Action Item Features

Recognition of the assignee and assignor of an action item when possible.
Recognition of the date and time specified in the insights.
Make use of the speaker context to enhance the quality of the insights.
Ability to accept the timezone to calculate the accurate date and time references.
Ability to control the confidence threshold for the insights.
Built-in punctuation and sentence boundary detection.
There are various types of actionable parts in the conversation between people and the platform can recognize these various connotations.

Tasks

Definitive action items that are not follow-ups are categorized as tasks.

Example:
"I will complete the presentation that needs to be presented to the management by the end of today". Here, a person is really committed to completing the presentation (task) by the end of today.

conversationType

sales: This conversationType optimizes Symbl algorithm to optimize Sales specific action-items in a meeting.
It focuses on 3 things : a very specific vocabulary used in Sales, the tense of the statement being made, and the presence of clear subjects, objects and verbs.
Example:
- I will send out that document after this call is over.
- As the next step, I will set up a follow up meeting for next week.

Follow Ups

The platform can recognize if an action item has a connotation, which requires following up in general or by someone in particular.

Examples:

"I will talk to my manager and find out the agreed dates with the vendor". Here, a person needs to follow up with their manager in order to complete this action.
"Perhaps I can submit the report today". Here, the action of submitting the report is indicated, but the overall connotation of it doesn't indicate the commitment.

Follow-ups can also be non-definitive

Example:

“We’ll need to sync up with the design team to find out more details”. Here, it’s clear that there needs to be a follow-up, but the details on when and how are not defined.

Type	Follow Up	Non-Follow Up
Non-Definitive	Follow Up (non defined)	Idea/Opinion
Definitive	Follow Up (defined data)	Task

Other Insight Types

Questions

Any explicit question or request for information that comes up during the conversation, whether answered or not, is recognized as a question.

Example:

“What features are most relevant for our use case?” “How are we planning to design the systems?”

Suggestive Actions

For each of the Action Items identified from the conversation, certain suggestive actions are recommended based on available worktool integrations. Action phrases within the action items can be also used to map to specific actions or trigger workflows based on custom business rules.

Example:

"Add to calendar" is recommended as a suggestive action for all follow up actions with recommendations for Jira.
Action Items can have an action phrase “create a Jira ticket” that can map to a Jira action.

Outbound Work Tool Integrations

The platform currently offers email as out-of-box integration with the NodeJS SDK and calendar integration on the pre-built post conversation summary UI. However, this can be extended to any work tool using extensible webhooks, where the actionable insights need to be pushed to enhance productivity and reduce the time taken by users to manually enter information from conversations. The same integrations can be enabled as suggestive actions to make this even quicker.

Some of the examples of these work tools that can be integrated using the extensible webhooks can be:

Sales platforms such as Salesforce, Copper
Task management solutions such as Trello, Google Tasks
Calendars
Project Management Tools such as Monday, Asana
Collaboration platforms such as Slack, Flock

Reusable and Customizable UI Components

The UI components can be widely divided into two areas:
1. Symbl JS Elements
2. Prebuilt Summary UI

Symbl JS Elements

Symbl JS elements helps developers embed customizable JS elements for transcription, insights and action items for both real-time and post conversation experience. These are customizable, embeddable components that can be used to simplify the process of building the experience with the desired branding, as applicable.

The Symbl JS elements are releasing in preview shortly, please send an email to devrelations@symbl.ai to get early access to the Symbl JS elements for:

Live captioning
Topics
Action Items
Suggestive Actions

Prebuilt Summary UI

Symbl provides a few prebuilt summary UI that can be used to generate a user experience of the understanding of the conversation after it has been processed. The pre-built summary UI is available as a URL that can be shared via email to all (or selected) participants or used to embed as a link as part of the conversation history within the application.

The prebuilt summary UI includes the following capabilities:

Title and details of the conversation including date, number of participants, etc.
Names of all the participants
Topics covered in the conversation in the order of importance
Full, searchable transcript of the conversation. Transcripts can be edited, copied and shared.
Any Insights, action items or questions from the transcript. Insights can also be edited, shared or dismissed, date/assignee for action item to be modified.
The prebuilt summary UI can also be customizable, as per the use case or product requirement.

Introduction

Getting Started

Postman

Explore Sample Apps

Symbl for Twilio Flex

Symbl for Zoom

Explore more

Authentication

How Tos

Get Live Transcription - Phone Call (Node.js) Telephony

Set up Symbl

Retrieve your credentials

Initialize SDK

Make a phone call

Subscribe to the Live Results

End the Call

Code

Test

Note

See also

How to use Symbl REST telephony API to get live transcription

Get Started

Retrieve your credentials

Authenticating

Calling phone or meeting using REST API

Connecting via websockets to get live transcriptions

How To get Live transcriptions and insights in a Telephone call using Symbl SDK

Realtime Output with PSTN Dialing using Voice SDK

Getting started

Insights

Actions

Getting connection Id.

Sending speaker events.

Realtime insights

How to access Symbl Node SDK through NextJS

Get Started

Retrieve your credentials

Create NextJS API routes

Using Symbl Node SDK

Symbl integration

Initializing sdk

startEndpoint

Usyng Symbl Node SDK

stopEndpoint

Subscribe to events from NextJS API route

Subscribing to Symbl realtime events

Server side summary

Client side setup

How to pass different audio codecs to Symbl endpoint

Getting started

How to push audio from NodeJS to Symbl and use symbl real-time API to subscribe to real-time data

Getting started

Initialize SDK

How To use different languages and timezones with Symbl

Getting started

How To Dial-in using Symbl Voice SDK, push speaker events and get summary URL

PSTN Dial-in using Voice SDK

Getting started

Connecting

Getting connection Id.

Sending speaker events.

Get real-time transcriptions with native WebSocket API

How to do text processing

Get Started

Retrieve your credentials

Authenticating

Using Symbl Text Async API

Text upload and processing

1. Get Relevant params for Video URL or file

2. Use Job API to poll for job status.

How to do audio processing

Get Started

Retrieve your credentials

Authenticating

Using Symbl audio Async API

File Upload and processing.

1. Transcoding audio file

2. Get Request parameters for either file or url

3. Use Job API to poll for job status.

Passing additional Async API parameters to get more granular insights

`startEndpoint`

`stopEndpoint`

The `email` identifier