Click to see new Symbl Docs!
Cleaner, prettier and easier to navigate.
Introduction
Base URL
https://api.symbl.ai/v1
Symbl is a comprehensive suite of APIs for analyzing natural human conversations - both for your team’s internal conversations and of course the conversations you are having with your customers. Built on our Contextual Conversation Intelligence (C2I) technology, the APIs enable you to rapidly incorporate human-level understanding that goes beyond simple natural language processing of voice and text conversations.
Get real-time analysis of free-flowing discussions like meetings, sales calls, support conversations, emails, chat, social conversations, transcripts, etc to automatically surface highly relevant summary topics, contextual insights, suggestive action items, follow-ups, decisions and questions - without the use of upfront training data, wake words or custom classifiers.
Getting Started
Postman
Here a quick Postman link for people who love to get their hands dirty.
Ready to test? When you've built out your integration, review our testing tips to ensure that your integration works end to end
Explore Sample Apps
Use any of the sample integrations as a starting point to extend your own custom applications
If you're interested in what all you can do with Symbl, check out our sample applications on GitHub that demonstrate how Symbl can be used to connect to Twilio Media Streams, Salesforce Dashboard, Outlook Calendar and many more.
Authentication
If you don't already have your app id or app secret, log in to the platform to get your credentials.
To invoke any API call, you must have a valid Access Token generated using the valid application credentials.
To generate the token using the appId
and appSecret
, the HTTP POST request needs to be made with these details.
POST https://api.symbl.ai/oauth2/token:generate
{
"type": "application",
"appId": "your_appId",
"appSecret": "your_appSecret"
}
curl -k -X POST "https://api.symbl.ai/oauth2/token:generate" \
-H "accept: application/json" \
-H "Content-Type: application/json" \
-d "{ \"type\": \"application\", \"appId\": \"<appId>\", \"appSecret\": \"<appSecret>\"}"
const request = require('request');
const authOptions = {
method: 'post',
url: "https://api.symbl.ai/oauth2/token:generate",
body: {
type: "application",
appId: "<appId>",
appSecret: "<appSecret>"
},
json: true
};
request(authOptions, (err, res, body) => {
if (err) {
console.error('error posting json: ', err);
throw err
}
console.log(JSON.stringify(body, null, 2));
});
JavaScript code to generate the Access Token. The code should work with NodeJS 7+ and browsers. You will need to install request for this sample code.
$ npm i request
For a valid appId
and appSecret
combination, the success response will be returned like this.
{
"accessToken": "your_accessToken",
"expiresIn": 3600
}
accessToken
- Token to be used for authorization in X-API-KEY header.expiresIn
- Duration in seconds after which the accessToken expires.
For any invalid appId
and appSecret
combination, HTTP 401 Unauthorized
response code will be returned.
How Tos
We know the fastest deterrent to adopting new technology is friction. We’ve created a series of step by step ‘How tos’ to get you to value quickly and easily.
Get Live Transcription - Phone Call (Node.js) Telephony
Get the live transcription in your Node.js application by making a call to a valid phone number.
Making a phone call is also the quickest way to test Symbl’s Telephony API. It can make an outbound call to a phone number using a traditional public switched telephony network (PSTN), any SIP trunks, or SIP endpoints that can be accessed over the internet using a SIP URI.
Set up Symbl
To get started, you’ll need your account credentials and Node.js installed (> v8.x) on your machine.
We’ll use the Symbl module for Node.js in this guide. Make sure you have a Node project set up. If you don’t have one, you can set one up using npm init.
From the root directory of your project, run the following command to add symbl-node
in your project dependencies.
$ npm i --save symbl-node
Retrieve your credentials
Your credentials include your appId and appSecret. You can find them on the home page of the platform.
Initialize SDK
const {sdk, SpeakerEvent} = require("symbl-node");
sdk.init({
// Your appId and appSecret https://platform.symbl.ai
appId: 'your_appId',
appSecret: 'your_appSecret'
}).then(async () => {
console.log('SDK initialized.');
try {
// You code goes here.
} catch (e) {
console.error(e);
}
}).catch(err => console.error('Error in SDK initialization.', err));
In a new file called index.js in the project, add the following lines to import and initialize the SDK. Replace the appId and appSecret in the code with yours.
Make a phone call
The quickest way to test the Telephony API is to make a phone call to any valid phone number. The Telephony API only works with phone numbers in the U.S. and Canada.
Your application uses the SDK to:
- make REST call to Telephony API with phone number details and then Symbl makes the call
- subscribe to live results over a WebSocket connection
Once the call is accepted on the phone, the Telephony API starts receiving audio from the call.
Live transcription is streamed back to your application using the subscribed WebSocket connection.
const connection = await sdk.startEndpoint({
endpoint: {
type: 'pstn', // when making a regular phone call
// Replace this with a real phone number
phoneNumber: '1XXXXXXXXXX' // include country code, example - 19998887777
}
});
const {connectionId} = connection;
console.log('Successfully connected. Connection Id: ', connectionId);
To make a phone call, call the startEndpoint
with type
set to ‘pstn’ and a valid U.S./Canadian phone number phoneNumber
.
Subscribe to the Live Results
To get the live transcription, you need to subscribe to the connection to get the results. You need to call the subscribeToConnection
method in the SDK and pass the connectionId
and a callback method which will be called on for every new event including the (live) transcription.
// Subscribe to connection using connectionId.
sdk.subscribeToConnection(connectionId, (data) => {
const {type} = data;
if (type === 'transcript_response') {
const {payload} = data;
// You get live transcription here!!
process.stdout.write('Live: ' + payload && payload.content + '\r');
} else if (type === 'message_response') {
const {messages} = data;
// You get processed messages in the transcript here!!! Real-time but not live! :)
messages.forEach(message => {
process.stdout.write('Message: ' + message.payload.content + '\n');
});
} else if (type === 'insight_response') {
const {insights} = data;
// You get any insights here!!!
insights.forEach(insight => {
process.stdout.write(`Insight: ${insight.type} - ${insight.text} \n\n`);
});
}
});
When you use this API, you can use a callback function within the subscribeToConnection
function. You pass an object and declare the type of response you want in the type field.
transcript_response
type is used for low latency live transcription resultsmessage_response
type is used for processed transcription results in real-time. These are not the same astranscript_response
which are low latency but typically generate a few seconds after the processing and contain messages split by sentence boundaries.insight_response
is used for any insights detected in real-time
End the Call
To end the call, you should make a stopEndpoint
call. The following code stops the call after 60 seconds. Your business logic should determine when the call should end.
// Stop call after 60 seconds to automatically.
setTimeout(async () => {
const connection = await sdk.stopEndpoint({ connectionId });
console.log('Stopped the connection');
console.log('Conversation ID:', connection.conversationId);
}, 60000); // Change the 60000 with higher value if you want this to continue for more time.
The stopEndpoint
will return an updated connection
object which will have the conversationId
in the response. You can use conversationId
to fetch the results even after the call using the Conversation API.
Code
You can find the complete code used in this guide here.
Test
To verify and check if the code is working:
- Run your code:
$ node index.js
You should receive a phone call to the number you used in the
startEndpoint
call. Accept the call.Start speaking in English (default language) and you should see the live transcription added to the console in real-time.
The call should automatically end after 60 seconds. If you end it sooner and don’t invoke
stopEndpoint
, you will not receive theconversationId
. If you need to access the results generated in the call, you should invokestopEndpoint
even if it was ended without explicitly invokingstopEndpoint
prior to this point.
Note
The example above invokes stopEndpoint
after a fixed timeout of 60 seconds. This is for demonstrating the stop functionality and it is not the recommended method of implementation for your application. In a real implementation, you should invoke startEndpoint
and stopEndpoint
as needed by the business logic of your application, i.e when you would like Symbl to start processing and stop processing.
See also
Congratulations! You finished your integration with Symbl’s Telephony API using PSTN.
How to use Symbl REST telephony API to get live transcription
In this how to you'll learn to use Symbl REST telephony API to call on the phone or meeting URL and get live transcription using websockets
Get Started
Retrieve your credentials
Your credentials include your appId and appSecret. You can find them on the home page of the platform.
add credentials to .env
file filling in APP_ID
and APP_SECRET
variables.
Authenticating
When using REST API, we would need to pass auth token in header. For that we've created component ProtectedPage
. This component executes Symbl specific REST endpoint, to retrieve auth token and store it in context. Later on we can retrieve this token from the helper hook useAuth
This is how we would retrieve the token:
async function loginToSymbl() {
const response = await fetch('https://api.symbl.ai/oauth2/token:generate', {
method: 'POST',
headers: {
'Content-Type': 'application/json',
},
mode: 'cors',
body: JSON.stringify({
type: 'application',
appId: process.env.APP_ID,
appSecret: process.env.APP_SECRET,
}),
})
const json = await response.json()
console.log('Access Token is: ', json)
setToken(json.accessToken)
Later on in any place in application we can use const { token } = useAuth()
to get the token.
Calling phone or meeting using REST API
First of all what we need to do is to collect parameters from the user to send to Symbl Telephoni REST API, so Symbl will know how to initiate the call and how to process the conversation.
type
- Symbl supports two meetings types.PSTN
andSIP
phoneNumber
- Naturally we need to have a phone number to call, but in case ofSIP
type we will need to call url. Something likesip:124@domain.com
. For simplicity we will pass this data underphoneNumber
parameter.dtmf
- This will be meeting code, which we need to pass in case meeting has such a codesummaryEmail
- you can pass a list of emails which will get summary of the call email.insightTypes
- there are several insight types that we can pass to Symbl.
Collecting these parameters is basically a set of React controlled input fields. You can check component PhoneConfigurations.tsx
if you are curious how it's implemented.
This component have two different behaviours. One is calling api/call
route, that we've covered in (how-to-nextjs-node-sdk.md)[#how-to-access-symbl-node-sdk-through-nextjs-app-to-call-on-the-phone-or-meeting-url-and-get-live-transcription] guide. The other version of it is basically instead of calling api/call
with user entered parameters, to call
https://api.symbl.ai/v1/endpoint:connect
This will look like this:
const phoneNumberOrUri =
type === 'sip' ? { uri: _phoneNumber } : { phoneNumber: _phoneNumber }
const res = await fetch('https://api.symbl.ai/v1/endpoint:connect', {
method: 'POST',
headers: {
'x-api-key': token as string,
'Content-Type': 'application/json',
},
mode: 'cors',
body: JSON.stringify({
operation: 'start',
endpoint: {
type,
...phoneNumberOrUri,
dtmf,
},
insightTypes,
actions: [
{
invokeOn: 'stop',
name: 'sendSummaryEmail',
parameters: {
emails: summaryEmails.split(','),
},
},
],
data: {
session: {
name: 'Call from nextjs Phone(Client only)',
},
},
}),
})
When we will execute fetch
call to endpoint:connect
endpoint, you will get a phone call either on your phone or sip
url according to what you've chosen in UI.
What we also need to do is to actually get a response of api call and set our connectionId in the app. We will do that by calling connectionResponseHandler
callback function that we pass to PhoneConfigurations.tsx
component. There we will set our response as conversationData
in our app.
Connecting via websockets to get live transcriptions
Now when we've got our phone call, we need to have a way to get realtime insights on the ongoing conversation. What we will do is monitor conversationData
changes and if it changes, we will start our websocket.
useEffect(() => {
let ws: any = null
//Check if we have conversationData and if we haven't already started websocket connection
if (conversationData && !ws) {
ws = new WebSocket(
`wss://api.symbl.ai/session/subscribe/${connectionId}?access_token=${token}`
)
ws.onmessage = (event: MessageEvent) => {
const data = JSON.parse(event.data)
if (data.type === 'message_response') {
setMessages(data.messages)
}
if (data.type === 'transcript_response') {
setLiveTranscript(data)
}
setRealTimeData(data)
}
}
// cleanup method which will be called before next execution. in your case unmount.
return () => {
ws && ws.close
}
}, [conversationData])
Here we will subscribe to Symbl websocket url wss://api.symbl.ai/session/subscribe/${connectionId}?access_token=${token}
to get realtime insights on ongoing conversation.
We will get either transcript_response
type or message_response
type or insight_response
type. For example message_response
will look like this:
{
"type": "message_response",
"messages": [
{
"from": {},
"payload": {
"content": "Yeah, tell us sir.",
"contentType": "text/plain"
},
"duration": {
"startTime": "2020-10-22T15:32:14.500Z",
"endTime": "2020-10-22T15:32:16.000Z"
}
}
]
}
Now what is left is to render UI based on what data we get.
How To get Live transcriptions and insights in a Telephone call using Symbl SDK
Realtime Output with PSTN Dialing using Voice SDK
In this example let's walk through how to get the live transcription and insights events in a Telephone call.
The code is available here:
Getting started
Open .env
file and add your APP_ID, APP_SECRET, SUMMARY_EMAIL. You can get
APP_ID and APP_SECRET from
https://platform.symbl.ai
We will use symbl-node
package:
require('dotenv').config()
const {sdk} = require('symbl-node')
Let's start by initialising symbl-node
sdk:
await sdk.init({
appId: process.env.APP_ID,
appSecret: process.env.APP_SECRET,
basePath: 'https://api.symbl.ai',
})
Now let's start our connection:
const connection = await sdk.startEndpoint(endpointConfig)
First of all let's provide phone number and endpoint type:
endpoint: {
type: 'pstn',
phoneNumber: process.env.DEFAULT_PHONE_NUMBER,
},
In case you want to use sip
connection, you can use type: sip
and provide
SIP URI to dial-in to. This should be unique for an active call/meeting in your
system. You can also provide dtmf
code if you have one. You can find this code
on meeting platform invite. You can leave it blank if not connecting to meeting
platform
dtmf: "<code>",
type: 'sip',
uri: 'sip:124@domain.com'
You can also pass custom audioConfig
. If not provided, it uses PCMU with 800
sample rate. If you want to provide it, you can do it like so:
audioConfig: {
encoding: 'OPUS',
sampleRate: 48000
},
Insights
Symbl provide various insights from the call. Main insights categories are
question
and action_item
. In order to include insights in processing, you
need to specify them in configuration like so:
insightTypes: ['action_item', 'question'],
Actions
You can specify different actions to happen during the call. We will define just one:
actions: [
{
invokeOn: 'stop',
name: 'sendSummaryEmail',
parameters: {
emails: [process.env.SUMMARY_EMAIL], // Add valid email addresses to received email
},
},
],
When our connection will be stopped, Symbl will send summary mail provided in
.env
file
Let's also name our session by providing:
data: {
session: {
name: 'My Test Meeting',
},
},
Getting connection Id.
For subscribing to the data, we will need to use connectionId
unique to each
active connection. to get it you can simply retrieve it from connection
response:
const connectionId = connection.connectionId
Sending speaker events.
We can send different speaker events to our connection indicating that different speakers started speaking. That will give us more personalised insights and get better meeting summary
In our example we will do it by calling helper getScheduleEvent
function, that
we will review in a bit. We pass SpeakerEvent type to it by using
SpeakerEvent.types
enum from symbl-node
, passing user data and timestamp:
const scheduleEvent = getScheduleEvent(sdk, connectionId)
setTimeout(() => {
// Schedule all the events to be sent.
scheduleEvent(SpeakerEvent.types.startedSpeaking, users.john, 0)
scheduleEvent(SpeakerEvent.types.stoppedSpeaking, users.john, 5)
}, 1000)
We retrieve users just from the global array of users but in real world example that might be users data retrieved from database:
const users = {
john: {
userId: 'john@example.com',
name: 'John',
},
mary: {
userId: 'mary@example.com',
name: 'Mary',
},
}
In order to push event to our connection we will create an event like so:
const speakerEvent = new SpeakerEvent({
type: eventType,
user,
})
speakerEvent.timestamp = new Date().toISOString()
And push it using pushEventOnConnection
function provided by SDK:
sdk.pushEventOnConnection(connectionId, speakerEvent.toJSON(), (err) => {
if (err) {
console.error('Error during push event.', err)
} else {
console.log('Event pushed!')
}
})
Realtime insights
In order to get real time data, we need to subscribe to the connection. We can do so by using:
sdk.subscribeToConnection(connectionId, (data) => {})
data has type
field, so we will check for transcript_response
,
insight_response
and message_response
transcript_response
are our live transcriptions that we want to preview in the
console:
if (type === 'transcript_response') {
const {payload} = data
process.stdout.write('Live: ' + payload && payload.content + '\r')
}
Transcripts are changing all the time, but once they are processed into
reasonable message and not just words, we will get message_response
with
messages array:
if (type === 'message_response') {
const {messages} = data
messages.forEach((message) => {
process.stdout.write('Message: ' + message.payload.content + '\n')
})
}
And finally if there will be any question or action item during conversation,
you will get insight_response
:
if (type === 'insight_response') {
const {insights} = data
insights.forEach((insight) => {
process.stdout.write(`Insight: ${insight.type} - ${insight.text} \n\n`)
})
}
As you can see with this data there is a wide array of implementations you can do.
How to access Symbl Node SDK through NextJS
In this how to you'll learn to access Symbl Node SDK through NextJS app to call on the phone or meeting url and get live transcription
Symbl comes with REST api, so same thing we can get by calling specific endpoint on the client side, but there are some use cases that you would want to use particularly server side.For example if you want to do some business logic decisions based on the data received from symbl.
Get Started
Retrieve your credentials
Your credentials include your appId and appSecret. You can find them on the home page of the platform.
add credentials to .env
file filling in APP_ID
and APP_SECRET
variables.
Create NextJS API routes
Let's create /api/call.ts
file which will be our api endpoint. We will use it both for calling to our phone or ending the call.
NextJS API endpoints will reside under pages/api
and will have the following structure
export default (req: any, res: any) => {
res.statusCode = 200
res.setHeader('Content-Type', 'application/json')
const params = JSON.parse(req.body)
res.end(params)
}
we will get bunch of params that we will pass from the client and will pass to Symbl node sdk. One of such params will be start
param.
export default (req: any, res: any) => {
res.statusCode = 200
res.setHeader('Content-Type', 'application/json')
const {
phoneNumber,
dtmf,
type,
summaryEmails,
start,
connectionId,
insightTypes,
} = JSON.parse(req.body)
if (start) {
console.log('Calling ', phoneNumber)
startEndpoint(
{ phoneNumber, dtmf, type, summaryEmails, insightTypes },
onSpeechDetected
).then((connectionId: string) => {
console.log('Connection Started', connectionId)
res.end(JSON.stringify({ connectionId }))
})
} else {
if (connectionId) {
console.log('Connection Stopped', connectionId)
stopEndpoint(connectionId).then((connection) => {
res.end(JSON.stringify(connection))
})
}
}
}
We will use start
and connectionId
parameters to determine whether we are starting or ending the call.
Basically what happens is every time we start the call, we will get back the connectionId
from Symbl, that we will pass to the client. Then later if we want to end the call, we need to pass this connectionId
along with start: false
to end the call.
Using Symbl Node SDK
If you look at the code the most dominant part will be this one:
startEndpoint(
{ phoneNumber, dtmf, type, summaryEmails, insightTypes },
onSpeechDetected
)
That is actually a function that we will write, which will pass parameters to symbl node sdk and trigger onSpeechDetected
callback. This callback can be your custom business logic, but in our case we will simply log the response.
async function onSpeechDetected(data: any) {
const { type } = data
console.log('Response type:', type)
}
Symbl integration
Let's create a file called integrations/symbl/utils.ts
and create startEndpoint
and stopEndpoint
functions, but before we need to initialize our sdk
Initializing sdk
const sdkInit = async () => {
await sdk.init({
appId: process.env.APP_ID,
appSecret: process.env.APP_SECRET,
basePath: apiBase,
})
}
We are using .env
file in this app, so we need to add our APP_ID
and APP_SECRET
there.
startEndpoint
startEndpoint
will get several parameters from the client:
type
- Symbl supports two meetings types.PSTN
andSIP
phoneNumber
- Naturally we need to have a phone number to call, but in case ofSIP
type we will need to call url. Something likesip:124@domain.com
. For simplicity we will pass this data underphoneNumber
parameter.dtmf
- This will be meeting code, which we need to pass in case meeting has such a codesummaryEmail
- you can pass a list of emails which will get summary of the call email.insightTypes
- there are several insight types that we can pass to Symbl.
export const startEndpoint = async (
{
type,
phoneNumber,
dtmf,
summaryEmails,
insightTypes,
}: {
type: string
phoneNumber: string
dtmf: string
summaryEmails: string
insightTypes: string
},
callback: (data: any) => Promise<void>,
endCallAfterInSeconds = 300
): Promise<any> => {
//init our sdk
await sdkInit()
}
Usyng Symbl Node SDK
Now let's import symbl-node
import { sdk } from 'symbl-node'
And use startEndpoint
method by passing all the parameters we've got from the client.
const phoneNumberOrUri = type === 'sip' ? { uri: phoneNumber } : { phoneNumber }
const connection = await sdk.startEndpoint(
{
endpoint: {
type,
...phoneNumberOrUri,
dtmf,
},
intents,
insightTypes,
actions: [
{
invokeOn: 'stop',
name: 'sendSummaryEmail',
parameters: {
emails: summaryEmails.split(','),
},
},
],
data: {
session: {
name: `Live Intent Detection NextJS demo - ${phoneNumber}`,
},
},
},
callback
)
You can see here in the code, that we either pass phoneNumber
or uri
, based on whether we have SIP
or PSTN
configuration
When executing this function we will initiate the call and tell Symbl to analyse it. What will be returned is connectionId
in which we will be particalarly interested for both subscribing to events and stopping our call.
stopEndpoint
This function will get connectionId
and call sdk.stopEndpoint
function to stop the connection.
const stopEndpoint = async (connectionId: string) => {
console.log('Stopping connection for ' + connectionId)
try {
const connection = await sdk.stopEndpoint({
connectionId,
})
console.log('Summary Info:', connection.summaryInfo)
console.log('Conversation ID:', connection.conversationId)
return {
summaryInfo: connection.summaryInfo,
conversationId: connection.conversationId,
}
} catch (e) {
console.error('Error while stopping the connection.', e)
throw e
}
}
Subscribe to events from NextJS API route
Now we want to create an endpoint that we can call to subscribe for realtime events. For our example we will use socket.io
library to use websockets and emit changes from the server to the client.
Let's create a new endpoint by creating a file /api/subscribeToPhoneEvents.ts
We will use Socket.io
specific setup within our handler
if (!res.socket.server.io) {
const io = new Server(res.socket.server)
io.on('connection', (socket: any) => {
socket.on('subscribeToEvents', (msg: any) => {
subscribeToRealtimeEvents(msg.connectionId, (data) => {
socket.emit('realtimeEvent', data)
})
})
socket.on('endCall', (msg: any) => {
stopEndpoint(msg.connectionId)
})
})
res.socket.server.io = io
} else {
console.log('socket.io already running')
}
res.end()
So what we are doing here?
- Starting socket.io server if it doesn't exist
const io = new Server(res.socket.server)
- When client is connected, we initiate subscription:
socket.on('subscribeToEvents', (msg: any) => {
subscribeToRealtimeEvents(msg.connectionId, (data) => {
socket.emit('realtimeEvent', data)
})
})
Here we basically pass connectionId
that was created after our call to telephony API to helper function subscribeToRealtimeEvents
. We also pass a callback to trigger realtimeEvent
that will push the event and the data from the server to client via websockets.
Subscribing to Symbl realtime events
To subscribe to realtime events, we will create another helper function under integrations/symbl/utils.ts
export async function subscribeToRealtimeEvents(
connectionId: string,
handler: (data: any) => void
) {
console.log(`Subscribe to events of connection: ${connectionId}`)
await sdkInit()
sdk.subscribeToConnection(connectionId, handler)
}
Here we will instantiate sdk and call subscribeToConnection
method within NodeSDK.
Server side summary
At this point we both have /api/call
api route to initiate the call and api/subscribeToPhoneEvents
to subscribe to events using socket.io
Client side setup
In order to set up our client properly we need two parts.
- Creating
PhoneConfigurations
component - we won't dive into this React component, but the main idea of it is to get user input for the following parameters and call/api/call
endpoint with these params.
const params = {
type,
phoneNumber,
dtmf,
summaryEmails,
insightTypes,
}
- Subscribing based on
connectionId
. We've seen already that we are getting backconnectionId
fromapi/call
endpoint, so in order to subscribe to real time events we will use this:
useEffect(() => {
console.log('Connection Id:', connectionId)
if (connectionId) {
fetch('/api/subscribeToPhoneEvents', {
method: 'POST',
}).finally(() => {
socket = io()
socket.on('connect', () => {
console.log('connect')
socket.emit('subscribeToEvents', { connectionId })
})
socket.on('realtimeEvent', (data: any) => {
console.log('Real time event', data)
if (data.type === 'transcript_response') {
setLiveTranscript(data.payload.content)
}
if (data.type === 'message_response') {
const oldMsgs = messagesList.current
setMessages([...oldMsgs, ...data.messages])
}
if (data.type === 'insight_response') {
const oldInsights = insightsList.current
setInsights([...oldInsights, ...data.insights])
}
})
socket.on('disconnect', () => {
console.log('disconnect')
})
})
}
}, [connectionId])
As you can see in the code, we set different state variables based on different responses we are getting from socket.io realtimeEvent
message.
Response that we will get will be either transcript_response
type or message_response
type or insight_response
type. For example message_response
will look like this:
{
"type": "message_response",
"messages": [
{
"from": {},
"payload": {
"content": "Yeah, tell us sir.",
"contentType": "text/plain"
},
"duration": {
"startTime": "2020-10-22T15:32:14.500Z",
"endTime": "2020-10-22T15:32:16.000Z"
}
}
]
}
Now what is left is to render UI based on what data we get.
How to pass different audio codecs to Symbl endpoint
The code is available here:
Sometimes you need to pass different audio codecs when passing the audio. This example shows how to pass them. The ones currently supported are
OPUS:
- Supported Sample Rates -- 16000Hz, 24000Hz,48000Hz
- Both CBR (Constant Bitrate) and VBR (Variable Bitrate) are supported
- Support for in-band FEC
SPEEX:
- Supported Sample Rates -- 16000Hz
- VBR is not supported
LINEAR16
- Supported Sample Rates -- 44100Hz
MULAW
- Supported Sample Rates -- 8000Hz
Getting started
Open .env
file and add your APP_ID, APP_SECRET, SUMMARY_EMAIL. You can get
APP_ID and APP_SECRET from
https://platform.symbl.ai
we will use symbl-node
package.
require('dotenv').config()
const {sdk} = require('symbl-node')
Let's start by initialising symbl-node
sdk
await sdk.init({
appId: process.env.APP_ID,
appSecret: process.env.APP_SECRET,
basePath: 'https://api.symbl.ai',
})
Now after we initialised, we need to start the connection by running
const connection = await sdk.startEndpoint(endpointConfig)
So how do you pass custom codecs? It's as simple as passing custom audio config
endpoint: {
//*****************Custom Audio Config******************
audioConfig: {
encoding: 'OPUS',
sampleRate: 16000,
},
//******************************************************
type: 'pstn',
phoneNumber: process.env.DEFAULT_PHONE_NUMBER,
},
If you have a requirement to use a codec not included in the ones above or have any other queries, please drop an e-mail to support@symbl.ai
How to push audio from NodeJS to Symbl and use symbl real-time API to subscribe to real-time data
Getting started
This example runs on node server, so we will use symbl-node
package.
Open .env
file and add your APP_ID, APP_SECRET, SUMMARY_EMAIL. You can get
APP_ID and APP_SECRET from https://platform.symbl.ai
require('dotenv').config()
const {sdk} = require('symbl-node')
For demo purposes, we're using mic
to simply get audio from microphone and pass
it on to websocket connection
We use mic
module so make sure you check that all requirements for the package
are met. For instance we need sox
package or ALSA
tools to be installed.
mic
installation instructions: https://www.npmjs.com/package/mic#installation
on Mac it can be as simple as running brew install sox
const mic = require('mic')
const sampleRateHertz = 16000
const micInstance = mic({
rate: sampleRateHertz,
channels: '1',
debug: false,
exitOnSilence: 6,
})
Initialize SDK
(async () => {
try {
await sdk.init({
appId: process.env.APP_ID,
appSecret: process.env.APP_SECRET,
basePath: 'https://api.symbl.ai',
})
} catch (e) {}
})()
We will also need a unique id to associate to our symbl request. We will create
this Id using uuid
package
const id = uuid();
Now we can start the connection using sdk.startRealtimeRequest
. We need to
provide several important options to our realtime request.
insightTypes
- we need to provide which insights will be detected. Supported ones areaction_item
andquestion
.-
config
object with meeting title, confidence treshold and the sampleRate speaker
object to define who is the speaker. To distinguish between different speakers we also need to provideuserId
with valid email, so after meeting will end, we will receive summary emailhandlers
- these handlers are used to detect speach, messages and insights.onSpeechDetected
- This will return live transcription of the callonMessageResponse
- When processed messages are available, this callback will be calledonInsightResponse
- When Symbl detects an insight, this callback will be called.
So our complete code will look like this:
const connection = await sdk.startRealtimeRequest({
id,
insightTypes: ['action_item', 'question'],
config: {
meetingTitle: 'My Test Meeting',
confidenceThreshold: 0.7,
timezoneOffset: 480, // Offset in minutes from UTC
languageCode: 'en-US',
sampleRateHertz,
},
speaker: {
// Optional, if not specified, will simply not send an email in the end.
userId: 'valid@email.com', // Update with valid email
name: 'My name',
},
handlers: {
onSpeechDetected: (data) => {
console.log(JSON.stringify(data))
// For live transcription
if (data) {
const {punctuated} = data
console.log('Live: ', punctuated && punctuated.transcript)
}
},
onMessageResponse: (data) => {
// When a processed message is available
console.log('onMessageResponse', JSON.stringify(data, null, 2))
},
onInsightResponse: (data) => {
// When an insight is detected
console.log('onInsightResponse', JSON.stringify(data, null, 2))
},
},
})
How To use different languages and timezones with Symbl
This example shows how to use languages other than English and also how to pass
in timezone
in which the conversation is taking place. For languages:
Currently the following languages are supported
['en-US', 'en-AU', 'en-GB', 'fr-CA', 'fr-FR', 'de-DE', 'it-IT', 'nl-NL', 'es-US', 'ja-JP']
The above are all BCP-47 standard language codes and currently ONLY 1 should be
passed in the languages
array as shown below. Support for detecting multiple
languages in the same conversation will be added soon!
For timezone: Please refer to https://en.wikipedia.org/wiki/List_of_tz_database_time_zones for a list of timeZones
You can also use moment-timezone
package to obtain a list of timeZones like
the following const timeZones = moment.tz.names()
NOTE: If languages
is NOT passed in the startEndpoint
call the API will
fallback to 'en-US'. If timezone
is NOT passed the API will fall back to
'UTC'.
Getting started
Open .env
file and add your APP_ID, APP_SECRET, SUMMARY_EMAIL. You can get
APP_ID and APP_SECRET from
https://platform.symbl.ai
we will use symbl-node
package.
require('dotenv').config()
const {sdk} = require('symbl-node')
Let's start by initialising symbl-node
sdk
await sdk.init({
appId: process.env.APP_ID,
appSecret: process.env.APP_SECRET,
basePath: 'https://api.symbl.ai',
})
Now start your endpoint and provide language
and timezone
properties:
const connection = await sdk.startEndpoint({
...config,
languages: ['ja-JP'],
timezone: 'Asia/Tokyo',
})
How To Dial-in using Symbl Voice SDK, push speaker events and get summary URL
PSTN Dial-in using Voice SDK
In this example, it establishes a connection using a phone number through PSTN, to send speaker events, generate insights, and display a summary URL with the output. You can see how to initialize the Voice SDK, connect to the endpoint, push speaker events and get the summary URL.
Getting started
Open .env
file and add your APP_ID, APP_SECRET, SUMMARY_EMAIL. You can get
APP_ID and APP_SECRET from
https://platform.symbl.ai
Let's start by inintialising symbl-node
sdk
await sdk.init({
appId: process.env.APP_ID,
appSecret: process.env.APP_SECRET,
basePath: 'https://api.symbl.ai',
})
Connecting
const connection = await sdk.startEndpoint(endpointConfig)
First of all let's provide phone number and endpoint type:
endpoint: {
type: 'pstn',
phoneNumber: process.env.DEFAULT_PHONE_NUMBER,
},
In case you want to use sip
connection, you can use type: sip
and provide
SIP URI to dial-in to. This should be unique for an active call/meeting in your
system. You can also provide dtmf
code if you have one. You can find this code
on meeting platform invite. You can leave it blank if not connecting to meeting
platform
dtmf: "<code>",
type: 'sip',
uri: 'sip:124@domain.com'
You can also pass custom audioConfig
. If not provided, it uses PCMU with 800
sample rate. If you want to provide it, you can do it like so:
audioConfig: {
encoding: 'OPUS',
sampleRate: 48000
},
Getting connection Id.
To send speaker events we will need connectionId
unique to each active
connection. to get it you can simply retrieve it from connection response:
const connectionId = connection.connectionId
Sending speaker events.
We can send different speaker events to our connection indicating that different speakers started speaking. That will give us more personalised insights and get better meeting summary
In our example we will do it by calling helper getScheduleEvent
function, that
we will review in a bit. We pass SpeakerEvent type to it by using
SpeakerEvent.types
enum from symbl-node
, passing user data and timestamp
const scheduleEvent = getScheduleEvent(sdk, connectionId)
setTimeout(() => {
// Schedule all the events to be sent.
scheduleEvent(SpeakerEvent.types.startedSpeaking, users.john, 0)
scheduleEvent(SpeakerEvent.types.stoppedSpeaking, users.john, 5)
}, 1000)
We retrieve users just from the global array of users but in real world example that might be users data retrieved from database.
const users = {
john: {
userId: 'john@example.com',
name: 'John',
},
mary: {
userId: 'mary@example.com',
name: 'Mary',
},
}
In order to push event to our connection we will create an event like so
const speakerEvent = new SpeakerEvent({
type: eventType,
user,
})
speakerEvent.timestamp = new Date().toISOString()
And push it using pushEventOnConnection
function provided by SDK
sdk.pushEventOnConnection(connectionId, speakerEvent.toJSON(), (err) => {
if (err) {
console.error('Error during push event.', err)
} else {
console.log('Event pushed!')
}
})
Get real-time transcriptions with native WebSocket API
In this how to, you will learn how to get started with Symbl’s most accurate conversation analysis api, native WebSocket API.
const accessToken = await getAccessToken();
const symblEndpoint = 'wss://api.symbl.ai/v1/realtime/insights/' +
UNIQUE_MEETING_ID +
'?accessToken=' + SYMBL_ACCESS_TOKEN;
Create the WebSocket endpoint by tagging on a unique meeting identifier and a query param, accessToken to the url:
const uniqueMeetingId = btoa(EMAIL_ADDRESS)
const accessToken = await getAccessToken();
const symblEndpoint = 'wss://api.symbl.ai/v1/realtime/insights/' +
UNIQUE_MEETING_ID +
'?accessToken=' + accessToken;
Now that we have constructed the endpoint, let's create a new WebSocket!
const ws = new WebSocket(symblEndpoint);
Before we connect the WebSocket to the endpoint we first want to subscribe to its event listeners so we don’t miss any messages.
// Fired when a message is received from the WebSocket server
ws.onmessage = (event) => {
console.log(event);
};
// Fired when the WebSocket closes unexpectedly due to an error or lost connetion
ws.onerror = (err) => {
console.error(err);
};
// Fired when the WebSocket connection has been closed
ws.onclose = (event) => {
console.info('Connection to websocket closed');
};
Now we are ready to start the WebSocket connection.
const ws = new WebSocket(
ws = send(JSON.stringify(
{
type: 'start_reqest',
meetingTitle: 'Websockets How-to', // Conversation name
insightTypes: ['question', 'action_item'], // Will enable insight generation
config: {
confidenceThreshold: 0.5,
languageCode: 'en-US',
speechRecognition: {
encoding: 'LINEAR16',
sampleRateHertz: 44100,
},
speaker: {
userId: 'example@symbl.ai',
name: 'Example Sample',
}
});
To pass the audio stream to the WebSocket API, we first need to grant access to peripherals.
We can do this with the Navigator
API by accessing mediaDevices
and calling getUserMedia
.
const stream = await navigator.mediaDevices.getUserMedia({ audio: true, video: false });
Now that we have granted access to the microphone we can actually use the
WebSocket to handle the the WebSocket the data streamed to Symbl’s cloud
so transcripts and insights can be analyzed in real-time. We can do this by
creating a new AudioContext
and using the microphones stream we retrieved from the Promise resolution above to create a new source, and processor.
const handleSuccess = (stream) => {
const AudioContext = window.AudioContext;
const context = new AudioContext();
const source = context.createMediaStreamSource(stream);
const processor = context.createScriptProcessor(1024, 1, 1);
const gainNode = context.createGain();
source.connect(gainNode);
this.gainNode.connect(processor);
processor.connect(context.destination);
processor.onaudioprocess = (e: any) => {
// convert to 16-bit payload
const inputData = e.inputBuffer.getChannelData(0) || new Float32Array(this.bufferSize);
const targetBuffer = new Int16Array(inputData.length);
for (let index = inputData.length; index > 0; index--) {
targetBuffer[index] = 32767 * Math.min(1, inputData[index]);
}
// Send to websocket
if (ws.readyState === WebSocket.OPEN) {
ws.send(targetBuffer.buffer);
}
};
};
handleSuccess(stream);
Know more about Real-time-websocket.
How to do text processing
Get Started
Retrieve your credentials
Your credentials include your appId and appSecret. You can find them on the home page of the platform.
add credentials to .env
file filling in APP_ID
and APP_SECRET
variables.
Authenticating
When using REST API, we would need to pass auth token in header. For that we've created component ProtectedPage
. This component executes Symbl specific REST endpoint, to retrieve auth token and store it in context. Later on we can retrieve this token from the helper hook useAuth
This is how we would retrieve the token:
async function loginToSymbl() {
const response = await fetch('https://api.symbl.ai/oauth2/token:generate', {
method: 'POST',
headers: {
'Content-Type': 'application/json',
},
mode: 'cors',
body: JSON.stringify({
type: 'application',
appId: process.env.APP_ID,
appSecret: process.env.APP_SECRET,
}),
})
const json = await response.json()
console.log('Access Token is: ', json)
setToken(json.accessToken)
Later on in any place in application we can use const { token } = useAuth()
to get the token.
Using Symbl Text Async API
The Async Text API allows you to process any text payload to get the transcription and conversational insights. It can be useful in any use case where you have access to the textual content of a type of conversation, and you want to extract the insightful items supported by the Conversation API. If you want to add more content to the same conversation, use PUT Async Text API.
Symbl Text async API will return jobId
which we can monitor for completion by using polling mechanism. Once job is completed, we will get conversationId
which we can use to retrieve data with Conversation API.
Text upload and processing
Text async API is built for particular format of messages list that you can upload and parse to get conversational insights. The format should look something like that:
[
{
"payload": {
"content": "Hi Mike, Natalia here. Hope you don’t mind me reaching out. Who would be the best possible person to discuss internships and student recruitment at ABC Corp? Would you mind pointing me toward the right person and the best way to reach them? Thanks in advance for your help, I really appreciate it!"
},
"from": {
"userId": "natalia@example.com",
"name": "Natalia"
},
"duration": {
"startTime": "2020-07-21T16:02:19.01Z",
"endTime": "2020-07-21T16:04:19.99Z"
}
},
{
"payload": {
"content": "Hey Natalia, thanks for reaching out. I am connecting you with Steve who handles recruitements for us."
},
"from": {
"userId": "mike@abccorp.com",
"name": "Mike"
},
"duration": {
"startTime": "2020-07-21T16:04:19.99Z",
"endTime": "2020-07-21T16:04:20.99Z"
}
}
]
So we will have a textarea on the page where you can paste you content. Then in order to process it, you need to click Submit for processing
button. Once you do that, there are several things that happen
1. Get Relevant params for Video URL or file
You will see the following params defined within useEffect
const requestTextOptions = {
method: 'POST',
headers: {
'x-api-key': token,
'Content-Type': 'application/json',
},
body: JSON.stringify({
messages: JSON.parse(data),
}),
}
const requestOptions = {
method: 'GET',
headers: {
'x-api-key': token,
},
}
2. Use Job API to poll for job status.
Previously we mentioned that there should be some kind of polling mechanism to check whether the job is finished or not. Symbl has Job API That we can use for that.
async function check(jobId: string) {
const checkJob = await fetch(
`https://api.symbl.ai/v1/job/${jobId}`,
requestOptions
)
const checkJobJson = await checkJob.json()
setJobStatus(checkJobJson)
if (checkJobJson.status === 'in_progress') {
check(jobId)
return
} else {
setSentForProcessing(false)
}
}
So the flow will be:
// Execute the request
const processingResponse = await fetch(
'https://api.symbl.ai/v1/process/text',
requestTextOptions
)
const processingResponseJson = await processingResponse.json()
// Check Job Status
check(processingResponseJson.jobId)
After Job is finished we can get data from Conversation API. We will do that by using <ConversationCard/>
component.
On this page you will also see this component <Transcripts conversationId={conversationData.conversationId} />
This is prebuilt component from @symblai/react-elements package. As soon as you provide it with conversationId
, It will nicely render conversation transcripts. There is also <Topics conversationId={conversationData.conversationId}/>
component that will do the same but for topics.
How to do audio processing
Get Started
Retrieve your credentials
Your credentials include your appId and appSecret. You can find them on the home page of the platform.
add credentials to .env
file filling in APP_ID
and APP_SECRET
variables.
Authenticating
When using REST API, we would need to pass auth token in header. For that we've created component ProtectedPage
. This component executes Symbl specific REST endpoint, to retrieve auth token and store it in context. Later on we can retrieve this token from the helper hook useAuth
This is how we would retrieve the token:
async function loginToSymbl() {
const response = await fetch('https://api.symbl.ai/oauth2/token:generate', {
method: 'POST',
headers: {
'Content-Type': 'application/json',
},
mode: 'cors',
body: JSON.stringify({
type: 'application',
appId: process.env.APP_ID,
appSecret: process.env.APP_SECRET,
}),
})
const json = await response.json()
console.log('Access Token is: ', json)
setToken(json.accessToken)
Later on in any place in application we can use const { token } = useAuth()
to get the token.
Using Symbl audio Async API
Symbl Audio async API is used to process audio file or url and return jobId
which we can monitor for completion by using polling mechanism. Once job is completed, we will get conversationId
which we can use to retrieve data with Conversation API.
File Upload and processing.
If you look at pages/audio/index.tsx file, you will notice that for file upload we are using
<FileOrUrlInput onSubmit={onAudioSubmit} />
We won't dive into how it's implemented but basically the idea is to toggle between URL or file input. Once you click on the button with caption Submit for processing
, it will call onAudioSubmit
function passing either URL or file.
const onAudioSubmit = async (data: FileOrUrlInputSubmitType) => {
if (data.type === 'file') {
setFile(data?.file as any)
} else {
setAudioUrl(data?.url as string)
}
}
This will set our state.
All the processing is done within useAudioAsyncAPI
hook. This hook monitors the state change
const { jobStatus, sentForProcessing } = useAudioAsyncAPI(
audioUrl !== '' ? audioUrl : file,
asyncApiParams.query
)
and based on file/url change will start the following flow
1. Transcoding audio file
You will see the following function defined within useEffect
async function transcodeFile(_file: File) {
// tslint:disable-next-line
const transcoder = new (Transcoder as any)()
const transcodedFile = await transcoder.load(_file)
return transcodedFile
}
We will also create a new AbortController
to abort transcoding and also to send controller.signal
to Symbl API
const controller = new AbortController()
2. Get Request parameters for either file or url
We will also define a function to conditionally get different request parameters that we need to send in our REST call. After all there is a difference whether we send a file or URL
async function getFileOrUrlOptions() {
if (isFile) {
const file = data
const transcodedFile: any = await transcodeFile(file as File)
const requestOptionsAudio = {
method: 'POST',
headers: {
'x-api-key': token,
'Content-Type': MIME_TYPES['mp3'],
},
body: transcodedFile,
signal: controller.signal,
}
return requestOptionsAudio
} else {
const url = data
const requestOptionsAudio = await {
method: 'POST',
headers: {
'x-api-key': token,
'Content-Type': 'application/json',
},
body: JSON.stringify({
url: url,
confidenceThreshold: 0.6,
timezoneOffset: 0,
}),
}
return requestOptionsAudio
}
}
3. Use Job API to poll for job status.
Previously we mentioned that there should be some kind of polling mechanism to check whether the job is finished or not. Symbl has Job API That we can use for that.
async function check(jobId: string) {
const checkJob = await fetch(
`https://api.symbl.ai/v1/job/${jobId}`,
requestOptions
)
const checkJobJson = await checkJob.json()
setJobStatus(checkJobJson)
if (checkJobJson.status === 'in_progress') {
check(jobId)
return
} else {
setSentForProcessing(false)
}
}
So the flow will be:
//Transcode and get Parameters
const requestOptionsAudio = await getFileOrUrlOptions()
// Execute the request
const processingResponse = await fetch(urlAudio, requestOptionsAudio)
const processingResponseJson = await processingResponse.json()
// Check Job Status
check(processingResponseJson.jobId)
After Job is finished we can get data from Conversation API. We will do that by using <ConversationCard/>
component.
On this page you will also see this component <Transcripts conversationId={conversationData.conversationId} />
This is prebuilt component from @symblai/react-elements package. As soon as you provide it with conversationId
, It will nicely render conversation transcripts. There is also <Topics conversationId={conversationData.conversationId}/>
component that will do the same but for topics.
Passing additional Async API parameters to get more granular insights
How to do video processing
Get Started
Retrieve your credentials
Your credentials include your appId and appSecret. You can find them on the home page of the platform.
add credentials to .env
file filling in APP_ID
and APP_SECRET
variables.
Authenticating
When using REST API, we would need to pass auth token in header. For that we've created component ProtectedPage
. This component executes Symbl specific REST endpoint, to retrieve auth token and store it in context. Later on we can retrieve this token from the helper hook useAuth
This is how we would retrieve the token:
async function loginToSymbl() {
const response = await fetch('https://api.symbl.ai/oauth2/token:generate', {
method: 'POST',
headers: {
'Content-Type': 'application/json',
},
mode: 'cors',
body: JSON.stringify({
type: 'application',
appId: process.env.APP_ID,
appSecret: process.env.APP_SECRET,
}),
})
const json = await response.json()
console.log('Access Token is: ', json)
setToken(json.accessToken)
Later on in any place in application we can use const { token } = useAuth()
to get the token.
Using Symbl Video Async API
Symbl Video async API is used to process video file or url and return jobId
which we can monitor for completion by using polling mechanism. Once job is completed, we will get conversationId
which we can use to retrieve data with Conversation API.
File Upload and processing.
If you look at pages/video/index.tsx
file, you will notice that for file upload we are using
<FileOrUrlInput onSubmit={onVideoSubmit} />
We won't dive into how it's implemented but basically the idea is to toggle between URL or file input. Once you click on the button with caption Submit for processing
, it will call onVideoSubmit
function passing either URL or file.
const onVideoSubmit = async (data: FileOrUrlInputSubmitType) => {
setConversationData(null)
if (data.type === 'file') {
setFile(data?.file as any)
} else {
setVideoUrl(data?.url as string)
}
}
This will set our state.
All the processing is done within useAsyncVideoApi
hook. This hook monitors the state change
const { jobStatus, sentForProcessing } = useAsyncVideoApi(
videoUrl !== '' ? videoUrl : file,
asyncApiParams.query
)
and based on file/url change will start the following flow
1. Get Relevant params for Video URL or file
You will see the following function defined within useEffect
async function getFileOrUrlOptions() {
if (isFile) {
const file = data
const requestOptionsVideo = {
method: 'POST',
headers: {
'x-api-key': token,
'Content-Type': 'video/mp4',
},
body: file,
json: true,
}
return requestOptionsVideo
} else {
const url = data
const requestOptionsVideo = await {
method: 'POST',
headers: {
'x-api-key': token,
'Content-Type': 'application/json',
},
body: JSON.stringify({
url: url,
confidenceThreshold: 0.6,
timezoneOffset: 0,
}),
}
return requestOptionsVideo
}
}
Here we define a function to conditionally get different request parameters that we need to send in our REST call. After all there is a difference whether we send a file or URL
2. Use Job API to poll for job status.
Previously we mentioned that there should be some kind of polling mechanism to check whether the job is finished or not. Symbl has Job API That we can use for that.
async function check(jobId: string) {
const checkJob = await fetch(
`https://api.symbl.ai/v1/job/${jobId}`,
requestOptions
)
const checkJobJson = await checkJob.json()
setJobStatus(checkJobJson)
if (checkJobJson.status === 'in_progress') {
check(jobId)
return
} else {
setSentForProcessing(false)
}
}
So the flow will be:
//Transcode and get Parameters
const requestOptionsVideo = await getFileOrUrlOptions()
// Execute the request
const processingResponse = await fetch(urlAudio, requestOptionsVideo)
const processingResponseJson = await processingResponse.json()
// Check Job Status
check(processingResponseJson.jobId)
After Job is finished we can get data from Conversation API. We will do that by using <ConversationCard/>
component.
On this page you will also see this component <Transcripts conversationId={conversationData.conversationId} />
This is prebuilt component from @symblai/react-elements
package. As soon as you provide it with conversationId
, It will nicely render conversation transcripts. There is also <Topics conversationId={conversationData.conversationId}/>
component that will do the same but for topics.
Passing additional Async API parameters to get more granular insights
How to retrieve Conversation Data
Get Started
Retrieve your credentials
Your credentials include your appId and appSecret. You can find them on the home page of the platform.
add credentials to .env
file filling in APP_ID
and APP_SECRET
variables.
Authenticating
When using REST API, we would need to pass auth token in header. For that we've created component ProtectedPage
. This component executes Symbl specific REST endpoint, to retrieve auth token and store it in context. Later on we can retrieve this token from the helper hook useAuth
This is how we would retrieve the token:
async function loginToSymbl() {
const response = await fetch('https://api.symbl.ai/oauth2/token:generate', {
method: 'POST',
headers: {
'Content-Type': 'application/json',
},
mode: 'cors',
body: JSON.stringify({
type: 'application',
appId: process.env.APP_ID,
appSecret: process.env.APP_SECRET,
}),
})
const json = await response.json()
console.log('Access Token is: ', json)
setToken(json.accessToken)
Later on in any place in application we can use const { token } = useAuth()
to get the token.
Getting data from Conversation API
In order to get data from conversation API we will need to get conversationId
. We can either use a demo id: 6055920157065216
or we can initiate the phone call using either Symbl Node SDK or Symbl REST api. Both flows will update conversationId
parameter in the app and we will be able to retrieve data that we want.
In this demo app we have <ConversationCard/>
component that will accept the following parameters:
const ConversationCard = ({
title,
children,
conversationId: cId,
type,
}: {
title: string
conversationId: string
children: JSX.Element
type:
| 'messages'
| 'insights'
| 'topics'
| 'members'
| 'questions'
| 'follow-ups'
| 'action-items'
}) => {}
This card component can render different data retrieved from Conversation API. The functionality is rather simple. Based on different types, card will render different data retrieved from Conversation API.
const fetchData = async () => {
const res = await fetch(`${CONVERSATION_API}/${conversationId}/${endpoint}`, {
method: 'GET',
headers: {
'x-api-key': token as string,
'Content-Type': 'application/json',
},
mode: 'cors',
})
const json = await res.json()
setResponseData(json)
}
You can read more about conversation API here
Get Speaker Separated Transcripts - Diarization with Async API
Enable the Speaker Diarization (Speaker Separation) for the Async Audio or Async Video APIs to get speaker separated transcripts and insights.
Enabling the Diarization
Enabling Speaker Separation in the Async Audio/Video API is as simple as adding the enableSpeakerDiarization=true
and diarizationSpeakerCount=<NUMBER_OF_UNIQUE_SPEAKERS>
query-parameters below:
$ curl --location --request POST 'https://api.symbl.ai/v1/process/video/url?enableSpeakerDiarization=true&diarizationSpeakerCount=2&webhookUrl=<WEBHOOK_URL>' --header 'Content-Type: application/json' --header 'x-api-key: <X-API-KEY> --data-raw '{
"url": "https://storage.googleapis.com/demo-conversations/interview-prep.mp4"
}'
The above snippet shows a cURL command for consuming the Async Video URL based API which takes in the url for a publicly available URL of a Video File.
The
<X-API-KEY>
needs to be replaced with the token generated token:generate endpoint.The
<WEBHOOK_URL>
can be replaced with a WebHook URL for receiving the status for the Job created after calling the API.For accuracy, the
<NUMBER_OF_UNIQUE_SPEAKERS>
should match the number of unique speakers in the Audio/Video data.
The above URL has two query-parameters:
enableSpeakerDiarization=true
which will enable the speaker separation for the Audio/Video data under consideration.diarizationSpeakerCount=2
which sets the number of unique speakers in the Audio/Video data under consideration.
Getting the uniquely identified speakers (Members)
Invoking the members
call in the Conversation API will return the uniquely identified speakers for this conversation when Speaker Diarization is enabled. View a sample output below:
{
"members": [
{
"id": "9d6d34d9-5019-4694-9c9a-8ba7bfc8cfab",
"name": "Speaker 1"
},
{
"id": "2f69f1c8-bf0a-48ef-b47f-95ae5a4de325",
"name": "Speaker 2"
}
]
}
The name
assigned to a uniquely identified speaker/member from a Diarized Audio/Video will follow the format Speaker <number>
where <number>
is arbitrary and does not necessarily reflect in what order someone spoke.
The id
can be used to identify a speaker/member for that specific conversation and can be used to update the details for the specific member demonstrated below in the Updating Detected Members section.
Getting the Speaker Separated Results
Invoking the messages
call in the Conversation API would return the speaker separated results. View a snippet for the above URL below:
{
"messages": [
{
"id": "4591723946704896",
"text": "You're hired two words, everybody loves to hear.",
"from": {
"id": "2f69f1c8-bf0a-48ef-b47f-95ae5a4de325",
"name": "Speaker 2"
},
"startTime": "2020-08-04T07:18:17.573Z",
"endTime": "2020-08-04T07:18:21.573Z",
"conversationId": "5105430690791424"
},
{
"id": "6328236401229824",
"text": "But before we hear these words comes the interview today's video is part one in a series.",
"from": {
"id": "2f69f1c8-bf0a-48ef-b47f-95ae5a4de325",
"name": "Speaker 2"
},
"startTime": "2020-08-04T07:18:21.973Z",
"endTime": "2020-08-04T07:18:30.473Z",
"conversationId": "5105430690791424"
},
...
]
}
The above snippet shows the speaker in the from
object with a unique-id. These are the uniquely identified members
of this conversation.
Similarly, invoking the insights
call in the Conversation API would also reflect the identified speakers in the detected insights. The response below demonstrates this:
{
"insights": [
{
"id": "5501181057040384",
"text": "We need to go over three more common interview questions.",
"type": "action_item",
"score": 1,
"messageIds": [
"5710067261243392"
],
"entities": [],
"phrases": [
{
"type": "action_phrase",
"text": "go over three more common interview questions"
}
],
"from": {
"id": "2f69f1c8-bf0a-48ef-b47f-95ae5a4de325",
"name": "Speaker 2"
},
"definitive": true,
"assignee": {
"id": "2f69f1c8-bf0a-48ef-b47f-95ae5a4de325",
"name": "Speaker 2"
}
},
{
"id": "5519156904460288",
"text": "How did you hear about this position?",
"type": "question",
"score": 0.999988666660899,
"messageIds": [
"4616389407014912"
],
"from": {
"id": "2f69f1c8-bf0a-48ef-b47f-95ae5a4de325",
"name": "Speaker 2"
}
},
...
]
}
Updating the Detected Members
The detected members (unique speakers) would have names like Speaker 1
as the ASR wouldn’t have any context to who this speaker is (name or other details of the speaker). Therefore, it is important to update the details of the detected speakers after the Job
is complete
.
The members
call in the Conversation API returns the uniquely identified speakers as shown in the Getting the uniquely identified speakers (Members) section above when the Diarization is enabled.
Let’s consider the same set of members that can be retrieved by calling the GET members call in the Conversation API.
{
"members": [
{
"id": "9d6d34d9-5019-4694-9c9a-8ba7bfc8cfab",
"name": "Speaker 1"
},
{
"id": "2f69f1c8-bf0a-48ef-b47f-95ae5a4de325",
"name": "Speaker 2"
}
]
}
We can now use the PUT members
call to update the details of a specific member as shown below. This call would update the Speaker 2
as shown in the above section with the values in the cURL’s request-body
$ curl --location --request PUT 'https://api.symbl.ai/v1/conversations/<CONVERSATION_ID>/members/2f69f1c8-bf0a-48ef-b47f-95ae5a4de325' --header 'Content-Type: application/json' --header 'x-api-key: <X-API-KEY> --data-raw '{
"id": "2f69f1c8-bf0a-48ef-b47f-95ae5a4de325",
"email": "john@example.com",
"name": "John Doe"
}'
The
<CONVERSATION_ID>
needs to be replaced with the actualconversationId
The
<X-API-KEY>
needs to be replaced with the token generated with thetoken:generate
endpoint.
The URL in line 1 above has the id
of the member
we want to append to /members
with the request body containing the updated name
of this member
.
There is also the option to include the email
of the member. The email
will be used as an identifier for tracking those specific members uniquely in that conversation. (Refer the Appending to an existing conversation with Diarization section below for more details)
After the above call is successful, we will receive the following response:
{
"message": "Member with id: 2f69f1c8-bf0a-48ef-b47f-95ae5a4de325 for conversationId: <CONVERSATION_ID> updated successfully! The update should be reflected in all messages and insights along with this conversation"
}
The message
is self-explanatory and tells us that all the references to the member
with the id
of 2f69f1c8-bf0a-48ef-b47f-95ae5a4de325
in the conversation should now reflect the new values we updated this member
with. That includes insights
, messages
and the conversation’s members
as well.
So if we call the members
API now, we would see the following result:
{
"members": [
{
"id": "9d6d34d9-5019-4694-9c9a-8ba7bfc8cfab",
"name": "Speaker 1"
},
{
"id": "2f69f1c8-bf0a-48ef-b47f-95ae5a4de325",
"email": "john@example.com",
"name": "John Doe"
}
]
}
And similarly, with the messages
API call, we would see the updates reflected below as well:
{
"messages": [
{
"id": "4591723946704896",
"text": "You're hired two words, everybody loves to hear.",
"from": {
"id": "2f69f1c8-bf0a-48ef-b47f-95ae5a4de325",
"email": "john@example.com",
"name": "John Doe"
},
"startTime": "2020-08-04T07:18:17.573Z",
"endTime": "2020-08-04T07:18:21.573Z",
"conversationId": "5105430690791424"
},
{
"id": "6328236401229824",
"text": "But before we hear these words comes the interview today's video is part one in a series.",
"from": {
"id": "2f69f1c8-bf0a-48ef-b47f-95ae5a4de325",
"email": "john@example.com",
"name": "John Doe"
},
"startTime": "2020-08-04T07:18:21.973Z",
"endTime": "2020-08-04T07:18:30.473Z",
"conversationId": "5105430690791424"
},
...
]
}
Curious about the insights
API? It would reflect these updates as well!
{
"insights": [
{
"id": "5501181057040384",
"text": "We need to go over three more common interview questions.",
"type": "action_item",
"score": 1,
"messageIds": [
"5710067261243392"
],
"entities": [],
"phrases": [
{
"type": "action_phrase",
"text": "go over three more common interview questions"
}
],
"from": {
"id": "2f69f1c8-bf0a-48ef-b47f-95ae5a4de325",
"email": "john@example.com",
"name": "John Doe"
},
"definitive": true,
"assignee": {
"id": "2f69f1c8-bf0a-48ef-b47f-95ae5a4de325",
"name": "Speaker 2"
}
},
{
"id": "5519156904460288",
"text": "How did you hear about this position?",
"type": "question",
"score": 0.999988666660899,
"messageIds": [
"4616389407014912"
],
"from": {
"id": "2f69f1c8-bf0a-48ef-b47f-95ae5a4de325",
"email": "john@example.com",
"name": "John Doe"
}
},
...
]
}
Appending to an existing conversation with Diarization
Because conversations don’t neatly end at once and may resume later, our Async API allows you to update/append an existing conversation. You can read more about this capability here.
To enable Diarization with the append capability, the request structure is the same as shown above for creating a new Conversation. You would need to pass in enableSpeakerDiarization=true
and diarizationSpeakerCount=<NUMBER_OF_UNIQUE_SPEAKERS>
query-parameters.
However, there is one caveat in how the ASR works with Diarization. Consider the below:
An example scenario
We send a recorded conversation to the Async API with 2 speakers John
and Alice
with enableSpeakerDiarization=true
. The diarization identifies them as Speaker 1
and Speaker 2
respectively. We then update the above speakers with their email
as john@example.com
and alice@example.com
Now we use the append call for appending another conversation with 2 speakers John
and May
with enableSpeakerDiarization=true
. Let’s assume that the diarization would now identify these as Speaker 1
and Speaker 2
respectively. As discussed before, these numbers are arbitrary and have nothing to do with the order in which the speakers spoke in the conversation.
After this job is complete we will have 4 members in this conversation:
John
Alice
Speaker 1
(Which isJohn
again)Speaker 2
(Which isMay
)
Since John
and Speaker 1
refer to the same speaker but are labeled as different speakers, their member
references would be different for all messages
and insights
that they are a part of.
The email
identifier
This is where the email
identifier comes in. The PUT members
call can uniquely identify and merge a member
with the same email
parameter and eliminate any duplicate references with a single reference across the entire conversation which would update all the references including the members
, messages
and insights
.
If we were to execute a PUT members
call with the below body where 74001a1d-4e9e-456a-84ed-81bbd363333a
refers to the id
of Speaker 1
from the above scenario, this would eliminate this member
and would update all the references with member represented by 2f69f1c8-bf0a-48ef-b47f-95ae5a4de325
which we know is John Doe
.
$ curl --location --request PUT 'https://api.symbl.ai/v1/conversations/<CONVERSATION_ID>/members/74001a1d-4e9e-456a-84ed-81bbd363333a' --header 'Content-Type: application/json' --header 'x-api-key: <X-API-KEY> --data-raw '{
"id": "74001a1d-4e9e-456a-84ed-81bbd363333a",
"email": "john@example.com",
"name": "John Doe"
}'
This is possible because the email
uniquely identifies that user.
Best Practices
The
diarizationSpeakerCount
should be equal to the number of unique speakers present in the conversation for best results as the Diarization model uses this number to probabilistically determine the speakers. If this number is different than the actual speakers, then it might introduce false-positives for some part of the transcriptions.For the best experience, the Sample Rate of the data should be greater than or equal to
16000Hz
.
That’s it!
You have the steps to start separating the speakers in any Audio/Video-based conversation.
You can visit docs.symbl.ai to learn more about the following APIs:
Symbl Adapter for Chime SDK
The Symbl Conversation AI Adapter for Chime SDK is the simplest way to get started with Symbl in your Amazon Chime video platform.
Currently, the Symbl Conversational AI Adapter has the following features:
Realtime closed-captioning.
Realtime conversational insights (Action Items, Questions, Follow-ups).
Transcription.
Prerequisites
You must have the following installed:
[NPM 6.11 or higher]
Set Up for Symbl
Install symbl-chime-adapter
To get started you will need to add the symbl-chime-adapter to your node dependencies.
$ npm install --save symbl-chime-adapter
Symbl Credentials
Create an account in the Symbl Console if you have not done so already.
After you login, you will find your appId and appSecret on the home page.
Create a
.env
file indemos/browser
anddemos/serverless/src
that includes your appId and appSecret as shown below.
SYMBL_APP_ID=<xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx>
SYMBL_APP_SECRET=<xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx>
The App ID and App Secret are used to authenticate your session with Symbl by generating an access token.
Initialization
The Symbl adapter should be initialized after connecting to your Chime video conference.
To connect each Chime participant to Symbl, it is best practice to generate the client’s access token on your backend service.
After the token is generated it can be returned along with the chime meeting configuration as shown below. For a full example of this, please see this page of our chime demo application.
const res = await fetch(`https://api.symbl.ai/oauth2/token:generate`, {
method: 'POST',
mode: 'cors',
cache: 'no-cache',
credentials: 'same-origin',
headers: {
'Content-Type': 'application/json',
},
redirect: 'follow',
referrerPolicy: 'no-referrer',
body: JSON.stringify({
type: 'application',
appId: symblAppId,
appSecret: symblAppSecret,
}),
});
accessToken = await res.json();
return response(200, 'application/json', JSON.stringify({
JoinInfo: {
Meeting: meeting,
Attendee: attendee,
Symbl: accessToken,
},
}, null, 2));
};
After the token is generated and returned by your server, the Symbl Access Token can be applied by setting the static property ACCESS_TOKEN
of the Symbl class.
Symbl.ACCESS_TOKEN = joinInfo.Symbl.accessToken;
After the access token has been set and your client has joined the Chime meeting, you can instantiate the Symbl class.
The Symbl constructor takes two parameters:
Chime
Field | Type | Required | Description |
---|---|---|---|
attendeeId | string | true | ID for video conference participant |
userName | string | false | Human readable name for video conference participant |
meetingId | string | true | Unique identifier for the meeting |
meeting | string | true | Name of the meeting |
Options
Field | type | Required | Supported Values | Default Value | Description |
---|---|---|---|---|---|
confidenceThreshold | double | false | 0.5-1.0 | 0.5 | Your access token retrieved through authentication with |
languageCode | boolean | false | en-US, en-GB, en-AU, it-IT, nl-NL, fr-FR, fr-CA, de-DE, es-US, ja-JP | en-US | The language code as per the BCP 47 specification |
insightsEnabled | boolean | false | true | Whether or not to generate insights. |
Example Config
/**
@param {object} chime - chime configuration
{
@property {string} attendeeId -- Client attendee id
@property {string} userName - User name
@property {string} meetingId -- UUID of the meeting
@property {string} meeting meeting name
},
@param {object} config - Symbl Configuration
{
@property {number} confidenceThreshold optional | default: 0.5 | 0.0 - 1.0 minimum confidence value produce valid insight
@property {string} languageCode optional - default: 'en-US' | The language code as per the BCP 47 specification
@property {boolean} insightsEnabled optional - default: true -- false if language code is not english.
}
*/
this.symbl = new Symbl(
{
attendeeId: this.configuration.credentials.attendeeId,
userName: this.configuration.credentials.externalUserId.split('#').pop(),
meetingId: this.configuration.credentials.meetingId,
meeting: this.meeting,
}, {
confidenceThreshold: 0.5,
languageCode: 'en-US',
insightsEnabled: true,
}
);
Subscribe to Realtime Events
Learn how to subscribe to Symbl’s conversational insights and transcription.
There are three different event types that you can subscribe through with Symbl’s Conversational AI Adapter for Chime.
Closed Captions
Transcripts
Insights
Each subscriber function, subscribeToCaptioningEvents
Closed Captions
Real-time closed captioning is enabled by default and provides realtime transcription of your audio content. Closed captioning tracks can be added to any video element through the callback handler to your Symbl instance.
Handler
Closed caption handler has 3 callback functions:
onCaptioningToggled
- Will be called whenever closed captioning is toggled on or off.oncCaptionCreated
- Event that occurs after speech is first detecting, having the caption object created after as a parameter.onCaptionUpdated
- Event that occurs after speech is no longer detected, indicating the end of transcriipton.
The handler can be added using the subscribeToCaptioningEvents
function of the Symbl instance.
const captioningHandler = {
onCaptioningToggled: (ccEnabled: boolean) => {
// Implement
},
onCaptionCreated: (caption: Caption) => {
console.warn('Caption created', caption);
// Retrieve the video element that you wish to add the caption tracks to.
const activeVideoElement = getActiveVideoElement() as HTMLVideoElement;
if (activeVideoElement) {
subtitle.setVideoElement(activeVideoElement);
}
},
onCaptionUpdated: (caption: Caption) => {
const activeVideoElement = getActiveVideoElement() as HTMLVideoElement;
subtitle.setVideoElement(activeVideoElement);
},
};
symbl.subscribeToCaptioningEvents(captioningHandler);
Setting the video element that subtitles will be superimposed over should be done by calling the setVideoElement
function on the Caption
class.
If your video chat application has alternating primary video tiles, this can be used to change which element is active.
Realtime Insights
Realtime insights are generated as Symbl processes the conversation in your video chat platform.
When an Insight is detected by Symbl and the onInsightCreated
event is emitted, an Insight object is passed to the callback function provided in the Insight handler.
The Insight
class holds data about the insight generated.
Insight Class Attributes
Field | Type | Description | ||
---|---|---|---|---|
type | string | Insight type. Can beaction_item, question, or follow_up. | ||
text | string | Insight content | ||
id | string | ID of the Insight | ||
confidence | number |
Assignee
field | type | description |
---|---|---|
name | string | Insight assignee name |
userId | string | Insight assignee email address |
id | string | Insight assignee ID |
Hints (follow-up's only)
field | type | description |
---|---|---|
key | string | Insight assignee name |
value | string | Insight assignee email address |
id | string | Insight assignee ID |
Tags
Field | Type | Description |
---|---|---|
key | string | |
value |
Handler
The Symbl adapter exposes a handler function, subscribeToInsightEvents
, that has a callback function onInsightCreated
.
new Symbl(chimeConfiguration, {insightsEnabled: true});
Subscribing to the Insight publisher is achieved by passing a handler to the subscribeToInsightEvents
function of your Symbl
instance.
Example
When an insight event is emitted from the onInsightCreated
handler, you can
use the Insight object returned and either use the createElement
function to
create a default element or you can use the data included in the Insight
object returned in the handlers callback to create your own element,
capture the data and store as a metric, etc.
const insightHandler = {
onInsightCreated: (insight: Insight) => {
// Creates a predesigned insight element
const element = insight.createElement();
// Customize any styling
element.classList.add('mx-auto');
element.style.width = '98%';
/** OR create your own element
insight.element = document.createElement('div');
insight.element.innerHTML = `<div style="width: auto; height: 400px">
<h3 style="font-width: 400;">
${insight.type}
</h3>
<br>
<h2>
${insight.text}
</h2>
</div>`;
**/
// Retrieve container you wish to add insights to.
const insightContainer = document.getElementById('receive-insight');
// Call add on the insight object to add it to DIV
insight.add(insightContainer);
}
};
// Subscribe to realtime insight events using the handler created above
this.symbl.subscribeToInsightEvents(insightHandler);
Realtime Transcripts
Realtime transcripts differ slightly from realtime closed-captioning. A Transcript object is created when after speech is no longer detected.
Attributes
Field | Type | Description | ||
---|---|---|---|---|
message | string | Speaker content | ||
userName | string | Speaker name | ||
userId | string | Speaker ID | ||
timeStamp | Date | Time of the transcript event |
Handler
The Transcript handler has only one event, onTranscriptCreated
. Each time a speaker finishes speaking a Transcript
object is generated.
Subscribing to the transcript publisher is achieved by passing a handler to the subscribeToTranscriptEvents
function of your Symbl
instance.
const transcriptHandler = {
onTranscriptCreated: (transcript: Transcript) => {
// Handle transcript item
}
};
this.symbl.subscribeToTranscriptEvents(transcriptHandler);
Generating Meeting Summary URL
To generate a meeting summary URL, you need only to call the getSummaryUrlfunction
of your Symbl instance.
const meetingSummaryUrl = await symbl.getSummaryUrl();
Symbl SDK (Node.js)
The Programmable Symbl SDK allows you to add Conversational Intelligence directly into your web applications and meeting platforms. With the Symbl SDK , you can generate intelligent insights such as action items, topics and questions.
This section demonstrates how to use the SDK to add voice integration to your existing application.
- Initialize the Symbl SDK
- Connect to Endpoints
- Active Speaker Events
- Subscribe to Real-time Events
- Language and Timezone
- Complete Example
- Send Summary Email
- Tuning your Summary Page
- Streaming Audio in Real-time
System requirement: NodeJS 7+
Initialize the SDK
After installing, initialize the SDK
sdk.init({
appId: 'yourAppId',
appSecret: 'yourAppSecret',
basePath: 'https://api.symbl.ai'
})
.then(() => console.log('SDK Initialized.'))
.catch(err => console.error('Error in initialization.', err));
1. After getting your appId and appSecret, use the command below to install the SDK and add it to your npm project's package.json.
$ npm install --save symbl-node
2. Reference the SDK in either the ES5 or ES6 way.
The ES5 way
var sdk = require('symbl-node').sdk;
The ES6 way
import { sdk } from 'symbl-node';
Connect to Endpoints
The code snippet below dials in using PSTN and hangs up after 60 seconds.
const {
sdk
} = require('symbl-node');
sdk.init({
appId: 'yourAppId',
appSecret: 'yourAppSecret',
basePath: 'https://api.symbl.ai'
}).then(() => {
sdk.startEndpoint({
endpoint: {
type: 'pstn', // This can be pstn or sip
phoneNumber: '<number_to_call>', // include country code
dtmf: '<meeting_id>' // if password protected, use "dtmf": "<meeting_id>#,#<password>#"
}
}).then(connection => {
console.log('Successfully connected.', connection.connectionId);
// Scheduling stop endpoint call after 60 seconds for demonstration purposes
// In real adoption, sdk.stopEndpoint() should be called when the meeting or call actually ends
setTimeout(() => {
sdk.stopEndpoint({
connectionId: connection.connectionId
}).then(() => {
console.log('Stopped the connection');
console.log('Summary Info:', connection.summaryInfo);
console.log('Conversation ID:', connection.conversationId);
}).catch(err => console.error('Error while stopping the connection', err));
}, 60000);
}).catch(err => console.error('Error while starting the connection', err));
}).catch(err => console.error('Error in SDK initialization.', err));
We recommend using SIP whenever possible instead of PSTN as it provides higher audio quality options as compared to PSTN. SIP endpoint provides an optional audio configuration as well. Contact us for your specific requirements.
This SDK supports dialing through a simple phone number - PSTN
or a Voice Over IP system - SIP
endpoint. If you don't have your own voice over IP system, you will want to use a phone number to make the connection.
PSTN (Public Switched Telephone Networks)
The Publicly Switched Telephone Network (PSTN) is the network that carries your calls when you dial in from a landline or cell phone. It refers to the worldwide network of voice-carrying telephone infrastructure, including privately-owned and government-owned infrastructure.
endpoint: {
type: 'pstn',
phoneNumber: '14083380682', // Phone number to dial in
dtmf: '6155774313#' // Joining code
}
SIP (Session Initiation Protocol)
Session Initiation Protocol (SIP) is a standardized communications protocol that has been widely adopted for managing multimedia communication sessions for voice and video calls. SIP may be used to establish connectivity between your communications infrastructures and Symbl's communications platform.
endpoint: {
type: 'sip',
uri: 'sip:555@your_sip_domain', // SIP URI to dial in
audioConfig: { // Optionally any audio configuration
sampleRate: 16000,
encoding: 'PCMU',
sampleSize: '16'
}
}
Active Speaker Events
The example on the right shows how to connect to a PSTN endpoint, create a speakerEvent instance and push events on connection
const {
sdk,
SpeakerEvent
} = require('symbl-node');
sdk.init({
appId: 'yourAppId',
appSecret: 'yourAppSecret',
basePath: 'https://api.symbl.ai'
}).then(() => {
sdk.startEndpoint({
endpoint: {
type: 'pstn',
phoneNumber: '<number_to_call>', // include country code
dtmf: '<meeting_id>' // if password protected, use "dtmf": "<meeting_id>#,#<password>#"
}
}).then(connection => {
const connectionId = connection.connectionId;
console.log('Successfully connected.', connectionId);
const speakerEvent = new SpeakerEvent();
speakerEvent.type = SpeakerEvent.types.startedSpeaking;
speakerEvent.user = {
userId: 'john@example.com',
name: 'John'
};
speakerEvent.timestamp = new Date().toISOString();
sdk.pushEventOnConnection(
connectionId,
speakerEvent.toJSON(),
(err) => {
if (err) {
console.error('Error during push event.', err);
} else {
console.log('Event pushed!');
}
}
);
// Scheduling stop endpoint call after 60 seconds for demonstration purposes
// In real adoption, sdk.stopEndpoint() should be called when the meeting or call actually ends
setTimeout(() => {
sdk.stopEndpoint({
connectionId: connection.connectionId,
}).then(() => {
console.log('Stopped the connection');
console.log('Summary Info:', connection.summaryInfo);
console.log('Conversation ID:', connection.conversationId);
}).catch(err => console.error('Error while stopping the connection.', err));
}, 60000);
}).catch(err => console.error('Error while starting the connection', err));
}).catch(err => console.error('Error in SDK initialization.', err));
NOTE: Setting the timestamp for speakerEvent is optional but it's recommended to provide accurate timestamps in the events when they occurred to get more precision.
Events can be pushed to an on-going connection to have them processed. The code snippet to the right shows a simple example.
Every event must have a type to define the purpose of the event at a more granular level, usually to indicate different activities associated with the
event resource. For example - A "speaker" event can have type as started_speaking
. An event may have additional fields specific to the event.
Currently, Symbl only supports the speaker event which is described below.
Speaker Event
The speaker event is associated with different individual attendees in the meeting or session. An example of a speaker event is shown below.
In the code example the user needs to have userId
field to uniquely identify the user.
Speaker Event has the following types:
started_speaking
This event contains the details of the user who started speaking with the timestamp in ISO 8601 format when he started speaking.
const speakerEvent = new SpeakerEvent({
type: SpeakerEvent.types.startedSpeaking,
timestamp: new Date().toISOString(),
user: {
userId: 'john@example.com',
name: 'John'
}
});
stopped_speaking
This event contains the details of the user who stopped speaking with the timestamp in ISO 8601 format when he stopped speaking.
const speakerEvent = new SpeakerEvent({
type: SpeakerEvent.types.stoppedSpeaking,
timestamp: new Date().toISOString(),
user: {
userId: 'john@example.com',
name: 'John'
}
});
As shown in the above examples, it's ok to reuse the same speakerEvent
instance per user, by changing the event's type to optimize
by reducing the number of instances for SpeakerEvent
.
A startedSpeaking
event is pushed on the on-going connection. You can use pushEventOnConnection()
method from the SDK to push the events.
Subscribe to Real Time Events
Symbl SDK also lets you subscribe to real-time events when you connect to one of the Endpoints specified in the above sections. These include
- Real-Time Transcription
- Real-Time Insights
- Real-Time Messages
- Real-Time Intents
The below example shows how to achieve this.
Initialize SDK
const {sdk, SpeakerEvent} = require("symbl-node");
sdk.init({
// Your appId and appSecret https://platform.symbl.ai
appId: 'your_appId',
appSecret: 'your_appSecret'
}).then(async () => {
console.log('SDK initialized.');
try {
// You code goes here.
} catch (e) {
console.error(e);
}
}).catch(err => console.error('Error in SDK initialization.', err));
Add the above lines to import and initialize the SDK. Replace the appId
and appSecret
in the code.
You can find the them by signing up on Symbl Developer Platform
Make a phone call
const connection = await sdk.startEndpoint({
endpoint: {
type: 'pstn', // when making a regular phone call
// Replace this with a real phone number
phoneNumber: '1XXXXXXXXXX' // include country code
}
});
const {connectionId} = connection;
console.log('Successfully connected. Connection Id: ', connectionId);
The above snippet makes a phone call, by calling the startEndpoint
with type set to pstn
and a valid US/Canada Phone Number phoneNumber.
You can also call in via type sip
as well and the steps below will still remain the same.
Subscribe to the Live Events
// Subscribe to connection using connectionId.
sdk.subscribeToConnection(connectionId, (data) => {
const {type} = data;
if (type === 'transcript_response') {
const {payload} = data;
// You get live transcription here!!
process.stdout.write('Live: ' + payload && payload.content + '\r');
} else if (type === 'message_response') {
const {messages} = data;
// You get processed messages in the transcript here!!! Real-time but not live! :)
messages.forEach(message => {
process.stdout.write('Message: ' + message.payload.content + '\n');
});
} else if (type === 'insight_response') {
const {insights} = data;
// See <link here> for more details on Insights
// You get any insights here!!!
insights.forEach(insight => {
process.stdout.write(`Insight: ${insight.type} - ${insight.text} \n\n`);
});
}
});
The above snippet calls the subscribeToConnection
which requires the connectionId
of the call and a callback function to be passed as the second argument which will be invoked when any of the above events are available to be consumed.
The data
received will contain type
of the event. It can be one of transcript_response
, message_response
, insight_response
.
Lets go over them one by one.
transcript_response
: This contains the real-time transcription data which is availabe as soon as its detected.message_response
: This will contain the array of the transcripts of all the speakers which will be logically separated by punctuations or the speakers if Active Speaker Events are pushed.insight_response
: This will contain the array of all the insights detected in real-time. These can be action-items or questions.
There is also a 4th type of event which is intent_response
covered in a separate example.
End the call
// Stop call after 60 seconds to automatically.
setTimeout(async () => {
const connection = await sdk.stopEndpoint({ connectionId });
console.log('Stopped the connection');
console.log('Conversation ID:', connection.conversationId);
}, 60000); // Change the 60000 with higher value if you want this to continue for more time.
To end the call gracefully, we call the stopEndpoint
call to stop the call.
The code snippet above simply stops the call after 60 seconds.
And we're done! That's how you can consume real-time events using Symbl SDK !
The complete code for the example above can be found here
Language and TimeZone
You can also specify languages
other than 'English' from the ones supported to be used for calls made via 'PSTN' or 'SIP'.
With this since the call can take place in a different timeZone you can also pass in the timeZone
which will be used to render the Summary UI specific to that language and timeZone.
The below sub-sections specify how to use these capabilities with Symbl SDK with a complete example.
Utilising other languages
Symbl SDK allows you to work with audio from multiple different languages. The currently supported languages are given below with the regions they are spoken in:
- English (United States)
- English (United Kingdom)
- English (Australia)
- French (Canada)
- German (Germany)
- Italic (Italy)
- Dutch (Netherlands)
- Japanese (Japan)
- Spanish (Latin America)
- French (France)
You can pass in the languages
array in the startEndpoint
as shown in the example linked below.
Please note that currently only one language can be specified per call and support for detecting multiple languages in the same call will be added soon.
Specifying Time Zone
With calls taking place in different regions around the world it can get important to capture that information and utilise it.
Symbl SDK allows you to pass in the timeZone
in the startEndpoint
call which will render the Summary UI in that specific Time Zone.
A list of all the Time Zones is available here
Passing Language and TimeZone
A complete example showcasing this capability can be found here
Complete Example
const {
sdk,
SpeakerEvent
} = require('symbl-node');
sdk.init({
appId: 'yourAppId',
appSecret: 'yourAppSecret',
basePath: 'https://api.symbl.ai'
}).then(() => {
console.log('SDK Initialized');
sdk.startEndpoint({
endpoint: {
type: 'pstn',
phoneNumber: '14087407256',
dtmf: '6327668#'
}
}).then(connection => {
const connectionId = connection.connectionId;
console.log('Successfully connected.', connectionId);
const speakerEvent = new SpeakerEvent({
type: SpeakerEvent.types.startedSpeaking,
user: {
userId: 'john@example.com',
name: 'John'
}
});
setTimeout(() => {
speakerEvent.timestamp = new Date().toISOString();
sdk.pushEventOnConnection(
connectionId,
speakerEvent.toJSON(),
(err) => {
if (err) {
console.error('Error during push event.', err);
} else {
console.log('Event pushed!');
}
}
);
}, 2000);
setTimeout(() => {
speakerEvent.type = SpeakerEvent.types.stoppedSpeaking;
speakerEvent.timestamp = new Date().toISOString();
sdk.pushEventOnConnection(
connectionId,
speakerEvent.toJSON(),
(err) => {
if (err) {
console.error('Error during push event.', err);
} else {
console.log('Event pushed!');
}
}
);
}, 12000);
// Scheduling stop endpoint call after 60 seconds
setTimeout(() => {
sdk.stopEndpoint({
connectionId: connection.connectionId
}).then(() => {
console.log('Stopped the connection');
console.log('Summary Info:', connection.summaryInfo);
console.log('Conversation ID:', connection.conversationId);
}).catch(err => console.error('Error while stopping the connection.', err));
}, 90000);
}).catch(err => console.error('Error while starting the connection', err));
}).catch(err => console.error('Error in SDK initialization.', err));
Below is a quick simulated speaker event example that
- Initializes the SDK
- Initiates a connection with an endpoint
- Sends a speaker event of type
startedSpeaking
for user John - Sends a speaker event of type
stoppedSpeaking
for user John - Ends the connection with the endpoint
Strictly for illustration and understanding purposes, the code to the right pushes events by simply using setTimeout()
method periodically, but
in real usage, they should be pushed as they occur.
const {
sdk,
SpeakerEvent
} = require('symbl-node');
sdk.init({
appId: 'yourAppId',
appSecret: 'yourAppSecret',
basePath: 'https://api.symbl.ai'
}).then(() => {
console.log('SDK Initialized');
sdk.startEndpoint({
endpoint: {
type: 'pstn',
phoneNumber: '14087407256',
dtmf: '6327668#'
},
actions: [{
"invokeOn": "stop",
"name": "sendSummaryEmail",
"parameters": {
"emails": [
"john@exmaple.com",
"mary@example.com",
"jennifer@example.com"
]
}
}],
data: {
session: {
name: 'My Meeting Name' // Title of the Meeting
},
users: [{
user: {
name: "John",
userId: "john@example.com",
role: "organizer"
}
},
{
user: {
name: "Mary",
userId: "mary@example.com"
}
},
{
user: {
name: "John",
userId: "jennifer@example.com"
}
}
]
}
}).then((connection) => {
console.log('Successfully connected.');
// Events pushed in between
setTimeout(() => {
// After successful stop endpoint, an email with summary will be sent to "john@example.com" and "jane@example.com"
sdk.stopEndpoint({
connectionId: connection.connectionId
}).then(() => {
console.log('Stopped the connection');
console.log('Summary Info:', connection.summaryInfo);
console.log('Conversation ID:', connection.conversationId);
}).catch(err => console.error('Error while stopping the connection.', err));
}, 30000);
}).catch(err => console.error('Error while starting the connection', err));
}).catch(err => console.error('Error in SDK initialization.', err));
Send Summary Email
This is an example of the summary page you can expect to receive at the end of your call
Take a look at the Sample Summary UI which is generated after a meeting is concluded.
Tuning your Summary Page
You can choose to tune your Summary Page with the help of query parameters to play with different configurations and see how the results look.
Query Parameters
You can configure the summary page by passing in the configuration through query parameters in the summary page URL that gets generated at the end of your meeting. See the end of the URL in this example:
https://meetinginsights.symbl.ai/meeting/#/eyJ1...I0Nz?insights.minScore=0.95&topics.orderBy=position
Query Parameter | Default Value | Supported Values | Description |
---|---|---|---|
insights.minScore |
0.8 | 0.5 to 1.0 | Minimum score that the summary page should use to render the insights |
insights.enableAssignee |
false | [true, false] | Enable to disable rending of the assignee and due date of the insight |
insights.enableAddToCalendarSuggestion |
true | [true, false] | Enable to disable add to calendar suggestion when applicable on insights |
insights.enableInsightTitle |
true | [true, false] | Enable or disable the title of an insight. The title indicates the originating person of the insight and if assignee of the insight. |
topics.enabled |
true | [true, false] | Enable or disable the summary topics in the summary page |
topics.orderBy |
'score' | ['score', 'position'] | Ordering of the topics. score - order topics by the topic importance score. position - order the topics by the position in the transcript they surfaced for the first time |
Streaming Audio in Real Time
This section talks about streaming the audio in real-time using the Symbl SDK . We can use this API to pass in audio via a single stream and it also supports sending multiple isolated streams of audio, each of which can contain one or more speaker's audio data.
You can also consume the processed results in real-time, which include
- Real Time Transcription
- Real Time Insights (Action Items and Questions)
- When using multiple audio streams (Each stream for 1 speaker) you also get access to speaker-separated data (including transcription and messages)
Example with Single Stream
The example below utilises mic
package to stream audio in real-time. This will be a single stream of audio obtained through mic
which may have one or more than one speaker's audio.
The link to the complete example below can be found here
Import required packages
const {sdk} = require('symbl-node');
const uuid = require('uuid').v4;
// For demo purposes, we're using mic to simply get audio from microphone and pass it on to websocket connection
const mic = require('mic');
In the above snippet we import the sdk
, uuid
and mic
npm packages. The uuid
package is used for generating a unique ID to represent this stream and it's strongly recommended to use it.
The mic
package is used to obtain the audio stream in real-time to pass to the SDK.
Initialise an instance of mic
const sampleRateHertz = 16000;
const micInstance = mic({
rate: sampleRateHertz,
channels: '1',
debug: false,
exitOnSilence: 6
});
We now declare the sampleRateHertz
variable to specify the Sample Rate of the audio obtained from the mic
.
It is imperative to use the same Sample Rate used for initialising the mic
package and for passing in to the startRealtimeRequest
of Symbl SDK as we will see below.
Otherwise the transcription will be completely in-accurate.
We also initialise mic
with channels: '1'
(mono channel) audio as currently only mono channel audio data is supported.
Initialise the Symbl SDK
// Initialize the SDK
await sdk.init({
appId: '__appId__',
appSecret: '__appSecret__',
basePath: 'https://api.symbl.ai'
});
// Need unique Id
const id = uuid();
Next we initialise a helper function to execute our code in the async/await
style. The following code snippets (including the one just above) will be a part of the same function.
We now initialise the Symbl SDK with the init
call, passing in appId
and appSecret
which you can be obtain by signing up on Symbl Developer Platform
We also initialise variable id
with uuid
function for the unique ID required for this stream as was also mentioned above in the import section snippet.
Call the startRealtimeRequest
// Start Real-time Request (Uses Realtime WebSocket API behind the scenes)
const connection = await sdk.startRealtimeRequest({
id,
insightTypes: ["action_item", "question"],
config: {
meetingTitle: 'My Test Meeting',
confidenceThreshold: 0.7,
timezoneOffset: 480, // Offset in minutes from UTC
languageCode: "en-US",
sampleRateHertz
},
speaker: { // Optional, if not specified, will simply not send an email in the end.
userId: 'john.doe@example.com', // Update with valid email
name: 'John'
},
handlers: {
'onSpeechDetected': (data) => {
console.log(JSON.stringify(data));
// For live transcription
if (data) {
const {punctuated} = data;
console.log('Live: ', punctuated && punctuated.transcript);
}
},
'onMessageResponse': (data) => {
// When a processed message is available
console.log('onMessageResponse', JSON.stringify(data));
},
'onInsightResponse': (data) => {
// When an insight is detected
console.log('onInsightResponse', JSON.stringify(data));
}
}
});
The next call is made to startRealtimeRequest
of the Symbl SDK and includes various parameters passed in.
Lets breakdown the configuration and take a look at them one by one.
id
: The unique ID that represents this stream. (This needs to be unique, which is why we are usinguuid
)insightTypes
: This array represents the type of insights that are to be detected. Today the supported ones areaction_item
andquestion
.config
: This configuration object encapsulates the properties which directly relate to the conversation generated by the audio being passed.a.
meetingTitle
: This optional parameter specifies the name of the conversation generated. You can get more info on conversations hereb.
confidenceThreshold
: This optional parameter specifies the confidence threshold for detecting the insights. Only the insights that haveconfidenceScore
more than this value will be returned.c.
timezoneOffset
: This specifies the actual timezoneOffset used for detecting the time/date related entities.d.
languageCode
: It specifies the language to be used for transcribing the audio in BCP-47 format. (Needs to be same as the language in which audio is spoken)e.
sampleRateHertz
: It specifies the sampleRate for this audio stream.speaker
: Optionally specify the details of the speaker whose data is being passed in the stream. This enables an e-mail with the Summary UI URL to be sent after the end of the stream.handlers
: This object has the callback functions for different events a.onSpeechDetected
: To retrieve the real-time transcription results as soon as they are detected. We can use this callback to render live transcription which is specific to the speaker of this audio stream.b.
onMessageResponse
: This callback function contains the "finalized" transcription data for this speaker and if used with multiple streams with other speakers this callback would also provide their messages. The "finalized" messages mean that the ASR has finalised the state of this part of transcription and has declared it "final".c.
onInsightResponse
: This callback would provide with any of the detected insights in real-time as they are detected. As with theonMessageCallback
above this would also return every speaker's insights in case of multiple streams.
Retrieve audio data from mic
console.log('Successfully connected.');
const micInputStream = micInstance.getAudioStream();
micInputStream.on('data', (data) => {
// Push audio from Microphone to websocket connection
connection.sendAudio(data);
});
console.log('Started listening to Microphone.');
After the startRealtimeRequest
returns successfully, it signifies that the connection has been established successfully with the passed configuration.
In the above snippet we now obtain the audio data from the micInputStream
and as it's received we relay it to the active connection instance we now have with Symbl SDK .
Stop the stream
setTimeout(async () => {
// Stop listening to microphone
micInstance.stop();
console.log('Stopped listening to Microphone.');
try {
// Stop connection
await conversationData = connection.stop();
console.log('Conversation ID: ' + conversationData.conversationId);
console.log('Connection Stopped.');
} catch (e) {
console.error('Error while stopping the connection.', e);
}
}, 60 * 1000); // Stop connection after 1 minute i.e. 60 secs
For the purpose of demoing a continuous audio stream we now simulate a stop
on the above stream after 60 seconds.
The connection.stop()
would close the active connection and will trigger the optional email if the speaker
config is included.
Here the conversationData
variable includes the conversationId
you can use with the Conversation API to retrieve this conversation's data.
And that's it! This marks the completion of streaming audio in real-time (Single Audio Stream) with Symbl SDK . The complete code for the example explained above can be found here
With Multiple Streams
The same example explained above can be deployed on multiple machines, each with one speaker to simulate the multiple streams use-case.
The only thing common needs to be the unique ID created in the above example which is used to initialize startRealtimeRequest
request.
Having this unique ID in common across all different ensures that the audio streams of all the speakers are bound the context of a single conversation.
This conversation can be retrieved by the conversationId
via the Conversation API which will include the data of all the speakers connecting using the same common ID.
Real Time Telephony API
The Voice API provides the REST interface for adding Symbl to your call and generating actionable insights from your conversations. Telephony API only allows phone numbers in the USA and Canada
POST Telephony
Example API Call
curl -k -X POST "https://api.symbl.ai/v1/endpoint:connect" \
-H "accept: application/json" \
-H "Content-Type: application/json" \
-H "x-api-key: <your_auth_token>" \
-d @location_of_fileName_with_request_payload
const request = require('request');
const payload = {
"operation": "start",
"endpoint": {
"type" : "pstn",
"phoneNumber": "<number_to_call>", // include country code
"dtmf": "<meeting_id>" // if password protected, use "dtmf": "<meeting_id>#,#<password>#"
},
"actions": [{
"invokeOn": "stop",
"name": "sendSummaryEmail",
"parameters": {
"emails": [
"joe.symbl@example.com"
]
}
}],
"data" : {
"session": {
"name" : "My Meeting"
}
}
}
const your_auth_token = "<your_auth_token>";
request.post({
url: 'https://api.symbl.ai/v1/endpoint:connect',
headers: {'x-api-key': your_auth_token},
body: payload,
json: true
}, (err, response, body) => {
console.log(body);
});
The above command returns an object structured like this:
{
"eventUrl": "https://api.symbl.ai/v1/event/771a8757-eff8-4b6c-97cd-64132a7bfc6e",
"resultWebSocketUrl": "wss://api.symbl.ai/events/771a8757-eff8-4b6c-97cd-64132a7bfc6e",
"connectionId": "771a8757-eff8-4b6c-97cd-64132a7bfc6e",
"conversationId": "51356232423"
}
The Telephony Voice API allows you to easily use Symbl's Language Insights capabilities.
It exposes the functionality of Symbl to dial-in to the conference. Supported endpoints are given below. Additionally, events can be passed for further processing. The supported types of events are discussed in detail in the section below.
- Symbl will connect to your SIP or PSTN exposed endpoints. The Audio stream will be fetched from the SIP or PSTN connection once connected.
HTTP REQUEST
POST https://api.symbl.ai/v1/endpoint:connect
Request Parameters
Parameter | Type | Description |
---|---|---|
operation |
string | enum([start, stop]) - Start or Stop connection |
endpoint |
object | Object containing Type of the session - either pstn or sip , phoneNumber which is the meeting number symbl should call with country code prepended and dtmf which is the conference passcode. |
actions |
list | actions that should be performed while this connection is active. Currently only one action is supported - sendSummaryEmail |
data |
object | Object containing a session object which has a field name corresponding to the name of the meeting |
Response Object
Field | Description |
---|---|
eventUrl |
REST API to push speaker events as the conversation is in progress, to add additional speaker context in the conversation. Example - In an on-going meeting, you can push speaker events |
resultWebSocketUrl |
Same as eventUrl but over WebSocket. The latency of events is lower with a dedicated WebSocket connection. |
connectionId |
Ephemeral connection identifier of the request, to uniquely identify the telephony connection. Once the connection is stopped using “stop” operation, or is closed due to some other reason, the connectionId is no longer valid |
conversationId |
Represents the conversation - this is the ID that needs to be used in conversation api to access the conversation |
To play around with a few examples, we recommend a REST client called Postman. Simply tap the button below to import a pre-made collection of examples.
Try it out
When you have started the connection through the API, try speaking the following sentences and view the summary email that gets generated:
"Hey, it was nice meeting you yesterday. Let's catch up again next week over coffee sometime in the evening. I would love to discuss the next steps in our strategic roadmap with you."
"I will set up a meeting with Roger to discuss our budget plan for the next quarter and then plan out how much we can set aside for marketing efforts. I also need to sit down with Jess to talk about the status of the current project. I'll set up a meeting with her probably tomorrow before our standup."
Specifying Language and Timezone
Language
The Localized Summary UI provides users a translated meeting summary page chosen based on one of eight currently supported languages chosen when a session is initiated.
Language is specified by passing an array of language codes to the lanaguage parameter when making a call to the startEndpoint. Although the languages parameter takes an array of language codes, only the first language code in the array is used to specify the session’s language.
Supported Languages
Currently 8 language codes are supported stemming from 6 language locales.
Language | Language Code |
---|---|
English(US) | en-US |
English(UK) | en-UK |
English(Australia) | en-AU |
French(Canada) | fr-CA |
French(France) | fr-FR |
Spanish(US) | es-us |
Italian(Italy) | it-IT |
Jaspanese(Japan) | ja-JP |
Dutch(Netherlands) | nl-NL |
Example
Below is an example of specifying that the meeting’s language should be in Spanish.
sdk.startEndpoint({
endpoint: {
audioConfig: {
encoding: 'OPUS',
sampleRate: 16000
},
type: 'sip',
uri: 'sip:1_firebird_1367062@am-fs-lo-05.anymeeting.com'
},
languages: ['es-US']
});
}
Timezone
Specifying a timezone when initiating a session will result in the Summary UI displaying the meeting start time for that given region.
curl --location --request POST 'https://api.symbl.ai/v1/endpoint:connect' \
--header 'x-api-key: <access_token>' \
--header 'Content-Type: application/json' \
--data-raw '{
"operation": "start",
"endpoint": {
"type" : "pstn",
"phoneNumber": "__phone_number__",
"dtmf": "__dtmf_sequence__"
},
"actions": [{
"invokeOn": "stop",
"name": "sendSummaryEmail",
"parameters": {
"emails": [
"__email_address__"
]
}
}],
"data" : {
"session": {
"name" : "__name_of_this_call__"
}
},
"languages": ["it-IT"]
}'
Example
Below is an example of setting the timezone to the Pacific Timezone by passing the TZ database name to the timezone parameter while initiating a session:
sdk.startEndpoint({
endpoint: {
audioConfig: {
encoding: 'OPUS',
sampleRate: 16000
},
type: 'sip',
uri: 'sip:1_firebird_1367062@am-fs-lo-05.anymeeting.com'
},
timezone: ' US/Pacific'
});
Full Code Example
To specify a language and timezone while initiating a meeting, two parameters required - languages and timezone.
The following code snippet would initiate a session with Japanese as the language and JST as the timezone.
sdk.startEndpoint({
endpoint: {
audioConfig: {
encoding: 'OPUS',
sampleRate: 16000
},
type: 'sip',
uri: 'sip:1_firebird_1367062@am-fs-lo-05.anymeeting.com'
},
languages: ['ja-JP'],
timezone: 'Asia/Tokyo'
});
Real Time WebSocket API
In the example below, we've used the websocket npm package for WebSocket Client, and mic for getting the raw audio from microphone.
$ npm i websocket mic
For this example, we are using your
mic
to stream audio data. You will most likely want to use other inbound sources for this
const WebSocketClient = require('websocket').client;
const mic = require('mic');
const micInstance = mic({
rate: '44100',
channels: '1',
debug: false,
exitOnSilence: 6
});
// Get input stream from the microphone
const micInputStream = micInstance.getAudioStream();
let connection = undefined;
Create a websocket client instance
const ws = new WebSocketClient();
ws.on('connectFailed', (err) => {
console.error('Connection Failed.', err);
});
ws.on('connect', (connection) => {
// Start the microphone
micInstance.start();
connection.on('close', () => {
console.log('WebSocket closed.')
});
connection.on('error', (err) => {
console.log('WebSocket error.', err)
});
connection.on('message', (data) => {
if (data.type === 'utf8') {
const {
utf8Data
} = data;
console.log(utf8Data); // Print the data for illustration purposes
}
});
console.log('Connection established.');
connection.send(JSON.stringify({
"type": "start_request",
"insightTypes": ["question", "action_item"],
"config": {
"confidenceThreshold": 0.9,
// "timezoneOffset": 480, // Your timezone offset from UTC in minutes
"languageCode": "en-US",
"speechRecognition": {
"encoding": "LINEAR16",
"sampleRateHertz": 44100 // Make sure the correct sample rate is provided for best results
},
"meetingTitle": "Client Meeting"
},
"speaker": {
"userId": "jane.doe@example.com",
"name": "Jane"
}
}));
micInputStream.on('data', (data) => {
connection.send(data);
});
});
For this example, we timeout our call after 2 minutes but you would most likely want to make the
stop_request
call when your websocket connection ends
// Schedule the stop of the client after 2 minutes (120 sec)
setTimeout(() => {
micInstance.stop();
// Send stop request
connection.sendUTF(JSON.stringify({
"type": "stop_request"
}));
connection.close();
}, 120000);
Generate the token and replace it in the placeholder
<accessToken>
. Once the code is running, start speaking and you should see themessage_response
andinsight_response
messages getting printed on the console.
ws.connect(
'wss://api.symbl.ai/v1/realtime/insights/1',
null,
null,
{ 'X-API-KEY': '<accessToken>' }
);
In the example below, we've used a Websocket that is compatible with most browsers and doesn't need any additional npm packages. Generate the token and replace it in the placeholder
<accessToken>
. Then, create the Websocket.
let url = `wss://api.symbl.ai/v1/realtime/insights/1?access_token=${'<accessToken>'}`
let ws = new WebSocket(url);
ws.onerror = (err) => {
console.error('Connection Failed.', err);
};
ws.onopen = () => {
console.log('Websocket open.')
ws.onmessage = (event) => {
if (event.type === 'message') {
console.log(event.data); // Print the data for illustration purposes
}
};
ws.onclose = () => {
console.log('WebSocket closed.')
};
ws.onerror = (err) => {
console.log('WebSocket error.', err)
};
console.log('Connection established.');
ws.send(JSON.stringify({
"type": "start_request",
"insightTypes": ["question", "action_item"],
"config": {
"confidenceThreshold": 0.9,
// "timezoneOffset": 480, // Your timezone offset from UTC in minutes
"languageCode": "en-US",
"speechRecognition": {
"encoding": "LINEAR16",
"sampleRateHertz": 44100 // Make sure the correct sample rate is provided for best results
},
"meetingTitle": "Client Meeting"
},
"speaker": {
"userId": "jane.doe@example.com",
"name": "Jane"
}
}));
}
To get direct access to the mic, we're going to use an API in the WebRTC specification called
getUserMedia()
.Once the code is running, start speaking and you should see the
message_response
andinsight_response
messages getting printed on the console.
const handleSuccess = function(stream) {
const context = new AudioContext();
const source = context.createMediaStreamSource(stream);
const processor = context.createScriptProcessor(1024, 1, 1);
source.connect(processor);
processor.connect(context.destination);
processor.onaudioprocess = function(e) {
// convert to 16-bit payload
const inputData = e.inputBuffer.getChannelData(0) || new Float32Array(this.options.bufferSize);
const targetBuffer = new Int16Array(inputData.length);
for (let index = inputData.length; index > 0; index--)
targetBuffer[index] = 32767 * Math.min(1, inputData[index]);
// Send to websocket
if(ws.readyState === WebSocket.OPEN){
ws.send(targetBuffer.buffer);
}
};
};
navigator.mediaDevices.getUserMedia({ audio: true, video: false })
.then(handleSuccess);
// Schedule the stop of the client after 2 minutes (120 sec)
setTimeout(() => {
// Send stop request
ws.send(JSON.stringify({
"type": "stop_request"
}));
ws.close();
}, 120000);
Introduction
The WebSocket based real-time API by Symbl provides the direct, fastest and most accurate of all other interfaces to push the audio stream in real-time, and get the results back as soon as they're available.
Learn more about how to get real-time transcriptions with native WebSocket API.
Connection Establishment
This is a WebSocket endpoint, and hence it starts as an HTTP request that contains HTTP headers that indicate the client's desire to upgrade the connection to a WebSocket instead of using HTTP semantics. The server indicates its willingness to participate in the WebSocket connection by returning an HTTP 101 Switching Protocols response. After the exchange of this handshake, both client and service keep the socket open and begin using a message-based protocol to send and receive information. Please refer to WebSocket Specification RFC 6455 for the more in-depth understanding of the Handshake process.
Message Formats
Client and Server both can send messages after the connection is established. According to RFC 6455, WebSocket messages can have either a text or a binary encoding. The two encodings use different on-the-wire formats. Each format is optimized for efficient encoding, transmission, and decoding of the message payload.
Text Message
Text message over WebSocket must use UTF-8 encoding. Text Message is the serialized JSON message. Every text message has a type field to specify the type or the purpose of the message.
Binary Message
Binary WebSocket messages carry a binary payload. For the Real-time API, audio is transmitted to the service by using binary messages. All other messages are the Text messages.
Client Messages
This section describes the messages that originate from the client and are sent to service.
The types of messages sent by the client are start_request
, stop_request
and binary messages containing audio.
Configuration
Main Message Body
Field | Required | Supported Values | Description |
---|---|---|---|
type |
true | start_request, stop_request | Type of message |
insightTypes |
false | action_item, question | Types of insights to return. If not provided, no insights will be returned. |
config |
false | Configuration for this request. See the config section below for more details. | |
speaker |
false | Speaker identity to use for audio in this WebSocket connection. If omitted, no speaker identification will be used for processing. See below. |
config
Field | Required | Supported Values | Default Value | Description |
---|---|---|---|---|
confidenceThreshold |
false | 0.0 - 1.0 | 0.5 | Minimum Confidence score that should be met for API to consider it as valid insight, if not provided defaults to 0.5 i.e. 50% or more |
languageCode |
false | en-US | The language code as per the BCP 47 specification | |
speechRecognition |
false | Speaker identity to use for audio in this WebSocket connection. If omitted, no speaker identification will be used for processing. See below. |
speechRecognition
Field | Required | Supported Values | Default Value | Description |
---|---|---|---|---|
encoding |
false | LINEAR16, FLAC, MULAW | LINEAR16 | Audio Encoding in which the audio will be sent over the WebSocket. |
sampleRateHertz |
false | 16000 | The rate of the incoming audio stream. |
speaker
Field | Required | Description |
---|---|---|
userId |
false | Any user identifier for the user. |
name |
false | Display name of the user. |
Messages
Start Request
{
"type": "start_request",
"insightTypes": ["question", "action_item"],
"config": {
"confidenceThreshold": 0.9,
"languageCode": "en-US",
"speechRecognition": {
"encoding": "LINEAR16",
"sampleRateHertz": 16000
}
},
"speaker": {
"userId": "jane.doe@example.com",
"name": "Jane"
}
}
This is a request to start the processing after the connection is established. Right after this message has been sent, the audio should be
streamed, any binary audio streamed before the receipt of this message will be ignored.
Stop Request
{
"type": "stop_request"
}
This is a request to stop the processing. After the receipt of this message, the service will stop any processing and close the WebSocket
connection.
Example of the
message_response
object
{
"type": "message_response",
"messages": [
{
"from": {
"name": "Jane",
"userId": "jane.doe@example.com"
},
"payload": {
"content": "I was very impressed by your profile, and I am excited to know more about you.",
"contentType": "text/plain"
}
},
{
"from": {
"name": "Jane",
"userId": "jane.doe@example.com"
},
"payload": {
"content": "So tell me, what is the most important quality that you acquired over all of your professional career?",
"contentType": "text/plain"
}
}
]
}
Sending Binary Messages with Audio
The client needs to send the audio to Service by converting the audio stream into a series of audio chunks. Each chunk of audio carries a segment of audio that needs to be processed. The maximum size of a single audio chunk is 8,192 bytes.
Service Messages
This section describes the messages that originate in Service and are sent to the client.
Service sends mainly two types of messages (message_response
, insight_response
) to the client as soon as they're available.
Message Response
The message_response
contains the processed messages as soon as they're ready and available, in the processing of continuous audio
stream. This message does not contain any insights.
Insight Response
Example of the
insight_response
object
{
"type": "insight_response",
"insights": [
{
"type": "question",
"text": "So tell me, what is the most important quality that you acquired over all of your professional career?",
"confidence": 0.9997962117195129,
"hints": [],
"tags": []
},
{
"type": "action_item",
"text": "Jane will look into the requirements on the hiring for coming financial year.",
"confidence": 0.9972074778643447,
"hints": [],
"tags": [
{
"type": "person",
"text": "Jane",
"beginOffset": 0,
"value": {
"value": {
"name": "Jane",
"alias": "Jane",
"userId": "jane.doe@symbl.ai"
}
}
}
]
}
]
}
The insight_response
contains the insights from the ongoing conversation as soon as they are available. This message does not contain any
messages.
Async API
The Async API provides a REST interface to allow you to run a job asynchronously in order to process insights out of audio and text files.
Text API
POST Async Text API
The Async Text API allows you to process any text payload to get the transcription and conversational insights. It can be useful in any use case where you have access to the textual content of a type of conversation, and you want to extract the insightful items supported by the Conversation API.
If you want to add more content to the same conversation, use PUT Async Text API.
Use the POST API to upload your content and generate a Conversation ID. If you want to append additional content to the same Conversation ID, use the PUT API.
Example API call
curl --location --request POST 'https://api.symbl.ai/v1/process/text' \
--header 'x-api-key: <generated_valid_token>' \
--header 'Content-Type: application/json' \
--data-raw '{
"name": "Your meeting name",
"metadata" : { "conversationType": ["sales"] },
"entities": [{"customType": "Hiring Process", "text": "internships"}],
"detectActionPhraseForMessages":true,
"messages": [
{
"payload": {
"content": "Hi Mike, Natalia here. Hope you don’t mind me reaching out. Who would be the best possible person to discuss internships and student recruitment at ABC Corp? Would you mind pointing me toward the right person and the best way to reach them? Thanks in advance for your help, I really appreciate it!"
},
"from": {
"userId": "natalia@example.com",
"name": "Natalia"
},
"duration":{
"startTime":"2020-07-21T16:02:19.01Z",
"endTime":"2020-07-21T16:04:19.99Z"
}
},
{
"payload": {
"content": "Hey Natalia, thanks for reaching out. I am connecting you with Steve who handles recruitements for us."
},
"from": {
"userId": "mike@abccorp.com",
"name": "Mike"
},
"duration":{
"startTime":"2020-07-21T16:04:19.99Z",
"endTime":"2020-07-21T16:04:20.99Z"
}
}
]
}'
const request = require('request');
const options = {
'method': 'POST',
'url': 'https://api.symbl.ai/v1/process/text',
'headers': {
'Content-Type': 'application/json',
'x-api-key': '<your_auth_token>'
},
body: JSON.stringify({
"name": "Your meeting name",
"metadata" : { "conversationType": ["sales"] },
"entities": [{"customType": "Hiring Process", "text": "internships"}],
"detectActionPhraseForMessages":true,
"messages": [
{
"payload": {
"content": "Hi Mike, Natalia here. Hope you don’t mind me reaching out. Who would be the best possible person to discuss internships and student recruitment at ABC Corp? Would you mind pointing me toward the right person and the best way to reach them? Thanks in advance for your help, I really appreciate it!"
},
"from": {
"userId": "natalia@example.com",
"name": "Natalia"
},
"duration":{
"startTime":"2020-07-21T16:02:19.01Z",
"endTime":"2020-07-21T16:04:19.99Z"
}
},
{
"payload": {
"content": "Hey Natalia, thanks for reaching out. I am connecting you with Steve who handles recruitements for us."
},
"from": {
"userId": "mike@abccorp.com",
"name": "Mike"
},
"duration":{
"startTime":"2020-07-21T16:04:19.99Z",
"endTime":"2020-07-21T16:04:20.99Z"
}
}
],
"confidenceThreshold": 0.5
})
};
request(options, function (error, response) {
if (error) throw new Error(error);
console.log(response.body);
});
The above request returns a response structured like this:
{
"conversationId": "5815170693595136",
"jobId": "9b1deb4d-3b7d-4bad-9bdd-2b0d7b3dcb6d"
}
HTTP REQUEST
POST https://api.symbl.ai/v1/process/text
Request Headers
Header Name | Required | Value |
---|---|---|
x-api-key |
Yes | your_auth_token |
Content-Type |
Yes | application/json |
Request Body
Field | Required | Type | Default | Description |
---|---|---|---|---|
name |
No | string | conversationId |
Your meeting name. |
messages |
Yes | list | Input Messages to look for insights. See the messages section below for more details. | |
confidenceThreshold |
No | double | 0.5 | Minimum required confidence for the insight to be recognized. Range is between 0.0 to 1.0 |
detectActionPhraseForMessages |
No | boolean | false | When true is passed it shows Actionable Phrases in each sentence of conversation. These sentences can be found in the Conversation's Messages API. |
entities |
No | list | Input custom entities which can be detected in conversation using Entities' API. For examples, check the sample code on the right. | |
metadata |
No | JSON | By providing details about your conversation in metadata our algorithm optimizes data discovery. Currently, metadata supports conversationType which optimizes action-items, for more details see the metadata section below. |
messages
Field | Required | Type | Description |
---|---|---|---|
payload |
Yes | object | Input Messages to look for insights. See the messages section below for more details. |
from |
No | object | Information about the User information produced the content of this message. |
duration |
No | object | Duration object containing startTime and endTime for the transcript. |
payload
Field | Required | Type | Default | Description |
---|---|---|---|---|
contentType |
No | string (MIME type) | text/plain | To indicate the type and/or format of the content. Please see RFC 6838 for more details. Currently only text/plain is supported. |
from |
No | object | The content of the message in the specified MIME type in the contentType field. |
from (user)
Field | Required | Type | Description |
---|---|---|---|
name |
No | string | Name of the user. |
userId |
No | string | A unique identifier of the user. E-mail ID is usually a preferred identifier for the user. |
duration
Field | Required | Type | Description |
---|---|---|---|
startTime |
No | DateTime | The start time for the particular text content. |
endTime |
No | DateTime | The start time for the particular text content. |
metadata
Field | Required | Type | Description |
---|---|---|---|
conversationType |
No | list | Currently, we only support ['sales'] , this optimizes Symbl algorithm to detect sales specific action-items in a meeting. To know more when to use this conversationType, click here. |
Query Params
Parameter | Required | Value |
---|---|---|
webhookUrl |
No | Webhook url on which job updates to be sent. (This should be post API) |
WebhookUrl will be used to send the status of job created for uploaded file. Every time the status of the job changes it will be notified on the WebhookUrl
Response Object
Field | Description |
---|---|
conversationId |
ID to be used with Conversation API |
jobId |
ID to be used with Job API |
Response on reaching limit
Field | Description |
---|---|
Payload |
{ "message" : "This API has a limit of maximum of 5 number of concurrent jobs per account. If you are looking to scale, and need more concurrent jobs than this limit, please contact us at support@symbl.ai" } |
Header |
{ "statusCode" : 429 } |
Webhook Payload
Field | Description |
---|---|
jobId |
ID to be used with Job API |
status |
Current status of the job. (Valid statuses - [ scheduled , in_progress , completed ]) |
PUT Async Text API
The Async Text API allows you to process any text payload to append the transcription of the previous conversation and get updated conversational insights.
It can be useful in any use case where you have access to the textual content of a type of conversation,
and you want to extract the insightful items supported by the Conversation API.
Use the POST API to upload your content and generate a Conversation ID. If you want to append additional content to the same Conversation ID, use the PUT API.
Example API call
curl --location --request POST 'https://api.symbl.ai/v1/process/text/:conversationId' \
--header 'x-api-key: <generated_valid_token>' \
--header 'Content-Type: application/json' \
--data-raw '{
"name": "Your meeting name",
"metadata" : { "conversationType": ["sales"] },
"entities": [{"customType": "Hiring Process", "text": "internships"}],
"detectActionPhraseForMessages":true,
"messages": [
{
"payload": {
"content": "Hi Mike, Natalia here. Hope you don’t mind me reaching out. Who would be the best possible person to discuss internships and student recruitment at ABC Corp? Would you mind pointing me toward the right person and the best way to reach them? Thanks in advance for your help, I really appreciate it!"
},
"from": {
"userId": "natalia@example.com",
"name": "Natalia"
},
"duration":{
"startTime":"2020-07-21T16:02:19.01Z",
"endTime":"2020-07-21T16:04:19.99Z"
}
},
{
"payload": {
"content": "Hey Natalia, thanks for reaching out. I am connecting you with Steve who handles recruitements for us."
},
"from": {
"userId": "mike@abccorp.com",
"name": "Mike"
},
"duration":{
"startTime":"2020-07-21T16:04:19.99Z",
"endTime":"2020-07-21T16:04:20.99Z"
}
}
]
}'
const request = require('request');
const options = {
'method': 'PUT',
'url': 'https://api.symbl.ai/v1/process/text/' + your_conversationId,
'headers': {
'Content-Type': 'application/json',
'x-api-key': '<your_auth_token>'
},
'body': JSON.stringify({
"name": "Your meeting name",
"metadata" : { "conversationType": ["sales"] },
"entities": [{"customType": "Hiring Process", "text": "internships"}],
"detectActionPhraseForMessages":true,
"messages": [
{
"payload": {
"content": "Hi Mike, Natalia here. Hope you don’t mind me reaching out. Who would be the best possible person to discuss internships and student recruitment at ABC Corp? Would you mind pointing me toward the right person and the best way to reach them? Thanks in advance for your help, I really appreciate it!"
},
"from": {
"userId": "natalia@example.com",
"name": "Natalia"
},
"duration":{
"startTime":"2020-07-21T16:02:19.01Z",
"endTime":"2020-07-21T16:04:19.99Z"
}
},
{
"payload": {
"content": "Hey Natalia, thanks for reaching out. I am connecting you with Steve who handles recruitements for us."
},
"from": {
"userId": "mike@abccorp.com",
"name": "Mike"
},
"duration":{
"startTime":"2020-07-21T16:04:19.99Z",
"endTime":"2020-07-21T16:04:20.99Z"
}
}
],
"confidenceThreshold": 0.5
})
};
request(options, function (error, response) {
if (error) throw new Error(error);
console.log(response.body);
});
The above request returns a response structured like this:
{
"conversationId": "5815170693595136",
"jobId": "9b1deb4d-3b7d-4bad-9bdd-2b0d7b3dcb6d"
}
HTTP REQUEST
PUT https://api.symbl.ai/v1/process/text/:conversationId
Request Headers
Header Name | Required | Value |
---|---|---|
x-api-key |
Yes | your_auth_token |
Content-Type |
Yes | application/json |
Path Params
Parameter | Value |
---|---|
conversationId |
ConversationId got from the POST Async API for the text content |
Request Body
Field | Required | Type | Default | Description |
---|---|---|---|---|
name |
No | string | conversationId |
Your meeting name. |
messages |
Yes | list | Input Messages to look for insights. See the messages section below for more details. | |
confidenceThreshold |
No | double | 0.5 | Minimum required confidence for the insight to be recognized. Range is from 0.0 to 1.0. |
detectActionPhraseForMessages |
No | boolean | false | It shows Actionable Phrases in each sentence of conversation. Values supported are true and false . These sentences can be found in the Conversation's Messages API. |
entities |
No | list | Input custom entities which can be detected in your conversation using Entities' API. For example, check the sample code on the right. | |
metadata |
No | JSON | By providing details about your conversation in metadata our algorithm optimizes data discovery. Currently, metadata supports conversationType which optimizes action-items, for more details see the metadata section below. |
messages
Field | Required | Type | Description |
---|---|---|---|
payload |
Yes | object | Input Messages to look for insights. See the messages section below for more details. |
from |
No | object | Information about the User information produced the content of this message. |
duration |
No | object | Duration object containing startTime and endTime for the transcript. |
payload
Field | Required | Type | Default | Description |
---|---|---|---|---|
contentType |
No | string (MIME type) | text/plain | To indicate the type and/or format of the content. Please see RFC 6838 for more details. Currently only text/plain is supported. |
from |
No | object | The content of the message in the specified MIME type in the contentType field. |
from (user)
Field | Required | Type | Description |
---|---|---|---|
name |
No | string | Name of the user. |
userId |
No | string | A unique identifier of the user. E-mail ID is usually a preferred identifier for the user. |
duration
Field | Required | Type | Description |
---|---|---|---|
startTime |
No | DateTime | The start time for the particular text content. |
endTime |
No | DateTime | The start time for the particular text content. |
metadata
Field | Required | Type | Description |
---|---|---|---|
conversationType |
No | list | Currently, we only support ['sales'] , this optimizes Symbl algorithm to detect specific action-items present in sales meetings. To know more when to use this conversationType, click here. |
Query Params
Parameter | Required | Value |
---|---|---|
webhookUrl |
No | Webhook url on which job updates to be sent. (This should be post API) |
WebhookUrl will be used to send the status of job created for uploaded file. Every time the status of the job changes it will be notified on the WebhookUrl
Response Object
Field | Description |
---|---|
conversationId |
ID to be used with Conversation API |
jobId |
ID to be used with Job API |
Webhook Payload
Field | Description |
---|---|
jobId |
ID to be used with Job API |
status |
Current status of the job. (Valid statuses - [ scheduled , in_progress , completed , failed ]) |
Audio API
- POST Async Audio API
- PUT Async Audio API
- POST Async Audio URL API
- PUT Async Audio URL API
- Speaker Diarization
System requirement: NodeJS 7+
POST Async Audio API
The Async Audio API allows you to process an audio file and return the full text transcript along with conversational insights. It can be utilized for any use case where you have access to recorded audio and want to extract insights and other conversational attributes supported by Symbl's Conversation API.
Use the POST API to upload your file and generate a Conversation ID. If you want to append additional audio information to the same Conversation ID, use the PUT API.
Example API call - The sample request accepts just the raw audio file from the data with the MIME typeset in the Content-Type Header. The audio file should only have Mono Channel.
# Wave file
curl --location --request POST 'https://api.symbl.ai/v1/process/audio?name="Your meeting name"&webhookUrl=<your_webhook_url>&entities= [{"customType": "Custom_Entity_Type", "text": "Custom Entity to be searched in transcript"}]' \
--header 'Content-Type: audio/wav' \
--header 'x-api-key: <generated_valid_token>' \
--data-binary '@/file/location/audio.wav'
# MP3 File
curl --location --request POST 'https://api.symbl.ai/v1/process/audio?name="Your meeting name"&webhookUrl=<your_webhook_url>&entities= [{"customType": "Custom_Entity_Type", "text": "Custom Entity to be searched in transcript"}]' \
--header 'Content-Type: audio/mpeg' \
--header 'x-api-key: <generated_valid_token>' \
--data-binary '@/file/location/audio.mp3'
const request = require('request');
const fs = require('fs');
const accessToken = "<your_auth_token>";
const audioFileStream = fs.createReadStream('/file/location/audio.wav');
const audioOption = {
url: 'https://api.symbl.ai/v1/process/audio',
headers: {
'x-api-key': accessToken,
'Content-Type': 'audio/wav'
},
qs: {
'name':'Your meeting name',
'webhookUrl': 'https://your_webhook_url',
'entitie's: [{'customType': 'Custom_Entity_Type', 'text': 'Custom Entity to be searched in transcript'}]
},
json: true,
};
audioFileStream.pipe(request.post(audioOption, (err, response, body) => {
console.log(err, body);
}));
The above request returns a response structured like this:
{
"conversationId": "5815170693595136",
"jobId": "9b1deb4d-3b7d-4bad-9bdd-2b0d7b3dcb6d"
}
HTTP REQUEST
POST https://api.symbl.ai/v1/process/audio
Request Headers
Header Name | Required | Value |
---|---|---|
x-api-key |
Yes | your_auth_token |
Content-Type |
No | Describes the format and codec of the provided audio data. Accepted values are audio/wav , audio/mpeg , audio/mp3 and audio/wave . |
** Content-Type: This field is optional.
If you're not sure about audio format, you can omit it since the API will automatically detect the content type.
But when it's mentioned, audio format is validated.
Query Params
Parameter | Required | Value |
---|---|---|
name |
No | Your meeting name. (When not specified, it will default to the conversationId .) |
webhookUrl |
No | Webhook url on which job updates to be sent. (This should be post API) |
customVocabulary |
No | Contains a list of words and phrases that provide hints to the speech recognition task. |
entities |
No | Input custom entities which can be detected in your conversation using Entities' API. For example, check the sample code on right. |
detectActionPhraseForMessages |
No | Accepted values are true & false . It shows Actionable Phrases in each sentence of conversation. These sentences can be found in the Conversation's Messages API. |
conversationType |
No | Currently, we only support ['sales'] , this optimizes Symbl algorithm to detect sales specific action-items present in a meeting. To know more when to use this conversationType, click here. |
WebhookUrl will be used to send the status of job created for uploaded audio. Every time the status of the job changes it will be notified on the WebhookUrl
Response Object on Success
Field | Description |
---|---|
conversationId |
ID to be used with Conversation API |
jobId |
ID to be used with Job API |
Response on reaching limit
Field | Description |
---|---|
Payload |
{ "message" : "This API has a limit of maximum of 5 number of concurrent jobs per account. If you are looking to scale, and need more concurrent jobs than this limit, please contact us at support@symbl.ai" } |
Header |
{ "statusCode" : 429 } |
Webhook Payload
Field | Description |
---|---|
jobId |
ID to be used with Job API |
status |
Current status of the job. (Valid statuses - [ scheduled , in_progress , completed , failed ]) |
PUT Async Audio API
The Async Audio API allows you to process an additional audio file to the previous conversation, append the transcription and get conversational insights for updated conversation.
It can be useful in any use case where you have access to multiple audio files of any type of conversation,
and you want to extract the insightful items supported by the Conversation API.
Use the POST API to upload your file and generate a Conversation ID. If you want to append additional audio information to the same Conversation ID, use the PUT API.
Example API call - The sample request accepts just the raw audio file from the data with the MIME typeset in the Content-Type Header. The audio file should only have Mono Channel.
# Wave file
curl --location --request PUT 'https://api.symbl.ai/v1/process/audio/:conversationId?name="Your meeting name"&webhookUrl=<your_webhook_url>&entities= [{"customType": "Custom_Entity_Type", "text": "Custom Entity to be searched in transcript"}]' \
--header 'Content-Type: audio/wav' \
--header 'x-api-key: <generated_valid_token>' \
--data-binary '@/file/location/audio.wav'
# MP3 File
curl --location --request PUT 'https://api.symbl.ai/v1/process/audio/:conversationId?name="Your meeting name"&webhookUrl=<your_webhook_url>&entities= [{"customType": "Custom_Entity_Type", "text": "Custom Entity to be searched in transcript"}]' \
--header 'Content-Type: audio/mpeg' \
--header 'x-api-key: <generated_valid_token>' \
--data-binary '@/file/location/audio.mp3'
const request = require('request');
const fs = require('fs');
const accessToken = "<your_auth_token>";
const audioFileStream = fs.createReadStream('/file/location/audio.wav');
const audioOption = {
url: 'https://api.symbl.ai/v1/process/audio' + your_conversationId,
headers: {
'x-api-key': accessToken,
'Content-Type': 'audio/wav'
},
qs: {
'name':'Your meeting name',
'webhookUrl': 'https://your_webhook_url',
'entities': [{'customType': 'Custom_Entity_Type', 'text': 'Custom Entity to be searched in transcript'}]
},
json: true,
};
audioFileStream.pipe(request.put(audioOption, (err, response, body) => {
console.log(err, body);
}));
The above request returns a response structured like this:
{
"conversationId": "5815170693595136",
"jobId": "9b1deb4d-3b7d-4bad-9bdd-2b0d7b3dcb6d"
}
HTTP REQUEST
PUT https://api.symbl.ai/v1/process/audio/:conversationId
Request Headers
Header Name | Required | Value |
---|---|---|
x-api-key |
Yes | your_auth_token |
Content-Type |
No | Describes the format and codec of the provided audio data. Accepted values are audio/wav , audio/mpeg , audio/mp3 and audio/wave . |
** Content-Type: This field is optional.
If you're not sure about audio format, you can omit it since the API will automatically detect the content type.
But when it's mentioned, audio format is validated.
Path Params
Parameter | value |
---|---|
conversationId |
conversationId which is provided by the first request submitted using POST async audio API |
Query Params
Parameter | Required | Value |
---|---|---|
name |
No | Your meeting name. (When not specified, it will default to the conversationId .) |
webhookUrl |
No | Webhook url on which job updates to be sent. (This should be post API) |
customVocabulary |
No | Contains a list of words and phrases that provide hints to the speech recognition task. |
detectActionPhraseForMessages |
No | Accepted values are true & false . It shows Actionable Phrases in each sentence of conversation. These sentences can be found in the Conversation's Messages API. |
entities |
No | Input custom entities which can be detected in your conversation using Entities' API. For example, check the sample code on right. |
conversationType |
No | Currently, we only support ['sales'] , this optimizes Symbl algorithm to detect sales specific action-items present in a meeting. To know more when to use this conversationType, click here. |
WebhookUrl will be used to send the status of job created for uploaded audio. Every time the status of the job changes it will be notified on the WebhookUrl
Response Object
Field | Description |
---|---|
conversationId |
ID to be used with Conversation API |
jobId |
ID to be used with Job API |
Webhook Payload
Field | Description |
---|---|
jobId |
ID to be used with Job API |
status |
Current status of the job. (Valid statuses - [ scheduled , in_progress , completed , failed ]) |
POST Async Audio URL API
The Async Audio URL API takes in a url link of your audio and returns the full text transcript along with conversational insights. It can be utilized for any use case where you have access to recorded audio stored publicly as URL and want to extract insights and other conversational attributes supported by Symbl's Conversation API.
Use the POST API to upload your file and generate a Conversation ID. If you want to append additional audio information to the same Conversation ID, use the PUT API.
Example API call
curl --location --request POST 'https://api.symbl.ai/v1/process/audio/url?name="Your meeting name"&webhookUrl=<your_webhook_url>&entities= [{"customType": "Custom_Entity_Type", "text": "Custom Entity to be searched in transcript"}]' \
--header 'Content-Type: application/json' \
--header 'x-api-key: <generated_valid_token>' \
--data-raw '{
"url": "https://symbltestdata.s3.us-east-2.amazonaws.com/sample_audio_file.wav",
"confidenceThreshold": 0.6,
"timezoneOffset": 0
}'
const request = require('request');
const fs = require('fs');
const accessToken = "<your_auth_token>";
const audioOption = {
url: 'https://api.symbl.ai/v1/process/audio/url',
headers: {
'x-api-key': accessToken,
'Content-Type': 'application/json'
},
qs: {
'name':'Your meeting name',
'webhookUrl': 'https://your_webhook_url',
'entities': [{'customType': 'Custom_Entity_Type', 'text': 'Custom Entity to be searched in transcript'}]
},
body: JSON.stringify({
"url": "https://symbltestdata.s3.us-east-2.amazonaws.com/sample_audio_file.wav",
"confidenceThreshold": 0.6,
"timezoneOffset": 0
})
};
request.post(audioOption, (err, response, body) => {
console.log(err, body);
});
The above request returns a response structured like this:
{
"conversationId": "5815170693595136",
"jobId": "9b1deb4d-3b7d-4bad-9bdd-2b0d7b3dcb6d"
}
HTTP REQUEST
POST https://api.symbl.ai/v1/process/audio/url
Request Headers
Header Name | Required | Value |
---|---|---|
x-api-key |
Yes | your_auth_token |
Content-Type |
Yes | Accepted value application/json |
Request Body
Field | Required | Type | Supported Values | Default | Description |
---|---|---|---|---|---|
url |
Yes | String | [] | A valid url string. The URL must be a publicly accessible url. | |
customVocabulary |
No | list | [] | Contains a list of words and phrases that provide hints to the speech recognition task. | |
confidenceThreshold |
No | double | 0.5 to 1.0 | 0.5 | Minimum required confidence for the insight to be recognized. |
Query Params
Parameter | Required | Value |
---|---|---|
name |
No | Your meeting name. (When not specified, it will default to the conversationId .) |
webhookUrl |
No | Webhook url on which job updates to be sent. (This should be post API) |
detectActionPhraseForMessages |
No | Accepted values are true & false . It shows Actionable Phrases in each sentence of conversation. These sentences can be found in the Conversation's Messages API. |
entities |
No | Input custom entities which can be detected in your conversation using Entities' API. For example, check the sample code on right. |
conversationType |
No | Currently, we only support ['sales'] , this optimizes Symbl algorithm to detect sales specific action-items present in a meeting. To know more when to use this conversationType, click here. |
WebhookUrl will be used to send the status of job created for uploaded audio url. Every time the status of the job changes it will be notified on the WebhookUrl
Response Object on Success
Field | Description |
---|---|
conversationId |
ID to be used with Conversation API |
jobId |
ID to be used with Job API |
Response on reaching limit
Field | Description |
---|---|
Payload |
{ "message" : "This API has a limit of maximum of 5 number of concurrent jobs per account. If you are looking to scale, and need more concurrent jobs than this limit, please contact us at support@symbl.ai" } |
Header |
{ "statusCode" : 429 } |
Webhook Payload
Field | Description |
---|---|
jobId |
ID to be used with Job API |
status |
Current status of the job. (Valid statuses - [ scheduled , in_progress , completed , failed ]) |
PUT Async Audio URL API
The Async Audio URL API allows you to append an additional audio url to the previous conversation, append the transcription and get conversational insights for updated conversation.
It can be useful in any use case where you have access to multiple recorded audio stored publicly as URL of any type of conversation,
and you want to extract the insightful items supported by the Conversation API.
Use the POST API to add your URL and generate a Conversation ID. If you want to append additional audio information to the same Conversation ID, use the PUT API.
Example API call - The sample request accepts just the raw audio file from the data with the MIME typeset in the Content-Type Header. The audio file should only have Mono Channel.
curl --location --request PUT 'https://api.symbl.ai/v1/process/audio/url/:conversationId?name="Your meeting name"&webhookUrl=<your_webhook_url>' \
--header 'Content-Type: application/json' \
--header 'x-api-key: <generated_valid_token>' \
--data-raw '{
"url": "https://symbltestdata.s3.us-east-2.amazonaws.com/sample_audio_file.wav",
"confidenceThreshold": 0.6,
"timezoneOffset": 0
}'
const request = require('request');
const fs = require('fs');
const accessToken = "<your_auth_token>";
const audioOption = {
url: 'https://api.symbl.ai/v1/process/audio/url/' + your_conversationId,
headers: {
'x-api-key': accessToken,
'Content-Type': 'application/json'
},
qs: {
'name':'Your meeting name',
'webhookUrl': 'https://your_webhook_url',
},
body: JSON.stringify({
"url": "https://symbltestdata.s3.us-east-2.amazonaws.com/sample_audio_file.wav",
"confidenceThreshold": 0.6,
"timezoneOffset": 0
})
};
request.put(audioOption, (err, response, body) => {
console.log(err, body);
});
The above request returns a response structured like this:
{
"conversationId": "5815170693595136",
"jobId": "9b1deb4d-3b7d-4bad-9bdd-2b0d7b3dcb6d"
}
HTTP REQUEST
PUT https://api.symbl.ai/v1/process/audio/url/:conversationId
Request Headers
Header Name | Required | Value |
---|---|---|
x-api-key |
Yes | your_auth_token |
Content-Type |
Yes | Accepted values is application/json |
Request Body
Field | Required | Type | Supported Values | Default | Description |
---|---|---|---|---|---|
url |
Yes | String | [] | A valid url string. The URL must be a publicly accessible url. | |
customVocabulary |
No | list | [] | Contains a list of words and phrases that provide hints to the speech recognition task. | |
confidenceThreshold |
No | double | 0.5 to 1.0 | 0.5 | Minimum required confidence for the insight to be recognized. |
Path Params
Parameter | value |
---|---|
conversationId |
conversationId which is provided by the first request submitted using POST async audio API |
Query Params
Parameter | Required | Value |
---|---|---|
name |
No | Your meeting name. (When not specified, it will default to the conversationId .) |
webhookUrl |
No | Webhook url on which job updates to be sent. (This should be post API) |
detectActionPhraseForMessages |
No | Accepted values are true & false . It shows Actionable Phrases in each sentence of conversation. These sentences can be found in the Conversation's Messages API. |
entities |
No | Input custom entities which can be detected in your conversation using Entities' API. For example, check the sample code on right. |
conversationType |
No | Currently, we only support ['sales'] , this optimizes Symbl algorithm to detect sales specific action-items present in a meeting. To know more when to use this conversationType, click here. |
WebhookUrl will be used to send the status of job created for uploaded audio. Every time the status of the job changes it will be notified on the WebhookUrl
Response Object
Field | Description |
---|---|
conversationId |
ID to be used with Conversation API |
jobId |
ID to be used with Job API |
Webhook Payload
Field | Description |
---|---|
jobId |
ID to be used with Job API |
status |
Current status of the job. (Valid statuses - [ scheduled , in_progress , completed , failed ]) |
Speaker Diarization
The Async Audio APIs can detect and separate unique speakers in a single stream of audio without need of separate speaker events.
To enable this capability with either of the APIs the enableSpeakerDiarization
and diarizationSpeakerCount
query-params need to be passed with the request.
The diarizationSpeakerCount
should be equal to the number of unique speakers in the conversation. If the number varies then this might introduce false positives in the diarized results.
Refer to the How-To Get Speaker Separated Transcripts - Diarization with Async API to get started with this capability!
If you’re looking for similar capability in Real-Time APIs, please refer to Speaker Events and Speaker Separation in WebSocket API sections.
Query Params
Parameter | Required | Value |
---|---|---|
enableSpeakerDiarization |
Yes | Whether the diarization should be enabled for this conversation. Pass this as true to enable this capability. |
diarizationSpeakerCount |
Yes | The number of unique speakers in this conversation. |
Video API
- POST Async Video API
- PUT Async Video API
- POST Async Video URL API
- PUT Async Video URL API
- Speaker Diarization
System requirement: NodeJS 7+
POST Async Video API
The Async Video API takes your video and returns the full text transcript along with conversational insights. It can be useful in any use case where you have access to the video file of any type of conversation, and you want to extract the insightful items supported by the Conversation API.
Example API call - The sample request accepts a raw video file from the data with the MIME typeset in the Content-Type Header.
curl --location --request POST 'https://api.symbl.ai/v1/process/video?name="Your meeting name"&webhookUrl=<your_webhook_url>&entities= [{"customType": "Custom_Entity_Type", "text": "Custom Entity to be searched in transcript"}]' \
--header 'Content-Type: video/mp4' \
--header 'x-api-key: <generated_valid_token>' \
--data-binary '@/file/location/your_video.mp4'
const request = require('request');
const fs = require('fs');
const accessToken = "<your_auth_token>";
const videoFileStream = fs.createReadStream('/file/location/video.mp4');
const videoOption = {
url: 'https://api.symbl.ai/v1/process/video',
headers: {
'x-api-key': accessToken,
'Content-Type': 'video/mp4'
},
qs: {
'name':'Your meeting name',
'webhookUrl': 'https://your_webhook_url',
'entities': [{'customType': 'Custom_Entity_Type', 'text': 'Custom Entity to be searched in transcript'}]
},
json: true,
};
videoFileStream.pipe(request.post(videoOption, (err, response, body) => {
console.log(err, body);
}));
The above request returns a response structured like this:
{
"conversationId": "5815170693595136",
"jobId": "9b1deb4d-3b7d-4bad-9bdd-2b0d7b3dcb6d"
}
HTTP REQUEST
POST https://api.symbl.ai/v1/process/video
Request Headers
Header Name | Required | Value |
---|---|---|
x-api-key |
Yes | your_auth_token |
Content-Type |
Yes | Describes the format and codec of the provided video. Accepted value video/mp4 |
Query Params
Parameter | Required | Value |
---|---|---|
name |
No | Your meeting name. (When not specified, it will default to the conversationId .) |
webhookUrl |
No | Webhook url on which job updates to be sent. (This should be post API) |
customVocabulary |
No | Contains a list of words and phrases that provide hints to the speech recognition task. |
detectActionPhraseForMessages |
No | Accepted values are true & false . It shows Actionable Phrases in each sentence of conversation. These sentences can be found in the Conversation's Messages API. |
entities |
No | Input custom entities which can be detected in your conversation using Entities' API. For example, check the sample code on right. |
conversationType |
No | Currently, we only support ['sales'] , this optimizes Symbl algorithm to detect sales specific action-items present in a meeting. To know more when to use this conversationType, click here. |
WebhookUrl will be used to send the status of job created for uploaded video. Every time the status of the job changes it will be notified on the WebhookUrl
Response Object on Success
Field | Description |
---|---|
conversationId |
ID to be used with Conversation API |
jobId |
ID to be used with Job API |
Response on reaching limit
Field | Description |
---|---|
Payload |
{ "message" : "Too Many Requests" } |
Header |
{ "statusCode" : 429 } |
Webhook Payload
Field | Description |
---|---|
jobId |
ID to be used with Job API |
status |
Current status of the job. (Valid statuses - [ scheduled , in_progress , completed , failed ]) |
PUT Async Video API
The Async Audio API allows you to append an additional audio to the previous conversation, append the transcription and get conversational insights for updated conversation. It can be useful in any use case where you have access to multiple video files of any type of conversation, and you want to extract the insightful items supported by the Conversation API.
Example API call - The sample request accepts just the raw video file from the data with the MIME typeset in the Content-Type Header.
# MP4 File
curl --location --request PUT 'https://api.symbl.ai/v1/process/video/:conversationId?name="Your meeting name"&webhookUrl=<your_webhook_url>&entities= [{"customType": "Custom_Entity_Type", "text": "Custom Entity to be searched in transcript"}]' \
--header 'Content-Type: video/mp4' \
--header 'x-api-key: <generated_valid_token>' \
--data-binary '@/file/location/your_video.mp4'
const request = require('request');
const fs = require('fs');
const accessToken = "<your_auth_token>";
const videoFileStream = fs.createReadStream('/file/location/video.mp4');
const videoOption = {
url: 'https://api.symbl.ai/v1/process/video/' + your_conversationId,
headers: {
'x-api-key': accessToken,
'Content-Type': 'video/mp4'
},
qs: {
'name':'Your meeting name',
'webhookUrl': 'https://your_webhook_url',
'entities': [{'customType': 'Custom_Entity_Type', 'text': 'Custom Entity to be searched in transcript'}]
},
json: true,
};
videoFileStream.pipe(request.put(videoOption, (err, response, body) => {
console.log(err, body);
}));
The above request returns a response structured like this:
{
"conversationId": "5815170693595136",
"jobId": "9b1deb4d-3b7d-4bad-9bdd-2b0d7b3dcb6d"
}
HTTP REQUEST
PUT https://api.symbl.ai/v1/process/video/:conversationId
Request Headers
Header Name | Required | Value |
---|---|---|
x-api-key |
Yes | your_auth_token |
Content-Type |
Yes | Describes the format and codec of the provided video data. Accepted value video/mp4 |
Path Params
Parameter | value |
---|---|
conversationId |
conversationId which is provided by the first request submitted using POST async video API |
Query Params
Parameter | Required | Value |
---|---|---|
name |
No | Your meeting name. (When not specified, it will default to the conversationId .) |
webhookUrl |
No | Webhook url on which job updates to be sent. (This should be post API) |
customVocabulary |
No | Contains a list of words and phrases that provide hints to the speech recognition task. |
detectActionPhraseForMessages |
No | Accepted values are true & false . It shows Actionable Phrases in each sentence of conversation. These sentences can be found in the Conversation's Messages API. |
entities |
No | Input custom entities which can be detected in your conversation using Entities' API. For example, check the sample code on right. |
conversationType |
No | Currently, we only support ['sales'] , this optimizes Symbl algorithm to detect sales specific action-items present in a meeting. To know more when to use this conversationType, click here. |
WebhookUrl will be used to send the status of job created for uploaded video. Every time the status of the job changes it will be notified on the WebhookUrl
Response Object
Field | Description |
---|---|
conversationId |
ID to be used with Conversation API |
jobId |
ID to be used with Job API |
Webhook Payload
Field | Description |
---|---|
jobId |
ID to be used with Job API |
status |
Current status of the job. (Valid statuses - [ scheduled , in_progress , completed , failed ]) |
POST Async Video URL API
The Async Video URL API allows you to process a mp4 video and return the full text transcript along with conversational insights. It can be utilized for any use case where you have access to recorded video stored publicly as URL and want to extract insights and other conversational attributes supported by Symbl's Conversation API.
Use the POST API to upload your file and generate a Conversation ID. If you want to append additional video information to the same Conversation ID, use the PUT API.
Example API call
curl --location --request POST 'https://api.symbl.ai/v1/process/video/url?name="Your meeting name"&webhookUrl=<your_webhook_url>&entities= [{"customType": "Custom_Entity_Type", "text": "Custom Entity to be searched in transcript"}]' \
--header 'Content-Type: application/json' \
--header 'x-api-key: <generated_valid_token>' \
--data-raw '{
"url": "https://symbltestdata.s3.us-east-2.amazonaws.com/sample_video_file.mp4",
"confidenceThreshold": 0.6,
"timezoneOffset": 0
}'
const request = require('request');
const fs = require('fs');
const accessToken = "<your_auth_token>";
const videoOption = {
url: 'https://api.symbl.ai/v1/process/video/url',
headers: {
'x-api-key': accessToken,
'Content-Type': 'application/json'
},
qs: {
'name':'Your meeting name',
'webhookUrl': 'https://your_webhook_url',
'entities': [{'customType': 'Custom_Entity_Type', 'text': 'Custom Entity to be searched in transcript'}]
},
body: JSON.stringify({
"url": "https://symbltestdata.s3.us-east-2.amazonaws.com/sample_video_file.mp4",
"confidenceThreshold": 0.6,
"timezoneOffset": 0
})
};
request.post(videoOption, (err, response, body) => {
console.log(err, body);
});
The above request returns a response structured like this:
{
"conversationId": "5815170693595136",
"jobId": "9b1deb4d-3b7d-4bad-9bdd-2b0d7b3dcb6d"
}
HTTP REQUEST
POST https://api.symbl.ai/v1/process/video/url
Request Headers
Header Name | Required | Value |
---|---|---|
x-api-key |
Yes | your_auth_token |
Content-Type |
Yes | Accepted value application/json |
Request Body
Field | Required | Type | Supported Values | Default | Description |
---|---|---|---|---|---|
url |
Yes | String | [] | A valid url string. The URL must be a publicly accessible url. | |
customVocabulary |
No | list | [] | Contains a list of words and phrases that provide hints to the speech recognition task. |
Query Params
Parameter | Required | Value |
---|---|---|
name |
No | Your meeting name. (When not specified, it will default to the conversationId .) |
webhookUrl |
No | Webhook url on which job updates to be sent. (This should be post API) |
detectActionPhraseForMessages |
No | Accepted values are true & false . It shows Actionable Phrases in each sentence of conversation. These sentences can be found in the Conversation's Messages API. |
entities |
No | Input custom entities which can be detected in your conversation using Entities' API. For example, check the sample code on right. |
conversationType |
No | Currently, we only support ['sales'] , this optimizes Symbl algorithm to detect sales specific action-items present in a meeting. To know more when to use this conversationType, click here. |
WebhookUrl will be used to send the status of job created for uploaded video url. Every time the status of the job changes it will be notified on the WebhookUrl
Response Object on Success
Field | Description |
---|---|
conversationId |
ID to be used with Conversation API |
jobId |
ID to be used with Job API |
Response on reaching limit
Field | Description |
---|---|
Payload |
{ "message" : "This API has a limit of maximum of 5 number of concurrent jobs per account. If you are looking to scale, and need more concurrent jobs than this limit, please contact us at support@symbl.ai" } |
Header |
{ "statusCode" : 429 } |
Webhook Payload
Field | Description |
---|---|
jobId |
ID to be used with Job API |
status |
Current status of the job. (Valid statuses - [ scheduled , in_progress , completed , failed ]) |
PUT Async Video URL API
The Async Video URL API allows you to process an additional video file to the previous conversation, append the transcription and get conversational insights for updated conversation.
It can be useful in any use case where you have access to multiple recorded video stored publicly as URL of any type of conversation,
and you want to extract the insightful items supported by the Conversation API.
Use the POST API to add your URL and generate a Conversation ID. If you want to append additional video information to the same Conversation ID, use the PUT API.
Example API call
curl --location --request PUT 'https://api.symbl.ai/v1/process/video/url/:conversationId?name="Your meeting name"&webhookUrl=<your_webhook_url>&entities= [{"customType": "Custom_Entity_Type", "text": "Custom Entity to be searched in transcript"}]' \
--header 'Content-Type: application/json' \
--header 'x-api-key: <generated_valid_token>' \
--data-raw '{
"url": "https://symbltestdata.s3.us-east-2.amazonaws.com/sample_video_file.mp4",
"confidenceThreshold": 0.6,
"timezoneOffset": 0
}'
const request = require('request');
const fs = require('fs');
const accessToken = "<your_auth_token>";
const videoOption = {
url: 'https://api.symbl.ai/v1/process/video/url/' + your_conversationId,
headers: {
'x-api-key': accessToken,
'Content-Type': 'application/json'
},
qs: {
'name':'Your meeting name',
'webhookUrl': 'https://your_webhook_url',
'entities': [{'customType': 'Custom_Entity_Type', 'text': 'Custom Entity to be searched in transcript'}]
},
body: JSON.stringify({
"url": "https://symbltestdata.s3.us-east-2.amazonaws.com/sample_video_file.mp4",
"confidenceThreshold": 0.6,
"timezoneOffset": 0
})
};
request.put(videoOption, (err, response, body) => {
console.log(err, body);
});
The above request returns a response structured like this:
{
"conversationId": "5815170693595136",
"jobId": "9b1deb4d-3b7d-4bad-9bdd-2b0d7b3dcb6d"
}
HTTP REQUEST
PUT https://api.symbl.ai/v1/process/video/url/:conversationId
Request Headers
Header Name | Required | Value |
---|---|---|
x-api-key |
Yes | your_auth_token |
Content-Type |
Yes | Accepted values is application/json |
Request Body
Field | Required | Type | Supported Values | Default | Description |
---|---|---|---|---|---|
url |
Yes | String | [] | A valid url string. The URL must be a publicly accessible url. | |
customVocabulary |
No | list | [] | Contains a list of words and phrases that provide hints to the speech recognition task. | |
confidenceThreshold |
No | double | 0.0 to 1.0 | 0.5 | Minimum required confidence for the insight to be recognized. |
Path Params
Parameter | value |
---|---|
conversationId |
conversationId which is provided by the first request submitted using POST async video API |
Query Params
Parameter | Required | Value |
---|---|---|
name |
No | Your meeting name. (When not specified, it will default to the conversationId .) |
webhookUrl |
No | Webhook url on which job updates to be sent. (This should be post API) |
detectActionPhraseForMessages |
No | Accepted values are true & false . It shows Actionable Phrases in each sentence of conversation. These sentences can be found in the Conversation's Messages API. |
entities |
No | Input custom entities which can be detected in your conversation using Entities' API. For example, check the sample code on right. |
conversationType |
No | Currently, we only support ['sales'] , this optimizes Symbl algorithm to detect sales specific action-items present in a meeting. To know more when to use this conversationType, click here. |
WebhookUrl will be used to send the status of job created for uploaded video url. Every time the status of the job changes it will be notified on the WebhookUrl
Response Object
Field | Description |
---|---|
conversationId |
ID to be used with Conversation API |
jobId |
ID to be used with Job API |
Webhook Payload
Field | Description |
---|---|
jobId |
ID to be used with Job API |
status |
Current status of the job. (Valid statuses - [ scheduled , in_progress , completed , failed ]) |
Speaker Diarization Async
The Async Video APIs can detect and separate unique speakers in a single stream of video without need of separate speaker events.
To enable this capability with either of the APIs the enableSpeakerDiarization
and diarizationSpeakerCount
query-params need to be passed with the request.
The diarizationSpeakerCount
should be equal to the number of unique speakers in the conversation. If the number varies then this might introduce false positives in the diarized results.
Refer to the How-To Get Speaker Separated Transcripts - Diarization with Async API to get started with this capability!
If you’re looking for similar capability in Real-Time APIs, please refer to Speaker Events and Speaker Separation in WebSocket API sections.
Query Params
Parameter | Required | Value |
---|---|---|
enableSpeakerDiarization |
Yes | Whether the diarization should be enabled for this conversation. Pass this as true to enable this capability. |
diarizationSpeakerCount |
Yes | The number of unique speakers in this conversation. |
Other Languages
The Async Audio and Async Video APIs can work with languages other than English. The following list of languages (with their BCP-47 language-codes) are currently supported:
- English (United States) –
en-US
- English (United Kingdom) –
en-GB
- English (Australia) –
en-AU
- English (Ireland) -
en-IE
- English (India) -
en-IN
- Chinese (China) -
zh-CN
- Russian (Russian Federation) -
ru-RU
- French (Canada) -
fr-CA
- French (France) -
fr-FR
- German (Germany) -
de-DE
- Italian (Italy) –
it-IT
- Dutch (Netherlands) –
nl-NL
- Japanese (Japan) –
ja-JP
- Spanish (United States) –
es-US
- Spanish (Spain) -
es-ES
- Arabic (Saudi Arabia) -
ar-SA
- Hindi (India) -
hi-IN
- Portuguese (Brazil) -
pt-BR
- Portuguese (Portugal) -
pt-PT
To use one of the supported languages use the query-parameter languageCode
with the language-codes specified above.
curl --location --request POST 'https://api.symbl.ai/v1/process/video/url?languageCode=en-US&webhookUrl=<your_webhook_url>' \
--header 'Content-Type: application/json' \
--header 'x-api-key: <generated_valid_token>' \
--data-raw '{
"url": "https://symbltestdata.s3.us-east-2.amazonaws.com/sample_video_file.mp4",
"confidenceThreshold": 0.6,
"timezoneOffset": 0
}'
If the query-parameter languageCode
is not specified then en-US
is used as the default language.
Currently only the messages endpoint of Conversation API will return the transcribed data and insights will be return an empty array.
Query Params
Parameter | Required | Value |
---|---|---|
languageCode |
Yes | The BCP-47 language-code of the language. example: fr-CA |
Job API
The Job Status API is used to retrieve the status of an ongoing Async audio and Async video request. You can use the jobId received in the successful response of the Async API.
API Endpoint
https://api.symbl.ai/v1/job/{jobId}
Example API call
curl --location --request GET 'https://api.symbl.ai/v1/job/{jobId}' \
--header 'Content-Type: application/json' \
--header 'x-api-key: <generated_valid_token>'
const request = require('request');
const your_auth_token = '<your_auth_token>';
request.get({
url: 'https://api.symbl.ai/v1/job/{jobId}',
headers: { 'x-api-key': your_auth_token },
json: true
}, (err, response, body) => {
console.log(body);
});
The above request returns a response structured like this:
{
"id": "9b1deb4d-3b7d-4bad-9bdd-2b0d7b3dcb6d",
"status": "in_progress"
}
HTTP REQUEST
GET https://api.symbl.ai/v1/job/{jobId}
Response Parameters
Parameter | Description |
---|---|
id |
The ID of the Job |
status |
Is one of type scheduled , in_progress , completed , failed |
Conversation API
The Conversation API provides the REST API interface for the management and processing of your conversations
GET Conversation
This API returns the conversation meta-data like meeting name, member name and email, start and end time of the meeting, meeting type and meeting id.
API Endpoint
https://api.symbl.ai/v1/conversations/{conversationId}
Example API call
curl "https://api.symbl.ai/v1/conversations/{conversationId}" \
-H "x-api-key: <api_token>"
const request = require('request');
const your_auth_token = '<your_auth_token>';
request.get({
url: 'https://api.symbl.ai/v1/conversations/{conversationId}',
headers: { 'x-api-key': your_auth_token },
json: true
}, (err, response, body) => {
console.log(body);
});
The above request returns a response structured like this:
{
"id": "5179649407582208",
"type": "meeting",
"name": "Project Meeting #2",
"startTime": "2020-02-12T11:32:08.000Z",
"endTime": "2020-02-12T11:37:31.134Z",
"members": [
{
"name": "John",
"email": "John@example.com"
},
{
"name": "Mary",
"email": "Mary@example.com"
},
{
"name": "Roger",
"email": "Roger@example.com"
}
]
}
HTTP REQUEST
GET https://api.symbl.ai/v1/conversations/{conversationId}
Response Object
Field | Description |
---|---|
id |
unique conversation identifier |
type |
conversation type. default is meeting |
name |
name of the conversation |
startTime |
DateTime value |
endTime |
DateTime value |
members |
list of member objects containing name and email if detected |
DELETE Conversation
This API permanently deletes conversation and all related entities such as messages, insights, topics etc. associated with the conversationId.
NOTE: Once delete operation is successful, all information associated with the conversationId is permanently lost and cannot be recovered later.
curl --location --request DELETE "https://api.symbl.ai/v1/conversations/{conversationId}" \
-H "x-api-key: <api_token>"
const request = require('request');
const your_auth_token = '<your_auth_token>';
request.delete({
url: 'https://api.symbl.ai/v1/conversations/{conversationId}',
headers: { 'x-api-key': your_auth_token },
json: true
}, (err, response, body) => {
console.log(body);
});
The above request returns a response structured like this:
When conversationId is right.
{
"message": "successfully deleted the conversation"
}
When conversationId is wrong.
{
"message": "The conversationId is either invalid or does not exist."
}
HTTP REQUEST
DELETE https://api.symbl.ai/v1/conversations/{conversationId}
GET Messages/Transcript
This API returns a list of all the messages in a conversation. You can use this for providing transcription for video conference, meeting or telephone call.
Sentiment Analysis in messages Beta
In Messages API you can enable sentiment analysis over each message which is being spoken in the conversation.
All you need to do is pass sentiment=true
as a query parameter. Read more about it here.
API Endpoint
https://api.symbl.ai/v1/conversations/{conversationId}/messages
Example API call
curl "https://api.symbl.ai/v1/conversations/{conversationId}/messages" \
-H "x-api-key: <api_token>"
const request = require('request');
const your_auth_token = '<your_auth_token>';
request.get({
url: 'https://api.symbl.ai/v1/conversations/{conversationId}/messages',
headers: { 'x-api-key': your_auth_token },
json: true
}, (err, response, body) => {
console.log(body);
});
The above request returns a response structured like this:
{
"messages": [
{
"id": "6412283618000896",
"text": "Best package for you is $69.99 per month.",
"from": {
"name": "Roger",
"email": "Roger@example.com"
},
"startTime": "2020-07-10T11:16:21.024Z",
"endTime": "2020-07-10T11:16:26.724Z",
"conversationId": "6749556955938816",
"phrases": [
{
"type": "action_phrase",
"text": "$69.99 per month"
}
],
"sentiment": {
"polarity": {
"score": 0.6
},
"suggested": "positive"
},
"words": [
{
"word": "Best",
"startTime": "2020-08-18T11:10:14.536Z",
"endTime": "2020-08-18T11:10:15.536Z",
},
{
"word": "package",
"startTime": "2020-08-18T11:10:16.536Z",
"endTime": "2020-08-18T11:10:17.536Z",
},
{
"word": "for",
"startTime": "2020-08-18T11:10:18.536Z",
"endTime": "2020-08-18T11:10:19.536Z",
},
{
"word": "you",
"startTime": "2020-08-18T11:10:20.536Z",
"endTime": "2020-08-18T11:10:22.536Z",
},
{
"word": "is",
"startTime": "2020-08-18T11:10:22.536Z",
"endTime": "2020-08-18T11:10:25.536Z",
},
{
"word": "$69.99",
"startTime": "2020-08-18T11:10:25.536Z",
"endTime": "2020-08-18T11:10:27.536Z",
},
{
"word": "per",
"startTime": "2020-08-18T11:10:27.536Z",
"endTime": "2020-08-18T11:10:29.536Z",
},
{
"word": "month.",
"startTime": "2020-08-18T11:10:30.536Z",
"endTime": "2020-08-18T11:10:32.536Z",
}]
},
{
"id": "5661493169225728",
"text": "Okay, Where is the file?",
"from": {
"name": "John",
"email": "John@example.com"
}
"startTime": "2020-08-18T11:11:14.536Z",
"endTime": "2020-08-18T11:11:18.536Z",
"conversationId": "5139780136337408",
"phrases": [],
"sentiment": {
"polarity": {
"score": 0.2
},
"suggested": "neutral"
},
"words": [
{
"word": "Okay,",
"startTime": "2020-08-18T11:11:14.536Z",
"endTime": "2020-08-18T11:11:14.936Z"
},
{
"word": "Where",
"startTime": "2020-08-18T11:11:14.936Z",
"endTime": "2020-08-18T11:11:15.436Z"
},
{
"word": "is",
"startTime": "2020-08-18T11:11:16.236Z",
"endTime": "2020-08-18T11:11:16.536Z"
},
{
"word": "the",
"startTime": "2020-08-18T11:11:16.536Z",
"endTime": "2020-08-18T11:11:16.936Z"
},
{
"word": "file?",
"startTime": "2020-08-18T11:11:16.936Z",
"endTime": "2020-08-18T11:11:17.236Z"
}
]
}
HTTP REQUEST
GET https://api.symbl.ai/v1/conversations/{conversationId}/messages
Query Params
Parameter | Required | Value | Description |
---|---|---|---|
verbose |
No | true | Gives you word level timestamps of each sentence. |
sentiment |
No | true | Give you sentiment analysis on each message. |
Response Object
Field | Description |
---|---|
id |
unique message identifier |
text |
message text |
from |
user object with name and email |
startTime |
DateTime value |
endTime |
DateTime value |
conversationId |
unique conversation identifier |
words |
words object with word, startTime and endTime. |
phrases |
It shows the most important action phrases in each sentence. It's enabled when you pass detectActionPhraseForMessages=true during submiting the request in Async and Websocket API. |
sentiment |
Shows the sentiment polarity(intensity of negativity or positivity of a sentene) and suggested sentiment type (positive, negative and neutral) |
PUT Speaker Events
Speaker Events API provides the functionality to update Speakers who spoke in the conversation after it has been processed.
This is achieved by sending the API the list of Speaker Events for that conversation which the API then uses for associating it with the content of the same. This API can be used when you have access to these events for a recorded conversation.
For realtime speaker events integration take a look at Active Speaker Events.
Once the API completes the processing the results are reflected for the Messages and Insights in the conversation.
Speaker Events from Zoom Timeline
We have an open-source utility written in JS to convert the Timeline file from Zoom APIs to Speaker Events API request format available here
This utility also does pre-processing on the events to ensure less noise while associating these with a conversation.
Speaker Event Object
"speakerEvents": [
{
"type": "started_speaking",
"user": {
"id": "4194eb50-357d-4712-a02d-94215ead1064",
"name": "Derek",
"email": "Derek@example.com"
},
"offset": {
"seconds": 0,
"nanos": 5000000000
}
},
{
"type": "stopped_speaking",
"user": {
"id": "4194eb50-357d-4712-a02d-94215ead1064",
"name": "Derek",
"email": "Derek@example.com"
},
"offset": {
"seconds": 15,
"nanos": 5000000000
}
}]
Parameter | Value | Description |
---|---|---|
type |
started_speaking, stopped_speaking | Refers to when a speaker starts speaking and stops. |
user |
JSON | Contains user details |
offset |
JSON | Contains the seconds and nanos at which this speaker event occurred. |
user
Parameter | Value | Description |
---|---|---|
id |
string | Uniquely identifies the speaker within the conversation. |
name |
string | Name of the speaker. |
email |
string | Email id of the speaker. |
API Endpoint
https://api.symbl.ai/v1/conversations/{conversationId}/speakers
Example API call
curl --location --request PUT 'https://api.symbl.ai/v1/conversations/{conversationId}/speakers' \
--header '{api-key}' \
--header 'Content-Type: application/json' \
--data-raw '{
"speakerEvents": [
{
"type": "started_speaking",
"user": {
"id": "4194eb50-357d-4712-a02d-94215ead1064",
"name": "Derek",
"email": "Derek@example.com"
},
"offset": {
"seconds": 0,
"nanos": 5000000000
}
},
{
"type": "stopped_speaking",
"user": {
"id": "4194eb50-357d-4712-a02d-94215ead1064",
"name": "Derek",
"email": "Derek@example.com"
},
"offset": {
"seconds": 15,
"nanos": 5000000000
}
},
{
"type": "started_speaking",
"user": {
"id": "4194eb50-357d-4712-a02d-94215ead2104",
"name": "Richard",
"email": "Richard@example.com"
},
"offset": {
"seconds": 10,
"nanos": 5000000000
}
},
{
"type": "stopped_speaking",
"user": {
"id": "4194eb50-357d-4712-a02d-94215ead2104",
"name": "Richard",
"email": "Richard@example.com"
},
"offset": {
"seconds": 20,
"nanos": 5000000000
}
}
]
}'
const request = require('request');
const your_auth_token = '<your_auth_token>';
request.put({
url: ' https://api.symbl.ai/v1/conversations/{conversationId}/speakers',
headers: { 'x-api-key': your_auth_token },
body: {
"speakerEvents": [
{
"type": "started_speaking",
"user": {
"id": "4194eb50-357d-4712-a02d-94215ead1064",
"name": "Derek",
"email": "Derek@example.com"
},
"offset": {
"seconds": 0,
"nanos": 5000000000
}
},
{
"type": "stopped_speaking",
"user": {
"id": "4194eb50-357d-4712-a02d-94215ead1064",
"name": "Derek",
"email": "Derek@example.com"
},
"offset": {
"seconds": 15,
"nanos": 5000000000
}
},
{
"type": "started_speaking",
"user": {
"id": "4194eb50-357d-4712-a02d-94215ead2104",
"name": "Richard",
"email": "Richard@example.com"
},
"offset": {
"seconds": 10,
"nanos": 5000000000
}
},
{
"type": "stopped_speaking",
"user": {
"id": "4194eb50-357d-4712-a02d-94215ead2104",
"name": "Richard",
"email": "Richard@example.com"
},
"offset": {
"seconds": 20,
"nanos": 5000000000
}
}
]
},
json: true
}, (err, response, body) => {
console.log(body);
});
The above request returns a response structured like this:
When API is successfully called.
{
"message": "Speaker events associated for conversationId: {conversationId} successfully! The update should be reflected in all messages and insights along with this conversation"
}
HTTP REQUEST
PUT https://api.symbl.ai/v1/conversations/{conversationId}/speakers
GET Members
Returns a list of all the members in a conversation. A Member is referred to a participant in the conversation that is uniquely identified as a speaker. Identifying different participants in the meetings can be done by implementing speaker separation.
For more details on identifying members by Speaker Events or Active Talker events in Real-time using Voice SDK - here.
For more details on identifying members by independent audio stream integration using Websocket - here.
API Endpoint
https://api.symbl.ai/v1/conversations/{conversationId}/members
Example API call
curl "https://api.symbl.ai/v1/conversations/{conversationId}/members" \
-H "x-api-key: <api_token>"
const request = require('request');
const your_auth_token = '<your_auth_token>';
request.get({
url: 'https://api.symbl.ai/v1/conversations/{conversationId}/members',
headers: { 'x-api-key': your_auth_token },
json: true
}, (err, response, body) => {
console.log(body);
});
The above request returns a response structured like this:
{
"members": [
{
"id": "fc9b35cd-361f-41c6-9029-0944d21c7150",
"name": "John",
"email": "John@example.com"
},
{
"id": "382362a2-eeec-46a3-8891-d50508293851",
"name": "Mary",
"email": "Mary@example.com"
},
{
"id": "b7de3a33-a16c-4926-9d4d-a904c88271c2",
"name": "Roger",
"email": "Roger@example.com"
}
]
}
HTTP REQUEST
GET https://api.symbl.ai/v1/conversations/{conversationId}/members
Response Object
Field | Description |
---|---|
id |
member's unique identifier |
name |
member's name |
email |
member's email |
PUT Update Members
Update an existing member in an conversation. This API can be used for updating the unique speakers detected as members from diarization as well.
To diarize/separate speakers in a single audio/video stream refer to the How-To Get Speaker Separated Transcripts - Diarization with Async API
For more details on identifying members by Speaker Events or Active Talker events in Real-time using Voice SDK - here.
For more details on identifying members by independent audio stream integration using Websocket - here.
API Endpoint
https://api.symbl.ai/v1/conversations/{conversationId}/members/{id}
Example API call
curl --location --request PUT 'https://api.symbl.ai/v1/conversations/{conversationId}/members/fc9b35cd-361f-41c6-9029-0944d21c7150' --header 'Content-Type: application/json' --header 'x-api-key: <valid-generated-token' --data-raw '{
"id": "fc9b35cd-361f-41c6-9029-0944d21c7150",
"email": "john@example.com",
"name": "John"
}'
const request = require('request');
const your_auth_token = '<your_auth_token>';
request.put({
url: 'https://api.symbl.ai/v1/conversations/{conversationId}/members/fc9b35cd-361f-41c6-9029-0944d21c7150',
headers: { 'x-api-key': your_auth_token },
body: {
id: 'fc9b35cd-361f-41c6-9029-0944d21c7150',
name: 'John',
email: 'john@example.com'
},
json: true
}, (err, response, body) => {
console.log(body);
});
The above request returns a response structured like this:
{
"message": "Member with id: fc9b35cd-361f-41c6-9029-0944d21c7150 for conversationId: <conversationId> updated successfully! The update should be reflected in all messages and insights along with this conversation"
}
HTTP REQUEST
PUT https://api.symbl.ai/v1/conversations/{conversationId}/members/{id}
Request Body
Field | Required | Type | Supported Values | Default | Description |
---|---|---|---|---|---|
id |
Yes | string | The unique identifier of the member for this conversation. This can be retrieved from the members endpoint. | ||
name |
Yes | string | The name of the member. | ||
email |
No | string | The email-id of the member. If specified this can be used to correctly identify and merge the existing user in case the conversation is appended with a new diarized conversation which has one or more same speakers as the conversation it's being appended to. |
Response Object
Field | Description |
---|---|
message |
A description of the update. This message indicates that the member details have now been updated across the conversation for all the messages and action-items, follow-ups and questions. You can also get the updated member from the members endpoint. |
GET Topics
The most relevant topics of discussion from the conversation that are generated based on the combination of the overall scope of the discussion.
This API returns all the topics generated from a conversation.
Sentiment Analysis in TopicsBeta
You can enable sentiment analysis over each topics which is being discussed in the conversation.
All you need to do is pass sentiment=true
as a query parameter. Read more about it here.
Topic HierarchyBeta
You can enable topic hierarchy in Topics API by passing parentRefs=true
. Topic Hierarchy breaks conversation
in parent and child topics which helps outline the entire conversation faster. Read more about it here
API Endpoint
https://api.symbl.ai/v1/conversations/{conversationId}/topics
Example API call
curl "https://api.symbl.ai/v1/{conversationId}/topics" \
-H "x-api-key: <api_token>"
const request = require('request');
const your_auth_token = '<your_auth_token>';
request.get({
url: 'https://api.symbl.ai/v1/{conversationId}/topics',
headers: { 'x-api-key': your_auth_token },
json: true
}, (err, response, body) => {
console.log(body);
});
Topic Hierarchy Sample Response (parentRefs=true):
{
"topics": [
{
"id": "5907389282779136",
"text": "interns",
"type": "topic",
"score": 0.7178597920690242,
"messageIds": [
"4600982711304192",
"5487363432120320",
"6109794119188480"
],
"parentRefs": [
{
"type": "topic",
"text": "company-wise hiring"
}
]
},
{
"id": "5776859730018304",
"text": "company-wise hiring",
"type": "topic",
"score": 0.788856914361565,
"messageIds": [
"6298570346987520",
"6330577953226752"
],
"parentRefs": []
},
{
"id": "6697188878974976",
"text": "new regulations",
"type": "topic",
"score": 0.6968750176932417,
"messageIds": [
"5356560840654848",
"5663440783802368",
"5263998490509312",
"6082396449406976",
"4925138187321344",
],
"parentRefs": [
{
"type": "topic",
"text": "company-wise hiring"
}
]
}
]
}
Sentiment Sample Response (sentiment=true):
{
"topics": [
{
"id": "5907389282779136",
"text": "interns",
"type": "topic",
"score": 0.7178597920690242,
"messageIds": [
"4600982711304192",
"5487363432120320",
"6109794119188480"
],
"sentiment": {
"polarity": {
"score": 0.389
},
"suggested": "positive"
},
"parentRefs": []
},
{
"id": "5776859730018304",
"text": "company-wise hiring",
"type": "topic",
"score": 0.788856914361565,
"messageIds": [
"6298570346987520",
"6330577953226752"
],
"sentiment": {
"polarity": {
"score": 0.012
},
"suggested": "neutral"
},
"parentRefs": []
},
{
"id": "6697188878974976",
"text": "new regulations",
"type": "topic",
"score": 0.6968750176932417,
"messageIds": [
"5356560840654848",
"5663440783802368",
"5263998490509312",
"6082396449406976",
"4925138187321344",
],
"sentiment": {
"polarity": {
"score": -0.809
},
"suggested": "negative"
},
"parentRefs": []
}
]
}
HTTP REQUEST
GET https://api.symbl.ai/v1/conversations/{conversationId}/topics
Query Params
Parameter | Required | Value | Description |
---|---|---|---|
sentiment |
No | true | Give you sentiment analysis on each topic in conversation. |
parentRefs |
No | true | Gives you topic hierarchy. |
Response Object
Field | Description |
---|---|
id |
unique conversation identifier |
text |
conversation text |
type |
response type. default is topics |
score |
confidence score of the generated topic. value from 0 - 1 |
messageIds |
unique message identifiers of the corresponding messages |
entities |
list of detected entity objects in the insight with type - entity type and text - corresponding text |
parentRefs |
This is enabled when parentRefs is set to true in request. For more information read Topic Hierarchy. |
sentiment |
Shows the sentiment polarity(intensity of negativity or positivity of a sentene) and suggested sentiment type (positive, negative and neutral) |
GET Questions
Any explicit question or request for information that comes up during the conversation, whether answered or not, is recognized as a question.
This API returns all the questions generated from the conversation.
API Endpoint
https://api.symbl.ai/v1/conversations/{conversationId}/questions
Example API call
curl "https://api.symbl.ai/v1/conversations/{conversationId}/questions" \
-H "x-api-key: <api_token>"
const request = require('request');
const your_auth_token = '<your_auth_token>';
request.get({
url: 'https://api.symbl.ai/v1/conversations/{conversationId}/questions',
headers: { 'x-api-key': your_auth_token },
json: true
}, (err, response, body) => {
console.log(body);
});
The above request returns a response structured like this:
{
"questions": [
{
"id": "5179649407582208",
"text": "Push them for the two weeks delivery, right?",
"type": "question",
"score": 0.9730208796076476,
"messageIds": [
"5019269922291712"
],
"entities": []
},
{
"id": "5642466493464576",
"text": "I think what is the Bahamas?",
"type": "question",
"score": 0.9119608386876195,
"messageIds": [
"5019269922291712"
],
"entities": []
},
{
"id": "5756718797553664",
"text": "Okay need be detained, or we can go there in person and support them?",
"type": "question",
"score": 0.893303149769215,
"messageIds": [
"5019269922291712"
],
"entities": []
},
{
"id": "6235991715086336",
"text": "Why is that holiday in US from 17?",
"type": "question",
"score": 0.9998053310511206,
"messageIds": [
"5019269922291712"
],
"entities": []
}
]
}
HTTP REQUEST
GET https://api.symbl.ai/v1/conversations/{conversationId}/questions
Response Object
Field | Description |
---|---|
id |
unique conversation identifier |
text |
conversation text |
type |
response type. default is question |
score |
confidence score of the generated topic. value from 0 - 1 |
messageIds |
unique message identifiers of the corresponding messages |
entities |
list of detected entity objects in the insight with type - entity type and text - corresponding text |
GET Action-items
An action item is a specific outcome recognized in the conversation that requires one or more people in the conversation to act in the future.
These actions can be definitive in nature and owned with a commitment to working on a presentation, sharing a file, completing a task, etc.
Or they can be non-definitive like an idea, suggestion or an opinion that could be worked upon.
All action items are generated with action phrases, assignees and due dates so that you can build workflow automation with your own tools.
This API returns list of all the action items generated from the conversation.
API Endpoint
https://api.symbl.ai/v1/conversations/{conversationId}/action-items
Example API call
curl "https://api.symbl.ai/v1/conversations/{conversationId}/action-items" \
-H "x-api-key: <api_token>"
const request = require('request');
const your_auth_token = '<your_auth_token>';
request.get({
url: 'https://api.symbl.ai/v1/conversations/{conversationId}/action-items',
headers: { 'x-api-key': your_auth_token },
json: true
}, (err, response, body) => {
console.log(body);
});
The above request returns a response structured like this:
{
"actionItems": [
{
"id": "5633940379402240",
"text": "Mary thinks we need to go ahead with the TV in Bangalore.",
"type": "action_item",
"score": 0.8659442937321238,
"messageIds": [
"4972726972317696"
],
"phrases": [
{
"type": "action_phrase",
"text": "create small file of like mp44 testing purpose"
}
],
"definitive": false,
"entities": [],
"assignee": {
"name": "Mary",
"email": "Mary@example.com"
}
},
{
"id": "5668855401676800",
"text": "Call and Stephanie also brought up something to check against what Ison is given as so there's one more test that we want to do.",
"type": "action_item",
"score": 0.8660254037845785,
"messageIds": [
"6531517035577342"
],
"phrases": [],
"definitive": true,
"entities": [],
"assignee": {
"name": "John",
"email": "John@example.com"
}
},
{
"id": "5690029162627072",
"text": "Checking the nodes with Eisner to make sure we covered everything so that will be x.",
"type": "action_item",
"score": 0.8657734634985154,
"messageIds": [
"6531517035577244"
],
"phrases": [
{
"type": "action_phrase",
"text": "Checking the nodes with Eisner to make sure we covered everything"
}
],
"definitive": true,
"entities": [],
"assignee": {
"name": "John",
"email": "John@example.com"
}
},
{
"id": "5707174000984064",
"text": "Roger is going to work with the TV lab and make sure that test is also included, so we are checking to make sure not only with our complaints.",
"type": "action_item",
"score": 0.9999962500210938,
"messageIds": [
"6531517035527344"
],
"phrases": [
{
"type":"action_phrase",
"text":"Roger is going to work with the TV lab"
}
],
"definitive": true,
"entities": [],
"assignee": {
"name": "Roger",
"email": "Roger@example.com"
}
},
{
"id": "5757280188366848",
"text": "Mary thinks it really needs to kick start this week which means the call with UV team and our us team needs to happen the next couple of days.",
"type": "action_item",
"score": 0.9999992500008438,
"messageIds": [
"6521517035577344"
],
"phrases": [],
"definitive": false,
"entities": [],
"assignee": {
"name": "Mary",
"email": "Mary@example.com"
},
"dueBy": "2020-02-10T07:00:00.000Z"
}
]
}
HTTP REQUEST
GET https://api.symbl.ai/v1/conversations/{conversationId}/action-items
Response Object
Field | Description |
---|---|
id |
unique conversation identifier |
text |
conversation text |
type |
response type. default is action_item |
score |
confidence score of the generated topic. value from 0 - 1 |
messageIds |
unique message identifiers of the corresponding messages |
entities |
list of detected entity objects in the insight with type - entity type and text - corresponding text |
definitive |
Boolean indicating if the action-item is definitive or not. More about definitive here. |
phrases |
list of detected phrases with type - phrase type and text - corresponding text. The action_phrase type represents the actionable part of an insight. |
assignee |
this field contains the name and email of the person assigned to the action item |
GET Follow-ups
This is a category of action items with a connotation to follow-up a request or
a task like sending an email or making a phone call or booking an appointment
or setting up a meeting.
This API returns list of all the follow ups generated from the conversation.
API Endpoint
https://api.symbl.ai/v1/conversations/{conversationId}/follow-ups
Example API call
curl "https://api.symbl.ai/v1/conversations/{conversationId}/follow-ups" \
-H "x-api-key: <api_token>"
const request = require('request');
const your_auth_token = '<your_auth_token>';
request.get({
url: 'https://api.symbl.ai/v1/conversations/{conversationId}/follow-ups',
headers: { 'x-api-key': your_auth_token },
json: true
}, (err, response, body) => {
console.log(body);
});
The above request returns a response structured like this:
{
"followUps": [
{
"id": "4526427164639111",
"text": "We need to have the meeting today, and we're going to talk about how to run a product strategy Workshop is by Richard Holmes.",
"type": "follow_up",
"score": 0.8660254037851491,
"messageIds": [
"4675554024357888"
],
"entities": [
{
"type": "date",
"text": "today",
"offset": 28,
"value": "2020-06-22"
},
{
"type": "person",
"text": "Richard Holmes",
"offset": 110,
"value": {
"name": "Richard Holmes"
}
}
],
"phrases": [
{
"text": "need to have the meeting today",
"type": "action_phrase"
},
{
"text": "talk about how to run a product strategy Workshop is by Richard Holmes",
"type": "action_phrase"
}
],
"from": {},
"assignee": {},
"dueBy": "2020-06-22T07:00:00.000Z"
}
]
}
HTTP REQUEST
GET https://api.symbl.ai/v1/conversations/{conversationId}/follow-ups
Response Object
Field | Description |
---|---|
id |
unique conversation identifier |
text |
conversation text |
type |
response type. default is follow_up |
score |
confidence score of the generated topic. value from 0 - 1 |
messageIds |
unique message identifiers of the corresponding messages |
entities |
list of detected entity objects in the insight with type - entity type and text - corresponding text |
from |
user object with name and email |
assignee |
this field contains the name and email of the person assigned to the follow up |
phrases |
list of detected phrases with type - phrase type and text - corresponding text. The action_phrase type represents the actionable part of an insight |
GET AnalyticsBeta
Analytics API provides you with functionality like finding speaker ratio, talk time, silence, pace and overlap in a conversation.
For each conversation it returns:
- Speaker Ratio - Speaker’s talk and listen ratio and time.
- Talk Time - Overall duration of the conversation.
- Silence - Overall duration of silence.
- Pace - Words per minute spoken in the conversation.
- Overlap - When more than 1 speaker are speaking the same time, then conversation has overlap.
API Endpoint
https://api.symbl.ai/v1/conversations/{conversationId}/analytics
Example API call
curl "https://api.symbl.ai/v1/conversations/{conversationId}/analytics" \
-H "x-api-key: <api_token>"
const request = require('request');
const your_auth_token = '<your_auth_token>';
request.get({
url: 'https://api.symbl.ai/v1/conversations/{conversationId}/analytics',
headers: { 'x-api-key': your_auth_token },
json: true
}, (err, response, body) => {
console.log(body);
});
The above request returns a response structured like this:
{
"metrics": [
{
"type": "total_silence",
"percent": 29.061,
"seconds": 23.432
},
{
"type": "total_talk_time",
"percent": 70.939,
"seconds": 57.199
},
{
"type": "total_overlap",
"percent": 55.071,
"seconds": 31.5
}
],
"members": [
{
"id": "acc63fbe-c3cd-4daa-8ab0-b088142e5a0f",
"name": "Speaker 1",
"pace": {
"wpm": 68
},
"talkTime": {
"percentage": 40.912,
"seconds": 23.401
},
"listenTime": {
"percentage": 59.088,
"seconds": 33.798
},
"overlap": {
"overlapDuration": 31.5,
"overlappingMembers": [
{
"id": "a52def45-be6e-484f-908b-9ac66eaecabb",
"name": "Speaker 2",
"percent": 61.58,
"seconds": 24.94
},
{
"id": "a52def45-be6e-484f-908b-9ac66eaecacb",
"name": "Speaker 3",
"percent": 7.51,
"seconds": 1.9
},
{
"id": "a52def45-be6e-484f-908b-9ac56eaecabb",
"name": "Speaker 4",
"percent": 12.199,
"seconds": 4.66
}
]
}
},
{
"id": "a52def45-be6e-484f-908b-9ac66eaecabb",
"name": "Speaker 2",
"pace": {
"wpm": 132
},
"talkTime": {
"percentage": 29.894,
"seconds": 17.099
},
"listenTime": {
"percentage": 70.106,
"seconds": 40.1
},
"overlap": {
"overlapDuration": 24.94,
"overlappingMembers": [
{
"id": "acc63fbe-c3cd-4daa-8ab0-b088142e5a0f",
"name": "Speaker 1",
"percent": 61.58,
"seconds": 24.94
}
]
}
},
{
"id": "a52def45-be6e-484f-908b-9ac66eaecacb",
"name": "Speaker 3",
"pace": {
"wpm": 189
},
"talkTime": {
"percentage": 3.322,
"seconds": 1.9
},
"listenTime": {
"percentage": 96.678,
"seconds": 55.299
},
"overlap": {
"overlapDuration": 1.9,
"overlappingMembers": [
{
"id": "acc63fbe-c3cd-4daa-8ab0-b088142e5a0f",
"name": "Speaker 1",
"percent": 7.51,
"seconds": 1.9
}
]
}
},
{
"id": "a52def45-be6e-484f-908b-9ac56eaecabb",
"name": "Speaker 4",
"pace": {
"wpm": 152
},
"talkTime": {
"percentage": 25.873,
"seconds": 14.799
},
"listenTime": {
"percentage": 74.127,
"seconds": 42.4
},
"overlap": {
"overlapDuration": 4.66,
"overlappingMembers": [
{
"id": "acc63fbe-c3cd-4daa-8ab0-b088142e5a0f",
"name": "Speaker 1",
"percent": 12.199,
"seconds": 4.66
}
]
}
}
]
}
HTTP REQUEST
GET https://api.symbl.ai/v1/conversations/{conversationId}/analytics
Response Object
Fields | Description |
---|---|
type | metrics sub-type |
id | unique user identifier |
name | user name |
user email | |
wpm | words per minutes for each speaker |
talkTime | individual speaker’s total talk time |
listenTime | individual speaker’s total listen time |
overlap | member wise talk overlap details |
GET Entities Beta
Entities API provides you with a functionality to extract entities(custom, location, person, date, number, organization,datetime,daterange, etc ) from the conversation.
API Endpoint
https://api.symbl.ai/v1/conversations/{conversationId}/entities
Example API call
curl "https://api.symbl.ai/v1/conversations/{conversationId}/entities" \
-H "x-api-key: <api_token>"
const request = require('request');
const your_auth_token = '<your_auth_token>';
request.get({
url: 'https://api.symbl.ai/v1/conversations/{conversationId}/entities',
headers: { 'x-api-key': your_auth_token },
json: true
}, (err, response, body) => {
console.log(body);
});
The above request returns a response structured like this:
Custom Entity
{
"type": "custom",
"customType": "Company Executives",
"value": "marketing director",
"text": "marketing director",
"messageRefs": [
{
"id": "5118221462011904",
"text": "The marketing director is out-of-town due to some work.",
"offset": 4
}
]
}
Person Entity
{
"type": "person",
"value": "jamie smith",
"text": "jamie smith",
"messageRefs": [
{
"id": "5979280332816384",
"text": "The name is Jamie Smith.", "offset": 12
} ]
}
Organization Entity
{
"type": "organization",
"value": "Vodafone",
"text": "Vodafone",
"messageRefs": [
{
"id": "6141473464516608",
"text": "Hello, this is Peter from Vodafone, I help you today.", "offset": 26
} ]
}
Date Entity:
{
"type": "date",
"value": "2020-07-15",
"text": "today",
"messageRefs": [
{
"id": "6141473464516608",
"text": "Hello, this is Peter from Vodafone, I help you today.", "offset": 47
},
{
"id": "4603163403354112",
"text": "Being a loyal customer at the three types of plan options that I can offer you today.",
"offset": 79
},
{
"id": "5936512994639872",
"text": "Is there anything else I may assist you with today?",
"offset": 45
}
]
}
Number Entity
{
"type": "number",
"value": "two",
"text": "two",
"messageRefs": [
{
"id": "6137109238775808",
"text": "What do you two think of that?",
"offset": 12
}
]
}
HTTP REQUEST
GET https://api.symbl.ai/v1/conversations/{conversationId}/entities
Note:
In Async Text custom entities needs to be passed in API body.
In Async Audio/Async Video API needs to be passed in Query Parameter.
Response Object
Field | Description |
---|---|
type | Defines the type of entity present. |
value | The value of entity. |
text | The text string matched by the algorithm |
messageRefs | Contains message references like id, text and offset of entity in message. |
customType | Optional. Show the custom entity type which was defined. |
Experience API
POST Create Experience
This API returns the URL of the different Summary UIs depending upon the user request body.
Example API Call
# Video Summary
curl --location --request POST 'https://api.symbl.ai/v1/conversations/{conversationId}/experiences' \
--header 'Content-Type: application/json' \
--header 'x-api-key: <generated_valid_token>' \
--data-raw '{
"name": "video-summary",
"videoUrl": "https://storage.googleapis.com/rammer-transcription-bucket/small.mp4",
"logo": "https://symblsanitydataset.s3.us-east-2.amazonaws.com/googleImage.webp"
}'
# Verbose text summary
curl --location --request POST 'https://api.symbl.ai/v1/conversations/{conversationId}/experiences' \
--header 'Content-Type: application/json' \
--header 'x-api-key: <generated_valid_token>' \
--data-raw '{
"name": "verbose-text-summary",
"logo": "https://symblsanitydataset.s3.us-east-2.amazonaws.com/googleImage.webp"
}'
> Video Summary
const request = require('request');
const your_auth_token = '<your_auth_token>';
request.post({
url: 'https://api.symbl.ai/v1/conversations/{conversationId}/experiences',
headers: {
'x-api-key': your_auth_token,
'Content-type': 'application/json',
},
body: JSON.stringify({
"name": "video-summary",
"videoUrl": "https://storage.googleapis.com/rammer-transcription-bucket/small.mp4",
"logo": "https://symblsanitydataset.s3.us-east-2.amazonaws.com/googleImage.webp"
}),
}, (err, response, body) => {
console.log(body);
});
The above request returns a response structured like this:
Response for verbose-text-summary
{
"name": "verbose-text-summary",
"url": "https://meetinginsights.symbl.ai/meeting/#/eyJzZXNzaW9uSWQiOiI0NTMyNTY2NDc2NDU1OTM2In0="
}
Response for video-summary
{
"name": "video-summary",
"url": "https://meetinginsights.symbl.ai/meeting/#/eyJzZXNzaW9uSWQiOiI1ODU5NjczMDg1MzEzMDI0IiwidmlkZW9VcmwiOiJodHRwczovL3N0b3JhZ2UuZ29vZ2xlYXBpcy5jb20vcmFtbWVyLXRyYW5zY3JpcHRpb24tYnVja2V0L3NtYWxsLm1wNCJ9?showVideoSummary=true"
}
HTTP REQUEST
POST
https://api.symbl.ai/v1/conversations/{conversationId}/experiences
Request Headers
Method | Path | Value |
---|---|---|
x-api-key |
true | "Valid Token" |
Content-Type |
true | application/json |
Request Body
Field | Required | Type | Supported Values | Description |
---|---|---|---|---|
name |
true | string | verbose-text-summary, video-summary | For Summary UI use 'verbose-text-summary' and for Video Summary UI use 'video-summary'. |
videoUrl |
false | string | This field is only required when the field 'name' is set to 'video-summary'. | |
logo |
false | string | This field accepts public URL for setting custom logo in Video Summary UI(video-summary) or Summary UI(verbose-text-summary). | |
favicon |
false | string | This field accepts public URL for setting custom favicon in Video Summary UI(video-summary) or Summary UI(verbose-text-summary). |
Video Summary UI
The Video Summary UI provides users the ability to interact with the Symbl elements(transcripts section, Insights, Filters) from a audio and video. It surfaces a screen where users can select key elements like topics, transcripts, and insights and the interface will surface the timestamp where this occurred and begin playback from there.
Features currently supported:
1. Auto scroll: The transcript auto scrolls while the video plays:
2. Video Playback: You can begin playback from a particular timestamp in the transcript by clicking on it:
3. Transcript Navigation: Clicking on an Insight takes you to the location related to the insight in the transcript and begins autoplay of the video from there:
4.Topic Highlights: Selecting topics highlights the topics in the Transcript and generated Insights. In the search bar you will be able to toggle to other Topics along with the ability to autoplay the video from those points as well:
5. Audio file : This is supported by providing an audio file’s url to the videoUrl
param. When an audio file is used, you will be provided a slightly different user experience -- while playing, the current speaker’s name, if available, is displayed where the video would otherwise be. Current audio formats supported are: mp3
and wav
.
How to Enable the Video Summary UI:
You need to add a query-parameter to the existing summary URL:
&showVideoSummary=true
What if there is no video in my url ?
Localized Summary UI
The Localized Summary UI provides users a translated meeting summary page chosen based on one of eight currently supported languages chosen when a session is initiated.
A timezone may also be specified that will change how the start time of the session is rendered in the summary UI.
Summary UI Comparison
German Summary UI
Japanese Summary UI(no insights or topics)
The language code specified will also change the language of the follow-up email after a session has ended.
Testing
Now that you've built out an integration using either the Voice SDK or Voice API, let's test to make sure your integration is working as expected.
- If you are dialed in with your phone number, try speaking the following sentences to see the generated output
"Hey, it was nice meeting you yesterday. Let's catch up again next week over coffee sometime in the evening. I would love to discuss the next steps in our strategic roadmap with you."
"I will set up a meeting with Roger to discuss our budget plan for the next quarter and then plan out how much we can set aside for marketing efforts. I also need to sit down with Jess to talk about the status of the current project. I'll set up a meeting with her probably tomorrow before our standup."
2. If you are dialed into a meeting, try running any of the following videos with your meeting platform open and view the summary email that gets generated:
3. Try tuning your summary page with query parameters to customize your output.
Errors
// example auth token is incorrect
{
"message": "Token validation failed for provided token."
}
Symbl uses the following HTTP codes:
Error Code | Meaning |
---|---|
200 | OK -- Success. |
201 | Accepted -- Your request is successfully accepted. |
400 | Bad Request -- Your request is invalid. |
401 | Unauthorized -- Your API key is invalid. |
403 | Forbidden |
404 | Not Found -- The specified resource does not exist. |
405 | Method Not Allowed -- You tried to access an api with an invalid method. |
413 | Request Entity Too Large. |
429 | Too Many Requests -- Too many requests hit the API too quickly. |
500 | Internal Server Error -- We had a problem with our server. Try again later. |
Resources
Async Audio Conversion
The below snippet shows how you can convert a file from .mp4 to .mp3 using the fluent-ffmpeg node module
const ffmpeg = require('fluent-ffmpeg');
ffmpeg('/my/path/to/original/file.mp4')
.format('mp3')
.on('end', () => {
console.log('end');
})
.on('error', (err) => {
// error
})
.save('/path/to/output/file.mp3');
Async Audio API supports files of either .wav
or .mp3
and the file must have mono-channel audio
only. Any other file formats can be converted using the code snippet from FFmpeg
$ npm install --save fluent-ffmpeg
Automatic Speech Recognition (ASR) Evaluation
This is a simple utility to perform a quick evaluation on the results generated by any Speech to text (STT) or Automatic Speech Recognition (ASR) System.
This utility can calculate following metrics:
Word Error Rate (WER), which is a most common metric of measuring the performance of a Speech Recognition or Machine translation system
Levenshtein Distance calculated at word level.
Number of Word level insertions, deletions and mismatches between the original file and the generated file.
Number of Phrase level insertions, deletions and mismatches between the original file and the generated file.
Color Highlighted text Comparison to visualize the differences.
General Statistics about the original and generated files (bytes, characters, words, new lines etc.)
Installation
$ npm install -g speech-recognition-evaluation
Usage
Simplest way to run your first evaluation is by simply passing original and generated options to asr-eval command. Where, original is a plain text file containing original transcript to be used as reference; usually this is generated by human beings. And generated is a plain text file containing generated transcript by the STT/ASR system.
$ asr-eval --original ./original-file.txt --generated ./generated-file.txt
For more information please visit this.
Media convertor
You can quickly transcode an audio file using transcodeMediaFile method.
const {transcodeMediaFile} = require('symbl-media');
(async () => {
try {
const result = await transcodeMediaFile('./my-input-file.wav', 'my-output-file.mp3', 'mp3');
console.log('Successfully transcoded to: ', result.outPath);
} catch (e) {
console.error(e);
}
})();
Currently this utility only supports one feature:
- Transcode Audio file
This utility can be used as a library in your NodeJS code. You can simply install it in your local project.
$ npm install symbl-media --save
Use the transcode command to transcode the file.
$ media transcode -i ./my-input-file.wav -o ./my-output-file.mp3 -f mp3
For more information please visit this.
Overview
Symbl is an API platform for developers and businesses to rapidly deploy conversational intelligence at scale – on any channel of communication. Our comprehensive suite of APIs unlock proprietary machine learning algorithms that can ingest any form of conversation data to identify actionable insights across domains, timelines, and channels (voice, email, chat, social) contextually – without the need for any upfront training data, wake words, or custom classifiers.
What is conversational intelligence?
In its pure form, conversation intelligence refers to the ability to communicate in ways that create a shared concept of reality. It begins with trust and transparency to remove biases in decisions, enable participants, such as knowledge workers, to be more effective at their core function, eradicate mundane and repetitive parts of their work and empower participants both at work and beyond.
Here at Symbl.ai, we are using hybird approaches of machine learning and deep learning to augment human capability by analyzing conversations and surface knowledge and actions that matter.
How is Symbl.ai different from chatbot platforms?
In short: chatbots are intent-based, rule-based, often launched by ‘wake words’, and enable short conversations between humans and machines. Symbl.ai is a developer platform and service capable of understanding context and meaning in natural conversations between humans. It can surface the things that matter in real-time, e.g. questions, action items, insights, contextual topics, signals, etc.
Additionally:
Chatbots or virtual assistants are commonly command-driven and often referred to as conversation AI systems. They add value to direct human-machine interaction via auditory or textual methods, and attempt to convincingly simulate how a human would behave in a conversation.
You can build chatbots by using existing intent-based systems like RASA, DialogFlow, Watson, Lex, etc. These systems identify intent based on the training data you provide, and these systems enable you to create rule-based conversation workflows between humans and machines.
We are building a platform that can contextually analyze natural conversations between two or more humans based on the meaning as opposed to keywords or wake words. We are also building it using models that require no training, so you can analyze conversations on both audio or text channels to get recommendations of outcomes without needing to train a custom engine for every new intent.
Next: explore supported use cases
Use Cases
Using Symbl you can build use cases for support, sales, collaboration apps and for workflow automation for single or multiple conversations to identify real-time growth opportunities, create indexed knowledge from conversations and drive productivity.
Meetings & UCaaS
Applying primarily to unified communication and collaboration platforms (UCaaS), you can add real-time recommendations of action items and next steps as part of your existing workflow. This would meaningfully improve meeting productivity by surfacing the things that matter, as the meeting occurs. Beyond real-time action items, take advantage of automated meetings summaries delivered to your preferred channel, like email, chat, Slack, calendar, etc.
Use real-time contextual recommendations to enable participants to drive efficiencies in their note-taking, save time and focus more on the meeting itself. Action items are surfaced contextually and in real-time and can be automated to trigger your existing workflows.
Post-meeting summaries are helpful for users that like to get more involved in the conversation as it happens, and prefer re-visiting information and action items post-meeting.
Benefits:
- Humans tend to be slightly subjective when taking notes. Remove bias from notes with an objective conversation intelligence service that contextually surfaces what matters.
- Humans are actually not that great at multitasking. Increase participation and engagement by adding a highly accurate note-taking AI service to the meeting.
- Humans share… most of it. Access and search through complete meeting transcripts, meeting notes, action items, summaries, insights, contextual topics, questions, signals, etc.
- Humans create culture. Understand patterns and trends in your organization’s meeting culture - sentiment, talk ratios, most productive conversations, etc.
Customer Care & CCaaS
As we understand it, customer care performance can be measured by 3 proxy metrics: customer satisfaction, time spent on call, and the number of calls serviced.
What if the introduction of a real-time passive conversation intelligence service into each call was to improve all 3 metrics at once? Real-time contextual understanding leads to suggested actions that a customer care agent can act upon during the call, enabling the agent to:
- Focus on the human connection with the customer.
- Come to a swifter resolution thanks to task automation
- Serve more customers with elevated experience during a shift.
Further, the Symbl.ai platform is also capable of automating post-call data collection. This enables analysis of support conversations over time, agents, shifts,and groups, which leads to a better understanding of pain-points, topics of customer support conversation, etc.
Benefits: Support Organization
- Improved customer experience thanks to more engaged support conversations.
- Reduce average call handling time thanks to automated real-time actions.
- Better data for coaching and benchmarking support staff.
- High-level understanding of topics and summaries of support conversation.
- Emotional analysis of conversational data.
Sales Enablement & CRM
Digital communication platforms used for sales engagements and customer interactions need to capture conversational data for benchmarking performance, improve net sales, and for identifying and replicating the best-performing sales scripts.
Use Symbl.ai to identify top-performing pitches by leveraging real-time insights. Accelerate the sales cycle by automating suggested action items in real-time, such as scheduling tasks and follow-ups via outbound work tool integrations. Keep your CRM up to date by automating the post-call entry with useful summaries.
Benefits: Sales Agent
- Real-time suggested actions.
- Real-time analysis & insights from the conversation.
- Auto-scheduling tasks and follow-ups through outbound work tool integrations.
- Full, searchable transcript of sales conversations.
- Automate the post-call entry into your CRM.
Benefits: Sales Enablement / VP of Sales
- A high-level performance view of the sales function.
- Customizable dashboard to view calls in different filters.
- Understand what works best in a sales call: topics, questions, competitor mentions, etc.
- Replicate best performing scripts to train and coach your whole team to success.
Social Media Conversations
Customers interact a lot with Brands on social media and other digital channels. These interactions include feedback, reviews, complaints, and a lot of other mentions. This is valuable data if used properly to derive insights for the business.
Symbl's APIs can be used along with social listening tools to extract, categorize all of this into actionable insights. For example, topics can be very helpful in abstracting data from product reviews, threads of conversation, and social media comments. Questions and requests from social interactions and forums can be identified to build a knowledge base and direct the customer conversations to the right resources.
With the right integrations to CRM tools and Knowledgebase, insights from social conversations can lead to better understanding customer sentiment towards the brand and more efficient customer service on social channels.
Benefits for Brands
- Extract topics from reviews based on different levels of ratings and identify what leads to good/bad ratings.
- Evaluate influencers, affiliates to work with the brand and ensure right messaging throughout the campaign.
- Understand customer voice from comments and live interactions on Facebook, youtube channels.
- Identify and document questions and requests from specific customers on product forums, comments, and replies to social media posts.
- Guide customers to relevant knowledge base articles or support streams based on their complaints and queries on social media.
- Enrich customer data on CRM based on insights identified from customer-specific social interactions.
Next: Learn more about the capabilities of the platform
Capabilities
Transcript
The platform provides a searchable transcript with word level timestamps and speaker information. The transcript is a refined output of the speech-to-text conversion. The transcript is one of the easiest ways to navigate through the entire conversation. It can be sorted using speaker-specific or topic-specific filters. Additionally, each insight or action item can also lead to related parts of the transcript. Transcripts can generated in both real-time and asynchronous manner for voice and video conversations. They can also be accessed through the pre-built post-conversation summary UI. The pre-built UI enables editing, copying and sharing of transcripts from the conversation and can be enhanced to support the needs of desired user experience.
Summary Topics
Summary topics provide a quick overview of the key things that were talked about in the conversation.
Topic Hierarchy Beta
Topic hierarchy understands customer conversation and returns the parent (global) topic and it's child topics.
Parent Topic: Identify the highest level abstraction of a meeting. These are the key points on which the speakers of the meeting expanded and discussed at length.
Child topic: These are the subtopics that aggregate or are originated from the parent topic itself. Child Topics are the topics which are linked to the parent topic as they form the time chunks of the parent topics in a certain way.
Request object
Query Params
Parameter | Required | Value | Description |
---|---|---|---|
parentRefs |
No | true | Gives you topic hierarchy when passed in Topic API. |
Response object
Field | Description |
---|---|
parentRefs |
For each topic in the response, it has a parentRefs field which point to its parent topic. If parentRefs is NULL it signifies that the topic is a parent. |
Sample Response
{
"topics": [
{ //Child Topic
"id": "5907389282779136",
"text": "salesforce",
"type": "topic",
"score": 0.7178597920690242,
"messageIds": [
"4600982711304192",
"5487363432120320",
"6109794119188480"
],
"parentRefs": [
{
"type": "topic",
"text": "sales conversation"
}
]
},
{ //Parent Topic
"id": "6697188878974976",
"text": "sales conversation",
"type": "topic",
"score": 0.6968750176932417,
"messageIds": [
"5356560840654848",
"5663440783802368",
"5263998490509312",
"6082396449406976",
"4925138187321344",
],
"parentRefs": []
}
]
}
Topic Hierarchy on a Conversation:
In sample diagram below that we created for your understanding. This shows how one conversation can have multiple parent (global) topics and each parent topic can have multiple child-topics in it.
For example, if in a meeting “Sales Conversation” was talked about and after it “Activity Call logs”, “Salesforce”, “Draft”, “Custom Integration” and “Jira” were talked.
So, the topic hierarchy algorithm will find a pattern in the conversation and make Sales Conversation as the parent topics and the rest of the topics as the child topics under it.
Sentiment AnalysisBeta
Sentiment analysis is the interpretation of the general thought, feeling, or sense of an object or a situation.
Sentiment Analysis works over sentence and topic(aspect) level. For topic level, the sentiment is a calculate over the topic messages scope i.e. it factors in the sentiment of messages where the topic was talked about.
Response object
Field | Description |
---|---|
polarity |
show the intensity of the sentiment. It ranges from -1.0 to 1.0, where -1.0 is the most negative sentiment and 1.0 is the most positive sentiment. |
suggested |
display suggested sentiment type (negative, neutral and positive). |
Suggested sentiment
polarity | Suggested Sentiment |
---|---|
-1.0 => x > -0.3 | negative |
-0.3 => x <= 0.3 | neutral |
0.3 < x <= 1.0 | positive |
Action Items
An action item is a specific outcome recognized in the conversation that requires one or more people in the conversation to take a specific action, e.g. set up a meeting, share a file, complete a task, etc.
Action Item Features
- Recognition of the assignee and assignor of an action item when possible.
- Recognition of the date and time specified in the insights.
- Make use of the speaker context to enhance the quality of the insights.
- Ability to accept the timezone to calculate the accurate date and time references.
- Ability to control the confidence threshold for the insights.
- Built-in punctuation and sentence boundary detection.
- There are various types of actionable parts in the conversation between people and the platform can recognize these various connotations.
Tasks
Definitive action items that are not follow-ups are categorized as tasks. Example: "I will complete the presentation that needs to be presented to the management by the end of today". Here, a person is really committed to completing the presentation (task) by the end of today.
conversationType
sales
: ThisconversationType
optimizes Symbl algorithm to optimize Sales specific action-items in a meeting.
It focuses on 3 things : a very specific vocabulary used in Sales, the tense of the statement being made, and the presence of clear subjects, objects and verbs.
Example:
- I will send out that document after this call is over.
- As the next step, I will set up a follow up meeting for next week.
- I will send out that document after this call is over.
Follow Ups
The platform can recognize if an action item has a connotation, which requires following up in general or by someone in particular.
Examples:
- "I will talk to my manager and find out the agreed dates with the vendor". Here, a person needs to follow up with their manager in order to complete this action.
- "Perhaps I can submit the report today". Here, the action of submitting the report is indicated, but the overall connotation of it doesn't indicate the commitment.
Follow-ups can also be non-definitive
Example:
- “We’ll need to sync up with the design team to find out more details”. Here, it’s clear that there needs to be a follow-up, but the details on when and how are not defined.
Type | Follow Up | Non-Follow Up |
---|---|---|
Non-Definitive | Follow Up (non defined) | Idea/Opinion |
Definitive | Follow Up (defined data) | Task |
Other Insight Types
Questions
Any explicit question or request for information that comes up during the conversation, whether answered or not, is recognized as a question. Example:
- “What features are most relevant for our use case?” “How are we planning to design the systems?”
Suggestive Actions
For each of the Action Items identified from the conversation, certain suggestive actions are recommended based on available worktool integrations. Action phrases within the action items can be also used to map to specific actions or trigger workflows based on custom business rules.
Example:
- "Add to calendar" is recommended as a suggestive action for all follow up actions with recommendations for Jira.
- Action Items can have an action phrase “create a Jira ticket” that can map to a Jira action.
Outbound Work Tool Integrations
The platform currently offers email as out-of-box integration with the NodeJS SDK and calendar integration on the pre-built post conversation summary UI. However, this can be extended to any work tool using extensible webhooks, where the actionable insights need to be pushed to enhance productivity and reduce the time taken by users to manually enter information from conversations. The same integrations can be enabled as suggestive actions to make this even quicker.
Some of the examples of these work tools that can be integrated using the extensible webhooks can be:
- Sales platforms such as Salesforce, Copper
- Task management solutions such as Trello, Google Tasks
- Calendars
- Project Management Tools such as Monday, Asana
- Collaboration platforms such as Slack, Flock
Reusable and Customizable UI Components
The UI components can be widely divided into two areas: 1. Symbl JS Elements 2. Prebuilt Summary UI
Symbl JS Elements
Symbl JS elements helps developers embed customizable JS elements for transcription, insights and action items for both real-time and post conversation experience. These are customizable, embeddable components that can be used to simplify the process of building the experience with the desired branding, as applicable.
The Symbl JS elements are releasing in preview shortly, please send an email to devrelations@symbl.ai to get early access to the Symbl JS elements for:
- Live captioning
- Topics
- Action Items
- Suggestive Actions
Prebuilt Summary UI
Symbl provides a few prebuilt summary UI that can be used to generate a user experience of the understanding of the conversation after it has been processed. The pre-built summary UI is available as a URL that can be shared via email to all (or selected) participants or used to embed as a link as part of the conversation history within the application.
The prebuilt summary UI includes the following capabilities:
- Title and details of the conversation including date, number of participants, etc.
- Names of all the participants
- Topics covered in the conversation in the order of importance
- Full, searchable transcript of the conversation. Transcripts can be edited, copied and shared.
- Any Insights, action items or questions from the transcript. Insights can also be edited, shared or dismissed, date/assignee for action item to be modified.
- The prebuilt summary UI can also be customizable, as per the use case or product requirement.