Get detected entities

This page describes how to get entities that are detected in an existing conversation with the Conversations API, and the message format for detected entity responses from the Streaming API. Both custom and default entities are returned in the responses.

For more information about the Entity Detection feature, see Entity Detection.


Entity Detection - Conversations API

This section provides basic examples of how to get entities that are detected in a conversation using the Conversations API. This request requires a conversation ID. You receive a conversation ID when you process a conversation with the Symbl.ai APIs.

Authentication

These requests require an access token, as described in Authenticate.

Get detected entities

To get entities that are detected in a conversation, use the GET <https://api.symbl.ai/v1/conversations/{conversationId}/entities> operation.

import fetch from 'node-fetch';

const accessToken = '<ACCESS_TOKEN>';
const conversationId = '<CONVERSATION_ID>';

const fetchResponse = await fetch(`https://api.symbl.ai/v1/conversations/${conversationId}/entities`, {
  method: get,
  headers: {
    'Authorization': `Bearer ${accessToken}`,
    'Content-Type': 'application/json'
  }
});

const responseBody = await fetchResponse.json();

console.log(responseBody);

Where:

  • <ACCESS_TOKEN> is a valid API access token.
  • <CONVERSATION_ID> is the ID of a conversation that you previously processed.

Response

The following table describes the response body that is returned by this request.

FieldDescription
entitiesA list of entity objects that are detected in the conversation.
idThe unique identifier of the entity.
typeThe type of the entity.
subTypeThe subtype of the entity.
categoryThe category of the entity. For custom entities, this value is Custom. For default entities, this value is Managed.
matchesA list of match objects that correspond to vocabulary for the entity.
matches.valueThe exact value that matched the entity vocabulary.
matches.messageRefsA list of message objects that contain the detected entity.
matches.messageRefs.idThe message ID.
matches.messageRefs.startTimeThe start time of the message.
matches.messageRefs.endTimeThe end time of the message.
matches.messageRefs.textThe content of the message where the entity was detected.
matches.messageRefs.offsetThe location of the match in the message. The value is the number of characters from the start of the message.

Example response

{
  "entities": [
    {
      "id": "4476908732794496",
      "type": "Vehicle",
      "subType": "TeslaModel",
      "category": "Custom",
      "matches": [
        {
          "value": "Model X",
          "messageRefs": [
            {
              "id": "4667009028587520",
              "startTime": "2020-08-18T11:10:14.536Z",
              "endTime": "2020-08-18T11:10:14.536Z",
              "text": "We are interested in Tesla Model X",
              "offset": 20
            }
          ]
        }
      ]
    },
    {
      "id": "4476908732794496",
      "type": "PolicyNumber",
      "subType": "HealthInsurance",
      "category": "Managed",
      "matches": [
        {
          "value": "998899789ABCD",
          "messageRefs": [
            {
              "id": "4667009028587520",
              "startTime": "2020-08-18T11:10:14.536Z",
              "endTime": "2020-08-18T11:10:14.536Z",
              "text": "The number on my insurance card is 998899789ABCD",
              "offset": 35
            }
          ]
        }
      ]
    }
  ]
}

Entity Detection - Streaming API

This section provides the message format for the response when entities are detected using WebSockets and the Streaming API.

For the Streaming API, the Entity Detection feature is disabled by default. To get detected entities with the Streaming API, you must:

To get started with the Streaming API, see Streaming API.

For reference information, see the Streaming API reference docs.

Configure streaming conversation

To get detected entities with the Streaming API, you must enable the Entity Detection feature for your streaming conversation. You can enable the Entity Detection feature when you start a streaming conversation or during the conversation.

Enable the Entity Detection feature at the start of a conversation

To enable the Entity Detection feature when you start a streaming conversation, use the detectEntities parameter of the Streaming API config object in your start_request:

ws.send(JSON.stringify({
  'type': 'start_request',
  'config': {
    'speechRecognition': {
      'encoding': '<ENCODING>',
      'sampleRateHertz': <SAMPLE_RATE>
    },
    'detectEntities': true
  }
}));

Where:

  • <ENCODING> and SAMPLE_RATE are the audio encoding and sample rate for the device you are using to capture audio. For an example of how to obtain the encoding and sample rate from the browser, see Live speech-to-text and AI insights in browser.

This example includes only the required parameters to configure your streaming conversation to get detected entities. For other request parameters that can be included when you start your streaming conversation, see the Streaming API reference docs.

Message parameters

The following table describes the parameters that can be used with this message.

ParameterRequiredDescription
typeYesValue must be start_request.
configYesStores configuration values for your streaming conversation.
config.speechRecognitionYesStores configuration values for your audio device.
config.speechRecognition.encodingYesSpecifies the type of audio encoding for your audio device.

For more information, see the Streaming API reference docs.
config.speechRecognition.sampleRateHertzYesSpecifies the sample rate of your audio device.

For more information, see the Streaming API reference docs.
config.detectEntitiesYesValue must be true.

If not included, the default value fordetectEntities is false and entities are not detected.

Enable the Entity Detection feature during a conversation

To enable the Entity Detection feature for a streaming conversation that is already in progress, send a modify_request message:

ws.send(JSON.stringify({
  'type': 'modify_request',
  'detectEntities': true
}));
Message parameters

The following table describes the parameters that can be used with this message.

ParameterRequiredDescription
typeYesValue must be modify_request.
detectEntitiesYesValue must be true.

Get detected entities

When an entity is detected during a streaming conversation, the Entity Detection feature sends an entity_response message via the WebSocket connection.

The following table describes the message response that is returned by the server when an entity is detected.

FieldDescription
typeThe value is entity_response.
entitiesA list of entity objects that are detected in the conversation.
idThe unique identifier of the entity.
typeThe type of the entity.
subTypeThe subtype of the entity.
categoryThe category of the entity. For custom entities, this value is Custom. For default entities, this value is Managed.
matchesA list of match objects that correspond to vocabulary for the entity.
matches.valueThe exact value that matched the entity vocabulary.
matches.messageRefsA list of message objects that contain the detected entity.
matches.messageRefs.idThe message ID.
matches.messageRefs.startTimeThe start time of the message.
matches.messageRefs.endTimeThe end time of the message.
matches.messageRefs.textThe content of the message where the entity was detected.
matches.messageRefs.offsetThe location of the match in the message. The value is the number of characters from the start of the message.

Example response

{
  "type": "entity_response",
  "entities": [
    {
      "id": "4476908732794496",
      "type": "Vehicle",
      "subType": "TeslaModel",
      "category": "Custom",
      "matches": [
        {
          "value": "Model X",
          "messageRefs": [
            {
              "id": "4667009028587520",
              "startTime": "2020-08-18T11:10:14.536Z",
              "endTime": "2020-08-18T11:10:14.536Z",
              "text": "We are interested in Tesla Model X",
              "offset": 20
            }
          ]
        }
      ]
    },
    {
      "id": "4476908732794496",
      "type": "PolicyNumber",
      "subType": "HealthInsurance",
      "category": "Managed",
      "matches": [
        {
          "value": "998899789ABCD",
          "messageRefs": [
            {
              "id": "4667009028587520",
              "startTime": "2020-08-18T11:10:14.536Z",
              "endTime": "2020-08-18T11:10:14.536Z",
              "text": "The number on my insurance card is 998899789ABCD",
              "offset": 35
            }
          ]
        }
      ]
    }
  ]
}