Skype Speech Recognition API with node.js Bot Framework

Question

I'm using the skype client for bot application. I'm using the speech-to-text

There are two patterns for dealing with audio speech: Mention enter link description here BY dandriscoll

client records a small audio file (like a WAV) and uploads it as an attachment to the bot
client sends an audio stream to the bot

Most Bot Framework channels support pattern 1, although client support varies. In this situation, you can upload the WAV to the Bing Speech API and it will return a transcribed result.

The only Bot Framework channel to support pattern 2 is Skype Calling. In this situation, you receive an audio stream and can use the Bing Speech client libraries to get a live transcription feed.

Here want to use the pattern 2 Example code

var restify = require('restify');
var builder = require('botbuilder');
var calling = require('botbuilder-calling');
var prompts = require('./prompts');
var speechService = require('./speech-service.js');

//=========================================================
// Bot Setup
//=========================================================

// Setup Restify Server
var server = restify.createServer();
server.listen(process.env.port || process.env.PORT || 3978, function () {
   console.log('%s listening to %s', server.name, server.url); 
});

// Create chat bot
var chatConnector = new builder.ChatConnector({
    appId: process.env.MICROSOFT_APP_ID,
    appPassword: process.env.MICROSOFT_APP_PASSWORD
});
var chatBot = new builder.UniversalBot(chatConnector);
server.post('/api/messages', chatConnector.listen());

// Create calling bot
var connector = new calling.CallConnector({
    callbackUrl: 'https://example.in/api/calls',
    appId: process.env.MICROSOFT_APP_ID,
    appPassword: process.env.MICROSOFT_APP_PASSWORD
});
var bot = new calling.UniversalCallBot(connector);
server.post('/api/calls', connector.listen());

//=========================================================
// Chat Dialogs
//=========================================================
// Add root dialog

bot.dialog('/', function (session) {
    session.send('Headfitted Bot application... !');
    session.beginDialog('/record');
});

bot.dialog('/record', [
    function (session) {
        session.send(prompts.record.intro);
        calling.Prompts.record(session, prompts.record.prompt, { playBeep: true });
    },
    function (session, results) {
        if (results.response) {
            console.log(results.response.recordedAudio);
            session.endDialog(prompts.record.result, results.response.lengthOfRecordingInSecs);

        } else {
            session.endDialog(prompts.canceled);
        }
    }
]);

When I run the above demo I received the response in my command prompt

{ recordedAudio: <Buffer 30 26 b2 75 8e 66 cf 11 a6 d9 00 aa 00 62 ce 6c 3d 13 00 00 00 00 00 00 06 00 00 00 01 02 a1 dc ab 8c 47 a9 cf 11 8e e4 00 c0 0c 20 53 65 68 00 00 00 ... >,
lengthOfRecordingInSecs: 2.581 }

Now I wanted to use the recordedAudio and pass to speech to text API function. stream will be my recordedAudio

speechService.getTextFromAudioStream(stream)
    .then(function (text) {
        session.send(processText(text));
    })
    .catch(function (error) {
        session.send('Oops! Something went wrong. Try again later.');
        console.error(error);
    });

I Did search on google and bot Framework but no luck.

@PaulSturm Question is How I can pass the recordedAudio to speechService in stream. — Mukesh S
After spend week to solve this issue I found the Bing Speech API only supports WAV/PCM. So I have started to convert the wma to wav file and pass the wav stream to speech to text web api. using ffmpeg. thank to github.com/Microsoft/BotBuilder-Samples/issues/52 user pcostantini post — Mukesh S

Gultekin Gultekin · Accepted Answer · 2017-11-25T06:41:14

I believe that you need to send audio as Base64. At least the rest api of BING SPEECH uses audio as base64. You can test their browser demos and sniff the network. They send something like below.

Request URL:https://api.mixpanel.com/track/?data=eyJldmVudCI6ICIkY3JlYXRlX2FsaWFzIiwicHJvcGVydGllcyI6IHsiJG9zIjogIk1hYyBPUyBYIiwiJGJyb3dzZXIiOiAiQ2hyb21lIiwiJHJlZmVycmVyIjogImh0dHBzOi8vYXp1cmUubWljcm9zb2Z0LmNvbS90ci10ci9zZXJ2aWNlcy9jb2duaXRpdmUtc2VydmljZXMvc3BlZWNoLyIsIiRyZWZlcnJpbmdfZG9tYWluIjogImF6dXJlLm1pY3Jvc29mdC5jb20iLCIkY3VycmVudF91cmwiOiAiaHR0cHM6Ly9henVyZS5taWNyb3NvZnQuY29tL3RyLXRyL3NlcnZpY2VzL2NvZ25pdGl2ZS1zZXJ2aWNlcy9zcGVlY2gvIiwiJGJyb3dzZXJfdmVyc2lvbiI6IDYyLCIkc2NyZWVuX2hlaWdodCI6IDEwNTAsIiRzY3JlZW5fd2lkdGgiOiAxNjgwLCJtcF9saWIiOiAid2ViIiwiJGxpYl92ZXJzaW9uIjogIjIuMTQuMCIsImRpc3RpbmN0X2lkIjogIjE1ZmYxZGY2YjIwYWZlLTA0MzVjYmVmNDA2N2IxLTE3M2Y2ZDU2LTFhZWFhMC0xNWZmMWRmNmIyMWQxMSIsIiRzZWFyY2hfZW5naW5lIjogImdvb2dsZSIsIiRpbml0aWFsX3JlZmVycmVyIjogImh0dHBzOi8vd3d3Lmdvb2dsZS5jb20udHIvIiwiJGluaXRpYWxfcmVmZXJyaW5nX2RvbWFpbiI6ICJ3d3cuZ29vZ2xlLmNvbS50ciIsImFsaWFzIjogIjk1NTI3ZTI1LTA2M2MtNDE5ZS04NzkzLTNkYTNkMGVmNDQ4YSIsInRva2VuIjogIjFkOTJlM2FiZDE0YTJkNjVmNzQ4ZDEzMTRkZDI0Yjk5In19&ip=1&_=1511591715549

Check out the link: Azure sample page

Skype Speech Recognition API with node.js Bot Framework

1 Answers