How do I integrate a Xamarin forms app with Azure Directline speech bot?

Question

Good day guys, I have a bot built on the azure bot framework and have enabled the direct line speech channel. I have a xamarin forms client that communicates with the bot and is able to receive text responses from the bot. I am unable to play the raw audio responses from the bot. Does anyone have experience doing this? I'd really appreciate the help. Thanks!

Code sample below

private void Connector_ActivityReceived
(object sender, ActivityReceivedEventArgs e)
{
    try
    {
        var json = e.Activity;
        var activity = JsonConvert.DeserializeObject<Activity>(json);
        
        if (e.HasAudio && activity.Speak != null)
        { 
           var audio = e.Audio; // How do I play the contents of e.Audio in xamarin forms?
        }

Hi Jason, the format of the raw audio response is a PullAudioOutputStream object. I'm editing the issue details on this post as well to show a code sample. Thanks! — AFAMEFUNA MILLAINE ANIGBO
Does the Cognitive Services Voice Assistant sample help? (Since there are multiple other people in this thread, you will need to @ mention me if you want me to see your reply.) — Kyle Delaney
Hi @KyleDelaney. Thanks for your comment. The Cognitive Services Voice Assistant samples do help to a large extent. However, they have examples for UWP, .net core, and some other languages. Nothing exists for Xamarin Forms — AFAMEFUNA MILLAINE ANIGBO
Hi @Jason. the format of the raw audio response is a PullAudioOutputStream object. I'm editing the issue details on this post as well to show a code sample. Thanks! — AFAMEFUNA MILLAINE ANIGBO

Alexey Strakh Alexey Strakh · Accepted Answer · 2021-01-07T16:26:06

Based on the documentation description, PullAudioOutputStream represents a memory backed pull audio output stream. Having a stream with the audio data, you can use something like a cross-platform audio player to play it back:

var audio = Plugin.SimpleAudioPlayer.CrossSimpleAudioPlayer.Current;
audio.Load(stream );
audio.Play();

Example found here.

The CrossSimpleAudioPlayer is just an abstraction over native iOS/Android functionality for audio playback, and if you have other preferred ways to implement it natively, you can use dependency service to inject it and use it in your cross-platform Xamarin.Forms core code.

How do I integrate a Xamarin forms app with Azure Directline speech bot?

1 Answers