Microsoft Bot Framework, Adaptive Card 1.1, media play button doesn't display correctly at all, doesn't respond to click correctly on mobile

Question

Microsoft Bot Framework 4.9.1 (version of NuGet packages).

C#, .Net Core 2.1.

Client is simple web page. Test version here:

https://dnctestbot.z13.web.core.windows.net

Type any message (eg. hi). Click button to send message to bot.

Bot shows an Adaptive Card (version 1.1). Markup for video taken from "product video" sample on adaptive card site.

Two problems:

1) In web chat (Win10, all major browsers), the "play" icon does not appear correctly. It is clipped on the right site.

Manually editing the height and width values in Chrome Dev Tools and setting the values to 1px, just as a test, causes the icon to display correctly, but

a) shouldn't have to do this b) no reasonable way to hack this with a CSS rule because the Bot Framework puts the width and height in a style element on the div, so a CSS rule won't change it

2) More important, on mobile, clicking the clipped icon doesn't do the right thing.

On desktop web, clicking the icon plays the video. Expected behavior.

On iPhone 6 and iPhone 7+ (latest iOS), clicking the icon causes the video to appear like it's going to play, but it never does. Not a matter of loading. It just sits there. If you click the icon again, then click it again, the video plays immediately. This is worse.

Things I have tried that do NOT fix or cover over the bugs:

1) Setting a poster image in the card. Image is displayed but doesn't change the clipped icon or the click to play not working on mobile.

2) Setting a playButton element in the card markup, to specify an image to display for the play button. Doesn't do anything. Yes, I declared the adaptive card version 1.1.

Here is the markup for the adaptive card:

{
      "$schema": "http://adaptivecards.io/schemas/adaptive-card.json",
      "type": "AdaptiveCard",
      "version": "1.1",
      "fallbackText": "This card requires Media to be viewed. Ask your platform to update to Adaptive Cards v1.1 for this and more!",
      "body": [
        {
          "type": "Media",
          "sources": [
            {
              "mimeType": "video/mp4",
              "url": "https://adaptivecardsblob.blob.core.windows.net/assets/AdaptiveCardsOverviewVideo.mp4"
            }
          ]
        }
      ],
      "actions": [
        {
          "type": "Action.OpenUrl",
          "title": "Learn more",
          "url": "https://adaptivecards.io"
        }
      ]
    }

How to fix these two bugs?

Thanks!

Adam Leffert

Yes. I'm happy to learn how to customize the web chat client with CSS, and JavaScript if necessary, but I don't want to get into React. Thanks! — MindModel

Kyle Delaney Kyle Delaney · Accepted Answer · 2020-05-18T20:25:57

This kind of bug should be raised in the Adaptive Cards repo. Stack Overflow is not really for bug reports.

That said, I did a little digging and it looks like the Adaptive Cards SDK is trying to render a play button using the left border of a div. You can see their code here:

let playButtonInnerElement = document.createElement("div");
playButtonInnerElement.className = this.hostConfig.makeCssClassName("ac-media-playButton-arrow");
playButtonInnerElement.style.width = playButtonArrowWidth + "px";
playButtonInnerElement.style.height = playButtonArrowHeight + "px";
playButtonInnerElement.style.borderTopWidth = (playButtonArrowHeight / 2) + "px";
playButtonInnerElement.style.borderBottomWidth = (playButtonArrowHeight / 2) + "px";
playButtonInnerElement.style.borderLeftWidth = playButtonArrowWidth + "px";
playButtonInnerElement.style.borderRightWidth = "0";
playButtonInnerElement.style.borderStyle = "solid";
playButtonInnerElement.style.borderTopColor = "transparent";
playButtonInnerElement.style.borderRightColor = "transparent";
playButtonInnerElement.style.borderBottomColor = "transparent";
playButtonInnerElement.style.transform = "translate(" + (playButtonArrowWidth / 10) + "px,0px)";

It seems the reason you're seeing a bad-looking play button is because they're assuming the page will have some kind of universal border-box style. That should be raised as a bug, but it also gives you a way to fix the play button for yourself.

Another workaround may be to replace the whole play button graphic with something you make yourself, perhaps an SVG. I believe you can do this in Web Chat using attachment middleware. This will probably be necessary if you end up needing to replace the whole media player to get it to work on iOS. You can try to get that to work yourself by researching iOS media playback or you can wait for the Adaptive Cards team to take care of it. There seem to be other people who had this kind of issue years ago but their tips might still be relevant: HTML5 Video tag not working in Safari , iPhone and iPad

Microsoft Bot Framework, Adaptive Card 1.1, media play button doesn't display correctly at all, doesn't respond to click correctly on mobile

1 Answers