Get User Input from Adaptive Card using Waterfall Dialog in Botframework v4

Question

I'm trying to put a simple adaptive card in my chatbot that collects the user's name and email. I can't figure out how to actually get the input from the card.

In the waterfall step where I display the dialog. I can't figure out what property should have the JSON string returned from the Action.Submit button.

I've included the json dialog and my TypeScript files. My MainDialog starts ClientCheckDialog on line 146, ClientCheckDialog starts GetContactInfoDialog on line 86

This is the json file dialog:

{
  "$schema": "https://adaptivecards.io/schemas/adaptive-card.json",
  "type": "AdaptiveCard",
  "version": "1.0",
  "body": [
    {
      "type": "TextBlock",
      "text": "Name",
      "wrap": true
    },
    {
      "type": "Input.Text",
      "id": "id_name"
    },
    {
      "type": "TextBlock",
      "text": "Email Address",
      "wrap": true
    },
    {
      "type": "Input.Text",
      "id": "id_email",
      "style": "email",
      "placeholder": "[email protected]"
    }
  ],
  "actions": [
    {
      "type": "Action.Submit",
      "title": "Submit",
      "data": {
        "clickedSubmit" : true
      }
    }
  ]
}

Bot File

import {
    ActivityHandler,
    BotTelemetryClient,
    ConversationState,
    EndOfConversationCodes,
    Severity,
    TurnContext } from 'botbuilder';
import {
    Dialog,
    DialogContext,
    DialogSet,
    DialogState } from 'botbuilder-dialogs';

export class DialogBot<T extends Dialog> extends ActivityHandler {
    private readonly telemetryClient: BotTelemetryClient;
    private readonly solutionName: string = 'tcsBot';
    private readonly rootDialogId: string;
    private readonly dialogs: DialogSet;

    public constructor(
        conversationState: ConversationState,
        telemetryClient: BotTelemetryClient,
        dialog: T) {
        super();

        this.rootDialogId = dialog.id;
        this.telemetryClient = telemetryClient;
        this.dialogs = new DialogSet(conversationState.createProperty<DialogState>(this.solutionName));
        this.dialogs.add(dialog);
        this.onTurn(this.turn.bind(this));
        this.onDialog(this.activityToText.bind(this));
    }

    // eslint-disable-next-line @typescript-eslint/no-explicit-any, @typescript-eslint/tslint/config
    public async turn(turnContext: TurnContext, next: () => Promise<void>): Promise<any> {

        // Client notifying this bot took to long to respond (timed out)
        if (turnContext.activity.code === EndOfConversationCodes.BotTimedOut) {
            this.telemetryClient.trackTrace({
                message: `Timeout in ${ turnContext.activity.channelId } channel: Bot took too long to respond`,
                severityLevel: Severity.Information
            });
            return;
        }

        const dc: DialogContext = await this.dialogs.createContext(turnContext);

        if (dc.activeDialog !== undefined) {
            await dc.continueDialog();
        } else {
            await dc.beginDialog(this.rootDialogId);
        }

        await next();
    }

    public async activityToText(turnContext: TurnContext, next: () => Promise<void>): Promise<any> {
        const activity = turnContext.activity;

        if (!activity.text.trim() && activity.value) {
            activity.text = JSON.stringify(activity.value);
        }
        turnContext.activity.text = JSON.stringify(turnContext.activity.value);
        await next();
    }
}

index.ts file

import {
    BotFrameworkAdapterSettings,
    BotTelemetryClient,
    ConversationState,
    NullTelemetryClient,
    TurnContext,
    UserState
} from 'botbuilder';
import { ApplicationInsightsTelemetryClient, ApplicationInsightsWebserverMiddleware } from 'botbuilder-applicationinsights';
import { LuisApplication } from 'botbuilder-ai';
import {
    CosmosDbStorage,
    CosmosDbStorageSettings
} from 'botbuilder-azure';
import { Dialog } from 'botbuilder-dialogs';
import {
    ISkillManifest
} from 'botbuilder-skills';
import {
    ICognitiveModelConfiguration,
    Locales
} from 'botbuilder-solutions';;
import i18next from 'i18next';
import i18nextNodeFsBackend from 'i18next-node-fs-backend';
import * as path from 'path';
import * as restify from 'restify';
import { DefaultAdapter } from './adapters/defaultAdapter';
import * as appsettings from './appsettings.json';
import { DialogBot } from './bots/dialogBot';
import * as cognitiveModelsRaw from './cognitivemodels.json';
import { MainDialog } from './dialogs/mainDialog';
import { IBotSettings } from './services/botSettings';
import { skills as skillsRaw } from './skills.json';

import { WelcomeDialog } from './dialogs/welcomeDialog'
import { GetContactInfoDialog } from './dialogs/getContactInfoDialog'
import { ServicesDialog } from './dialogs/servicesDialog'
import { ClientCheckDialog } from './dialogs/clientCheckDialog'

// Configure internationalization and default locale
// tslint:disable-next-line: no-floating-promises
i18next.use(i18nextNodeFsBackend)
    .init({
        fallbackLng: 'en',
        preload: ['en', 'fr'],
        backend: {
            loadPath: path.join(__dirname, 'locales', '{{lng}}.json')
        }
    })
    .then(async (): Promise<void> => {
        await Locales.addResourcesFromPath(i18next, 'common');
    });

const skills: ISkillManifest[] = skillsRaw;
const cognitiveModels: Map<string, ICognitiveModelConfiguration> = new Map();
const cognitiveModelDictionary: { [key: string]: Object } = cognitiveModelsRaw.cognitiveModels;
const cognitiveModelMap: Map<string, Object> = new Map(Object.entries(cognitiveModelDictionary));
cognitiveModelMap.forEach((value: Object, key: string): void => {
    cognitiveModels.set(key, <ICognitiveModelConfiguration>value);
});

const botSettings: Partial<IBotSettings> = {
    appInsights: appsettings.appInsights,
    blobStorage: appsettings.blobStorage,
    cognitiveModels: cognitiveModels,
    cosmosDb: appsettings.cosmosDb,
    defaultLocale: cognitiveModelsRaw.defaultLocale,
    microsoftAppId: appsettings.microsoftAppId,
    microsoftAppPassword: appsettings.microsoftAppPassword,
    skills: skills
};

function getTelemetryClient(settings: Partial<IBotSettings>): BotTelemetryClient {
    if (settings !== undefined && settings.appInsights !== undefined && settings.appInsights.instrumentationKey !== undefined) {
        const instrumentationKey: string = settings.appInsights.instrumentationKey;

        return new ApplicationInsightsTelemetryClient(instrumentationKey);
    }

    return new NullTelemetryClient();
}

const telemetryClient: BotTelemetryClient = getTelemetryClient(botSettings);

const adapterSettings: Partial<BotFrameworkAdapterSettings> = {
    appId: botSettings.microsoftAppId,
    appPassword: botSettings.microsoftAppPassword
};

let cosmosDbStorageSettings: CosmosDbStorageSettings;
if (botSettings.cosmosDb === undefined) {
    throw new Error();
}
cosmosDbStorageSettings = {
    authKey: botSettings.cosmosDb.authKey,
    collectionId: botSettings.cosmosDb.collectionId,
    databaseId: botSettings.cosmosDb.databaseId,
    serviceEndpoint: botSettings.cosmosDb.cosmosDBEndpoint
};

const storage: CosmosDbStorage = new CosmosDbStorage(cosmosDbStorageSettings);
const userState: UserState = new UserState(storage);
const conversationState: ConversationState = new ConversationState(storage);

const adapter: DefaultAdapter = new DefaultAdapter(
    botSettings,
    adapterSettings,
    telemetryClient,
    userState,
    conversationState
);

let bot: DialogBot<Dialog>;
try {
    const luisConfig: LuisApplication = { applicationId: appsettings.luis.appId, endpointKey: appsettings.luis.key, endpoint: appsettings.luis.endpoint };

    const welcomeDialog: WelcomeDialog = new WelcomeDialog();
    const servicesDialog: ServicesDialog = new ServicesDialog();
    const getContactInfoDialog: GetContactInfoDialog = new GetContactInfoDialog()
    const clientCheckDialog: ClientCheckDialog = new ClientCheckDialog(getContactInfoDialog)


    const mainDialog: MainDialog = new MainDialog(
        luisConfig, welcomeDialog, servicesDialog, clientCheckDialog
    );

    bot = new DialogBot(conversationState, telemetryClient, mainDialog);


} catch (err) {
    throw err;
}

// Create server
const server: restify.Server = restify.createServer();

// Enable the Application Insights middleware, which helps correlate all activity
// based on the incoming request.
server.use(restify.plugins.bodyParser());
// tslint:disable-next-line:no-unsafe-any
server.use(ApplicationInsightsWebserverMiddleware);

server.listen(process.env.port || process.env.PORT || '3979', (): void => {
    // tslint:disable-next-line:no-console
    console.log(`${server.name} listening to ${server.url}`);
    // tslint:disable-next-line:no-console
    console.log(`Get the Emulator: https://aka.ms/botframework-emulator`);
    // tslint:disable-next-line:no-console
    console.log(`To talk to your bot, open your '.bot' file in the Emulator`);
});

// Listen for incoming requests
server.post('/api/messages', async (req: restify.Request, res: restify.Response): Promise<void> => {
    // Route received a request to adapter for processing
    await adapter.processActivity(req, res, async (turnContext: TurnContext): Promise<void> => {
        // route to bot activity handler.
        await bot.run(turnContext);
    });
});

mainDialog.ts

import { InputHints, MessageFactory, StatePropertyAccessor, TurnContext } from 'botbuilder';
import { LuisApplication, LuisRecognizer } from 'botbuilder-ai';

import {
    ComponentDialog,
    DialogSet,
    DialogState,
    DialogTurnResult,
    DialogTurnStatus,
    TextPrompt,
    WaterfallDialog,
    WaterfallStepContext,
    ChoicePrompt,
    ListStyle,
    ConfirmPrompt
} from 'botbuilder-dialogs';

import { WelcomeDialog } from '../dialogs/welcomeDialog'
import { ClientCheckDialog } from '../dialogs/clientCheckDialog'
import { ServicesDialog } from '../dialogs/servicesDialog'
import { Conversation } from './conversation'

import msg from '../resources/enMsg.json';
import { ClientInfo } from './clientInfo';

const CHOICE_PROMPT = 'choicePrompt';
const MAIN_WATERFALL_DIALOG = 'mainWaterfallDialog';
const TEXT_PROMPT = 'textPrompt';
const CONFIRM_PROMPT = 'confirmPrompt';

export class MainDialog extends ComponentDialog {
    private luisRecognizer: LuisRecognizer;
    private conversation: Conversation;
    private clientInfo: ClientInfo;


    public constructor(config: LuisApplication, welcomeDialog: WelcomeDialog, servicesDialog: ServicesDialog, clientCheckDialog: ClientCheckDialog) {
        super('MainDialog');


        const luisIsConfigured = config && config.applicationId && config.endpoint && config.endpointKey;
        if (luisIsConfigured) {
            this.luisRecognizer = new LuisRecognizer(config, {}, true);
        }
        else {
            throw new Error('[MainDialog]: Missing parameter \'luisRecognizer\' is required');
        }

        this.conversation = new Conversation()
        this.clientInfo = new ClientInfo()

        const choicePrompt = new ChoicePrompt(CHOICE_PROMPT);
        choicePrompt.style = ListStyle.suggestedAction;

        this.addDialog(new TextPrompt(TEXT_PROMPT))
            .addDialog(new ConfirmPrompt(CONFIRM_PROMPT))
            .addDialog(choicePrompt)
            .addDialog(welcomeDialog)
            .addDialog(servicesDialog)
            .addDialog(clientCheckDialog)
            .addDialog(new WaterfallDialog(MAIN_WATERFALL_DIALOG, [
                this.introStep1.bind(this),
                this.introStep2.bind(this),
                this.getIntentStep.bind(this),
                this.followUpStep.bind(this),
                this.checkForContactInfo.bind(this),
                this.checkIfHelpfulStep.bind(this),
                this.finalStep.bind(this)
            ]));

        this.initialDialogId = MAIN_WATERFALL_DIALOG;
    }

    public async run(context: TurnContext, accessor: StatePropertyAccessor<DialogState>) {
        const dialogSet = new DialogSet(accessor);
        dialogSet.add(this);

        const dialogContext = await dialogSet.createContext(context);
        const results = await dialogContext.continueDialog();
        if (results.status === DialogTurnStatus.empty) {
            await dialogContext.beginDialog(this.id);
        }
    }

    private async introStep1(stepContext: WaterfallStepContext): Promise<DialogTurnResult> {

        if (!this.luisRecognizer) {
            const luisConfigMsg = 'NOTE: LUIS is not configured. To enable all capabilities, add `LuisAppId`, `LuisAPIKey` and `LuisAPIHostName` to the .env file.';
            await stepContext.context.sendActivity(luisConfigMsg);
            return await stepContext.next();
        }


        const messageText = (stepContext.options as any).restartMsg ? (stepContext.options as any).restartMsg : msg.welcome;
        this.conversation.addSpeech(Conversation.Speaker.Bot, messageText)

        return await stepContext.beginDialog('welcomeDialog', { messageText: messageText })
    }

    private async introStep2(stepContext: WaterfallStepContext): Promise<DialogTurnResult> {


        var messageText = msg.clickOrType
        const promptMessage = MessageFactory.text(messageText, messageText, InputHints.ExpectingInput);
        return await stepContext.prompt(TEXT_PROMPT, { prompt: promptMessage });
    }


    private async getIntentStep(stepContext: WaterfallStepContext): Promise<DialogTurnResult> {

        this.conversation.addSpeech(Conversation.Speaker.Client, stepContext.result)

        this.clientInfo.question = stepContext.result

        if (this.luisRecognizer) {
            const luisResult = await this.luisRecognizer.recognize(stepContext.context);
            switch (LuisRecognizer.topIntent(luisResult)) {
                case 'Services':
                    this.clientInfo.intent = ClientInfo.Intent.Services
                    break

                default:
                    this.clientInfo.intent = ClientInfo.Intent.Other
                    // Catch all for unhandled intents
                    return await stepContext.replaceDialog(this.initialDialogId, { restartMsg: msg.didNotUnderstandIntent });

            }

            if (this.clientInfo.intent === ClientInfo.Intent.Services) {
                return await stepContext.beginDialog('servicesDialog', { clientInfo: this.clientInfo, repeat: false })
            }

        }
        return await stepContext.next();
    }

    private async followUpStep(stepContext: WaterfallStepContext): Promise<DialogTurnResult> {
        if (stepContext.result) {
            var getIntentResult = stepContext.result as { clientInfo: ClientInfo | undefined; conversation: Conversation };
            if (getIntentResult.clientInfo)
                this.clientInfo = getIntentResult.clientInfo

            this.conversation.addSubConversation(getIntentResult.conversation)
            if (getIntentResult.clientInfo) {
                if (getIntentResult.clientInfo.intent === ClientInfo.Intent.Services) {
                    return await stepContext.beginDialog('checkClientDialog', this.clientInfo)
                }
            }
        }

        return await stepContext.next();
    }

    private async checkForContactInfo(stepContext: WaterfallStepContext): Promise<DialogTurnResult> {

        if (stepContext.result) {
            var followUpResult = stepContext.result as { clientInfo: ClientInfo | undefined; conversation: Conversation };
            this.conversation.addSubConversation(followUpResult.conversation)

        }

        return await stepContext.next();
    }

    //ask user if bot was able to help them
    private async checkIfHelpfulStep(stepContext: WaterfallStepContext): Promise<DialogTurnResult> {
        const messageText = msg.wasThisHelpful
        const message = MessageFactory.text(messageText, messageText, InputHints.ExpectingInput);
        this.conversation.addSpeech(Conversation.Speaker.Bot, messageText)
        return await stepContext.prompt(CONFIRM_PROMPT, { prompt: message });
    }


    //restart
    private async finalStep(stepContext: WaterfallStepContext): Promise<DialogTurnResult> {
        this.clientInfo.wasHelpful = stepContext.result
        // Restart the main dialog waterfall with a different message the second time around
        return await stepContext.replaceDialog(this.initialDialogId, { restartMsg: msg.restartMain });
    }
}

clientCheck.ts

import {
    ComponentDialog,
    DialogTurnResult,
    WaterfallDialog,
    WaterfallStepContext,
    ChoiceFactory,
    ConfirmPrompt
} from 'botbuilder-dialogs';
import { ClientInfo } from './clientInfo';

import { InputHints, MessageFactory } from 'botbuilder';

import { GetContactInfoDialog } from '../dialogs/getContactInfoDialog'
import { Conversation } from './conversation'

import msg from '../resources/enMsg.json';
const CONFIRM_PROMPT = 'confirmPrompt'
const WATERFALL_DIALOG = 'waterfallDialog';

export class ClientCheckDialog extends ComponentDialog {

    private conversation: Conversation;

    // Constructor
    public constructor(getContactInfoDialog: GetContactInfoDialog) {
        super('ClientCheckDialog');

        this.conversation = new Conversation()

        this.addDialog(new ConfirmPrompt(CONFIRM_PROMPT))
            .addDialog(getContactInfoDialog)
            .addDialog(new WaterfallDialog(WATERFALL_DIALOG, [
                this.introStep.bind(this),
                this.generalInfoStep.bind(this),
                this.getContactInfoStep.bind(this),
                this.finalStep.bind(this)
            ]));

        this.initialDialogId = WATERFALL_DIALOG;
    }

    private async introStep(stepContext: WaterfallStepContext): Promise<DialogTurnResult> {
        const messageText = msg.workWithUs
        const message = MessageFactory.text(messageText, messageText, InputHints.ExpectingInput);
        this.conversation.addSpeech(Conversation.Speaker.Bot, messageText)

        return await stepContext.prompt(CONFIRM_PROMPT, { prompt: message });
    }

    private async generalInfoStep(stepContext: WaterfallStepContext): Promise<DialogTurnResult> {

        const clientInfo = stepContext.options as ClientInfo;
        this.conversation.addSpeech(Conversation.Speaker.Client, stepContext.result)
        clientInfo.isQualified = stepContext.result

        //start list of recources
        var bulletPoints = [msg.benefit1, msg.benefit2, msg.benefit3]

        //check for more cases to add info 
        const messageText1 = msg.general
        const message = ChoiceFactory.list(bulletPoints, messageText1, InputHints.IgnoringInput);

        //collecting bot output for conversation 
        var botOutput = messageText1
        for (var point in bulletPoints) {
            botOutput.concat(" -", point)
        }
        this.conversation.addSpeech(Conversation.Speaker.Bot, botOutput)
        await stepContext.context.sendActivity(message);

        if (clientInfo.isQualified) {
            const messageText2 = msg.becomeAClient
            const messageContact = MessageFactory.text(messageText2, messageText2, InputHints.ExpectingInput);
            this.conversation.addSpeech(Conversation.Speaker.Bot, messageText2)
            return await stepContext.prompt(CONFIRM_PROMPT, { prompt: messageContact });
        }
        else {
            return await stepContext.endDialog({ clientInfo: clientInfo, conversation: this.conversation });
        }
    }

    private async getContactInfoStep(stepContext: WaterfallStepContext): Promise<DialogTurnResult> {
        this.conversation.addSpeech(Conversation.Speaker.Client, stepContext.result)
        const clientInfo = stepContext.options as ClientInfo;
        if (stepContext.result) {
            return await stepContext.beginDialog("getContactInfoDialog")
        }
        return await stepContext.endDialog({ clientInfo: clientInfo, conversation: this.conversation });
    }

    private async finalStep(stepContext: WaterfallStepContext): Promise<DialogTurnResult> {
        const clientInfo = stepContext.options as ClientInfo;
        return await stepContext.endDialog({ clientInfo: clientInfo, conversation: this.conversation });
    }
}

getContactInfoDialog.ts :

import {
    ComponentDialog,
    DialogTurnResult,
    WaterfallDialog,
    WaterfallStepContext,
    TextPrompt
} from 'botbuilder-dialogs';
import { CardFactory, MessageFactory } from 'botbuilder';

const WATERFALL_DIALOG = 'waterfallDialog';
const TEXT_PROMPT = 'textPrompt';

import getContactInfoCard from '../cards/getContactInfoCard.json'

export class GetContactInfoDialog extends ComponentDialog {
    public constructor() {
        super('getContactInfoDialog')

        this.addDialog(new TextPrompt(TEXT_PROMPT))
        this.addDialog(new WaterfallDialog(WATERFALL_DIALOG, [
            this.firstStep.bind(this),
            this.secondStep.bind(this)
        ]))
        this.initialDialogId = WATERFALL_DIALOG;
    }

    public async firstStep(stepContext: WaterfallStepContext): Promise<DialogTurnResult> {
        const cardPrompt = MessageFactory.text('');
        cardPrompt.attachments = [CardFactory.adaptiveCard(getContactInfoCard)];
        return await stepContext.prompt(TEXT_PROMPT, cardPrompt);
    }

    public async secondStep(stepContext: WaterfallStepContext): Promise<DialogTurnResult> {
        //process adaptive card input here
        const messageText = 'What else can I do for you?'
        const messageContact = MessageFactory.text(messageText, messageText);
        return await stepContext.prompt(TEXT_PROMPT, { prompt: messageContact });
    }
}

Thank you in advance

mdrichardson mdrichardson · Accepted Answer · 2019-10-25T16:39:59

UPDATE

Now that I've reviewed your code, I've found the answer. First, a few points:

Virtual Assistant is still in Preview and the TypeScript version of it is a lower priority than C#. Expect bugs. Also, there's a better-than-zero chance it's going to change pretty drastically in the future. Personally, I'd use it for ideas, but develop by starting with something like CoreBot and then integrating in the stuff you actually need.
I have no idea why adding this.onDialog(this.activityToText.bind(this)); results in a double-welcome message. I think it has to do with point 1.
The reason this didn't work is my own fault. I included trim() because it mirrors the C# version of this answer, but it actually breaks it because it doesn't result in a falsy value. Sorry for the confusion and I'll edit my other answers.
You don't need turnContext.activity.text = JSON.stringify(turnContext.activity.value); at the end.

More info about Point 4

I'm going to expand on this because it's an important aspect of Javascript to understand.

When you have:

const activity = turnContext.activity;
if (!activity.text && activity.value) {
    activity.text = JSON.stringify(activity.value);
}
turnContext.activity.text = JSON.stringify(turnContext.activity.value);

...turnContext.activity.text = JSON.stringify(turnContext.activity.value); is redundant because setting

const activity = turnContext.activity

is NOT saying:

const activity = copyOf(turnContext.activity)

Instead, it's saying:

const activity = memoryLocationOf(turnContext.activity)

So, when you call activity.text = JSON.stringify(activity.value);, it's changing the memory location of activity.text (which is turnContext.activity.text) to JSON.stringify(activity.value). So, you're actually changing both activity.text and turnContext.activity.text at the same time because they refer to the same location in memory.

I bring this up not because it's super super relevant, but because if you don't learn this now, you'll likely run into some real head-scratchers in the future.

The Answer

With all that being said:

Delete this.onDialog(this.activityToText.bind(this));
Delete activityToText()
Make the ending of turn() look like this:

if (dc.activeDialog !== undefined) {
    const activity = turnContext.activity;
    if (!activity.text && activity.value) {
        activity.text = JSON.stringify(activity.value);
    }
await dc.continueDialog();
} else {
    await dc.beginDialog(this.rootDialogId);
}

await next();

and then you will get:

I'm only not going to flag this as a duplicate because it's in TypeScript and it's been awhile since I've answered this for TypeScript. That being said, I've answered this before and edited the original answer to keep it up-to-date (last link in this answer).

That being said, the principle is the same. See this answer.

I highly highly recommend reading the blog post linked at the bottom of that answer.

A slightly shorter version can be found here.

Specific to your code:

You can't simply use await stepContext.context.sendActivity(<card>) and then return await stepContext.next(undefined);. There's nothing telling the bot to wait for a reply. You either need to send a blank text prompt after the card (like in my example in the last link) or attach the card to a text prompt and send them both at the same time. Something like:

async displayCard(step) {
    const cardPrompt = MessageFactory.text('');
    cardPrompt.attachments = [yourAdaptiveCard];
    return await step.prompt('textPrompt', cardPrompt);
}

Get User Input from Adaptive Card using Waterfall Dialog in Botframework v4

1 Answers

UPDATE

The Answer