Data driven using Robot framework - How to create PASS/FAIL in reports for each testcase read from file

Question

I am new to Robot, and trying to implement a data-driven test case, where I read data from a file. The data looks like this:

TC1,user1,password1
TC2,user2,password2
TC3,user3,password3
TC4,user4,password4

My test case and keywords are as below:

*** Test Cases ***
Login TestCase
    ${data}=    Read Data From File    ${testFile}
    Process Test Data    ${data}

*** Keywords ***
Process Test Data
    [Arguments]    ${data}
    : FOR    ${row}    IN    @{data}
    \    ${status}=    Run Keyword And Return Status    Perform Login ${row}

Perform Login
    [Arguments]    ${row}
    Login using    ${row[1]}   ${row[2]}

Now while running the 'Login Testcase', the report generated shows

'Total 1 Pass 1 Fail 0'. 

My question is how can I get the report the generate Pass/Fail for each row, ie

'Total N Pass X Fail Y' , where N=number of rows in data file and X+Y=N

I tried putting the For loop inside 'Login Testcase', but still getting the same result. Any ideas/help is highly appreciated!

Update: So, I tried modifying the tests (without reading the data file) as below:

*** Settings ***
Test Template     Perform Login

*** Test Cases ***
Login TestCase
    TC1    user1    password1
    TC2    user2    password2
    TC3    user3    password3
    TC4    user4    password4

*** Keywords ***
Perform Login
    [Arguments]    ${tc#}    ${username}    ${password}
    Login using    ${username}    ${password}

While doing so, Perform Login is run 4 times, but the report output still shows
'Total 1 Pass 1 Fail 0'.

I am not sure if I am using the test template the correct way, and also how to read data from file and use that for each test case iteration.

Update2:

*** Settings ***
Test Template     Perform Login

*** Test Cases ***
TC1    user1    password1
TC2    user2    password2
TC3    user3    password3
TC4    user4    password4

*** Keywords ***
Perform Login
    [Arguments]    ${username}    ${password}
    Login using    ${username}    ${password}

When I modify the testcases this way, I get

'Total 4 Pass 3 Fail 1', which is what I expect.

But in this case, I am not sure how to use the data read from file. Please help/share your ideas on how I could use data file to feed my testcases!!

Answer 1

Just follow documentation: http://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#data-driven-style You have to define Test Template: http://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#test-templates

Answer 2

I'll try to answer the questions you've had here in total, starting from the beginning.

The reason your test only showed one success is because you only had one test. Let me break it down for you.

Each .robot file that you run is actually a Test Suite. Each Suite can have multiple tests, defined under *** Test Cases ***. Each test case can have multiple Keywords Visually, that looks like this:

Suite
|-Test
|  |-Keyword
|  |-Keyword
|-Test
|-Test

You were running first one test with one keyword, then one test with four keywords. The test will only pass if all keywords pass. You can add other logic to make it pass if none, some, or all of the keywords pass, but that's the basic structure of a Robot Framework Test Suite. The output of the Robot Framework Test Suite will show exactly one Pass/Fail for each Test, not Keyword. In the Log it'll show a Pass/Fail for each Keyword individually, but not in the Console output.

As for reading from a file, that's a particularly complicated problem in Robot Framework, but an easier solution is to create a list and a dictionary in a Variable Table, as follows:

*** Variables ***
@{users} =  user1  user2  user3  user4
&{passwords} =  {'user1':'password1', 'user2':'password2', 'user3':'password3', 'user4':'password4'}

Then, in your loop, you can use :FOR ${user} IN @{users} and call the password with ${passwords["${user}"]} to only refer to the current user's password.

The only two ways I know of in base Robot Framework to directly read from a file are with a custom Python Keyword and the Operating System Library. It's not a full programming language, more like a specialized shell for Python.

Answer 3

I believe this question hit a scenario that Robot cannot currently solve. The scenario is:

1. You have a bunch of data points in a file, that you want to load dynamically.
2. You want to run identical tests on each of the data points.
3. You want Robot to report how many of these tests succeeded and failed.

There are ways to fulfill (1) and (2), by loading the data into a list and iterating through it in a for loop, as demonstrated in the question. (For completeness sake: this can also be done as a template with a for loop.) However, this for loop will always be lumped into one test case in the summary.

You can fulfill (2) and (3) with a data-driven Robot suite which lists the data points as test cases. (This method was mentioned in the question update). However this method cannot dynamically load data points during Robot execution.

The only solution I see, is to convert your data into a data-driven Robot suite with some script, and then run that suite. This would require some wrapper script to run Robot. This is a little disappointing, since my company has put its hopes into Robot being the test framework to supersede and envelop all other frameworks.