Is it possible to create a report of the most failing keywords used over all test cases in Robot Framework?

Question

Because of improving the automated test quality of our test cases in Robot Framework, we want to get a clear vision on the quality of our keywords that are used.

So, can somebody explain or help me with getting a report or a list with all failing keywords and if possible to create a "top 10 most failing keywords".

Currently I have written a Python script that can count all keywords over all test cases and test suites. However, when I want to filter them by status (count of status = PASS and status = FAIL), the totals aren't correct.

This has probably to do with the fact that the status of child-keywords in parent-keywords are counted as well...

for i in files_output:
    dom = ElementTree.parse(i)
    root = dom.getroot()
    print("Adding Keywords")
    for kw in root.iter('kw'):
        count+=1
        kw_element = kw.attrib["name"]
        keyword_list_total[kw_element] = 1

        if kw_element not in keyword_list_failed:
            for item in kw.iter('status'):
                if "FAIL" in item.attrib["status"]:
                    keyword_list_failed[kw_element] = 1

        elif kw_element in keyword_list_failed:
            for item in kw.iter('status'):
                keyword_list_total[kw_element] += 1
                if "FAIL" in item.attrib["status"]:
                    keyword_list_failed[kw_element] += 1

for example: a keyword in the last has a TOTAL: 30 but will give FAILED: 523. It clearly doesn't make sense.

Thank you already for helping me out!

Joonas Köppä Joonas Köppä · Accepted Answer · 2019-09-19T06:44:25

In this part of the code:

for kw in root.iter('kw'):
        count+=1
        kw_element = kw.attrib["name"]
        keyword_list_total[kw_element] = 1

You are setting the keyword_list_total[kw_element] to 1 for the iterator element on each XML node. Therefore every time the same keyword comes up in the XML, its total count is reset to 1, whereas you never reset the keyword_list_failed count.

Example:

You run the keyword My Keyword 10 times in the test case and it fails on every time. With your script, it would initialize total keyword_list_total['My Keyword'] = 1 10 times, but add the keyword_list_failed['My Keyword'] all the way up to 10. You would end up with Total: 2, Failed: 10.

I would suggest you add a check to the keyword_list_total initializer:

for kw in root.iter('kw'):
    count+=1
    kw_element = kw.attrib["name"]
    if keyword_list_total[kw_element] == None:
        keyword_list_total[kw_element] = 1

Is it possible to create a report of the most failing keywords used over all test cases in Robot Framework?

1 Answers