Running a test suite with over a million test cases

55
votes

We have a distributed test environment with the robotremoteserver starting a bunch of other applications and working with them as part of the test. The test that I am trying to run requires me to run over a million test cases in a single suite. The issue here is, when the pybot starts running, it blocks up 8 GB of RAM and results in slow performance. After a while it freezes up.

To overcome this issue I was planning to create separate suites with less than 100 cases in each. But in this case I am not able to use the keywords from the other remoteservers initiated from other suites. The only way I can use it is by disconnecting from the remoteserver and reconnecting it in each suite - which would defeat the purpose of the test.

I am not sure if anybody has come across this scenario. I would appreciate if anybody can think of a solution to this issue.

Additional information

Another thing that helps solve this: Is it is possible to call a keyword from a library (with state) that is initiated by another suite?

Can I use get library instance when using RIDE? I am importing the library in one suite, then try to get the library instance in other suite. Is this possible?

I keep getting the following error:

AttributeError: Remote instance has no attribute 'replace'

In Suite 1 this is what I am doing:

Import Library    Remote    ${verifix_xmlrpc_url}    WITH NAME    Verifix
${lib}=    BuiltIn.Get Library Instance    Verifix
Set Global Variable    ${lib}

In Suite 2:

${lib}.remove messages    ${VenueNSDQ}
2
pythonautomated-testsrobotframework
Hi Amol, have you had any success thus far? To be honest it sounds pretty stupid to have that many test cases linked under a test suite, I can't imagine it's possible to understand the suite as a unit. I think I would advise splitting the test suite up as you have been doing in logical units so you avoid issues. Have you tried doing the same with tags? One additional suggestion would be to use pabot once you've split out some test suites. Just be careful if you need some shared resource for any test runs – shicky
@shicky no reason to call anyone stupid... – Monkpit
@Monkey I didn't call him stupid, in fact I went to great lengths to try and help Amol. I said it was stupid to have a million test cases under one test suite, it's completely unmanageable and asking for trouble, hence the problematic situation Amol finds himself in. I doubt Amol is solely responsible for putting a million test cases in one suite, as it's a problem Amol is trying to solve, most likely he inherited it. Why don't you try and help also instead of trying to create conflict where none exists? Given the upvotes my comment had, clearly other people agree with the sentiment. – shicky
@shicky: it may not be so stupid to have a million test cases, if they are auto-generated. Perhaps they have a large matrix of products which need to be tested against a large matrix of inputs and expected outputs for which they automatically generate the test cases. Permutations can easily reach into the millions. Without knowing how these test cases were created and managed, it's a little bit insensitive to call it "stupid". – Bryan Oakley
Fair enough @BryanOakley, I bow to your superior knowledge and apologise to Amol and anyone else offended by my words. As previously stated it was not my intent to offend as I was referring to the test case situation and in no way was it directed at Amol. I still believe having anything close to one million test cases in a suite is a very very bad idea, as Amol highlights with his question. I offered a bounty on the question, did my own research and offered a few steps I would take, not much more I can do than that. – shicky

2 Answers

3
votes

I don't know any specifics about Python, but you might want to try spinning up each test case as a separate process. This will allow you to recover the memory from each process as it completes, and may allow you to run your million test cases safely on one machine.

Generally, when a program needs more memory from the OS, it takes it, but it cannot give it back until it exits. This is likely why your process falls over.

Quick processes that you can spin up and then kill, will alleviate this memory problem.... and then it is a pretty short step from there to running each of these test cases through rsh on a different machine entirely.

I hope that helps a little.

0
votes

OK. First, I do not program in Python, and I also have no idea what kind of computers you are using or how much memory the computer has or anything else about your situation with your computer(s). So this is just general information gleaned from writing system programs and working across multiple computers.

With that said: If you are going to run huge datasets or a huge number of programs you will have to partition up what you want to do so it will fit into the system(s) that you are using. That is the first part.

The second part is that if you want to talk across programs to other programs or systems then you need to set up some type of common memory usage. I know that PHP has such a thing built in to it so I have to assume that Python also has such a thing. You set up the common memory area and then send information back and forth between the various programs. Usually using some form of tokens to identify each program and/or system. In this way, you can set things up so once one suite of tests completes, it can tell the next set to begin and thus not take over or bog down your system.

If you are really good - you could also create some kind of controller program that starts up a test, watches it until the test ends/exits, and then start the next test. (So instead of a daisy chain way of doing this you have a master/slave way of doing this just like a client/server relationship.)

If you are using a Windows box/OS I would suggest looking at AutoIt. AutoIt can easily handle watching a program run and starting up a new task once an old task has finished. It also allows you full control over both your system as well as the remote system and was developed to help automate tasks such as the one you are trying to do. I found (by experimenting) that AutoIt can also be used with Unix/Linux boxes such as Macs and others. Although you only then have the commands you send over rather than access to the Windows features.

If you are more proficient with Python that you can do the above using it - then more power to you. Even though it has been two years since you asked this - I hope this helps you out in your endeavors. :-)