Suitable approach to test an API with 100 million records of multiple csvs having same column headers using Jmeter

Question

I am conducting rest API testing with JMeter using multiple CSV files having the same headers. The query parameters to request are 5 columns out of 30 columns of CSV file. Each CSV file has close to 1 million rows.

What would be the best approach to handle multiple CSV files with the same headers using CSV data set config and __CSVRead function to handle this scenario?

I am trying to come up an approach on below lines.

• Thread group = 1000(Concurrent, not simultaneous)
• Loop count = 10000
• For each file in a directory ( for loop iterating through directory)
  • CSV data set config as a child with below options: - Filename: ... (each file) - Variable Names: param1,param2,param3,param4,param5, - Delimiter: , - Recycle on EOF = False - Stop Thread on EOF = True - Sharing Mode = All threads
• while loop controller condition (param1 ne EOF)
  • HTTP request

Questions are: How can I pass dynamically a CSV file to CSV data set config using BeanShell pre-processor scripting? Is there any plugin that can be used to handle multiple files as a JMeter variable and pass that variable to csv data set config during run time?

Answer 1

You cannot pass dynamic file names to the CSV Data Set Config, you should consider going for __CSVRead() function instead.

Check out How to Pick Different CSV Files at JMeter Runtime article for more information on the implementation if needed.