Use gnuplot via python (python-gnuplot) to plot all files at once

Question

The requirement is to use a python script to plot two columns from all specific .csv files in a directory at once. This works great:

plot_str = 'plot for [i=0:*] file=sprintf("cpu%03d.csv",i) title_s=sprintf("cpu%03d",i) file using 16:7 pt 7 ps 1 title title_s'
g_plot.__call__(plot_str)

Where g_plot is:

g_plot = Gnuplot.Gnuplot(persist=1)

However, additionally, sometimes a cpu mask has been involved in previous steps, resulting in non-sequential file names. For example:

$ ls -1 cpu???.csv
cpu001.csv
cpu003.csv
cpu005.csv
cpu007.csv

In which case the above command aborts upon the first missing file in the sequence.
I have tried several methods, most obtained from here in stackoverflow. My second to latest attempt (including debug flags and such):

#    the following commands are the current experiment.
    file_list = subprocess.check_output('ls -1 cpu???.csv',shell=True)
    title_list = subprocess.check_output('ls -1 cpu???.csv | sed -e \'s/.csv//\'',shell=True)

    plot_str = "plot for [i=1:words(file_list)] 'word(file_list,i)' using 16:7 pt 7 ps 1 title word(title_list,i)"

    print('flag 1')
    print(file_list)
    print('flag 2')
    print(title_list)
    print('flag 3')
    print(plot_str)

    g_plot('file_list = {0}'.format(file_list))
    g_plot('title_list = {0}'.format(title_list))
#    g_plot.__call__('file_list = "{0}"'.format(file_list))
#    g_plot('print file_list')
#    g_plot.__call__('title_list = "{0}"'.format(title_list))
#    g_plot('print title_list')

    g_plot.__call__(plot_str)

Resulting in:

flag 1
cpu001.csv
cpu003.csv
cpu005.csv
cpu007.csv

flag 2
cpu001
cpu003
cpu005
cpu007

flag 3
plot for [i=1:words(file_list)] 'word(file_list,i)' using 16:7 pt 7 ps 1 title word(title_list,i)
         line 0: undefined variable: cpu001
gnuplot> cpu003.csv
         ^
         line 0: invalid command
gnuplot> cpu005.csv
         ^
         line 0: invalid command
gnuplot> cpu007.csv
         ^
         line 0: invalid command
         line 0: undefined variable: cpu001
gnuplot> cpu003
         ^
         line 0: invalid command
gnuplot> cpu005
         ^
         line 0: invalid command
gnuplot> cpu007
         ^
         line 0: invalid command
         line 0: undefined variable: file_list

My latest attempt:

#    the following commands are the current experiment.
    file_list = subprocess.check_output('ls -1 cpu???.csv',shell=True).splitlines()
    title_list = subprocess.check_output('ls -1 cpu???.csv | sed -e \'s/.csv//\'',shell=True).splitlines()
#    file_list = file_list.splitlines()

    plot_str = "plot for [i=1:words(file_list)] 'word(file_list,i)' using 16:7 pt 7 ps 1 title word(title_list,i)"
#    plot_str = 'plot for [i=1:words(file_list)] file=word(file_list,i) title_s=word(title_list,i) file using 16:7 pt 7 ps 1 title title_s'

    print('flag 1')
    print(file_list)
    print('flag 2')
    print(title_list)
    print('flag 3')
    print(plot_str)

    g_plot('file_list = {}'.format(file_list))
    g_plot('title_list = {}'.format(title_list))
#    g_plot.__call__('file_list = "{0}"'.format(file_list))
#    g_plot('print file_list')
#    g_plot.__call__('title_list = "{0}"'.format(title_list))
#    g_plot('print title_list')

    g_plot.__call__(plot_str)

Resulting in:

flag 1
['cpu001.csv', 'cpu003.csv', 'cpu005.csv', 'cpu007.csv']
flag 2
['cpu001', 'cpu003', 'cpu005', 'cpu007']
flag 3
plot for [i=1:words(file_list)] 'word(file_list,i)' using 16:7 pt 7 ps 1 title word(title_list,i)
gnuplot> file_list = ['cpu001.csv', 'cpu003.csv', 'cpu005.csv', 'cpu007.csv']
                     ^
         line 0: invalid expression
gnuplot> title_list = ['cpu001', 'cpu003', 'cpu005', 'cpu007']
                      ^
         line 0: invalid expression
         line 0: undefined variable: file_list

EDIT: Attempt to use the answer from Christoph resulted in:

gnuplot> cpu003.csv
         ^
         line 0: invalid command
gnuplot> cpu005.csv
         ^
         line 0: invalid command
gnuplot> cpu007.csv
         ^
         line 0: invalid command
gnuplot> ""
         ^
         line 0: invalid command
gnuplot> cpu003
         ^
         line 0: invalid command
gnuplot> cpu005
         ^
         line 0: invalid command
gnuplot> cpu007
         ^
         line 0: invalid command
gnuplot> ""
         ^
         line 0: invalid command

Other notes:

• It is desirable that this work with python version 2.7 and up. i.e. I tried stuff using "run" from subprocess, but version 2.7 doesn't have it.
• The number of CPUs can be quite high, over 100. A small number is being used to test with, for now.
• While the same python script does know the CPU mask from earlier steps, the preference is not to rely on it here, because there may be other reasons the related .csv file is missing.

Question: How can I pass the file list and title list strings from python to gnuplot so that they can be plotted?
Even an array of integers, i.e. 1,3,5,7 in the above example, would probably be good enough, as the file names and titles could be re-constructed on the gnuplot side.

Answer 1

You must quote the string you assign to the gnuplot variables, but not have quotes around the word function when plotting

file_list = subprocess.check_output('ls -1 cpu???.csv',shell=True)
title_list = subprocess.check_output('ls -1 cpu???.csv | sed -e \'s/.csv//\'',shell=True)

plot_str = "plot for [i=1:words(file_list)] word(file_list,i) using 16:7 pt 7 ps 1 title word(title_list,i)"

g_plot('file_list = "{0}"'.format(file_list))
g_plot('title_list = "{0}"'.format(title_list))
g_plot.__call__(plot_str)

Answer 2

The problem with the first approach was that the newlines are still included and therefore approximately this was being passed to gnuplot:

file_list = cpu001.csv
cpu003.csv
cpu005.csv
cpu007.csv

And so gnuplot is complaining about the 3 subsequent lines that it thinks are commands. (Similarly for the second string).

The problem with the second approach is the rather bizarre, in my opinion, string format returned from the .splitlines() operation. And so gnuplot is complaining because it doesn't understand what is being passed to it.

Since the desire is to pass a nice clean string to gnuplot, then the challenge was to find a function that would do it. .replace('\n', ' ') seems to do the job. One string is enough, as we can rebuild the full file name so easily with the actual plot for loop command. (Note also the change from an integer iterator to a string iterator in the plot for loop):

    title_list = subprocess.check_output('ls -1 cpu???.csv | sed -e \'s/.csv//\'',shell=True).replace('\n', ' ')
    plot_str = "plot for [i in title_list] i.'.csv' using 16:7 pt 7 ps 1 title i"

#debug
    print('flag 2')
    print(title_list)
    print('flag 3')

    g_plot('title_list = "{}"'.format(title_list))
    g_plot(plot_str)