Programming multiple devices parallelly using Vivado

Question

We have a setup where two Xilinx FPGA devices are individually connected to a computer. We use Vivado Tcl Console to program the FPGAs in our automation environment. I am able to successfully program the two devices one after the other. However, I am trying to reduce the programming time by launching two vivado instances in parallel and program each device simultaneously. While doing so, I get errors like below:

Targets(s) ", jsn-JTAG-SMT2-XXXXXXXXjsn-JTAG-HS3-XXXXXXXX" may be locked by another hw_server.

Error Message: ERROR: [Common 17-39] 'get_hw_targets' failed due to earlier errors.

Here is my python script that launches hw_server and cs_server on different ports and programs FPGAs parallely.

def program(hw_server_cmd, cs_server_cmd, cmd):
    cs_server_process = subprocess.Popen(cs_server_cmd, stdin=subprocess.PIPE, stdout=subprocess.PIPE, encoding='utf8')
    cs_server_process.wait()
    print("cs_server started")

    hw_server_process = subprocess.Popen(hw_server_cmd, stdin=subprocess.PIPE, stdout=subprocess.PIPE, encoding='utf8')
    hw_server_process.wait()
    print("hw_server started")
    p = subprocess.Popen(cmd, stdin=subprocess.PIPE, stdout=subprocess.PIPE, encoding='utf8')        
    p.wait()
    stdout, stderr = p.communicate()
    p.wait()
    hw_server_process.kill()
    cs_server_process.kill()
    print('Command [{}] return value: {}'.format(cmd, p.returncode))
    print('stdout for cmd [{}]: {}'.format(cmd, stdout))
    print('stderr for cmd [{}]: {}'.format(cmd, stderr))


if __name__ == '__main__':
    cs_server_cmd_1 = r"C:\Xilinx\Vivado\2020.1\bin\cs_server.bat -d -stcp::3041"
    hw_server_cmd_1 = r"C:\Xilinx\Vivado\2020.1\bin\hw_server.bat -d -stcp::3121"
    cmd_1 = r'C:\Xilinx\Vivado\2020.1\bin\vivado.bat -mode tcl -source ProgFPGA_1.tcl'

    cs_server_cmd_2 = r"C:\Xilinx\Vivado\2020.1\bin\cs_server.bat -d -stcp::3042"
    hw_server_cmd_2 = r"C:\Xilinx\Vivado\2020.1\bin\hw_server.bat -d -stcp::3122"
    cmd_2 = r'C:\Xilinx\Vivado\2020.1\bin\vivado.bat -mode tcl -source ProgFPGA_2.tcl'

    fpga_prog_threads = {}    
    fpga_prog_threads[0] = threading.Thread(name=f'Programming FPGA on Platform 1', target=program, args=[hw_server_cmd_1, cs_server_cmd_1,  cmd_1])
    fpga_prog_threads[1] = threading.Thread(name=f'Programming FPGA on Platform 2', target=program, args=[hw_server_cmd_2, cs_server_cmd_2, cmd_2])
    for t in fpga_prog_threads.values():
        t.start()
        #t.join()

    join_all_threads(list(fpga_prog_threads.values()), 3600)

Here is one of the tcl script I used. The other file also contains same set of commands but different port numbers and Serial number.

    open_hw_manager
    connect_hw_server -url 127.0.0.1:3121 -cs_url 127.0.0.1:3041 -allow_non_jtag
    current_hw_server 127.0.0.1:3121

    current_hw_target [get_hw_targets */xilinx_tcf/Digilent/XXXXXXXXXX]
    open_hw_target

    current_hw_device [lindex [get_hw_devices] 0]
    refresh_hw_device -update_hw_probes false [lindex [get_hw_devices] 0]
    
    set_property PROBES.FILE {} [lindex [get_hw_devices] 0]
    set_property FULL_PROBES.FILE {} [lindex [get_hw_devices] 0]
    set_property PROGRAM.FILE {C:/temp/bitfile2.bit} [lindex [get_hw_devices] 0]
    program_hw_devices [lindex [get_hw_devices] 0]
    refresh_hw_device [lindex [get_hw_devices] 0]    
    exit

Could someone point out what I am doing wrong here that is stopping me from programming the devices simultaneously?

P.S: Initially we used Xilinx ISE. I tried to program the devices simultaneously using impact.exe. In that case one FPGA or the other always returned error message like "Pin did not go HIGH".

Looks like there's things locking stuff out at the driver level, possibly during some sort of enumeration of the JTAG devices. — Donal Fellows
Just wanted to mention that I power off the devices and power them on again before programming them. — Kannan D
Perhaps there is a way to tell the system to only open one JTAG device instead of probing them all (as it looks like that is what is going wrong) but I really don't know how you might do that. If you can do that, you can get two processes going in parallel. — Donal Fellows

Cuauhtémoc Cuauhtémoc · Accepted Answer · 2020-09-18T15:19:24

I was wondering if you have 1 JTAG cable per device. If you do then you can start an instance of a hw_server per cable. Details on how to do this are described here:

https://forums.xilinx.com/t5/FPGA-Configuration/Multiple-hw-server-instances-on-same-PC/m-p/749154#M5849

In your case one hw_server instances could be launched something like:

hw_server -stcp::3122 -e "set jtag-port-filter Digilent/xxx"

where you replace the xxx value with the respective id from the cable.

Note that for windows I am not sure the quotes work. So in this case you may need to use an init file passed in to the hw_server.

Using this example, the init text file would look something like this:

# set JTAG port filter
set jtag-port-filter Digilent/xxx

assuming you call the file 'cable1.init' then you can launch the server as follows:

hw_server -stcp::3122 --init=cable1.init

The second instance would be similar to this this setup but you would specify a different port using the stcp option and cable configuration to point to the 2nd cable.

When programming, you will then use the port numbers to talk to the devices and now you should be able to program in parallel.

Programming multiple devices parallelly using Vivado

1 Answers