1
votes

I am trying to use GNU Parallel in order to execute a set of commands in parallel on a local cluster. I am using two client nodes which have 12 cores each but only 4 cores each are being used for computation. Here is what the test command looks like -

cut -d " " -f -2 $PE_HOSTFILE > hostfile
parallel --gnu --jobs 4 --sshloginfile hostfile --workdir $SGE_O_WORKDIR echo {} ::: A B C D E F G H

However, I get the error saying -

/bin/bash: node30: command not found
/bin/bash: node30: command not found
/bin/bash: node30: command not found
/bin/bash: node30: command not found
/bin/bash: node31: command not found
/bin/bash: node31: command not found
/bin/bash: node31: command not found
/bin/bash: node31: command not found

I have logged into both the nodes and checked if the command works and it does. I have installed a local copy of GNU Parallel and work with it. Can anyone please tell me what is wrong here?

Update: Here is the output of parallel -vv -

merlinc30.psi.ch 4 -- exec perl -e @GNU_Parallel\\\=split/_/,\\\"use_IPC::Open3\\\;_use_MIME::Base64\\\"\\\;eval\\\"@GNU_Parallel\\\"\\\;\\\$SIG\\\{CHLD\\\}\\\=\\\"IGNORE\\\"\\\;my\\\$zip\\\=\\\(grep\\\{-x\\\$_\\\}\\\"/usr/local/bin/bzip2\\\"\\\)\\\[0\\\]\\\|\\\|\\\"bzip2\\\"\\\;open3\\\(\\\$in,\\\$out,\\\"\\\>\\\&STDERR\\\",\\\$zip,\\\"-dc\\\"\\\)\\\;if\\\(my\\\$perlpid\\\=fork\\\)\\\{close\\\$in\\\;\\\$eval\\\=join\\\"\\\",\\\<\\\$out\\\>\\\;close\\\$out\\\;\\\}else\\\{close\\\$out\\\;print\\\$in\\\(decode_base64\\\(join\\\"\\\",@ARGV\\\)\\\)\\\;close\\\$in\\\;exit\\\;\\\}wait\\\;eval\\\$eval\\\; QlpoOTFBWSZTWWFW/uIAAWgfgHV/+//u538ev////kAB7OGgohU9CPSNkmmmjQyGhpkAAGgBo0BoHGTJoxDE0wEDAmmCMExNNNABhCKnmpk1PSeoANBkaAAxAAAAaADJAptAyp5NFPJ6am1Nqh6Q9EbTKaDTJ6jQB6il1up9ItZGm0TFUNBCc5EM2fqnR+LKMi/tlHRqKjYrRJ/pi5U/oFKnhTUbQ+0YKzYIYUIGgWKpYpCroUeacSMtx3ggS5EgjSPsadVOgJbpDEpJdHA8dKl0AZdgkJ97qO0SiDMoQaJYacAQU8J1sCmUYHo1lg5qz+TYWCMRpohYrViTBuLrdgH+Jlll3bjsSrKb+aIpXztA5FjelrlQdbod2CWpSDBCqi5nmZd+rqHtDcdPPKNrVPzp5wvJo7BT0Kh0pCw7wPSB2jY3u4d/egE5HDe6YIv5eFCw94G0DwIVOFEfSoRHVwDYPBUxvqqShgGVCEQUtVYSPCuAWR9DuEqQx5Lkd4SN8sIegWFkO4oDH26etSVjWPeVjpDWlVxWdn9G6IgWbXiiztxmZScRa3IUugNGIsDOKUiTIp1h9tZmH+8hXETAaVAMLyGgRi/LbabiHEWD7LtN6s1kDy5zlSBWtdQWuZECeMTnrLS7iLwtC4MKSHSGBsrlyT0H7qNhd+EMixfXjfyLF5nhyp1L0pVmgVIE4yC45RNeJxjKFhtpCxKMw16vRB6ZtSsC80pMqA6YSyZoOBjPOeKQjooAFNoZbfEGH5DWJEVzvuAZjB+gesfEM+DD/xdyRThQkGFW/uI\\\=; /bin/bash: merlinc30.psi.ch: command not found merlinc29.psi.ch 4 -- exec perl -e @GNU_Parallel\\\=split/_/,\\\"use_IPC::Open3\\\;_use_MIME::Base64\\\"\\\;eval\\\"@GNU_Parallel\\\"\\\;\\\$SIG\\\{CHLD\\\}\\\=\\\"IGNORE\\\"\\\;my\\\$zip\\\=\\\(grep\\\{-x\\\$_\\\}\\\"/usr/local/bin/bzip2\\\"\\\)\\\[0\\\]\\\|\\\|\\\"bzip2\\\"\\\;open3\\\(\\\$in,\\\$out,\\\"\\\>\\\&STDERR\\\",\\\$zip,\\\"-dc\\\"\\\)\\\;if\\\(my\\\$perlpid\\\=fork\\\)\\\{close\\\$in\\\;\\\$eval\\\=join\\\"\\\",\\\<\\\$out\\\>\\\;close\\\$out\\\;\\\}else\\\{close\\\$out\\\;print\\\$in\\\(decode_base64\\\(join\\\"\\\",@ARGV\\\)\\\)\\\;close\\\$in\\\;exit\\\;\\\}wait\\\;eval\\\$eval\\\; QlpoOTFBWSZTWfIkiOsAAWgfgHV/+//+538ev////kAB7O6dBohU9EMjyJoDQaAAeoAANAPSaGj1AhCNTTR6TTRoD0RoaNAAAAAAAIqeamTUyepoBoGmgABtQAAADQASSCVP2ppplT2mink01Pap5Q9EYh6magaMnqNAHqJ7x1+z+0lJxUyxTbkMMEU0aXl53xorY3XYrCtFka9V8/4stZf6BSscKKTK0UDBVdBDKRAz6xUrFIVdCTtnEkx2IoH2KcJ9qL9abqHQEh4hiXmHhoKOwldAGXZJAwK6jtEohLjiDhKzlIBBMhtsB0Kdi5eRQPDYvyrVArrNNNVrtRrhNtjrJ+5SovdXUJrhWpYvUGQYNSVAI5gykScsY6XD4wXZKQKhBKh1Sqo75dRKJHFg8V49cXSzN5R+Pj6TByOq52tW4PMHtLOZvjv7Oxgvo37pqBn/LtpeSkG0O1Dr45H0wlZ0byNUhOhHz4WpgBmQQIFs00EuDKALAwC6K0MU5byTiRzniCAGhbDwKgpe6+hzWGw9yec5GKdoT1zf06aypQ2yKbfZiVDm4GeONNoqOPvKBDg2AnxumtW6xUH+8CfWUBpcEFJEQrrlkz51oRMi07n51c7LbUuLMsjVJ734R76GYb61rNYoqbgXD2UxBNOtJAjemfPfpX2crVN92U7y6zFLvvLlfq4m6H6Ww5ak5huE5oWQouL5iTKCz4B5eMps1ctUmy6na1JYFZkUH0CsdkHgvMNVlilfEwBOSK7m0G4CJuqDIYgeBDdv9nrWyrIe+C/4u5IpwoSHkSRHWA\\\=\\\=; /bin/bash: merlinc29.psi.ch: command not found merlinc30.psi.ch 4 -- exec perl -e @GNU_Parallel\\\=split/_/,\\\"use_IPC::Open3\\\;_use_MIME::Base64\\\"\\\;eval\\\"@GNU_Parallel\\\"\\\;\\\$SIG\\\{CHLD\\\}\\\=\\\"IGNORE\\\"\\\;my\\\$zip\\\=\\\(grep\\\{-x\\\$_\\\}\\\"/usr/local/bin/bzip2\\\"\\\)\\\[0\\\]\\\|\\\|\\\"bzip2\\\"\\\;open3\\\(\\\$in,\\\$out,\\\"\\\>\\\&STDERR\\\",\\\$zip,\\\"-dc\\\"\\\)\\\;if\\\(my\\\$perlpid\\\=fork\\\)\\\{close\\\$in\\\;\\\$eval\\\=join\\\"\\\",\\\<\\\$out\\\>\\\;close\\\$out\\\;\\\}else\\\{close\\\$out\\\;print\\\$in\\\(decode_base64\\\(join\\\"\\\",@ARGV\\\)\\\)\\\;close\\\$in\\\;exit\\\;\\\}wait\\\;eval\\\$eval\\\; QlpoOTFBWSZTWX9KUYEAAWgfgHV/+//u538ev////kAB7Lc0Gwyqn6n6pHtRtCTNCaZMTamACNMmADQmI9TQhE1PU00ek00aA9EaGmQAADQAAARU81Miek9TQDQNNAAD0QAAANABkmiU9qZNpTaaJk9TTagAZAGgGTQANHlwtr38sqqNJGNNszQuOKjYvngud+ym55s6PBxPN0+tXlc7NzGl/wKVHCmkytFIwV3QYZSIGgWKlYpCrsJO2cSTnYzwQIqxIGo/1pusdgJSiGJgQoVPHQldgDLskhPv9R2iUQmyRBwllyoAglY7vZApGWTbtFBTlnoXygeeFRjr6qwgTvxR4A1xK5MjJkG8RmcLsWmwC5qpwCUoGciTlknS2fKCyCkCuQSocNYM55cQ9wdbm2LA6HMffTdC3NbpE2+yGNKTkA4QOwUYk8M+XKgEpGfJFIEXszWVD3gc4GYhkoYR52hOOPOHQeCxruK0lloY6ISG4PSFVgywKx6nSWo5i97UVwqpbb46iUZ4+zGMXHfyMSgbw7SoxhzizAVOL9HTOIFvO8Waeu2ZGJoFTrhTBAa2kWhuhJhHcZFSclBkH9qFZxIBhYA0vIcBPO+7VUcBERFI9lqNbL1MDq3zdSBVVtBVkRAnrOb9BcYtArCoYg0pGcIaHWIljlsn3YaTF8IY1FdFt+morMsNVONcKWr0CxAmiMYDdElYlFsoWmqYKLAvDe2eCD0vbLNovM0SfIB5AqnYB0MC7L6pSvCoAUUis3uBqAib5hlrwOghq7+0vvreCaEP+LuSKcKEg/pSjAg\\\=;

The contents of the hostfile are -

merlinc29.psi.ch 4 merlinc30.psi.ch 4

2

2 Answers

1
votes

I think your hostfile should be:

4/merlinc29.psi.ch
4/merlinc30.psi.ch
1
votes

The content of hostfile is not what you think it is.

Maybe there are spaces or formatting characters in it?

Run parallel --vv for debugging.

The warning:

Warning: No xauth data; using fake authentication data for X11 forwarding.

is caused by your ssh defaulting to forward X display. You can override ssh's default with -x. So this should work:

parallel --ssh 'ssh -x' ...

As you are probably going to use that all the time you can put it in ~/.parallel/config:

--ssh "ssh -x"