0
votes

I implemented the between-graph replication and asynchronous training according to the example at ‘https://www.tensorflow.org/deploy/distributed’.

Then, I set two servers and one worker as follows.

python dnn.py --ps_hosts=localhost:19000,localhost:18000 --worker_hosts=localhost:11000 --job_name=ps --task_index=0

python dnn.py --ps_hosts=localhost:19000,localhost:18000 --worker_hosts=localhost:11000 --job_name=ps --task_index=1

python dnn.py --ps_hosts=localhost:19000,localhost:18000 --worker_hosts=localhost:11000 --job_name=worker --task_index=0

I have three questions for distributed tensorflow.

Firstly, according to the tensorflow timeline of my program as shown below, all the computing and variable updating operations are executed at ps nodes, while the worker node is idle. This is confusing for me, since I thought the computing operation should be executed at worker nodes instead of ps nodes. Would someone help me on this?

distributed tensorflow timeline

Secondly, only CPU is assigned to servers by using tf.train.replica_device_setter in my program. However, the operations execute both at CPU and GPU. What is the correct way to assign CPU/GPU to servers?

Last but not the least, if I start two servers and three workers, will the two servers save the same copy of parameters? Also, I wonder whether the three workers will update gradients of the same graph. Would anyone tell me?

p.s. I assigned devices to both servers and workers by using tf.train.replica_device_setter. However, in the example (https://www.tensorflow.org/deploy/distributed), there’s no device assignment to the local server. In my case, if I didn't assign devices to local server, there will be errors like:

“Operation was explicitly assigned to /job:ps/task:0 but available devices are [ /job:localhost/replica:0/task:0/device:CPU:0, /job:localhost/replica:0/task:0/device:GPU:0, /job:localhost/replica:0/task:0/device:GPU:1 ...]. Make sure the device specification refers to a valid device.”

My code:

def train():
tl = TimeLiner()
#get current servers
ps_hosts = FLAGS.ps_hosts.split(",")
#get current workers
worker_hosts = FLAGS.worker_hosts.split(",")
cluster = tf.train.ClusterSpec({"ps": ps_hosts,
                                "worker": worker_hosts})
graph_options = tf.GraphOptions(enable_bfloat16_sendrecv=True)
gpu_options = tf.GPUOptions(per_process_gpu_memory_fraction=0.3, allow_growth=True)
config = tf.ConfigProto(graph_options=graph_options, gpu_options=gpu_options, log_device_placement=False,
                        allow_soft_placement=False)
#start a server
server = tf.train.Server(cluster,
                         job_name=FLAGS.job_name,
                         task_index=FLAGS.task_index,
                         config=config)
if FLAGS.job_name == "ps":
    server.join()
elif FLAGS.job_name == "worker":
    with tf.variable_scope(tf.get_variable_scope()):
        with tf.device(tf.train.replica_device_setter(ps_device="/job:localhost/replica:0/task:%d/device:CPU:0" % FLAGS.task_index,
                                                      worker_device="/job:localhost/replica:0/task:%d/device:GPU:0" % FLAGS.task_index,
                                                      cluster=cluster)):
            loss = ...
            global_step = tf.train.get_or_create_global_step()
            train_op = tf.train.AdagradOptimizer(0.01).minimize(loss, global_step=global_step)
            sys.stdout.flush()
            init_op = tf.group(tf.global_variables_initializer(), tf.local_variables_initializer())
            summary_op = tf.summary.merge_all()
        hooks = [tf.train.StopAtStepHook(last_step=FLAGS.max_steps)]
        total_training = 0
        graph_options = tf.GraphOptions(enable_bfloat16_sendrecv=True)
        gpu_options = tf.GPUOptions(per_process_gpu_memory_fraction=0.9, allow_growth=True)
        config = tf.ConfigProto(graph_options=graph_options, gpu_options=gpu_options, log_device_placement=False,
                                allow_soft_placement=True)
        with tf.train.MonitoredTrainingSession(master=server.target,
                                               is_chief=(FLAGS.task_index == 0),
                                               checkpoint_dir=FLAGS.log_dir,
                                               log_step_count_steps=100000,
                                               hooks=hooks,
                                               config=config) as mon_sess:
            mon_sess.run(init_op)
            options = tf.RunOptions(trace_level=tf.RunOptions.FULL_TRACE)
            run_metadata = tf.RunMetadata()

            while not mon_sess.should_stop():
                # run a training step asynchronously
                [_, tot_loss, step, summary] = mon_sess.run([train_op, loss, global_step, summary_op],
                                                            options=options,
                                                            run_metadata=run_metadata)

                fetched_timeline = timeline.Timeline(run_metadata.step_stats)
                chrome_trace = fetched_timeline.generate_chrome_trace_format()
                tl.update_timeline(chrome_trace)
        tl.save('timeline.json')

Thanks in advance!

Ying

2

2 Answers

0
votes

You've probably heard of setting device with tf.Device(cpu:0) or something like that you set right before starting session. Tried that?

0
votes

It probably is because you put your worker's flag index in ps task's index in your replica_device_setter?