0
votes

I want to parallel iterate over dask dataframe these code:

    from dask import dataframe
    dataset_df = dataframe.read_csv('input.csv')

    for x, row in dataset_df.iterrows():
       similarity = []
       for y, item in dataset_df.iterrows():
          similarity.append(similarity_text(row['text'], item['text']))
       result['similarity'] = similarity

Many thanks for your help,

1

1 Answers

0
votes

If you have access to a GPU you can try cudf: https://developer.nvidia.com/blog/pandas-dataframe-tutorial-beginners-guide-to-gpu-accelerated-dataframes-in-python/.

If not, you're going to run into trouble due to global interpreter lock. You could always use numpy and take advantage of its vectorised operations.