Google DataPrep - Apparently Limited Table Size

Question

I'm trying to prepare SEO data from Screaming Frog, Majestic and Ahrefs, join it before importing said data into BigQuery for analysis.

The Majestic and Ahrefs csv files import after some pruning down to the 100MB limit.

The Screaming Frog CSV file however doesn't fully load, only displaying approx 37,000 rows of 193,000. By further pruning less important cols in Excel and reducing the filesize(from 44MB to 39MB) , the number of rows loaded increases slightly. This would indicate to me that it's not an errant character or cell.

I've made sure(resaved via text editor) that the CSV file is saved in UTF8, checked the limitations of Dataprep to see if there is a limit on the number of cells per Flow/Wrangle and can find nothing.

The Majestic and AHREFS files are larger and load completely with no issue. There is no data corruption in the Screaming Frog file. Is there something common I'm missing?

Is the total limit for all files 100MB?

Any advice or insight would be appreciated.

Answer 1

To get the full transformation of your files, you should run the recipe.

What you see in the Dataprep Transformer Page is a head sample.

You can take a look about how the sampling works here.