Missing delimiter error when importing html text

Question

Playing with Azure Machine Learning using the Designer and am getting a "Delimiter not found" error when importing my data.

I originally started with a few hundred html files stored as azure blobs. Each file would be considered a single row of text, however, I had no luck importing these files for further text analytics.

I created a Data Factory job that imported each file, stripped all the tabs, quotes, cr/lf from the text, added a column for the file name and stored it all as a combined tab-delimited file. In notepad++ I can confirm that the format is FileName tab HtmlText. This is the file I'm trying to import into ML and getting the missing delimiter message as I'm trying to define the import module.

Here is the error when I try and create a dataset:

{
  "message": "'Delimiter' is not specified or invalid."
}

Question 1: Is there a better way to do text analytics on a large collection of html files?

Question 2: Is there a format I need to use in my combined .tsv file that works?

Question 3: Is there maybe a max length to the string column? My html can be 10's of thousands of characters long.

can you share a mockup of what your tsv looks like? also, what is left if you've already stripped all the tabs, quotes, and line breaks? it sounds like you have large file of text where each line is the text from one webpage? — Anders Swanson
Sample of the tsv: /dir/file1.htm \t <div>Text<p>Lots of text!</p></div>\r\n etc. I'm thinking it's the length of the text field that is giving me problems. When I create a tsv file by hand with a few hundred characters, the import module is working. — Wavel

Anders Swanson Anders Swanson · Accepted Answer · 2020-08-25T23:37:20

you're right that it might be line length, but my guess is that there are still some special characters (i.e. anything starting with \ that aren't properly escaped or removed. How did you scrape and strip the text data? Have you tried using beautifulsoup?

Missing delimiter error when importing html text

1 Answers