"input string 1 is invalid UTF-8" in removeWords function

Question

I have looked through all of the questions on Stack Overflow regarding why removeWords isn't working and for me, it is none of the reasons already posted.

Here's what I have:

setwd("not shown")
filenames<-list.files(getwd(),pattern="*.txt")
files<-lapply(filenames,readLines)
docs<-Corpus(VectorSource(files))
writeLines(as.character(docs[[1]]))
docs <- tm_map(docs, removePunctuation)
docs <- tm_map(docs, removeNumbers)
docs<-tm_map(docs, content_transformer(tolower))
docs <- tm_map(docs, stripWhitespace)

docs <- tm_map(docs, removeWords, c(stopwords("english"))

Everything works and runs fine until the last line for the removeWords. Here's the error I get:

Error in gsub(sprintf("(*UCP)\b(%s)\b", paste(words, collapse = "|")), : input string 1 is invalid UTF-8

Any thoughts on why I am receiving this error?

Could you paste a minimal input text file that causes this error? Have you checked that you input file contains no non-UTF-8 characters? — Tomasz Maczyński
Welcome to StackOverflow:) Have you check if your new lines are consistent within your input file (in terms of CR and LF characters)? Do you have an empty line at the end of file or something unusual? — Tomasz Maczyński

Artem Moskalev Artem Moskalev · Accepted Answer · 2017-06-15T19:30:00

0

votes

Check that your files are saved as UTF-8, often they are ANSI or Unicode by default

"input string 1 is invalid UTF-8" in removeWords function

1 Answers