Removing Sparsity in matrix

0
votes

What does this sparsity parameter in removeSparseTerms do?

This works.

TrigramTokenizer <- function(x) NGramTokenizer(x, Weka_control(min = 3, max = 3))
BigramTokenizer <- function(x) NGramTokenizer(x, Weka_control(min = 2, max = 2))
tdm <- DocumentTermMatrix(corpus, control = list(tokenize = TrigramTokenizer))
x <- as.matrix(removeSparseTerms(tdm, 0.99991))

However, when i change to 

x <- as.matrix(removeSparseTerms(tdm, 0.10))

This does not work. The matrix returns a sparse matrix. Am i not understanding it correctly? The 0.10 parameter determines the term must occur in at least 10% of the documents in the corpus?

Is this due to the sparsity of my matrix?

Running corpus returns this result 

corpus
<<DocumentTermMatrix (documents: 42695, terms: 326740)>>
Non-/sparse entries: 393990/13949770310
Sparsity           : 100%
Maximal term length: 97
Weighting          : term frequency (tf)
1
rmatrixsparse-matrix

1 Answers

3
votes

The sparsity parameter helps you to removes those terms which have at least a certain percentage of sparse elements. (very) Roughly speaking if you want to keep the terms that appear 3% of of the time, set the parameter to 0.97. If you want the terms that occur in 70% of the time, set the parameter to 0.3. The values must be bigger than 0 and smaller than 1.

In your case, if you want the term to appear in at least 10% of the time, you need to set the sparsity to 0.9.

see code example.

  libary(tm)

  data("crude")
  crude <- as.VCorpus(crude)
  crude <- tm_map(crude, stripWhitespace)
  crude <- tm_map(crude, removePunctuation)
  crude <- tm_map(crude, content_transformer(tolower))
  crude <- tm_map(crude, removeWords, stopwords("english"))
  crude <- tm_map(crude, stemDocument)
  dtm <- DocumentTermMatrix(crude)
  sdtm <- removeSparseTerms(dtm, 0.3)
  sdtm2 <- removeSparseTerms(dtm, 0.7)

  sdtm$ncol
  inspect(sdtm) # 4 words returned 
  sdtm2$ncol      
  inspect(sdtm2) # 24 words returned