How to convert corpus to data.frame with meta data in R

Question

how can I convert a corpus into a data frame in R which contains also meta data? I already tried the suggestion from convert corpus into data.frame in R, but the resulting data frame only contains the text lines from all docs in the corpus. I need also the document ID and maybe the line number of the text line in two columns. So, how can I extend this command: dataframe <- data.frame(text=unlist(sapply(mycorpus,[, "content")), stringsAsFactors=FALSE) to get the data?

I already tried

    dataframe <- 
data.frame(id=sapply(corpus, meta(corpus, "id")), 
text=unlist(sapply(corpus, `[`, "content")), 
stringsAsFactors=F)

but it didn't help; I only got an error message "Error in match.fun(FUN) : 'meta(corpus, "id")' ist nicht Funktion, Zeichen oder Symbol"

The corpus is extracted from plain text files; here is an example:

> str(corpus)
[...]
$ 1178531510 :List of 2
  ..$ content: chr [1:67] " uberrasch sagt [...] gemacht echt schad verursacht" ...
  ..$ meta   :List of 7
  .. ..$ author       : chr(0) 
  .. ..$ datetimestamp: POSIXlt[1:1], format: "2015-08-16 14:44:11"
  .. ..$ description  : chr(0) 
  .. ..$ heading      : chr(0) 
  .. ..$ id           : chr "1178531510" # <--- This is the ID i want in the data.frame
  .. ..$ language     : chr "de"
  .. ..$ origin       : chr(0) 
  .. ..- attr(*, "class")= chr "TextDocumentMeta"
  ..- attr(*, "class")= chr [1:2] "PlainTextDocument" "TextDocument"
[...]

Many thanks in advance :)

sapply(corpus, meta(corpus, "id")) should be sapply(corpus, meta, "id") — scoa
Thanks, this seems to work little better, but now I got this error: Error in data.frame(id = sapply(corpus, meta, "id"), text = unlist(sapply(corpus, : Argumente implizieren unterschiedliche Anzahl Zeilen: 323, 10012 — Azrael
well, your code and my correction work for the example dataset data(acq) ; corpus <- acq, so the problem probably is in your data : you get 323 id, but 10012 text content (I would guess, by my german is rusty)... What is the length of your corpus? Could you post a sample part of the corpus that reproduces the problem? — scoa
The corpus contains 323 documents, but 10012 lines of text. In the dataframe, every line of text is a row. — Azrael
you could try this : unlist(lapply(sapply(corpus, [, "content"),paste,collapse="\n")) — scoa

scoa scoa · Accepted Answer · 2015-08-18T10:51:07

There are two problems : you should not repeat the argument corpus in sapply, and multi-paragraphs texts are turned to character vectors of length > 1 which you should paste together before unlisting.

dataframe <- 
    data.frame(id=sapply(corpus, meta, "id"),
               text=unlist(lapply(sapply(corpus, '[', "content"),paste,collapse="\n")),
               stringsAsFactors=FALSE)

How to convert corpus to data.frame with meta data in R

1 Answers