An R package for reading text files in all their various formats, by Ken Benoit, Adam Obeng, Paul Nulty, Aki Matsuo, Kohei Watanabe, and Stefan Müller.

Introduction

readtext is a one-function package that does exactly what it says on the tin: It reads files containing text, along with any associated document-level metadata, which we call “docvars”, for document variables. Plain text files do not have docvars, but other forms such as .csv, .tab, .xml, and .json files usually do.

readtext accepts filemasks, so that you can specify a pattern to load multiple texts, and these texts can even be of multiple types. readtext is smart enough to process them correctly, returning a data.frame with a primary field “text” containing a character vector of the texts, and additional columns of the data.frame as found in the document variables from the source files.

As encoding can also be a challenging issue for those reading in texts, we include functions for diagnosing encodings on a file-by-file basis, and allow you to specify vectorized input encodings to read in file types with individually set (and different) encodings. (All encoding functions are handled by the stringi package.)

How to Install

  1. From CRAN

  2. From GitHub, if you want the latest development version.

Linux note: There are a couple of dependencies that may not be available on linux systems. On Debian/Ubuntu try installing these packages by running these commands at the command line:

Demonstration: Reading one or more text files

readtext supports plain text files (.txt), data in some form of JavaScript Object Notation (.json), comma-or tab-separated values (.csv, .tab, .tsv), XML documents (.xml), as well as PDF and Microsoft Word formatted files (.pdf, .doc, .docx). readtext also handles multiple files and file types using for instance a “glob” expression, files from a URL or an archive file (.zip, .tar, .tar.gz, .tar.bz).

The file formats are determined automatically by the filename extensions. If a file has no extension or is unknown, readtext will assume that it is plain text. The following command, for instance, will load in all of the files from the subdirectory txt/UDHR/:

For files that contain multiple documents, such as comma-separated-value documents, you will need to specify the column name containing the texts, using the text_field argument:

For a more complete demonstration, see the package vignette.

Inter-operability with other packages

Text Interchange Format compatibility

readtext returns a data.frame that is formatted as per the corpus structure of the Text Interchange Format, it can easily be used by other packages that can accept a corpus in data.frame format.

If you only want a named character object, readtext also defines an as.character() method that inputs its data.frame and returns just the named character vector of texts, conforming to the TIF definition of the character version of a corpus.