Installation

v1.15.0 on CRAN 30 Jan 2024

Linux, Mac and Windows

install.packages("data.table")        # install it
library(data.table)                   # attach it
example(data.table)                   # run the examples section of ?data.table
?data.table                           # read
?fread                                # read
update.packages()                     # keep up to date

---

---

v1.14.1 in development

Please click and read latest news and keep up to date. That page is updated live as we push changes in development. When we release to CRAN we are saying that we are happy with the changes and in general you can rely on them being there in future. It is safe to simply upgrade to development version and give it a go. If it doesn't work out, you can very easily revert back to the CRAN version by removing the package and reinstalling using the commands above.

Windows

A compiled Windows binary .zip file is published. If you are using a recent R version (your major version / first digit number of R matches the current latest major version of R) you can install the package from from a pre-compiled binary file. If you are using an older version of R, then you will need to compile the development version from source, which will require Rtools.

On Windows, when upgrading any package that uses compiled code, it appears to be important to close all R sessions before upgrading. This releases all locks that Windows holds on dlls. To be really sure, reboot too. Then open a new fresh R session. Some users have an .Rprofile file that automatically loads data.table whenever any R sessions starts. You probably do not have an .Rprofile file but if you do, and it loads any packages (which themselves might load data.table), either temporarily rename .Rprofile or pass --vanilla when starting R to prevent it from running .Rprofile. Once you have a single fresh R session without data.table loaded proceed to installation.

install.packages("data.table", repos="https://Rdatatable.gitlab.io/data.table") ## defaults to binary version and only works if you are using a recent version of R
install.packages("data.table", type="source", repos="https://Rdatatable.gitlab.io/data.table") ## installs from source, which requires Rtools; use this for older versions of R
# or, which install only if newer version (by git commit hash) is available
data.table::update.dev.pkg()

To test your installation

require(data.table)
test.data.table()

The full suite of over 5,000 tests runs locally on your machine and should not take more than 2 minutes.

Linux and Mac

Linux users usually have all tools already installed by default. Mac users will need the correct version of Xcode Command Line Tools, see OpenMP below.

From a clean session (make sure data.table package isn't loaded):

remove.packages("data.table")
install.packages("data.table", type = "source",
    repos = "https://Rdatatable.gitlab.io/data.table")

If you would like to revert to CRAN version, from a clean session, do:

remove.packages("data.table")
install.packages("data.table")

Why install.packages above rather than devtools::install_github

This way you fetch the last version pushed to master which also passed the full test suite. This protects you from pulling master if it happens to be in a broken state. devtools always pulls master even if a recent push is failing tests. To do that for some reason (e.g. if we've just pushed a fix and you can't wait 15 mins for tests to complete, or if Travis or AppVeyor themselves are not running) :
library(devtools)
install_github("Rdatatable/data.table", build_vignettes=FALSE)

data.table not detecting OpenMP on Linux

Sometimes (this has been observed on Amazon Linux AMIs), it might happen that installation of data.table will be without any issue, but while loading the package, you may be told the OpenMP support does not exist. This could be because data.table might not be able to make use of the latest version of gcc installed on these machines to detect OpenMP, so you will need to create a Makevars file in ~/.R (in your home directory) with the following contents to refer to an older version of gcc, as so:

CC = /usr/bin/gcc64
CXX = /usr/bin/g++
SHLIB_OPENMP_CFLAGS = -fopenmp

Prior to this, make sure that gcc64 and libgomp (or equivalent) are present on your system.

Enable OpenMP for macOS

The clang compiler included in XCode and Command Line Tools does not include an OpenMP runtime. However, the OpenMP runtime can be installed with homebrew or from https://mac.r-project.org/openmp/. Or, you can install an OpenMP enabled compiler like LLVM's clang or GCC.

The basic steps are:

1. Install Command Line Tools
2. Install OpenMP support, either:
   • install OpenMP runtime for clang (libomp), recommended
   • or, install LLVM (clang), GFortran, or GCC
3. Update ~/.R/Makevars with flags or the preferred compiler.

Install Command Line Tools

First, ensure that you have command line tools installed. Do NOT skip this step. It is essential. See https://github.com/Rdatatable/data.table/issues/1692. From the terminal, type:

xcode-select --install

If you get an error message: xcode-select: error: command line tools are already installed, use "Software Update" to install updates, then you already have command line tools and can proceed to the next step. Else please follow the onscreen instructions and install it first.

NOTE: Some Big Sur updates may cause a condition where xcode command line tools may need to be updated but are not showing as updateable in Software Updater. To check and correct this condition, see the following from Macports

Option 1: Add the OpenMP runtime for clang (preferred)

Tested on Big Sur & Monterey.

Install libomp

Option 1.1 Install libomp with brew

Install homebrew if you have not already. After that install the OpenMP runtime (libomp):

brew update && brew install libomp

Option 1.2 brew-less install

Follow instructions here: https://mac.r-project.org/openmp/. Option 1 may be preferable if you already use brew.

Update Makevars

add the following lines to ~/.R/Makevars.

On Intel Macs:

CPPFLAGS += -Xclang -fopenmp
LDFLAGS += -lomp

On Apple Silicon Macs Issue 5419 (make sure this goes at the end of the file):

LDFLAGS += -L/opt/homebrew/opt/libomp/lib -lomp
CPPFLAGS += -I/opt/homebrew/opt/libomp/include -Xclang -fopenmp

or simply compile data.table directly from the command line with:

PKG_CPPFLAGS='-Xclang -fopenmp' PKG_LIBS=-lomp R CMD INSTALL path/to/data.table_<VERSION>.tar.gz

Option 2: Use an OpenMP enabled compiler

Choose one of the following OpenMP enabled compiler installs:

brew LLVM/clang

brew update && brew install llvm

Macports clang (tested on macOS 10.15.6)

sudo port install clang-10 pkgconfig
sudo port select clang mp-clang-10
## if you get: Object `datatable.so` not found
# sudo ln -s /opt/local/bin/pkg-config /usr/local/bin/pkg-config

GNU Fortran complier (tested on macOS Big Sur 11.1)

Download and install GNU Fortran complier from CRAN, which contains full GCC. CRAN recommends official GNU Fortran 8.2 from F.X.Coudert. Then add GCC version Makevars presented below to the file ~/.R/Makevars.

This options requires less disk storage (<300MB) than llvm (1.4GB).

GNU Fortran compiler (tested on Apple Silicon M1 with macOS Ventura 13.4.1 and GFortran 12.2)

I had a working data.table with openMP support in macOS Monterey but upgraded to Ventura, needed to install MacPorts for Ventura again, had to install XCode 14.3.1 CLT again.

xcode-select --install

I also had an old version of GCC which caused the error

<built-in>: error: unknown value '13.4.0' of '-mmacosx-version-min'

I downloaded gfortran-12.2-universal.pkg which includes GCC and adjusted my .R/Makevars to the actual directories and file names in lines 1--3:

LOC = /opt/gfortran
CC=$(LOC)/bin/aarch64-apple-darwin20.0-gcc -fopenmp
CXX=$(LOC)/bin/aarch64-apple-darwin20.0-cpp -fopenmp
CXX11 = $(LOC)/bin/g++ -fopenmp # for fst package
CFLAGS=-g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe
CXXFLAGS=-g -O3 -Wall -pedantic -std=c++11 -mtune=native -pipe
LDFLAGS=-L$(LOC)/lib -Wl,-rpath,$(LOC)/lib
CPPFLAGS=-I$(LOC)/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include

This worked to install data.table 1.14.9 from source in RStudio and also in a terminal R session:

install.packages("data.table", type = "source", repos = "https://Rdatatable.gitlab.io/data.table")

GCC

brew install gcc

Update Makevars with compiler path and flags

Add the following lines to the file ~/.R/Makevars using your favourite text editor. It's likely you need to create the .R directory and the file Makevars in it if it doesn't already exist. Note that the path to homebrew can vary depending on your Mac's architecture (see also #5271).

LLVM clang

# Newly installed Homebrew is located in
#  - /opt/homebrew for ARM Macs (M1 and its successors)
#  - /usr/local for Intel Macs
HOMEBREW_LOC=/opt/homebrew
# If you downloaded llvm manually above, replace with your chosen NEW_PATH/clang
LLVM_LOC=$(HOMEBREW_LOC)/opt/llvm
CC=$(LLVM_LOC)/bin/clang -fopenmp
CXX=$(LLVM_LOC)/bin/clang++ -fopenmp
# -O3 should be faster than -O2 (default) level optimisation ..
CFLAGS=-g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe
CXXFLAGS=-g -O3 -Wall -pedantic -std=c++11 -mtune=native -pipe
LDFLAGS=-L$(HOMEBREW_LOC)/opt/gettext/lib -L$(LLVM_LOC)/lib -Wl,-rpath,$(LLVM_LOC)/lib
CPPFLAGS=-I$(HOMEBREW_LOC)/opt/gettext/include -I$(LLVM_LOC)/include

For macOS 10.14.5+ you may need to supplement the last line as follows (see #3698) :

CPPFLAGS=-I$(HOMEBREW_LOC)/opt/gettext/include -I$(LLVM_LOC)/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include

If you also intend to use the fst package, please refer to this issue.

Variant for Macports users With OpenMP enabled as above, simply install r-data.table with OpenMP option. On Intel mac with OS 14.6.1, this works without any Makevars setting.

sudo port install r-data.table +openmp

CC=/opt/local/bin/clang -fopenmp
CXX=/opt/local/bin/clang++ -fopenmp
# -O3 should be faster than -O2 (default) level optimisation ..
CFLAGS=-isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe
CCFLAGS=-isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk
CXXFLAGS=-isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -g -O3 -Wall -pedantic -std=c++11 -mtune=native -pipe
LDFLAGS=-L/opt/local/lib -Wl,-rpath,/opt/local/lib
CPPFLAGS=-I/opt/local/include

See also variant for Macports users on Ventura on Apple Silicon above.

GCC (Official GNU fortran) ver

LOC = /usr/local/gfortran
CC=$(LOC)/bin/gcc -fopenmp
CXX=$(LOC)/bin/g++ -fopenmp
CXX11 = $(LOC)/bin/g++ -fopenmp # for fst package

CFLAGS=-g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe
CXXFLAGS=-g -O3 -Wall -pedantic -std=c++11 -mtune=native -pipe
LDFLAGS=-L$(LOC)/lib -Wl,-rpath,$(LOC)/lib
CPPFLAGS=-I$(LOC)/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include

GCC Brew (Option 5 above)

VER=9  # set the version of brew gcc. (as of today: 11)
CC=gcc-$(VER) -fopenmp   # brew gcc nicely creates gcc-9 as symlink 
CXX=g++-$(VER) -fopenmp
CFLAGS=-g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe
CXXFLAGS=-g -O3 -Wall -pedantic -std=c++11 -mtune=native -pipe

Save and close the file. Then you should be all set on Mac. The -fopenmp tag on CC and CXX lines might cause compile problems for other packages, notably stringi. Simply remove -fopenmp or comment # after compiling data.table. RStudio IDE reads those flags thus its definition jump functionality for C or C++ codes may get affected. Remove the above added lines if you need that.

Installing R via homebrew might cause additional problems during installation and is not recommended. However, if you want to install it using Homebrew you can find extended instructions to install R and data.table in this blog post.

Step 4 (only if installation error persists)

gettext issues

It has been reported that gettext is needed to be installed too. There are several posts online on how to do that, and it may depend on which version of MacOS you have.

<stdlib.h> or <stdio.h> file not found

If you encounter a stdio.h file not found error after following the above instructions, install the package at /Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg so as to create a /usr/include directory (this no longer works for MacOS 10.15+, see here).

To install this package, you can either double click the file or run

sudo installer -pkg /Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg -target /

Error "linker command failed"

clang-10: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [data.table.so] Error 1

Check the solution described in this issue.

Object datatable.so not found

If you encounter a shared object ‘datatable.so’ not found error after following the above instructions, you may try installing pkg-config using homebrew:

brew install pkg-config

and then re-attempting the installation from a clean R-session (see #4574).

error: 'TARGET_OS_IPHONE' is not defined

On macOS 15 you might encounter this error while compiling stdio.h and/or stdlib.h. This is due to the fact that LLVM/Clang 11 automatically adds -Werror,-Wundef-prefix=TARGET_OS_ as default compiler. Apparently TARGET_OS_MAC isn't defined in this compilation environment. Adding -Wno-undef-prefix to the last line of your Makevars file solves the issue.

error: Library not loaded: /opt/X11/lib/libSM.6.dylib

On macOS you might encounter this error Library not loaded: /opt/X11/lib/libSM.6.dylib during installation. You may try installing XQuartz using homebrew (see here):

brew install xquartz --cask