memshare: Shared memory multithreading in R via C++17, with zero-copy access through ALTREP data structures.

Overview

memshare enables multicore computation in R without redundant memory copies. Large vectors, matrices, or lists are stored once in shared memory and exposed to R processes as ALTREP views. This allows workers in a PSOCK cluster to operate on the same physical data while avoiding serialization overhead.

Key features:

• Shared memory allocation via C++17 back-end (shm_open on Unix, MapViewOfFile on Windows).
• ALTREP wrappers so R sees shared objects as native vectors or matrices.
• High-level parallels to parallel::parApply and parallel::parLapply:
  • memApply() — apply a function row/column-wise over a matrix in shared memory.
  • memLapply() — apply a function element-wise over a list in shared memory.
• Automatic cleanup of shared objects when all views are released or when the package is unloaded.
• Tested on Linux, macOS, and Windows.

Installation

From CRAN:

install.packages("memshare")

From GitHub (development version):

remotes::install_github("mthrun/memshare")

System requirements: R ≥ 4.0 and a C++17 compiler.

Quick start

Example 1: Parallel correlations with a matrix

library(memshare)
library(parallel)

set.seed(1)
n = 10000
p = 2000

X = matrix(rnorm(n * p), n, p)
y = rnorm(n)

res = memApply(
  X = X, MARGIN = 2,
  FUN = function(v, y) cor(v, y),
  VARS = list(y = y)
)
str(res)

Example 2: List operations

library(memshare)
library(parallel)

list_length = 1000
matrix_dim = 100

ListV = lapply(
     1:list_length,
     function(i) matrix(rnorm(matrix_dim * matrix_dim),
     nrow = matrix_dim, ncol = matrix_dim))

y = rnorm(matrix_dim)

namespace = "ID123"
res = memshare::memLapply(ListV, function(el, y) {
   el %*% y
}, NAMESPACE=namespace, VARS=list(y=y), MAX.CORES = 1)

Each element el of ListV is multiplied by y in parallel. The list resides once in shared memory.

Concepts

• Pages: memory regions owned by the current R session that loaded the package.

• Views: ALTREP wrappers exposing shared memory variables (read/write capable).

• Namespaces: string identifiers defining a shared memory context across sessions.

When the package is detached, all handles and associated shared memory pages are released, unless another R process still holds references.

Shared-memory lifecycle

memshare exposes explicit lifecycle functions so you can control when data is placed in shared memory and when it is freed.

registerVariables(namespace, variableList)

Allocate shared memory and copy R objects (matrices or vectors, or lists for memLapply) into it.

• namespace: character(1). Identifier of the shared memory context shared across processes.
• variableList: a named list of objects to register. Names become the keys under which you can later retrieve views.

Example

library(memshare)

ns <- "my_namespace"
X  <- matrix(rnorm(1e4), 100, 100)
y  <- rnorm(100)

registerVariables(ns, list(X = X, y = y))
# Now X and y live once in shared memory and can be accessed from other R processes

releaseVariables(namespace, variableNames)

Delete variables from the shared memory space. Shared regions are only removed when no active views remain.

• namespace: character(1) used above.
• variableNames: character vector of variable names to free.

Example

# After all workers have released their views:
releaseVariables(ns, c("X", "y"))

Working with views: retrieveViews() and releaseViews()

To avoid duplication, workers attach to shared memory by views:

• retrieveViews(namespace, c("X","y")) returns ALTREP-backed objects that behave like ordinary R matrices/vectors.
• Always call releaseViews(namespace, ...) when finished so that the backing memory can be reclaimed.

Example (worker-side)

vlist <- retrieveViews(ns, c("X","y"))
# use vlist$X and vlist$y
releaseViews(ns, c("X","y"))

Tip: memApply() and memLapply() manage views for you automatically, but the low-level API above is useful for custom workflows.

Manual

The full manual for users or developers is available here: Package documentation

References

[Thrun and Märte, 2025] Thrun, M.C., Märte, J.: Memshare: Memory Sharing for Multicore Computation in R with an Application to Feature Selection by Mutual Information using PDE, 2025.

[Thrun et al., 2020] Thrun, M.C., Gehlert, T., & Ultsch, A.: Analyzing the Fine Structure of Distributions, PLOS ONE, 15(10), e0238835, 2020.

[Ultsch, 2005] Ultsch, A.: Pareto Density Estimation: A Density Estimation for Knowledge Discovery, Proceedings of the 28th Annual Conference of the German Classification Society, Springer, 2005