Last chance! 50% off unlimited learning
Sale ends in
The following options for repartition are possible:
1. Return a new SparkDataFrame that has exactly numPartitions
.
2. Return a new SparkDataFrame hash partitioned by
the given columns into numPartitions
.
3. Return a new SparkDataFrame hash partitioned by the given column(s),
using spark.sql.shuffle.partitions
as number of partitions.
repartition(x, ...)# S4 method for SparkDataFrame
repartition(x, numPartitions = NULL, col = NULL,
...)
a SparkDataFrame.
additional column(s) to be used in the partitioning.
the number of partitions to use.
the column by which the partitioning will be performed.
Other SparkDataFrame functions: SparkDataFrame-class
,
agg
, arrange
,
as.data.frame
, attach
,
cache
, coalesce
,
collect
, colnames
,
coltypes
,
createOrReplaceTempView
,
crossJoin
, dapplyCollect
,
dapply
, describe
,
dim
, distinct
,
dropDuplicates
, dropna
,
drop
, dtypes
,
except
, explain
,
filter
, first
,
gapplyCollect
, gapply
,
getNumPartitions
, group_by
,
head
, histogram
,
insertInto
, intersect
,
isLocal
, join
,
limit
, merge
,
mutate
, ncol
,
nrow
, persist
,
printSchema
, randomSplit
,
rbind
, registerTempTable
,
rename
, sample
,
saveAsTable
, schema
,
selectExpr
, select
,
showDF
, show
,
storageLevel
, str
,
subset
, take
,
union
, unpersist
,
withColumn
, with
,
write.df
, write.jdbc
,
write.json
, write.orc
,
write.parquet
, write.text
# NOT RUN {
sparkR.session()
path <- "path/to/file.json"
df <- read.json(path)
newDF <- repartition(df, 2L)
newDF <- repartition(df, numPartitions = 2L)
newDF <- repartition(df, col = df$"col1", df$"col2")
newDF <- repartition(df, 3L, col = df$"col1", df$"col2")
# }
Run the code above in your browser using DataLab