Perform Weighted Random Sampling on a Spark DataFrame

Draw a random sample of rows (with or without replacement) from a Spark DataFrame If the sampling is done without replacement, then it will be conceptually equivalent to an iterative process such that in each step the probability of adding a row to the sample set is equal to its weight divided by summation of weights of all rows that are not in the sample set yet in that step.

Usage

sdf_weighted_sample(x, weight_col, k, replacement = TRUE, seed = NULL)

Arguments

x

An object coercable to a Spark DataFrame.

weight_col

Name of the weight column

k

Sample set size

replacement

Whether to sample with replacement

seed

An (optional) integer seed

See also

Other Spark data frames: sdf_copy_to(), sdf_distinct(), sdf_random_split(), sdf_register(), sdf_sample(), sdf_sort()