Fast (Grouped, Weighted) Sum for Matrix-Like Objects
fsum.Rdfsum is a generic function that computes the (column-wise) sum of all values in x, (optionally) grouped by g and/or weighted by w (e.g. to calculate survey totals). The TRA argument can further be used to transform x using its (grouped, weighted) sum.
Usage
fsum(x, ...)
# Default S3 method
fsum(x, g = NULL, w = NULL, TRA = NULL, na.rm = .op[["na.rm"]],
use.g.names = TRUE, fill = FALSE, nthreads = .op[["nthreads"]], ...)
# S3 method for class 'matrix'
fsum(x, g = NULL, w = NULL, TRA = NULL, na.rm = .op[["na.rm"]],
use.g.names = TRUE, drop = TRUE, fill = FALSE, nthreads = .op[["nthreads"]], ...)
# S3 method for class 'data.frame'
fsum(x, g = NULL, w = NULL, TRA = NULL, na.rm = .op[["na.rm"]],
use.g.names = TRUE, drop = TRUE, fill = FALSE, nthreads = .op[["nthreads"]], ...)
# S3 method for class 'grouped_df'
fsum(x, w = NULL, TRA = NULL, na.rm = .op[["na.rm"]],
use.g.names = FALSE, keep.group_vars = TRUE, keep.w = TRUE, stub = .op[["stub"]],
fill = FALSE, nthreads = .op[["nthreads"]], ...)Arguments
- x
a numeric vector, matrix, data frame or grouped data frame (class 'grouped_df').
- g
a factor,
GRPobject, atomic vector (internally converted to factor) or a list of vectors / factors (internally converted to aGRPobject) used to groupx.- w
a numeric vector of (non-negative) weights, may contain missing values.
- TRA
an integer or quoted operator indicating the transformation to perform: 0 - "na" | 1 - "fill" | 2 - "replace" | 3 - "-" | 4 - "-+" | 5 - "/" | 6 - "%" | 7 - "+" | 8 - "*" | 9 - "%%" | 10 - "-%%". See
TRA.- na.rm
logical. Skip missing values in
x. Defaults toTRUEand implemented at very little computational cost. Ifna.rm = FALSEaNAis returned when encountered.- use.g.names
logical. Make group-names and add to the result as names (default method) or row-names (matrix and data frame methods). No row-names are generated for data.table's.
- fill
logical. Initialize result with
0instead ofNAwhenna.rm = TRUEe.g.fsum(NA, fill = TRUE)returns0instead ofNA.- nthreads
integer. The number of threads to utilize. See Details.
- drop
matrix and data.frame method: Logical.
TRUEdrops dimensions and returns an atomic vector ifg = NULLandTRA = NULL.- keep.group_vars
grouped_df method: Logical.
FALSEremoves grouping variables after computation.- keep.w
grouped_df method: Logical. Retain summed weighting variable after computation (if contained in
grouped_df).- stub
character. If
keep.w = TRUEandstub = TRUE(default), the summed weights column is prefixed by"sum.". Users can specify a different prefix through this argument, or set it toFALSEto avoid prefixing.- ...
arguments to be passed to or from other methods. If
TRAis used, passingset = TRUEwill transform data by reference and return the result invisibly.
Details
The weighted sum (e.g. survey total) is computed as sum(x * w), but in one pass and about twice as efficient. If na.rm = TRUE, missing values will be removed from both x and w i.e. utilizing only x[complete.cases(x,w)] and w[complete.cases(x,w)].
This all seamlessly generalizes to grouped computations, which are performed in a single pass (without splitting the data) and are therefore extremely fast. See Benchmark and Examples below.
When applied to data frames with groups or drop = FALSE, fsum preserves all column attributes. The attributes of the data frame itself are also preserved.
Since v1.6.0 fsum explicitly supports integers. Integers are summed using the long long type in C which is bounded at +-9,223,372,036,854,775,807 (so ~4.3 billion times greater than the minimum/maximum R integer bounded at +-2,147,483,647). If the value of the sum is outside +-2,147,483,647, a double containing the result is returned, otherwise an integer is returned. With groups, an integer results vector is initialized, and an integer overflow error is provided if the sum in any group is outside +-2,147,483,647. Data needs to be coerced to double beforehand in such cases.
Multithreading, added in v1.8.0, applies at the column-level unless g = NULL and nthreads > NCOL(x). Parallelism over groups is not available because sums are computed simultaneously within each group. nthreads = 1L uses a serial version of the code, not parallel code running on one thread. This serial code is always used with less than 100,000 obs (length(x) < 100000 for vectors and matrices), because parallel execution itself has some overhead.
Value
The (w weighted) sum of x, grouped by g, or (if TRA is used) x transformed by its (grouped, weighted) sum.
Examples
## default vector method
mpg <- mtcars$mpg
fsum(mpg) # Simple sum
#> [1] 642.9
fsum(mpg, w = mtcars$hp) # Weighted sum (total): Weighted by hp
#> [1] 84362.7
fsum(mpg, TRA = "%") # Simple transformation: obtain percentages of mpg
#> [1] 3.266449 3.266449 3.546430 3.328667 2.908695 2.815368 2.224296 3.795303
#> [9] 3.546430 2.986468 2.768704 2.550941 2.690932 2.364287 1.617670 1.617670
#> [17] 2.286514 5.039664 4.728574 5.272982 3.344221 2.410950 2.364287 2.068751
#> [25] 2.986468 4.246384 4.044175 4.728574 2.457614 3.064240 2.333178 3.328667
fsum(mpg, mtcars$cyl) # Grouped sum
#> 4 6 8
#> 293.3 138.2 211.4
fsum(mpg, mtcars$cyl, mtcars$hp) # Weighted grouped sum (total)
#> 4 6 8
#> 23743.0 16873.0 43746.7
fsum(mpg, mtcars[c(2,8:9)]) # More groups..
#> 4.0.1 4.1.0 4.1.1 6.0.1 6.1.0 8.0.0 8.0.1
#> 26.0 68.7 198.6 61.7 76.5 180.6 30.8
g <- GRP(mtcars, ~ cyl + vs + am) # Precomputing groups gives more speed !
fsum(mpg, g)
#> 4.0.1 4.1.0 4.1.1 6.0.1 6.1.0 8.0.0 8.0.1
#> 26.0 68.7 198.6 61.7 76.5 180.6 30.8
fmean(mpg, g) == fsum(mpg, g) / fnobs(mpg, g)
#> 4.0.1 4.1.0 4.1.1 6.0.1 6.1.0 8.0.0 8.0.1
#> TRUE TRUE TRUE TRUE TRUE TRUE TRUE
fsum(mpg, g, TRA = "%") # Percentages by group
#> [1] 34.035656 34.035656 11.480363 27.973856 10.354374 23.660131
#> [7] 7.918051 35.516739 33.187773 25.098039 23.267974 9.080842
#> [13] 9.579181 8.416390 5.758583 5.758583 8.139535 16.314199
#> [19] 15.307150 17.069486 31.295488 8.582503 8.416390 7.364341
#> [25] 10.631229 13.746224 100.000000 15.307150 51.298701 31.928687
#> [31] 48.701299 10.775428
## data.frame method
fsum(mtcars)
#> mpg cyl disp hp drat wt qsec vs
#> 642.900 198.000 7383.100 4694.000 115.090 102.952 571.160 14.000
#> am gear carb
#> 13.000 118.000 90.000
fsum(mtcars, TRA = "%")
#> mpg cyl disp hp drat wt
#> Mazda RX4 3.266449 3.030303 2.1671114 2.343417 3.388652 2.544875
#> Mazda RX4 Wag 3.266449 3.030303 2.1671114 2.343417 3.388652 2.792564
#> Datsun 710 3.546430 2.020202 1.4628002 1.981253 3.345208 2.253477
#> Hornet 4 Drive 3.328667 3.030303 3.4944671 2.343417 2.676166 3.122815
#> Hornet Sportabout 2.908695 4.040404 4.8760006 3.728164 2.736988 3.341363
#> Valiant 2.815368 3.030303 3.0475004 2.236898 2.398123 3.360789
#> Duster 360 2.224296 4.040404 4.8760006 5.219429 2.789122 3.467635
#> Merc 240D 3.795303 2.020202 1.9869702 1.320835 3.206186 3.098531
#> Merc 230 3.546430 2.020202 1.9070580 2.023860 3.406030 3.059678
#> Merc 280 2.986468 3.030303 2.2700492 2.620366 3.406030 3.341363
#> Merc 280C 2.768704 3.030303 2.2700492 2.620366 3.406030 3.341363
#> Merc 450SE 2.550941 4.040404 3.7355582 3.834683 2.667478 3.953299
#> Merc 450SL 2.690932 4.040404 3.7355582 3.834683 2.667478 3.623048
#> Merc 450SLC 2.364287 4.040404 3.7355582 3.834683 2.667478 3.671614
#> Cadillac Fleetwood 1.617670 4.040404 6.3929786 4.367277 2.545834 5.099464
#> Lincoln Continental 1.617670 4.040404 6.2304452 4.580315 2.606656 5.268475
#> Chrysler Imperial 2.286514 4.040404 5.9595563 4.899872 2.806499 5.191740
#> Fiat 128 5.039664 2.020202 1.0659479 1.406050 3.545052 2.136918
#> Honda Civic 4.728574 2.020202 1.0253146 1.107797 4.283604 1.568692
#> Toyota Corolla 5.272982 2.020202 0.9630101 1.384746 3.666696 1.782384
#> Toyota Corona 3.344221 2.020202 1.6266880 2.066468 3.214875 2.394320
#> Dodge Challenger 2.410950 4.040404 4.3071339 3.195569 2.398123 3.419069
#> AMC Javelin 2.364287 4.040404 4.1175116 3.195569 2.736988 3.336506
#> Camaro Z28 2.068751 4.040404 4.7405561 5.219429 3.240942 3.729894
#> Pontiac Firebird 2.986468 4.040404 5.4177784 3.728164 2.676166 3.734750
#> Fiat X1-9 4.246384 2.020202 1.0700112 1.406050 3.545052 1.879517
#> Porsche 914-2 4.044175 2.020202 1.6293969 1.938645 3.849162 2.078639
#> Lotus Europa 4.728574 2.020202 1.2880768 2.407329 3.275697 1.469617
#> Ford Pantera L 2.457614 4.040404 4.7541006 5.624201 3.666696 3.079105
#> Ferrari Dino 3.064240 3.030303 1.9639447 3.728164 3.145364 2.690574
#> Maserati Bora 2.333178 4.040404 4.0768783 7.136770 3.075854 3.467635
#> Volvo 142E 3.328667 2.020202 1.6388780 2.322113 3.571118 2.700288
#> qsec vs am gear carb
#> Mazda RX4 2.881854 0.000000 7.692308 3.389831 4.444444
#> Mazda RX4 Wag 2.979901 0.000000 7.692308 3.389831 4.444444
#> Datsun 710 3.258281 7.142857 7.692308 3.389831 1.111111
#> Hornet 4 Drive 3.403600 7.142857 0.000000 2.542373 1.111111
#> Hornet Sportabout 2.979901 0.000000 0.000000 2.542373 2.222222
#> Valiant 3.540164 7.142857 0.000000 2.542373 1.111111
#> Duster 360 2.773303 0.000000 0.000000 2.542373 4.444444
#> Merc 240D 3.501646 7.142857 0.000000 3.389831 2.222222
#> Merc 230 4.009384 7.142857 0.000000 3.389831 2.222222
#> Merc 280 3.204006 7.142857 0.000000 3.389831 4.444444
#> Merc 280C 3.309055 7.142857 0.000000 3.389831 4.444444
#> Merc 450SE 3.046432 0.000000 0.000000 2.542373 3.333333
#> Merc 450SL 3.081448 0.000000 0.000000 2.542373 3.333333
#> Merc 450SLC 3.151481 0.000000 0.000000 2.542373 3.333333
#> Cadillac Fleetwood 3.147980 0.000000 0.000000 2.542373 4.444444
#> Lincoln Continental 3.119966 0.000000 0.000000 2.542373 4.444444
#> Chrysler Imperial 3.049933 0.000000 0.000000 2.542373 4.444444
#> Fiat 128 3.408852 7.142857 7.692308 3.389831 1.111111
#> Honda Civic 3.242524 7.142857 7.692308 3.389831 2.222222
#> Toyota Corolla 3.484138 7.142857 7.692308 3.389831 1.111111
#> Toyota Corona 3.503397 7.142857 0.000000 2.542373 1.111111
#> Dodge Challenger 2.953638 0.000000 0.000000 2.542373 2.222222
#> AMC Javelin 3.028924 0.000000 0.000000 2.542373 2.222222
#> Camaro Z28 2.698018 0.000000 0.000000 2.542373 4.444444
#> Pontiac Firebird 2.985153 0.000000 0.000000 2.542373 2.222222
#> Fiat X1-9 3.309055 7.142857 7.692308 3.389831 1.111111
#> Porsche 914-2 2.923874 0.000000 7.692308 4.237288 2.222222
#> Lotus Europa 2.958891 7.142857 7.692308 4.237288 2.222222
#> Ford Pantera L 2.538693 0.000000 7.692308 4.237288 4.444444
#> Ferrari Dino 2.713775 0.000000 7.692308 4.237288 6.666667
#> Maserati Bora 2.556201 0.000000 7.692308 4.237288 8.888889
#> Volvo 142E 3.256531 7.142857 7.692308 3.389831 2.222222
fsum(mtcars, g)
#> mpg cyl disp hp drat wt qsec vs am gear carb
#> 4.0.1 26.0 4 120.3 91 4.43 2.140 16.70 0 1 5 2
#> 4.1.0 68.7 12 407.6 254 11.31 8.805 62.91 3 0 11 5
#> 4.1.1 198.6 28 628.6 564 29.04 14.198 130.90 7 7 29 10
#> 6.0.1 61.7 18 465.0 395 11.42 8.265 48.98 0 3 13 14
#> 6.1.0 76.5 24 818.2 461 13.68 13.555 76.86 4 0 14 10
#> 8.0.0 180.6 96 4291.4 2330 37.45 49.249 205.71 0 0 36 37
#> 8.0.1 30.8 16 652.0 599 7.76 6.740 29.10 0 2 10 12
fsum(mtcars, g, TRA = "%")
#> mpg cyl disp hp drat
#> Mazda RX4 34.035656 33.333333 34.408602 27.848101 34.150613
#> Mazda RX4 Wag 34.035656 33.333333 34.408602 27.848101 34.150613
#> Datsun 710 11.480363 14.285714 17.181037 16.489362 13.257576
#> Hornet 4 Drive 27.973856 25.000000 31.532633 23.861171 22.514620
#> Hornet Sportabout 10.354374 8.333333 8.388871 7.510730 8.411215
#> Valiant 23.660131 25.000000 27.499389 22.776573 20.175439
#> Duster 360 7.918051 8.333333 8.388871 10.515021 8.571429
#> Merc 240D 35.516739 33.333333 35.991168 24.409449 32.625995
#> Merc 230 33.187773 33.333333 34.543670 37.401575 34.659593
#> Merc 280 25.098039 25.000000 20.483989 26.681128 28.654971
#> Merc 280C 23.267974 25.000000 20.483989 26.681128 28.654971
#> Merc 450SE 9.080842 8.333333 6.426807 7.725322 8.197597
#> Merc 450SL 9.579181 8.333333 6.426807 7.725322 8.197597
#> Merc 450SLC 8.416390 8.333333 6.426807 7.725322 8.197597
#> Cadillac Fleetwood 5.758583 8.333333 10.998742 8.798283 7.823765
#> Lincoln Continental 5.758583 8.333333 10.719113 9.227468 8.010681
#> Chrysler Imperial 8.139535 8.333333 10.253064 9.871245 8.624833
#> Fiat 128 16.314199 14.285714 12.519885 11.702128 14.049587
#> Honda Civic 15.307150 14.285714 12.042634 9.219858 16.976584
#> Toyota Corolla 17.069486 14.285714 11.310850 11.524823 14.531680
#> Toyota Corona 31.295488 33.333333 29.465162 38.188976 32.714412
#> Dodge Challenger 8.582503 8.333333 7.410169 6.437768 7.369826
#> AMC Javelin 8.416390 8.333333 7.083935 6.437768 8.411215
#> Camaro Z28 7.364341 8.333333 8.155847 10.515021 9.959947
#> Pontiac Firebird 10.631229 8.333333 9.320968 7.510730 8.224299
#> Fiat X1-9 13.746224 14.285714 12.567611 11.702128 14.049587
#> Porsche 914-2 100.000000 100.000000 100.000000 100.000000 100.000000
#> Lotus Europa 15.307150 14.285714 15.128858 20.035461 12.982094
#> Ford Pantera L 51.298701 50.000000 53.834356 44.073456 54.381443
#> Ferrari Dino 31.928687 33.333333 31.182796 44.303797 31.698774
#> Maserati Bora 48.701299 50.000000 46.165644 55.926544 45.618557
#> Volvo 142E 10.775428 14.285714 19.249125 19.326241 14.152893
#> wt qsec vs am gear
#> Mazda RX4 31.699940 33.605553 NaN 33.33333 30.769231
#> Mazda RX4 Wag 34.785239 34.748877 NaN 33.33333 30.769231
#> Datsun 710 16.340330 14.216960 14.28571 14.28571 13.793103
#> Hornet 4 Drive 23.718185 25.292740 25.00000 NaN 21.428571
#> Hornet Sportabout 6.984913 8.273783 NaN NaN 8.333333
#> Valiant 25.525636 26.307572 25.00000 NaN 21.428571
#> Duster 360 7.248878 7.700160 NaN NaN 8.333333
#> Merc 240D 36.229415 31.791448 33.33333 NaN 36.363636
#> Merc 230 35.775128 36.401208 33.33333 NaN 36.363636
#> Merc 280 25.378089 23.809524 25.00000 NaN 28.571429
#> Merc 280C 25.378089 24.590164 25.00000 NaN 28.571429
#> Merc 450SE 8.264127 8.458510 NaN NaN 8.333333
#> Merc 450SL 7.573758 8.555734 NaN NaN 8.333333
#> Merc 450SLC 7.675283 8.750182 NaN NaN 8.333333
#> Cadillac Fleetwood 10.660115 8.740460 NaN NaN 8.333333
#> Lincoln Continental 11.013422 8.662680 NaN NaN 8.333333
#> Chrysler Imperial 10.853012 8.468232 NaN NaN 8.333333
#> Fiat 128 15.495140 14.873950 14.28571 14.28571 13.793103
#> Honda Civic 11.374842 14.148205 14.28571 14.28571 13.793103
#> Toyota Corolla 12.924356 15.202445 14.28571 14.28571 13.793103
#> Toyota Corona 27.995457 31.807344 33.33333 NaN 27.272727
#> Dodge Challenger 7.147353 8.200865 NaN NaN 8.333333
#> AMC Javelin 6.974761 8.409897 NaN NaN 8.333333
#> Camaro Z28 7.797113 7.491128 NaN NaN 8.333333
#> Pontiac Firebird 7.807265 8.288367 NaN NaN 8.333333
#> Fiat X1-9 13.628680 14.438503 14.28571 14.28571 13.793103
#> Porsche 914-2 100.000000 100.000000 NaN 100.00000 100.000000
#> Lotus Europa 10.656430 12.910619 14.28571 14.28571 17.241379
#> Ford Pantera L 47.032641 49.828179 NaN 50.00000 50.000000
#> Ferrari Dino 33.514822 31.645570 NaN 33.33333 38.461538
#> Maserati Bora 52.967359 50.171821 NaN 50.00000 50.000000
#> Volvo 142E 19.580223 14.209320 14.28571 14.28571 13.793103
#> carb
#> Mazda RX4 28.571429
#> Mazda RX4 Wag 28.571429
#> Datsun 710 10.000000
#> Hornet 4 Drive 10.000000
#> Hornet Sportabout 5.405405
#> Valiant 10.000000
#> Duster 360 10.810811
#> Merc 240D 40.000000
#> Merc 230 40.000000
#> Merc 280 40.000000
#> Merc 280C 40.000000
#> Merc 450SE 8.108108
#> Merc 450SL 8.108108
#> Merc 450SLC 8.108108
#> Cadillac Fleetwood 10.810811
#> Lincoln Continental 10.810811
#> Chrysler Imperial 10.810811
#> Fiat 128 10.000000
#> Honda Civic 20.000000
#> Toyota Corolla 10.000000
#> Toyota Corona 20.000000
#> Dodge Challenger 5.405405
#> AMC Javelin 5.405405
#> Camaro Z28 10.810811
#> Pontiac Firebird 5.405405
#> Fiat X1-9 10.000000
#> Porsche 914-2 100.000000
#> Lotus Europa 20.000000
#> Ford Pantera L 33.333333
#> Ferrari Dino 42.857143
#> Maserati Bora 66.666667
#> Volvo 142E 20.000000
## matrix method
m <- qM(mtcars)
fsum(m)
#> mpg cyl disp hp drat wt qsec vs
#> 642.900 198.000 7383.100 4694.000 115.090 102.952 571.160 14.000
#> am gear carb
#> 13.000 118.000 90.000
fsum(m, TRA = "%")
#> mpg cyl disp hp drat wt
#> Mazda RX4 3.266449 3.030303 2.1671114 2.343417 3.388652 2.544875
#> Mazda RX4 Wag 3.266449 3.030303 2.1671114 2.343417 3.388652 2.792564
#> Datsun 710 3.546430 2.020202 1.4628002 1.981253 3.345208 2.253477
#> Hornet 4 Drive 3.328667 3.030303 3.4944671 2.343417 2.676166 3.122815
#> Hornet Sportabout 2.908695 4.040404 4.8760006 3.728164 2.736988 3.341363
#> Valiant 2.815368 3.030303 3.0475004 2.236898 2.398123 3.360789
#> Duster 360 2.224296 4.040404 4.8760006 5.219429 2.789122 3.467635
#> Merc 240D 3.795303 2.020202 1.9869702 1.320835 3.206186 3.098531
#> Merc 230 3.546430 2.020202 1.9070580 2.023860 3.406030 3.059678
#> Merc 280 2.986468 3.030303 2.2700492 2.620366 3.406030 3.341363
#> Merc 280C 2.768704 3.030303 2.2700492 2.620366 3.406030 3.341363
#> Merc 450SE 2.550941 4.040404 3.7355582 3.834683 2.667478 3.953299
#> Merc 450SL 2.690932 4.040404 3.7355582 3.834683 2.667478 3.623048
#> Merc 450SLC 2.364287 4.040404 3.7355582 3.834683 2.667478 3.671614
#> Cadillac Fleetwood 1.617670 4.040404 6.3929786 4.367277 2.545834 5.099464
#> Lincoln Continental 1.617670 4.040404 6.2304452 4.580315 2.606656 5.268475
#> Chrysler Imperial 2.286514 4.040404 5.9595563 4.899872 2.806499 5.191740
#> Fiat 128 5.039664 2.020202 1.0659479 1.406050 3.545052 2.136918
#> Honda Civic 4.728574 2.020202 1.0253146 1.107797 4.283604 1.568692
#> Toyota Corolla 5.272982 2.020202 0.9630101 1.384746 3.666696 1.782384
#> Toyota Corona 3.344221 2.020202 1.6266880 2.066468 3.214875 2.394320
#> Dodge Challenger 2.410950 4.040404 4.3071339 3.195569 2.398123 3.419069
#> AMC Javelin 2.364287 4.040404 4.1175116 3.195569 2.736988 3.336506
#> Camaro Z28 2.068751 4.040404 4.7405561 5.219429 3.240942 3.729894
#> Pontiac Firebird 2.986468 4.040404 5.4177784 3.728164 2.676166 3.734750
#> Fiat X1-9 4.246384 2.020202 1.0700112 1.406050 3.545052 1.879517
#> Porsche 914-2 4.044175 2.020202 1.6293969 1.938645 3.849162 2.078639
#> Lotus Europa 4.728574 2.020202 1.2880768 2.407329 3.275697 1.469617
#> Ford Pantera L 2.457614 4.040404 4.7541006 5.624201 3.666696 3.079105
#> Ferrari Dino 3.064240 3.030303 1.9639447 3.728164 3.145364 2.690574
#> Maserati Bora 2.333178 4.040404 4.0768783 7.136770 3.075854 3.467635
#> Volvo 142E 3.328667 2.020202 1.6388780 2.322113 3.571118 2.700288
#> qsec vs am gear carb
#> Mazda RX4 2.881854 0.000000 7.692308 3.389831 4.444444
#> Mazda RX4 Wag 2.979901 0.000000 7.692308 3.389831 4.444444
#> Datsun 710 3.258281 7.142857 7.692308 3.389831 1.111111
#> Hornet 4 Drive 3.403600 7.142857 0.000000 2.542373 1.111111
#> Hornet Sportabout 2.979901 0.000000 0.000000 2.542373 2.222222
#> Valiant 3.540164 7.142857 0.000000 2.542373 1.111111
#> Duster 360 2.773303 0.000000 0.000000 2.542373 4.444444
#> Merc 240D 3.501646 7.142857 0.000000 3.389831 2.222222
#> Merc 230 4.009384 7.142857 0.000000 3.389831 2.222222
#> Merc 280 3.204006 7.142857 0.000000 3.389831 4.444444
#> Merc 280C 3.309055 7.142857 0.000000 3.389831 4.444444
#> Merc 450SE 3.046432 0.000000 0.000000 2.542373 3.333333
#> Merc 450SL 3.081448 0.000000 0.000000 2.542373 3.333333
#> Merc 450SLC 3.151481 0.000000 0.000000 2.542373 3.333333
#> Cadillac Fleetwood 3.147980 0.000000 0.000000 2.542373 4.444444
#> Lincoln Continental 3.119966 0.000000 0.000000 2.542373 4.444444
#> Chrysler Imperial 3.049933 0.000000 0.000000 2.542373 4.444444
#> Fiat 128 3.408852 7.142857 7.692308 3.389831 1.111111
#> Honda Civic 3.242524 7.142857 7.692308 3.389831 2.222222
#> Toyota Corolla 3.484138 7.142857 7.692308 3.389831 1.111111
#> Toyota Corona 3.503397 7.142857 0.000000 2.542373 1.111111
#> Dodge Challenger 2.953638 0.000000 0.000000 2.542373 2.222222
#> AMC Javelin 3.028924 0.000000 0.000000 2.542373 2.222222
#> Camaro Z28 2.698018 0.000000 0.000000 2.542373 4.444444
#> Pontiac Firebird 2.985153 0.000000 0.000000 2.542373 2.222222
#> Fiat X1-9 3.309055 7.142857 7.692308 3.389831 1.111111
#> Porsche 914-2 2.923874 0.000000 7.692308 4.237288 2.222222
#> Lotus Europa 2.958891 7.142857 7.692308 4.237288 2.222222
#> Ford Pantera L 2.538693 0.000000 7.692308 4.237288 4.444444
#> Ferrari Dino 2.713775 0.000000 7.692308 4.237288 6.666667
#> Maserati Bora 2.556201 0.000000 7.692308 4.237288 8.888889
#> Volvo 142E 3.256531 7.142857 7.692308 3.389831 2.222222
fsum(m, g)
#> mpg cyl disp hp drat wt qsec vs am gear carb
#> 4.0.1 26.0 4 120.3 91 4.43 2.140 16.70 0 1 5 2
#> 4.1.0 68.7 12 407.6 254 11.31 8.805 62.91 3 0 11 5
#> 4.1.1 198.6 28 628.6 564 29.04 14.198 130.90 7 7 29 10
#> 6.0.1 61.7 18 465.0 395 11.42 8.265 48.98 0 3 13 14
#> 6.1.0 76.5 24 818.2 461 13.68 13.555 76.86 4 0 14 10
#> 8.0.0 180.6 96 4291.4 2330 37.45 49.249 205.71 0 0 36 37
#> 8.0.1 30.8 16 652.0 599 7.76 6.740 29.10 0 2 10 12
fsum(m, g, TRA = "%")
#> mpg cyl disp hp drat
#> Mazda RX4 34.035656 33.333333 34.408602 27.848101 34.150613
#> Mazda RX4 Wag 34.035656 33.333333 34.408602 27.848101 34.150613
#> Datsun 710 11.480363 14.285714 17.181037 16.489362 13.257576
#> Hornet 4 Drive 27.973856 25.000000 31.532633 23.861171 22.514620
#> Hornet Sportabout 10.354374 8.333333 8.388871 7.510730 8.411215
#> Valiant 23.660131 25.000000 27.499389 22.776573 20.175439
#> Duster 360 7.918051 8.333333 8.388871 10.515021 8.571429
#> Merc 240D 35.516739 33.333333 35.991168 24.409449 32.625995
#> Merc 230 33.187773 33.333333 34.543670 37.401575 34.659593
#> Merc 280 25.098039 25.000000 20.483989 26.681128 28.654971
#> Merc 280C 23.267974 25.000000 20.483989 26.681128 28.654971
#> Merc 450SE 9.080842 8.333333 6.426807 7.725322 8.197597
#> Merc 450SL 9.579181 8.333333 6.426807 7.725322 8.197597
#> Merc 450SLC 8.416390 8.333333 6.426807 7.725322 8.197597
#> Cadillac Fleetwood 5.758583 8.333333 10.998742 8.798283 7.823765
#> Lincoln Continental 5.758583 8.333333 10.719113 9.227468 8.010681
#> Chrysler Imperial 8.139535 8.333333 10.253064 9.871245 8.624833
#> Fiat 128 16.314199 14.285714 12.519885 11.702128 14.049587
#> Honda Civic 15.307150 14.285714 12.042634 9.219858 16.976584
#> Toyota Corolla 17.069486 14.285714 11.310850 11.524823 14.531680
#> Toyota Corona 31.295488 33.333333 29.465162 38.188976 32.714412
#> Dodge Challenger 8.582503 8.333333 7.410169 6.437768 7.369826
#> AMC Javelin 8.416390 8.333333 7.083935 6.437768 8.411215
#> Camaro Z28 7.364341 8.333333 8.155847 10.515021 9.959947
#> Pontiac Firebird 10.631229 8.333333 9.320968 7.510730 8.224299
#> Fiat X1-9 13.746224 14.285714 12.567611 11.702128 14.049587
#> Porsche 914-2 100.000000 100.000000 100.000000 100.000000 100.000000
#> Lotus Europa 15.307150 14.285714 15.128858 20.035461 12.982094
#> Ford Pantera L 51.298701 50.000000 53.834356 44.073456 54.381443
#> Ferrari Dino 31.928687 33.333333 31.182796 44.303797 31.698774
#> Maserati Bora 48.701299 50.000000 46.165644 55.926544 45.618557
#> Volvo 142E 10.775428 14.285714 19.249125 19.326241 14.152893
#> wt qsec vs am gear
#> Mazda RX4 31.699940 33.605553 NaN 33.33333 30.769231
#> Mazda RX4 Wag 34.785239 34.748877 NaN 33.33333 30.769231
#> Datsun 710 16.340330 14.216960 14.28571 14.28571 13.793103
#> Hornet 4 Drive 23.718185 25.292740 25.00000 NaN 21.428571
#> Hornet Sportabout 6.984913 8.273783 NaN NaN 8.333333
#> Valiant 25.525636 26.307572 25.00000 NaN 21.428571
#> Duster 360 7.248878 7.700160 NaN NaN 8.333333
#> Merc 240D 36.229415 31.791448 33.33333 NaN 36.363636
#> Merc 230 35.775128 36.401208 33.33333 NaN 36.363636
#> Merc 280 25.378089 23.809524 25.00000 NaN 28.571429
#> Merc 280C 25.378089 24.590164 25.00000 NaN 28.571429
#> Merc 450SE 8.264127 8.458510 NaN NaN 8.333333
#> Merc 450SL 7.573758 8.555734 NaN NaN 8.333333
#> Merc 450SLC 7.675283 8.750182 NaN NaN 8.333333
#> Cadillac Fleetwood 10.660115 8.740460 NaN NaN 8.333333
#> Lincoln Continental 11.013422 8.662680 NaN NaN 8.333333
#> Chrysler Imperial 10.853012 8.468232 NaN NaN 8.333333
#> Fiat 128 15.495140 14.873950 14.28571 14.28571 13.793103
#> Honda Civic 11.374842 14.148205 14.28571 14.28571 13.793103
#> Toyota Corolla 12.924356 15.202445 14.28571 14.28571 13.793103
#> Toyota Corona 27.995457 31.807344 33.33333 NaN 27.272727
#> Dodge Challenger 7.147353 8.200865 NaN NaN 8.333333
#> AMC Javelin 6.974761 8.409897 NaN NaN 8.333333
#> Camaro Z28 7.797113 7.491128 NaN NaN 8.333333
#> Pontiac Firebird 7.807265 8.288367 NaN NaN 8.333333
#> Fiat X1-9 13.628680 14.438503 14.28571 14.28571 13.793103
#> Porsche 914-2 100.000000 100.000000 NaN 100.00000 100.000000
#> Lotus Europa 10.656430 12.910619 14.28571 14.28571 17.241379
#> Ford Pantera L 47.032641 49.828179 NaN 50.00000 50.000000
#> Ferrari Dino 33.514822 31.645570 NaN 33.33333 38.461538
#> Maserati Bora 52.967359 50.171821 NaN 50.00000 50.000000
#> Volvo 142E 19.580223 14.209320 14.28571 14.28571 13.793103
#> carb
#> Mazda RX4 28.571429
#> Mazda RX4 Wag 28.571429
#> Datsun 710 10.000000
#> Hornet 4 Drive 10.000000
#> Hornet Sportabout 5.405405
#> Valiant 10.000000
#> Duster 360 10.810811
#> Merc 240D 40.000000
#> Merc 230 40.000000
#> Merc 280 40.000000
#> Merc 280C 40.000000
#> Merc 450SE 8.108108
#> Merc 450SL 8.108108
#> Merc 450SLC 8.108108
#> Cadillac Fleetwood 10.810811
#> Lincoln Continental 10.810811
#> Chrysler Imperial 10.810811
#> Fiat 128 10.000000
#> Honda Civic 20.000000
#> Toyota Corolla 10.000000
#> Toyota Corona 20.000000
#> Dodge Challenger 5.405405
#> AMC Javelin 5.405405
#> Camaro Z28 10.810811
#> Pontiac Firebird 5.405405
#> Fiat X1-9 10.000000
#> Porsche 914-2 100.000000
#> Lotus Europa 20.000000
#> Ford Pantera L 33.333333
#> Ferrari Dino 42.857143
#> Maserati Bora 66.666667
#> Volvo 142E 20.000000
## method for grouped data frames - created with dplyr::group_by or fgroup_by
mtcars |> fgroup_by(cyl,vs,am) |> fsum(hp) # Weighted grouped sum (total)
#> cyl vs am sum.hp mpg disp drat wt qsec gear carb
#> 1 4 0 1 91 2366.0 10947.3 403.13 194.740 1519.70 455 182
#> 2 4 1 0 254 5764.3 34121.1 960.08 736.135 5356.47 919 411
#> 3 4 1 1 564 15612.7 52945.4 2301.27 1165.914 10456.79 2369 838
#> 4 6 0 1 395 8067.5 60575.0 1491.50 1089.200 6395.30 1755 1930
#> 5 6 1 0 461 8805.5 93234.6 1592.92 1563.190 8837.10 1629 1199
#> 6 8 0 0 2330 34550.5 846042.0 7323.40 9689.735 39807.80 6990 7480
#> 7 8 0 1 599 9196.2 193499.0 2299.98 2032.830 8719.00 2995 3736
mtcars |> fgroup_by(cyl,vs,am) |> fsum(TRA = "%")
#> cyl vs am mpg disp hp drat
#> Mazda RX4 6 0 1 34.035656 34.408602 27.848101 34.150613
#> Mazda RX4 Wag 6 0 1 34.035656 34.408602 27.848101 34.150613
#> Datsun 710 4 1 1 11.480363 17.181037 16.489362 13.257576
#> Hornet 4 Drive 6 1 0 27.973856 31.532633 23.861171 22.514620
#> Hornet Sportabout 8 0 0 10.354374 8.388871 7.510730 8.411215
#> Valiant 6 1 0 23.660131 27.499389 22.776573 20.175439
#> Duster 360 8 0 0 7.918051 8.388871 10.515021 8.571429
#> Merc 240D 4 1 0 35.516739 35.991168 24.409449 32.625995
#> Merc 230 4 1 0 33.187773 34.543670 37.401575 34.659593
#> Merc 280 6 1 0 25.098039 20.483989 26.681128 28.654971
#> Merc 280C 6 1 0 23.267974 20.483989 26.681128 28.654971
#> Merc 450SE 8 0 0 9.080842 6.426807 7.725322 8.197597
#> Merc 450SL 8 0 0 9.579181 6.426807 7.725322 8.197597
#> Merc 450SLC 8 0 0 8.416390 6.426807 7.725322 8.197597
#> Cadillac Fleetwood 8 0 0 5.758583 10.998742 8.798283 7.823765
#> Lincoln Continental 8 0 0 5.758583 10.719113 9.227468 8.010681
#> Chrysler Imperial 8 0 0 8.139535 10.253064 9.871245 8.624833
#> Fiat 128 4 1 1 16.314199 12.519885 11.702128 14.049587
#> Honda Civic 4 1 1 15.307150 12.042634 9.219858 16.976584
#> Toyota Corolla 4 1 1 17.069486 11.310850 11.524823 14.531680
#> Toyota Corona 4 1 0 31.295488 29.465162 38.188976 32.714412
#> Dodge Challenger 8 0 0 8.582503 7.410169 6.437768 7.369826
#> AMC Javelin 8 0 0 8.416390 7.083935 6.437768 8.411215
#> Camaro Z28 8 0 0 7.364341 8.155847 10.515021 9.959947
#> Pontiac Firebird 8 0 0 10.631229 9.320968 7.510730 8.224299
#> Fiat X1-9 4 1 1 13.746224 12.567611 11.702128 14.049587
#> Porsche 914-2 4 0 1 100.000000 100.000000 100.000000 100.000000
#> Lotus Europa 4 1 1 15.307150 15.128858 20.035461 12.982094
#> Ford Pantera L 8 0 1 51.298701 53.834356 44.073456 54.381443
#> Ferrari Dino 6 0 1 31.928687 31.182796 44.303797 31.698774
#> Maserati Bora 8 0 1 48.701299 46.165644 55.926544 45.618557
#> Volvo 142E 4 1 1 10.775428 19.249125 19.326241 14.152893
#> wt qsec gear carb
#> Mazda RX4 31.699940 33.605553 30.769231 28.571429
#> Mazda RX4 Wag 34.785239 34.748877 30.769231 28.571429
#> Datsun 710 16.340330 14.216960 13.793103 10.000000
#> Hornet 4 Drive 23.718185 25.292740 21.428571 10.000000
#> Hornet Sportabout 6.984913 8.273783 8.333333 5.405405
#> Valiant 25.525636 26.307572 21.428571 10.000000
#> Duster 360 7.248878 7.700160 8.333333 10.810811
#> Merc 240D 36.229415 31.791448 36.363636 40.000000
#> Merc 230 35.775128 36.401208 36.363636 40.000000
#> Merc 280 25.378089 23.809524 28.571429 40.000000
#> Merc 280C 25.378089 24.590164 28.571429 40.000000
#> Merc 450SE 8.264127 8.458510 8.333333 8.108108
#> Merc 450SL 7.573758 8.555734 8.333333 8.108108
#> Merc 450SLC 7.675283 8.750182 8.333333 8.108108
#> Cadillac Fleetwood 10.660115 8.740460 8.333333 10.810811
#> Lincoln Continental 11.013422 8.662680 8.333333 10.810811
#> Chrysler Imperial 10.853012 8.468232 8.333333 10.810811
#> Fiat 128 15.495140 14.873950 13.793103 10.000000
#> Honda Civic 11.374842 14.148205 13.793103 20.000000
#> Toyota Corolla 12.924356 15.202445 13.793103 10.000000
#> Toyota Corona 27.995457 31.807344 27.272727 20.000000
#> Dodge Challenger 7.147353 8.200865 8.333333 5.405405
#> AMC Javelin 6.974761 8.409897 8.333333 5.405405
#> Camaro Z28 7.797113 7.491128 8.333333 10.810811
#> Pontiac Firebird 7.807265 8.288367 8.333333 5.405405
#> Fiat X1-9 13.628680 14.438503 13.793103 10.000000
#> Porsche 914-2 100.000000 100.000000 100.000000 100.000000
#> Lotus Europa 10.656430 12.910619 17.241379 20.000000
#> Ford Pantera L 47.032641 49.828179 50.000000 33.333333
#> Ferrari Dino 33.514822 31.645570 38.461538 42.857143
#> Maserati Bora 52.967359 50.171821 50.000000 66.666667
#> Volvo 142E 19.580223 14.209320 13.793103 20.000000
#>
#> Grouped by: cyl, vs, am [7 | 5 (3.8) 1-12]
mtcars |> fgroup_by(cyl,vs,am) |> fselect(mpg) |> fsum()
#> cyl vs am mpg
#> 1 4 0 1 26.0
#> 2 4 1 0 68.7
#> 3 4 1 1 198.6
#> 4 6 0 1 61.7
#> 5 6 1 0 76.5
#> 6 8 0 0 180.6
#> 7 8 0 1 30.8
## This compares fsum with data.table and base::rowsum
# Starting with small data
library(data.table)
#>
#> Attaching package: ‘data.table’
#> The following objects are masked from ‘package:dplyr’:
#>
#> between, first, last
#> The following object is masked from ‘package:collapse’:
#>
#> fdroplevels
opts <- set_collapse(nthreads = getDTthreads())
mtcDT <- qDT(mtcars)
f <- qF(mtcars$cyl)
library(microbenchmark)
microbenchmark(mtcDT[, lapply(.SD, sum), by = f],
rowsum(mtcDT, f, reorder = FALSE),
fsum(mtcDT, f, na.rm = FALSE), unit = "relative")
#> Unit: relative
#> expr min lq mean median
#> mtcDT[, lapply(.SD, sum), by = f] 107.120960 83.273878 61.524921 63.828363
#> rowsum(mtcDT, f, reorder = FALSE) 3.754335 3.434847 3.224078 3.311696
#> fsum(mtcDT, f, na.rm = FALSE) 1.000000 1.000000 1.000000 1.000000
#> uq max neval
#> 49.027905 44.517159 100
#> 3.081299 3.651943 100
#> 1.000000 1.000000 100
# Now larger data
tdata <- qDT(replicate(100, rnorm(1e5), simplify = FALSE)) # 100 columns with 100.000 obs
f <- qF(sample.int(1e4, 1e5, TRUE)) # A factor with 10.000 groups
microbenchmark(tdata[, lapply(.SD, sum), by = f],
rowsum(tdata, f, reorder = FALSE),
fsum(tdata, f, na.rm = FALSE), unit = "relative")
#> Unit: relative
#> expr min lq mean median uq
#> tdata[, lapply(.SD, sum), by = f] 3.444597 3.806938 3.363075 3.866758 3.397583
#> rowsum(tdata, f, reorder = FALSE) 2.884303 2.816264 2.613085 2.554351 2.455199
#> fsum(tdata, f, na.rm = FALSE) 1.000000 1.000000 1.000000 1.000000 1.000000
#> max neval
#> 2.951398 100
#> 10.560802 100
#> 1.000000 100
# Reset options
set_collapse(opts)