Split up a dataframe by number of rows

Question

I have a dataframe made up of 400\'000 rows and about 50 columns. As this dataframe is so large, it is too computationally taxing to work with. I would like to split this d

Answer 1

Make your own grouping variable.

d <- split(my_data_frame,rep(1:400,each=1000))

You should also consider the ddply function from the plyr package, or the group_by() function from dplyr.

edited for brevity, after Hadley's comments.

If you don't know how many rows are in the data frame, or if the data frame might be an unequal length of your desired chunk size, you can do

chunk <- 1000
n <- nrow(my_data_frame)
r  <- rep(1:ceiling(n/chunk),each=chunk)[1:n]
d <- split(my_data_frame,r)

You could also use

r <- ggplot2::cut_width(1:n,chunk,boundary=0)

For future readers, methods based on the dplyr and data.table packages will probably be (much) faster for doing group-wise operations on data frames, e.g. something like

(my_data_frame 
   %>% mutate(index=rep(1:ngrps,each=full_number)[seq(.data)])
   %>% group_by(index)
   %>% [mutate, summarise, do()] ...
)

There are also many answers here