Import MySQL dump into R (without requiring MySQL server)

Question

Packages like RMySQL and sqldf allow one to interface with local or remote database servers. I\'m creating a portable project which involves import

Answer 1

I don't think you will find a way to import a sql dump (which contains multiple tables with references) and then perform arbitrary sql queries on them within R. This would basically require the R package to run a complete database server (compatible with the one creating the dump) within R.

I would suggest exporting the tables/select statements you need as CSV from your database (see here). If you can only work from the dump and don't want to setup a server for the conversion you could use some simple regular expressions to turn the insert statements in your dump into a bunch of CSV files for the tables using a tool of your choosing like sed or awk (or even R as suggested by the other answer but that might be rather slow for this file size).

Answer 2

depending on what you want to extract from the table, here is how you can play around with the data

numLines <- R.utils::countLines("sportsdb_sample_mysql_20080303.sql")
# [1] 81266

linesInDB <- readLines("sportsdb_sample_mysql_20080303.sql",n=60)

Then you can do some regex to get tables names (after CREATE TABLE), column names (between first brackets) and VALUES (lines after CREATE TABLE and between second brackets)

Reference: Reverse engineering a mysqldump output with MySQL Workbench gives "statement starting from pointed line contains non UTF8 characters" error

---

EDIT: in response to OP's answer, if i interpret the python script correct, it is also reading it line by line, filter for INSERT INTO lines, parse as csv, then write to file. This is very similar to my original suggestion. My version below in R. If the file size is too large, it would be better to read in the file in chunks using some other R package

options(stringsAsFactors=F)
library(utils)
library(stringi)
library(plyr)

mysqldumpfile <- "sportsdb_sample_mysql_20080303.sql"

allLines <- readLines(mysqldumpfile)
insertLines <- allLines[which(stri_detect_fixed(allLines, "INSERT INTO"))]
allwords <- data.frame(stri_extract_all_words(insertLines, " "))
d_ply(allwords, .(X3), function(x) {
    #x <- split(allwords, allwords$X3)[["baseball_offensive_stats"]]
    print(x[1,3])

    #find where the header/data columns start and end
    valuesCol <- which(x[1,]=="VALUES")
    lastCols <- which(apply(x, 2, function(y) all(is.na(y))))
    datLastCol <- head(c(lastCols, ncol(x)+1), 1) - 1

    #format and prepare for write to file
    df <- data.frame(x[,(valuesCol+1):datLastCol])
    df <- setNames(df, x[1,4:(valuesCol-1)])
    #type convert before writing to file otherwise its all strings
    df[] <- apply(df, 2, type.convert)
    #write to file
    write.csv(df, paste0(x[1,3],".csv"), row.names=F)
})

Answer 3

I'll reluctantly answer my own question, using the input from +bnord and +chinsoon12 (who both contributed pieces of the puzzle).

Short answer: there is no out of the box solution. As +bnord notes, it would be preferred to fix it server-side (e.g., by exporting to CSV format with mysqldump). However, as my question indicated, I'm looking for a solution that allows me to work with the sql dump, bypassing the server.

So if we have to work with the dump, how? The hardcore, manual way is to use regular expressions to convert INSERT statements to CSV, either (1) outside R using sed and awk on the .sql text file (+bnord), or (2) inside R with grep and gsub on strings loaded with readLines (+chinsoon12).

Some good soul wrote a python script that can convert sql dumps to CSV. This requires yet another piece of (potentially non-trivial to install/maintain) software, so it's not the answer I was hoping for, but it does look like a good model in case anyone wants to reinvent the wheel in R.

For now I'll stick with my modus operandi of (on Windows) running MySQL Community Server and using WorkBench to import the dump, then talk to the local server from R. A very indirect method that is a pain in the ass because of the inscrutable access rights system of MySQL (especially annoying since it's all just there in an ASCII text file), but the only way for now, it seems. Thanks all for your input!

(If a better, more complete answer comes along I'll gladly accept that, turning this into a comment if possible.)