How to export a MySQL database to JSON?

Question

I am interested in exporting a subset of values from a MySQL database into a JSON-formatted file on disk.

I found a link that talks about a possible way to do this:

Answer 1

The simplest solution I found was combination of mysql and jq commands with JSON_OBJECT query. Actually jq is not required if JSON Lines format is good enough.

Dump from remote server to local file example.

ssh remote_server \
    "mysql \
        --silent \
        --raw \
        --host "" --port 3306 \
        --user "" --password="" \
        table \
        -e \"SELECT JSON_OBJECT('key', value) FROM table\" |
    jq --slurp --ascii-output ." \
> dump.json

books table example

+----+-------+
| id | book  | 
+----+-------+
| 1  | book1 | 
| 2  | book2 | 
| 3  | book3 | 
+----+-------+

Query would looks like:

SELECT JSON_OBJECT('id', id, 'book', book) FROM books;

dump.json output

[
    {
        "id": "1",
        "book": "book1"
    },
    {
        "id": "2",
        "book": "book2"
    },
    {
        "id": "3",
        "book": "book3"
    }
]

Answer 2

You can export any SQL query into JSON directly from PHPMyAdmin

Answer 3

Another solution, if you are using Ruby, is to write a connection script to the database with ActiveRecord. You will need to install it first

gem install activerecord

# ruby ./export-mysql.rb
require 'rubygems'
require 'active_record'

ActiveRecord::Base.establish_connection(
  :adapter => "mysql",
  :database => "database_name",
  :username => "root",
  :password => "",
  :host => "localhost"
)

class Event < ActiveRecord::Base; end
class Person < ActiveRecord::Base; end

File.open("events.json", "w") { |f| f.write Event.all.to_json }
File.open("people.json", "w") { |f| f.write Person.all.to_json }

You can also add methods to the ActiveRecord classes if you want to manipulate data first or include or exclude certain columns.

Person.all.to_json(:only => [ :id, :name ])

With ActiveRecord you are not limited to JSON. You can just as easily export as XML or YAML

Person.all.to_xml
Person.all.to_yaml

You are not limited to MySQL. Any database supported by ActiveRecord (Postgres, SQLite3, Oracle... etc).

And it's worth mentioning you could open another handle to a database

require 'active_record'

ActiveRecord::Base.configurations["mysql"] = {
  :adapter  => 'mysql',
  :database => 'database_name',
  :username => 'root',
  :password => '',
  :host     => 'localhost'
}


ActiveRecord::Base.configurations["sqlite3"] = {
  :adapter  => 'sqlite3',
  :database => 'db/development.sqlite3'
}

class PersonMySQL < ActiveRecord::Base
  establish_connection "mysql"
end

class PersonSQLite < ActiveRecord::Base
  establish_connection "sqlite3"
end


PersonMySQL.all.each do |person|
    PersonSQLite.create(person.attributes.except("id"))
end

Here is a quick little blog post about it http://www.seanbehan.com/how-to-export-a-mysql-database-to-json-csv-and-xml-with-ruby-and-the-activerecord-gem

Answer 4

as described in the link, it only returns (as I mentioned) 15 results. (fwiw, I checked these results against the 4000 I'm supposed to get, and these 15 are the same as the first 15 of the 4000)

That's because mysql restricts the length of the data returned by group concat to the value set in @@group_concat_max_len as soon as it gets to the that amount it truncates and returns what it's gotten so far.

You can set @@group_concat_max_len in a few different ways. reference The mysql documentation...

Answer 5

If you have Ruby, you can install the mysql2xxxx gem (not the mysql2json gem, which is a different gem):

$ gem install mysql2xxxx

and then run the command

$ mysql2json --user=root --password=password --database=database_name --execute "select * from mytable" >mytable.json

The gem also provides mysql2csv and mysql2xml. It's not as fast as mysqldump, but also doesn't suffer from some of mysqldump's weirdnesses (like only being able to dump CSV from the same computer as the MySQL server itself)

Answer 6

For anyone that wants to do this using Python, and be able to export all tables without predefinining field names etc, I wrote a short script for this the other day, hope someone finds it useful:

from contextlib import closing
from datetime import datetime
import json
import MySQLdb
DB_NAME = 'x'
DB_USER = 'y'
DB_PASS = 'z'

def get_tables(cursor):
    cursor.execute('SHOW tables')
    return [r[0] for r in cursor.fetchall()] 

def get_rows_as_dicts(cursor, table):
    cursor.execute('select * from {}'.format(table))
    columns = [d[0] for d in cursor.description]
    return [dict(zip(columns, row)) for row in cursor.fetchall()]

def dump_date(thing):
    if isinstance(thing, datetime):
        return thing.isoformat()
    return str(thing)


with closing(MySQLdb.connect(user=DB_USER, passwd=DB_PASS, db=DB_NAME)) as conn, closing(conn.cursor()) as cursor:
    dump = {}
    for table in get_tables(cursor):
        dump[table] = get_rows_as_dicts(cursor, table)
    print(json.dumps(dump, default=dump_date, indent=2))