JSONStore error when push a collection with documents with files

Question

I have a JSONStore for a list of customers, the user can add documents to those customers using the app.

The list of cusotmers and its data (also attached documents) must be synch with the backend.

When I add a ppt document 774KB (that is the size in binary, I transform it to base64) to the json store and execute the push() it fails with the error:

E/CursorWindow(32705): need to grow: mSize = 1048576, size = 1056310, freeSpace() = 1048464, numRows = 1
E/CursorWindow(32705): Attempting to grow window beyond max size (1048576)
E/Cursor(32705): Failed allocating 1056310 bytes for text/blob at 0,1
D/Cursor(32705): finish_program_and_get_row_count row 0
E/CursorWindow(32705): need to grow: mSize = 1048576, size = 1056310, freeSpace() = 1048464, numRows = 1
E/CursorWindow(32705): Attempting to grow window beyond max size (1048576)
E/Cursor(32705): Failed allocating 1056310 bytes for text/blob at 0,1
D/Cursor(32705): finish_program_and_get_row_count row 0
E/CursorWindow(32705): Bad request for field slot 0,0. numRows = 0, numColumns = 4
E/jsonstore-core(32705): error while dispatching action "allDirty"
E/jsonstore-core(32705): java.lang.IllegalStateException: get field slot from row 0 col 0 failed
E/jsonstore-core(32705):    at net.sqlcipher.CursorWindow.getLong_native(Native Method)
E/jsonstore-core(32705):    at net.sqlcipher.CursorWindow.getLong(CursorWindow.java:381)
E/jsonstore-core(32705):    at net.sqlcipher.AbstractWindowedCursor.getLong(AbstractWindowedCursor.java:110)
E/jsonstore-core(32705):    at net.sqlcipher.AbstractCursor.moveToPosition(AbstractCursor.java:195)
E/jsonstore-core(32705):    at net.sqlcipher.AbstractCursor.moveToNext(AbstractCursor.java:257)
E/jsonstore-core(32705):    at android.database.CursorWrapper.moveToNext(CursorWrapper.java:166)
E/jsonstore-core(32705):    at com.worklight.androidgap.plugin.storage.AllDirtyActionDispatcher$AllDirtyAction.performAction(AllDirtyActionDispatcher.java:148)
E/jsonstore-core(32705):    at com.worklight.androidgap.plugin.storage.AllDirtyActionDispatcher$AllDirtyAction.performAction(AllDirtyActionDispatcher.java:119)
E/jsonstore-core(32705):    at com.worklight.androidgap.plugin.storage.DatabaseActionDispatcher$Context.performReadableDatabaseAction(DatabaseActionDispatcher.java:141)
E/jsonstore-core(32705):    at com.worklight.androidgap.plugin.storage.AllDirtyActionDispatcher.dispatch(AllDirtyActionDispatcher.java:64)
E/jsonstore-core(32705):    at com.worklight.androidgap.plugin.storage.DatabaseActionDispatcher.dispatch(DatabaseActionDispatcher.java:56)
E/jsonstore-core(32705):    at com.worklight.androidgap.plugin.storage.BaseActionDispatcher.dispatch(BaseActionDispatcher.java:87)
E/jsonstore-core(32705):    at com.worklight.androidgap.plugin.storage.DispatchingPlugin$ActionDispatcherRunnable.run(DispatchingPlugin.java:113)
E/jsonstore-core(32705):    at com.worklight.androidgap.plugin.storage.DispatchingPlugin$SerialExecutor$1.run(DispatchingPlugin.java:147)
E/jsonstore-core(32705):    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1076)
E/jsonstore-core(32705):    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:569)
E/jsonstore-core(32705):    at java.lang.Thread.run(Thread.java:856)
E/myApp (32705): [wl.jsonstore] {"src":"push","err":8,"msg":"FAILED_TO_GET_UNPUSHED_DOCUMENTS_FROM_DB","col":"Documentos","usr":"jsonstore","doc":{},"res":{}}

I can add the document, the error is executing the push() method.

All the information I have seen in stackoverflow and infocenter about JSONStore is that there is no size limit. I have more than enough free space in my mobile.

Any idea?

Thank you.

Answer 1

"Don't store blobs [...] in your database. Store identifiers in your database and put the blobs as files onto the storage." - Source

Cordova has a File API you can use.

Here's a quick example:

//Code to write customer-1-file1.ppt and customer-1-file2.ppt to disk.
//See Cordova's File API.

//Pseudocode to get the blobsCollection and add metadata to be able to find the files.
//This would be inside the success callback for writing the files.
WL.JSONStore.get('blobsCollection')
  .add([{fileName: 'customer-1-file1.ppt'}, {fileName: 'customer-1-file2.ppt'}]);

//Some time has passed...

//Pseudocode to get %customer-1% from disk
//% are wildcards characters and match any string
WL.JSONStore.get('blobsCollection')
  .find({fileName: 'customer-1'}, {exact: false})
  .then(function (listOfFiles) {
    //listOfFiles => [{_id: 1, json: { fileName: 'customer-1-file1.ppt'} }, 
    //                {_id: 2, json: { {fileName: 'customer-1-file2.ppt'} }]

    var firstFile = listOfFiles[0].json.fileName;

    //Code to read firstFile. See Cordova's File API.
  });

JSONStore is backed by SQLite (well, technically SQLCipher which is a wrapper for SQLite that adds data encryption). Read Internal Versus External BLOBs in SQLite. The takeaway is "For BLOBs smaller than 100KB, reads are faster when the BLOBs are stored directly in the database file. For BLOBs larger than 100KB, reads from a separate file are faster".

If you need to store blobs bigger than the default SQLite Cursor size (1048576 bytes), I suggest a feature request here.

I'll make sure this is mentioned in the documentation.

Note that there's a getPushRequired API you can use to get the list of documents that the push API will try to send to the Worklight Adapter. You will need to send file changes yourself to the Worklight Adapter using WL.Client.invokeProcedure, or directly to a backend using something like jQuery.ajax.