Best practice for storing large amounts of data with J2ME

Question

I am developing a J2ME application that has a large amount of data to store on the device (in the region of 1MB but variable). I can\'t rely on the file system so I\'m stuck

Answer 1

It is quite simple, use JSR75 (FileConnection) and remember to sign your midlet with a valid (trusted) certificate.

Answer 2

For read only I'm arriving at acceptable times (within 10s), by indexing a resource file. I've got two ~800KB CSV price list exports. Program classes and both those files compress to a 300KB JAR.

On searching I display a List and run a two Threads in the background to fill it, so the first results come pretty quickly and are viewable immediately. I first implemented a simple linear search, but that was too slow (~2min).

Then I indexed the file (which is alphabetically sorted) to find the beginnings of each letter. Now before parsing line by line, I first InputStreamReader.skip() to the desired position, based on first letter. I suspect the delay comes mostly from decompressing the resource, so splitting resources would speed it up further. I don't want to do that, not to loose the advantage of easy upgrade. CSV are exported without any preprocessing.

Answer 3

Thanks everyone for useful commments. In the end the simplest solution was to limit the amount of data being stored, implementing code that adjusts the data according to how large the store is, and fetching data from the server on demand if its not stored locally. Thats interesting that the limit is increased in OS 4.6, with any luck my code will simply adjust on its own and store more data :)

Developing a J2ME application for Blackberry without using the .cod compiler limits the use of JSR 75 some what since we can't sign the archive. As pointed out by Carlos this is a problem on any platform and I've had similar issues using the PIM part of it. The RMS seems to be incredibly slow on the Blackberry platform so I'm not sure how useful a inode/b-tree file system on top would be, unless data was cached in memory and written to RMS in a low priority background thread.

Answer 4

Under Blackberry OS 4.6 the RMS store size limit has been increased to 512Kb but this isn't much help as many devices will likely not have support for 4.6. The other option on Blackberry is the Persistent Store which has a record size limit of 64kb but no limit on the size of the store (other than the physical limits of the device).

I think Carlos and izb are right.

Answer 5

For anything past a few kilobytes you need to use either JSR 75 or a remote server. RMS records are extremely limited in size and speed, even in some higher end handsets. If you need to juggle 1MB of data in J2ME the only reliable, portable way is to store it on the network. The HttpConnection class and the GET and POST methods are always supported.

On the handsets that support JSR 75 FileConnection it may be valid alternative but without code signing it is an user experience nightmare. Almost every single API call will invoke a security prompt with no blanket permission choice. Companies that deploy apps with JSR 75 usually need half a dozen binaries for every port just to cover a small part of the possible certificates. And this is just for the manufacturer certificates; some handsets only have carrier-locked certificates.

Answer 6

RMS performance and implementation varies wildly between devices, so if platform portability is a problem, you may find that your code works well on some devices and not others. RMS is designed to store small amounts of data (High score tables, or whatever) not large amounts.

You might find that some platforms are faster with files stored in multiple record stores. Some are faster with multiple records within one store. Many are ok for storage, but become unusably slow when deleting large amounts of data from the store.

Your best bet is to use JSR-75 instead where available, and create your own file store interface that falls back to RMS if nothing better is supported.

Unfortunately when it comes to JavaME, you are often drawn into writing device-specific variants of your code.