Comments on: Indexing and Query Performance in SparseMap 1.0, indicative tests

By: Erik Froese

Erik Froese — Tue, 04 Oct 2011 13:41:08 +0000

The MongoDB driver supports native multi-term queries as well as the simple OR queries when the $orset key is found. It will also use native MongoDB sorting if the _sort key is present in the query.

Paging could be added with the skip() and limit() functions. I didn’t see sparse ever using it so there wasn’t a clear way to add it. If SMC ever supports those operations it should be pretty straightforward to support them.

By: Harry Wang

Harry Wang — Mon, 26 Sep 2011 17:50:36 +0000

Thanks a lot for these very useful information!

By: Ian

Ian — Fri, 23 Sep 2011 21:32:54 +0000

Hi Harry, At the moment I dont think the NoSQL DBs have the necessary level or query support to support OAE. OAE needs multi query terms with paging and sorting on large sets and at the moment we dont have that capability in any of the NoSQL DBs. As you can see from the posts MySQL has problems with the length of a row and query performance and indexing, all of which I hope are solvable. That leaves Derby, PostgreSQL and Oracle as potential production DBs. Derby may not sound like a good production DB, but the tests indicate its quite happy at 200K records with wide columns so it should be considered for single instance pilots. Eventually, once we have written native implementations of the query engines it should be possible to use the NoSQL DBs, however do not forget that there are some massive systems out there that use RDBMS’s (YouTubes metadata system is sharded MySQL quite simular in structure to the non indexing part of SparseMap, bits of Facebooks message system are PostgreSQL and FlockDB, twitters followers DB is sharded MySQL). Its perfectly possible to use a RDBMS in a way that works for a wide cluster.

I am working with Erik Froese to get the MongoDB driver merged. It has some nice additional features like GridFS and I think it may have better support for native querying.

HTH Ian

By: Harry Wang

Harry Wang — Fri, 23 Sep 2011 13:54:54 +0000

Hi Ian,

I have always been interested in using NoSQL database for OAE in production. I asked a similar question before and got an impression that the best DB choice for OAE production is PostgreSQL (is that still true?). Have you done any of the tests shown here using NoSQL databases? As you may know, Erik just developed a driver for MongoDB (which I have not tried yet). Is using NoSQL db for OAE production an ultimate goal for SparseMap?

Thanks,

Harry