http://blog.tfd.co.uk/2012/10/11/sakai-cle-elasticsearch/ Open Source Open Thought Thu, 24 Jan 2013 05:22:45 +0000 hourly 1 http://wordpress.com/ http://blog.tfd.co.uk/2012/10/11/sakai-cle-elasticsearch/#comment-1763 Thu, 11 Oct 2012 20:16:35 +0000 http://blog.tfd.co.uk/?p=748#comment-1763 Yes, AuthZ is always a problem. Fortunately we have a scheme that started in CLE and evolved in OAE. In the original scheme in CLE we used the site realm as the metadata indicating a user could see the items, and that works well with minimal performance impact for most CLE use cases. In OAE it was harder, since the number of “reader” tokens owned by a user is far higher and performing 100s of AuthZ queries (via Lucene’s bitmaps) for each query is noticeable. Initially in OAE this caused problems. We did eventually develop a custom Lucene Query class that was benchmarked with a flat query profile for very large numbers of user reader tokens. (I guess I should call those “principals”). Although this was coded for Solr, I believe we can use it inside ElasticSearch. Unlike OAE which eventually (mistakenly IMHO) relied 100% of Solr for all queries, CLE will only be using search for search, and consequently the queries will not be insane. The limiting factor in the AuthZ resolution will become the cardinality of the AuthZ tokens as that drives the size of the inverted index bitmaps, which in turn is related to heap usage and performance.

]]> http://blog.tfd.co.uk/2012/10/11/sakai-cle-elasticsearch/#comment-1761 Thu, 11 Oct 2012 12:07:54 +0000 http://blog.tfd.co.uk/?p=748#comment-1761 This sounds great. I am looking forward to how this all fits together. Search is frankly one of the harder problems to solve in an LMS and this would be a nice competitive advantage for the Sakai CLE IMHO. Search in an LMS is far more difficult than search in nearly any other application because of the AUTHZ intricacies that force scaling in another dimension not typically a problem in classic large-corpus search applications.

]]>