Google has recently open sourced a library that is designed to be a fast query engine for column-oriented database backends. The Supersonic Query Engine is their latest open source offering.
Last year, Google has open sourced another library for database developers called LevelDB, a non-relational database that could be embedded in any application that needed to store non-relational data. The Supersonic Query Engine isn’t a database in itself, nor is it non-relations. It is an engine that can be used by those developing their own column oriented database management systems.
Column Oriented databases are relational databases that optimise access for columns, rather than rows. For systems where it is more likely that an aggregate of data contained across many rows of the same column will be required more than data on all columns of a row. Imagine an analytics solution that records the time spent by a user on different parts of your software, the screen size of their computer etc. In such a case you are likely more interested in the aggregate data of one metric (what is the average / median screen size, or how long on average do people spend using a particular feature) rather than all the data on one single user. The way data is stored, and queried can be optimised in this scenario by using a column-oriented database.
As with most things Google, their claim is that the focus of this library is speed. The library uses “cache-aware algorithms, SIMD instructions and vectorised execution” which allows it to “exploit the capabilities and resources of modern, hyper pipelined CPUs”.
Here are the functionalities it claims to provide:
- speed
- cache consciousness
- instruction pipelining
- SIMD use
- efficient memory allocation
- custom data structures
- reliability
- failure handling
- high test coverage (under development)
- robustness
- support for standard columnar database operations
- a wide range of specialised expressions (including many math, string and date manipulation functionalities)
For more information about the SuperSonic Query Engine, you can head over to its project page on Google Code.
