vmx

2010-05-21 22:35

I did it! I'll speak at the FOSS4G Conference 2010 (Free and Open Source Software for Geospatial Conference), 6th–9th September in Barcelona about “GeoCouch: A spatial index for CouchDB”. As soon as the abstract is available online I'll link to it. Hope to see you there!

Categories: en, GeoCouch, CouchDB, Erlang, geo

2010-05-14 22:35

The upcoming OpenSearch Geo specification will add support for querying with WKT (Well-Known Text). As I plan to support this specification in GeoCouch, I was in need of a WKT parser written in Erlang. I tried several ways to write this parser, but I ended up with writing it manually, based on the ideas of the fabulous MochiWeb JSON2 Parser

The parser is meant for fast parsing, it is non-validating. This means that it parses only valid WKT and all other strings that seem to be valid, but are not. The grammar is simplified to (in EBNF as used for the XML spec):

wkt ::= item | string  '(' space* item (comma item)* ')'
item ::= string (geom | list | nested_list | item | 'EMPTY')
nested_list ::= space* '(' list (comma list)* ')' | '(' nested_list+ ')'
list ::= '(' geom (comma geom)* ')'
geom ::= space* '(' coord (comma coord)* ')'
coord ::= space* number (space+ number)*
number ::= integer | float 
integer ::=  ('-' | '+')? [0-9]+
float ::= ('-' | '+')? [0-9]+ '.' [0-9]+ exponent?
exponent = 'E' ('-' | '+')? [0-9]+
string ::= [a-zA-Z]+ (space* [a-zA-Z])*
space :== #x20
comma :== ',' space*

I hope I got the grammar right, leave a comment if not. This means also strings like this(is(10 20), a test EMPTY) would be parsed to:

{this,[{is,[{10,20}]},{'a test',[]}]}

A validating parser would be much slower as it would also need to perform checks on the geometry, e.g. for polygons whether interiors are really within the exterior ring or not.

The general rule is, a list of coordinates is transformed to a tuple, a list of coordinates to a list. The geometry name will be an atom. Here's an example for a polygon:

wkt:parse("POLYGON ((102 103, 204 205, 306 107, 102 103),
                    (12 13, 24 25, 36 17, 12 13),
                    (62 63, 74 75, 86 67, 62 63))").
{polygon,[[{102,103},{204,205},{306,107},{102,103}],
          [{12,13},{24,25},{36,17},{12,13}],
          [{62,63},{74,75},{86,67},{62,63}]]}

In case you're getting excited now, the source is available at Github, realeased under the MIT License.

If someone plans to write a validating WKT parser for Erlang (please let me know), I propose using neotoma it's really a nice "packrat parser-generator for Erlang for Parsing Expression Grammars (PEGs)".

Categories: en, GeoCouch, Erlang, geo

2010-05-03 22:35

Update: This blog entry is outdated and kepts for historical reasons. Please do always check for newer blog posts. The up to date information on how to install and use GeoCouch can be found in its README.

An idea has become reality. Exactly two years after the blog post with the initial vision, a new version of GeoCouch is finished. It's a huge step forward. The first time the dependencies were narrowed down to CouchDB itself. No Python, no SpatiaLite any longer, it's pure Erlang. GeoCouch is tightly integrated with CouchDB, so you'll get all the nice features you love about CouchDB.

Current implementation

Thanks to the feedback after the FOSS4G 2009 and "GeoCouch: The future" blog entry" it was clear that people prefer a simple, yet powerful and tightly integrated approach, rather than having to many external dependencies (which was a showstopper for quite a few people).

I implemented an R-tree (I call it vtree as the implementation is subject to change a lot) from scratch. The reason why I haven't used the already existing R-Tree implementation available at Github is that I needed something to learn Erlang, it doesn't contain test or examples and that it is always a good idea to implement a data structure yourself to understand the details/problems. My implementation is far from being perfect but works good enough for now. The vtree is implemented as an append-only data structure just as CouchDB's B-trees are. Currently it doesn't support bulk insertion.

If you want to know details on how to create your own indexer, have a look at my Indexer tutorial.

Feature set

Following the "Release early, release often" philosophy currently only points can be inserted, the only supported query is a bounding box search. Though other geometries should follow soon.

Using GeoCouch

GeoCouch is now hosted at Github. Giving GeoCouch a go is easy:

git clone http://github.com/vmx/couchdb.git
cd couchdb
./bootstrap
./configure
make dev
./utils/run

To try the spatial features when it's up and running is easy as well. Just add a spatial property and a named function to your Design Document as you would to for show or list functions:

function(doc) {
    if (doc.loc) {
        emit(doc._id, {
            type: "Point",
            coordinates: [doc.loc[0], doc.loc[1]]
        });
    }
};

All you need to do is emitting GeoJSON as the value (Remember that point is the only supported geometry at the moment), the key is currently ignored.

curl -X PUT http://127.0.0.1:5984/places
curl -X PUT -d '{"spatial":{"points":"function(doc) {\n    if (doc.loc) {\n        emit(doc._id, {\n            type: \"Point\",\n            coordinates: [doc.loc[0], doc.loc[1]]\n        });\n    }};"}}' http://127.0.0.1:5984/places/_design/main

Before a bounding box query can return anything, you need to insert Documents that contain a location.

curl -X PUT -d '{"loc": [-122.270833, 37.804444]}' http://127.0.0.1:5984/places/oakland
curl -X PUT -d '{"loc": [10.898333, 48.371667]}' http://127.0.0.1:5984/places/augsburg

And finally you can make a bounding box request:

curl -X GET 'http://localhost:5984/places/_design/main/_spatial/points/%5B0,0,180,90%5D'

This one should return only augsburg:

{"query1":[{"id":"augsburg","loc":[10.898333,48.371667]}]}

Next steps

The development of GeoCouch was quite slow in the past, but it gets up to speed as my diploma thesis (comparable to a master's thesis) will be about GeoCouch. Additionally Couchio kindly supports the development.

The next steps are (in no particular order):

• Better R-tree (better splitting algorithm, bulk operations)
• Supporting more geometries
• Polygon search
• Improving CouchDB's plugin capabilities

Thanks

I'd like to thank all the people that kept me motivated over the past two years with their tremendous feedback. Special thanks go to Jan Lehnardt for getting me onto the Couch, Cameron Shorter for introducing me into the geospatial open source business and all people from Couchio for the great two weeks in Oakland.

Categories: en, CouchDB, Python, Erlang, geo