vmx

the blllog.

GeoCouch: The future

2009-12-20 22:27

GeoCouch started as a proof of concept and was heavily rewritten for the 0.10 release. As more and more people got interested, I got feedback to see what people really want/need. And now it's time to determine the future of GeoCouch. It's your chance to shape the future. In this blog entry I'll explain my ideas for the future, but I'm more than happy to get further ideas/complains from you. So please check if my ideas match your use-cases for GeoCouch.

Stripping it down

GeoCouch needs an external spatial index, at the moment I use SpatiaLite for it, but a PostGIS backend would be easily possible. My inital idea was that it is better to use the existing power of spatial databases, rather than reinventing the wheel. I though I could use all the power they have, that I can even use them for complex analytics, but I can't. As I only store the geometries, I need to “ask” CouchDB for the attributes (no, I don't want to store attributes in my spatial index).

If I don't use the full power of the spatial databases, but only a small fraction, there might be better solution. Therefore I propose that GeoCouch will use a simple spatial index for storing the geometries, not a full blown spatial database. I haven't decided yet which one it'll be, but I really think about moving this part to Erlang (I know that quite a few people would love that move).

You will loose functionality like reprojection. The spatial index won't know anything about projections. So GeoCouch won't be projection aware anymore, but you application still can be. For example if you want to return your data in a different projection than it was stored, you do the transformation after you've queried GeoCouch.

You would also loose fancy things for geometries, like boolean operations on them. But this is something I'd call complex analytics, and not simple querying.

GeoCouch would only support three simple queries: bounding search, polygon search and radius/distance search. If the search would be within a union of polygons, let's say all countries of the European Union, you would simply make the union operation before you query GeoCouch.

Complex analytics

What I call “complex analytics” is things like: “return all apple trees that are located with a 10km range around buildings that have are over 100m high, but only in countries with a population over 50 million people” is not possible with GeoCouch as you would need the attribute values as well. Those are stored in CouchDB, so you would need to request them. What GeoCouch only supports is a simple: give me all IDs within a bounding box/polygon/radius.

Conclusion

Simple requests are needed for everyday use, thus they should be incredibly fast. Complex analytics don't necessarily need to handle thousands of requests per second, in most cases they don't even need to be processed in real-time. I'd like to see some layer build above GeoCouch, so CouchDB can even be used for analytics (which is a thing I wanted to have right from the start).

This means that GeoCouch will be mainly for high performance and massive sized projects that need some simple spatial bits, what I think the majority of users need.

If you either think you really need only those simple queries, but you want them to be fast, or you think this is wrong, that you need dynamic reprojection I can only invite you to leave a comment below or drop a mail to volker.mische@gmail.com. Thanks.

Categories: en, CouchDB, Python, geo

FOSS4G 2009: “Geodata and CouchDB” presentation is online

2009-11-17 22:27

The final wrap-up of the FOSS4G 2009, my presentation on “Geodata and CouchDB” is available online in several formats. It should also be of interest for people who are new to CouchDB as huge parts of the talk are an introduction into CouchDB.

Categories: en, CouchDB, Python, geo

Drag as long as you want

2009-11-11 22:27

It has been a very long outstanding bug (officially it was a missing feature) in OpenLayers that annoyed me from the first time I’ve been using OpenLayers. I’m talking about ticket #39: “Allow pan-dragging while outside map until mouseup”.

Normally when you drag the map in OpenLayers it will stop dragging as soon as you hit the edge of the map viewport (the div that contains the map). Whenever you have a small map, but a huge window and a loooong way to drag, it can get quite annoying, as the maximum distance you can drag at once is the size of that viewport.

But yesterday it finally happend. A patch to fix it landed in trunk. A first rough cut was made at the OpenLayers code sprint at the FOSS4G. Andreas Hocevar reviewed the code and made a more unobtrusive version of it (thanks, again).

Try these two examples to see the difference. Click on the map an drag it a long way to the right and back to the left again (you might need to zoom it a bit to see the full effect):

As it is a new feature, it isn’t enabled by default (and only available on current SVN trunk, it will be available in OpenLayers 2.9). To enable it on your map, just use the following code to add the documentDrag parameter to the DragPan control (you obviously need a recent SVN checkout).

Update (2009-11-18): It got even easier with r9805:

// Use default controls but with documentDrag enabled.
var controls = [
    new OpenLayers.Control.Navigation({documentDrag: true}),
    new OpenLayers.Control.PanZoom(),
    new OpenLayers.Control.ArgParser(),
    new OpenLayers.Control.Attribution()]
map = new OpenLayers.Map('map', {controls: controls});

For a full working version have a look at the source of the documentDrag example.

Categories: en, OpenLayers, JavaScript, geo

Australia: Getting home

2009-11-02 22:27

This one goes out to all the people that want some news from me. I’m finally back in Germany, but getting home wasn’t as easy as excpected. Now I know why you really should be at the airport 2-3 hours before your departure.

I have to admit I wasn’t too early at the airport, probably 2.5 hours prior to my departure, as everything on my previous flights went smoothly every time. But this time there was something different.

Originally I wanted to stay only for s week, but I decided to extend my stay to two weeks. This was about 7 weeks ago. Everything seemed to be alright, I got my new flight details via email (as I did for other flights as well).

Houston, we have a problem

“Sir, have you changed your booking recently?”

“Hmm, well, no, errr, no, sorry I changed it about 7 weeks ago.”

The conversation went on for a bit, with the conclusion that I neither have an e-ticket number, nor that I’m listed on the flight. There's only a booking with mine name for last week’s date. Excellent.

So I couldn’t check-in and the lady at the counter couldn’t do anything. I should contact my travel agency. Aaaaalright, it’s not a problem it’s 5:30 am in Berlin (where my travel agency is). If I can’t reach them I should go to the service point.

Dialing from Australia

I’ve always wondered, why international phone numbers start with a “+”, when I always need to call “00” instead of the plus anyway, why not appending it automatically.

I couldn’t call out from a call box. Whenever I was calling 0049 I got an error message. But luckily the madame at the service point told me that Australia has a different dial-out code (international call prefix) . It’s 0011.

So I was finally able to call the agency. Still plenty of time, 70 minutes to departure. And there was actually someone there. After another call and $20 spent, it was clear that it was the fault of another company that does the actual booking and that Qantas can’t access their stock of tickets.

A glimmer of hope

“You are already the third one today where the changes of the booking weren’t done properly”, the guy at the Qantas service point said. “We need to get you on the flight quickly, check-in closes soon”. Still 40 mins to go. So I got a new e-ticket and checked-in. That’s it for me, the Qantas people will try to find out what went wrong.

Getting through all the airport stuff seems to take longer when you are in a hurry, but I was right on time for the boarding. And obviously, I made it.

Lessons learned

  • Get your e-ticket number
  • Be early at the airport
  • If someone screws it up, you are likely to make it, though
  • Different countries have different international call prefixes

Categories: en, life

FOSS4G 2009: It was great

2009-10-25 22:27

The FOSS4G 2009 (Free and Open Source Software for Geospatial Conference) is over now, it was great. I've finally met many people that I've previously only chatted or discussed on mailing lists with.

Organisation and venue

The Sydney Convention & Exhibition Centre Darling Harbour really is an amazing venue and Arinex did a great job as well. We had good food, the technicians were keeping everything up and running, even the wireless internet didn't break down and performed well.

The Organising Committee did an excellent job (especially Mark), too. I exclude myself a bit, I was more the code monkey before the conference, rather than keeping that conference running smoothly. But because of that I had the chance to visit quite a few presentations.

Presentations

Probably the most favoured presentation was Paul Ramsey's Keynote speech. It was just incredibly insightful and entertaining (watch it at YouTube). it here.

There were to other excellent presentations as well. First the Mapping interviews with open source technologies by Chris McDowall. He is using a video projector and a Wii remote control in order to map locations people are pointing at during an interview (just watch this video to get a better idea).

And second the Visualising animal movements in ‘near’ real time by Ben Madin. It was about a project where they try to track the movements if cows in Southeast Asia. The idea is to place GSM transmitters in one of the cows' stomach to track their position. But they are facing problem like "How to get a GSM signal through 40cm of meat". Really interesting.

Geodata and CouchDB

So how did my talk go? I'm very happy with it. I haven't expected so much positive feedback and so many good conversations about CouchDB and GeoCouch afterwards and during the next days.

After show parties

After the talks it's time to socialise while having a few beers. It was again great, every single night.

One outstanding event was the Ignite Spatial on Wednesday. 10 high paced talks with 20 slides displayed 15 secs each. My favourite one was the Pie charts are evil talk by Glen Bell. Another result of the night is that I'll always think about short green skirts whenever someone is mentioning Google Wave.

The code sprint

I was code sprinting OpenLayers. It was well organised and we got some cool new stuff in. Sadly, I haven't reached my goal of fixing Ticket 39, but hopefully soon (or next year in Barcelona). But I was discussing with Roald de Wit and Andreas Hocevar the implementation details of the abstraction of the UI in OpenLayers (that idea was discussed in the Openlayers BOF).

Final words

Yes, it really was great. I hope to see you all again in Barcelona at the FOSS4G 2010.

Categories: en, geo

Benchmarking is not easy

2009-09-23 22:27

There are so many ways to have a play with CouchDB. This time I thought about using CouchDB as a TileCache storage. Sounds easy, so it was.

What is a tilecache

Everyone knows Google Maps and its small images, called tiles. Rendering those tiles for the whole world for every zoom level can be quite time consuming, therefore you can render them on demand and cache them once they are rendered. This is the business of a tilecache.

You can use the tilecache as a proxy to a remote tile server as well, that's what I did for this benchmark.

Coding

The implementation looks quite similar to the memcache one. I haven't implemented locking as I was just after something working, not a full-fledged backend.

When I finished coding, it was time to find out how it performs. That should be easy, as there's a tilecache_seeding script bundled with TileCache to fill the cache. So you fill the cache, then you switch the remote server off and test how long it takes if all requests are hits without any fails (i.e. all tiles are in your cache and don't need to be requested from a remote server).

The two contestants for the benchmark are the CouchDB backend and the one that stores the tiles directly on the filesystem.

Everyone loves numbers

We keep it simple and measure the time for seeding with time. How long will it take to request 780 tiles? The first number is the average (in seconds), the one in brackets the standard deviation.

  • Filesystem:

    real 0.35 (0.04)
    user 0.16 (0.02)
    sys  0.05 (0.01)
    
  • CouchDB:

    real 3.03 (0.18)
    user 0.96 (0.05)
    sys  0.21 (0.03)
    

Let's say CouchDB is 10 times slower that the file system based cache. Wow, CouchDB really sucks! Why would you use it as tile storage? Although you could:

  • easily store metadata with every tile, like a date when it should expire.
  • keep a history of tiles and show them as "travel through time layers" in your mapping application
  • easy replication to other servers

You just don't want such a slow hog. And those CouchDB people try to tell me that CouchDB would be fast. Pha!

Really??

You might already wonder, where the details are, the software version numbers, the specification of the system and all that stuff? These things are missing with a good reason. This benchmark just isn't right, even if I would add these details. The problem lies some layers deeper.

This benchmark is way to far away from a real-life usage. You would request much more tiles and not the same 780 ones with every run. When I was benchmarking the filesystem cache, all tiles were already in the system's cache, therefore it was that fast.

Simple solution: clear the system cache and run the tests again. Here are the results after as echo 3 > /proc/sys/vm/drop_caches

  • Filesystem:

    real 8.36 (0.71)
    user 0.29 (0.04)
    sys  0.18 (0.03)
    
  • CouchDB:

    real 6.64 (0.15)
    user 1.13 (0.07)
    sys  0.29 (0.06)
    

Wow, the CouchDB cache is faster than the filesystem cache. Too nice to be true. The reason is easy: loading the CouchDB database file, thus one file access on the disk, is way faster that 780 accesses.

Does it really matter?

Let's take the first benchmark, if CouchDB would be that much slower, but isn't it perhaps fast enough? Even with those measures (ten times slower than the filesystem cache) it would mean your cache can take 250 requests per second. Let's say a user requests 9 tiles per second it would be about 25 users at the same time. With every user staying 2 minutes on the map it would mean 18 000 users per day. Not bad.

Additionally you gain some nice things you won't have with other caches (as outlined above). And if you really need more performance you could always dump the tiles to the filesystem with a cron job.

Conclusion

  1. Benchmarking is not easy, but easy to get wrong.
  2. Slow might be fast enough.
  3. Read more about benchmarking on Jan's blog.

Categories: en, CouchDB, Python, TileCache, geo

GeoCouch: New release (0.10.0)

2009-09-19 22:27

Notice: This blog post is outdated, please move on :)

It has been way to long since the initial release, but it’s finally there: a new release of GeoCouch. For all first time visitors, GeoCouch is an extension for CouchDB to support geo-spatial queries like bounding box or polygon searches.

I keep this blog entry relatively short and only outline the highlights and requirements for the new release as GeoCouch finally has a real home at http://gitorious.org/geocouch/. Feel free to contribute to the wiki or fork the source.

Highlights

  • Many geometries are supported: points, lines, polygons (using Shapely).
  • Queries are largely along the lines of the OpenSearch-Geo extension draft. Currently supported are bounding box and polygon searches.
  • Adding new backends (in addition to SpatiaLite) is easily possible.

Requirements

Other versions might work.

Download

If you don’t like Git, you can download GeoCouch 0.10.0 here.

Categories: en, CouchDB, Python, geo

CouchDB: Returning all design documents with Python

2009-08-21 22:27

I just wanted to get all design documents of a CouchDB database with couchdb-python. I couldn’t find any hints how to do it, it took longer to find out than expected. Therefore this blog entry, perhaps I save someone a few minutes of research.

from couchdb.client import Server
couch_server = Server('http://localhost:5984/')
for designdoc in couch_server['yourdatabase']\
        .view('_all_docs', startkey='_design', endkey='_design0'):
    print 'designdoc: %s' % designdoc

Update: even simpler with slicing:

from couchdb.client import Server
couch_server = Server('http://localhost:5984/')
for designdoc in couch_server['yourdatabase']\
        .view('_all_docs')['_design':'_design0']:
    print 'designdoc: %s' % designdoc

Categories: en, CouchDB, Python

FOSS4G 2009: I'm speaking

2009-07-21 22:27

I did it! I'll speak on the FOSS4G Conference 2009 (Free and Open Source Software for Geospatial Conference), 20th–23rd October in Sydney about “CouchDB and Geodata”. More information is available at the official website.

Categories: en, CouchDB, geo

Poor man’s bounding box queries with CouchDB

2009-07-19 22:27

Several people store geographical points within CouchDB and would like to make a bounding box query on them. This isn’t possible with plain CouchDB _views. But there’s light at the end of the tunnel. One solution will be GeoCouch (which can do a lot more than simple bounding box queries), once there’s a new release, the other one is already there: you can use a the list/show API (Warning: the current wiki page (as at 2009-07-19) applies to CouchDB 0.9, I use the new 0.10 API).

You can either add a _list function as described in the documentation or use my futon-list branch which includes an interface for easier _list function creation/editing.

Your data

The _list function needs to match your data, thus I expect documents with a field named location which contains an array with the coordinates. Here’s a simple example document:


{
   "_id": "00001aef7b72e90b991975ef2a7e1fa7",
   "_rev": "1-4063357886",
   "name": "Augsburg",
   "location": [
       10.898333,
       48.371667
   ],
   "some extra data": "Zirbelnuss"
}

The _list function

We aim at creating a _list function that returns the same response as a normal _view would return, but filtered with a bounding box. Let’s start with a _list function which returns the same results as plain _view (no bounding box filtering, yet). The whitespaces of the output differ slightly.

function(head, req) {
    var row, sep = '\n';

    // Send the same Content-Type as CouchDB would
    if (req.headers.Accept.indexOf('application/json')!=-1)
      start({"headers":{"Content-Type" : "application/json"}});
    else
      start({"headers":{"Content-Type" : "text/plain"}});

    send('{"total_rows":' + head.total_rows +
         ',"offset":'+head.offset+',"rows":[');
    while (row = getRow()) {
        send(sep + toJSON(row));
        sep = ',\n';
    }
    return "\n]}";
};

The _list API allows to you add any arbitrary query string to the URL. In our case that will be bbox=west,south,east,north (adapted from the OpenSearch Geo Extension). Parsing the bounding box is really easy. The query parameters of the request are stored in the property req.query as key/value pairs. Get the bounding box, split it into separate values and compare it with the values of every row.

var row, location, bbox = req.query.bbox.split(',');
while (row = getRow()) {
    location = row.value.location;
    if (location[0]>bbox[0] && location[0]<bbox[2] &&
            location[1]>bbox[1] && location[1]<bbox[3]) {
        send(sep + toJSON(row));
        sep = ',\n';
    }
}

And finally we make sure that no error message is thrown when the bbox query parameter is omitted. Here’s the final result:

function(head, req) {
    var row, bbox, location, sep = '\n';

    // Send the same Content-Type as CouchDB would
    if (req.headers.Accept.indexOf('application/json')!=-1)
      start({"headers":{"Content-Type" : "application/json"}});
    else
      start({"headers":{"Content-Type" : "text/plain"}});

    if (req.query.bbox)
        bbox = req.query.bbox.split(',');

    send('{"total_rows":' + head.total_rows +
         ',"offset":'+head.offset+',"rows":[');
    while (row = getRow()) {
        location = row.value.location;
        if (!bbox || (location[0]>bbox[0] && location[0]<bbox[2] &&
                      location[1]>bbox[1] && location[1]<bbox[3])) {
            send(sep + toJSON(row));
            sep = ',\n';
        }
    }
    return "\n]}";
};

An example how to access your _list function would be: http://localhost:5984/geodata/_design/designdoc/_list/bbox/viewname?bbox=10,0,120,90&limit=10000

Now you should be able to filter any of your point clouds with a bounding box. The performance should be alright for a reasonable number of points. A usual use-case would something like displaying a few points on a map, where you don’t want to see zillions of them anyway.

Stay tuned for a follow-up posting about displaying points with OpenLayers.

Categories: en, CouchDB, JavaScript, geo

By Volker Mische

Powered by Kukkaisvoima version 7