vmx

the blllog.

Goodbye Couchbase

2016-07-04 22:35

I'm no longer with Couchbase. That's a pretty significant change for me. I've been with Couchbase (Couchio, CouchOne) for over 6 years, it was on May 12th 2010 when I got my Couchio email address. I was even still studying back then. It was a great time, I learnt a lot and it was a pleasure to work with so many skilled people.

Though there are exciting times ahead. I'm working with Damien Katz on a fancy new project which includes the technologies I really like to work with. It's build with Rust and the storage back-end is RocksDB. More on that as we go.

Surely some may wonder what will happen to GeoCouch. I don't know about the future of it when it comes to Couchbase, that's beyond my control. Though I do control GeoCouch in regards to Apache CouchDB. I surely want to get it working with the upcoming Apache CouchDB 2.0 release. GeoCouch might look pretty different from what it is now. I'd like to base the back-end on RocksDB. That's the reason why I'm implementing an R-tree on top of RocksDB. This also means that GeoCouch again is a free-time project of me, though that isn't necessarily a bad thing.

Categories: en, Couchbase, GeoCouch

FOSS4G 2014

2014-09-16 22:35

The FOSS4G 2014 conference was a blast as every year. I really enjoyed meeting old friends as well as people that I’ve know through the web only.

The talks

As I was in the program committee myself, I won’t say much about the selection of the talks (please see the “Things to improve” section, though), but I’ve heard only few complaints so far. This might be due to publishing the review process that we used. But if you have any complaints or ideas to improve it in coming years, please get in touch with me.

I haven’t spend all my time it talks but saw quite a few. As always you might end up in some decent talk where you expect it the least. Notable ones that I’ve attended:

  • “Gimme some YeSQL! – and a GIS” by Vincent Picavet: It was a good overview what is hot and new in PostgreSQLl. It’s good to see the Josh Berkus is getting closer to his envisioned CouchgreSQL.
  • “Spatial in Lucene and Solr” by David Smiley: For me it’s always interesting to hear from other spatial indexing solutions.
  • “Accurate polygon search in Lucene Spatial (with performance benefits to boot!)” by Jeffrey Gerard: That one was of interest for me as the problems that need to be solved are similar to the ones I have with GeoCouch.
  • “An Open Source Approach to Communicating Weather Risks” by Jonathan Wolfe: A talk about the NWS Enhanced Data Display, which is a huge web portal. There are a lot possible through that web interface, which contains a lot of information. Although they use a lot of open source I’d really love to see the portal itself to be open sourced.
  • “OnEarth: NASA’s Boundless Solution to Rapidly Serving Geographic Imagery” by Joe Roberts: They showed WorldView which is another example of a huge web portal, but this time the source code is available as open source on Github: https://github.com/nasa-gibs/worldview
  • “Introduction to the geospatial goodies in Elasticsearch” by Simon Hope and Jerome Anthony: It was a good introductory talk with a great live example at the end.

I certainly had fun with my own talk “GeoCouch: A distributed multidimensional index”, I had a good feeling afterwards. I hope the people in the audience enjoyed it as well. I’m still working on getting the Terminal output into a PDF.

During my talk I also announced that MapQuery is officially retired. Steven Ottens and I don’t really use it anymore, there weren’t much users. The JavaScript world has moved on with OpenLayers 3 as well as new kinds of web frameworks.

The venue

The catering was great. I heard a few people that weren’t happy about last year’s catering in Nottingham, I have to say that I was.

What I really enjoyed this year was that after the first day there was even food after the last session. On the second day there was the gala event (with food) and on the last day everyone was heading off anyway.

This year the venue was great as all sessions were close to each other. It was easily possible to switch rooms (unlike it was in some of the previous FOSS4Gs).

Everything was well organized, there were plenty of volunteers, you saw them at every corner. Also the guidance to the gala event was great. I think everyone who wanted to make it was easily able get onto the right lightrail.

The social events

There was an JS.Geo after-party I liked meeting some people that I haven’t seen in a while that weren’t even attending the FOSS4G. We then moved on to the FOSS4G Welcome Reception hosted by Ecotrust and Point 97.

On the first day there was the LocationTech Meetup which had plenty of free drinks and a lot of the people from the conference that just went over to that bar.

Second day was the gala event at the World Forestry Center. I think it was the best one from any of the FOSS4Gs that I’ve been to (since 2009). What I really enjoyed was that it wasn’t the normal “Gala Dinner Setup” with huge round tables you kind of feel locked onto. Instead there was a wide open space and you grabbed the food at some counter (kind like a food cart). You were able to walk around and chat with people, but if you’d like to be seated you could also sit down (at one of those round tables).

The last event was at the Refuge (hosted by MapBox. It was excited to have trains run by so closely. After the event some of us headed over to Boke Bowl which was really served great food.

The field trip

I booked a field trip on Saturday to Mount St. Helens. It was really great. Our guide Jill was just as enthusiastic as Darrell Fuhriman described her. It was a fun group, a lot to learn, beautiful views and certainly worth a visit. I was impressed by scale of the 1980 Mount St. Helens eruption. It looks way smaller on the pictures that I’ve seen in the past.

Miscellaneous

I really liked that you didn’t get huge conference bags with all sorts of things you never need and throw away anyway. Instead you were just pointed to a table with those things where you could take them. You would then proceed to pick a t-shirt of your size (if you wanted to).

I also really like the program brochure, I don’t think I’ve seen one done that well before. It’s small and handy and was insanely well designed. Having the talks split into tracks which fit one page and having the time axis horizontally is a great idea. Also having the abstracts right behind every day, rather than having the full schedule first and the abstracts next is good to keep things easy to browse. You don’t really care about yesterdays abstracts, do you? But even if you do, you can easily find them as the individual days had colored markers on the side of the page, very much like telephone books have. So it was again easy to browse. Perhaps the local team could upload it for reference for future conferences.

Thing to improve

There wasn’t much that could be done better. Though there’s one thing that I’ve discussed with another member of the program committee (that I was also part of). The conference is about Free and Open Source Software. For me this means that you are not only using, but also contributing something back. For me the conference talks should create value for the community.

Of course there should also be talks about “How we migrated our department from proprietary software to open source”, I don’t have a problem with that. Though things should be clearer. What I generally don’t want to see is talks about how people use open source software, build upon it, even improve it, but then not contributing it back. Such a talk has no real value for the attendees. It’s too much “Look what we’ve done but you can’t use it”. I’m well aware that there are cases where open sourcing things is not possible due to the contracts. It strikes me that we might have rejected a talk that would have been in the open source spirit.

One solution I came up together with Jacob Greer is that for future FOSS4Gs you need to include a link in your abstract submission to the source code. This could either be to the project itself, or to upstream projects that you’ve contributed to (and not only used).

Conclusion

It was an awesome, well organized event. I’d like to thank the local organizing committee very much for all the huge amount of work they’ve put into this. You’ve set the bar really high.

Categories: en, Couchbase, GeoCouch, MapQuery, conference, geo

OSGeo Code Sprint Bolsena 2012

2012-06-16 22:35

The OSGeo Code Sprint in Bolsena/Italy was great. Many interesting people sitting the whole day in front of their laptops surrounded by a beautiful scenery. This year I spent most of my time on GeoCouch, CouchDB, Couchbase, deegree, Imposm and GeoNetwork.

Already on the first hacking day we had a result, a Couchbase backend for the deegree blob storage. This means that you can now store your rich features (e.g. from INSPIRE) in Couchbase and serve them up as a WFS with deegree. In case you wonder what rich features are, it's the new, shiny and more descriptive name for complex features.

In the following days I worked together with Oliver Tonnhofer on a CouchDB/GeoCouch backend for Imposm. You are now able to store your OpenStreetMap data in CouchDB and make queries on it through GeoCouch. I've created a small demo that displays the some data import from Andorra with directly with MapQuery, without the need of any additional server/service. The CouchDB backend should be easily adaptable to Couchbase, if you want to help, let me know.

I've then spent some time on the GeoNetwork project and helped translating the language file to German. I cleaned it up a bit and fixed major wrong translations. It's not perfect yet, as I've only spent little time on it, but at least it should be way more understandable (and sadly less funny) than the original version which was generated by Google Translate.

When it was time for presentations, I give a quick overview over the Couch* ecosystem. From CouchDB to GeoCouch, BigCouch, PouchDB, TouchDB (TouchDB-iOS, TouchDB-Android), Couchbase Syncpoint and Couchbase. You can find the slides as PDF here.

On the last day I've spent my time on polishing GeoCouch a bit and getting it ready for the Couchhack in Vienna. I've backported all changes from Couchbase to the CouchDB 1.2.x branch and also ported the geometry search into an experimental branch. You can now search your GeoCouch with any Geometry GEOS supports.

The event was fun as always and I also get to know some new people (hello B3Partners guys). Thanks Jeroen from GeoCat for organizing it, and thanks all other hackers that made it such a awesome event. Hope to see you all next year!

Categories: en, GeoCouch, CouchDB, Couchbase, conference, geo

The future of GeoCouch and CouchDB

2012-01-06 22:35

The CouchDB world is currently full of “The future of CouchDB” blog posts. It started with the blog post from Damien Katz the creator of CouchDB. Of course people were also concerned about the future of GeoCouch. No worries, it will be good.

The future of Apache CouchDB

The reactions were quite different. People who are not deeply involved with the CouchDB community think that this means the end of Apache CouchDB. My reaction was positive, I tweeted:

“It’s good to see the Damien is so open to [the] world”

The reason was, that for me it was pretty clear that it would happen, and I was just happy that Damien officially made the cut.

The reactions from CouchDB community members where pretty much what Till Klampäckel describes in his blog post. You could see it comming after Couchbase announced that they are not the CouchDB company and that their product won’t be Apache CouchDB compatible.

I agree with Till here, the way Damien wrote his blog post, isn’t the best imaginable. For outsiders, it really seems to be the end of Apache CouchDB, but it is not. For me it just shows, why foundations like the Apache Foundation are such a great idea. Even if the original creator leaves the project, it still lives on.

Apache CouchDB has a lot of contributers and the mailing lists and IRC channel is busy as always. That CouchDB has a future is also shown by the blog post from Cloudant. They will keep supporting Apache CouchDB.

The future of GeoCouch

After this quick recap what happened so far, it’s time to talk about the future of GeoCouch. As you may know, I work for Couchbase on the integration of spatial functionality into their product.

Currently the overlap between Apache CouchDB and the version Couchbase uses internally is still quite huge, but it will diverge more and more in the future. Thus it will get harder and harder to maintain a single version that supports Apache CouchDB and Couchbase.

The good news is, that GeoCouch is pretty much a data structure only. It's an R-tree that stores JSON documents. This can easily be used by CouchDB and Couchbase. Perhaps small wrappers will be needed, but those should be minimal.

The easiest way to understand how the future looks like is in a small illustration:

Illustration of GeoCouch and its relation to CouchDB and Couchbase

GeoCouch's core is the R-tree, it's the same code for CouchDB and Couchbase. On top of it there will be code that is specific to either CouchDB or Couchbase.

This means that the majority of the devlopment I do for Couchbase will also improve the GeoCouch you can use for CouchDB.

Conclusion

The future of all three, Apache CouchDB, Couchbase and GeoCouch looks bright.

Categories: en, CouchDB, GeoCouch, Erlang, geo

FOSS4G 2011: Report

2011-09-20 22:35

The FOSS4G 2011 is over now. Time for a small report. The crowd was amazing and it was again the ultimate gathering of the Free and Open Source for Geospatial developer tribe. Solid presentations and great evenings.

My talk: The State of GeoCouch

I'm really happy how my talk went, I really enjoyed it. The were lots of people (although there was a talk from Frank Warmerdam at the same time) asking interesting questions at the end.

The talk is not only about GeoCouch but also gives you an overview of some of the features it leverages from Apache CouchDB. In the end you should have an overview why you might want to use GeoCouch for your next project.

You can get the slides right here.

Other talks

I was happy to see that there was another talk about GeoCouch. Other talks I really enjoyed were:

And of course there were also great talks from in the plenary sessions from Paul Ramsey about Why do you do that? An exploration of open source business models and Schuyler Erle's so funny lightning talk about Pivoting to Monetize Mobile Hyperlocal Social Gamification by Going Viral

Code Sprint

At the code sprint I was working on MapQuery together with Steven Ottens and Justin Penka. Steven was working on TMS support, Justin on a 6 minutes tutorial and I on making manual adding of features possible.

The OpenLayers developers did the migration from Subversion to Git for their development. OpenLayers is now available on Github.

And luckily there was a fire alarm in between to take a group photograph.

Future of the FOSS4G

I really hope there won't be a yearly FOSS4G conference for the whole of the US. There should be regional events, as I think one big one would draw the attention away from the international conference. Why should you fly to Beijing for the FOSS4G 2012 if you can meet the majority of the developers in the US as well?

Final words

The FOSS4G was great. It was organized well and people were always out in the evenings. The only minor nitpick is that many people working remote had the city of their company in the name badge and not the one they live in. It seems that the original for you had to fill was confusing. So for next year it should perhaps say “Location where you live”. Hence I still don't believe that there were more Dutch than German people at the conference (Tik hem aan, ouwe! ;)

Categories: en, CouchDB, GeoCouch, MapQuery, Erlang, JavaScript, geo

WhereCampEU 2011 Berlin

2011-05-29 22:35

The WhereCampEU 2011 is just over now, so it's time to get my impression and notes from the sessions written down. But first I'd like to thank all the organizers and attendees for this great event and the geobeers in the evenings.

Geo Databases

The WhereCamp is also a chance to meet people you haven't seen for a long time. Exactly this was the case with Shoaib Burq. We know each other from being a member of the FOSS4G 2009 organizing committee.

As there were still a few free slots, we decided to make session that is simply called "Geo Databases". I didn't expect many people to show up, as there were other sessions with way catchier names.

I ran late as I didn't find the room where my session was supposed to be. But luckily Shoaib was already presenting a rough outline on what will be talked about (we haven't talked about what we want to do upfront). So I happily joined the crowd and started to moderate things a bit. And it was just amazing, the room was packed with people even sitting on the floor. A rough guess would b,e that it was half of the attendees.

We discussed a bit about one of my favourite topics, the question of how much should be done within a geospatial database and what should be done outside of it. It then led to the problem that doing a lot within the database is easy (and fast) on a single instance, but as soon as you want to distribute your database, you might get into trouble.

To conclude, the atmosphere was good and people seemed to have enjoyed it (at least some were telling me so :) I'm happy that it went this well. Shoaib has kindly taken some notes and put on Github.

Web Mapping Libraries

On the second day the program wasn't as packed as on the first day, so I decided to make another session called "Web Mapping Libraries". The Again I didn't expect too many people to show up and this time I was right. It was a good crowd nonetheless.

What I wanted to find out was why people create new web mapping libraries instead of just improving existing ones like OpenLayers. This was also triggered by Slide 54 from Peter Batty's talk, where he compares OpenLayers to Google Maps. The two things that were catching my attention were "lightweight" and "extra mobile features". Those two can be solved. Eric Lemoine demonstrated a 29KB (gzipped) version of OpenLayers including the mobile features that were added at the Lausanne Code Sprint.

In the session it was just great to hear the impressions for what I call "web neogeo people" (people are web developers, designers and general people that like to visualize data) as opposed to "old school GIS people" (that know all kinds of OGC standards and dream about the axis confusion at night).

Things we've identified were (photo version):

  • Documentation
  • Default Look
  • Usability
  • Examples are not good
  • No explanation of the general architecture

Documentation

The main source of documentation are the API docs. They describe every function of the library. One problem is that it isn't really clear what exactly you can pass in when one parameter is an options object (Example.

The lack of documentation with a nice introduction was also mentioned. Though I don't fully agree, there is some really nice documentation in prose, but it seems that people easily miss it (I especially like e.g. the page about Spherical Mercator).

Default look

I agree that the default look of OpenLayers looks a bit out dated. The main buttons look like from the early 2000's and the layer switcher isn't nice either.

On the one hand, you can change the look if you want to (example (scroll down)), or code the controls with any UI library you'd like to use (this is what GeoExt/MapQuery does).

Usability

I'm not really sure what this one was about, but I think it was about the complexity of OpenLayers. You need to have quite a bit GIS knowledge before you can get started. Don't get me wrong, I love OpenLayers for what it does and it's huge amount of features, though it comes to a price for newbies.

Again this is where I think GeoExt/MapQuery can help. They can build an easy to use API around the most common functionality like: display a GeoJSON file on a Google base map that shows a popup when I click on the geometry.

Examples are not good

It's nice to have a whole bunch of examples, though in the case of OpenLayers they are a bit misleading. I use them a lot to see how things work. Though they are not always well drafted idiomatic OpenLayers examples, but rather some source showing how a specific feature works. And this is exactly what they are. If you implement a new feature, you create API documentation and a working example.

For me they are more like acceptance tests, rather than good howtos for newbies.

No explanation of the general architecture

This one was mostly about "how do things work". One point was about vector layers. They are really powerful and you can compose them out of several parts with things like strategies and formats. It's not easy to understand how they fit together.

I think there was also another point, but I don't recall it. Let me know if you were at the session and remember it :)

Conclusion

I had a great time at the WhereCampEU 2011 and it was definitely worth going there. If you have anything to add to the notes about the session, please do so.

Categories: en, GeoCouch, OpenLayers, MapQuery, conference, geo

FOSSGIS, GeoCouch and MapQuery

2011-04-19 22:35

Two weeks ago I had the chance to give a talk about GeoCouch and MapQuery at the FOSSGIS 2011. Most of the people who read this Blog are probably aware of GeoCouch, but not so much of MapQuery. For me these two projects are tightly connected and therefore deserve a quick introduction/update.

GeoCouch

GeoCouch, a spatial index for CouchDB gains, more and more attention. One of the reason is that the installation recently got way easier for developers as well as for normal users. You now can install GeoCouch as an extension right next to your already existing CouchDB instance. You may also download a binary of Couchbase-Server, which already includes GeoCouch. And finally there's the brand new Iris Couch hosting as well (previously known as the CouchOne hosting). So getting started with GeoCouch is easier than ever before.

Some people might have wondered about the state/future of GeoCouch, especially after the merger of CouchOne with Membase to Couchbase. I will keep on developing GeoCouch at Couchbase and it is (as it always was) fully open source licensed under the Apache 2.0 License.

The new home for the latest source is the Couchbase Github repository.

OpenStreetMap

The FOSSGIS was also about OpenStreetMap. The idea to put OpenStreetMap data into GeoCouch is very sensible, but wasn't really done (AFAIK) in a big fashion. Luckily Jochen Topf from Geofabrik told me about his Projekt Osmium, which makes it possible to process OSM data with JavaScript. There is already a script to output a Shapefile, so it should be really easy to output GeoJSON, which could be consumed by GeoCouch. So if you (who are currently reading this) have some spare time, please give it a go :)

MapQuery

MapQuery is a web mapping framework that builds on OpenLayers and jQuery. The goal is a framework that is just as easy to use as jQuery combined with the power of OpenLayers. It's meant for people that just want to get started with web mapping, but also for those who have already knowledge about OpenLayers and want to have easy integration into their jQuery application.

I was able to show a quick demo of the MapQuery API at the FOSSGIS. I won't publish it here, as things are about to move fast. After over one year of discussions about MapQuery and only little code contributions, it seems that we are finally getting somewhere. That feels so good :)

The wonderful EduGIS is build on an early version of MapQuery (source code), but will be merged with the most recent version of my fork.

Other big news is that the WhereGroup hired Christian Wygoda, who is a committer of the MapQuery project. This also means that Mapbender 3 will use MapQuery.

And finally I've also met a developer of a another company that was building a big mapping application based on OpenLayers and jQuery. I don't want disclose it here, as the code isn't open source yet, but the developer told me that it should be easily possible. I will keep in touch with them and hope they will contribute their code to MapQuery.

To get to a conclusion about MapQuery. If you want to stay in touch with the project, please subscribe to the official mailing list, this is where things are happening (there's also the little attended IRC channel #mapquery on freenode). If you want to be a user of MapQuery, you should be patient and wait a bit. If you plan to contribute, you can start now. The currently biggest item is moving the EduGIS MapQuery code base over to the MapQuery version of my fork. The "documententation" are the demos.

FOSSGIS

As people started to asked about the slides from my presentaion at FOSSGIS, here they are.

FOSSGIS was a really awesome event, where I met a lot of new people, but also a lot of friends I haven't seen in a while. I'm really looking forward to next year's conference, but also hope that I might see many of the people at this year's FOSS4g in Denver.

Categories: en, CouchDB, GeoCouch, MapQuery, geo

GeoCouch Vortrag in Augsburg

2010-07-07 22:35

Im Rahmen des Diplomandencolloquium des Lehrstuhl für Humangeographie und Geoinformatik halte ich am 19.07.2010 um 17:30 Uhr (Raum 2125) an der Uni Augsburg einen Votrag über GeoCouch. Der genaue Titel lautet:

GeoCouch: Eine Erweiterung für CouchDB zur Abfrage räumlicher Daten

Er richtet sich an Geographen, wird also nicht zu sehr ins Detail der Implementierung gehen. Es sind auch keine Vorkenntnisse zum Thema CouchDB nötig. Wer also mehr über CouchDB und GeoCouch wissen will, ist herzlich dazu eingeladen. Danach stehe ich natürlich zu Fragen zur Verfügung.

Ich habe keine Ahnung wie groß die CouchDB Community im Raum Augsburg ist, aber sollte jemand dieser Einladung folgen, spricht auch nichts gegen ein anschließendes kleines CouchDB/GeoCouch/NoSQL Meetup. Am besten meldet ihr euch bei mir per Mail, denn wenn ein paar Leute sicher kommen, werden es sich andere bestimmt auch überlegen.

Sorry Planet CouchDB for writing in German, but this is about a talk in German.

Categories: de, CouchDB, GeoCouch, Erlang, geo

FOSS4G 2010: I'm speaking

2010-05-21 22:35

I did it! I'll speak at the FOSS4G Conference 2010 (Free and Open Source Software for Geospatial Conference), 6th–9th September in Barcelona about “GeoCouch: A spatial index for CouchDB”. As soon as the abstract is available online I'll link to it. Hope to see you there!

Categories: en, GeoCouch, CouchDB, Erlang, geo

Non-validating WKT parser for Erlang

2010-05-14 22:35

The upcoming OpenSearch Geo specification will add support for querying with WKT (Well-Known Text). As I plan to support this specification in GeoCouch, I was in need of a WKT parser written in Erlang. I tried several ways to write this parser, but I ended up with writing it manually, based on the ideas of the fabulous MochiWeb JSON2 Parser

The parser is meant for fast parsing, it is non-validating. This means that it parses only valid WKT and all other strings that seem to be valid, but are not. The grammar is simplified to (in EBNF as used for the XML spec):

wkt ::= item | string  '(' space* item (comma item)* ')'
item ::= string (geom | list | nested_list | item | 'EMPTY')
nested_list ::= space* '(' list (comma list)* ')' | '(' nested_list+ ')'
list ::= '(' geom (comma geom)* ')'
geom ::= space* '(' coord (comma coord)* ')'
coord ::= space* number (space+ number)*
number ::= integer | float 
integer ::=  ('-' | '+')? [0-9]+
float ::= ('-' | '+')? [0-9]+ '.' [0-9]+ exponent?
exponent = 'E' ('-' | '+')? [0-9]+
string ::= [a-zA-Z]+ (space* [a-zA-Z])*
space :== #x20
comma :== ',' space*

I hope I got the grammar right, leave a comment if not. This means also strings like this(is(10 20), a test EMPTY) would be parsed to:

{this,[{is,[{10,20}]},{'a test',[]}]}

A validating parser would be much slower as it would also need to perform checks on the geometry, e.g. for polygons whether interiors are really within the exterior ring or not.

The general rule is, a list of coordinates is transformed to a tuple, a list of coordinates to a list. The geometry name will be an atom. Here's an example for a polygon:

wkt:parse("POLYGON ((102 103, 204 205, 306 107, 102 103),
                    (12 13, 24 25, 36 17, 12 13),
                    (62 63, 74 75, 86 67, 62 63))").
{polygon,[[{102,103},{204,205},{306,107},{102,103}],
          [{12,13},{24,25},{36,17},{12,13}],
          [{62,63},{74,75},{86,67},{62,63}]]}

In case you're getting excited now, the source is available at Github, realeased under the MIT License.

If someone plans to write a validating WKT parser for Erlang (please let me know), I propose using neotoma it's really a nice "packrat parser-generator for Erlang for Parsing Expression Grammars (PEGs)".

Categories: en, GeoCouch, Erlang, geo

By Volker Mische

Powered by Kukkaisvoima version 7