First of all, read his blog post and follow the many links, there is so much to discover.

One re-occurring thing in the documents linked from the “issues on the earlier stages of the Open Data pipeline” section is that for most portals a static site should be sufficient. I fully agree with that. When it’s done properly, an automated rebuild of some parts when new data is added should work well. These days even powerful client-sided search is possible.

It’s a bit off-topic, but David’s Barefoot Data Platforms page links to Maggie Appleton’s Home Cooked Software and Barefoot Developers talk linked. I highly recommend watching it, it was one of my favourite talks at the Local-first Conference 2024. I always wanted to blog about it, but never found the time.

But now to the concrete points David mentions. If anyone has ideas on how to make those things happen with Matadisco, please open issues on the main Matadisco repo.

Take inspiration from existing flexible standards like Data Package, Croissant, and GEO ones for the core fields. Start with the smallest shared lexicon while leaving room for specialized extensions (sidecars?).

I don’t think Matadisco should go into too much detail on specifying what the metadata should look like. Making one metadata standard to rule them all is destined to fail from my experience (ISO 19115/19139 anyone?). Though there might be a lowest common denominator, similar to what Standard.site is doing for long-form publishing. In order to find out what that looks like, I propose that individual communities start by specifying Lexicons for their own needs. This could be done through tags, which I’ve outlined in the Matadisco issue “Introducing tags for filtering and extension point”.

Split datasets from “snapshots”. Say, io.datonic.dataset holds long-term properties like description and points to io.datonic.dataset.release or io.datonic.dataset.snapshot, which point to the actual resources.

Some kind of hierarchical relationship would be useful. FROST, which Matadisco drew a lot of inspiration from, is centred around IceChunk, which also has the concept of snapshots. But I don’t think we should stop at the concept of snapshots. In my original demo, I scrape a STAC catalogue for Sentinel-2 imagery. Every new image is a new record. They are all part of the same STAC collection, so we could use a similar concept in Matadisco as well.

Add an optional DASL-CID field for resources so we “pin” the bytes.

Yes, that’s something @mosh is keen to have. It’s not only useful for pinning things to a specific version, but also to make it possible to verify that the data you received is the one you expected. It sounds trivial, but the problem would be where to put it. Do you only hash the metadata record it points to? Do you hash the data container (if there’s one)? Or each resource a metadata record points to?

Core lexicon should be as agnostic as possible!

As mentioned above, it might be out of scope for Matadisco and for now it’s left to the individual communities.

Bootstrap the catalog. There are many open indexes and organizations. Crawl them!

Indeed! My first two Matadisco producers are sentinel-to-atproto crawling Element 84’s Earth Search STAC catalogue and gdi-de-csw-to-atproto crawling the GeoNetwork instance of the official German geo metadata catalogue.

Integrate with external repositories. E.g., a service that creates JSON-LD files from the datasets it sees appearing on the Atmosphere so Google Datasets picks them up. The same cron job could push data into Hugging Face or any other tool that people are already using in their fields.

At first this would need to happen for each individual type of record, see the tags proposal above.

Convince and work with high quality organizations doing something like this! I’d definitely collaborate with source.coop for example.

That surely is the goal!

While being the biggest edition ever (1000 registrations on-site, 300 online) it was well run and organized as every year. It didn’t even feel larger than usual. The CCC video team streamed live and published the cut videos the same day in outstanding quality as always.

I split this post into two sections, one about interesting talks for the geo world in general and then follow up discussions on my Matadisco talk and ATProto in general.

Talks

I’ve spent most of my time in hallway chatting with people as this is what matters most to me when I’m attending a conference in person. Nonetheless I’ve still managed to see some excellent talks.

Panel discussion on digital sovereignty in the cloud

The conference started with a high-class panel discussion on digital sovereignty in the cloud. The public discussion on that topic is often centered around where servers are located. Though that doesn’t actually matter. US companies can be forced by their government to give access to the data independent of their physical location.

Other topics touched were best practices on switching from proprietary to open source systems.

Barrier-free travelling thanks to paid mappers

Public transport in Germany must be accessible to disabled individuals (reality is far away from that). For routing, you need the data basis for it. This talk gets into the details on how Baden-Württemberg, a federal state in south Germany, works on enabling barrier-free travelling. They decided to add that information of all their 1100 train stations directly to OpenStreetMap. In order to achieve the required high quality they’ve hired through a third party company several experienced mappers from the community.

I really like the idea that OpenStreetMap can now be used as source of truth for that data set. I hope other federal states follow this lead.

Routing talks

I’ve seen two talks about routing. The one about Valhalla routing engine with MapLibre Native was interesting because it was about a special case, where you want to re-route bus lines in case of construction. Although the resulting system is not open source, they’ve contributed upstream to Valhalla, to make it work well with MapLibre Native. Those contributions can be more valuable than a one time source code dump of forked repositories, just to call it open source.

Another one was about Real-time mobility analytics for disaster relief operations, which was interesting to see how routing is used in such cases. The limitations and how such systems really help on the ground.

Matadisco and ATProto

My talk on Matadisco was about the current status of metadata catalogues, the problems and how ATProto can make things better. What I should have made clearer is what Matadisco actually is. I didn’t make it clear that it’s just a schema/convention people would use to announce their data on ATProto. It could’ve been mistaken as a piece of software or a service. You would use Matadisco in order to implement something for your pipeline.

Nonetheless people got the idea and I had good conversations afterwards. I talked with Olivia Guyot about the possible ways on how to integrate Matadisco record publishing into GeoNetwork. With Christian Willmes about creating a portal for combining paleoenvironmental and archaeological data.

While chatting about ATProto at one of the social events Klaus Stein talked about how he would like a social network to be. Users would just put static files somewhere. I agree that having static webspace somewhere without any server component is not only cheap, but also the easiest to get. He is not bothered about other components being operated by other parties, e.g. for indexing. That kept me thinking how far ATProto is away from that. I’d like to build a prototype that is like a static site generator for ATProto records. It won’t be able to act as a full PDS, you would need a WebSocket connection to get the data to a relay. But there could be a minimal service operated by a third party that polls those static PDS for updates and forwards them to a relay.

It’s early stage and experimental, but the potential is significant. To see it in action, visit the matadisco-viewer demo. It listens to the incoming stream of ATProto events and renders them. At the moment it’s satellite images only, but that will hopefully change soon.

Above is an example of what crossed my screen while developing (metadata, download at full resolution (253MiB)).

Motivation

Metadata records can be very diverse. They might describe geodata, your favourite news site or your favourite podcasts. What they all have in common is that users usually rely on centralized platforms in order to find them. For geodata, this is often a government-run open data or geo portal. These platforms decide which data gets published.

You might generate a derived dataset or clean up an existing one. If you operate from outside of the original creators, you probably won’t even be able to get your data linked from there. So how will anyone find out about it? That’s a problem of metadata discovery.

The other side of the problem is that even when metadata is available, it can be hard to find. There are large metadata aggregation portals like the portal for European data, with almost 2 million records. How do you find what exactly you are looking for. What if there were specialized portals tailored to specific communities?

For even more details, see the companion blog post of the IPFS Foundation.

The idea

The idea is to support both: an easy way for anyone to publish discoverable metadata, and a way to make that metadata widely accessible to build both large aggregators and specialized portals tailored to specific communities.

The central building block is ATProto. It allows anyone to publish and subscribe to records. Rather than defining a single metadata schema to rule them all, the approach here is more meta-meta. Each record contains a link to the actual metadata. That’s the absolute minimum. Though it could make sense to go beyond this minimalism and store additional information to make it easier to build custom portals.

One example of such additional information is a preview. It’s nice to get a quick sense of the underlying data that the metadata describes. For satellite imagery, this could be a true color thumbnail of the scene. For long form articles, a summary or excerpt. For podcasts it may be a brief audio snippet or trailer.

The implementation

As part of my work at the IPFS Foundation, I started with geodata. The first prototype focuses on [Copernicus Sentinel-2 L2A satellite images].

The metadata is sourced from Element 84’s Earth Search STAC catalogue. It provides free, publicly accessible HTTP links to the images (the official Copernicus STAC does not). A [Cloudflare Worker] checks the STAC instance every few minutes for updates. When new records appear, a link to the metadata along with a preview is ingested into ATProto. The source code for the worker is available at https://github.com/vmx/sentinel-to-atproto/.

Below is the Lexicon schema for this ATProto meta-metadata record, which I call Matadisco. To improve readability, the MLF syntax is used:

/// A Matadisco record
record matadisco {
    /// The time the original metadata/data was published
    publishedAt!: Datetime,
    /// A URI that links to resource containing the metadata
    resource!: Uri,
    /// Preview of the data
    preview: {
        /// The media type the preview has
        mimeType!: string,
        /// The URL to the preview
        url: Uri,
    },
}

Once records are available on ATProto, they can be processed and displayed. I built a simple viewer that renders records conforming to the cx.vmx.matadisco Lexicon schema defined above. A Bluesky Jetstream instance streams newly added records directly into the browser. A demo is available at https://vmx.github.io/matadisco-viewer/.

If you’re interested in seeing the raw records, you can find them on my ATProto dev account.

Prior art

This work builds on ideas by Tom Nicholas, who started a project called FROST. His motivating blog post is an excellent read about data-sharing challenges in a scientific context. His presentation on FROST explains why such a system should remain simple, with the metadata URL as the only required field.

Edward Silverton, who works in the GLAM space, explored a similar idea for publishing IIIF data. We refined his approach to align it more closely with FROST. He published further details on the complete workflow for his use case, which has a broader scope.

There was also a discussion thread on Bluesky about metadata for long-form content to build cross-platform discovery.

What’s next

Possible future steps I want to look into:

• Adding another, different geodata source, such as metadata from the German geodata catalogue.
• Including another image-based source, for example GLAM catalogues that use IIIF.
• Integrating a completely different data source, such as metadata from the podcasts of the German public broadcasting.

As mentioned in the introduction, this is deliberately experimental, things may break or change dramatically. The upside is that no one needs to worry about breakage. Please experiment with these ideas and let us now about them at the Matadisco GitHub repository. Publish records under your own namespace, or even reuse the one I am currently using.

The organization and venue

Wow. It’s been my favourite venue of all FOSS4Gs I’ve been to so far. The exhibition hall was a great place to hang out, combined with the excellent idea of a 24h bar. I’m not sure if it was used at all times, but definitely for more than 20h a day. Outside, there was plenty of space and tables to hang out, and very close by another set of tables that formed the “work area”. Which was another great place to hang out, with enough power sockets and shade for the hot days.

The main stage was an open air stage with enough seating for everyone. It was converted for the gala dinner to a stage with an excellent live band and the usual big round tables.

For me, the best part was that even the accommodation was on-site. The barracks of the former military basis, which now serve as student dorms, were our home for a week. Pretty spartan, but at a conference I don’t really spend much time in my room, I mostly need just some place to sleep.

Having everything, the talks, exhibition, social events and accommodations on-site makes it easy to maximize the time for socializing, which for me is the number one reason to attend a conference.

Everything was well organized, and it was great to see so many volunteers around.

The talks

I haven’t really selected the talks I went to. I rather joined others where they were going, or listened to recommendations. Often, I just stayed in the rest of the slot to see what else is there. My favourite talks were:

• Smart Maps for the UN and All - keeping web maps open: For me, it was the first time I saw someone speaking at a FOSS4G about using IPFS that wasn’t me. It’s great to see that it gains traction for the offline use case, where it just makes a lot of sense. UN Smart Maps is part of the UN OpenGIS initiative, it features a wide range of things, even an AI chatbot called TRIDENT that transforms the text into Overpass API calls. Try TRIDENT it out yourself, when you open the developer console, you can see the resulting Overpass API calls.
• Offline web map server “UNVT Portable”: This talk got into more detail about using Raspberry Pis to have map data stored in IPFS for offline use. It’s very similar to what I envision, the only difference is that I’d also like to keep the storage in the browser. But I surely see a future, where those efforts are combined, to have a small easy server you can deploy, with in browser copies of subsets of the data to be able to work completely offline in the field. The original UNVT Portable repository doesn’t use IPFS, but Smart Maps Bazaar does, which seems to be its successor.
• B6, Diagonal’s open source geospatial analysis engine: A presentation of the B6 tool for geospatial analysis for urban planning. It has a beautiful interface. I really like the idea of doing things directly on the map in a notebook-style way, where you perform certain steps after each other.
• Elephant in the room: A talk about how many resources to computations take? Do we always need it? It’s very hard, often impossible, to find out how environmentally friendly some cloud services are. One of the conclusions was that cheaper providers likely use less power, hence are harming the environment less. I would like if there would be better ways (e.g. it misses things like economies of scale of large providers), but I agree that this might be the best metric we currently have. And I also hope there will be more economic pressure to save resources.
• There was a closing keynote from Kyoung-Soo Eom, who was talking about his long journey in open source GIS, but also his history with Kosovo, where he was also on a mission in 1999. Quite inspiring.

My talk

My talk about Collaborative mapping without internet connectivity was about a browser based offline-first prototype that uses IPFS to enable replication to other peers. The project is called Colleemap and is dual-licensed under the MIT and Apache 2.0 license. Although I tried the demo bazillion times before my talk, it sadly didn’t work during my talk. Though, trying it later with various people, I was able to get 4 peers connected once. I even saw it working on a Windows machine. So it really works cross-platform.

For the future I hope to work closer with the people from the UN OpenGIS initiative, it would be great to combine it with their Raspberry Pi based prototype.

Things I’ve learnt

The Sentinel-2 satellite imagery is available from multiple sources, directly from Copernicus Open Access Hub or through cloud providers like AWS, Azure of Google Cloud. From the cloud providers you only get the level-2 data. They might use the original level-2 data or do their own atmospheric correction based on the level-1 data. Or even re-encode the data. So it’s hard to tell which kind of data you actually get.

As far as I know (please let me know if I’m wrong), there isn’t any mirror of the full level-1c data. You can only get it through the Copernicus Open Access Hub and there the older images are stored in the long term archive on tape, where it can take up to 24h for the data to be available for download (if it works).

Ideally, there would be a mirror of the full level-1c data (where the ESA would provide checksums of their files) and a level-2 version, where the exact process is openly published, so that you can verify how it was created. The problem is the storage cost. The current level-2 data is about 25 PiB, which leads to storage costs of over $500k USD a month if you would store it on AWS S3 Standard at the current pricing (I used the $0.021 per GB).

Final thoughts

It was great to meet Gresa and Valmir from the local organizing committee before the FOSS4G in March at the OSGeo German language chapter conference FOSSGIS in Berlin. That made it easy for me to connect to the event right from the start. If there’s one thing future FOSS4Gs should adapt, it’s the cheap on-site (or close by) accommodation. I think that shared bathrooms is also much smoother to have, if you know that everyone in the accommodation is from the conference. We had something similar with the BaseCamp in Bonn during the FOSS4G 2016 and the international code spring in 2018 during the FOSSGIS conference, where the whole place was rented for the time of the events.

Though, of course, I also missed some of my longtime FOSS4G friends I hadn’t seen in a long time. I hope you’re all doing well and will meet again soon.

This year’s FOSSGIS 2021 conference was a pure online conference. Though it had the same format as every year. Three days of conference, with four tracks in parallel. This leads to about 100 talks. I joined the organizing team about 10 weeks before the conference took place. The task sounded easy. The speakers should be able to upload their talks prior to the conference, so that during the conference less could go wrong.

All scripts are available at https://github.com/vmx/conference-tools licensed under the MIT License.

The software

The speakers submitted their talks through pretalx, a conference management system I highly recommend. It is open source and has an active community. I’ve worked on/with it over the past few to make it suitable for OSGeo conferences. The latest addition is the public community voting plugin, which has been used for the FOSS4G 2021 as well as this conference. pretalx has a great HTTP API to get data out of the system. It doesn’t yet have much support for manipulating the data, but pull-requests are welcome.

For storing the video files, Seafile was used. I haven’t had any prior experience with it. It took me a bit to figure out, that the Python API is for local access only and that the public API is a pure HTTP API. You can clearly see that their API is tailored to their use in their web interface and not really designed for third party usage. Nonetheless, it guarantees that you can do everything via the HTTP API, that can be done through the web UI.

My scripts are heavily based on command line tools like b2sum, curl, cut, jq and jo, hence a lot of shell is used. For more complex data manipulation, like merging data, I use Python.

The task

The basic task is providing pre-recorded videos for a conference that were uploaded by the speakers themselves. The actual finer grained steps are:

• Sending the speakers upload links
• Looking through the videos to make sure they are good
• Re-organizing the files suitable to be played back according to the schedule
• Make the final files easily downloadable
• Create a schedule which lists the live/pre-recorded talks

Sending upload links

In Seafile you can create directories and make them publicly available so that people can upload files. Once uploaded, you won’t see what else in that directory. In order to be able to easily reference the uploaded videos back to the corresponding talk, it was important to create one dedicated directory per talk, as you won’t know which filenames people will use for their videos.

The speakers will receive an email containing dedicated upload links for each of their talks. See the email_upload_links directory for all the scripts that are needed for this step.

pretalx

First you need to get all the talks. In pretalx that’s easy, go to your conference, e.g. https://pretalx.com/api/events/democon/submissions/. We only care about the accepted talks, which can be done with selecting a filter. If you access it through curl, you’ll get a JSON response like that one: https://pretalx.com/api/events/democon/submissions/?format=json. pretalx returns 25 results per request. I’ve created a script called pretalx-get-all.py that automatically pages through all the results and concatenates them.

A talk might be associated with multiple speakers. Each speaker should get an email with an upload link. There were submissions that are not really talks in the traditional sense, so people shouldn’t get an email. The query for jq looks like that:

[.results[] | select((.submission_type[] | contains("Workshop")) or (.submission_type[] == "Anwendertreffen / BoF") | not) | { code: .code, speaker: .speakers[].code, title: .title, submission_type: .submission_type[]}]

The submissions contain only the speaker IDs and names, but not other details like their email address. We query the speakers API (e.g. https://pretalx.com/api/events/democon/speakers/) and post-process the data again with jq, as we care about their email addresses.

You can find all the requests and filter in the email_upload_links/upload_talks_to_seafile.sh script.

Seafile

Creating and upload link is a two-step process in Seafile. First create the directory, then creating a public accessible upload link for the directory. The directories are named after the pretalx ID of the talk (Full script for creating directories).

Creating emails

After acquiring the data, the next step is to process the data and creating the individual emails. Combining the data is done with the combine_talks_speakers_upload_links.py script, where the output is again post-processed with jq. The data_to_email.py script takes that data output and a template file to create the actual email as files. The template file is used as a Python format string, where the variables a filled with the data provided.

Those email files are then posted to pretalx, so that we can send them over their email system. That step is more complicated as currently there is no API in pretalx to do that. I logged in through the web interface and manually added a new email, while having the developer tools open. I then copied the POST request “as cURL” to have a look at the data it sent. There I manually extracted the session and cookie information in order to add emails from the command line. The script that takes the pre-generated emails and puts them into pretalx is called email_to_pretalx.sh.

Reviewing the uploaded videos

Once a video is uploaded, it gets reviewed. The idea was, that the speakers don’t need to care too much about the start and the end of the video, e.g. when they start the recording and there is a few seconds of silence while switching to the presentation. The reviewer will cut the beginning and end of the video and also convert it to a common format.

We wanted to preserve the original video quality, hence we use LosslessCut and converted it then to the Matroska format. The reviewers would also check that the video isn’t longer than the planned slot.

See the copy_uploads directory for all the scripts that are needed for this step.

pretalx

The reviewers get a file with things to check for each video file. We get the needed metadata again from pretalx and post-process it with jq. As above for the emails, there is again a template file which (this time) generates Markdown files with the information for the reviewers. The full script is called create_info_files.sh.

Seafile

Once videos are uploaded they should be available for the reviewers. The uploaded files are the primary source, hence it makes sense to always make copies of the talks, so that the original uploads are not lost. The sync_files_and_upload_info.sh script copies the talks into a new directory (together with the information files), which is then writeable for the reviewers. They will download the file, review it, cut it if needed, convert it to Matroska and upload it again. Once uploaded, they move the directory into one called fertig (“done” in German) as an indicator that no one else needs to review it.

I run the script daily as a cron job, it only copies the new uploads. Please note that it only checks the existence on a directory level. This means that if a talk was reviewed and a speaker uploads a new version of the talk, it won’t be copied. That case didn’t often happen often and speakers actually let us know about it, so it’s mostly a non-issue (also see the miscellaneous scripts section for more).

Last step is that someone looks through the filled out markdown files to check if everything was alright, respectively make sure that e.g. the audio volume is fixed, or asks the speaker for a new upload. The then checked videos are moved to yet another directory, which then contains all the talks that are ready to be streamed.

Re-org files for schedule

So far, the video files were organized by directories that are named after the pretalx ID of the talk. For running the conference we used OBS for streamer. The operator would need to play the right video at the right time. Therefore, it makes sense to sort them by the schedule. The cut_to_schedule.sh script does that re-organization, which can be found in the cut_to_schedule directory.

pretalx

To prevent accidental inconsistencies, the root directory is named after the current version of the pretalx schedule. So if you publish a new version of the schedule and run the script again, you’ll get a new directory structure. The video files still have an arbitrary name, chosen by the uploader/reviewer, we want a common naming scheme instead. The get_filepath.py script creates such a name that also sorts chronologically and contains all the information the OBS operators need. The current scheme is <room>/day<day-of-the-conference>/day<day-of-the-conference>_<day-of-the-week>_<date>_<time>_<pretalx-id>_<title>.mkv.

Seafile

The directories do not only contain the single final video, but also the metadata and perhaps the original video or a presentation. The file we actually copy is the *.mkv file which was modified last, which will be the cut video. The get_files_to_copy.sh script creates a list of the files that should be copied, it will only list the files that weren’t copied yet (based on the filename). The copy_files.sh script does the actual copying and is rather generic, it only depends on a file list and Seafile.

Easily downloadable files

Seafile has a feature to download a full directory as zip file. I originally planned to use that. It turns out that the size of the files can be too large, I got the error message Unable to download directory "day1": size is too large.. So I needed to provide another tool, as I didn’t want that people would need to click and download all individual talks.

The access to the files should as easy as possible, i.e. the operators that need the files shouldn’t need a Seafile account. As the videos also shouldn’t be public, the compromise was using a download link secured with a password. This means that an authentication step is needed, which isn’t trivial. The download_files.sh script does login and then downloads all the files in that directory. For simplicity, it doesn’t do recursively. This means that any stage would need to run this script for each day.

I also added a checksum check for more robustness. I created those checksums manually with running b2sum * > B2SUMS in each of the directories and then uploaded them to Seafile.

List of live/pre-recorded talks

Some talks are recorded and some are live, the list_recorded_talks.py script, creates a Markdown file that contains a schedule with that information, including the lengths of the talks if they are pre-recorded. This is useful for the moderators to know how much time for questions will be. At the FOSSGIS we have 5 minutes for questions, but if the talk runs longer, there will be less time.

You need the schedule and the length of the recorded talks. This time I haven’t fully automated the process, it’s a bit more manual than the other steps. All scripts can be found in the list_recorded_talks directory.

Get the schedule:

curl https://pretalx.com/<your-conference>/schedule.json > schedule.json

For getting the lengths of the videos, download them all with the download script from the Easily downloadable files section above. Then run the get_length.sh script in each of the directories and output then into a file. For example:

cd your-talks-day1
/path/to/get_lengths.sh > ../lengths/day1.txt

Then combine the lengths of all days into a single file:

cat ../lengths/*.txt > ../talk_lengths.txt

Now you can create the final schedule:

cd ..
python3 /path/to/list_recorded_talks.py schedule.json talk_lengths.txt

Here’s a sample schedule from the FOSSGIS 2021.

Miscellaneous Scripts

Speaker notification

The speakers didn’t get feedback whether their video was correctly uploaded/processed (other than seeing a successful upload in Seafile). A short time before the conference, we were sending out the latest information that speakers needs to know. We decided to take the chance to also add information whether their video upload was successful or not, so that they can contact us in case something with the upload didn’t go as they expected (there weren’t any issues :).

It is very similar to sending out the email with the upload links. You get the information about the speakers and talks in the same way. The only difference is we now also need the information whether the talk was pre-recorded or not. We get that from Seafile:

curl --silent -X GET --header 'Authorization: Token <seafile-token>' 'https://seafile.example.org/api2/repos/<repo-id>/?p=/<dir-with-talks>&t=d'|jq --raw-output '.[].name' > prerecorded_talks.txt

The full script to create the emails can be found at email_speaker_final.sh. In order to post them to pretalx, you can use the email_to_pretalx.sh script and follow the description in the creating emails section.

Number of uploads

It could happen that people upload a new version of the talk. The current scripts won’t recognize that if a previous version was already reviewed. Hence, I manually checked the directories for the ones with more than one file in it. This can easily be done with a single curl command to the Seafile HTTP API:

curl --silent -X GET --header 'Authorization: Token <seafile-token>' 'https://seafile.example.org/api2/repos/<repo-id>/dir/?p=/<dir-with-talks>&t=f&recursive=1'|jq --raw-output '.[].parent_dir'|sort|uniq -c|sort

The output is sorted by the number of files in that directory:

  1 /talks_conference/ZVAZQQ
  1 /talks_conference/DXCNKG
  2 /talks_conference/H7TWNG
  2 /talks_conference/M1PR79
  2 /talks_conference/QW9KTH
  3 /talks_conference/VMM8MX

Normalize volume level

If the volume of the talk was too low, it was normalized. I used ffmpeg-normalize for it:

ffmpeg_normalize --audio-codec aac --progress talk.mkv

Conclusion

Doing all this with scripts was a good idea. The less manual work the better. It also enabled me to process talks even during the conference in a semi-automated way. I created lots of small scripts and sometimes used just a subset of them, e.g. the copy_files.sh script, or quickly modified them to deal with a special case. For example, all lightning talks of a single slot (2-4) were merged together into one video file. That file of course then isn’t associated with a single pretalx ID any more.

During the conference, the volume level of the pre-recorded talks was really different. I think for next time I’d like to do some automated audio level normalization after the people have uploaded the file. It should be done before reviewers have a look, so that they can report in case the normalization broke the audio.

The speakers were confused whether the upload really worked. Seafile doesn’t have an “upload now” button or so, it does it’s JavaScript magic once you’ve selected a file. That’s convenient, but was also confusing me, when I used it for the first time. And if you reload the page, you also won’t see that something was uploaded already. So perhaps it could also be automated that speakers get an email “we received your upload” or so.

Overall I’m really happy how the whole process went, there weren’t major failures like lost videos. I also haven’t heard any complaints from the people that needed to use any of the videos at any stage of the pipeline. I’d also like to thank all the speakers that uploaded a pre-recorded video, it really helped a lot running the FOSSGIS conference as smooth as it was.

Introduction

In my case I wanted to pass some bytes into the WASM module, do some processing and returning some other bytes. I found all information I needed in this excellent A practical guide to WebAssembly memory from radu. There he mentions the WebAssembly multi-value proposal and links to a blog post from 2019 called Multi-Value All The Wasm! which explains its implementation for the Rust ecosystem.

As it's from 2019 I just went ahead and thought I can use multi-value returns in Rust.

The journey

My function signature for the FFI looks like this:

pub extern "C" fn decode(data_ptr: *const u8, data_len: usize) -> (*const u8, usize) { … }

When I compiled it, I got this warning:

warning: `extern` fn uses type `(*const u8, usize)`, which is not FFI-safe
 --> src/lib.rs:2:67
  |
2 | pub extern "C" fn decode(data_ptr: *const u8, data_len: usize) -> (*const u8, usize) {
  |                                                                   ^^^^^^^^^^^^^^^^^^ not FFI-safe
  |
  = note: `#[warn(improper_ctypes_definitions)]` on by default
  = help: consider using a struct instead
  = note: tuples have unspecified layout

Multi-value returns are certainly not meant for C APIs, but for WASM it might still work, I thought. Running wasm2wat shows:

(module
  (type (;0;) (func (param i32 i32 i32)))
  (func $decode (type 0) (param i32 i32 i32)
…

This clearly isn't a multi-value return. It doesn't even have a return at all, it takes 3 parameters, instead of the 2 the function definition has. I found an issue called Multi value Wasm compilation #73755 and was puzzled why it doesn't work. Is this a regression? Why did it work in that blog post from 2019? I gave the Multi-Value All The Wasm! blog post another read, and it turns out it explains all this in detail (look at the wasm-bindgen section). Back then it wasn't supported by the Rust compiler directly, but by wasm-bindgen.

So perhaps I can just use the wasm-bindgen command line tool and transform my compiled WASM binary into a multi-value return one. There is a command-line flag called WASM_BINDGEN_MULTI_VALUE=1 to enable that transformation. Sadly that doesn't really work as it needs some interface-types present in the WASM binary (which I don't have).

Thanks to open source, the blog post about the implementation of the transformation feature and some trial an error, I was able to extract the pieces I needed and created a tool called wasm-multi-value-reverse-polyfill. I didn't need to do any of the hard parts, just some wiring up. I was now able to transform my WASM binary into a multi-value return one simply by running:

$ multi-value-reverse-polyfill ./target/wasm32-unknown-unknown/release/wasm_multi_value_retun_in_rust.wasm 'decode i32 i32'
Make `decode` function return `[I32, I32]`.

The WAT disassembly now looks like that:

  (type (;0;) (func (param i32 i32) (result i32 i32)))
  (type (;1;) (func (param i32 i32 i32)))
  (func $decode_multivalue_shim (type 0) (param i32 i32) (result i32 i32)
…

There you go. There is now a shim function that has the multi-value return, which calls the original method. I can now use my newly created WASM binary with WebAssembly runtimes that support multi-value returns (like Wasmer or Node.js).

Conclusion

With wasm-multi-value-reverse-polyfill I'm now able to create multi-value return functions with the current Rust compiler without depending on all the magic wasm-bindgen is doing.

The problem

When you do an npm install heavy module deduplication and hoisting, which doesn't always behave the same way in all cases. For example if you npm link a package, the resulting node_modules tree is different. This may lead to unexpected runtime errors.

It happened to me recently and I thought I use exactly this real world example to illustrate that problem and a possible solution to it.

Real world example

Preparations

Start with cloning the js-ipfs-mfs and js-ipfs-unixfs-importer repository:

$ git clone https://github.com/ipfs/js-ipfs-mfs --branch v0.12.0 --depth 1
$ git clone https://github.com/ipfs/js-ipfs-unixfs-importer --branch v0.39.11 --depth 1

Our main module is js-ipfs-mfs and let's say you want to make local changes to js-ipfs-unix-importer, which is a direct dependency of js-ipfs-mfs.

First of all you of course make sure that currently the tests pass (we just run a subset, to get to the actual issue faster). I'm sorry that the installation takes so long and so much space, the dev dependencies are quite heavy.

$ cd js-ipfs-mfs
$ npm install
$ npx mocha test/write.spec.js
…
  53 passing (4s)
  1 pending

Ok, all tests passed.

Reproducing the issue

Before we even start modifying js-ipfs-unix-importer, we link it and check that the tests still pass.

$ cd js-ipfs-unixfs-importer
$ npm link
$ cd ../js-ipfs-mfs
$ npm link ipfs-unixfs-importer
$ npx mocha test/write.spec.js
…
  37 passing (2s)
  1 pending
  16 failing
…

Oh, no. The tests failed. But why? The reason is deep down in the code. The root cause is in the [hamt-sharding] module and it's not even a bug. It just checks if something is a Bucket:

static isBucket (o) {
  return o instanceof Bucket
}

instanceof only works if both instances we check on came from the exact same module. Let's see who is importing the hamt-sharding module:

$ npm ls hamt-sharding
ipfs-mfs@0.12.0 /home/vmx/misc/protocollabs/blog/when-npm-link-fails/js-ipfs-mfs
├── hamt-sharding@0.0.2
├─┬ ipfs-unixfs-exporter@0.37.7
│ └── hamt-sharding@0.0.2  deduped
└─┬ UNMET DEPENDENCY ipfs-unixfs-importer@0.39.11
  └── hamt-sharding@0.0.2  deduped

npm ERR! missing: ipfs-unixfs-importer@0.39.11, required by ipfs-mfs@0.12.0

Here we see that ipfs-mfs has a direct dependency on it, and an indirect dependency through ipfs-unixfs-exporter and ipfs-unixfs-importer. All of them use the same version (0.0.2), hence it's deduped and the instanceof call should work. But there's also an error about an UNMET DEPENDENCY, the ipfs-unixfs-importer module we linked to.

To make it clear what's happening inside Node.js. When you require('hamt-sharding') from the ipfs-mfs code base, it will load the module from the physical location js-ipfs-mfs/node_modules/hamt-sharding. When you require it from ipfs-unixfs-importer it will be loaded from js-ipfs-mfs/node_modules/ipfs-unixfs-importer/node_modules/hamt-sharding resp. from ipfs-unixfs-importer/node_modules/hamt-sharding, as js-ipfs-mfs/node_modules/ipfs-unixfs-importer is just a symlink to a symlink to that directory.

When you do a normal installation without linking, you won't have this issue as hamt-sharding will be properly deduplicated and only loaded once from js-ipfs-mfs/node_modules/hamt-sharding.

Possible workarounds that do not work

Though you still like to change ipfs-unixfs-importer locally and test those changes with ipfs-mfs without breaking anything. I had several ideas on how to workaround this. I start with the ones that didn't work:

1. Just delete the js-ipfs-unixfs-importer/node_modules/hamt-sharding directory. The module should still be found in the resolve paths of ipfs-mfs. No it doesn't. Tests fail because hamt-sharding can't be found.
2. Global linking runs an npm install when you run the initial npm link. What if we remove the js-ipfs-unixfs-importer/node_modules completely and symlink to the module manually. That also doesn't work, the hamt-sharding module also can't be found.
3. Install ipfs-unixfs-importer directly with a relative path (npm install ../js-ipfs-unixfs-importer). No, that doesn't work either, it will still have its own node_modules/hamt-sharding, it won't be properly deduplicated.

There must be a way to make local changes to a module and testing them without publishing it each time. Luckily there really is.

Working workaround

I'd like to thank my colleague Hugo Dias for this workaround that he has been using for a while already.

You can just replicate what a normal npm install <package> would be doing. You pack the module and then install that packed package. In our case that means:

$ cd js-ipfs-mfs
$ npm pack ../js-ipfs-unixfs-importer
…
ipfs-unixfs-importer-0.39.11.tgz
$ npm install ipfs-unixfs-importer-0.39.11.tgz
…
+ ipfs-unixfs-importer@0.39.11
added 59 packages from 76 contributors and updated 1 package in 31.698

Now all tests pass.

This is quite a manual process. Luckily Hugo created a module to automate exactly that workflow. It's called connect-deps.

Conclusion

Sometimes linking packages doesn't create the same structure of modules and you need to use packing instead. To automate this you can use connect-deps.

Intro

When I first saw #ShowYourStripes I immediately felt in love (thanks Stefan Münz for tweeting about it). I think it's a great and simple visualization by Ed Hawkins of what we are currently facing when it comes to climate change. You don't need to scroll through long tables or figure out the axis on some diagram. You can simply see that there's something massively changing.

After playing around a bit with the cities available on the #ShowYourStripes website I wanted to do the same for the city I live in, Augsburg, Germany. I looked at the website's source code first, in hope that it dynamically creates the data from some JSON or so. That isn't the case. I then searched Twitter, GitHub and the Web if I can find any related open source project. I wouldn't want to spend time figuring out the parameters that were used to create those. After all, I wanted mine to look exactly like those.

Luckily I found a Tweet from Zeke Hausfather saying that he could create those. I then asked him if he could please release the source code. And just 7h later he did.

Creating your own stripes

Now it's time for a quick tutorial on how you can create your own #ShowYourStripes with that source code.

Prerequisites

I did those steps on a Debian system that had the most common tools installed (like Python3, or Wget). I'm using Pipenv for installing the required Python packages, but you can use any other package management tool for Python.

Let's get the data file with the global temperature values first. It's 200MB so it might take a while.

wget http://berkeleyearth.lbl.gov/auto/Global/Gridded/Complete_TAVG_LatLong1.nc

Now retrieve the source code:

$ wget https://raw.githubusercontent.com/hausfath/scrape_global_temps/master/City%20Warming%20Strips%20.ipynb -O showyourstripes.ipynb

In order to run the script, we need to get a few Python packages first:

$ pipenv install matplotlib nbconvert netcdf4 numpy_indexed pandas

Running the script

The original script is a Jupyter Notebook, so we convert it to a plain Python script (you can ignore the warnings):

$ pipenv run jupyter-nbconvert --to python showyourstripes.ipynb

Next we need to make some changes to the showyourstripes.py file so that it works on your machine and plots the stripes for your location. We work on the current directory, so you can comment out changing the directory:

#os.chdir('/Users/hausfath/Desktop/Climate Science/GHCN Monthly/')

The other changes we need is the location the stripes should be plotted from. Here I use the values for Augsburg, Germany. Use your own values there. When I don't know the coordinates of my location, I usually check Wikipedia. In the top right corner of an article you can find the coordinate of a place (if it has one attached). If you click on those you get to the GeoHack page of the article. There on the top right you can find the coordinate in decimals in lat/lon order. In my case it's "48.366667, 10.9".

savename = 'augsburg'

lat = 48.366667
lon = 10.9

Now you're ready to run the script:

pipenv run python showyourstripes.py

Now you should have an output file called augsburg.png in the same directory which contains the stripes.

Conclusion

Have fun creating your own #ShowYourStripes. Thanks again Zeke Hausfather for making and publishing the source code so quickly.

Zunächst ein bisschen Hintergrund zur Urheberrechtsreform. Die Reform sieht mehrere Änderungen vor, die durchaus nicht alle schlecht sind, einen guten Überblick zum Thema ist [dieser Blog-Eintrag von Julia Reda]. Noch vor der Abstimmung gab es einen, meines Erachtens, sehr guten Gastbeitrag von Dorothee Bär bei der Main Post, bei dem auf die negativen Auswirkungen der Reform eingegangen wird.

Darüber hinaus gab es auch die Warnung vom UN-Sonderberichterstatter zur Meinungsfreiheit David Kayne davor, dass die Reform zur Einschränkung der Meinungsfreiheit führen wird.

Bereits einen Tag nach der Zustimmung des EU Parlaments am 26. März hatte der französische Kulturminister Franck Riester angekündigt, dass Frankreich in Filtertechnologie investieren will. Es wird also wohl trotz der Protokollnotiz Deutschlands zu Uploadfiltern kommen.

Was mich bei der gesamten Debatte wirklich gestört hat, war die Unwissenheit vieler Beteiligter. Auch ich habe Fehler gemacht, diesen aber umgehend korrigiert. Dabei ist es eben hilfreich auch die Argumente der Gegenseite zu hören. Besonders bei der Debatte im EU Parlament direkt vor der Abstimmung (direkt als Video) wurden deutlich, wie viele Abgeordnete nicht wirklich verstanden haben, um was es genau geht, bzw. welche Folgen die Reform hat. Es gab sogar plumpe Angriffe, die mit der eigentlichen Sache zu tun hatten. Es kam bei den Befürwortern wohl nicht an, dass auch die Gegner, wie ich, eine Urheberrechtsform wollen, es geht lediglich um deren Umsetzung. Der Redebeitrag von Julia Reda war (wie so oft) hervorragend. Sie fasst die Faktenlage noch einmal kurz zusammen und beschreibt auch den Frust der Teilnehmer der Massenproteste. Zur möglichen Folge der Politikverdrossenheit gibt es auch einen sehr guten Kommentar auf tagesschau.de.

Spannend ist auch noch die Frage, wer die eigentlichen Gewinner dieser Reform sind. Die Befürworter haben immer die Urheber als Gewinner der Reform ins Feld geführt. Allerdings handelte es dabei immer um Urheber die von Verwertungsgesellschaften vertreten werden. Es wurde dabei außer Acht gelassen, dass es gerade im Internetzeitalter eine Vielzahl von anderen Möglichkeiten gibt Urheber zu sein. Dazu gibt es zwei nette Geschichten, einmal der Versuch als Privatperson für Fotos Vertreten zu werden, das andere Mal als Videoproducer auf YouTube. Beides ist derzeit nicht möglich.

Ich hatte mich außerdem daran beteiligt bei den Brüsseler Büros der Abgeordneten anzurufen. Dabei war das Feedback sehr verschieden. Es reichte von einem freundlichen „ich werde dies so weitergeben“, über „die Abgeordnete stimmt wie ihre Kollegin, entgegen der Mehrheit ihrer Fraktion, bzw. der Bundesfraktion“ bis zu „es rufen so viele an, ich werde deshalb nicht mit Ihnen sprechen, die Abgeordnete wird sich aber eine fundierte Meinung bilden“ (im Endeffekt dann aber gar nicht abstimmen).

Zum Schluss möchte ich mich bei Allen bedanken die so hart dagegen gekämpft haben, Demos organisiert und natürlich auch den zahlreichen Teilnehmern.

There are many arguments against the EU Copyright Directive (more correct Directive on Copyright in the Digital Single Market) some I agree with, some I don't. Hence, here's my take on why I think that directive should be stopped. Short versions is: it strengthens the big platforms and weakens/destroys the small ones.

My hope is that this blog post will get more people interested in that topic and hopefully make you join the European wide protests on Saturday March 23rd 2019. If you want to join, there's an interactive map of all known protests created by the folks from stopACTA2.

Intro

It is confusing that platforms like YouTube are against the directive, it sounds like they have a lot to lose, hence they try everything they can against it. For me, this is normally a sign, that such a directive is exactly what it should do.

But it this case, it's not. YouTube will surely have its own reasons being against it. But what is more important for me is, that if the directive is approved by the European Parliament, the small platforms will almost have no chance to survive.

Why small platforms will die

There are exceptions in the directive for some platforms. You can find those in the full text of the final version at paragraph (38b), page 36. But those exceptions won't help all smaller platforms. For example a discussion board which is older than 3 years and has advertising to cover the server costs wouldn't be excluded. They would be liable for every copyright infringement.

It could be a small as a profile picture, let's say yours is Luke Skywalker. That platform could block custom profile pictures, but that still won't be enough. Someone could post some copyrighted text. But how would you make a discussion board without the users being able to post text? So the only way to not being liable would be to check for all infringements (how would you do that?), or close the platform.

Outro

I tried to keep it intentionally short and highlight the issue that matters to me most. Of course there's a lot more issues regarding the EU Copyright Directive, so if you want to know more, go to websites like savetheinternet.info, stopACTA2 or Julia Reda's website who is a Member of the European Parliament and puts lots of efforts in explaining and spreading the word on why the directive should be stopped (also follow her on Twitter). Thanks a lot Julia for doing such an amazing work!