Open Source Computing and GIS in the UK

Travels in a digital world

Don’t Be Afraid to Commit

| Comments

Back in June, which seems a long time ago now, we (OSGeo:UK) ran a mini FOSS4G event at the Ordnance Survey offices in Southampton, UK. It was a big success, well attended, and everyone seemed to enjoy themselves, which is always nice. This is not a post about the event, per se (perhaps in retrospect going straight on to another event the week after was a bad idea). Others have posted about this, and there’s always the #foss4guk hashtag on twitter if you’re interested. However, along with chairing the event, I foolishly decided to run a workshop around a topic that had been taking shape in my mind for a while- namely how to get new users of open source software to a point where they are comfortable contributing to the packages they use, as well as simply using them.

I began this workshop looking at my own perspective. I’m no more than an arm-chair coder at best (Python and SQL are fine but the rest requires some serious googling and sucking up to colleagues). I can, however, find and report bugs, and contribute to documentation, and that’s my way of giving back. This may not require an in depth knowledge of java or C++, and all the additional skills they require but there’s still a barrier to entry- namely navigating online repositories such as GitHub, and producing useful bug reports. Helping with documentation involves learning about ReadTheDocs and ReStructured Text, and proper bug reporting, where you try and isolate the cause of the problem requires virtualisation of some kind. Suddenly you’re looking at a fairly large toolkit, before you start with bug fixes and other code contributions!

Then, a few months ago some colleagues went to PyCon UK and came back with stories of the wonderous Don’t be afraid to commit workshops, so I decided to come up with something vaguely similar, but focused more on GitHub and bug reporting, and less on Python. I also included an adapted version of the wonderful How to Report a Bug site, but again focusing on open source geospatial projects. You can find my workshop attempt on GitHub at http://archaeogeek.github.io/foss4gukdontbeafraid/.

Some thoughts for those considering running something similar…

In retrospect it was incredibly ambitious to try and get through all the above in 2 hours- it was only possible by setting some fairly strict workshop prerequisites, which most people actually took note of. Given that this was a “Bring your own device” conference, I ran the Git sections using the command line tools rather than any of the graphical user interfaces. Although there was some resistance to this, it was the only sane approach because I had people using Windows, Mac, Linux and an Android tablet in the class! The bit that actually caused the most trouble was setting the default text editor for Git to use. If I was running the class again I would probably omit this section and use the short-hand method of adding commit messages at the command line instead.

It’s also really hard to manage a shared repository for people to use for trying out pull requests- I had a simple text file (generated using the superb hipster ipsum site to generate it), but that caused all sorts of conflicts when multiple people issued pull requests all at the same time. There are alternative approaches to this (countless online, interactive git tutorials) but they weren’t the approach I wanted to use in this workshop. If I’d had more time to prepare I would have come up with a more complex file, perhaps with some specific errors in it, that people could fix without conflicting with each other.

Finally, we barely had time to look at ReStructured Text, or virtualisation, and it probably wasn’t worth even trying to cover them in the same workshop!

Having said all that, I know of at least one workshop attendee who has used his new GitHub skills to good effect, and was recently seen contributing to some documentation, so that’s a start. It would be good if there were more workshops of this kind at open source conferences, not necessarily based on mine, but feel free to borrow/improve it if you like!

PortableGIS 5.6

| Comments

I’m pleased to announce the latest release of Portable GIS. This version (v5.6) has the following changes:

  • QGIS 2.14.1 LTR
  • By popular demand: Geoserver 2.8

You can download the setup exe and the md5 checksum here.

Older versions are still available but have been archived to avoid confusion.

As always, please let me know of any problems via the Portable GIS google group.

Note that I will shortly be publicising a GitLabs repository for the changed files, along with developer and user documentation, to allow people to roll their own versions or contribute to development. This work is nearly complete, so watch this space!

AGI Foresight 2020

| Comments

Over the summer I was asked to contribute a piece on what I thought the future of “Open Geospatial” would be in 2020, for the AGI (Association for Geographic Information) and their Foresight Study on the Geospatial Industry in 2020. This follows on from an earlier study in 2010, which attempted to predict what the state of the Geospatial Industry would be like in 2015, and that I also contributed to.

You can find a copy of the final study at the AGI website (when it’s launched, later this year). I’ve placed a copy of my contribution here on archaeogeek as a pdf.

I had what turned out to be an over-optimistic idea about crowd-sourcing some predictions, and asked on twitter and on the OSGeo Discuss Mailing List. Unfortunately, perhaps due to bad timing (everyone was off having fun at FOSS4G at the time), I had very few responses, however many thanks to Jody Garnett, who did take the time to come up with some ideas, and to my colleague Matt Walker who read through the final article. The rest is all mine, faults, mistakes, wrong ideas and all, so enjoy.

Hacking With APIs

| Comments

Since Astun Technology has quite a geographically distributed workforce (OK, we are all in the UK, but still), we make the effort to get together every three months or so for meetings, merriment, and some geeky hacking. Our “hackastun” events are a great chance for those of us who deal more with the finished software to do a bit of coding, even if it is just messing around with python.

For our most recent hack, I had a goal of investigating how we might start integrating some interesting new APIs with our software, iShare. This would use a relatively new feature in iShare, called Dynamic Datasources. These do exactly what they say on the tin, and allow us to call external datasources at the time the iShare page is loaded, to get up to date information.

My hopefully willing accomplice was new Astun hire (but not new to open source Geo) Ian Turton. As is traditional with these things, the first half of the hack was spent doing some serious yak-shaving getting a working iShare development environment set up on his linux laptop. However, contrary to other hacks, once we got that set up, progress was swift.

The first API we looked at was the UK Transport API. Developing a script to pass a unique property reference number (UPRN) from iShare, convert it to a Latitude/Longitude string, send it to the API and return the nearest bus stops was pretty easy- we used pyproj to convert a British National Grid Easting/Northing to Lat/Lon, and Untangle to wrangle any XML we came across.

The next stage for this will be to extend the script to make a second call to get the bus timetable for the stops. Train and tube information are also available from the API, as well as some journey planning, so no doubt we’ll be exploring this API further.

On the train on the way home I also looked at the What3Words API to see how straightforward it would be to work with. It turns out that it’s pretty easy, even for an entry-level pythonista like myself, so now we can offer the What3Words address for a given property in iShare.

It’s important when working with APIs to assess their resilience and sustainability, you don’t want one that you’re relying on to close down suddenly. It should also use open standards, of course. However, I think they are becoming increasingly important as they reduce the need for people to hold, manage and update the data themselves, which is a big plus from an Astun/iShare perspective. They are also a lot of fun to work with, if you’re a geo-geek!

Extending Portable GIS

| Comments

I was asked recently how someone could add new libraries, python programmes etc to their copy of Portable GIS. It’s fairly simple, but here’s a quick guide. I’ll use ogr2osm as an example.

Portable GIS 5.2

| Comments

I’m pleased to announce the latest release of Portable GIS. This version (v5.2) has only a couple of changes:

  • QGIS 2.8 (I’m going to try and do a release to coincide with each long-term release of QGIS)
  • Loader has been updated to the latest version

You can download the setup exe and the md5 checksum here.

Older versions are still available but have been archived to avoid confusion.

As always, please let me know of any problems via the Portable GIS google group.

This version also marks the start of an attempt to create a repository for Portable GIS, which will hold the additional and/or changed files and instructions for rolling your own version. This is something that I have wanted to do for some time, but it’s not ready for release yet. Watch this space!

Archaeology, APIs and Accessibility

| Comments

At the end of November (yes, I know it’s a while ago, life got in the way) I was involved in an event called #HackLancaster, part of a wider project in my home town to try and raise interest in the heritage of the old city area beyondthecastle. This was, I think, a great success on several levels, but it also throw up some issues with archaeological and geospatial data that I hadn’t really considered. This was going to be a much longer post, going through everything I think is wrong with trying to put archaeological data on-line as is, but since it’s been sitting in draft form, I think it’s time to just bite the bullet and get something out there…

So without further ado- my job for #HackLancaster was to take a stuffy old access database, some shape files and some geotiffs, and make this available as an api. Here is the toolkit I ended up using:

  • Various scripts for converting access databases to postgresql- collected and tinkered with here and here.
  • csvkit for dumping csv to postgresql with the bare minimum of effort.
  • Dirt Simple PostGIS API for opening up spatial functionality in my postgresql database over http.
  • PGRest for opening up standard non-spatial functionality in the database over http (had I been doing this now, I might have tried PostgREST instead)
  • Geoserver for serving historic base maps over WMS, and spatial data over WFS/WMS as needed.
  • Gitbook for documentation. You can see my documentation here. Live documentation meant it could be updated and added to as we went along.

Insight Number 1: The one thing I’d take away from this part of the project is that all of these pieces of software were easy to set up, use and configure. As a toolkit for making this kind of data available quickly and easily, you can’t go far wrong here.

Insight Number 2: You need to do more than simply make the data available via an api. Complex relational data structures and archaic terminology are a barrier to non-specialists, no matter how technically competent they are. What’s the solution here- collapse the relational structure somehow? It would be nice to have an automated method in case of updates to the source data.

Insight Number 3: Technologies that seemed like a really obvious fit to a GIS person are not so obvious to everyone else- the historic mapping WMS was completely ignored in favour of Google Maps or MapBox and the spatial RESTful endpoint was only used to convert data from British National Grid to Spherical Mercator, under some duress. Functionality that could have been provided by the spatial RESTful end point was generally fudged.

Insight Number 4: It’s important to make committing code part of the hack. While we asked the teams nicely, not one of them made their code available after the event. In retrospect, since we gave out prizes, we should have made committing their code a prerequisite for winning a prize!

Finally, despite my slight gripes above- the teams involved came up with some extremely nice visualisations of the data- if I can persuade them to make them publicly available I’ll definitely put out some links.

PostGIS Day 2014

| Comments

On 20th November this year we held the first PostGIS “Show-and-Tell” Day at the HQ of the British Computing Society. This was the first “official” OSGeo:UK event since hosting FOSS4G 2013 in Nottingham last year- it took us that long to recover from the stress!

We deliberately chose to make the event extremely light and informal, with the minimum of organisation, and this seemed to work pretty well. We had a range of speakers doing both longer talks and lightning talks, and then a session at the end where we discussed useful tools and add-ons for working with PostGIS.

The only caveats we placed on speakers were that the talk should (obviously) include PostGIS or PostgreSQL in some way, and that overt sales pitches would be considered uncool. Luckily our speakers obliged! There was also some really good audience interaction.

Highlights for me included Foreign Data Wrappers, for getting at information stored in other databases or behind other apis, but through PostgreSQL, and Raster processing. I was particularly excited by the Twitter FDW, but it turns out that it doesn’t work with the latest version of the twitter api, and it doesn’t look like a particularly well-maintained piece of code. You can see links to all the talks, and notes from the discussion that we had at the end, at our OSGeo:UK GitHub repository for the event: http://osgeouk.github.io/pgday/.

It transpired that there were a number of other events globally, also celebrating #PostGISDay, and perhaps next year it would be good to coordinate these better.

From an OSGeo:UK perspective, we have plans to hold more of these light-touch events, on a range of subjects (ideas gratefully received). Personally I would like to have charged a nominal fee for attending the event, not to make a profit but so that we could make a small donation back to PostGIS but charging money makes the whole process more complicated! We were extremely lucky in this case to get the venue for free, courtesy of the BCS Location Information Group, and to get the food sponsored by Astun Technology.

All in all, a really fun event, both to organise and to attend- thanks to all the speakers and attendees, and here’s to another event quite soon.

New Portable GIS Releases

| Comments

I’m pleased to announce not one, but two new releases of Portable GIS! The first, version 4.2, contains QGIS 2.4 and PostGIS 1.5 and will be the last release to include that version of PostGIS.

The second, version 5, contains QGIS 2.4 and PostGIS 2.1 and all future releases will be based on this. Get them here:

There are two important things about these two releases. The first is that the download size is almost twice what it was previously- this is out of my control and is due to the increased size of the QGIS installation. The second is that version 5.0 should be considered a beta release as it’s very much un-tested. If there are any problems let me know via the Portable GIS google group.

You might also be interested to see a short presentation I did about Portable GIS at GeoMob London in September: http://archaeogeek.github.io/geomob-portablegis/#/

OSGIS 2014

| Comments

Two weeks ago now saw the return of the OSGIS conference in Nottingham, after a year off in 2013 for FOSS4G. I think there had been mixed feelings about this event; those of us heavily involved in the organisation of FOSS4G 2013 had taken a back seat this year, and with FOSS4G 2014 imminent in Portland, it was clearly going to be a smaller scale get together.

I have to say that overall, my impression is that small is good! Small allows you to chat to everyone, see everything you want to see, and generally enjoy, rather than rush around like a mad man or woman. It was nice to see some new faces, and to see a number of papers from local government and business, belying the idea that OSGIS is primarily an academic event. Thanks as always to the chaps at Nottingham for organising.

Astun had a strong showing at the event, with two workshops and two presentations. My colleague Matt Walker did a workshop on OpenLayers3 and Leaflet, and I did one on WPS and PgRouting (a beginners guide). I did a quick introduction to Portable GIS, and another colleague Antony Scott did a comparison to web servers. You can see the workshops at the Astun Technology GitHub pages. As a slight techy aside, Matt and I collectively decided to try GitBook for preparing our workshops, and we’re both very impressed. If you’re interested in a cheeky workflow for pushing a gitbook directly to GitHub gh-pages, see this gist.

Steven Feldman, another Astun employee/adviser also did a talk entitled “There’s no such thing as a free lunch”, a continuation of an emerging theme in open source geospatial at the moment on getting companies to contribute more, or at the very least acknowledge and thank the open source components they use. As always this was very thought provoking (although I worry a little that making people feel responsible for the software they use might potentially back fire)– you can find Steven’s blog post on the talk here.

OSGIS has always had a strong relationship with the OSGeo:UK local chapter, but unfortunately that’s been a bit inactive since FOSS4G- sometimes it’s hard to know what to do next when you’ve fulfilled one of your primary goals! We’re going to have a go at rebooting the local chapter though now- a more detailed post on this will follow soon. The short version is, I’m back co-chairing after a two year absence, and we’ve got some good ideas going forward. Watch this space!