Open Source Computing and GIS in the UK

Travels in a digital world

AGI Foresight 2020

| Comments

Over the summer I was asked to contribute a piece on what I thought the future of “Open Geospatial” would be in 2020, for the AGI (Association for Geographic Information) and their Foresight Study on the Geospatial Industry in 2020. This follows on from an earlier study in 2010, which attempted to predict what the state of the Geospatial Industry would be like in 2015, and that I also contributed to.

You can find a copy of the final study at the AGI website (when it’s launched, later this year). I’ve placed a copy of my contribution here on archaeogeek as a pdf.

I had what turned out to be an over-optimistic idea about crowd-sourcing some predictions, and asked on twitter and on the OSGeo Discuss Mailing List. Unfortunately, perhaps due to bad timing (everyone was off having fun at FOSS4G at the time), I had very few responses, however many thanks to Jody Garnett, who did take the time to come up with some ideas, and to my colleague Matt Walker who read through the final article. The rest is all mine, faults, mistakes, wrong ideas and all, so enjoy.

Hacking With APIs

| Comments

Since Astun Technology has quite a geographically distributed workforce (OK, we are all in the UK, but still), we make the effort to get together every three months or so for meetings, merriment, and some geeky hacking. Our “hackastun” events are a great chance for those of us who deal more with the finished software to do a bit of coding, even if it is just messing around with python.

For our most recent hack, I had a goal of investigating how we might start integrating some interesting new APIs with our software, iShare. This would use a relatively new feature in iShare, called Dynamic Datasources. These do exactly what they say on the tin, and allow us to call external datasources at the time the iShare page is loaded, to get up to date information.

My hopefully willing accomplice was new Astun hire (but not new to open source Geo) Ian Turton. As is traditional with these things, the first half of the hack was spent doing some serious yak-shaving getting a working iShare development environment set up on his linux laptop. However, contrary to other hacks, once we got that set up, progress was swift.

The first API we looked at was the UK Transport API. Developing a script to pass a unique property reference number (UPRN) from iShare, convert it to a Latitude/Longitude string, send it to the API and return the nearest bus stops was pretty easy- we used pyproj to convert a British National Grid Easting/Northing to Lat/Lon, and Untangle to wrangle any XML we came across.

The next stage for this will be to extend the script to make a second call to get the bus timetable for the stops. Train and tube information are also available from the API, as well as some journey planning, so no doubt we’ll be exploring this API further.

On the train on the way home I also looked at the What3Words API to see how straightforward it would be to work with. It turns out that it’s pretty easy, even for an entry-level pythonista like myself, so now we can offer the What3Words address for a given property in iShare.

It’s important when working with APIs to assess their resilience and sustainability, you don’t want one that you’re relying on to close down suddenly. It should also use open standards, of course. However, I think they are becoming increasingly important as they reduce the need for people to hold, manage and update the data themselves, which is a big plus from an Astun/iShare perspective. They are also a lot of fun to work with, if you’re a geo-geek!

Extending Portable GIS

| Comments

I was asked recently how someone could add new libraries, python programmes etc to their copy of Portable GIS. It’s fairly simple, but here’s a quick guide. I’ll use ogr2osm as an example.

Portable GIS 5.2

| Comments

I’m pleased to announce the latest release of Portable GIS. This version (v5.2) has only a couple of changes:

  • QGIS 2.8 (I’m going to try and do a release to coincide with each long-term release of QGIS)
  • Loader has been updated to the latest version

You can download the setup exe and the md5 checksum here.

Older versions are still available but have been archived to avoid confusion.

As always, please let me know of any problems via the Portable GIS google group.

This version also marks the start of an attempt to create a repository for Portable GIS, which will hold the additional and/or changed files and instructions for rolling your own version. This is something that I have wanted to do for some time, but it’s not ready for release yet. Watch this space!

Archaeology, APIs and Accessibility

| Comments

At the end of November (yes, I know it’s a while ago, life got in the way) I was involved in an event called #HackLancaster, part of a wider project in my home town to try and raise interest in the heritage of the old city area beyondthecastle. This was, I think, a great success on several levels, but it also throw up some issues with archaeological and geospatial data that I hadn’t really considered. This was going to be a much longer post, going through everything I think is wrong with trying to put archaeological data on-line as is, but since it’s been sitting in draft form, I think it’s time to just bite the bullet and get something out there…

So without further ado- my job for #HackLancaster was to take a stuffy old access database, some shape files and some geotiffs, and make this available as an api. Here is the toolkit I ended up using:

  • Various scripts for converting access databases to postgresql- collected and tinkered with here and here.
  • csvkit for dumping csv to postgresql with the bare minimum of effort.
  • Dirt Simple PostGIS API for opening up spatial functionality in my postgresql database over http.
  • PGRest for opening up standard non-spatial functionality in the database over http (had I been doing this now, I might have tried PostgREST instead)
  • Geoserver for serving historic base maps over WMS, and spatial data over WFS/WMS as needed.
  • Gitbook for documentation. You can see my documentation here. Live documentation meant it could be updated and added to as we went along.

Insight Number 1: The one thing I’d take away from this part of the project is that all of these pieces of software were easy to set up, use and configure. As a toolkit for making this kind of data available quickly and easily, you can’t go far wrong here.

Insight Number 2: You need to do more than simply make the data available via an api. Complex relational data structures and archaic terminology are a barrier to non-specialists, no matter how technically competent they are. What’s the solution here- collapse the relational structure somehow? It would be nice to have an automated method in case of updates to the source data.

Insight Number 3: Technologies that seemed like a really obvious fit to a GIS person are not so obvious to everyone else- the historic mapping WMS was completely ignored in favour of Google Maps or MapBox and the spatial RESTful endpoint was only used to convert data from British National Grid to Spherical Mercator, under some duress. Functionality that could have been provided by the spatial RESTful end point was generally fudged.

Insight Number 4: It’s important to make committing code part of the hack. While we asked the teams nicely, not one of them made their code available after the event. In retrospect, since we gave out prizes, we should have made committing their code a prerequisite for winning a prize!

Finally, despite my slight gripes above- the teams involved came up with some extremely nice visualisations of the data- if I can persuade them to make them publicly available I’ll definitely put out some links.

PostGIS Day 2014

| Comments

On 20th November this year we held the first PostGIS “Show-and-Tell” Day at the HQ of the British Computing Society. This was the first “official” OSGeo:UK event since hosting FOSS4G 2013 in Nottingham last year- it took us that long to recover from the stress!

We deliberately chose to make the event extremely light and informal, with the minimum of organisation, and this seemed to work pretty well. We had a range of speakers doing both longer talks and lightning talks, and then a session at the end where we discussed useful tools and add-ons for working with PostGIS.

The only caveats we placed on speakers were that the talk should (obviously) include PostGIS or PostgreSQL in some way, and that overt sales pitches would be considered uncool. Luckily our speakers obliged! There was also some really good audience interaction.

Highlights for me included Foreign Data Wrappers, for getting at information stored in other databases or behind other apis, but through PostgreSQL, and Raster processing. I was particularly excited by the Twitter FDW, but it turns out that it doesn’t work with the latest version of the twitter api, and it doesn’t look like a particularly well-maintained piece of code. You can see links to all the talks, and notes from the discussion that we had at the end, at our OSGeo:UK GitHub repository for the event:

It transpired that there were a number of other events globally, also celebrating #PostGISDay, and perhaps next year it would be good to coordinate these better.

From an OSGeo:UK perspective, we have plans to hold more of these light-touch events, on a range of subjects (ideas gratefully received). Personally I would like to have charged a nominal fee for attending the event, not to make a profit but so that we could make a small donation back to PostGIS but charging money makes the whole process more complicated! We were extremely lucky in this case to get the venue for free, courtesy of the BCS Location Information Group, and to get the food sponsored by Astun Technology.

All in all, a really fun event, both to organise and to attend- thanks to all the speakers and attendees, and here’s to another event quite soon.

New Portable GIS Releases

| Comments

I’m pleased to announce not one, but two new releases of Portable GIS! The first, version 4.2, contains QGIS 2.4 and PostGIS 1.5 and will be the last release to include that version of PostGIS.

The second, version 5, contains QGIS 2.4 and PostGIS 2.1 and all future releases will be based on this. Get them here:

There are two important things about these two releases. The first is that the download size is almost twice what it was previously- this is out of my control and is due to the increased size of the QGIS installation. The second is that version 5.0 should be considered a beta release as it’s very much un-tested. If there are any problems let me know via the Portable GIS google group.

You might also be interested to see a short presentation I did about Portable GIS at GeoMob London in September:

OSGIS 2014

| Comments

Two weeks ago now saw the return of the OSGIS conference in Nottingham, after a year off in 2013 for FOSS4G. I think there had been mixed feelings about this event; those of us heavily involved in the organisation of FOSS4G 2013 had taken a back seat this year, and with FOSS4G 2014 imminent in Portland, it was clearly going to be a smaller scale get together.

I have to say that overall, my impression is that small is good! Small allows you to chat to everyone, see everything you want to see, and generally enjoy, rather than rush around like a mad man or woman. It was nice to see some new faces, and to see a number of papers from local government and business, belying the idea that OSGIS is primarily an academic event. Thanks as always to the chaps at Nottingham for organising.

Astun had a strong showing at the event, with two workshops and two presentations. My colleague Matt Walker did a workshop on OpenLayers3 and Leaflet, and I did one on WPS and PgRouting (a beginners guide). I did a quick introduction to Portable GIS, and another colleague Antony Scott did a comparison to web servers. You can see the workshops at the Astun Technology GitHub pages. As a slight techy aside, Matt and I collectively decided to try GitBook for preparing our workshops, and we’re both very impressed. If you’re interested in a cheeky workflow for pushing a gitbook directly to GitHub gh-pages, see this gist.

Steven Feldman, another Astun employee/adviser also did a talk entitled “There’s no such thing as a free lunch”, a continuation of an emerging theme in open source geospatial at the moment on getting companies to contribute more, or at the very least acknowledge and thank the open source components they use. As always this was very thought provoking (although I worry a little that making people feel responsible for the software they use might potentially back fire)– you can find Steven’s blog post on the talk here.

OSGIS has always had a strong relationship with the OSGeo:UK local chapter, but unfortunately that’s been a bit inactive since FOSS4G- sometimes it’s hard to know what to do next when you’ve fulfilled one of your primary goals! We’re going to have a go at rebooting the local chapter though now- a more detailed post on this will follow soon. The short version is, I’m back co-chairing after a two year absence, and we’ve got some good ideas going forward. Watch this space!

Portable GIS Update

| Comments

It’s taken slightly longer than I’d like, but I’ve updated Portable GIS to include QGIS 2.2. You can find a copy of this new version on the portable gis page. I’ve included a zip file of the qgis2 folder for those that don’t want to install a full new version. You should be able to simply copy this over the existing apps/qgis2 folder, but you will lose any personalisations, such as new plugins etc that you’ve installed, so you have been warned!

Note that there’s a slight regression with this version- as I’m no longer claiming that QGIS Server and Map Viewer will work. I’ve had all sorts of trouble configuring these to work in the windows environment, let alone portable gis, and I wanted to get this release out without additional delay. When I get chance, I’ll get it finished, and believe me it will deserve a blog post and fanfare all of it’s own.

Also note that this is my first move away from using Dropbox as my file hosting service. Please bear with me if there are any problems with the link!

Complexity vs Quality

| Comments

Recently I had need to evaluate a Proprietary Desktop GIS (PDG for short) to document the procedure for doing a Thing for a client. To avoid any mud-slinging and name calling , I’m naming neither the PDG or the Thing, I’ll just say that the Thing is something that the PDG claims to be able to do. This is not a blog post excoriating PDGs by the way, it’s a reflection on the virtues of simplicity, good documentation, and being honest and open.

So, I download a trial version of the PDG and spend 2 hours installing and licensing it. During this time I have to consult the documentation on exactly what licensing options I wanted for a TRIAL piece of software. I also have to consult the documentation on exactly how to apply the license. No mind, I get the software installed and working and try and do the Thing. I remember from several years ago, last time I tried to do the Thing with the PDG, that it was slightly tricky, but several versions have been and gone, all of which claim to be able to do the Thing. Consequently though, I cut the PDG a bit of slack when it can’t do the Thing, and I try the work-around. Yes, that still works, though I don’t know how you’d guess that from the error messages or the documentation. It’s not ideal to need two methods of doing the Thing but hey ho. I also cut the PDG some slack when it tells me that I can only do the Thing if I adhere to some very unusual naming conventions, which will mean that, should I need to do this for real, I will have a lot of work to do renaming a bunch of stuff.

Let’s take this up a level. I don’t only need to do the Thing, but also the related Slightly More Complicated Thing (SMCT for short) too. I confess that the documentation doesn’t really say out and out that the PDG can do this, but it certainly implies it. Only, it doesn’t seem to be able to without a license for it’s rather more expensive elder brother, the Proprietary Server GIS (PSG for short). However, to explain this to the client, I will need some documentary proof. I can find blog and forum posts admitting it’s true, and for all I know there might be lots of information in the knowledge base for the PDG and PSG but you have to have a customer number to access this and because I am only evaluating the software, I haven’t purchased it yet, so I don’t have one of those.

So, I ask some questions of colleagues, and while waiting for them to get back to me, I try some work-arounds for the SMCT. Needless to say, they don’t work either.

A colleague finally gets back to me. After some incredulity that the PDG really can’t do the SMCT when everything implies that it can, said colleague, in his other role as a re-seller for the PDG rings them up and asks. “Yes, we can do that” says the first person, let me find a Thing-specialist to explain how. “No, we can’t do that” says the Thing-specialist. “Our reasons are very complicated, but here’s some obscure documentation that actually admits that we can’t do it”. We let the client know the good news.

As I said earlier, this is not a post excoriating PDG, it’s a reflection on the virtues of simplicity, good documentation, and being honest and open.

Reflection One: The whole process of installing the PDG and discovering the various methods of doing the Thing was needlessly over-complicated. This may be due to the long history of the PDG, and the enormous feature-set, but it feels like bloat. Complexity and a huge feature-set do not necessarily equate to quality, and similarly simplicity and a smaller feature-set are not a bad thing.

Reflection Two: Why hide documentation behind what’s effectively a pay-wall? Had I actually been in the market for purchasing this software, I would have given up at that point. Documentation should be freely available to everyone.

Reflection Three: We really should not have needed to get a re-seller to ring up, and speak to two different people, just to get a definitive answer on the capabilities of a piece of software. This is wrong on so many levels.

In my opinion, these points have nothing to do with the license applied to the source code of the software, or the name on the box: Don’t fall prey to Zawinski’s Law, do make your documentation comprehensive and easily accessible, and do be honest about your capabilities. I’d pay good money for that.