Open Source Computing and GIS in the UK

Travels in a digital world

PostGIS Day 2014

| Comments

On 20th November this year we held the first PostGIS “Show-and-Tell” Day at the HQ of the British Computing Society. This was the first “official” OSGeo:UK event since hosting FOSS4G 2013 in Nottingham last year- it took us that long to recover from the stress!

We deliberately chose to make the event extremely light and informal, with the minimum of organisation, and this seemed to work pretty well. We had a range of speakers doing both longer talks and lightning talks, and then a session at the end where we discussed useful tools and add-ons for working with PostGIS.

The only caveats we placed on speakers were that the talk should (obviously) include PostGIS or PostgreSQL in some way, and that overt sales pitches would be considered uncool. Luckily our speakers obliged! There was also some really good audience interaction.

Highlights for me included Foreign Data Wrappers, for getting at information stored in other databases or behind other apis, but through PostgreSQL, and Raster processing. I was particularly excited by the Twitter FDW, but it turns out that it doesn’t work with the latest version of the twitter api, and it doesn’t look like a particularly well-maintained piece of code. You can see links to all the talks, and notes from the discussion that we had at the end, at our OSGeo:UK GitHub repository for the event: http://osgeouk.github.io/pgday/.

It transpired that there were a number of other events globally, also celebrating #PostGISDay, and perhaps next year it would be good to coordinate these better.

From an OSGeo:UK perspective, we have plans to hold more of these light-touch events, on a range of subjects (ideas gratefully received). Personally I would like to have charged a nominal fee for attending the event, not to make a profit but so that we could make a small donation back to PostGIS but charging money makes the whole process more complicated! We were extremely lucky in this case to get the venue for free, courtesy of the BCS Location Information Group, and to get the food sponsored by Astun Technology.

All in all, a really fun event, both to organise and to attend- thanks to all the speakers and attendees, and here’s to another event quite soon.

New Portable GIS Releases

| Comments

I’m pleased to announce not one, but two new releases of Portable GIS! The first, version 4.2, contains QGIS 2.4 and PostGIS 1.5 and will be the last release to include that version of PostGIS.

The second, version 5, contains QGIS 2.4 and PostGIS 2.1 and all future releases will be based on this. Get them here:

There are two important things about these two releases. The first is that the download size is almost twice what it was previously- this is out of my control and is due to the increased size of the QGIS installation. The second is that version 5.0 should be considered a beta release as it’s very much un-tested. If there are any problems let me know via the Portable GIS google group.

You might also be interested to see a short presentation I did about Portable GIS at GeoMob London in September: http://archaeogeek.github.io/geomob-portablegis/#/

OSGIS 2014

| Comments

Two weeks ago now saw the return of the OSGIS conference in Nottingham, after a year off in 2013 for FOSS4G. I think there had been mixed feelings about this event; those of us heavily involved in the organisation of FOSS4G 2013 had taken a back seat this year, and with FOSS4G 2014 imminent in Portland, it was clearly going to be a smaller scale get together.

I have to say that overall, my impression is that small is good! Small allows you to chat to everyone, see everything you want to see, and generally enjoy, rather than rush around like a mad man or woman. It was nice to see some new faces, and to see a number of papers from local government and business, belying the idea that OSGIS is primarily an academic event. Thanks as always to the chaps at Nottingham for organising.

Astun had a strong showing at the event, with two workshops and two presentations. My colleague Matt Walker did a workshop on OpenLayers3 and Leaflet, and I did one on WPS and PgRouting (a beginners guide). I did a quick introduction to Portable GIS, and another colleague Antony Scott did a comparison to web servers. You can see the workshops at the Astun Technology GitHub pages. As a slight techy aside, Matt and I collectively decided to try GitBook for preparing our workshops, and we’re both very impressed. If you’re interested in a cheeky workflow for pushing a gitbook directly to GitHub gh-pages, see this gist.

Steven Feldman, another Astun employee/adviser also did a talk entitled “There’s no such thing as a free lunch”, a continuation of an emerging theme in open source geospatial at the moment on getting companies to contribute more, or at the very least acknowledge and thank the open source components they use. As always this was very thought provoking (although I worry a little that making people feel responsible for the software they use might potentially back fire)– you can find Steven’s blog post on the talk here.

OSGIS has always had a strong relationship with the OSGeo:UK local chapter, but unfortunately that’s been a bit inactive since FOSS4G- sometimes it’s hard to know what to do next when you’ve fulfilled one of your primary goals! We’re going to have a go at rebooting the local chapter though now- a more detailed post on this will follow soon. The short version is, I’m back co-chairing after a two year absence, and we’ve got some good ideas going forward. Watch this space!

Portable GIS Update

| Comments

It’s taken slightly longer than I’d like, but I’ve updated Portable GIS to include QGIS 2.2. You can find a copy of this new version on the portable gis page. I’ve included a zip file of the qgis2 folder for those that don’t want to install a full new version. You should be able to simply copy this over the existing apps/qgis2 folder, but you will lose any personalisations, such as new plugins etc that you’ve installed, so you have been warned!

Note that there’s a slight regression with this version- as I’m no longer claiming that QGIS Server and Map Viewer will work. I’ve had all sorts of trouble configuring these to work in the windows environment, let alone portable gis, and I wanted to get this release out without additional delay. When I get chance, I’ll get it finished, and believe me it will deserve a blog post and fanfare all of it’s own.

Also note that this is my first move away from using Dropbox as my file hosting service. Please bear with me if there are any problems with the link!

Complexity vs Quality

| Comments

Recently I had need to evaluate a Proprietary Desktop GIS (PDG for short) to document the procedure for doing a Thing for a client. To avoid any mud-slinging and name calling , I’m naming neither the PDG or the Thing, I’ll just say that the Thing is something that the PDG claims to be able to do. This is not a blog post excoriating PDGs by the way, it’s a reflection on the virtues of simplicity, good documentation, and being honest and open.

So, I download a trial version of the PDG and spend 2 hours installing and licensing it. During this time I have to consult the documentation on exactly what licensing options I wanted for a TRIAL piece of software. I also have to consult the documentation on exactly how to apply the license. No mind, I get the software installed and working and try and do the Thing. I remember from several years ago, last time I tried to do the Thing with the PDG, that it was slightly tricky, but several versions have been and gone, all of which claim to be able to do the Thing. Consequently though, I cut the PDG a bit of slack when it can’t do the Thing, and I try the work-around. Yes, that still works, though I don’t know how you’d guess that from the error messages or the documentation. It’s not ideal to need two methods of doing the Thing but hey ho. I also cut the PDG some slack when it tells me that I can only do the Thing if I adhere to some very unusual naming conventions, which will mean that, should I need to do this for real, I will have a lot of work to do renaming a bunch of stuff.

Let’s take this up a level. I don’t only need to do the Thing, but also the related Slightly More Complicated Thing (SMCT for short) too. I confess that the documentation doesn’t really say out and out that the PDG can do this, but it certainly implies it. Only, it doesn’t seem to be able to without a license for it’s rather more expensive elder brother, the Proprietary Server GIS (PSG for short). However, to explain this to the client, I will need some documentary proof. I can find blog and forum posts admitting it’s true, and for all I know there might be lots of information in the knowledge base for the PDG and PSG but you have to have a customer number to access this and because I am only evaluating the software, I haven’t purchased it yet, so I don’t have one of those.

So, I ask some questions of colleagues, and while waiting for them to get back to me, I try some work-arounds for the SMCT. Needless to say, they don’t work either.

A colleague finally gets back to me. After some incredulity that the PDG really can’t do the SMCT when everything implies that it can, said colleague, in his other role as a re-seller for the PDG rings them up and asks. “Yes, we can do that” says the first person, let me find a Thing-specialist to explain how. “No, we can’t do that” says the Thing-specialist. “Our reasons are very complicated, but here’s some obscure documentation that actually admits that we can’t do it”. We let the client know the good news.

As I said earlier, this is not a post excoriating PDG, it’s a reflection on the virtues of simplicity, good documentation, and being honest and open.

Reflection One: The whole process of installing the PDG and discovering the various methods of doing the Thing was needlessly over-complicated. This may be due to the long history of the PDG, and the enormous feature-set, but it feels like bloat. Complexity and a huge feature-set do not necessarily equate to quality, and similarly simplicity and a smaller feature-set are not a bad thing.

Reflection Two: Why hide documentation behind what’s effectively a pay-wall? Had I actually been in the market for purchasing this software, I would have given up at that point. Documentation should be freely available to everyone.

Reflection Three: We really should not have needed to get a re-seller to ring up, and speak to two different people, just to get a definitive answer on the capabilities of a piece of software. This is wrong on so many levels.

In my opinion, these points have nothing to do with the license applied to the source code of the software, or the name on the box: Don’t fall prey to Zawinski’s Law, do make your documentation comprehensive and easily accessible, and do be honest about your capabilities. I’d pay good money for that.

Portable GIS V4

| Comments

Here’s a quick and overdue announcement to say that I’m making a new version of Portable GIS available today, including QGIS 2. Consider this one a beta release, since I really want to upgrade PostGIS and GDAL when I get time. Additional upgrades in this version: Astun Technology’s Loader has been upgraded to the latest version, and Psycopg2 is now included.

Before you click on the link, please take time to read the main Portable GIS page, and also do me the favour of reporting any problems that you find at the portable GIS google group, or via Twitter. If you can provide me with a screenshot of an error, and let me know the exact version of Windows that you’re using, then I’ll do my best to fix.

You can pick up the beta version here. This is a dropbox link, so if it’s not available, then I’ve exceeded my bandwidth for the moment, so please try again later.

I hope to get a release out fairly soon with PostGIS 2.1 and GDAL 1.10, but this is not my day job so bear with me!

It’s All About the Language

| Comments

There’s a lot more discussion about Open Source GIS these days, which is great (even if it took a global financial meltdown for some people to try it). However, one thing that has bothered me for a while is the language that gets used when discussing it. The short version- this is not about arguing that open source is “better” than proprietary, as some twitter debates I’ve participated in recently have assumed! Comparing open source and proprietary is no longer about comparing apples and oranges- it’s about comparing apples with different labels on the side, and the language we use to make these comparisons should reflect that.

Example 1:

Try it, it’s not as dodgy as you think!

This is usually used by people supporting open source software, but to my mind it actually undermines the point people are trying to make- in that it introduces the possibility of “dodginess” (aka risk) into the discussion. Admittedly a few years ago there was some possibility that people’s experience of free software was something that they downloaded from tucows, but that’s no longer the case. If you want to say something is good, then say it’s good rather than that it’s not dodgy!

Example 2:

It’s still very costly to learn how to use open source software

This one comes up fairly often in discussions about the pros and cons of open source. It’s true that learning any type of software takes time (and therefore money), but when you’re talking about the modern incarnations of QGIS for example, I would argue that the outlay is going to be absolutely comparable to any other established desktop GIS package. If what you mean is “My staff, who have been using Package A for years, will need to invest time in learning Package B”, then that is also just as applicable to any software package under the sun- even when B is actually A.2. If anyone can say truthfully that they coped seamlessly with the switch from Windows XP to a later version, or from Office 2003 onwards, then I will call you a big fibber! It’s fair to say that learning to use command-line tools is tricky if you’re used to a GUI, or that databases are difficult for beginners- but the software license has nothing to do with this. Ironically the reason this gets trotted out when discussing open source is that suddenly people have access to a much greater range of tools, and hence are learning packages (such as server-side databases) that they previously wouldn’t have had access to.

Example 3:

But what about the support?

This is often asked by people enquiring about open source software, but it’s too woolly, and easy to counter with corporate SLA-speak. There’s a huge disconnect between the expectations of corporate clients about what they get when they buy subscription to a proprietary software package, and what would happen if there was an actual problem with the software. Have a look at the small print in a EULA if you’re interested. What the actual end-users want, I would imagine, is a responsive support system where if they have a problem, someone fixes it for them. The difference between proprietary and open source software is purely where you go for the answers.

Going back to the apples and oranges analogy- by using these arguments that are just as applicable to any kind of software, you’re not making a valid comparison. I’m also a firm (possibly naive) believer that invalid comparisons are a lose-lose situation for both proprietary and open source software. By muddying the debate, it’s hard to concentrate on the actual, important differences, or indeed make a proper informed decision. Furthermore, if valid comparisons are made, and there do turn out to be differences in (say) support outcomes (rather than options or contracts or other legalese) then that’s something software suppliers of all persuasions can build and improve on.

Getting Into GitHub

| Comments

Yesterday I did a couple of talks at the AGI Northern Group Showcase in York, one of which was titled “A Beginners Guide to GitHub for Geospatial Folk”. Given that, for various reasons, I was managing on 3 hours sleep and fuelled by caffeine and jammy dodger biscuits, it seemed quite well-received, so I thought I’d expand on it a bit here.

There are definitely some major hurdles for “beginners” to overcome when faced with going public on GitHub. Yes, there are tutorials to help you understand the syntax and workflow, but they tend to be of the “Hello World” variety, or they are focused on the collaborative coding workflow, which might not be appropriate for the casual user. Then you have to beware of the trolls, or at the very least, the people who don’t understand the need for constructive criticism.

My quick suggestion for an easy way in, which also happens to do the world a great service, is to use GitHub for hosting presentations. Not only is this a safe way of learning to use Git/GitHub without exposing any coding inexperience, but you help rid the world of powerpoint, one presentation at a time!

So… go find Big, or Reveal.js and clone it to your local machine. Write your talk (yes write it- none of this gui nonsense). When you’re happy with it, push it up to your own GitHub repository. Use the nifty GitHub pages functionality to create a hosted version of your talk. There’s no need to worry about versions of powerpoint, you just access your hosted talk, or take along a local copy. This is just html, so every browser in the world will render it (in some form)– even lynx) *.

GitHub may or may not be the most important social network or place to put your CV, but if you’re put off from getting involved because of the learning curve or the public nature of it all, then this is one way to do it. Then once you’re happy with the interface, find yourself some non-controversial repositories to contribute to- my first pull request was to the Vaguely Rude Place Names Map of all things! Other good projects you can contribute to are related to documentation, such as the QGIS Training Manual. Again, these are an easy way in- no one will complain if you fix a typo!

*I know this because I’m nerdy enough to have installed Git on my Nexus 10 tablet via the super Terminal IDE app and instructions from DamGit

Company Hackathons

| Comments

A couple of weeks ago we had an Astun “Company Hackathon” for the first time, after the success of the MapAction Hackathon that we hosted a couple of months ago. I think it was a big success, and something that we will continue with in future. You might ask what the big deal is about a hackathon- we’re a technology company, what’s the difference between a hackathon and our normal work? There are a couple of key differences, which are also the things that made the event a success and worth persevering with. I think they are generally applicable to all hackathons, but definitely to company-based events, although as usual these are my own opinions and don’t represent Astun policy etc etc etc…

  • Do have a couple of ideas for what you want to achieve prepared in advance. These don’t need to be fully fleshed out, but should allow you to get to the end of the event with something concrete and visible to show for your efforts.

  • Do consider those people in the company with different skill-sets, and provide topics that they can get involved with or take ownership of.

  • Do appoint someone to informally lead the hack, even if all they do is ensure it actually happens and people don’t fall back to doing their normal work or checking email all day.

  • Ensure the work gets committed to your repository or the equivalent, and that it can be taken forward and not lost/forgotten. This is important even if it was just a proof-of-concept or something you don’t intend to pursue as you still need to record what you did and what the conclusions were.

  • Do make people present the results of their work. This provides an extra incentive to achieve something concrete within the allotted time, and also ensures that people get the recognition and ownership for what they have done.

So what did we look at in our hack? The overall focus was on a test suite, and a set of data to be included in an install for workshop or testing purposes. For the test suite we ended up using PyTest, basically because it’s very easy to install and configure (even on Windows), is actively maintained, and has good documentation. It’s also really easy to write tests! At the end of the hack we had a good initial set of tests, a workflow for installation on a client’s server, and a set of objectives for we want to implement in future.

The “data group” looked at putting together a dataset that we can supply with installations, and that demonstrates all the functionality of the our software. This is in no way less important than the test suite- good data that shows off your software is vital. You can have the best software in the world but if you can’t demonstrate it, or it looks superficially rubbish, then that won’t stand for anything.

In conclusion, hackathons (targeted and well managed) are good for rapid assessment/development of discrete ideas. They are expensive- getting everyone together to work on unbillable tasks for a day or two costs a fair bit, but if you plan them properly and take the ideas forward then the benefits will definitely outweigh the costs. They are also very good for company morale and cohesion, which has to be a plus!

UKQGIS Inaugural User Group Meeting

| Comments

On Friday I attended the inaugrual meeting of the UKQGIS User Group, set up by Simon Miles and Matt Travis in sunny Maidenhead. I’d been asked to do a presentation about Portable GIS, which both surprised and flattered me, and they actually gave out USB sticks with it on to all the delegates (kindly sponsored by Ordnance Survey). So anyhow, I gave a brief intro into Portable GIS, it’s history, what’s currently installed, and what plans I have for the next version. (Yes! There is going to be a new version as soon as I can put it together!). I also did a quick demo of it in use. The reaction was amazing- mind you it doesn’t take much to overwhelm me with Portable GIS- just hearing that someone uses it is enough!

You can see my talk on github (note it might not make much sense without the demo in the middle). To be honest, mine was probably the least relevant of the talks, after all it was only nominally about QGIS! It was followed by a couple of really interesting talks on using QGIS as part of super-fast broadband deployment, and a demo of a really innovative use of Anita Graser’s Time Manager plugin to show the movement of a train, passing through stations on a route. It was really cool and very simple to set up- when you think about it!

The afternoon’s talks were quite varied- we learnt about QGIS Server and the Mobile Web Viewer from Andreas Neumann (who also pointed out this really useful visual changelog for QGIS 2.0 and talked about the Swiss QGIS User Group). We also had a couple of presentations about using QGIS in a corporate environment, integrating with proprietary databases, and the general challenges of getting it accepted (hint, the non-existent initial price tag helps- though that’s not a comment on TCO so don’t flame me).

The general feel for the event was great- very relaxed and informal. There was some discussion about the potential structure of the User Group moving forward, drawing inevitable comparisons with things like the OSGeo UK Local Chapter. Having just surfaced from several years of OSGeo committees, I have to say I’m now a total convert to the loosely organised informal approach that we had today and long may it continue. Having said that, there’s some obvious grounds for cross-fertilisation and interaction between the two groups and I’m all for that as well!

All in all, a really good event- refreshing, fun, and informative. UK QGIS Users- keep an eye on the blog and google plus for future events.

Back to Portable GIS… all fired up and full of enthusiasm, I’m giving serious thought to how to make the development more sustainable. In an ideal world I would get it into an online repository so that others can contribute, and also figure out how to automate the process of updating it when new versions of the core packages come out. If anyone has any solid ideas as to how this could work *, then ping me- let’s talk!

*Note it needs to be fairly easy to set up, and not take too much time for me to manage…

I’m also ready to start thinking seriously about how it could be used for rapid deployment, for the likes of MapAction and HOT. Again, ping me if you have some ideas about this!