Open Source Computing and GIS in the UK

Travels in a digital world

Portable GIS Update

| Comments

It’s taken slightly longer than I’d like, but I’ve updated Portable GIS to include QGIS 2.2. You can find a copy of this new version on the portable gis page. I’ve included a zip file of the qgis2 folder for those that don’t want to install a full new version. You should be able to simply copy this over the existing apps/qgis2 folder, but you will lose any personalisations, such as new plugins etc that you’ve installed, so you have been warned!

Note that there’s a slight regression with this version- as I’m no longer claiming that QGIS Server and Map Viewer will work. I’ve had all sorts of trouble configuring these to work in the windows environment, let alone portable gis, and I wanted to get this release out without additional delay. When I get chance, I’ll get it finished, and believe me it will deserve a blog post and fanfare all of it’s own.

Also note that this is my first move away from using Dropbox as my file hosting service. Please bear with me if there are any problems with the link!

Complexity vs Quality

| Comments

Recently I had need to evaluate a Proprietary Desktop GIS (PDG for short) to document the procedure for doing a Thing for a client. To avoid any mud-slinging and name calling , I’m naming neither the PDG or the Thing, I’ll just say that the Thing is something that the PDG claims to be able to do. This is not a blog post excoriating PDGs by the way, it’s a reflection on the virtues of simplicity, good documentation, and being honest and open.

So, I download a trial version of the PDG and spend 2 hours installing and licensing it. During this time I have to consult the documentation on exactly what licensing options I wanted for a TRIAL piece of software. I also have to consult the documentation on exactly how to apply the license. No mind, I get the software installed and working and try and do the Thing. I remember from several years ago, last time I tried to do the Thing with the PDG, that it was slightly tricky, but several versions have been and gone, all of which claim to be able to do the Thing. Consequently though, I cut the PDG a bit of slack when it can’t do the Thing, and I try the work-around. Yes, that still works, though I don’t know how you’d guess that from the error messages or the documentation. It’s not ideal to need two methods of doing the Thing but hey ho. I also cut the PDG some slack when it tells me that I can only do the Thing if I adhere to some very unusual naming conventions, which will mean that, should I need to do this for real, I will have a lot of work to do renaming a bunch of stuff.

Let’s take this up a level. I don’t only need to do the Thing, but also the related Slightly More Complicated Thing (SMCT for short) too. I confess that the documentation doesn’t really say out and out that the PDG can do this, but it certainly implies it. Only, it doesn’t seem to be able to without a license for it’s rather more expensive elder brother, the Proprietary Server GIS (PSG for short). However, to explain this to the client, I will need some documentary proof. I can find blog and forum posts admitting it’s true, and for all I know there might be lots of information in the knowledge base for the PDG and PSG but you have to have a customer number to access this and because I am only evaluating the software, I haven’t purchased it yet, so I don’t have one of those.

So, I ask some questions of colleagues, and while waiting for them to get back to me, I try some work-arounds for the SMCT. Needless to say, they don’t work either.

A colleague finally gets back to me. After some incredulity that the PDG really can’t do the SMCT when everything implies that it can, said colleague, in his other role as a re-seller for the PDG rings them up and asks. “Yes, we can do that” says the first person, let me find a Thing-specialist to explain how. “No, we can’t do that” says the Thing-specialist. “Our reasons are very complicated, but here’s some obscure documentation that actually admits that we can’t do it”. We let the client know the good news.

As I said earlier, this is not a post excoriating PDG, it’s a reflection on the virtues of simplicity, good documentation, and being honest and open.

Reflection One: The whole process of installing the PDG and discovering the various methods of doing the Thing was needlessly over-complicated. This may be due to the long history of the PDG, and the enormous feature-set, but it feels like bloat. Complexity and a huge feature-set do not necessarily equate to quality, and similarly simplicity and a smaller feature-set are not a bad thing.

Reflection Two: Why hide documentation behind what’s effectively a pay-wall? Had I actually been in the market for purchasing this software, I would have given up at that point. Documentation should be freely available to everyone.

Reflection Three: We really should not have needed to get a re-seller to ring up, and speak to two different people, just to get a definitive answer on the capabilities of a piece of software. This is wrong on so many levels.

In my opinion, these points have nothing to do with the license applied to the source code of the software, or the name on the box: Don’t fall prey to Zawinski’s Law, do make your documentation comprehensive and easily accessible, and do be honest about your capabilities. I’d pay good money for that.

Portable GIS V4

| Comments

Here’s a quick and overdue announcement to say that I’m making a new version of Portable GIS available today, including QGIS 2. Consider this one a beta release, since I really want to upgrade PostGIS and GDAL when I get time. Additional upgrades in this version: Astun Technology’s Loader has been upgraded to the latest version, and Psycopg2 is now included.

Before you click on the link, please take time to read the main Portable GIS page, and also do me the favour of reporting any problems that you find at the portable GIS google group, or via Twitter. If you can provide me with a screenshot of an error, and let me know the exact version of Windows that you’re using, then I’ll do my best to fix.

You can pick up the beta version here. This is a dropbox link, so if it’s not available, then I’ve exceeded my bandwidth for the moment, so please try again later.

I hope to get a release out fairly soon with PostGIS 2.1 and GDAL 1.10, but this is not my day job so bear with me!

It’s All About the Language

| Comments

There’s a lot more discussion about Open Source GIS these days, which is great (even if it took a global financial meltdown for some people to try it). However, one thing that has bothered me for a while is the language that gets used when discussing it. The short version- this is not about arguing that open source is “better” than proprietary, as some twitter debates I’ve participated in recently have assumed! Comparing open source and proprietary is no longer about comparing apples and oranges- it’s about comparing apples with different labels on the side, and the language we use to make these comparisons should reflect that.

Example 1:

Try it, it’s not as dodgy as you think!

This is usually used by people supporting open source software, but to my mind it actually undermines the point people are trying to make- in that it introduces the possibility of “dodginess” (aka risk) into the discussion. Admittedly a few years ago there was some possibility that people’s experience of free software was something that they downloaded from tucows, but that’s no longer the case. If you want to say something is good, then say it’s good rather than that it’s not dodgy!

Example 2:

It’s still very costly to learn how to use open source software

This one comes up fairly often in discussions about the pros and cons of open source. It’s true that learning any type of software takes time (and therefore money), but when you’re talking about the modern incarnations of QGIS for example, I would argue that the outlay is going to be absolutely comparable to any other established desktop GIS package. If what you mean is “My staff, who have been using Package A for years, will need to invest time in learning Package B”, then that is also just as applicable to any software package under the sun- even when B is actually A.2. If anyone can say truthfully that they coped seamlessly with the switch from Windows XP to a later version, or from Office 2003 onwards, then I will call you a big fibber! It’s fair to say that learning to use command-line tools is tricky if you’re used to a GUI, or that databases are difficult for beginners- but the software license has nothing to do with this. Ironically the reason this gets trotted out when discussing open source is that suddenly people have access to a much greater range of tools, and hence are learning packages (such as server-side databases) that they previously wouldn’t have had access to.

Example 3:

But what about the support?

This is often asked by people enquiring about open source software, but it’s too woolly, and easy to counter with corporate SLA-speak. There’s a huge disconnect between the expectations of corporate clients about what they get when they buy subscription to a proprietary software package, and what would happen if there was an actual problem with the software. Have a look at the small print in a EULA if you’re interested. What the actual end-users want, I would imagine, is a responsive support system where if they have a problem, someone fixes it for them. The difference between proprietary and open source software is purely where you go for the answers.

Going back to the apples and oranges analogy- by using these arguments that are just as applicable to any kind of software, you’re not making a valid comparison. I’m also a firm (possibly naive) believer that invalid comparisons are a lose-lose situation for both proprietary and open source software. By muddying the debate, it’s hard to concentrate on the actual, important differences, or indeed make a proper informed decision. Furthermore, if valid comparisons are made, and there do turn out to be differences in (say) support outcomes (rather than options or contracts or other legalese) then that’s something software suppliers of all persuasions can build and improve on.

Getting Into GitHub

| Comments

Yesterday I did a couple of talks at the AGI Northern Group Showcase in York, one of which was titled “A Beginners Guide to GitHub for Geospatial Folk”. Given that, for various reasons, I was managing on 3 hours sleep and fuelled by caffeine and jammy dodger biscuits, it seemed quite well-received, so I thought I’d expand on it a bit here.

There are definitely some major hurdles for “beginners” to overcome when faced with going public on GitHub. Yes, there are tutorials to help you understand the syntax and workflow, but they tend to be of the “Hello World” variety, or they are focused on the collaborative coding workflow, which might not be appropriate for the casual user. Then you have to beware of the trolls, or at the very least, the people who don’t understand the need for constructive criticism.

My quick suggestion for an easy way in, which also happens to do the world a great service, is to use GitHub for hosting presentations. Not only is this a safe way of learning to use Git/GitHub without exposing any coding inexperience, but you help rid the world of powerpoint, one presentation at a time!

So… go find Big, or Reveal.js and clone it to your local machine. Write your talk (yes write it- none of this gui nonsense). When you’re happy with it, push it up to your own GitHub repository. Use the nifty GitHub pages functionality to create a hosted version of your talk. There’s no need to worry about versions of powerpoint, you just access your hosted talk, or take along a local copy. This is just html, so every browser in the world will render it (in some form)– even lynx) *.

GitHub may or may not be the most important social network or place to put your CV, but if you’re put off from getting involved because of the learning curve or the public nature of it all, then this is one way to do it. Then once you’re happy with the interface, find yourself some non-controversial repositories to contribute to- my first pull request was to the Vaguely Rude Place Names Map of all things! Other good projects you can contribute to are related to documentation, such as the QGIS Training Manual. Again, these are an easy way in- no one will complain if you fix a typo!

*I know this because I’m nerdy enough to have installed Git on my Nexus 10 tablet via the super Terminal IDE app and instructions from DamGit

Company Hackathons

| Comments

A couple of weeks ago we had an Astun “Company Hackathon” for the first time, after the success of the MapAction Hackathon that we hosted a couple of months ago. I think it was a big success, and something that we will continue with in future. You might ask what the big deal is about a hackathon- we’re a technology company, what’s the difference between a hackathon and our normal work? There are a couple of key differences, which are also the things that made the event a success and worth persevering with. I think they are generally applicable to all hackathons, but definitely to company-based events, although as usual these are my own opinions and don’t represent Astun policy etc etc etc…

  • Do have a couple of ideas for what you want to achieve prepared in advance. These don’t need to be fully fleshed out, but should allow you to get to the end of the event with something concrete and visible to show for your efforts.

  • Do consider those people in the company with different skill-sets, and provide topics that they can get involved with or take ownership of.

  • Do appoint someone to informally lead the hack, even if all they do is ensure it actually happens and people don’t fall back to doing their normal work or checking email all day.

  • Ensure the work gets committed to your repository or the equivalent, and that it can be taken forward and not lost/forgotten. This is important even if it was just a proof-of-concept or something you don’t intend to pursue as you still need to record what you did and what the conclusions were.

  • Do make people present the results of their work. This provides an extra incentive to achieve something concrete within the allotted time, and also ensures that people get the recognition and ownership for what they have done.

So what did we look at in our hack? The overall focus was on a test suite, and a set of data to be included in an install for workshop or testing purposes. For the test suite we ended up using PyTest, basically because it’s very easy to install and configure (even on Windows), is actively maintained, and has good documentation. It’s also really easy to write tests! At the end of the hack we had a good initial set of tests, a workflow for installation on a client’s server, and a set of objectives for we want to implement in future.

The “data group” looked at putting together a dataset that we can supply with installations, and that demonstrates all the functionality of the our software. This is in no way less important than the test suite- good data that shows off your software is vital. You can have the best software in the world but if you can’t demonstrate it, or it looks superficially rubbish, then that won’t stand for anything.

In conclusion, hackathons (targeted and well managed) are good for rapid assessment/development of discrete ideas. They are expensive- getting everyone together to work on unbillable tasks for a day or two costs a fair bit, but if you plan them properly and take the ideas forward then the benefits will definitely outweigh the costs. They are also very good for company morale and cohesion, which has to be a plus!

UKQGIS Inaugural User Group Meeting

| Comments

On Friday I attended the inaugrual meeting of the UKQGIS User Group, set up by Simon Miles and Matt Travis in sunny Maidenhead. I’d been asked to do a presentation about Portable GIS, which both surprised and flattered me, and they actually gave out USB sticks with it on to all the delegates (kindly sponsored by Ordnance Survey). So anyhow, I gave a brief intro into Portable GIS, it’s history, what’s currently installed, and what plans I have for the next version. (Yes! There is going to be a new version as soon as I can put it together!). I also did a quick demo of it in use. The reaction was amazing- mind you it doesn’t take much to overwhelm me with Portable GIS- just hearing that someone uses it is enough!

You can see my talk on github (note it might not make much sense without the demo in the middle). To be honest, mine was probably the least relevant of the talks, after all it was only nominally about QGIS! It was followed by a couple of really interesting talks on using QGIS as part of super-fast broadband deployment, and a demo of a really innovative use of Anita Graser’s Time Manager plugin to show the movement of a train, passing through stations on a route. It was really cool and very simple to set up- when you think about it!

The afternoon’s talks were quite varied- we learnt about QGIS Server and the Mobile Web Viewer from Andreas Neumann (who also pointed out this really useful visual changelog for QGIS 2.0 and talked about the Swiss QGIS User Group). We also had a couple of presentations about using QGIS in a corporate environment, integrating with proprietary databases, and the general challenges of getting it accepted (hint, the non-existent initial price tag helps- though that’s not a comment on TCO so don’t flame me).

The general feel for the event was great- very relaxed and informal. There was some discussion about the potential structure of the User Group moving forward, drawing inevitable comparisons with things like the OSGeo UK Local Chapter. Having just surfaced from several years of OSGeo committees, I have to say I’m now a total convert to the loosely organised informal approach that we had today and long may it continue. Having said that, there’s some obvious grounds for cross-fertilisation and interaction between the two groups and I’m all for that as well!

All in all, a really good event- refreshing, fun, and informative. UK QGIS Users- keep an eye on the blog and google plus for future events.

Back to Portable GIS… all fired up and full of enthusiasm, I’m giving serious thought to how to make the development more sustainable. In an ideal world I would get it into an online repository so that others can contribute, and also figure out how to automate the process of updating it when new versions of the core packages come out. If anyone has any solid ideas as to how this could work *, then ping me- let’s talk!

*Note it needs to be fairly easy to set up, and not take too much time for me to manage…

I’m also ready to start thinking seriously about how it could be used for rapid deployment, for the likes of MapAction and HOT. Again, ping me if you have some ideas about this!

FOSS4G 2013 Thoughts

| Comments

aka that mythical time post FOSS4G 2013

So where to start? This feels really weird. I first discussed bringing FOSS4G to the UK in 2006, at the first “official” (don’t start on the lineage of the name, this isn’t the place) FOSS4G with my colleagues at the time, Chris and Leif, and Tyler Mitchell. It was (is?) still on the list of objectives in the UK chapter pages on the OSGeo website. Fast forward 7 years and I’m on my way home from Nottingham, with more t-shirts than I started off with, achy feet, bone-tiredness, and a weird kind of numb feeling about what has just happened.

(Small self-congratulatory bit for the team) I think we did a good job! We could have done some things better, and we had a lot of luck, but in general it worked well. Thanks to the team. We couldn’t have done it without every single person. That’s all I’m going to say on that- lots has been said (go see Twitter and come back if you’re interested…). Back? OK. So this is just some general thoughts on the event, and what it’s like to organise something like FOSS4G. There will be a more formal debrief for the FOSS4G cookbook, this is just a non-objective brain-dump- you have been warned (and people from the Portland committee might want to look away in case you get put off)…

  1. Conference organising on this scale is tough. Take the level of toughness you expect, and multiply it by at least two. If you knew how tough it was, you wouldn’t do it. Really.
  2. You learn a lot about yourself and your fellow team members over the year. You spend a lot of time together, on and off-line, you get excited, frustrated, stressed and scared. You learn that everyone handles this differently, and at the end of it, you really are a team- and not just of colleagues and acquaintances, but firm friends.
  3. Don’t expect, realistically, to have either time or headspace to think of anything else in the run up to the event. Become a task-management ninja so that your boss isn’t forced to sack you.
  4. Only attempt this if you have an understanding boss!
  5. Get a thick skin, as “Conference-Organiser’s Law” states that “Someone will complain about EVERY.SINGLE.THING” and you can’t take every comment to heart, or indeed act on every complaint. Similarly, those people that troll the event on twitter? Don’t bother. Really.
  6. Everything costs a lot of money, and at the end of the day you have to a) balance quality with cost, and b) pass those costs onto the delegates. People will complain about the cost (see point 5) and some people will feel entitled to a free ticket, but honestly there’s no such thing unless they are happy to camp/sleep in their car, and bring their own packed meals/drinks every day, and not use any conference wifi, power, or other facilities.
  7. Anyway, who should decide who is entitled? Is it better to cater for established core developers who have undoubtedly contributed an enormous amount to OSGeo, or new blood who might be the core contributors, movers and shakers of the future if they get inspired by attending FOSS4G?
  8. For future organisers- take advice from previous events but don’t listen to it unless you want to- it’s your turn so, within the bounds of logistics and OSGeo requirements, make it your own.
  9. Also, take up a hobby that allows you to clear your mind when you’re really stressed. Mine has been rock-climbing (try thinking about work when you’re clinging onto a rock with only your fingers) but YMMV.

Some thoughts from a punter’s perspective (warning, I only got to see a few talks so this might be totally un-representative):

  • This event was about maturity- the big players are all well settled now, so what we had were talks about new releases and features. Previous bright-young projects are becoming more established too. This is all great! It’s getting even harder (let’s say ludicrous actually) to dismiss this stack as not fit for purpose, whatever that purpose is. Similarly we had keynotes/plenaries from some really big companies, talking about open source. There might have been complaints about getting guys in suits on the stage, but it demonstrates a growing confidence in our software and you’d be crazy not to see the advantages that might bring.
  • There’s some great work going on in the services part of the stack too- which is fantastic from an interoperability perspective and again, making our software an even more viable option for companies alongside their existing packages, which I definitely see as a “softly, softly catchee monkey” approach. I also saw a couple of decent talks on metadata- making life easier to connect and store, which is definitely an area I’m interested in at the moment.
  • Arnulf Christl won the Sol Katz award, which is not only totally deserved but dreadfully overdue. Paul Ramsey did one of the best closing keynotes that I’ve seen in a long time, as well as more presentations and workshops than all the other presenters put together, or so it seemed! Lots of people “from the internet” met each other face to face for the first time, in one case after 10 years of working together on the same project. People had fun.

So, what happens next? I tweeted recently that I’m looking forward to some “doing” rather than “organising”. This is not just related to FOSS4G, but a growing feeling that I am happier and more useful doing low-level tinkering in things that I’m interested in, than being on higher-level committees. So my plan for the next year is to step back from any committee stuff of any description. I want to continue my python learning. I want to get to grips with PgRouting and WPS, particularly from a UK ITN perspective. I want to do a release of Portable GIS that works with QGIS 2.0 and PostGIS 2.x. I want to blog about “some thing I found out that newbies might find useful”. Finally, I want to get back to giving talks about what I do (after all, who wants to listen to talks on conference organising?), and how others can get involved.

To those that came to our little “event”, thanks for visiting a smallish rainy island in the North Sea! Maybe see you in Portland next year. End Update.

Blimey, a New Post

| Comments

So after months of silence, I’m finally putting virtual pen to virtual paper again- Archaeogeek is not dead after all!

I could have written any number of posts over the last few months, on the trials and tribulations of organising a major international conference for one thing- but I think I’ll save those for after the event…

Instead- a couple of things that have caught my eye recently: firstly a fantastic post on the problems of cartography on Pluto in the new Wired MapLab. I confess I’d never thought about the technicalities of defining a coordinate system on anything other than Earth, or even defining which of the two poles on a planet/asteroid is North and which is South. For asteroids, the problem is exacerbated by the fact that their orientation can change dramatically over a short period of time, so it’s impractical to define a coordinate system pegged at the poles- instead you define one based on the direction of spin. Cue the “right-hand-rule” that you probably learnt in Physics at school for defining directions of magnetic forces and such like. There’s lots more detail in the article, but it’s a great read, and it blew my mind (in a good way) thinking about the arbitrary nature of things we take for granted in mapping.

Secondly, a frankly bizarre set of posts from the UKGovOSS blog. This started up in 2009 and has seemingly returned after a long hiatus with a set of posts on (as you’d expect) using Open Source in UK Government. “Yay” I thought, how useful! But what I found was a set of badly written, often contradictory posts that could have been written 5 years ago in terms of what they have to say about open source. The more I read these posts, the more something seemed suspicious, and a whois search for the domain shows it has been recently registered in Indonesia! I find the idea of a bunch of bloggers in Indonesia writing blog-posts on the use of Open Source in UK Government almost as mind-blowing as the Pluto Coordinate System problem, but perhaps the guys behind the original website ought to try and claim their domain back…

My First Hackathon

| Comments

In all the years that I’ve been involved with open source, I’ve been a committed advocate for the idea that you don’t need to be a coder to get involved. I’m definitely not a coder- I can write a script or two, and have been known to submit bugs, but that’s as far as it goes. My strengths are in identifying and fixing problems, and getting other people enthused. That’s all well and good, but hackathons- they are for real coders, right? Why expose my terrible deficiency of coding skills to the world? It turns out, I was wrong!

So, for various reasons I don’t need to go into here, I found myself at the first MapAction/AGI Technical SIG Hackathon at our company offices in Epsom, with a mix of 40 other people: coders, MapAction volunteers, Astun colleagues, and other interested parties. MapAction had done a lot of leg-work before the event, identifying particular problems that they wanted to try and solve, with enough variety that everyone could find something of interest. As one of my current work projects involves metadata gathering, when I saw a hack around implementing a metadata gathering script in Quantum GIS, working with an existing python script (thanks Tyler), I thought it was something I could have a go at. The guy I teamed up with had even less scripting experience than me, but was also interested in the end result. Other teams looked at data visualisation, and the fantastically named “Dirty Data Dashboard”, that deliberately creates invalid or “dirty” data for use in training.

With very little experience of implementing python plugins in Quantum GIS, we had a brief attempt at using the Script Runner plugin to access the script. This definitely seems like a useful first approach, but the learning curve of adapting the script to interact with Quantum GIS (for example to return results to the logging console) was more than we could manage in the time available. Furthermore, I think it would be useful to create a wrapper for Quantum GIS, rather than rewriting the original code and risking it no longer working as a stand-alone script. Instead, we worked on extending the script to return more “human-readable” results. For non-coders, this was definitely achievable in the time available, so we had the nice fuzzy feeling of actually having a result at the end of the day.

Time for another first! Well, technically a second, but I’m not sure this counts. I love GitHub, and use plenty of repositories, but as a non-coder, putting your own code on the site for all and sundry to see is quite intimidating, and there are tales of people getting criticised publicly for nothing more than producing imperfect code. Not only does that put me right off, but it also makes me quite angry because it’s deeply elitist and non-constructive. However, once we’d extended this python script, re-committing it back to GitHub was the natural thing to do. Cue much googling on forking a repository, committing back to GitHub from my local copy, then issuing a Pull Request. And yay, it was accepted!

All in all, the MapAction hackathon was definitely a success. Lots of good things were achieved, and I would imagine that it will be repeated fairly soon. There’s no moral to this story, and I still think of myself as a non-coder, but this was just a first toe in the water of engaging differently with the open source world. Finally though, I would say that if you do consider yourself a non-coder, don’t be put off from attending a hackathon, as there will definitely be something you can contribute to.