Warning: Declaration of description_walker::start_el(&$output, $item, $depth, $args) should be compatible with Walker_Nav_Menu::start_el(&$output, $item, $depth = 0, $args = Array, $id = 0) in /home1/mapbutch/public_html/blog/wp-content/themes/SCRN/functions.php on line 264

mapbutcher

My Baggage

Posted on 16 Dec 2011 in rib | 7 comments

Over the past few months I’ve been making an effort to drop my Esri baggage. I’m trying to get all Hugh Fearnley- Whittingstall on GIS – I want to start feeding my GIS needs on shit I know the provenance of, and perhaps learn how to manage it all myself too.

I’d like to think that I’ve been an advocate for Open Source GIS software for a while however not at a grass roots level, I’ve never committed a line of code or doco to an open source project. My motives are driven by a number of things; Firstly a self interest in finding out more about alternative open approaches to software development, secondly I’ve never been completely convinced that the traditional software vendor model was the best way forward from a client perspective and thirdly I wanted to understand the choices my clients had when it came to spending their pennies on software. In 2006 I was reading more and more about open source GIS software, I attended the 2007 Foss4g and I was part of the 2009 Foss4g organising committee, but my day job has always been centred around the needs of clients with Esri software so I’ve never been in a position to scratch my own itch.

I have always been frustrated by the polar views on both sides of the proprietary\open source fence. It’s as annoying to me to listen to a dim witted software sales person drop FUD as it is to listen to some super *geek* who believes you’ve just committed a heinous crime by running windows.  I’ve not succumb to the pusherman either – I have always made conscious choices towards proprietary GIS software mainly because its been the source of my mortgage repayments. I still like writing software on top of Esri stuff, but I’ve felt for a long time that I’ve been going through the motions, when Esri move, I move, when Esri change their minds I follow and to be perfectly honest I’m tired of listening to the same record.

So I’ve taken some early steps in untying the apron strings and thought I would share some of my opinions.

1. People love security

I’ve always wondered why more people don’t shop around for mortgage deals every few years, like there’s some unwritten rule that when you take a mortgage with a bank you’re signed up for the 25 year term – people tend to like security, or what they think is security. In environments which are naturally safe then adoption to new technology is generally sluggish – people tend to want to use something they’ve used before or perhaps something they’ve seen their neighbours use. From my experience local government is a good example of this, I’ve found that geographically coincident governments tend to use the same GIS software, because they can easily share experience and trade war stories and scars about this or that vendor.

I have a security blanket too, we all do. There’s not many people I know who have an intimate knowledge of a large spectrum of GIS software. However I’ve found it very rewarding going for a few hours a day without my security blanket, I feel free. If I was starting a self help group I would perhaps be chanting ‘It’s OK not to open ArcMap, you’re still a good person’. In some case security comes in the form of a group of people who can support you or help you deliver and there is an emerging market of providers in this space who are wrapping open source software in a blanket just for you.

2. Glue

Esri sells software partly because their components fit together neatly like jigsaw pieces. I’m sure Esri will continue to develop this approach, making it as seamless an experience as possible to work between these components – its a sensible approach. They use language like ‘system’ and ‘platform’ because this is a key selling point for them, it described something larger, more unified. As each version rolls off the production line we’re told ‘stories’ about how easy it will all be in this release – this is an appealing story to many people, and very valid in some environments.

Open Source is challenged in this space – projects indepently emerge and don’t automatically need to belong to an umbrella group, who will help manage this story. To bridge the gap they rely on collaboration, standards and well used interchange formats – essentially knowing your way around these will help you to glue the open source projects together. I’m not trying to paint an unfair picture by calling this glue, it just seems appropriate to me. Some providers are selecting a set of well established tools and standards to base a more unified ‘system’ on – this makes perfect sense to me, this reads like the ‘new and improved’ Open Source story, and I believe it has value.

Personally I’m discovering a love for tools and technology I’ve never used before – it seems exciting to be out of my comfort zone – one thing which has really struck me is that whilst things do occasionally stumble (see no. 3) I have found it on the whole very straightforward. Even when barriers appear the availability of decent resources to allow you to move forward is abundent.

3. Open Source Crashes too

I don’t know how many times I’ve seen and read about ArcMap crashing! People love a good bitch and moan about Esri software, some GIS people just love a good bitch and moan. Open Source software crashes too – I’ve been using QGIS a bit lately, it crashes too, however this is my take on it; Firstly I’m fairly tolerant when it comes to it, because I don’t use it all the time. I just tend to restart and move on. Secondly if it bothered me enough there are a myriad of places to go and seek a fix or I could just try and fix it myself. This is of course the freedom I have when I use these products. If I had the time I should really do some work and provide back diagnostics to the projects and support a fix, but as I said these squeaky wheels are all acceptable to me. My guess would be that if the adoption of QGIS rocketed to the same levels are ArcGIS then we’d be hearing a lot more about QGIS crashing (bitchy and moaning GIS people remember), my second guess would be that we’d be waiting a fraction of the time to see these issues resolved.

Software, proprietary or open source has its issues. Personally I like the idea of being able to resolve defects or amend the software myself (if I have the skills and the need to do so).

4. Home Brand GIS

Heinz baked beans are always front and centre when I buy my beans at the supermarket – and I buy Heinz (English variety of course). Now I won’t go into why I buy Heinz too deeply, but I suspect it’s something to do with child hood memories and a conditioning (see No. 1 above). It’s probably also fair to say I’m a sucker for packaging, so when i glance down at the bottom shelf at the home brand beans I’m simply unimpressed by the label, irrespective of the fact that I know the beans are good quality.  Open Source GIS in some respect has a home brand feel about it to me – I know its good quality, but the packaging can be a bit shite. Now if you’re not a shallow, little brained brand man like me then you’ve probably be eating home brand for some time and can’t get enough – you don’t care about the packaging you just love them beans.

But what does the packaging matter? Well firstly it matters only when the product and service that it is wrapping is good quality – if you’re brand loyal for a product and service which is inferior then you’re more of a dickhead than me. When I’ve moved across to open source projects like QGIS, GeoServer and TileMill recently, I’ve had to (in some cases) put aside the packaging and judge the results against what’s in my Esri toolkit, and I’m liking what I see. Once you’ve established an understanding of product quality I think its fair to look at the packaging -   is it mature? Does it have good support? (perhaps commercial if required). Does it have good documentation? Can I get my hands dirty if I need to? What is the user exeprience like?

5. Esri has given me pre-conceptions about GIS

Esri has given me preconceptions about GIS. I’ve never been particularly comfortable with Esri’s terminology such as the ‘geodatabase’ or ‘enterprise GIS’ or ‘geodesign’. I can’t help feeling these terms are manufactured. Going outside the Esri world for the first time in years has made me feel like I’m learning first principles again, without my Esri baggage. I don’t wish to spread FUD and perhaps Esri is legitimate in its desire to talk openly about the problem spaces which GIS can solve – however I feel now that when I look at these problem spaces I’m no longer focusing on an Esri technical approach to solving them, but rather looking at them with a much wider lens. Personally as a GIS consultant its no longer acceptable to me to restrict my view, and Open Source GIS is helping me open my eyes.

6. Organic GIS Software

After a while working with Esri software and for a distributor of Esri software I felt like I was going round and round on a mouse wheel, an endless cycle of release, service pack, service pack, (promise performance improvement), go to conference, cram more features in and start again. This is after all commercial software – you have to feed the beast right? Yes -  in amongst this the products evolved and improved – no argument or criticism, but it’s not for me. Somewhere along the line I felt that this endless cycle and growth begins to erode the software. Endless extensions, feature bloat, UX problems, quality issues, multiple divergent APIs, all constantly accruing technical debt.  Again don’t read this the wrong way – these problems are not unique to Esri, they seem to me to be a condition of growth, and perhaps like battery farmed chicken, there is a drive to keep your costs down and margins up and hey’ the punters just keep coming back for that tasteless bird!

Open Source GIS software I would argue comes from a different eco-system, where there are multiple smaller projects and competition thrives and drives improvement. Despite my comments above about quality – on a whole I’ve found the quality of open source GIS software to be as good, if not better than similar commercial products (See Update *). This eco system is maturing to provide choices in terms of software support and development and demand is increasing. I love the lack of marketing and sales tripe in Open Source too. Its straight up, no bullshit, does what is says on the tin, wholesome GIS.

Footnote:

The title for this post came from the following story:

Back at the 2009 Foss4g Volker Mische had just done a great presentation on CouchDB and was taking questions from the floor. A legitimate question came forth from an individual about how one could represent relationships a la RDBMS in these new fangled NoSQL DB’s, before Volker could answer the question, a friend of mine shouted from elsewhere in the crowd:

“Hey man, you’ve just got to leave all that relational baggage at home, man”

Update

* Thanks for the feedback from Anthony – I have no quantitative evidence to support this claim. FUD, guilty as charged.

7 comments

  1. Stephen Lead / December 16th, 2011 9:09

    GREAT post. Very well argued.

  2. Anthony Burgon / December 16th, 2011 9:59

    The way that they (esri) actually build ArcGIS software is often described as “organic” and that is why there are so many quality issues – because 35+ teams, of different abilities, building their capability organically into a large stack of software. There is limited process (though in the last 4 years Scrum has been adopted by many of the teams), there is limited involvement from marketing and sales departments, ideas evolve from within and from talking to the esri users, programmers post code into a large stack, that large stack gets tested and released – and hence why esri never meets its deadlines – cause the process is organic.

    You can go and work in Redlands and be part of an organic eco-system, or contribute to the open source community – either way scratch that itch.

  3. MicahWilli / December 17th, 2011 1:41

    I feel like I should defend local governments using Esri. The analogy of security is good and true. Infact just the other week i helped an adjacent county upgrade from ArcServer 9.3 to 10.0.
    In terms of ‘glue’ I think (at least for us) the issue does go a bit deeper. It’s not only glue, but reliability, consistency and progress. It’s not that we choose Esri over the other, it’s that sometimes Open Source isn’t even an option.

  4. mapbutcher / December 17th, 2011 7:28

    Micah,

    There is no need to defend local government. My comments weren’t intended as an attack on local government, rather an observation – and as you point out, having a neighbour who can help now and again has its advantages.

    I think everyone should have options around the software they use – but perhaps there are forces beyond your control that dictate the technology direction? In terms of reliability, consistency and progress. I’m not sure that open source is not reliable, or that it doesn’t progress, and of course there are paths which can be taken to deal with reliability and extension if required – however these paths, may not be as easy to traverse or as familiar as those provided by a traditional vendor, depending upon the software you’re using. In terms of consistency I’ve always thought that projects between clients and services providers benefit from consistent relationships, this does not mean the inverse is untrue of course – and I think this point is valid for services offered on both proprietary and open source software.

    Simon

  5. mapbutcher / December 19th, 2011 21:52

    Anthony,

    Within the context of this post the term ‘scracth the itch’ is used to describe having a demand for GIS software, rather than a desire to work here or there. In many of the consultancy jobs I have worked on the client has ‘the itch’ and has decided on the tool to scratch it with. In my case this has mostly been Esri software. Essentially I have rarely been faced with solving problems for clients where there has been a choice, in some cases it has been my job to promote Esri software as the only choice – and I have done this. In my experience the client is often saying ‘we want you to do this job with this software’.

    This throws up some interesting questions about how consultancies are generally engaged to undertake work. For example I tend to find many tenders skirt over the value which the organisation seeks to gain from undertaking a piece of work. If the value which can be delivered is placed at the very centre of the project, then perhaps the technology should (rightly?) take a back seat. A recent trend in ‘stick your nose in the trough’ licensing seems to perpetuate this trend of dictating technology over value, where we have software vendors providing licensing models which arguably encourage longer term software lock in – this is an unfortunate development IMO.

    It is great to have some time to be able to look at some common problems which GIS tends to solve and ask the question ‘how can i solve this?’ without having a software blue print. As I tried to convey in the post:

    “Personally as a GIS consultant it’s no longer acceptable to me to restrict my view, and Open Source GIS is helping me open my eyes.”

    ‘Open Source’ is often bounded around as being a single alternative option to a commercial product: “I hear they’re moving to ‘Open Source’”. I view it on the other hand as a collection of projects each one driven by a different motivation and purpose, each one in a different stage of maturity. It seems to me to be one of the best eco-systems to look at if you’re interested in broadening your understanding of GIS software.

    Simon

  6. TellyTubby of the North / January 25th, 2012 7:43

    “The good life” has never been the same for me after the young ones laundromat episode.

    Whether or not you consider it baggage, there is a just *a lot to know*, about any of these platforms. The personal investment that ESRI users have jointly made dwarfs ESRI the company.

    These days I’m more reluctant to invest in something that will never quite be mine.

    The other untold story imo -is the age/experience one. ESRI does good business supporting people without a computing background doing things with computers.

    [baggage on]
    When I was young and inexperienced, ArcINFO was a fun place to explore and get things running that I never could have built myself from scratch.

    But you grow up -and unless you retire into management, or are very domain (geo-science!) focussed then then the only way you can contribute is to do boring, trivial things with much pomp and ceremony -viz. enterprise computing.

    [bunch of crap deleted]

    SO -rather than fretting about your ESRI baggage I think you should just see yourself as growing up and moving out of your parent’s place.

    I’m presently poking at some ideas (and a possible contract) that might take me towards the elusive “interesting things to do back in the ESRI world” -but alas, I do fear my own trajectory as an ESRI user would be towards being a Douglas Crockford viz. perhaps correct, but increasingly old and crotchety -and a pain in the arse. I guess I might write a slim volume called “ArcGIS, the good parts”.

    Footnote: The guy who said “Hey man, you’ve just got to leave all that relational baggage at home, man” was clearly a complete tosser! No-sql amounts to a bunch of lemmings throwing themselves off a cliff, in the belief that they will suddenly need to scale to fifty million users in the two seconds before their frail bodies are smashed to pieces on the rocks below.

    The technology is seductive of course (I found a talk about using cassandra for spatial data the other day -but i personally fancy bucking the cassandra trend and leveraging voldemort views..) but the price in terms of inflexibility is frightening. And don’t get me started on how the (oh so lauded) Dynamo paper perverted real cloud computing… ;-)

    Footnote to footnote: have you heard about ESRI’s no-sql technology? I have seen it running recently (handling a “big data” problem that was killing ArcMap). Man was it ever fast! It is called Info. You should check it out.

    T.T

  7. matt wilkie / July 11th, 2012 7:11

    Simon, I like your style. Thanks for writing and sharing your views and experience. As someone who started with Arcinfo Workstation, works in a near 100% Esri office and understands that’s what has bought the house and fed the kids I found myself nodding in emphatic agreement many times. I’ve long been an aficionado of open source and it has a permanent location in my personal toolbox (gdal how love thee, let me count the ways…), but very little leakage to past my cubicle.

    TT North, yeah. Info. Aside from speed, I don’t miss it much, but my GIS work has always focussed more on geometry than db. What I really miss are Region sub-coverages (supported by Info). The datamodel and process for creating them was hell to understand and made my brain hurt. Dumping the coverage model and designing a new storage type which allows the whole globe to be held in one container without loss of precision is an evolution I’m truly grateful for. Dropping Regions at the same time though is something I don’t think I’ll ever forgive Esri for. It’s a classic baby bathwater story. I don’t think the open source world would have let that happen.

    (For those who don’t know what Regions are: imaging having *one* polygon feature class that let you individually and collectively draw *and* analyse lakes, rivers, wetlands, and intermittent shallows as the semi autonomous and grouped units they are without resorting to the jiggery pokery of symbol level drawing and/or adding the same feature class multiple times to your map. See http://gis.stackexchange.com/questions/8571/how-to-hide-internal-polygon-boundaries)

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>