My Baggage

Over the past few months I’ve been making an effort to drop my Esri baggage. I’m trying to get all Hugh Fearnley- Whittingstall on GIS – I want to start feeding my GIS needs on shit I know the provenance of, and perhaps learn how to manage it all myself too.

I’d like to think that I’ve been an advocate for Open Source GIS software for a while however not at a grass roots level, I’ve never committed a line of code or doco to an open source project. My motives are driven by a number of things; Firstly a self interest in finding out more about alternative open approaches to software development, secondly I’ve never been completely convinced that the traditional software vendor model was the best way forward from a client perspective and thirdly I wanted to understand the choices my clients had when it came to spending their pennies on software. In 2006 I was reading more and more about open source GIS software, I attended the 2007 Foss4g and I was part of the 2009 Foss4g organising committee, but my day job has always been centred around the needs of clients with Esri software so I’ve never been in a position to scratch my own itch.

I have always been frustrated by the polar views on both sides of the proprietary\open source fence. It’s as annoying to me to listen to a dim witted software sales person drop FUD as it is to listen to some super *geek* who believes you’ve just committed a heinous crime by running windows.  I’ve not succumb to the pusherman either – I have always made conscious choices towards proprietary GIS software mainly because its been the source of my mortgage repayments. I still like writing software on top of Esri stuff, but I’ve felt for a long time that I’ve been going through the motions, when Esri move, I move, when Esri change their minds I follow and to be perfectly honest I’m tired of listening to the same record.

So I’ve taken some early steps in untying the apron strings and thought I would share some of my opinions.

1. People love security

I’ve always wondered why more people don’t shop around for mortgage deals every few years, like there’s some unwritten rule that when you take a mortgage with a bank you’re signed up for the 25 year term – people tend to like security, or what they think is security. In environments which are naturally safe then adoption to new technology is generally sluggish – people tend to want to use something they’ve used before or perhaps something they’ve seen their neighbours use. From my experience local government is a good example of this, I’ve found that geographically coincident governments tend to use the same GIS software, because they can easily share experience and trade war stories and scars about this or that vendor.

I have a security blanket too, we all do. There’s not many people I know who have an intimate knowledge of a large spectrum of GIS software. However I’ve found it very rewarding going for a few hours a day without my security blanket, I feel free. If I was starting a self help group I would perhaps be chanting ‘It’s OK not to open ArcMap, you’re still a good person’. In some case security comes in the form of a group of people who can support you or help you deliver and there is an emerging market of providers in this space who are wrapping open source software in a blanket just for you.

2. Glue

Esri sells software partly because their components fit together neatly like jigsaw pieces. I’m sure Esri will continue to develop this approach, making it as seamless an experience as possible to work between these components – its a sensible approach. They use language like ‘system’ and ‘platform’ because this is a key selling point for them, it described something larger, more unified. As each version rolls off the production line we’re told ‘stories’ about how easy it will all be in this release – this is an appealing story to many people, and very valid in some environments.

Open Source is challenged in this space – projects indepently emerge and don’t automatically need to belong to an umbrella group, who will help manage this story. To bridge the gap they rely on collaboration, standards and well used interchange formats – essentially knowing your way around these will help you to glue the open source projects together. I’m not trying to paint an unfair picture by calling this glue, it just seems appropriate to me. Some providers are selecting a set of well established tools and standards to base a more unified ‘system’ on – this makes perfect sense to me, this reads like the ‘new and improved’ Open Source story, and I believe it has value.

Personally I’m discovering a love for tools and technology I’ve never used before – it seems exciting to be out of my comfort zone – one thing which has really struck me is that whilst things do occasionally stumble (see no. 3) I have found it on the whole very straightforward. Even when barriers appear the availability of decent resources to allow you to move forward is abundent.

3. Open Source Crashes too

I don’t know how many times I’ve seen and read about ArcMap crashing! People love a good bitch and moan about Esri software, some GIS people just love a good bitch and moan. Open Source software crashes too – I’ve been using QGIS a bit lately, it crashes too, however this is my take on it; Firstly I’m fairly tolerant when it comes to it, because I don’t use it all the time. I just tend to restart and move on. Secondly if it bothered me enough there are a myriad of places to go and seek a fix or I could just try and fix it myself. This is of course the freedom I have when I use these products. If I had the time I should really do some work and provide back diagnostics to the projects and support a fix, but as I said these squeaky wheels are all acceptable to me. My guess would be that if the adoption of QGIS rocketed to the same levels are ArcGIS then we’d be hearing a lot more about QGIS crashing (bitchy and moaning GIS people remember), my second guess would be that we’d be waiting a fraction of the time to see these issues resolved.

Software, proprietary or open source has its issues. Personally I like the idea of being able to resolve defects or amend the software myself (if I have the skills and the need to do so).

4. Home Brand GIS

Heinz baked beans are always front and centre when I buy my beans at the supermarket – and I buy Heinz (English variety of course). Now I won’t go into why I buy Heinz too deeply, but I suspect it’s something to do with child hood memories and a conditioning (see No. 1 above). It’s probably also fair to say I’m a sucker for packaging, so when i glance down at the bottom shelf at the home brand beans I’m simply unimpressed by the label, irrespective of the fact that I know the beans are good quality.  Open Source GIS in some respect has a home brand feel about it to me – I know its good quality, but the packaging can be a bit shite. Now if you’re not a shallow, little brained brand man like me then you’ve probably be eating home brand for some time and can’t get enough – you don’t care about the packaging you just love them beans.

But what does the packaging matter? Well firstly it matters only when the product and service that it is wrapping is good quality – if you’re brand loyal for a product and service which is inferior then you’re more of a dickhead than me. When I’ve moved across to open source projects like QGIS, GeoServer and TileMill recently, I’ve had to (in some cases) put aside the packaging and judge the results against what’s in my Esri toolkit, and I’m liking what I see. Once you’ve established an understanding of product quality I think its fair to look at the packaging -   is it mature? Does it have good support? (perhaps commercial if required). Does it have good documentation? Can I get my hands dirty if I need to? What is the user exeprience like?

5. Esri has given me pre-conceptions about GIS

Esri has given me preconceptions about GIS. I’ve never been particularly comfortable with Esri’s terminology such as the ‘geodatabase’ or ‘enterprise GIS’ or ‘geodesign’. I can’t help feeling these terms are manufactured. Going outside the Esri world for the first time in years has made me feel like I’m learning first principles again, without my Esri baggage. I don’t wish to spread FUD and perhaps Esri is legitimate in its desire to talk openly about the problem spaces which GIS can solve – however I feel now that when I look at these problem spaces I’m no longer focusing on an Esri technical approach to solving them, but rather looking at them with a much wider lens. Personally as a GIS consultant its no longer acceptable to me to restrict my view, and Open Source GIS is helping me open my eyes.

6. Organic GIS Software

After a while working with Esri software and for a distributor of Esri software I felt like I was going round and round on a mouse wheel, an endless cycle of release, service pack, service pack, (promise performance improvement), go to conference, cram more features in and start again. This is after all commercial software – you have to feed the beast right? Yes -  in amongst this the products evolved and improved – no argument or criticism, but it’s not for me. Somewhere along the line I felt that this endless cycle and growth begins to erode the software. Endless extensions, feature bloat, UX problems, quality issues, multiple divergent APIs, all constantly accruing technical debt.  Again don’t read this the wrong way – these problems are not unique to Esri, they seem to me to be a condition of growth, and perhaps like battery farmed chicken, there is a drive to keep your costs down and margins up and hey’ the punters just keep coming back for that tasteless bird!

Open Source GIS software I would argue comes from a different eco-system, where there are multiple smaller projects and competition thrives and drives improvement. Despite my comments above about quality – on a whole I’ve found the quality of open source GIS software to be as good, if not better than similar commercial products (See Update *). This eco system is maturing to provide choices in terms of software support and development and demand is increasing. I love the lack of marketing and sales tripe in Open Source too. Its straight up, no bullshit, does what is says on the tin, wholesome GIS.

Footnote:

The title for this post came from the following story:

Back at the 2009 Foss4g Volker Mische had just done a great presentation on CouchDB and was taking questions from the floor. A legitimate question came forth from an individual about how one could represent relationships a la RDBMS in these new fangled NoSQL DB’s, before Volker could answer the question, a friend of mine shouted from elsewhere in the crowd:

“Hey man, you’ve just got to leave all that relational baggage at home, man”

Update

* Thanks for the feedback from Anthony – I have no quantitative evidence to support this claim. FUD, guilty as charged.

Brewing Maps with TileMill

The Melbourne Open GIS meetup has been growing for a little while now and last Thursday it was great to see our biggest crowd yet assembled at OpenHub for some beer and banter. As usual it was a good melting pot of folks who share a common interest in maps.

After some shoddy technical ramblings on my part and a last minute dash to the *JOHN* (who incindently could have been the first person to invent tiling…) I did a short 50 metre sprint on TileMill – the slides are below, but you should probably just skipt them and download it.

A big thank you to all those who came along, looking forward to the next one.

Simon

GeoCoder

This post is inspired in part by Eric Wolf’s comments on the recent NoGIS post by Sean Gorman.

Whilst I’m not going to pretend to truly appreciate what’s pythonic or not I agree with Eric’s comments and have my own twopence to contribute to the conversation. You may ask:

“What qualifies you to talk to these matters, you mouthy Northern….?”

Well I’ve spent about 10 years working as a pseudo geographer\software developer. I say pseudo because I’m really neither of these things. I am ‘qualified’ as a human geographer (I know, I know), and I’ve been known to write some software, but like many people I meet in GIS, I am here entirely by accident. In my case the accident was a prolonged binge drink fueled university rampage. Anyway my point is that I staddle both camps, I have been a shit software engineer in my time and an even worse geographer, but I’ve also done some things in code that I’m proud of and on occasion I’ve been capable of placing space in the centre of things.

Eric states:

“To me, the nature of noGIS is people who’ve learned enough computer science to realize that the Goodchild/ESRI model is fundamentally flawed. Instead of teaching programming in the Geography Department, someone should be teaching a little Geography in Computer Science Departments.”

I couldn’t agree more, and here’s why. I’ve seen some car crash software developed under the banner of GIS and much of it rolled out by Geographers pretending to be software engineers. Now that’s not to say there aren’t great software engineers out there who are hardcore GIS people too, but I’d like to share some of what I think are some of the common traits of a ‘GeoCoder‘.

1. Reach for code….all the time

Some of the first code I came across was Magik. Now I’m not reflecting on the language, however during this time I learned that a disgraceful amount of code was being written by Smallworld application developers because GIS folks* would want something to work a certain way – this still goes on today – layers of ArcObjects is written all over the place to deal with ‘work flow’ issues or ‘productivity issues’ which often don’t really exist. It seems like as a profession we’re hell bent on creating new tools instead of using the ones we’ve got to drive new results. I’m not suggesting that we shouldn’t  recognize tooling deficiencies but rather when we do, evaluate the cost of re-tooling against the gain and if new tools are required then don’t become a blacksmith yourself.

*I make the distinction here between GIS folks and people whose primary motivation for using GIS is to achieve a goal. The former are often employed to administer systems and data – the latter just want to use the tools to get shit done.

2. Ugly code

Not Designed. Not tested. Not re factored. Not reviewed. Not defensive. Not Good – just ugly.

3. Heavy Lifting & Home cooking

GIS coders, don’t code enough to understand when they’re heavy lifting. I’ve seen this loads –  home cooked configuration, data access layers, user interface components etc. If you find yourself writing a lot of code to solve an issue – the first question in your head should be – hang on a second, what am I doing wrong here? What problem am I trying to solve and hasn’t someone else been here before me? Patterns & Frameworks my friend.

4. Over cooking

This is kind of related to point number one, but it’s about over complicating code – GIS coders are prevalent to over cooking code – they add complexity and configuration where it’s not required, they write verbose code which reeks. Is this down to a selfish desire to tinker with more code? Know when to stop, keeping it simple will aid your software usability and maintainability. If you hear the words ‘we might need this down the line’………..chances are you probably won’t.

5. “But you don’t understand spatial data is different.”

This old chestnut is total guff. I was recently working with a non GIS coder who was stumbling slightly on some geometry conversion issues, it took all of five minutes and a sketch on the back of a cigarette packet to explain all there is to know about Geometry. (That’s not an entirely accurate statement). Why do we cling onto this nonsense in desperation? Back to my human geography days – as geographers we seem to continue to struggle with our identity, especially in a climate where even more people are engaged with maps – this is no bad thing. The fact that spatial is not special is something we should be proud of – somewhere along the way we must have done our jobs right (Or did Google just do our job better?)

Of course not all ‘professional software’ developers are great coders either – shit  - is there no winning! On balance my experience makes me agree with Erik – Some geography basics taught in computer science courses would go a long way.

Now to NoGIS – #NeoGeographyReDux

Consultancy Myths

I’ve been quiet for the past few months. Things are hectic and I’m trying to hold down a new job, two small children, a marriage and a drinking habit – what can I say.

I recently departed development for management – a big step for me. I feel like I’m lost, waiting to be found at the moment. There are many reasons for the change, too many to list here; you’ll be relieved. Towards the end of my old life I was beginning to reflect on the role of consultants. I was and still am (to a degree) a consultant and over the past few years I’ve had the opportunity to gain an insight into some of the myths of consultancy and also some of the things I would want from a consultant if the shoe was on the other foot, which it now is. So here are some random ramblings about consultancy, in part inspired by Scott Berkun‘s ‘Calling bullshit’ posts.  I should also say that this is my myopic view based upon about 10 years in the GIS consultancy game.

Oh, I nearly forgot I also should thank my western beer buddies for an inspiring chat a few weeks ago.

1. Consultants know everything

Bad consultants act like they know everything, good consultants will tell you when the don’t. Learn to sniff out the bullshit. As a consultant I know where my strengths lie and I want to be the first to direct a client towards the right person should I not know the answer. I remember a proposal I once wrote for a client which was reviewed by a colleague of mine. In the proposal I had ruled a piece of work out on the basis that it was outside of the realm of our company profile, expertise and experience. Upon review my colleague suggested that ‘we should never say we can’t do something which a client requests’.  This ‘know it all’ consultancy is also very tiring for anybody who has to be on the receiving end because bad consultants, those who know it all, invariably love the sound of their own voices which is exhausting for those poor souls in the room having to listen.

In my experience I may have done something many times, but I rarely know everything – each project and client brings a new set of challenges. An implementation of the same software can be vastly different across two clients whose environments and processes differ.  What’s important is having consultants on your team who can navigate your differences using their skill and experience.

2. Divorcing responsibility

I am continually amazed by how many clients believe their job is done once the tender is awarded – its the most dissatisfying experience to arrive on site and feel like you’re expected to pull the rabbit from the hat without ever asking a question. I’ve never seen a project succeed when a client divorces responsibility from the project they’ve hired the consultant to deliver.  If you’re bringing in a consultancy practice to deliver a project and you’re expecting to leave them in a dark room and shut the door, then think again. Consultants are often specialists so they don’t necessarily see the whole, make sure you are around to fill in the gaps – you’ll make you’re investment go a whole lot further.

3. Bad consultants agree with you

A good consultant will challenge your assumptions and ask hard questions. But you don’t just want to hear that, so at the same time a good consultant will help you move towards a solution. The important point here is ‘help you’. I don’t necessarily want to have a consultant on staff full time – they’re expensive for one thing, so keep that in mind and have an expectation that a good consultant should tell you where you’re going wrong as well as guide you to corrective action.  Expect to be challenged, if you’re not looking for this, just hire some low cost yes-man and give yourself a pat on the back – you’ve done a great job. Well done.

4. Try before you buy

Don’t be afraid to try before you buy. Any consultant worth their weight will give you a little of their time so that you can understand who they are and what they’re about. Not all consultants will be a good fit for your organisation, team or project so its worth doing this – no in fact you should insist on this.  I want to at least chat to the consultant who will be working with my team, I want to know what other work they’ve been doing, what experience they have in the particular area I am interested in, and most importantly whether they are someone who can bring options to the table. Do your homework and don’t be afraid to make some calls and find out what others have experienced. A little effort up front can save you some sheckles.

5. Fix pricing is nonsense unless you’re building a shed

I generally think fix pricing is nonsense unless you know your dimensions very well. In my experience projects invariably lack requirements that constitute accurate dimensions making it very difficult to estimate and fix price. Consultants will add ‘fat’ because they know this, you know this to, so why do it. I know, I know, ……….your organisation has a policy…but try and convince whoever needs to be convinced that there are alternative models out there, which can be much more cost effective. Fix pricing can often lead to mis-trust and scope wrangling because the default position is one of risk mitigation – look at negotiating a good T&M rate and regular short iterations which allow you to monitor your costs and direction.

If you ever find me in your midst, make sure you call me out on these things

ST_MULTILINESTRING….ESRI Bug?

Recently our client found a bug They were testing the services and tried to create a long linear location – suddenly the server returned a nasty oracle error. Everything had worked fine up until the application server tried to write the geometry as WKT into the database at which point things went astray. To resolve this issue we built a small test which uses ODP.Net:

As a bit of background I’ve been working on a project which delivers a set location translation web services – basically the organisation collect data using a number of methods some of which have a linear aspect. They want to be able to translate between these methods as well as validate the locations to ensure a consistent data quality and finally enrich each location through spatial analysis. In effect we’ve been developing an advanced linear referencing locator.

We made a decision with the solution that we would utilise as much functionality at the database level as possible and that means we use ST_Geometry where we can. Unlike some implementations the ESRI ST implementation does not provide linear referencing functions therefore the web services which are served out from an application server communicates with our spatial server hosted within ArcGIS Server. The spatial server component provides dynamic segmentation and some other spatial functions. For linear locations the dynamic segmentation operations return ArcObjects geometry which is parsed and returned to the application server as WKT.

All has been fine until last week when our client found a bug They were testing the services and tried to create a long linear location – suddenly the server returned a nasty oracle error. Everything had worked fine up until the application server tried to write the geometry as WKT into the database at which point things went astray. To resolve this issue we built a small test which uses ODP.Net:

//wkt
string WKT = "some well known text"
 
//create parameters
OracleParameter p_wkt  = new OracleParameter();
p_wkt.OracleDbType = OracleDbType.Clob;
p_wkt.Value = WKT;
 
OracleParameter p_srid  = new OracleParameter();
p_srid.OracleDbType = OracleDbType.Int32;
p_srid.Value = 0;
 
//create command
OracleCommand cmd = new OracleCommand();
cmd.Connection = conn;
cmd.Parameters.Add(p_wkt);
cmd.Parameters.Add(p_srid);
 
//sql
string sql = "INSERT INTO GEOMETRY_TESTBED VALUES(1, SDE.MULTILINESTRING(:p_wkt, :p_srid))";
 
//execute
cmd.CommandText = sql;
cmd.ExecuteNonQuery();

So basically Oracle was spitting a dummy when I was inserting a lengthy piece of text as the WKT parameter, returning an ORA-01461: can bind a LONG value only for insert into a LONG column. Basically when the WKT was truncated to less than 4000 characters long the insert was working. I wanted to try and insert a very long geometry so I selected (ST_ASTEXT & ST_LENGTH) the geometry from the database with the highest number of points (ST_NUMPOINTS) which returned a 400,000+ character string using the ST_LINESTRING M constructor, I updated the code to:

string sql = "INSERT INTO GEOMETRY_TESTBED VALUES(1, SDE.LINESTRING M(:p_wkt, :p_srid))";

Insert worked fine. So lets try another geometry – this time using ST_NUMGEOMETRIES > 1 I selected the longest multi part with the highest number of verticies. This time the database returned a geometry with the ST_MULTILINESTRING M constructor, I updated the code to:

string sql = "INSERT INTO GEOMETRY_TESTBED VALUES(1, SDE.MULTILINESTRING M(:p_wkt, :p_srid))";

Insert worked fine! Ok then, now I tried my geometry as returned from my ArcObjects parser – this time amended to have a dummy M value appended to each coordinate, and using the ST_MULTILINESTRING M constructor. Worked!

Is this a bug with the ST_MULTILINESTRING constructor within the ESRI ST implementation on Oracle? Somewhere under the hood is the UDT using a varchar when it shouldn’t be?

Has anyone else seen this issue?