I think as GIS folks we’ve been educated to understand spatial information and analysis, and most importantly we see the benefits that both of these can provide if they’re used effectively. Arguably its been our goal to broadcast this benefit to others and raise the adoption of GIS technology. However to help people realise the benefits we need to step outside of our little GIS world and think like real users  - users who want to simply use spatial information, analysis and tools to achieve their own goals.

If my memory serves me right the origins of this rant go back a few years to a drunken night in a Nelson pub with the North Shore Telly Tubby and my mate Martin.

Anyway – there were some other really interesting folks there last night too and I particularly liked NeatStreets and our follow up chat about Viz and the profanosaurus! Get out there and tag some BSW’s!

I’m fully impressed!

A mate of mine has been making some suggestions to the OpenGeo team about their latest OpenGeo Suite Community Edition – they listened, asked for his input and released new functionality all within the space of a few days – this is agility at its best!

There’s so much that is right about this story:

1. A dude spoke up with a legitimate use case and someone real was listening

2. They talked about the idea and recorded the process openly

3. They implemented the change and released quickly

The willingness to listen, the ability to evaluate a good idea and quickly release something usable is an impressive level of service, and one OpenGeo should be proud of. On their getSatisfaction site it says ‘OpenGeo employees are here to help’.  Simple, no need for anything else – it’s just true.

It seems this sort of thing is all the rage these days.

PS. In the same thread there’s a link to a nice clean community edition site which is well worth a look

Paul is back on the attack – this time he’s likening proprietary software to crack. Brilliant

It’s worth taking Paul’s analogy a bit further. Being an open source ‘evangelist’ he believes that proprietary software is bad for you. The two main points he makes are:

1. Proprietary software is pushed onto unsuspecting audiences by software vendors

2. Proprietary software is addictive

Lets take the second point first. People generally get addicted to a drug for a number of reasons such as;

a) their conditions

b) something in the drug is physically addictive (e.g. nicotine in cigarettes)

c) they enjoy it

Software is not dissimilar. People go with proprietary products because of their environment – they may work for an organisation which has made an investment in a particular product, or they may select a proprietary product because they feel comfortable with it – not necessarily because of actual familiarity with the product, but often because it seems more familiar or it seems less risky than other options. Sometimes proprietary products include something addictive – data lock in is a good example, once you’re using format X is very difficult to move away from it. Finally people often use proprietary products because it is good at what it does. It’s a tool which helps them get their job done – I’m not saying that there are not issues with proprietary products but to not accept that they are competitive is naive.

The first issue, that proprietary software is pushed onto unsuspecting audiences by software vendors is not new. This happens everywhere – just this morning I walked out of the train station and a new brand of yoghurt was pushed into my hand. Free samples of this and that are given away by companies intentionally to get you interested, to get you hooked – this is marketing! What’s wrong with this? I think Paul makes this point because he’s an open source geek and open source geek’s generally don’t get marketing – they see straight through it. They view their non-geek ‘express’ users colleagues as helpless against the evil ‘pusherman’ whose marketing mechanisms lure them in and gets them hooked.  But is this an accurate picture – I didn’t particularly like my free sample yoghurt. I certainly won’t be buying any. People do have the ability to make informed choices – some more than others, and they make these choices based upon many factors.

People are interested in the ‘express’ editions because on the surface of it the marketing works on them, they’re familiar with the brand and are attracted by the ‘free’ carrot dangling from the end of the stick. Open Source software starts from that position – it’s already free so open source projects need another carrot to get us hooked.  I think Paul needs to get Marlo Stansfield on our asses  - you gotta get in the game bro – you’ve got to convince us your crack is better, gets you higher quicker and lasts longer……and like any great drug it doesn’t have any nasty side effects. Alternatively you could develop a 12 step program to help those of us who are addicted to sucking on that pipe.

Late last year I was really pleased when I read Martin Daly’s ‘Higher Education’ series of posts. Here was a real story about about a real project – which had a real resonance with me. As one large project draws to a close, (..and inspired by Martin’s posts) I thought I’d share some of my experience about the reality of projects.

It shouldn’t be surprising to hear that sometimes projects have their ups and downs –  no amount of GANTT charts can help this situation!  So it was little surprise to me that towards the end of this large project we had a few bumps – I should say that whilst I wasn’t surprised I was frustrated because we’d been putting  a lot of time and effort into making sure we tried to unearth such bumps early!!!

The bumps

1. Stability

The first bump was associated to a data loading component in our solution. This component automates the processing  of state wide source data into a complex data model. We knew it was a pretty intense process and we’d spent a lot of time making it perform – because performance was important to our client, however while we were looking one way we forgot to to look the other and we let a stability issue pass us by. Bump!

2. Functionality

The second was a particular piece of functionality which we’d implemented into our services tier. This functionality was fairly straightforward – we had to analyse a geometry that was defined and created in memory at runtime against some layers – again defined at runtime to return some values (once again defined at runtime). All our tests had been successful until we encountered a specific type of geometry, and a bug in the underlying technology. Bump!

Why did we bump

I don’t need to explain in detail about these two issues, but its useful to try and reflect on why we encountered them, after all we were using SCRUM and reflecting is an important part of the process:

1. Iterating is not enough

We had a series of sprints, at each sprint the usual scrum ceremonies took place – every few sprints a larger set of the client team joined us and we spent a full day going through all the functionality developed up to that point. But iterating is not enough – we were missing the final press of the GO button – deploying each iteration. There were reasons why we couldn’t deploy in the early stages. We were deploying onto our internal test infrastructure from our development infrastructure and we had our CI server doing the heavy lifting for us, but we underestimated the reality and value of the software being directly in the hands of users from as early as possible.

2. Look out for the warning signs
There were warning signs for the stability issues. I missed them. Not being able to deploy frequently onto client infrastructure stopped the issue bubbling up earlier, but even in our own testing and deployment environments we dropped the ball. I think our focus on the high priority features such as performance was a factor in this.

3. Testing the boundary

We did a lot of testing, but we missed a crucial boundary area. In fact its more accurate to say that we would have benefited from the client scrutinising those crazy boundary cases that only they’re familiar with earlier on. This omission was enough to let that functional issue slip through the cracks.

Getting over the bumps

It’s important to look at how we resolved these issues as well as understanding why they occurred in the first place:

1. Review

In the case of our stability issue with the data loader our solution lay in a review and memory profiling. We’d done code reviews during development as well as testing and review of UI functionality and of course end to end testing of the load process (which takes around 6-7 hours). When we encountered the issue we conducted some careful memory monitoring and we came to the conclusion that we had a leak or some nasty fragmentation going on, but we were struggling to isolate the real cause of the problem. I’m lucky enough to work with some very smart people and a phone call or two later and a great piece of review we had the issue nailed. I’ve got a growing belief that continuous review is an extremely important process –  from pair programming to formal code review – these processes are invaluable.

2. Good design…and a good backup

An iterative process  can lead you to design with change in mind, just like unit testing can lead to more carefully and considered design. Good OO design and constant refactoring can help you overcome changes as they occur. I like to think we did these things and this prepared us well for our functionality issue. I didn’t have to pull the system apart when we found the issue in our initial design, I’d given myself enough room to move. I also had a back up design. Its really important to record the design process IMO – notes, sketches, and ideas may all be useful at a later date. It’s also useful to not be afraid to throw away something. My design allowed me to throw away the problem without a knock on affect. I could fairly rapidly test new designs and when it came to it get a fresh perspective on the issue.

3. A great team

This was the most important factor in overcoming the bumps. Being able to bounce ideas of others in the immediate and not so immediate team helped a great deal. The commitment of the team to get through the issues was fantastic – the folks that worked on the project really wanted to see it succeed – this was a huge, huge factor. I think one of the main reasons why our team got over the hurdles was because our client worked with us, as part of the team. When we encountered the issues we didn’t brush them under the carpet  - we discussed the problems openly, assessed the options, provided advice, shared ideas and in the end helped each other to get it sorted. There is something deeply unsatisfying when you have to work in a client v’s vendor – us v’s them relationship –  I’m happy to say we left all that bullshit behind in about week 3 of the project.

The outcome

There’s lots of things I would change, but the project has been a success from my perspective – although we had a little stumble near the end its important to remember that we delivered a complex system comprising of database, server and client components. We listened to our client and hopefully they consider the project a success too. All projects have the odd bump here and there (some more so than others), if you hear otherwise you should be suspicious. Accepting that projects occasionally bump is not an acceptance of failure – it’s an acceptance of reality, an acceptance of change. When measuring the success of a project then I think we should include an assessment on how well the project manages its issues along the way.  I’m certainly going to take a great deal of this experience forward and hopefully there will be a few less bumps the next time.

I’m back from Palm Springs, and wanted to reflect on this years ESRI Developer Summit. In summary I had a really great week, meeting old and new friends and experiencing what is a really good conference. It’s a long way to go, but definitely worth it, especially given the nature of the 10.0 release.  Say what you like about ESRI, but I think post ADF they’ve lifted their game and there is no sign that things are changing on that front. So here are my personal highlights in no particular order.

1. Open Source Myths

Despite what people map think  ESRI are quite happy to have FOSS4G on display.

I also enjoyed a cool demonstration of GDAL analysis by the folks from the ESRI Prototype lab. I was disappointed to hear Jeremy Bartley’s reasoning behind not open sourcing the JavaScript API – apparently it’s an issue of ‘control’….?

2. arcpy.mapping

I’m not a pythonite but the new mapping API might put the very successful DS MapBook to bed. I thought about all those times I’ve sat there and done manual data audits at client sites – bye bye and good riddance to that manual overhead. Also the arcpy.sa includes some numpy goodness which pythonites will enjoy

3. GeoDesign

I may have had it wrong – I’m starting to come around to what this is about, I enjoyed the GeoDesign lightening talks especially the Azavea guys – they’ve done some really nice stuff. The concept of GIS as a scenario building tool is not new, but our ability to produce tools, especially on the web which quickly allow us to design and tweak our data to visualise our scenarios is a hurdle we need to leap.

4. The FeatureLayer everywhere

The new feature layer and feature service facility is implemented into each of the three web apis (JavaScript, Flex, Silverlight). A feature server is not a new idea, but ESRI’s implementation is powerful and integrates sweetly across the stack. I like the client side query features capability cutting down the need for round trips to the server plus feature layers are really the basic building block of the editing capability of the web apis.

5. Query Layer – you gotta love the SQL

I love SQL – nice and functional – the new query layer, low level within the geodatabase API (SQLWorkspace) and in the ArcMap UI is cool. It’s great that we no longer have to ‘register’ data with SDE to use it. I never got a chance to catch a hold of anyone to ask whether ST_ functionality can be used – but I understand pl\sql can not be executed via query layers yet.

6. Add Ins

It seems the add-in concept from ArcGIS Explorer has made its way into ArcGIS Desktop. This is an improved framework for creating customisations not only from a developer perspective, but also from an administrative perspective. Add in’s are compiled into a single compressed file with a .esriAddIn extension – these contain a configuration XML file, the binaries and any necessary resources (dependencies etc) and are registered with the client by being placed into a registered folder known to the client. There’s also a new installation utility but the old customise route is not disappearing. Interestingly there’s a new esriRegAsm utility as pre version 10 COM components have to be re-registered once 10 is installed – this may be a bit of a headache if you’ve got a lot of customisations in your desktop.

7. Eating your own dog food

Because ESRI are now running their own server in the cloud to support ArcGIS.com they have to ensure it’s a reliable stand up and high performance server able to run on hardware and cheap OS – They’ve been doing  a lot of testing across the board and have got some rather good results running on RHEL  - I hope they stand up and be counted at the 2010 WMS shootout?

8. REST binding for server extentions

There has always been a SOAP binding for server object extensions, but it was a bit dark and tricky around the edges – ESRI have addressed this and done more by adding in new classes for exposing server object extensions via their SOAP and REST channels – this is pretty powerful stuff. In a nutshell the XML and JSON serialisation/de-serialisation has been taken care of by some helper classes and a nice WSDL template is provided (you used to have to handcraft your own) for SOAP. New RestResource and RestOperation classes are provided for REST – at this stage the REST support is limited to mapping services. There’s also support for MSD services via flag with the ServerTypesExt.dat – and I liked the feature which allow methods to be exposed and configured as capabilities within ArcCatalog and (new at 10) ArcGIS Server Manager.

9. User Presentations and Developer To Developer Sessions

Love them – have more of them!

There were a lot more goodies on offer so if you didn’t make it or couldn’t make a session because of a clash then head to the developer summit site for the tech session and plenary videos. Finally thanks to all at DTSAgile for a great BBQ we used some sophisticated techniques to find the house.

Well not if you’re the the real daddy. How many examples are given here where a map is NOT used….

Recently our client found a bug They were testing the services and tried to create a long linear location – suddenly the server returned a nasty oracle error. Everything had worked fine up until the application server tried to write the geometry as WKT into the database at which point things went astray. To resolve this issue we built a small test which uses ODP.Net:

As a bit of background I’ve been working on a project which delivers a set location translation web services – basically the organisation collect data using a number of methods some of which have a linear aspect. They want to be able to translate between these methods as well as validate the locations to ensure a consistent data quality and finally enrich each location through spatial analysis. In effect we’ve been developing an advanced linear referencing locator.

We made a decision with the solution that we would utilise as much functionality at the database level as possible and that means we use ST_Geometry where we can. Unlike some implementations the ESRI ST implementation does not provide linear referencing functions therefore the web services which are served out from an application server communicates with our spatial server hosted within ArcGIS Server. The spatial server component provides dynamic segmentation and some other spatial functions. For linear locations the dynamic segmentation operations return ArcObjects geometry which is parsed and returned to the application server as WKT.

All has been fine until last week when our client found a bug They were testing the services and tried to create a long linear location – suddenly the server returned a nasty oracle error. Everything had worked fine up until the application server tried to write the geometry as WKT into the database at which point things went astray. To resolve this issue we built a small test which uses ODP.Net:

//wkt
string WKT = "some well known text"
 
//create parameters
OracleParameter p_wkt  = new OracleParameter();
p_wkt.OracleDbType = OracleDbType.Clob;
p_wkt.Value = WKT;
 
OracleParameter p_srid  = new OracleParameter();
p_srid.OracleDbType = OracleDbType.Int32;
p_srid.Value = 0;
 
//create command
OracleCommand cmd = new OracleCommand();
cmd.Connection = conn;
cmd.Parameters.Add(p_wkt);
cmd.Parameters.Add(p_srid);
 
//sql
string sql = "INSERT INTO GEOMETRY_TESTBED VALUES(1, SDE.MULTILINESTRING(:p_wkt, :p_srid))";
 
//execute
cmd.CommandText = sql;
cmd.ExecuteNonQuery();

So basically Oracle was spitting a dummy when I was inserting a lengthy piece of text as the WKT parameter, returning an ORA-01461: can bind a LONG value only for insert into a LONG column. Basically when the WKT was truncated to less than 4000 characters long the insert was working. I wanted to try and insert a very long geometry so I selected (ST_ASTEXT & ST_LENGTH) the geometry from the database with the highest number of points (ST_NUMPOINTS) which returned a 400,000+ character string using the ST_LINESTRING M constructor, I updated the code to:

string sql = "INSERT INTO GEOMETRY_TESTBED VALUES(1, SDE.LINESTRING M(:p_wkt, :p_srid))";

Insert worked fine. So lets try another geometry – this time using ST_NUMGEOMETRIES > 1 I selected the longest multi part with the highest number of verticies. This time the database returned a geometry with the ST_MULTILINESTRING M constructor, I updated the code to:

string sql = "INSERT INTO GEOMETRY_TESTBED VALUES(1, SDE.MULTILINESTRING M(:p_wkt, :p_srid))";

Insert worked fine! Ok then, now I tried my geometry as returned from my ArcObjects parser – this time amended to have a dummy M value appended to each coordinate, and using the ST_MULTILINESTRING M constructor. Worked!

Is this a bug with the ST_MULTILINESTRING constructor within the ESRI ST implementation on Oracle? Somewhere under the hood is the UDT using a varchar when it shouldn’t be?

Has anyone else seen this issue?

It’s been really busy at work, with the big project I’ve worked on for the past year coming to a close and I’ve started to work on some new projects. I’m looking forward to a change.  Amongst all the mayhem of tenders, my daily batch of Jira and the occasional bit of consultancy March and April are shaping up quite nicely thank you.

To begin with myself and my good friend Peter are heading to the Gold Coast to present at this years OZRI. We’re going to share our experiences of using SCRUM on a large GIS project. It’s been a great journey with plenty of ups and downs but I think that it’s been a success.  More than anything else I feel we’ve worked together as a team  to deliver on what the organisation wanted. I think SCRUM engenders a culture of working together which removes the old school vendor/client bullshit which often gets in the way of just getting on and getting the job done.  To top it all off I’ve managed to squeeze a pork pie into my slide deck – that alone is an achievement I am proud of.

Following OZRI I’m heading across to the ESRI Developer Summit – It’s my first ever ESRI event and I will be putting on my little c# badge, packing in the kool aid and soaking it up – I’m hoping to catch up with some of the people whose blogs I’ve read for years but never had the chance to meet  - i could go some Belgium BBQ action

Aside from all this I have an old friend visiting from the UK, tickets for Calexico and the legendary Buddy Guy and I’m heading to NZ to see my Brother – it really couldn’t get any better than this……..or could it

My good friends Tony and Harley are looking to add a their growing team.  Having worked at GBS I can recommend the team there – a great bunch and a really good attitude to work and life. Hey you might even get to fly on some very small planes over the best landscapes in the world, work on some really interesting projects, meet some great people and have the occasional social beer

Find out more here