SD Cards and beyond

Saturday, 23 February, 2008

SD cards are all the same right? You stick them in your camera, PDA, laptop etc and they just work? Well have a look at the above article and you’ll quickly realise that this isn’t the case. SD, SDHC, mini-SD… the list goes on. And its caught me out. I had a well loved Palm Zire 72 which, yes, has an SD card slot. And it supports SD, but not SDHC cards.

And what is SDHC? Well SD cards had a physical limit of 2Gb based upon the transport method used to access them. As capacities grew this became a problem and so SDHC was developed to overcome this. Exactly the same physical specifications, just a different transport method used, although with a disk format that could address above 2Gb. FAT16 had been the format of choice and was replaced by FAT32. OK, so new cameras could access these large capacities (at 16Gb now), but older devices couldn’t. If the device had an OS or firmware, then a new driver would be needed to recognise SDHC. And of course work with FAT32.

Fast-forward to a couple of weeks ago when I became the proud owner of a Palm TX and, yes, you’ve guessed, it doesn’t support SDHC. Its a great device, but having shipped originally in 2005, no SDHC support was needed. Unfortunately Palm have a history of NOT updating the OS of existing devices; you have to buy a new one. Except of course the TX is the most up-to-date Palm PDA. Added to this the sale of the OS to Access software and the extreme delay on their own Linux based replacement and we have this current problem, which means 2Gb cards are the maximum supported. Even more galling with their smartphones, using the same OS, do now support SDHC. One minor useful hack is at Palm PowerUps where a FAT32 driver has been back-ported to the TX. SD cards (not SDHC) are available upto 4Gb which means you can format it in FAT32 and access the entire volume in one go. Neat!

Cadcorp and GeoPDF

Wednesday, 20 February, 2008

Geoplace report on a recent tie up between Cadcorp and Terrago’s GeoPDF. I’ve already blogged about how much I liked Cadcorp’s layered PDF export and the opportunities for good quality output (a significant benefit over ArcGIS). Well TerraGo have been developing along similar lines including ArcGIS and Acrobat extensions. Well it seems that Cadcorp have licensed GeoPDF creation for its users in what should be a mutually beneficial tie up. It’ll be very interesting to see the results.

Free Our Data @ Kingston

Monday, 18 February, 2008

OK, well its not quite the Free Our Data Campaign roadshow, but Charles Arthur, Technology Editor at the Guardian and one of the campaign founders, was speaking at Kingston last week. I would love to say that his talk was excellent, well founded and thoroughly thought provoking, but unfortunately I was ill and my colleague Ken was covering for me. However Charles apparently did stimulate quite a bit of discussion which he mentions here. So yes, we will be able to post the video once its been processed (and I’ll be watching it!).

Super-duper-Zoom

Wednesday, 13 February, 2008

192.com have launched a beta version of their “Super Zoom” aerial imagery. I saw this on Mapperz last week and thought I would check it out. And yes, the UK is covered at 12.5cm (no surprise there) and central London at 4cm (although their press release doesn’t actually state the resolution, the date of acquisition or where the imagery came from)!! The Guardian have a thoughtful piece on the privacy implications. They interviewed 192.com’s CEO and note the 4cm resolution, flown during 2007 and licensed from the OS. Anyway, the data itself is amazing and well worth a look.

Pythonian Adventures 3

Wednesday, 6 February, 2008

Over the last two blogs I’ve briefly covered why we may want to use a scripting language and how this fitted with ESRI’s view of how we should use ArcGIS. Since late last year I’ve been working on a small project that has required a not too complex, but quite fiddly, process to run iteratively over ~1000 files. This was definitely a job for scripting so I bit the bullet and started playing around with Python. My colleague James O’Brien does much more on the programming front than me and he set me on the right path with a nice sample script to work from (thanks James!). He also fielded more than a few calls with me cursing Python and ArcGIS. The ESRI-L list was also very helpful in getting responses to quite a few queries.

And in fact the best place to start is not with Python but with Model Builder. As I noted previously, Model Builder is an excellent method by which you can visually develop your process and save it. You can also run it on a single file to test all the stages of the model. Its also nice and easy to use which is a bonus!

I did actually think I would be able to do all my processing in Model Builder entirely which would have been nice. However it is painfully obvious that Model Builder was NOT designed to work iteratively and loop through the processing of multiple files. This functionality has actually been added in ArcGIS 9.2 however it is a real kludge. You need to set up a count variable at the beginning, setting the loop properties on it. The model properties then need to be changed to reflect the iterative status. You then need to spend ages working out how to add multiple files to the input variable; another kludge. This did actually work, EXCEPT the processing I needed to do required setting of ArcToolBox environment variables on EACH loop. You CANNOT do this in Model Builder. I would consider the looping functionality to be still in beta and it may serve some purposes if you can fathom ESRI’s logic. Is it too much too ask for a fully functional, well designed, product?

As an aside, when you are using Model Builder you’ll notice there is a “Save” and “Export” button. If you want to send your model to someone else you will find that there is no SaveAs button which means you’ve got no way to access the file itself. Or so I thought…. In true ESRI fashion they have totally overlooked this possibility and don’t tell you that, in actual fact, your model needs to reside in a toolbox you have created. The whole toolbox will be stored in a single file and will reside in your Windows profile under Application Data/ESRI/ArcToolBox/MyToolBoxes. All you need to do is copy the .TBX file related to your toolbox.

With your model complete it can be exported to Python (although note that Model Builder also supports JScript and VBScript) and then simply becomes a text file which can be edited in any text editor. If you look at the code generated it is pretty simple and the exported model actually comes pretty close to the final Python code you need. I used a FOR looping routine that James gave me which worked very well. As I noted above, I wanted to set an ArcToolBox environment variables on EACH loop. This is very easy to do in Python but it didn’t seem to work entirely properly. The solution that worked for me was to reset the variable at the beginning of each iteration and then set it again, later on, to the value I wanted.

With Python script complete it can be loaded back in to ArcToolBox as a “New Script”. If you go into “Edit” mode it will load the script into the Python IDE; hitting F5 will run the script and give you any console messages, although this crashed quite regularly when I run my script iteratively. It appears more stable when run directly from ArcGIS. Something that James noted was that ArcCatalog appears more stable than ArcMap for running scripts and that was my experience as well. Although note that the licensing of ESRI extensions in ArcCatalog is entirely separate from ArcMap. Which means you will need to go in to Tools->Extensions to enable them.

Pythonian Adventures 2

Monday, 4 February, 2008

OK, ESRI had VBA, no scripting language and some frustrated “power users”. Thankfully ESRI realised that this was a problem and from v9.0, introduced scripting via any COM standards compliant language, although Python was the preferred choice and shipped with ArcGIS out of the box. For those not familiar with Python (and I wasn’t), the Wikipedia article is pretty good and to quote from the main Python site:

“Python is a dynamic object-oriented programming language that can be used for many kinds of software development. It offers strong support for integration with other languages and tools, comes with extensive standard libraries, and can be learned in a few days. Many Python programmers report substantial productivity gains and feel the language encourages the development of higher quality, more maintainable code.

Its also GPL and runs on just about any/every OS. Python allows scripting of pretty much all ArcToolBox tools which allows access to most of ARCGIS’s underlying power without the complexity of VBA. This change to ArcGIS 9.0 formed part of a major revamp of the “geoprocessing tools” and included a new graphical programming environment called Model Builder. Like its much older partner in ERDAS Imagine (Spatial Modeller), Model Builder lets you link tools (graphically) together to form complex, conditional, spatial processing tasks. And best of all it involves no coding at all! Models can be made available to others and simplify the whole issue of “repetitive processing” essentially letting you save “tasks”. And Model Building also allows export to Python, VBScript and JScript.

So ESRI finally has a scripting language and some moderately happy power users. How does it all work in practise?? That’s for the next blog entry.