Spaced Out :: 08 2009

Wikileaks has posted a presentation purporting to originate from the OS outlining a change to their business model. Posted 20 August, the document itself is simply dated 2009 so it’s hard to know when it was drafted but given the content it seems likely that it is relatively recently (assuming it does come from the OS). It certainly makes for interesting reading as it outlines three possible modes of operation: full commercial, free data and a hybrid. Not surprisingly, given all the criticism, it suggests adopting a hybrid mode, dismissing the “utility” model out of hand. Whilst drafted in such a way as to mark a “step change” in the way they do business, it comes across more as applying the patches to the trading fund model.

What are the largest areas of criticism of the OS? Well public access to data, derived data and cost? The “hybrid” model therefore tries to tackle these problems. In particular there is a focus upon easier public sector licensing (one license for England and Wales), non-commercial reuse of derived data (something the OS has been hammered for)and increased reuse of data through OpenSpace. Interestingly a cost cutting programme forms part of the package.

After the news last week of the death of TopSat, it is good to see that NASA have opened up EO-1 for tasking. EO-1 was “launched on November 21, 2000 as part of a one-year technology validation/demonstration mission.” Its been very successful and lasted considerably longer than most thought. It carries the ALI multi-spectral (10m Pan and 30m MS) and Hyperion hyperspectral (220 30m bands) sensors. If you are in need of data then visit the Data Acquisition Request page and submit a request; this will be reviewed and, if deemed appropriate, tasked.

Twitter announces the addition of geolocation to tweets. Its currently being added to the API (with it’s implementation being made available to developers) and thereafter to the interface. To be honest, that’s all the announcement says and I imagine lat/long will come out of the 140 characters. No information on how location will be implemented although All Points notes that its likely to use GeoRSS.

I blogged a while back about the availability of TopSat for academic research and whilst the data I received was not great it did provide very good data for many users. I had noticed that they had ceased tasking the satellite formally, but managed to get two late requests in. The first was completed but the data was not good. Whilst waiting for these to be re-collected I received the following news on 18th August:

Unfortunately QinetiQ have come to the decision to end TopSat operations by the end of the week. We would have continued to schedule up until tomorrow had it not been for an unexpected hardware failure here in the office.

The “end TopSat operations” seems pretty final. Let’s hope that the TopSat experiment will be followed up with something equally interesting.

I’ve been involved with a project looking at the reflectance of loess and seeing how well this correlates with traditional measures, including magnetic susceptibility and grain size. The data sets rapidly grow so I’ve written several scripts in R to process them. To give you an idea of the problem, we used a GER1500 (400-1100nm) to collect 40 point samples in the field; each sample collects 700 data points (28,000). A further 12 field samples were collected and re-measured in the lab using an SVC HR1024 (400-2500nm) giving a further 105,000 measurements. The samples were then powdered and re-measured using an ASD Field Spec Pro (400-2500) giving another 105,000 measurements. That’s a total of 238,000 and that’s before you move on to looking at first derivative or continuum removed spectra.

Clearly a good data processing environment is needed and R fits the bill very well, although Matlab is used in equal measure by many (and Alasdair MacArthur over at NERC FSF is currently porting many of their pre-processing scripts). Matlab has the benefit of being known as a dedicated data processing facility with good graphing capabilities and a lot of bespoke, application specific, scripts. R is a statistical programming environment and is easily scriptable and good at the statistical analysis of massive data sets. It’s horses for courses, but R is open source which is good for me. And there is a portable version to boot (and for those using Excel, yes it does work, but as soon as you need to do anything iterative you are better off using something designed for the job).

In order to expedite the project I was working on I used the standard NERC FSF Excel template to do all the initial pre-processing. I then needed to produce some initial plots of the raw and first derivative spectra at each data point on multi-graph plots, before generating corelogram plots (correlation line graph). Most of this is relatively straight forward, just requiring importing and iterating over the data sets to produce nice looking graphs. I was particularly interested in using continuum removal as a technique for analysing the absorption features in the lab spectra and couldn’t (obviously) find any software that did it. So one of the scripts specifically processes the data in pre-defined ranges and calculates corelograms. I’m hoping to get a general purpose importation routine running for the HR1024 and FIeld Spec Pro sometime this year.

All scripts are available on my webpage and include a description of the files and sample data. They are not “general purpose” in so far as you need to edit them to load your own data. However they should ten work fine. I hope they prove useful and if you use them for any published academic work then please reference them as:

Smith, M.J. (2009) Reflectance Spectroscopy Scripts [Online]. Available from: http://www.kingston.ac.uk/gge/staff/smith.htm, [Last accessed: Access Date]