Field spec processing scripts

Monday, 17 August, 2009

I’ve been involved with a project looking at the reflectance of loess and seeing how well this correlates with traditional measures, including magnetic susceptibility and grain size. The data sets rapidly grow so I’ve written several scripts in R to process them. To give you an idea of the problem, we used a GER1500 (400-1100nm) to collect 40 point samples in the field; each sample collects 700 data points (28,000). A further 12 field samples were collected and re-measured in the lab using an SVC HR1024 (400-2500nm) giving a further 105,000 measurements. The samples were then powdered and re-measured using an ASD Field Spec Pro (400-2500) giving another 105,000 measurements. That’s a total of 238,000 and that’s before you move on to looking at first derivative or continuum removed spectra.

Clearly a good data processing environment is needed and R fits the bill very well, although Matlab is used in equal measure by many (and Alasdair MacArthur over at NERC FSF is currently porting many of their pre-processing scripts). Matlab has the benefit of being known as a dedicated data processing facility with good graphing capabilities and a lot of bespoke, application specific, scripts. R is a statistical programming environment and is easily scriptable and good at the statistical analysis of massive data sets. It’s horses for courses, but R is open source which is good for me. And there is a portable version to boot (and for those using Excel, yes it does work, but as soon as you need to do anything iterative you are better off using something designed for the job).

In order to expedite the project I was working on I used the standard NERC FSF Excel template to do all the initial pre-processing. I then needed to produce some initial plots of the raw and first derivative spectra at each data point on multi-graph plots, before generating corelogram plots (correlation line graph). Most of this is relatively straight forward, just requiring importing and iterating over the data sets to produce nice looking graphs. I was particularly interested in using continuum removal as a technique for analysing the absorption features in the lab spectra and couldn’t (obviously) find any software that did it. So one of the scripts specifically processes the data in pre-defined ranges and calculates corelograms. I’m hoping to get a general purpose importation routine running for the HR1024 and FIeld Spec Pro sometime this year.

All scripts are available on my webpage and include a description of the files and sample data. They are not “general purpose” in so far as you need to edit them to load your own data. However they should ten work fine. I hope they prove useful and if you use them for any published academic work then please reference them as:

Smith, M.J. (2009) Reflectance Spectroscopy Scripts [Online]. Available from: http://www.kingston.ac.uk/gge/staff/smith.htm, [Last accessed: Access Date]

Add comment

Fill out the form below to add your own comments