Data maintenance and recovery

Friday, April 25, 2014 at 08:52:20

Several things landed across my desk this week which reminded me of the the thorny problem of file formats, data media, archival and redundancy. Computers make our life easier - they can do repetitive tasks easily, store data and control external devices such as printers amongst other things. A nice example of this is my MSc thesis which I wrote in the then industry standard of Word Perfect 5.1. Scarily, I blogged on this nearly a decade ago noting the relatively short space of time that Word Perfect files largely became redundant…. yes Microsoft Office can read the files but not complex formatting. So I could recover the data and, using the process of exporting to PDF, I could recover a printable page. What I couldnt easily recover was a fully formatted, editable, document.

This has strong echoes of the doomed BBC Doomsday Project….. an ambitious 2.5M 1986 project (about 10M in todays money!) designed to “capture the essence of life in the United Kingdom. Over a million people contributed to this digital snapshot of the country.” Yet within 15 years it was redundant (as opposed to the 900+ years of the original Doomsday project!) before the CAMiLEON Project was resourced to recover the data.

And in the news this week a similar tale about some artwork produced by Andy Warhol for Commodore. This was a 3 year project to recover a dozen unknown digital works from the artist… yes, magnetic imaging was used so there was no physical contact (or damage) to the original floppy discs. They then discovered an “unknown” file format which had to be reverse engineered in order to read the content.

That brings me to the two recent remote sensing examples which are nice:

The first noted over at DPReview shows recovery of “lost” NASA lunar orbiter photos from the 1960s. These were amazingly photography on to 70mm film and chemically processed automatically in the orbiter before being transmitted back to Earth in analogue signals. The original recordings have now been found and reprocessed to produce, arguably, better imagery than current digital sensors!

The second dates back a few years to the Soviet Venera missions to Venus in the 1980s. These also show access to the original data (no problems with media this time!) but this time analysing multiple analogue transmissions of the same data in order to improve the fidelity of the imagery.

Stark warnings indeed of:

1. the media you store your data on
Be careful WHERE you store your data and how you back it up/archive it. Dispersing data between different cloud based services makes identifying WHAT you own and WHERE it is can be difficult. My recommendation would be to keep a local coopy and use cloud based services for file sharing (e.g. Dropbox) or backup (e.g. Carbonite). However remember that data can be quite ephemeral - I use a portable HDD, but this has a finite life so I not only back this up but where it wears out I will “format shift” to a new device. Offloading data on to DVDs or “old” HDDs can solve running out of storage but be careful that the media doesn’t become unreadable. Think of your data as “belongings” and the media as a bag. The bag is unimportant, it is there for storage and transport and you will move your “belongings” from one bag to another. Take care of your data!

2. the file formats themselves (something I blogged about in terms of spatial data).
There are some formats we can choose to use that will likely be readable in the future (Word DOC/DOCX, TIFF, JPG etc) but as applications become more specialist this becomes more difficult. We all get caught out with old formats, but if its valuable to you make sure you have a future proof route to ensure its longevity.

Mount Everest Foundation

Thursday, April 17, 2014 at 14:11:22

Late last year I became a trustee at the Mount Everest Foundation, an charity (and company) dedicated to “provide grants and recognition to assist particularly deserving expeditions for the exploration of, and scientific work in, the mountain regions of the world.” It was set up after the successful 1953 expedition to climb Everest and uses income from surplus funds and subsequent media royalties to support these activities. The website is of a relatively old design (its currently being updated) but has a wealth of information on it including application information and details of supported expeditions and brief trip reports.

All applications for funding are passed through a screening committee which assesses them against the Memorandum of Association of the charity. On the MEF website it also notes:

“The aims of the Foundation are to encourage and support expeditions exploring mountain regions, and both education and research pertaining to a wide range of subjects in mountain areas, including geography, glaciology and the effects of altitude.”

It goes on to note the way the charity is incorporated:

“Affairs are controlled by a Committee of Management, of which half is appointed by the Alpine Club and half by the Royal Geographical Society.Unless an expedition has research as its primary objective, it must have a strong exploratory element to be deemed eligible for support. Expeditions planning geographical exploration, first ascents of, or major new routes on high or remote mountains are likely to qualify. Applications from expeditions proposing to visit little explored or formerly inaccessible areas are particularly encouraged, as are those pursuing worthwhile research. Normally, the MEF will only support expeditions where the majority of memberscome from Great Britain or New Zealand.”

However its worth quoting the relevant objects themselves:

1. To encourage, or support, (whether financially or otherwise) expeditions for the exploration of the mountain regions of the Earth

2. To encourage, or support, (whether financially or otherwise) education in or research into the geography, topography, geology, ethnology, meteorology, botany, zoology (including entomology), and glaciology of the mountain regions of the Earth and allied subjects relating thereto.

3. To encourage, or support, (whether financially or otherwise) education in or research into the effect of altitude upon the human organism and the means of countering such effect in so far as it may be harmful

4. To encourage, or support, (whether financially or otherwise) the dissemination of any information acquired in pursuance of the foregoing objects or any of them and the provision of publicity for the results of education research and exploration as aforesaid

5. To relieve or contribute relief, of persons injured or suffering from sickness contracted in pursuance of such researchers or explorations or to encourage or support either alone or in combination with others arrangements or instructions for the purpose of affording such relief

“Mountain regions of Earth” are generally taken to mean outside Europe for climbing expeditions. However note the wide remit of the charity to include “geographic” research, education, altitude based physiology and relief of injured persons. Its a wide remit and as a charity we have a responsibility to target all aspects of the Memorandum of Association. Grants are typically of the order of hundreds to several thousands of pounds and need to show financial support from other areas.

MEF is a valuable and worthwhile charity with a 60 year heritage of supporting work in high mountains built upon the foundations of the early Everest expeditions. It’s a heritage that should long continue and I would urge those that undertake research or expeditions in high mountains to submit a grant application.

For those interested the charity (208206) files its accounts with the Charities Commission and is also a listed company at Companies’ House.

Proba V Data Products

Friday, April 4, 2014 at 09:12:08

This one had slipped past my radar (O dear….). Those with slightly longer memories will remember the introduction of VEGETATION on board SPOT 4 and 5, with PROBA designed to provide a continuation of this data. PROBA was a technology micro-satellite demonstrator (with the CHRIS hyperspectral imager), so fast forward a decade and PROBA-V is not only operational but its data (300m 4-band multispectral, although 100m at nadir) is freely available. Nice PDF factsheet and direct to ESA for image downloads. (thanks to Paolo for the tip!)

Heavenly Sentinels

Wednesday, April 2, 2014 at 14:27:24

A nice summary article by Jonathan Amos over at the BBC on the EUs Sentinel series of satellites. Provides a good upbeat message for this ambitious EO programme, the impact of the Landsat programme (and how the removal of the $600 fee increased usage) and the future potential for the Sentinels, their use and space activities for the EU.

Top of Page