EGU 2020 Short Course: UAV Data Collection and Analysis: operating procedures, applications and standards

Tuesday, October 1, 2019 at 12:33:41

UAV Data Collection and Analysis: operating procedures, applications and standards Conveners: Paolo Paron; Co-conveners: Mike James, Michael Smith

UAVs have reached a tipping point in geoscience research such that they are near-ubiquitous and commonly used in data collection. In this way they are opening new ways to study and understand landforms, sediments, processes and other landscape properties at spatial and temporal scales that is close to the scale of the processes that operate. However this implies that non experts are entering the field of photography, image interpretation, photogrammetry and 3D modelling often without a solid grounding in the principles of surveying. This course aims at providing a solid foundation for UAV users in order to avoid simple mistakes that can lead to legal restrictions, UAV loss, operational problems and poor quality data.

We will introduce pre-flight, in-flight, and post-flight procedures that aim at optimizing the collection of high quality imagery for subsequent downstream processing. We will also demonstrate the analyses of data by means of existing state of the art commercial software, such as Pix4D and Metashape for point cloud analysis, and eCognition for object based image analysis. We will also demonstrate the use of open source/open access software like Cloud Compare and Orfeo Toolbox

Converting from Blosxom to Flatpress

Sunday, September 29, 2019 at 13:50:46

This blog has been offline for a little while as the original Blosxom implementation had been hacked. Blosxom was a wonderful CGI script that was elegant in its simplicity yet eminently extensible through the many plugins which existed and made it moderately feature rich. Best of all, it used plain text files to store all its entries which makes backup and conversion much simpler than a database. With my implementation of blosxom decommissioned, I needed to find a replacement. Google flat file blogging engines and there are a lot. However many of the projects have been orphaned, like blosxom, and no longer in active development. What I wanted to find was an engine that was simple, had some good features and an active community. Flatpress seems to fit the bill with a new maintainer - and active Flatpresser - Arvid Zimmerman.

The next step was to convert my archive of over 1000 blosxom blog entries to Flatpress. Big shout out to James O’Connor who wrote the Python script to convert the files. The process is broadly this:

  • download your Blosxom files, including all the sub-directories for categories, but make sure to maintain the date/time filestamp of individual files - this is used to timestamp the entry for Flatpress. WinSCP does this (Filezilla doesnt)
  • make sure the categories only ONE DIRECTORY DEEP. Move any sub-sub-directories up to the top level
  • rename all the directories to numbers. These are used to tag the entries and can then be recreated within FlatPress
  • copy the script.py and template files to the directory the folders are stored in
  • edit the template file to have the header/footer you want. The content, date and categories will be changed for the entries
  • run the script
  • a new fp-content directory will be created with all your entries
  • upload this to your flatpress site and rebuild the index

The script does the following

  • renames the file to entry‹date›-‹time›.txt based upon the date modified date
  • copies the file to a new subfolder in FlatPress /content folder based upon year and month
  • deletes the first line from the file (and deletes the first line break)
  • prefixes the file with: VERSION|fp-1.1|SUBJECT||CONTENT|
  • suffixes with: |AUTHOR|miksmith|DATE|<1566926569>|CATEGORIES||

Any updates will be posted over at Flatpress.

FREE EPRINT: Editorial: Perspectives on the contemporary art-geoscience interface, Journal of Maps

Saturday, September 28, 2019 at 22:46:04

Tooth, S., Smith, M.J., Viles, H.A. and Parrott, F.
Journal of Maps


This Special Issue of the Journal of Maps is devoted to highlighting contemporary examples of interdisciplinary collaborations between the arts and the geosciences (e.g. geomorphology, geology, Quaternary studies), with a specific focus upon the exploration of locations using, at least in part, some form of mapping. As previous contributions to the journal have exemplified, mapping is essential for the exploration of locations, particularly by supplying visual representation to help with the characterisation of three core geographical concepts (Matthews & Herbert, 2008): space (e.g. distances, directions), place (e.g. boundaries, territories), and environment (e.g. biophysical characteristics).

FREE EPRINT: Testing and application of a model for snow redistribution (Snow_Blow) in the Ellsworth Mountains, Antarctica, Journal of Glaciology

Saturday, September 28, 2019 at 22:42:25

Mills, S.C., Le Brocq, A.M., Winter, K., Smith, M.J., Hillier, J., Ardakova, E., Boston, C., Sugden, D. and Woodward, J.
Journal of Glaciology


Wind-driven snow redistribution can increase the spatial heterogeneity of snow accumulation on ice caps and ice sheets, and may prove crucial for the initiation and survival of glaciers in areas of marginal glaciation. We present a snowdrift model (Snow_Blow), which extends and improves the model of Purves et al. (1999). The model calculates spatial variations in relative snow accumulation that result from variations in topography, using a digital elevation model (DEM) and wind direction as inputs. Improvements include snow redistribution using a flux routing algorithm, DEM resolution independence and the addition of a slope curvature component. This paper tests Snow_Blow in Antarctica (a modern environment) and reveals its potential for application in palaeo-environmental settings, where input meteorological data are unavailable and difficult to estimate. Specifically, Snow_Blow is applied to the Ellsworth Mountains in West Antarctica where ablation is considered to be predominantly related to wind erosion processes. We find that Snow_Blow is able to replicate well the existing distribution of accumulating snow and snow erosion as recorded in and around Blue Ice Areas. Lastly, a variety of model parameters are tested, including depositional distance and erosion vs wind speed, to provide the most likely input parameters for palaeo-environmental reconstructions.

FREE EPRINT: Quantification of Hydrocarbon Abundance in Soils using Deep Learning with Dropout and Hyperspectral Data, Remote Sensing

Saturday, September 28, 2019 at 22:37:41

Asmau Ahmed, Olga Duran, Yahya Zweiri, Mike Smith
Remote Sensing


Terrestrial hydrocarbon spills have the potential to cause significant soil degradation across large areas. Identification and remedial measures taken at an early stage are therefore important. Reflectance spectroscopy is a rapid remote sensing method that has proven capable of characterizing hydrocarbon-contaminated soils. In this paper, we develop a deep learning approach to estimate the amount of Hydrocarbon (HC) mixed with different soil samples using a three-term backpropagation algorithm with dropout. The dropout was used to avoid overfitting and reduce computational complexity. A Hyspex SWIR 384 m camera measured the reflectance of the samples obtained by mixing and homogenizing four different soil types with four different HC substances, respectively. The datasets were fed into the proposed deep learning neural network to quantify the amount of HCs in each dataset. Individual validation of all the dataset shows excellent prediction estimation of the HC content with an average mean square error of ~2.2×10-4. The results with remote sensed data captured by an airborne system validate the approach. This demonstrates that a deep learning approach coupled with hyperspectral imaging techniques can be used for rapid identification and estimation of HCs in soils, which could be useful in estimating the quantity of HC spills at an early stage.

FREE EPRINT: Assessment of low altitude UAS flight strategy on DEM accuracy, Earth Science Informatics

Saturday, September 28, 2019 at 22:34:25

Anders, N.S., Smith, M.J., Suomalainen, J., Cammeraat, L.H., and Keesstra, S.D.
Earth Science Informatics


Soil erosion, rapid geomorphological change and vegetation degrada- tion are major threats to the human and natural environment. Unmanned Aerial Systems (UAS) can be used as tools to provide detailed and accurate estimations of landscape change. The effect of flight strategy on the accuracy of UAS image data products, typically a digital surface model (DSM) and orthophoto, is unknown. Herein different flying altitudes (126-235 m) and area coverage orientations (N-S and SW-NE) are assessed in a semi-arid and medium-relief area where terraced and abandoned agricultural fields are heavily damaged by piping and gully erosion. The assessment was with respect to cell size, vertical and horizontal accuracy, absolute difference of DSM, and registration of recognizable landscape features. The results show increasing cell size (5-9 cm) with increasing altitude, and differences between elevation values (10-20 cm) for different flight directions. Vertical accuracy ranged 4-7 cm but showed no clear relationship with flight strategy, whilst horizontal error was stable (2-4 cm) for the different orthophotos. In all data sets, geomorphological features such as piping channels, rills and gullies and vegetation patches could be labeled by a technician. Finally, the datasets have been released in a public repository.

FREE EPRINT: ‘Reading landscape’: interdisciplinary approaches to understanding, Journal of Maps

Wednesday, March 20, 2019 at 08:39:56

Mike J. Smith, Flora Parrott, Anna Monkman, James O’Connor and L. Rousham
Journal of Maps


This paper outlines a collaborative project between a group of Fine Art and Geography students who helped develop and contribute to a conversation about recording ‘place’. Introducing methodologies from both disciplines, the project started from the premise of all environmental ‘recordings’ being ‘inputs’ and so questioned what could be defined as ‘data’ when encountering a location. Brunel’s Grand Entrance to the Thames Tunnel (London) provided the motivation for 10 objective and subjective ‘recordings’ which were subsequently distilled into a smaller subset and then used to produce a short film that was presented at an international conference. Important to the collaborative nature of the project were ongoing opportunities to share equipment, techniques, material and references across disciplines. It was an experiment to measure the potential for ‘mapping’ to capture physical and historical information, as well as embodied experience.

FREE EPRINT: Land inundation and cropping intensity influences on organic carbon in the agricultural soils of Bangladesh, Catena

Thursday, March 7, 2019 at 11:13:34

M.J. Uddin, Peter S. Hooda, A.S.M. Mohiuddin, Mike J. Smith and Martyn Waller
Catena


Land inundation is a common occurrence in Bangladesh, mainly due to the presence of two major river systems -the Brahmaputra and the Ganges. Inundation influences land use and cropping intensity. However, there is little information on the influences of the extent of flooding and cropping intensity has on soil organic carbon (SOC),particularly at the landscape level. To investigate these influences, we collected 268 surface (0-30 cm) soil samples from 4 large sites within the two alluviums deposits (the Brahmaputra river and the Ganges river), on a regular grid (1600 m). The findings show that SOC levels are generally low, reflecting the intensity of agriculture and land management practices. SOC variability was higher across the medium high land (MHL) and medium low land (MLL) sites than in the high land (HL) and low land (LL) sites. The relatively low SOC levels and variability in the HL sites indicate soils here might have reached to equilibrium levels due to higher land use intensity. Topographically higher lands (HL and MHL), due to less of inundation, had higher cropping intensities and lower SOC’s than lower lands (MLL and LL), which had lower cropping intensities, as they remain inundated for longer periods of time. The findings clearly demonstrate the intrinsic influence of land inundation in driving cropping intensity, land management practices and SOC levels.

FREE EPRINT: Summary of activities 2018, Journal of Maps

Friday, February 22, 2019 at 14:32:00

Mike J. Smith (2019)
Journal of Maps


Creativity is one of those tropes that seems to do the rounds regularly in, well, creative circles. Almost by definition, it is levelled at the arts, in part because its base definition is along the lines of the ability to create. Withinthis context, cartography is well-poised because any map requires the cartographer to create a new, unrealised, graphic product.

OPEN ACCESS EPRINT: Demystifying academics to enhance university-business collaborations in environmental science

Tuesday, January 15, 2019 at 11:08:02

John K. Hillier, Geoffrey R. Saville, Mike J. Smith, Alister J. Scott, Emma K. Raven, Jonathon Gascoigne, Louise J. Slater, Nevil Quinn, Andreas Tsanakas, Claire Souch, Gregor C. Leckebusch, Neil Macdonald, Alice M. Milner, Jennifer Loxton13, Rebecca Wilebore, Alexandra Collins, Colin MacKechnie, Jaqui Tweddle, Sarah Moller, MacKenzie Dove, Harry Langford, and Jim Craig (2019)
Geoscience Communication


challenge posed by a heavily time-constrained culture; specifically, tension exists between opportunities presented by working with business and non-optional duties (e.g. administration and teaching). Thus, to justify the time to work with business, such work must inspire curiosity and facilitate future novel science in order to mitigate its conflict with the overriding imperative for academics to publish. It must also provide evidence of real-world changes (i.e. impact), and ideally other reportable outcomes (e.g. official status as a business’ advisor), to feed back into the scientist’s performance appraisals. Indicatively, amid 20-50 key duties, typical full-time scientists may be able to free up to 0.5 day per week for work with business. Thus specific, pragmatic actions, including short-term and time-efficient steps, are proposed in a “user guide” to help initiate and nurture a long-term collaboration between an early- to mid-career environmental scientist and a practitioner in the insurance sector. These actions are mapped back to a tailored typology of impact and a newly created representative set of appraisal criteria to explain how they may be effective, mutually beneficial and overcome barriers. Throughout, the focus is on environmental science, with illustrative detail provided through the example of natural hazard risk modelling in the insurance sector. However, a new conceptual model of academics’ behaviour is developed, fusing perspectives from literature on academics’ motivations and performance assessment, which we propose is internationally applicable and transferable between sectors. Sector-specific details (e.g. list of relevant impacts and user guide) may serve as templates for how people may act differently to work more effectively together.

Ricoh GRIII

Tuesday, September 25, 2018 at 11:04:54

I’ve waxed lyrical about the Ricoh GR being a great UAV camera before - well the long awaited successor has been announced (but hasn’t yet landed) and summarised over at DPReview. The interesting aspects of the uprated specs are IIS (inbody image stabilisation), 24MP sensor and touchscreen. The resolution boost and IIS will be of significant interest to UAV users so it will be interesting to see how it performs out in the field.

TED: Academic research is publicly funded - why isn’t it publicly available?

Thursday, March 29, 2018 at 08:52:24

An interesting talk from TEDxMileHighWomen. Worth a watch to get a short 10-min summary of some of the issues involved with publishing academic research - the comments are worth a look too.

As Erica Stone implies, she hasn’t got much experience in academic publishing and it unfortunately shows. There are some points well made, but there is and underlying naivety about the role of publishing, the cost, the requirements of universities and the amount of time academics have. As I noted in my editorial this year:

Academic publishing is a knowledge distribution and academic assessment system, partially funded by universities and research institutes.

To publish you have cross-subsidise, or go down an author or reader pays route - ironically (and perhaps to the chagrin of the OA camp), OA is currently costing the system more than a subscription model on an annual basis and probably on a pageview basis too. But, let’s keep the debate going!

Open Access UK Unit Postcodes

Thursday, March 29, 2018 at 08:32:12

The Ordnance Survey released their Code-Point Open product a few years ago that has the OSGB centroids of the unit postcodes. It’s very useful but is only points - if you want the postcodes areas as polygons then you need to license Code-Point with Polygons (snappily titled!). A number of people have derived unit postcode areas using Voronoi polygons including more recently Mike Spencer with some intro at his Scottish Snow site. It’s worth noting that Voronoi diagrams equally partition space between points and nothing more - they are not equivalent to unit postcodes (which can be arbitrary) but are a reasonable first guess. One dataset worth having for your arsenal of spatial data!

Waldo Tobler

Tuesday, March 6, 2018 at 12:57:46

I was sad to see that Waldo Tobler passed away last month - a lifetime geoscientist he contributed huge amounts to computational cartography but will be best remember for the First Law of Geography. Some more details at GeoLounge and in the original paper.

However I love the CSISS Classic which was a tongue-in-cheek experiment with Peter Gould on geocoding. Read it because its wonderfully powerful way of showing students about geocoding. Hopefully we’ll see an anthology of his work in the not too distant future.

QGIS 3 features

Monday, March 5, 2018 at 14:21:56

QGIS 3 is well and truly out now - download your copy here. And the good folks over at GIS Geography have put together a list of QGIS features that are in the new version. Some of the highlights include 3D (1), coordinate reference bounds (5), geopackage (7), background processing (8), new print composer (13), refined graphical modeler (25), but they are all worth taking a look at as it might just be a solution to the problem you have!

Pursuit by Mike Oblinkski

Monday, February 26, 2018 at 12:15:10

The only words to describe this are awe inspiring, spectacular, mesmerising - this is a visual tour de force that is matched by wonderful evocative music. Just the basic stats behind it say it all… completed over three months, 27 days of filming, traveling across 10 states, with 28,000 miles of driving and over 90,000 time-lapse frames. Produced to 4k this movie is simply begging to be viewed on a MASSIVE screen. Now if only my local cinema would show it as a short.

Utterly brilliant. Watch it.

Pursuit (4K) from Mike Olbinski on Vimeo.

You’re Waiting For a bus… and a SenseFly Comes Along

Saturday, January 27, 2018 at 17:17:04

I commented late last year about DJI becoming a camera manufacturer… this is an interesting and exciting move simply because the quality of imagery from drones has lagged significantly behind the rest of the camera industry. So whilst there are clearly regulatory challenges that need to be overcome through hardware and software engineering, the end-user is interested in the best visual imagery possible and drone manufacturers are starting to wake up to this fact.

So I was surprised to see that SeneseFly had announced back in 2016 its (then new) eBee Plus offered an hour of flight time within a ~1kg airframe that can incorporate RTK/PPK positioning (forget those ground control points!) and multi-spectral/thermal. But… they have also introduced their own homegrown air camera dubbed SODA (Sensor Optimised for Drone Applications). It is extremely light on specification however their website notes its a 20MP 1” fixed lens camera. This that likely utilises Sony’s standard 1” sensor, although it would be useful to clarify the exact size. Is it genuinely 13.2x8.8mm? Taking their GSD example figures, that breaks down to a 10mm focal length lens, equivalent 27mm on a full-frame camera. Interesting there is a single global electronic shutter which is a good thing for a fixed-wing aircraft: it should stop the problem of a rolling-shutter.

Fascinating to see the drone-camera market develop and it’d be great to see some results from this baby.

Earth-i Launches Vivid-i

Saturday, January 13, 2018 at 20:08:20

Earth-i has just launched a new SSTL built satellite which is claimed to be the first to provide full colour UHD video - UHD is 3840x2160 pixels (8MP), shooting at 50 fps (compared to Landsat 9’s ~12,000x12,000). That’s a lot of downstream data, although it would appear it’s not the video they’re interested in, but the multi-temporal data. Think super resolution to give them an effective ~70cm pixel size, but also stereo (and so 3D). This is the first of a planned 15 satellite constellation which could provide global coverage and much more agile mapping capabilities. Video is clearly the new high resolution!

Video file sort order on my Pico projector

Sunday, December 24, 2017 at 22:02:04

A (long) time back I wrote about the Acer C112 Pico Projector I was using. I’ve since upgraded to the Pico Genie P200 which gives a great 200 lumens of light and has microSD, USB and HDMI ports to allow all sorts of connectivity along with a built in battery. It’s not perfect - but its very good (and I still follow the same routine for transcoding with DVDDecryptor and TEncoder).

One minor irritant was that the sort order of the files on the memory stick appeared to be shown by date on the projector - I used Nico’s useful BulkFileChanger and… it made no difference. Obviously the embedded Linux system wasn’t using file dates. In fact on my (FAT32) USB stick it’s using the order of the files in the FAT - what was successful was using DriveSort. Problem solved - I suspect this affects a small number of embedded systems (mp3 players, projectors etc) that have had some lazy coding and don’t sort the files on portable drives.

Top of Page