It’s a , stupid!!

Wednesday, 2 November, 2016

Yes, one of the manuscript writing moments where I was using Endnote and wanted to cite a webpage for an organisation. Enter the oranisation name in and the European Geosciences Union gets turned into…

Union

This is one of those annoying diversions where you either go and work and the syntax for citing it or… do it manually.

In this instance I Googled it and found that all that was needed was the humble

,

at the end of the author field. It’s always easy when you know how!

Data transfer and memory cards

Monday, 17 October, 2016

James and I were in Norfolk a few weekends back completing data collection for his PhD studies (his blog has at least one post relating to this) and the whole topic of data transfer speeds came back to haunt me. Amongst a number of cameras, we had been shooting with the Nikon D700 using a Sandisk 16Gb Ultra card which has read speeds of 30MB/s. We actually filled the card on day 1 (1000 shots) and need to unload the data off it. I had brought with me my cheap and cheerful Integral CF->USB card reader which works fine. Except it took the best part of 30 minutes to copy the data off the card around <5MB/s. Painfully slow.

When we got back I thought I’d dig back into data transfer speeds again. Remember that the firmware (and hardware) in a digital camera will be able to use cards up to a certain specification. The D700 is at least 64Gb cards up to 90MBs (although it might not be able to utilise the full speed of the card). My Fuji XM1 can take 32Gb cards at 50MBs. Now to achieve these speeds during data transfer to a PC, all parts of the chain need to be quick - card, card reader, USB port, bus and hard disk. In this instance the card reader was the limiting factor as it was plugged into a USB3 port on a new laptop. And just as a reminder, USB3 has a throughout of (depending upon what you read), somewhere above 400MBs (and doubling for USB3.1), whilst USB2 somewhere around 35MBs.

So, one lowcost USB3 card reader later and a new (lowcost!) 50MBs SD card, plugged directly into the USB3 port and read speeds race along. Getting this kind of throughout is both cheap and easy, but its not hard to accidentally put a weak link in that chain and see those rates plummet.

Windows 10 Anniversary Update

Tuesday, 27 September, 2016

Well the Windows 10 Anniversary Update has landed and, after the big download, it comes with quite an array of tweaks and new features. To get the skinny on some of these head to your favourite IT site for their run down… for example cnet or How-to-Geek.

Perhaps the most interesting for techies out there is the arrival of a Linux Ubuntu subsytem. How-to-Geek has a great rundown for installation (and to note its not really Linux, as its the bash shell, so really GNU apps). Anyway, think of this as the reverse of WINE. Opens up a world of command line scripting.

And a final note on usability and interface. Yes, really yes, the start menu has changed AGAIN. Supposedly simpler, cleaner, nicer, fresher - pick your superlative. Except… usable?? My Mum went through the update, it all installed perfectly, no errors and then… she couldn’t work out how to shut down the machine. Sorry Microsoft, FAIL on that count.

Of VM woes…

Tuesday, 27 September, 2016

Actually not so much VM woes as out-of-date underlying OS. Yes, the new term is hitting so I thought it appropriate to upgrade ArcGIS to the same version used on campus - that meant going from 10.2.1 to 10.4. Simples I thought - just request the student/instructor license, download the EXE and away we go.

Of course in reality things are never quite so simple. I won’t allow ArcGIS near my native OS because it is such a behemoth of an application - so I run it in a VM. Its slower, but locked down and if it dies, well I can just spin it up in the VM again. I use the excellent (and cross-platform) Portable VirtualBox) and, because its portable, I can copy it onto my portable hard drive and take it with (just make sure your drive is exFAT formatted for those large VDI files). Anyway, after the EXE download and install fails telling me to upgrade to Windows 7 (yes, the VM is on 7, I have an 8.1 and the main system on 10) to SP1. One upgrade later and…. upgrade .NET to 4.5+. And then… my virtual disk runs out of space. Its sized to 25Gb, but with ArcGIS, Imagine and Cyclone on there its run out of space. A little head scratching (and Googling later) led me to this page outlining the VirtualBox command to increase the VDI file (why you can’t do this in the GUI I don’t know!):

VBoxManage modifyhd windows7.vdi —resize 30000

Finally, inside Windows, expand the partition to take up all the virtual drive space.

Bingo, ArcGIS finally installs.

Whilst these are quite frustrating hoops to jump through, it does at least make you aware of some of the hurdles that students face.

Got an old ISO disc image….

Friday, 23 September, 2016

…. as I have. I was interested in accessing the contents of an old ISO disc image from an age old data CD. There are a number of tools to look inside these, but at times you just want to emulate a CD/DVD drive and run it directly. There are a range of tools to do this but I wondered if there was an easy option… and there is! Under Windows 10 you can now directly “Mount” the ISO image and it will appear as a new CD-ROM on your PC. Just right-click and “Mount.” For older systems Microsoft do a device driver bundled as the Microsoft Virtual CDRom Control Panel (which I first came across here).

Voila! Problem solved.

word word count

Wednesday, 17 August, 2016

Yup, that’s right… how do you do a word count in Word? Well, the simple answer is Alt-T, W and particularly for the ribbon-challenged like me who prefer the keyboard. But what happens if you are working to a specific word count, have ancillary information in the file but only want the content word count? Well, that’s a little more tricky and requires some lateral thinking (or some DuckDuckGoing).

And the (or one) solution is to put your ancillary information in a different style, change the font attributes to include “hidden” for that style, but then in the Word options make sure “print hidden text is selected”. That way you can see it and print it - crucially though Word excludes hidden text from word counts.

Read in full here

Kickstarter Plug: Spooly

Saturday, 14 May, 2016

Wanted to give a plug to Spooly who are running a Kickstarter campaign at the moment. I’ve not link to them, but the product looks great - I’m always on the lookup for innovative products that solve problems or make life easier. And this is a little gem - its a USB charge cable but incredibly small and rolls up into a perfectly neat package. What can you say other than a fantastic piece of design, doesn’t cost much and worthy of the plug.

Column Numbers in Excel?

Thursday, 12 May, 2016

I’ve been doing some more programming in R recently and working on a large XLSX spreadsheet (using the excellent readxl package) where I needed to cross-reference columns. Excel painfully uses letters for column titles and I needed the column number; useful to know then that you can change this option in Excel:

Pre-ribbon
Go to Tools, Options, General, check ‘R1C1 reference style’

Ribbon
Go Options, Formulas, check ‘R1C1 reference style’

GodMode

Monday, 7 March, 2016

Seriously! In Windows 10, create a new folder and give it the following name:

GodMode.{ED7BA470-8E54-465E-825C-99712043E01C}

And you have a control panel link to ALL control panel settings.

I am God!!

Distraction free writing

Monday, 18 January, 2016

I wrote several years ago about distraction free writing and the portable Windows application Focus Writer, although over the last few years its Scrivener that has taken quite a few headlines.

Well over the last few months I’ve been doing quite a lot of writing and so went back to the idea of distraction free writing. One thing I’ve found over the years is that it’s not only the software you use, but the frame of mind you’re in. And frame of mind is closely linked to location. In short I like to write in a variety of locations, so having something portable to write with is important. Yes a smartphone could do it, but the screen is limited (although I do like the idea Microsoft has tried to bring together with Continuum by plugging your device in to a monitor - or just using a pico projector). My ageing Nexus 7 (7”!) tablet strikes the balance between size and portability which then leads to a keyboard. For a while I flirted with the Apple keyboard which is very nice, but the lack of cover was a problem. My current “goto” device is the Microsoft Wedge which is bluetooth, beautiful to type on and comes with a protective cover which doubles as a stand.

So, in looking for a distraction free editor on Android I came across JotterPad which is beautifully designed and does exactly what I want. A focus upon writing, research mode for searching dictionary/thesaurus and keyboard shortcuts for all the main types of formatting. BUT…

As it turns out, JotterPad uses Markdown. This is an extremely lightweight markup language designed for writing. Given it’s low key evolution, there has never been a “standard” and so there are various flavours (e.g. Github). CommonMark is an attempt to bring this together and markDownEdit is one of several Windows editors (portable!) that supports Markdown. Thankfully there is a converter (Pandoc) which is widely used (by both MDE and JotterPad) which translates in to a range of other formats (include DOCX, Latex and HTML). The latter is particularly important if you are then working to create a Kindle ebook.

However one of the key things I do is edit - that means being able to add comments and indicate edits. Currently markdown does not support this but CriticMarkup is a set of 5 tags that do. I’ve request that MDE add support for these which will be the final feature that will allow utilisation in academia.

Give markdown a go - it’ll really set your writing free.

Universal Naked Driver

Monday, 28 December, 2015

The first stage in replacing your stock Android phone, with a new ROM (for example CyanogenMod) is to replace the recovery partition on your device - for example here are the full instructions for my aging Moto e. When you attach your phone normally to a computer you need USB drivers to recognise it - and for the purposes of installing a new ROM you want to begin by using ADB to reboot your device to the bootloader. This is a pre-boot mode and gives you a number of limited menu options for your device. At this point you can change the recovery partition on your device using fastboot which can then be used to change the ROM (and so version of Android).

I was looking to put an unofficial port of CyanogenMod 11 (aka Android 4.4) doing this for my son’s ZTE Kis3 (cheap and chearful but extremely serviceable ZTE phone), however whenever I rebooted it to the bootloader it wasn’t recognised by my Windows PC. After a fair amount of searching I discovered two things:

1. the ZTE drivers clearly don’t support USB when rebooted to the bootloader
2. the drivers for fastboot should be generic

The latter is really helpful for this and potentially in the future - and led me to the Universal Naked Drivers which do what they say on the tin. However one big caveat is that they are unsigned and Microsoft (understandably) has made it very difficult to install unsigned drivers. So…. watch this YouTube demo that explains how to install them.

Worked for the ZTE Kis3 and the subsequent install of CM11.

FAT32 Redux

Wednesday, 23 December, 2015

My earlier blog on exFAT reminded me that, as a general principle, Android devices can’t access that filesystem (although there are some devices that have OEM support for it and some third party add-ons). That’s why microSD cards are formatted to FAT32 - Android, Windows and Mac can all access them. Of course, we then hit the 32Gb limit which is of course why all devices are listed as having this as the maximum external storage. But this is a FAT32 limit, not one imposed by the hardware. The simple solution is to use fat32format which gives a much higher limit. In fact, to quote from the page:

“Note that the 32GB limit is a limit of the formatter in Windows XP. FAT32 itselft should be OK to 2TB.”

hard to believe that the format command is the root of so much misinformation. Anyway, buy yourself a cheap microSD (£12!) and voila you have 64Gb of external storage. Something that Apple charges a premium for if you have it is internal storage.

Getting FATter

Tuesday, 15 December, 2015

Three years ago I blogged about using FAT32 beyond the 32Gb limit to format large external drives. The rationale for this was to have an disk file system that worked across operating systems and didn’t run in to the complete drag that is NTFS file permissions.

Well step forward three years and my 500Gb drive is starting to be (very) full and needed upping to 1Tb. That and my increasing need for security meant going in search of something more secure for my external drive - the iStorage diskAshur fitted the bill perfectly with a keypad locked drive, real-time Hardware encryption (256-bit AES) and some shock protection. Great bit of kit and only a little pricier. But…. what file system to have??

Well the reason for moving on from FAT32 was the limit to maximum filesize (4Gb) which is easily exceeded with satellite imagery or virtual machines. NTFS, as I note above, is just painful for file permissions and isn’t really needed for an external drive which has hardware encryption. Well, as it turns out the answer was staring me in the face - exFAT which is essentially FAT64. OK, so its a proprietary Microsoft file system, but solves the filesize limit, is relatively simple and fast, ideally suited to external media and particularly flash drives, but perfectly suited to this use. Compatible with add-ons to Windows XP and MacOS, whilst there appears to be driver support for Linux. Given 99.9% of the time I move between Windows systems that suits me.

So, 500Gb of data transfer later I now have an operational drive. Just got to move the other backup disk from 500Gb to 1Tb…

DOS For loop for iterating

Sunday, 13 December, 2015

I’m currently working on a pet project doing some timelapse (if you hadn’t guessed!) involved geocoding the photos (and a much longer blog on this process at a later date once it’s complete). However as part of this I needed to iterate over a series of files using the excellent nconvert. DOS batch files let you do this but the syntax (for me at least!) is a little confusing. So, in all it’s glory here’s the solution I rustled up after looking at various websites:

for /f %%a IN (‘dir /b .jpg’) do ..\nconvert -out jpeg -wmflag bottom-right -wmfile overlay\%%~na.png -o final\%%~na.jpg %%~na.jpg

As ever, much useful information can be gleaned by typing

for /?

The /f parses a list of files (denotes by the variable %%a) and this is fed to it by the ‘dir /b *.jpg’ command. In this case a “bare” directory listing of all JPG files. Once that is complete it runs the nconvert command inserting the variable at the %%a locations.

Crucially the statement is then finished with \%%~na.jpg which strips the file
name* from the 3-letter filetype (suffix). This is needed in the nconvert command.

One line of code, 1000 files processed very rapidly! Love it when this stuff works!!

DOS file listing

Wednesday, 9 December, 2015

Need to get a plain and simple file listing from the command line in DOS to pipe in to a programme??

As ever a good starting point is to get help for the command. So:

dir /?

If you peruse through this you’ll eventually work out that this is the command you want:

dir /a-d /b > ..\list.txt

The “>” symbol sends the output and stores it in a file. Make sure to specify “.." before the filename so the file is saved in the directory above the existing one otherwise that file will be listed as well. And if you want to sort on (for example) filename then you add the “\on” switch:

dir /a-d /b /on > ....\list.txt

This will now save me looking that command up everytime!

Visualisation Update

Monday, 7 December, 2015

Following on from my neogeography/big data blog Edoardo L’Astorina at Blu Frame (web designers) contact me to note a review he’s recently completed on 20 online visualisation tools. And it’s a useful review and worth a look to see where things are at.

Timelapse: timeshift your timestamp!!

Saturday, 5 December, 2015

AS I noted in my previous blog entry, if you forget to set your time/date on your camera and its wrong, you need to reset it in order to get XnView to pull our this metadata and overprint it on your JPGs. If you’ve got more than 1 or 2 to do (I had about 4000!) then you need an alternative…

And here the amazingly complete ExifTool does a brilliant job. For all things EXIF look no further. It’s power is fully opened up in the command line version (rather than the drag and drop). And yes- it supports a timeshift facility.

For the images I had, the camera had been dead for several months so when a battery was put back in it reset the time/date to a default value. All subsequent images were therefore out by the same offset. I ignored the days as I wanted a time overstamp. Running the command:

exiftool -AllDates+=12:18:28 .

worked perfectly. This instructs exiftool to shift forward all time fields by 12 hrs 19 min 28s. More details here. Job sorted.

Timelapse Update

Friday, 4 December, 2015

A little while back I blogged about producing a timelapse. That sorted out all of the main issues I had with setting a camera up and leaving it to record a timelapse of a scene.

However I was recently reminded that my daughter had taken a series of daily photos back in in 2010 over an 8 week period whilst a new canteen was built at her school. It was a camera project and something we could use to look back on. All the photos were hand held (and under a range of different times of day and lighting conditions) which created a challenge to producing a timelapse. Most particularly frames were not aligned so I filed the photos under “must do someday”.

Well that day arrived last week!! I realised that the excellent PTGUI which we used on the Studio of Objects project to produce the 360 panoramas has a module that performs image alignment. Now obviously I didn’t want to stitch those photos together (because they would all sit on top of each other!!), but I did want to align them. PTGUI puts the images in to layers and looks for high contrast points between layers to match them. This it did (bar one which I handled manually) and you can then export each layer separately.

Voila a set of aligned images! What this showed was that some of the images were only partially overlapping. Back to the excellent XnView where I specified the crop I wanted and batch processed the images. I now had an aligned timelapse.

I finally did a little bit of experimentation…

1. Colour Toning: I tried “auto-levels” and “auto-contrast” in XnView to try to help colour balance/tone the images. This kinda worked a little bit, but colour toning is complex and doing it across such a diverse range of images very difficult.

2. Overlay: I also wanted to overlay the date and an logo on to the timelapse. XnView has a fabulous overlay facility (“add text”!) which allows you to pull in information from the image itself. The EXIF data in the JPEG had the date and time of exposure, so I changed the font and then added the date to the lower left corner of the frame. After batch processing the images I then realised about 8 images had the wrong date - a friend took photos whilst we were away on holiday and had replaced the batteries in the camera but not reset the date. I thought changing the date in the EXIF header would be easy - but XnView doesn’t do it. That led me to the portable Geosetter which allows you to change a range of image headers including the date. The final step was to add the school logo (“watermark”) and we were done.

Not going to set Hollywood on fire, but satisfying!

VirtualBox Update

Friday, 27 November, 2015

Just a reminder that VirtualBox remains a great option for virtualisation and running a range of operating systems. In addition to being able to spin up various Linux OSes on my Windows box, it also allows me to run a Windows VM for testing purposes (and for those in HE Microsoft’s DreamSpark provides a no cost way to use a range of software, including the OSes). I run all my ERDAS Imagine and Esri ArcGIS work through here. I had been on the venerable v4 of VirtualBox but have since upgraded to 5 and speed boost is very noticeable.

And, of course, those who know me know my penchant for portable apps. And yes there is Portable VirtualBox. Run it, get it to download the install files and then youre away. A base install of Win7 is about 5Gb, but with Imagine and ArcGIS runs at about 25Gb. But it’s nicely sandboxed and anything that goes wrong can simply be vaped.

Turning Point Clickers Orphaned by Windows 10 @turningtech

Wednesday, 18 November, 2015

I was just doing some testing of a remote polling session for a class this week so took my Turning Point RF Receiver and plugged it in to a Windows 10 desktop - yes its marketed as driver free (and hassle free) installation. Just plug and play. And indeed it would have done if I wasn’t on a Windows 10 machine.

Yes Turning Point DONOT support Windows 10 for their software and, for my receiver model at least, it appears to been have orphaned. It remains to be seen whether their new receivers support Windows 10 (they don’t actually state that) or whether they will release a driver for Windows 10. Support were, of course, ambiguous in terms of promising any future compatibility.

Poor effort and I can only say that its definitely more reliable using an online service such as PollEverywhere.