Python Scripting [6C]: Layer Sidecar Duplication & Renaming for NZ Rail Maps 3

After the use of the duplicate.py script for a while, an amendment has been decided upon and implemented this week. This is to simply duplicate the sidecars where there is no change of pixel size involved.  For example it may be just a case of the prefix being changed.

To implement these changes the following alterations were made in the script:

# set up command line argument parser
parser = argparse.ArgumentParser(prog='duplicate')
parser.add_argument('-s', '--source', required=True)
parser.add_argument('-d', '--dest', required=True)
parser.add_argument('-m', '--multisuffix', type=str, default="")
parser.add_argument('-p', '--pixelsize', type=float, default=None)

The main change in the above is to allow the multiplier suffix parameter -m to default to an empty string if omitted (instead of "x2") and the pixel size parameter -p to default to the special value None (which is essentially a Null) if omitted, instead of a fixed value size. Having an empty string default for multiSuffix allows no suffix on the filename if none is specified at input. The original code that adds this string on to filenames didn't need to be changed because an empty string is still a string and processed accordingly.

The other change is in the part that copies the original world file and specifically relates to the pixelSize parameter.

           # write new world file
            worldFile = open(WFPNew, "w+")
            if  pixelSize is None:
                worldFile.write(worldFileLines[0] + "\n")
            else:
                worldFile.write(str(pixelSize) + "\n")
            worldFile.write(worldFileLines[1] + "\n")
            worldFile.write(worldFileLines[2] + "\n")
            if pixelSize is None:
                worldFile.write(worldFileLines[3] + "\n")
            else:
                worldFile.write("-" + str(pixelSize) + "\n")
            worldFile.write(worldFileLines[4] + "\n")
            worldFile.write(worldFileLines[5] + "\n")
            worldFile.close()

The changes here are a couple of if..else statements relating to pixelSize. If pixelSize is set to the None value, as will be the case if no value was passed on the command line, then the original value is passed through from the source file. Otherwise, the pixelSize value is assumed to be a valid value to be written into the new world file.

This allows this duplicate.py script to be used for more things than was originally planned, since there are a number of cases where the base layers are not resized and still need their sidecar files to be duplicated for exported derivative layers from the mosaics, which this script will speed up the work of doing.

Using a Raspberry Pi headless

I have used several different systems that connect to a tethered mobile connection for ministry related activities that shouldn't be tied to the shared internet connection that I use for most other things, out of respect for the people who own that internet connection and whose IP address is tagged to every activity. What one needs is a system that has both wired and wireless connections. This could be a laptop, or it could be a NUC, or a Raspberry Pi as they all have this capability built in. The second thing is that it can be run headless, because for occasional use we simply can't justify the space needed for a separate display, keyboard and mouse. 

As it happens I have all three types of device here and have experimented with them. The laptop and the NUC were able to be set up with Lubuntu which worked well with all the drivers involved, but I did try Debian with LXQt which I did refer to earlier and which was a bit more difficult to set up, so Lubuntu which now includes LXQt was a whole lot easier with this system and I will keep using Lubuntu on that computer. Both these computers were set up with X11VNC to begin with. That is quite an easy to install and configure type of VNC, but does inherently have a lag of a few seconds that can be annoying to work with when doing stuff on the VNC session with the computer.

At this point I have remembered that I have this Raspberry Pi and that I had trialled it in this use before. So I decided this would be the simplest step at this point to flash its microSD card with the latest version of Raspbian, which was released in April. It still uses LXDE, but it seems likely they will go over to LXQt because that is where the LXDE project has gone to. Hence that Lubuntu now installs LXQt by default. As I have made clear in previous posts I now have a preference of LXQt over XFCE because the latter is slow in development (they are just about to release version 4.14, when they really should be looking at 5.0 version) and I have lost the patience to assume they will keep up. This new version of Raspbian comes with RealVNC server built in, so it didn't need any additional installation to run headless with VNC once a few settings were done.

So the Raspberry Pi is now the ideal headless system that I can tether to my phone, for the ministry related activities or when the main internet connection is down, as does happen from time to time. It is connected to a spare input of the TV, but most of the time the TV is not set to display from it. The reason for having it connected is to enable the Pi to detect the screen, which it can do without the input being selected to it, and it will come up with VNC running and I can just connect to it from my regular computer and work on it. The best part is that with this VNC setup there is no lag at all. The Pi is set up with the wired ethernet connection being only usable on the LAN and no access to the internet, so VNC runs over that cable, and the Pi's wireless networking is used to connect to the shared connection from the phone. It all adds up to a very good use of resources with the very small cost that it takes to buy a Pi and that it is in its element doing this straightforward task that would be hard to do on a virtual machine which would still require a wireless connection in the host computer that most desktops would not have.

NZ Rail Maps: Optimising Gimp and using 4x4x4 grid for mosaics [4]

Last time I posted I wrote about my experiments rescaling layers in Gimp from 0.4 metres to 0.2 metres pixel spacing and 0.3 to 0.15 metres pixel spacing and seeing how these matched with the NZR station surveys.

I am still in two minds about whether to go with 0.2 metres from 0.4 metres source imagery because the 1:4300 layers in particular have to be scaled down to 70% of their original size, which is enough to lose quite a bit of sharpness. At 0.15 metres they would be about the same size, at 0.1333333333 metres (0.4/3) they need to be scaled up by 6%.

So at the moment I am redoing some of the Otago Central station survey mosaics with 0.4 metre base imagery scaled to 0.1333333333333 metres and then a layer that is 3x3 for Qgis that is 14400x21600 which is quite large but not too hard to load so we will have a go with that as well as continuing to experiment in the MSL coverage but that so far is generally 0.3 metres scaled to 0.15 metres so it does not have same challenges as 0.4 metre stuff.

But of course for the lower res corridor surveys and state highway surveys we can go at 0.3, 0.2 or whatever without too much resizing because it is a lower resolution so stations that come under those surveys aren't such a challenge.

Escaping Google's walled garden on handhelds [3]: Using Lineage OS - 1

So last time I wrote about the installation experience for Lineage, now it is time to have a look at early impressions of using it. Along the way I discovered some stuff about the Nexus 5X. It turns out that it was first shipped with Android and the final official release of Android for it was 8.1. Also that the last time Google sent a security update package for it was last December. So as far as Google is concerned they no longer feel a need to continue updating Android on this phone.

On the other hand the version of Lineage I was able to flash to it is 15.1 corresponding to Android 8.1 and they are currently working on a release called 16.0 which will be Android 9 so it is possible I could get a later version of Android for this phone that what Google is prepared to support, provided the developers are able to produce an Android 9 release for the "bullhead".

Although I am not using it as a full replacement for the Nexus running Google Android just yet, as I haven't yet switched back to using it as my day to day phone, initial impressions of Lineage are pretty good. It doesn't come with many apps of its own, but one can choose to install one's own from various sources. Of course you have to be sure the sources are reputable and are not going to contain spyware or malware or whatever. I was able to access a site called apkmirror to get first of all the Youversion Bible app and install that. I then didn't install any more apps for a while.

I am currently planning to install the F-Droid app to be able to access their app store (and possibly the Amazon app store as well, although I will be carefully considering that option since I want to steer clear of apps from big multinationals that could do unwanted things on the phone). For further security certain things that are built into Lineage will be activated and the app store apps will be disabled when they are not actually being used. In addition the phone will not be rooted as this is not really necessary as I only plan to use regular apps on it, not ones that need root permissions. All of these measures mean hopefully no security issues with apps from outside the Play Store. However there are many apps that use Google Play Services, and those simply will not run on Lineage.

At the same time I deliberately chose not to have social media apps on my phone and only one email account that is being checked regularly because I simply decided I don't need to have access to social media when away from home. This means apart from the phone and text messaging, the main thing the phone would be used for when not actually at home is reading my Bible or possibly listening to music (it depends on how well the battery lasts). Any time I was away from home for any period of time I can take my tablet with me and use all these apps on it but for regular day to day stuff I won't need any social media apps on my phone. Also I intend to use an Open Street Maps app instead of Google Maps.

So yes it is possible if one is willing to make a few sacrifices about what apps are really essential, to escape the "rat race" of having your phone taken over by gigantic multi national corporations to make enormous profits and just get back to the basics of the convenience of having a phone you can carry with you and which can perform a small number of additional functions.

One of the bonuses so far which is partly because Lineage gives control of the ability to run an app in the background is very much increased battery life. Although I haven't used my phone much in the last couple of days, it is currently 37 hours since the last charge and the battery capacity is 73% which is absolutely unheard of for this phone. If I used it I could expect it to go down a bit faster, but I am still going to get heaps more use out of the battery than I have before. So this phone will probably last a lot longer. Then I suppose I will have to work out what I can replace it with.

I am also planning to install the Android Studio app developer software and maybe some other F/OSS app developers to see what I can do with them, and whether I can develop an app to replace the Auto DND app that I use. But ultimately the replacement for a lot of apps is to use a web browser instead.

Escaping Google's walled garden on handhelds [2]: Installing Lineage OS

So after further investigation I decided Lineage OS would be the most straightforward means of getting my Nexus 5X off Google. This is quite a lengthy process that involves replacing the installed Google bloatware on the phone with the LineageOS installation images.

LineageOS provides guides, which are pretty good except that like me, you can get some steps wrong the first time. And of course the instructions differ depending on your version of Android.

The most important points found are as follows:

• When you install adb you will need to install the udev rules on Debian. Using the instructions for Ubuntu are what worked for me. But of course because Buster won't let us run all the commands in a bash shell, as usual we have to go out to a full screen terminal (Ctrl-Alt-F1) to do some of it, and then come back into KDE (Ctrl-Alt-F7) to finish things off.
• When you go into recovery mode, the Lineage instructions have missed out a step or two. The steps you need to follow to boot into recovery mode are
• Power off the Nexus
• Hold down Volume Down and Power to boot into recovery mode
• According to Lineage the next screen gives you "Recovery mode". This is not so. You actually have to press Volume Down button to get to Recovery Mode. By default it just comes up with "Start".
• Then you see a Android icon tipped on its side with a exclamation point hovering over it. 
• At that point you have to hold Power and then press and release Volume Up to get into recovery mode.
• The recovery environment install is tricky because if you don't reboot properly, it seems that Google will conveniently replace the TWRP recovery environment with its own (Android Recovery). How you know this has happened is if recovery comes up to TWRP it is a graphical touchscreen environment. But Android Recovery is all text based and using the volume up/down keys to select options and the listed options do not match the ones for TWRP at all.
• So you have to make sure you reboot properly to use TWRP to run the commands you need because Android Recovery will not let you flash anything into the system. So you can't use it to install.
• So after you have flashed TWRP into the system then reboot properly.
• On the recovery options DON'T CHOOSE REBOOT. Choose POWER OFF
• Once the Nexus is off then turn it on by pressing Volume Down and Power together.
• This will get you into TWRP with its graphical touchscreen and then you can get on with flashing Lineage and any other packages onto the phone.

I have chosen to flash only Lineage and the rooting package. I am no way absolutely never verboten from my cold dead hands etc NOT INSTALLING THE GOOGLE APPS PACKAGE. This means unfortunately not being able to use any app from the Play Store and anything that will rely on Google Play Services if from another source. However that is why I have other devices running regular Android but these devices are not a phone or do not have a cellular services SIM card installed or can be turned off when I am not using them thus preserving my privacy. 

After I had done the flashing I told TWRP to reboot the system. The most annoying thing at that stage was TWRP telling me that I had no OS installed and was I sure I wanted to reboot.  After rebooting you are going to see "Google" and then instead of the animated boot icon for regular Android you will see the Lineage one instead. Then there is a slightly scary moment when the whole screen is black and the system doesn't respond to pressing any buttons. Fortunately after a few more seconds your computer will pop up and tell you it recognises the Nexus (assuming the USB cable is still plugged in) and at that point checking the Nexus shows us that LineageOS boot screen has come up.

The setup steps are practically the same as for regular Android except you don't get asked to do any of the Google apps stuff but I was quite pleased to see it work with the fingerprint reader and allow me to enrol fingerprints just the same as before. After this it went into the home screen of Lineage and at that point was ready to go. I'll leave it there for this post and I am not going to do any more stuff on it right now, that will be covered in my next post after I have had plenty of time to play with Lineage and see what I can do with it. I will be using it initially with my tablet share SIM, which can make phone calls and send and receive texts but can't receive phone calls, until such a time as I might be ready to switch over to it becoming my regular phone again.

Escaping Google's walled garden on handhelds [1]: Intro

I've owned smartphones since 2012 when I was given one as part of a new employment role. It (HTC) was running Windows Phone 7. It was followed by a couple of Nokias running Windows Phone 8 (the later one since updated to 10) and then a low-end Motorola Moto E running Android 4. The next thing was a Samsung Galaxy J2 running Android 5 and a Galaxy Tab A 8" tablet also running 'Droid. My latest phone is a Nexus 5X which is currently on Android 8.

What has become particularly noticeable in handheld operating systems apart from Apple's (in a class of its own) in recent years is the extent to which vendors have sought to take ownership of the user's device and data. MS first did this with Windows 10 and went to extraordinary and obnoxious efforts to push users of older versions into upgrading to 10. 

Google has been sneaking upgrades into Android over the past few years, particularly in the "Google" app which in Android 5.1 has gone from 13 MB to 209 MB in size without any explanation or choice on the user's part as to the justification for installing this massive bloatware. But what we do know about Google's recent efforts is that voice recognition and the obnoxious Google assistant are now enabled by default with every new Android phone or upgrade and are very hard to disable. The assistant in particular is listening to everything that the microphone can pick up and transmitting it back to Google. In addition, it is becoming known that in Android 8, Google screen scrapes data from every app installed in the system, and that it tracks every location that the device travels to.

So we have entered an era where more than ever, operating system vendors are forcing updates onto systems to increase their ability to grab every possible piece of information they can collect on the device owners and transmit it back to the mothership so that it can be used to make increasingly bigger profits. And we are supposed to trust these gigantic multinational corporations that they have a benign intent and really are going to use our data to give us better products and services? LOL! 

So in much the same way as I have been through a journey of giving Windows the boot and installing open source software on my PCs, I am going to be doing the same sort of process with my handhelds. First of all, I will be switching my phone back to the Galaxy J2 with only a limited set of apps, and then the Nexus will be reinstalled with something else. At the moment, Lineage looks good, but I need to delve deeper into its architecture before making a final decision. The Galaxy Tab A will be left as is. If Lineage is what I want then it should be capable of becoming my day to day thing on the Nexus. Of course, it won't be running any of the Google apps or the Google Play Services. 

More to come...

Python Scripting [6B]: Layer Sidecar Duplication & Renaming for NZ Rail Maps 2

Since yesterday the script has been completed and tested OK. Here is the complete script. It is similar to the segments script but a lot less complex with a total of 77 lines.

---

 # declarations
import glob
import argparse
import os
import shutil
import sys

rootPath = os.getcwd()

# set up command line argument parser
parser = argparse.ArgumentParser(prog='duplicate')
parser.add_argument('-s', '--source', required=True)
parser.add_argument('-d', '--dest', required=True)
parser.add_argument('-m', '--multisuffix', type=str, default="x2")
parser.add_argument('-p', '--pixelsize', type=float, default=0.15)

# get arguments
argList = sys.argv[1:]                              # drop the script name parameter
args = parser.parse_args(argList)
dests = args.dest.split(" ")                   # multiple dests supported
source = args.source
multiSuffix = args.multisuffix
pixelSize = args.pixelsize

This initial section is basically declarations and initialisations including setting up the command line input parser and receiving the input. I mentioned the command input in the previous post, and how the way to run the script is from the directory that contains all of the input files and receives the output files. To achieve this, rootPath is set near the top to be the current working directory, instead of being a fixed path as has been the case in other scripts. However the previous scripts were much easier to use for their particular purpose with specially designated hard coded directories instead of specifying a long path containing spaces for each of the source and destination parameters. So there is no right or wrong way concerning paths; there is just whatever is the easiest one to use for the particular situation.

---

# search for source 
filessourcePath = os.path.normpath(rootPath + "/" + source + "*.jpg")
for file in glob.glob(sourcePath):
    fileName = os.path.basename(file)
    fileNameParts = os.path.splitext(fileName)
    fileNameBase = fileNameParts[0]
    fileNameExt = fileNameParts[1]
    fileNameSects = fileNameBase.split("-")
   
    # loop through each possible destination
    for destName in dests:
        fileNameNew = destName + fileNameSects[1] + multiSuffix + "-" + fileNameSects[2] + multiSuffix
        FNNJpg = fileNameNew + ".jpg"
        FilePathNew = os.path.normpath(rootPath + "/" + FNNJpg)
        if os.path.isfile(FilePathNew):

This section contains code for finding source files and creating destination file specs. It initialises two processing loops. The first loop is initialised by getting a list of all jpg files in the directory and then slicing each file name into components (separated by a dash delimiter). The loop processes each source file name found. The second loop simply processes in turn each possible destination filename prefix. It uses this information to create a jpg file name by combining the components of the previous file name with the destination filename prefix and the multiplier suffix. It then enters an if statement loop which tests for the existence of a layer that has that file name (that was created by exporting from a Gimp mosaic).

---

            # File copying / writing loop
            print fileName + " ^^ " + FNNJpg
           
            # read world file
            worldFileName = fileNameBase + ".jgw"
            worldFilePath = os.path.normpath(rootPath + "/" + worldFileName)
            worldFile = open(worldFilePath, "r")
            worldFileLines = worldFile.readlines()
            worldFile.close()
            WFNNew = fileNameNew + ".jgw"
            WFPNew = os.path.normpath(rootPath + "/" + WFNNew)
            print worldFileName + " -> " + WFNNew
           
            # write new world file
            worldFile = open(WFPNew, "w+")
            worldFile.write(str(pixelSize) + "\n")
            worldFile.write(worldFileLines[1] + "\n")
            worldFile.write(worldFileLines[2] + "\n")
            worldFile.write("-" + str(pixelSize) + "\n")
            worldFile.write(worldFileLines[4] + "\n")
            worldFile.write(worldFileLines[5] + "\n")
            worldFile.close()

This section of code is concerned with using the destination file spec to create the world file for the destination layer. It accomplishes this by reading the world file for the original source layer, and copying all of the data except for the two lines which specify the pixel size of the destination layer. It also prints status messages to show what is happening (at the top is the message showing a matching destination layer has been found, and in about the middle is the message showing a world file is being copied).

---

            # copy xml files            
            auxFileName = fileNameBase + ".jpg.aux.xml"
            auxFilePath = os.path.normpath(rootPath + "/" + auxFileName)
            AFNNew = fileNameNew + ".jpg.aux.xml"
            AFPNew = os.path.normpath(rootPath + "/" + AFNNew)
            print auxFileName + " -> " + AFNNew
            shutil.copyfile(auxFilePath,AFPNew)
            auxFileName = fileNameBase + ".xml"
            auxFilePath = os.path.normpath(rootPath + "/" + auxFileName)
            AFNNew = fileNameNew + ".xml"
            AFPNew = os.path.normpath(rootPath + "/" + AFNNew)
            print auxFileName + " -> " + AFNNew
            shutil.copyfile(auxFilePath,AFPNew)

The script is concluded by a simple block of code to directly copy the xml sidecars without modifying them in any way. It writes a status message for each of the two files.

The script has been tested for real world situations and is performing a stellar job.

Lubuntu 19.04 with LXQt

It should generally be known by now that the Lubuntu project, which produces a version of Ubuntu that utilises the LX desktop environment for low resource usage, has switched from LXDE to LXQt in its more recent releases. The use of LXQt provides for a refreshing modern interface design in this application and for me, it makes it possible to consider LXQt based distros as being a worthwhile replacement for XFCE, which is falling too far behind in development.

Although these days I prefer to use Debian as my base distro and install one of the desktop environments provided by the installer (KDE, XFCE and LXQt are all available in Debian 10, along with several other environments), for some of my computers extra hardware drivers such as for Wifi simply make it more convenient to install a Ubuntu-based distro that has these drivers built in. An example would be the NUC and the R500 laptop. 

The Lubuntu installer is very straightforward with a smooth GUI that breaks down the installation steps into a small number and it installs very quickly from the DVD ISO image on a pen drive.

The best part is that it works flawlessly compared to my experience of missing GUI interface tools for network connections in Debian 10/LXQt. The connection manager supplied in Lubuntu is interesting in that it uses a text based interface to set the parameters, but everything worked exactly as it should have for the requirement of a wired connection only working on the local network and a wireless connection for Internet. It would seem in the Debian installation I may have had two connection managers conflicting with each other but this is difficult as there was not any installed by default and I had to find which packages to install.

I will be using Lubuntu more in future on the wireless devices that I only use occasionally and on VMs where I need to test against a Ubuntu based distro so that I don't need to use up too much resources in the host system. LXQt while still at version number 0.1x has had a fair bit of updating lately since the earlier version that I installed over XFCE on systems for testing and then ended up discarding because of too many missing features so it is good to see this progressing and becoming more user friendly.

Python Scripting [6A]: Layer Sidecar Duplication & Renaming for NZ Rail Maps 1

Our new scripting project as of present is a script called duplicate.py which is specifically with the NZ Rail Maps project and it aims to achieve duplication and renaming of the sidecar files that are associated with raster layers.

Suppose that we have a base raster 4800x7200 pixels named in the following pattern:

• Timbuctoo-930W8-92NN9.jpg

We have created a Gimp mosaic project in which these layers are stretched to double dimensions in both directions. The mosaic incorporates historical aerial imagery from 1942, 1961, 1974 and 1984. As a result we have exported the following files from Gimp:

• T1942-930W8x2-92NN9x2.jpg
• T1961-930W8x2-92NN9x2.jpg
• T1974-930W8x2-92NN9x2.jpg
• T1984-930W8x2-92NN9x2.jpg

The next steps are to find the sidecar files for the original base raster mentioned above (named Timbuctoo-) and copy these files and rename the copies so that there are four sets of them to match the four exported rasters from Gimp. You can see as the exports have had "x2" tagged onto the row and column names, this string has to be inserted at two places into the original filename. 

In addition to this straightforward copy exercise, the world file (xx.jgw) has to have its pixel size entries changed. These are on the first and fourth lines and these have to be halved to reflect the fact that the original file is now doubled in each dimension, so that Qgis will draw it to occupy the same space on the canvas as the original.

There is a complication in that one of the sidecar files is the xx.jpg.aux.xml file as the issue is that the extension in this case is considered to be .xml rather than .jpg.aux.xml and therefore this has to be taken account of when determining where to add the "x2" string into the original file name.

A new aspect of this script that is possible because of the way it works is to run it within the directory that contains all of these files so we cd into that directory and then invoke the script passing its full path. And here we can make life easier for ourselves by using the ln -s command to create a symlink to the Scripts directory. So that to invoke the script we only have to type

• python ~/MapScripts/duplicate.py

Within the script itself we can use os.getcwd() to find out the directory we actually started in and then add that path to filenames when we do file operations. 

 

A typical invocation could look like 

 

• python ~/MapScripts/duplicate.py -s Timbuctoo- -d "T1942- T1961- T1974- T1984-" -m x2 -p 0.15

 The parameter -s refers to the source filename prefix and the parameter -d refers to one or more destination filename prefixes, which can be placed within quotes if there is more than one. -m refers to the multiplier suffix (the string added to the row and column names signifying the resize factor) and -p to the new pixel size to be inserted in the world file. The -m and -p parameters can be omitted as they will default (in this case, to the values actually shown).

 

The script will search for source .jpg files whose names start with the source prefix and end with .jpg and then look for export files whose names start with each of the specified destination prefixes and end with .jpg. If such a file is found, the sidecar files for the source are copied/renamed/altered as mentioned above, to each destination file name.

 

By next posting I expect this relatively simple script will be completed.

NZ Rail Maps: Optimising Gimp and using 4x4x4 grid for mosaics [3]

Since last writing on this subject I have further determined that I can scale 0.3 and 0.4 metre pixel resolution background Linz aerial images to double the scale (0.15 and 0.2 metres) and these scales work very well with the 1:4300/4325, 1:5500 and 1:8000 scale Retrolens aerial photos which are the best ones for creating yard layouts. Furthermore Qgis can handle a tile size that is four times the original (halving the original pixel size) without difficulty when render caching is disabled.

The result is that I don't need to use grid segmenting of the original layers to create the original tile size of 4800x7200 pixels, in which each of the original tiles was split into four segments with an index grid loaded into Gimp to ensure the correct naming pattern was used. This also facilitated processing the segments in a Python script to automatically generate the correct world files and copy all the sidecar files to the segment names.

I expect this grid segmenting system will only be needed with 0.75 metre or larger pixel sizes which are most practically scaled to 5 times original size resulting in a very large 5x5 grid. At the moment I can't think of one area in NZ for which there is only 0.75 metre imagery available although having said that it was only last year that 0.4 metres was available for the whole of Central Otago and when I started mapping the Cromwell Gorge only 0.75 m was available.

Whilst it is now considerably simpler for most cases to be able to eliminate grid segmenting, there is still a need for a script that can copy all the sidecar files from the original file names because in most maps there are multiple generations of historical imagery resulting in sidecar files needing to be duplicated for all these generations and the world file's pixel size needing to be scaled accordingly. I am now working on a script to perform these tasks as a straightforward step and unlike other scripts so far which are based on serverpc, this one will be hosted on mainpc as it will be working with files off the References directory in the maps share.

NZ Rail Maps: Optimising Gimp and using 4x4x4 grid for mosaics [2]

In the previous post in this section I wrote at some length about the use of grid segmentation of Linz base layers that were at resolutions of 0.3 to 0.4 metres pixel size. The grid segmentation being used to rescale these layers to 0.1 metre pixel size and then the original grid references were assigned additional letters and numbers to reflect that each original tile could be in 9 or 16 segments depending on the rescale factor.

This is felt to be necessary when using NZR station surveys that have a scale of 1:4300 to prevent losing too much detail on the ground when they have to be downsampled. In practice at 0.1 metre or smaller pixel size (0.075 is generally the smallest pixel size available from Linz) I am actually upsampling the historical imagery and it may be more evenly matched to a base layer of 0.15 metre pixel resolution. However I chose to continue with 0.1 metre as it is an even multiple of all the common sizes - 0.1, 0.2, 0.3 and 0.4 metres that Linz carries.

Lately working with other Retrolens scales, specifically the SH1 and NZR corridor surveys at a somewhat smaller scale, I have taken a look at how much detail loss occurs when downsampling these scales to match 0.3 metre base imagery and after doing a trial I have found there is not that much loss of detail and the result is still generally acceptable, especially in comparison to the 0.3 metre base images.

So I will only use segmentation in future for 1:4300 station surveys, and for the larger scale stuff that is still under 1:8000, which covers the NZR corridor surveys and state highway corridors, I expect to use the 0.3 metre or in some cases 0.2 metre base imagery without rescaling and segmenting. There are some areas where the base imagery is 0.4 metres and I expect to lose too much detail downsampling and will look at a compromise of rescaling this to 0.2 metres and a 4x tile area to avoid the need for segmentation in these cases. Since I discovered how to turn off render caching in Qgis and as there are likely to be a small number of areas using 0.4 metres and they will be small areas of a station, I think Qgis will be able to handle the larger tile sizes. Anyway we will see how that goes. Segmentation is a good system but it does take a lot more work to implement so it will just be used to get the best out of NZR station surveys which are in a small number of areas in any case, that aren't in urban areas with coverage of 0.075 metre to 0.15 metre resolution.

Eliminating the need for segmentation will speed up and reduce resource usage for mosaics for the MNL and MSL lines where I am currently working on including as many small stations as possible in the maps of these corridors. The development of more detailed maps for these corridors is now a priority in terms of volume production for these routes and will make extensive use of the SH1 maps as well as the NZR corridor surveys so as to get good quality maps completed as quickly as I can for these two lines in the South Island.