Python Scripting [7A]: Change File Modification Time on Map Images

This scripting project is a part of the NZ Rail Maps project. The aim is to change file modification times for a group of maps so that they follow a certain sequence when imported into Google Photos.

In the NZ Rail Maps project, maps published at what is defined as the "Basic" level are produced in two formats. These are "Aerial Maps" with the filename being in the format:

• First character is the capital letter "A"
• Next three characters are the numerical digits "000" to "999"
• Followed by the suffix ".jpg"

This means the filenames are a sequence and all operating systems will sort them in sequence alphabetically.

The other format is "Diagram Maps" which is the same as above except the first character is the letter "D".

When working with programs that recognise alphabetical sorting of file names there is generally no problem. However, Google Photos only sorts according to age which is assumed to be file modification time. 

The purpose of the script is to achieve these objectives which are both achievable simultaneously:

• Ensure that all A series maps are given a file modification time that follows the same sequence as their filename, and then follow the same process for the D series maps
• Ensure that for each numerical sequence number, the A series map is directly succeeded by the D series map.

Example: A001.jpg, D001.jpg, A002.jpg, D002.jpg will be the strict order by file modification time after running the script.

This means where both types of map are placed in the same disk folder, Google Photos will interleave them when imported. The user can also use the alphanumerical sorting order in the file dialog box to only import one series instead of both at the same time, if the Google Photo album should only contain one type of map instead of both.

This means that in addition to discrete "Aerial Maps" and "Diagram Maps" photo albums there can also be a "Combined Maps" photo album easily.

There can be variations of the filename format such as Axxx-2015-Old.jpg so basically the steps required are extract the first letter as a letter and then the next three characters as digits and then save the rest of the filename after that but getting the exact sort order is a little tricky as it does require taking into account more than just the first four characters of the filename. All we guarantee is that the first four characters are Axxx or Dxxx where xxx is a numerical digit sequence from 000 to 999.

Python Scripting: Planned Scripting Projects Update

Back in February this year I set out a list of planned scripting projects following my acquisition of Python programming skills. The sequence of planned or implemented projects to date is:

1. NZ Rail Maps project script to copy a set of GIS raster layer files based on reading a QLR file produced by Qgis. Currently used occasionally.
2. Auto sync script for creating an audio-only clone of a collection of video files. Not yet started.
3. A script to produce layer fractional segment sidecar files for GIS raster layers. Not used at present due to this feature not currently being used for mosaic tile generation.
4. EXIF based image rename script for digital camera photos / movies. Used almost daily.
5. I'm not sure what scripting project was going to have the number 5 because I went straight to 6 instead, and it's a mystery at the moment why I did this. It would have made sense at the time, because project 5 may not have been written down, or it appeared in one of the previous articles that I haven't re-read thoroughly enough to determine.
6. Taking the script from (3) above and changing it into a straight duplication script for simply copying sidecars where layers are duplicated. This is currently used quite regularly when mosaic tiles are exported for use with Qgis.

It is now time to pick up item (2) and start work on it, this week hopefully.  I now have a situation where I need to be able to play back music from a phone and base it on automating the audio extraction from video files to produce a music-only tree clone of a video file collection.

Python Scripting [6D]: Layer Sidecar Duplication & Renaming for NZ Rail Maps 4

Since we last looked at this topic a couple of months have gone by with everything working as expected when the duplicate script is used, both for duplicating sidecar files for layers that are the same size as the original and ones where the destination layer has been increased in size. The former is the most common usage although the latter case was the original reason the script was devised, but being able to carry out simple duplication for layers that are the same size as the original has proved extremely useful as this was originally a manual task requiring copying the files and renaming them which was tedious even allowing for Thunar's built in bulk renaming.

There has now been one refinement added to the original script which is an additional command line parameter -o or --overwrite which is for the purpose of specifying that existing files can be overwritten. If this parameter is included in one of the forms shown (no value required) then overwriting will be allowed, otherwise it will not be allowed.

To implement this in Python we have to add an extra line to the argument parser initialisation code block:

parser = argparse.ArgumentParser(prog='duplicate')
parser.add_argument('-s', '--source', required=True)
parser.add_argument('-d', '--dest', required=True)
parser.add_argument('-o','--overwrite', action = 'store_true')
parser.add_argument('-m', '--multisuffix', type=str, default="")
parser.add_argument('-p', '--pixelsize', type=float, default=None)

(the new line being the one in italics, the rest are existing)

The difference being that as no parameter value is passed with the command line parameter, an action must be specified, which is 'store_true'. The way this works is that if the parameter is specified then its value is set to True, otherwise it is set to False. (I am not sure how it comes to be False but this must be some sort of default or something ???)

Then we collect the parameter value with all of the others:

overWrite = args.overwrite

Since there is a block of a number of lines of code used to write the world file that would have to be duplicated, we put this block into a function and then call it as required by function name. So at the top of the script there is this function definition:

 # function to write new world file
def writeWorldFile(wFPName,wFLines,pSize):
    # write new world file
    wFile = open(wFPName, "w+")
    if  pSize is None:
        wFile.write(wFLines[0] + "\n")
    else:
        wFile.write(str(pSize) + "\n")
    wFile.write(wFLines[1] + "\n")
    wFile.write(wFLines[2] + "\n")
    if pSize is None:
        wFile.write(wFLines[3] + "\n")
    else:
        wFile.write("-" + str(pSize) + "\n")
    wFile.write(wFLines[4] + "\n")
    wFile.write(wFLines[5] + "\n")
    wFile.close()

Then when we are going to write the world file it looks like this:

           # copy world file
           #  first read existing world file
            worldFileName = fileNameBase + ".jgw"
            worldFilePath = os.path.normpath(rootPath + "/" + worldFileName)
            worldFile = open(worldFilePath, "r")
            worldFileLines = worldFile.readlines()
            worldFile.close()
            WFNNew = fileNameNew + ".jgw"
            WFPNew = os.path.normpath(rootPath + "/" + WFNNew)
            if overWrite:
                    print worldFileName + " -> " + WFNNew
                    writeWorldFile(WFPNew,worldFileLines,pixelSize)
            elif not os.path.exists(WFPNew):
                print worldFileName + " -> " + WFNNew
                writeWorldFile(WFPNew,worldFileLines,pixelSize)
            else:
                print WFNNew + " exists --------------------"

That could be simplified and made more efficient by putting the read operation into another function and only calling it when actually needed. At the moment the read is performed even if the write is not performed. 

The code for duplicating the two xml files is basically the same for both and the code block looks like this:

            # copy xml files
            auxFileName = fileNameBase + ".jpg.aux.xml"
            auxFilePath = os.path.normpath(rootPath + "/" + auxFileName)
            AFNNew = fileNameNew + ".jpg.aux.xml"
            AFPNew = os.path.normpath(rootPath + "/" + AFNNew)
            if overWrite:
                print auxFileName + " -> " + AFNNew
                shutil.copyfile(auxFilePath,AFPNew)
            elif not os.path.exists(AFPNew):
                print auxFileName + " -> " + AFNNew
                shutil.copyfile(auxFilePath,AFPNew)
            else:
                print AFNNew + " exists --------------------"

 

The same block is duplicated in the script with the .xml extension instead of .jpg.aux.xml

This could be made more efficient by putting this entire block of code into a function and simply passing the two different file extensions to it in two separate function calls.

Probably after doing more testing I will tidy the code up further along the suggested lines.

There is also an issue with some of the aux files possibly not being copied because every so often I will see a "CRS was undefined" message from Qgis when I load a duplicated layer. I have not yet tried to work out what is happening but the information about the CRS is stored in one of the xml files and there must be an issue happening with the file copy operation or something else happening that I haven't worked out yet.

Python Scripting [4D]: Exif Based Image Renaming 4

After testing the Exif renaming script for the past couple of weeks I have added a piece of code to check for collisions and rename any file that collides with an existing filename. With Exif data, the most likely cause of a filename collision is when a camera is set to shoot continuously and it takes several pictures a second. If the camera doesn't set the subsec time field then the exif string for the time the photo was taken will be identical for several photos and if used as the basis of a filename the result can be identical filenames. Another possibility is you own two cameras of the same type and you let someone else use the second one at the same time as you are using the first one (for example there are two of you covering a big event) so you can possibly get duplicates that way.

When I used to use IrfanView it had some sort of collision handling function built into it based on the one that Windows Explorer uses, which as some will be aware handles copying files to a new destination when it puts an extra number in brackets on the end of a filename. Apart from the fact that on Linux we obviously don't have the use of Windows Explorer, this wasn't able to be accessed for exif string renaming, so I wasn't able to use it and had to manually rename colliding filenames. There are a few older directories in my archive of photos on the computer that do have duplicates which in that case came about by the files being copied to somewhere else and then being copied back to the original folder, in this case the duplicate has an extra string of numerical digits on the end of the filename, which was what happened after a collision had occurred with the first time of copying some files, a second copy operation was run using the IrfanView capability to add a sequence number to the end of the filename.

So for this script I planned on adding a filename collision handling functionality to the code to deal with these possibilities. This has consisted of writing a function and calling the function from within the main script because it is needed in two places. Functions are fairly easy to do in Python and like any other programming language they are for more than just where you need to reuse a block of code. There is in fact a strong case to write all of your code in functions and then the main execution block just consists of a series of function calls and is very neat and tidy. I haven't done this yet with any of my scripts but I will start doing it with the next project.

The function is fairly straightforward and looks like this:

def fixCollision(fn): # handle filename collisions. fn is full filename with path
    p = os.path.splitext(fn)
    f = p[0]
    e = p[1]
    x = 1
    ff = fn
    while os.path.exists(ff):
        ff = f + "-" + str(x) + e
        x += 1
    return ff  

The first line is the file definition and an explanatory comment. The subsequent lines of code are focused on splitting the filename into its base and extension, and setting the initial value of the collision counter x. We then set the initial value of ff, the complete filename string that is tested in the while loop, to the filename that was passed into the function, and then enter the loop where the while clause checks to see if the filename exists. If it does then it makes a new filename by inserting the value of x with a dash separator between the base and extension, increments x for the next time around, and then goes back to the top of the loop. As soon as a filename is found which doesn't collide (which may include the filename that was passed into the function) then it exits the loop and it returns the filename to the calling code.

This is called within the script in two places (the code blocks that handle images and non images) as follows:

                    destFile = os.path.normpath(destPath + "/" + destName)
                    if not os.path.exists(destPath):
                        print "Create destination path " + destPath
                        os.mkdir(destPath)
                    destFile = fixCollision(destFile)

This code block is just before the move operation that moves the source file to its destination and renames it at the same time. In other words where we have a destination to move the file to and we move it to the new location and give it its new name as well. The first four lines in that block are creating the destination file path from the new direction and the new exif-based file name string (or a different process for non-exif files), checking for and if necessary creating the destination directory, and calling the collision resolution function. Essentially this results in ensuring the destination filename is unique within the destination folder.

The only other issue which has come about from the script is with the file permissions for the originals off the camera. As far as the computer and me are concerned, I only have read permissions to the photos and that is unchanged by the script. I am currently considering whether the script should change the permissions on each file as it is renamed. However there is an advantage in preventing the files from being changed in the Photos folder as they should be copied before any alterations are made and it may be that I will just leave things as they are.

Python Scripting [4C]: Exif Based Image Renaming 3

After letting this one slide for quite a while, I picked it up again yesterday and have completed most of the coding. This is the full code to date (barring any further modifications found to be needed after testing).

The script is run from the directory that is to be acted upon and takes no parameters as it works by looking for all the files off the source path.

  

# declarations

import glob
import argparse
import os
import shutil
import sys
import time
from PIL import Image

This is the basic declarations block. There may be some declarations that are irrelevant as it is copied from another script without checking.

# script is run from Photos subdirectory and processes subfolders
rootPath = os.getcwd()
exifDateMonth = None
exifDateYear = None

sourcePath = os.path.normpath(rootPath + "/" + "*")
for subDir in glob.glob(sourcePath):
    if os.path.isdir(subDir):
        if ("_" in subDir):

In here you just have the start of things which is to look for subdirectories off the source. The subdirectories that it scans for source file must have at least one underline character in their name. This basically matches the format of directory names that Canon uses on the camera by default (generally named xxx_dd_mm although sometimes it will be xxx___mm, depending on the camera or its settings).

            # process files from source directories
            imagePath = os.path.normpath(subDir + "/*")
            for sourceFile in glob.glob(imagePath):
                sourceExt = os.path.splitext(sourceFile)[1]
                try:
                    image = Image.open(sourceFile)
                    image.verify()
                except:
                    image = None # not an exif processable image

In this block we are getting the source file (hopefully a JPEG image with exif tags) and reading its exif data. We are using a try..except block to catch any exceptions that are raised in the process of reading the exif data block, in case there is no exif data or the source file is not an image. For example it could be a movie file which fits neither possibility.

                if (image is not None):
                    exif = image._getexif()
                    try:
                         # get the exif tags from the image
                        exifDateTime = exif[36867]
                        exifName = exif[272]
                    except:
                        image = None
                    try:
                        exifSubSec = exif[37521]
                    except:
                        exifSubSec = "00"

This is the first of two blocks of code that only works if an Exif file has been read and the tags have been extracted from it. We are using two try..except blocks. The first is trapping the possibility that this image doesn't have any Exif tags (which could happen with some image files) and if that happens we set the image to None (null). The second exception handling block deals with the possibility there is no SubSecTime field which could be the case with some older cameras that don't set this field, in which case we set it to a string value of "00". When I first started batch renaming photos off my cameras I never used to use SubSecTime and didn't know it existed, but it has become very useful for dealing with cameras that can take multiple frames per second in order to avoid file renaming collisions. The script doesn't have any way of detecting collisions and what the next iteration of it needs to do is to ensure where SubSecTime is not provided by a camera or is always 0, that it doesn't overwrite an existing file of the same name (an actual collision between two different files). Because right now it will happily overwrite an existing file without even blinking.

                if (image is not None):
                    # turn the tags into a filename
                    exifDateTimeParts = exifDateTime.split(" ")
                    exifDateParts = exifDateTimeParts[0].split(":")
                    exifDateYear = exifDateParts[0]
                    exifDateMonth = exifDateParts[1]
                    exifTimeParts = exifDateTimeParts[1].split(":")
                    exifDateStr = "".join(exifDateParts)
                    exifTimeStr = "".join(exifTimeParts) + exifSubSec
                    destPath = os.path.normpath(rootPath + "/" + exifDateYear + "/" + exifDateMonth)
                    destName = exifDateStr + " " + exifTimeStr + " " + exifName + sourceExt
                    destFile = os.path.normpath(destPath + "/" + destName)
                    print destFile
                    if not os.path.exists(destPath):
                        os.mkdir(destPath)
                    shutil.move(sourceFile,destFile)

This block works on valid Exif tagged files and it extracts from the date/time fields enough data to make the destination directory path. We use a path of year followed by month e.g. 2019/03 as a subdirectory of the current directory (the one from which the script is being run). We then make the new file name by combining the date, time and camera model name and the original extension. After creating the new path if it does not exist we then move the file from its original path and name to its new path and name. So the file gets renamed and moved to its new location at the same time.

                if (image is None): # non exif or unreadable exif
                    sourceDirName = os.path.split(os.path.dirname(sourceFile))[1]
                    sourceDirDtParts = sourceDirName.split("_")[-1]
                    SourceDirDtMonth = sourceDirDtParts[-2:]
                    sourceDateTime = time.localtime(os.path.getctime(sourceFile))
                    sourceDateYear = sourceDateTime[0]
                    sourceDateMonth = SourceDirDtMonth
                    destName = os.path.basename(sourceFile)
                    destPath = os.path.normpath(rootPath + "/" + str(sourceDateYear) + "/" + str(sourceDateMonth))
                    destFile = os.path.normpath(destPath + "/" + destName)
                    print destFile
                    if not os.path.exists(destPath):
                        os.mkdir(destPath)
                    shutil.move(sourceFile,destFile)

The final block of code is to deal with files that don't have Exif data in them. There is an inherent difficulty in processing these files that means this code block doesn't rename the files, instead it leaves the file with its original name. When the files are copied off the camera onto the PC, the original timestamps are lost and replaced with new ones. The system creates the copy of the file on the PC with the current date and time as the creation time, which means we just don't have the ability to know what the correct date to use for the destination file name is. However there is a clue in the way the camera creates the source folder with a day and month in the path. So this code basically gets the year from the file itself and then gets the month from the source directory name. It then moves the file to the new location, but without giving it a new name.

One way around this issue would be to create a script that copies directly off the camera. The problem is that the way a camera is interfaced into the operating system in Debian doesn't create a path to access it like it was a removable hard drive. I'm not sure if taking the card out and putting it into a USB card reader will make it appear like a full file path. The script will work correctly as long as the non-image file (e.g. movie) gets copied off the camera in the same year as it was created otherwise it will end up with the wrong year.

Python Scripting [6C]: Layer Sidecar Duplication & Renaming for NZ Rail Maps 3

After the use of the duplicate.py script for a while, an amendment has been decided upon and implemented this week. This is to simply duplicate the sidecars where there is no change of pixel size involved.  For example it may be just a case of the prefix being changed.

To implement these changes the following alterations were made in the script:

# set up command line argument parser
parser = argparse.ArgumentParser(prog='duplicate')
parser.add_argument('-s', '--source', required=True)
parser.add_argument('-d', '--dest', required=True)
parser.add_argument('-m', '--multisuffix', type=str, default="")
parser.add_argument('-p', '--pixelsize', type=float, default=None)

The main change in the above is to allow the multiplier suffix parameter -m to default to an empty string if omitted (instead of "x2") and the pixel size parameter -p to default to the special value None (which is essentially a Null) if omitted, instead of a fixed value size. Having an empty string default for multiSuffix allows no suffix on the filename if none is specified at input. The original code that adds this string on to filenames didn't need to be changed because an empty string is still a string and processed accordingly.

The other change is in the part that copies the original world file and specifically relates to the pixelSize parameter.

           # write new world file
            worldFile = open(WFPNew, "w+")
            if  pixelSize is None:
                worldFile.write(worldFileLines[0] + "\n")
            else:
                worldFile.write(str(pixelSize) + "\n")
            worldFile.write(worldFileLines[1] + "\n")
            worldFile.write(worldFileLines[2] + "\n")
            if pixelSize is None:
                worldFile.write(worldFileLines[3] + "\n")
            else:
                worldFile.write("-" + str(pixelSize) + "\n")
            worldFile.write(worldFileLines[4] + "\n")
            worldFile.write(worldFileLines[5] + "\n")
            worldFile.close()

The changes here are a couple of if..else statements relating to pixelSize. If pixelSize is set to the None value, as will be the case if no value was passed on the command line, then the original value is passed through from the source file. Otherwise, the pixelSize value is assumed to be a valid value to be written into the new world file.

This allows this duplicate.py script to be used for more things than was originally planned, since there are a number of cases where the base layers are not resized and still need their sidecar files to be duplicated for exported derivative layers from the mosaics, which this script will speed up the work of doing.

Python Scripting [6B]: Layer Sidecar Duplication & Renaming for NZ Rail Maps 2

Since yesterday the script has been completed and tested OK. Here is the complete script. It is similar to the segments script but a lot less complex with a total of 77 lines.

---

 # declarations
import glob
import argparse
import os
import shutil
import sys

rootPath = os.getcwd()

# set up command line argument parser
parser = argparse.ArgumentParser(prog='duplicate')
parser.add_argument('-s', '--source', required=True)
parser.add_argument('-d', '--dest', required=True)
parser.add_argument('-m', '--multisuffix', type=str, default="x2")
parser.add_argument('-p', '--pixelsize', type=float, default=0.15)

# get arguments
argList = sys.argv[1:]                              # drop the script name parameter
args = parser.parse_args(argList)
dests = args.dest.split(" ")                   # multiple dests supported
source = args.source
multiSuffix = args.multisuffix
pixelSize = args.pixelsize

This initial section is basically declarations and initialisations including setting up the command line input parser and receiving the input. I mentioned the command input in the previous post, and how the way to run the script is from the directory that contains all of the input files and receives the output files. To achieve this, rootPath is set near the top to be the current working directory, instead of being a fixed path as has been the case in other scripts. However the previous scripts were much easier to use for their particular purpose with specially designated hard coded directories instead of specifying a long path containing spaces for each of the source and destination parameters. So there is no right or wrong way concerning paths; there is just whatever is the easiest one to use for the particular situation.

---

# search for source 
filessourcePath = os.path.normpath(rootPath + "/" + source + "*.jpg")
for file in glob.glob(sourcePath):
    fileName = os.path.basename(file)
    fileNameParts = os.path.splitext(fileName)
    fileNameBase = fileNameParts[0]
    fileNameExt = fileNameParts[1]
    fileNameSects = fileNameBase.split("-")
   
    # loop through each possible destination
    for destName in dests:
        fileNameNew = destName + fileNameSects[1] + multiSuffix + "-" + fileNameSects[2] + multiSuffix
        FNNJpg = fileNameNew + ".jpg"
        FilePathNew = os.path.normpath(rootPath + "/" + FNNJpg)
        if os.path.isfile(FilePathNew):

This section contains code for finding source files and creating destination file specs. It initialises two processing loops. The first loop is initialised by getting a list of all jpg files in the directory and then slicing each file name into components (separated by a dash delimiter). The loop processes each source file name found. The second loop simply processes in turn each possible destination filename prefix. It uses this information to create a jpg file name by combining the components of the previous file name with the destination filename prefix and the multiplier suffix. It then enters an if statement loop which tests for the existence of a layer that has that file name (that was created by exporting from a Gimp mosaic).

---

            # File copying / writing loop
            print fileName + " ^^ " + FNNJpg
           
            # read world file
            worldFileName = fileNameBase + ".jgw"
            worldFilePath = os.path.normpath(rootPath + "/" + worldFileName)
            worldFile = open(worldFilePath, "r")
            worldFileLines = worldFile.readlines()
            worldFile.close()
            WFNNew = fileNameNew + ".jgw"
            WFPNew = os.path.normpath(rootPath + "/" + WFNNew)
            print worldFileName + " -> " + WFNNew
           
            # write new world file
            worldFile = open(WFPNew, "w+")
            worldFile.write(str(pixelSize) + "\n")
            worldFile.write(worldFileLines[1] + "\n")
            worldFile.write(worldFileLines[2] + "\n")
            worldFile.write("-" + str(pixelSize) + "\n")
            worldFile.write(worldFileLines[4] + "\n")
            worldFile.write(worldFileLines[5] + "\n")
            worldFile.close()

This section of code is concerned with using the destination file spec to create the world file for the destination layer. It accomplishes this by reading the world file for the original source layer, and copying all of the data except for the two lines which specify the pixel size of the destination layer. It also prints status messages to show what is happening (at the top is the message showing a matching destination layer has been found, and in about the middle is the message showing a world file is being copied).

---

            # copy xml files            
            auxFileName = fileNameBase + ".jpg.aux.xml"
            auxFilePath = os.path.normpath(rootPath + "/" + auxFileName)
            AFNNew = fileNameNew + ".jpg.aux.xml"
            AFPNew = os.path.normpath(rootPath + "/" + AFNNew)
            print auxFileName + " -> " + AFNNew
            shutil.copyfile(auxFilePath,AFPNew)
            auxFileName = fileNameBase + ".xml"
            auxFilePath = os.path.normpath(rootPath + "/" + auxFileName)
            AFNNew = fileNameNew + ".xml"
            AFPNew = os.path.normpath(rootPath + "/" + AFNNew)
            print auxFileName + " -> " + AFNNew
            shutil.copyfile(auxFilePath,AFPNew)

The script is concluded by a simple block of code to directly copy the xml sidecars without modifying them in any way. It writes a status message for each of the two files.

The script has been tested for real world situations and is performing a stellar job.

Python Scripting [6A]: Layer Sidecar Duplication & Renaming for NZ Rail Maps 1

Our new scripting project as of present is a script called duplicate.py which is specifically with the NZ Rail Maps project and it aims to achieve duplication and renaming of the sidecar files that are associated with raster layers.

Suppose that we have a base raster 4800x7200 pixels named in the following pattern:

• Timbuctoo-930W8-92NN9.jpg

We have created a Gimp mosaic project in which these layers are stretched to double dimensions in both directions. The mosaic incorporates historical aerial imagery from 1942, 1961, 1974 and 1984. As a result we have exported the following files from Gimp:

• T1942-930W8x2-92NN9x2.jpg
• T1961-930W8x2-92NN9x2.jpg
• T1974-930W8x2-92NN9x2.jpg
• T1984-930W8x2-92NN9x2.jpg

The next steps are to find the sidecar files for the original base raster mentioned above (named Timbuctoo-) and copy these files and rename the copies so that there are four sets of them to match the four exported rasters from Gimp. You can see as the exports have had "x2" tagged onto the row and column names, this string has to be inserted at two places into the original filename. 

In addition to this straightforward copy exercise, the world file (xx.jgw) has to have its pixel size entries changed. These are on the first and fourth lines and these have to be halved to reflect the fact that the original file is now doubled in each dimension, so that Qgis will draw it to occupy the same space on the canvas as the original.

There is a complication in that one of the sidecar files is the xx.jpg.aux.xml file as the issue is that the extension in this case is considered to be .xml rather than .jpg.aux.xml and therefore this has to be taken account of when determining where to add the "x2" string into the original file name.

A new aspect of this script that is possible because of the way it works is to run it within the directory that contains all of these files so we cd into that directory and then invoke the script passing its full path. And here we can make life easier for ourselves by using the ln -s command to create a symlink to the Scripts directory. So that to invoke the script we only have to type

• python ~/MapScripts/duplicate.py

Within the script itself we can use os.getcwd() to find out the directory we actually started in and then add that path to filenames when we do file operations. 

 

A typical invocation could look like 

 

• python ~/MapScripts/duplicate.py -s Timbuctoo- -d "T1942- T1961- T1974- T1984-" -m x2 -p 0.15

 The parameter -s refers to the source filename prefix and the parameter -d refers to one or more destination filename prefixes, which can be placed within quotes if there is more than one. -m refers to the multiplier suffix (the string added to the row and column names signifying the resize factor) and -p to the new pixel size to be inserted in the world file. The -m and -p parameters can be omitted as they will default (in this case, to the values actually shown).

 

The script will search for source .jpg files whose names start with the source prefix and end with .jpg and then look for export files whose names start with each of the specified destination prefixes and end with .jpg. If such a file is found, the sidecar files for the source are copied/renamed/altered as mentioned above, to each destination file name.

 

By next posting I expect this relatively simple script will be completed.

NZ Rail Maps: Optimising Gimp and using 4x4x4 grid for mosaics [3]

Since last writing on this subject I have further determined that I can scale 0.3 and 0.4 metre pixel resolution background Linz aerial images to double the scale (0.15 and 0.2 metres) and these scales work very well with the 1:4300/4325, 1:5500 and 1:8000 scale Retrolens aerial photos which are the best ones for creating yard layouts. Furthermore Qgis can handle a tile size that is four times the original (halving the original pixel size) without difficulty when render caching is disabled.

The result is that I don't need to use grid segmenting of the original layers to create the original tile size of 4800x7200 pixels, in which each of the original tiles was split into four segments with an index grid loaded into Gimp to ensure the correct naming pattern was used. This also facilitated processing the segments in a Python script to automatically generate the correct world files and copy all the sidecar files to the segment names.

I expect this grid segmenting system will only be needed with 0.75 metre or larger pixel sizes which are most practically scaled to 5 times original size resulting in a very large 5x5 grid. At the moment I can't think of one area in NZ for which there is only 0.75 metre imagery available although having said that it was only last year that 0.4 metres was available for the whole of Central Otago and when I started mapping the Cromwell Gorge only 0.75 m was available.

Whilst it is now considerably simpler for most cases to be able to eliminate grid segmenting, there is still a need for a script that can copy all the sidecar files from the original file names because in most maps there are multiple generations of historical imagery resulting in sidecar files needing to be duplicated for all these generations and the world file's pixel size needing to be scaled accordingly. I am now working on a script to perform these tasks as a straightforward step and unlike other scripts so far which are based on serverpc, this one will be hosted on mainpc as it will be working with files off the References directory in the maps share.

Python Scripting [4B]: Exif Based Image Renaming 2

Today I am going to have a look at Pillow and its interface for reading EXIF data from images. As I mentioned previously, Pillow is a fork of PIL (the Python Imaging Library) and contains capabilities to read and write EXIF data from images.

Although there are official documents for Pillow at Readthedocs, it is not very good at describing the objects and methods. I am attempting to pull together a description of the way things work in this post. I used sample code from https://developer.here.com/blog/getting-started-with-geocoding-exif-image-metadata-in-python3 in this post, which is possibly a reasonably comprehensive listing of how to access the data.

from PIL import Image

def get_exif(filename):
    image = Image.open(filename)
    image.verify()
    return image._getexif()

exif = get_exif('image.jpg')

Essentially you are invoking the Image object to open and read an image, and then calling its _getxif() method to retrieve exif data. What is returned from that call is a dictionary with all the EXIF tags in it. A dictionary is a specific data type in Python that contains a list of key:value pairs. Thus, the exif data is stored as a list of pairs where the key in each case is the numerical index of the tag. So you could look up a specific tag by passing its numerical index. The page listed above goes on to document other ways you can look up the data. Because I can get the list of tag numbers off another page and in fact already know the ones I passed in to IrfanView, I am just going to use the following in my code:

exif[tag_number]

which will give me the values I need.

Listing what I used from the previous post, the decimal tag codes used were as follows:

• 36867 - DateTimeDigitised. This appears to be returned as a string in the format YYYY:MM:DD HH:MM:SS
• 37520 - SubSecTime. This appears to be returned as a numerical value.
• 272 - Model. Appears to be a string.

There is a bit of manipulation of these values needed for my purposes. The date time data needing to be turned into two strings consisting of the date followed by the time with no extra characters and a space between the two. Followed by the subsectime as 3 characters, then a space, then the model name, and then the original extension.

In other words the file name should be changed to the format: YYYYMMDD HHMMSSmmm <camera name> where mmm is the subsec time in milliseconds.

The other considerations are as follows:

• The script needs to consider whether it just scans for JPEG files or all files in a directory.
• It will be run in directories where files have already been renamed so it needs to be able to skip the ones that have already been renamed, or avoid renaming where the code produces a filename that the file already has.
• If it looks for all files, it needs to be able to handle files that don't have EXIF data in them. I experimentally changed the script to scan a movie file that was in the same directory as the one that had my sample image in it. The result of attempting to read that file was an IOError raised by python. Another possible outcome is an empty directory object. Therefore I need to determine how the script will handle these instances.
• If a file doesn't have exif data we need to decide whether to rename it to something. For example a movie file could possibly be renamed based on the file date/time data rather than exif date/time.
• One of the things I would like this script to be able to do is to handle filename collisions. The generic way to handle a filename collision is to generate something to add on to the end of a new file name such as a sequence number. This is why I use the SubSecTime value to deal with shooting multiple images with the camera's drive mode setting on Continuous, where it can take a number of images within milliseconds of each other. Mainly a file collision detection will be needed where the SubSecTime is not set, as is the case on older cameras that I have owned.

I expect all of these considerations will be easy to implement but will require quite a bit of code of course.

So that is the end of this part, so next time I will do some serious coding to bring this together.

Python Scripting [4A]: Exif Based Image Renaming

This is now a new scripting project I am starting for Python. I need this script to rename all my photos off the camera, replacing the use of IrfanView which I used on my Windows 10 computer. Whilst I still have that computer and the software, I am looking to do something with Python scripting under Linux to achieve the same outcome automatically.

First thing is to look at the Exif string we use under IrfanView. 

This is the current string:

• $E36867(%Y%m%d %H%M%S)$E37520 $E272$O

 That is obviously specific to IrfanView in that it incorporates some parameters that are specified in their software design. All parameters start with a $. The breakdown is

• $E36867 - DateTimeDigitised
• (%Y%m%d %H%M%S) when following a parameter means to extract the year, month, day, hour, minute and second out of the parameter
• $E37520 - SubSecTime
• $E272 - Model
• $O - original extension of the filename including the period.

When put together that will rename a file to have a name that is based on the date, time and the name of the camera. SubSecTime is something that not all cameras support. It is basically the  millisecond component of the time and my current camera will provide this, but not all of my cameras have been able to provide it.

Then we have to translate these into something that is related to the standard. After doing some investigation, I found a list of tags on the Internet, and it listed tag codes that correspond to the above without the $E characters in front. These codes are in decimal, so every single one of them can be looked up in the list, and it turns out they are all part of the standard, and not manufacturer specific.

The second thing is to look at Exif read support for Python. This is generally implemented using a third party library. To install these libraries, a tool called pip is available. Once I installed that, I was able to install a library called exifread. To edit my script I am using KDevelop, which is the KDE supplied tool for development support, and it recognises Python out of the box.

Looking at exifread, I can start with a sample script that they provide:

import exifread
f = open("/home/xxx/Media/Pictures/Photos/2019/142_0803/IMG_2864.JPG",'rb')
tags = exifread.process_file(f, details=False)
for tag in tags.keys():
        print "Key: %s, value %s" % (tag, tags[tag])

This is a pretty simple example, which is hardcoded with a path to a specific image file I am using as an example. It just gets the tags for that file and outputs them. The exifread.process_file call gets passed a "details=False" parameter, which just limits the tags that come out, such as a thumbnail binary blob, and MakerNotes, which are manufacturer specific. The output looks like this:

Key: EXIF ApertureValue, value 29/8
Key: Image ExifOffset, value 388
Key: EXIF ComponentsConfiguration, value YCbCr
Key: EXIF CustomRendered, value Normal
Key: EXIF FlashPixVersion, value 0100
Key: EXIF RecommendedExposureIndex, value 250
Key: Image DateTime, value 2019:03:08 11:23:29
Key: EXIF ShutterSpeedValue, value 53/16
Key: EXIF ColorSpace, value sRGB
Key: EXIF MeteringMode, value Pattern
Key: EXIF ExifVersion, value 0230
Key: EXIF LensSpecification, value [15, 45, 0, 0]
Key: EXIF ISOSpeedRatings, value 250
Key: Thumbnail YResolution, value 180
Key: EXIF SubSecTime, value 87
Key: Interoperability InteroperabilityVersion, value [48, 49, 48, 48]
Key: Image Model, value <deleted>
Key: Image Orientation, value Horizontal (normal)
Key: EXIF DateTimeOriginal, value 2019:03:08 11:23:29
Key: Image YCbCrPositioning, value Co-sited
Key: EXIF InteroperabilityOffset, value 15836
Key: Thumbnail JPEGInterchangeFormat, value 20468
Key: Interoperability RelatedImageWidth, value 6000
Key: EXIF FNumber, value 7/2
Key: EXIF FileSource, value Digital Camera
Key: EXIF ExifImageLength, value 4000
Key: Image ResolutionUnit, value Pixels/Inch
Key: GPS GPSVersionID, value [2, 3, 0, 0]
Key: EXIF CompressedBitsPerPixel, value 3
Key: Thumbnail XResolution, value 180
Key: EXIF LensSerialNumber, value 000006a8dd
Key: EXIF ExposureProgram, value Program Normal
Key: Image GPSInfo, value 16078
Key: EXIF BodySerialNumber, value 495050000006
Key: Image Copyright, value
Key: Thumbnail JPEGInterchangeFormatLength, value 5322
Key: EXIF Flash, value Flash did not fire, compulsory flash mode
Key: Thumbnail Compression, value JPEG (old-style)
Key: EXIF ExposureMode, value Auto Exposure
Key: EXIF FocalPlaneYResolution, value 2000000/293
Key: EXIF FocalPlaneXResolution, value 2000000/293
Key: EXIF ExifImageWidth, value 6000
Key: Image Artist, value
Key: EXIF SceneCaptureType, value Standard
Key: EXIF SensitivityType, value Recommended Exposure Index
Key: Interoperability RelatedImageLength, value 4000
Key: Image ImageDescription, value                               
Key: EXIF DigitalZoomRatio, value 1
Key: EXIF SubSecTimeOriginal, value 87
Key: EXIF LensModel, value EF-M15-45mm f/3.5-6.3 IS STM
Key: EXIF DateTimeDigitized, value 2019:03:08 11:23:29
Key: EXIF FocalLength, value 15
Key: EXIF ExposureTime, value 1/10
Key: Image XResolution, value 180
Key: Image Make, value Canon
Key: EXIF WhiteBalance, value Manual
Key: Thumbnail ResolutionUnit, value Pixels/Inch
Key: Image YResolution, value 180
Key: EXIF FocalPlaneResolutionUnit, value 2
Key: Interoperability InteroperabilityIndex, value R98
Key: EXIF ExposureBiasValue, value 0
Key: EXIF SensingMethod, value One-chip color area
Key: EXIF SubSecTimeDigitized, value 87

It looks like most of what I am interested in are in those tags. The exact name would have to be flagged in a tag search to get a value, and then the value transcribed from a string into the data that gets fed into a rename algorithm. In practice this will be a call to a filename move because that is the rename allegory in Linux.

There are other libraries that do exif stuff. Some examples I found are:

• piexif
• exif
• Python Imaging Library (PIL)
• pyexiv2

Out of these examples I  chose to evaluate PIL (using its Pillow fork) as well. Next time around I will make a decision which of the two libraries (Pillow or exifread) will be more useful for my project.

• HERE Developer - Using Pillow

Python Scripting [3F]: Layer Fractional Segments for NZ Rail Maps 6

Today's little bit of fun and games has been to change the script so that it can read in a list file and process a list of source layers and produce the world and auxiliary files for them. Last night I discovered there was some additional aerial photos available for OtiraNorth area, for which I opened up the existing xcf project in Gimp that covered four existing areas with a total of 10 base tiles, I then added two more 0.4m base tiles rescaled to 0.1m, added the historical aerial photo, and as a result had six segments to export from the project. The result was a 17.3 GB Gimp file which it was able to handle OK with the extra swap space that it now has. If I can get a bigger swap disk for the computer then it should be able to handle very large files in future.

This means as there were three segments from each of two base tiles, I could put the parameters for the two base tiles into a file and save that, and then pass it to the script with a few modifications. These turned out to be rather more complex than I expected due to the different data types involved. 

Basically when you get arguments off the command line from sys.argv this is not a string containing arguments. It is a list of arguments. parse_args expects to be passed a list. But if you are reading from a file, with readlines, you get a list of strings, and when you loop through them, you have a string to pass in, which isn't what parse_args expects. So you have to turn that into a list, which typically you'd do by calling the string's split method.

So I have had to do some extra work to make sure parse_args is getting the list it expects to receive, because  otherwise it doesn't work as expected.

Here is the first part of the script up to the point where it reads the world files, showing the extra code needed to handle the extra and different parsing:

# declarations
import argparse
import os
import shutil
import sys

rootPath = "/home/patrick/Sources/Segments/"

# set up command line argument parser
parser = argparse.ArgumentParser(prog='segments')
parser.add_argument('-l', '--listfile')
parser.add_argument('-b', '--base')
parser.add_argument('-r', '--right')
parser.add_argument('-d', '--down')
parser.add_argument('-c', '--counter', type=int, default=4)
parser.add_argument('-p', '--pixelsize', type=float, default=0.1)

# check first for list file and handle if found otherwise assume single line input
argList = sys.argv[1:]                              # drop the script name parameter
args = parser.parse_args(argList)
if args.listfile == None:                           # single line input from command line
    listData = [" ".join(argList)]
else:                                               # multi line input from file
    listDataFileName = rootPath + args.listfile
    listDataFile = open(listDataFileName, "r")
    listData = listDataFile.readlines()
    listDataFile.close()

for listLine in listData:

    # parse arguments
    listLine = listLine.strip("\n")
    listItems = listLine.split(" ")
    args = parser.parse_args(listItems)
    #save the parameters
    baseName = args.base
    rightName = args.right
    downName = args.down
    counter = args.counter
    pixelSize = args.pixelsize

The major differences in this script therefore are:

• import sys is needed in order to use sys.argv which is the parameters typed on a command line.
• parser.add_argument calls are different. -b -d and -r are no longer mandatory and we put a new one in which is -l for the listfile.
• The next block is to parse the arguments to look for the list file parameter (-l). This time instead of using the default to parse_args which gets sys.argv itself, I have saved this into a string. We used sys.argv[1:] in order to drop the script name which is actually passed in as the first parameter; parse_args itself uses the same syntax to achieve the same thing.
• If a list file name was passed in then we read the list file into a list of strings. Each string contains one set of parameters. Otherwise we create a list containing one set of parameters from the command line. This means we have to first turn the command line parameter list into a string, with spaces between each parameter, and then turn that string into a list, which in this case will only have one string in it.
• We then enter a loop which loops through our list of parameter lines. We get each list item into a string.
• The first thing is to strip any newline characters (which is what we get as part of the input when we use readlines() to read multiple lines from a file, there will be a newline at the end of each line).
• Next is to split the string into a list whose items are the parameters themselves, using the split function with a space as input.
• Then finally we can call parse_args with this list as input.

From there the rest of the script is the same as before.

So we have about 20 more lines of code to handle the differences, which include discovering if there is a list file specified, and reading it, and then handling the conversions needed between different data formats. 

So I tested both types of input and it has worked as expected.

Python Scripting [3E]: Layer Fractional Segments for NZ Rail Maps 5

I made a lot of progress on this on Wednesday, mainly due to dropping nearly everything else and pushing on to finish it. After testing it, which so far has worked well, I decided to add an extra step to the workflow. The source layer (base layer) is a jpg file and we are making a jgw file for each of its segments. It also has a .xml file and a .jpg.aux.xml file with it, which we want to copy automatically.

So the extra code for this is

auxFileNameSource = rootPath + baseNameBase + ".jpg.aux.xml"

auxFileNameDest = rootPath + rootNameBase + ".jpg.aux.xml"

if os.path.isfile(auxFileNameSource):

    shutil.copyfile(auxFileNameSource,auxFileNameDest)

auxFileNameSource = rootPath + baseNameBase + ".xml"

auxFileNameDest = rootPath + rootNameBase + ".xml"

if os.path.isfile(auxFileNameSource):

    shutil.copyfile(auxFileNameSource,auxFileNameDest)

Again testing this has worked out. The script has now been tested and worked with all 10 segments that I had produced. One small change made near the top of the script is to reduce the amount of typing and have the script automatically add the .jgw extensions to the filenames that were typed in on the command line. So we are now only specifying layer names, not layer file names. This means changing another line further down in the script (now 82 lines) to eliminate the splitting of the extension off the input parameter. A few more lines now print out status messages, and comments have been added as well as whitespace. One thing I have done differently from my previous effort is to start camelCasing variable names. I don't know what convention the Python people recommend, but that's my personal preference, compared to underlines and other things that people sometimes use.

If I wanted to make the script even more useful I could have it work out the right and down layers automagically and it's possible I could do that but I would have to put in code to work out Linz's rules for naming sequences and that could be a lot more work so at the moment I will leave it here for the script and just put in those parameters but it is taking a bit of work to remember to put in -b and -d and -r before typing those layer names.

The other idea I am having is to be able to feed in a parameter list in a file and have it process all  source tiles at the same time, in a case where more than one or two source tiles are ready to be processed together. So if I modify my original script then I can have a different command line parameter and that will be -l or --layerlist that will read a list that is basically the command line parameters and the script runs through the whole file and then processes the entire list. It turns out that argparse can process a list of arguments that you pass to the parse_args function call, instead of the command line which it processes by default.

Here is the full script as it is for now. I expect that I will pursue the idea of adding an option to feed in an input file, and will look at it next time I need to use the script, so there may be an additional part added to this series then.

import argparse
import os
import shutil

#parse command line parameters
parser = argparse.ArgumentParser(prog='segments')
parser.add_argument('-b', '--base', required=True)
parser.add_argument('-r', '--right', required=True)
parser.add_argument('-d', '--down', required=True)
parser.add_argument('-c', '--counter', type=int, default=4)
parser.add_argument('-p', '--pixelsize', type=float, default=0.1)
args = parser.parse_args()

#save the parameters
baseName = args.base
rightName = args.right
downName = args.down
counter = args.counter
pixelSize = args.pixelsize

#Input file names
rootPath = "/home/patrick/Sources/Segments/"
baseFileName = rootPath + baseName + ".jgw"
rightFileName = rootPath + rightName + ".jgw"
downFileName = rootPath + downName + ".jgw"

#read input files
baseFile = open(baseFileName, "r")
baseData = baseFile.readlines()
baseFile.close()
rightFile = open(rightFileName, "r")
rightData = rightFile.readlines()
rightFile.close()
downFile = open(downFileName, "r")
downData = downFile.readlines()
downFile.close()

#save input file data
baseX = float(baseData[4])
baseY = float(baseData[5])
baseSkewX = float(baseData[1])
baseSkewY = float(baseData[2])
rightX = float(rightData[4])
rightY = float(rightData[5])
downX = float(downData[4])
downY = float(downData[5])

#Main calculation and processing section
#Initialisation
for colNum in range(counter):
    for rowNum in range(counter):
        segmentX = (((rightX-baseX)/counter)*colNum)+baseX
        segmentY = (((downY-baseY)/counter)*rowNum)+baseY
        gridX = "x" + str(counter) + "." + str(colNum + 1)
        gridY = "x" + str(counter) + "." + str(rowNum + 1)
       
        #Generate segment tile filename
        baseNameBase = baseName
        baseNameSplit = baseNameBase.split("-")
        baseColDescriptor = baseNameSplit[0]
        baseRowDescriptor = baseNameSplit[1]
        gridRowDescriptor = baseRowDescriptor + gridY
        gridColDescriptor = baseColDescriptor + gridX
        gridFileName = gridColDescriptor + "-" + gridRowDescriptor + ".jpg"
        # Next line for debugging output only
        #print(gridX + " " + gridY + " : " + str(segmentX) + " , " + str(segmentY))
       
        #Look for segment tiles that match current grid position
        rootFilesList = os.listdir(rootPath)
        for rootFile in rootFilesList:
            if rootFile.endswith(gridFileName):
               
                #Generate world file name
                rootNameBase = os.path.splitext(rootFile)[0]
                segmentName = rootNameBase + ".jgw"
                segmentFileName = rootPath + segmentName
                print(rootFile + " -> " + segmentName)
               
                # Write world file
                segmentFile = open(segmentFileName, "w+")
                segmentFile.write(str(pixelSize) + "\n")
                segmentFile.write(str(baseSkewX) + "\n")
                segmentFile.write(str(baseSkewY) + "\n")
                segmentFile.write("-" + str(pixelSize) + "\n")
                segmentFile.write(str(segmentX) + "\n")
                segmentFile.write(str(segmentY) + "\n")
                segmentFile.close()
               
                # find xml files for base layer and copy to segment
                auxNameSource = baseNameBase + ".jpg.aux.xml"
                auxFileNameSource = rootPath + auxNameSource
                auxNameDest = rootNameBase + ".jpg.aux.xml"
                auxFileNameDest = rootPath + auxNameDest
                if os.path.isfile(auxFileNameSource):
                    print("   " + auxNameSource + " -> " + auxNameDest)
                    shutil.copyfile(auxFileNameSource,auxFileNameDest)
                auxNameSource = baseNameBase + ".xml"
                auxFileNameSource = rootPath + auxNameSource
                auxNameDest = rootNameBase + ".xml"
                auxFileNameDest = rootPath + auxNameDest
                if os.path.isfile(auxFileNameSource):
                    print("   " + auxNameSource + " -> " + auxNameDest)
                    shutil.copyfile(auxFileNameSource,auxFileNameDest)         

Python Scripting [3D]: Layer Fractional Segments for NZ Rail Maps 4

Continuing from our last post, we now want to find tile segments that need world files written for them and generate those files. 

The code to do this (from the steps outlined previously is:

1. Strip the extension off the base file (92XJ7-92MKF.jgw) so that we get 92XJ7-92MKF:
   baseNameBase = os.path.splitext(baseName)[0]
2. Split the filename at the hyphen so we get two pieces, baseColDescriptor and baseRowDescriptor respectively being 92XJ7 and 92MKF in this example:
   baseNameSplit = baseNameBase.split("-")
   baseColDescriptor = baseNameSplit[0]
   baseRowDescriptor = baseNameSplit[1]
3. Set gridRowDescriptor to be a concatenation of baseRowDescriptor and gridY e.g. 92XJ7x4.1
   gridRowDescriptor = baseRowDescriptor + gridY
4. Set gridColDescriptor to be a concatenation of baseColDescriptor and gridX e.g. 92MKFx4.3
   gridColDescriptor = baseColDescriptor + gridX
5. Set gridFilename to be gridColDescriptor + "-" + gridRowDescriptor + ".jpg"
   gridFileName = gridColDescriptor + "-" + gridRowDescriptor + ".jpg"
6. Search for a filename that ends with gridFileName (it will start with something else like K1977full-)
   rootFilesList = os.listdir(rootPath)
   for rootFile in rootFilesList:
          if rootFile.endswith(gridFileName):
7. If the file name exists then change the full name into ".jgw" extension and write the 6 lines needed in the file:
   rootNameBase = os.path.splitext(rootFile)[0]
   segmentName = rootNameBase + ".jgw"
   segmentFileName = rootPath + segmentName
   segmentFile = open(segmentFileName, "w+")
   segmentFile.write(str(pixelSize) + "\n")
   segmentFile.write(str(baseSkewX) + "\n")
   segmentFile.write(str(baseSkewY) + "\n")
   segmentFile.write("-" + str(pixelSize) + "\n")
   segmentFile.write(str(segmentX) + "\n")
   segmentFile.write(str(segmentY) + "\n")
   segmentFile.close()

Testing so far indicates that the jgw files that the script wrote so far in testing are correct. Only one layer has been tested so far but the segment tile was presented in the right place relative to the base tile.

After looking at the workflow it would be ideal to provide the .xml and .jpg.aux.xml files for each jpeg file and have the script copy these automatically for each jgw file that it writes, so that all four files needed for each segment are complete. I expect to write a piece of code to perform that function and then present the complete script in the next part. So far the script is 72 lines total.

Python Scripting [3C]: Layer Fractional Segments for NZ Rail Maps 3

Last time we had a look at how to read data from our files and store it in a list. This time we are going to get that data and perform the calculations we need on it.

First thing is to convert the numbers read out as strings to floats. The numbers we need are in lines 5 and 6 in each of the three files.

baseSkewX = float(baseData[1])

baseSkewY = float(baseData[2])

baseX = float(baseData[4])

baseY = float(baseData[5])

rightX = float(rightData[4])

rightY = float(rightData[5])

downX = float(downData[4])

downY = float(downData[5])

Note we also have two more numbers from the base file only: baseSkewX and baseSkewY which we need to write out to every .jgw file that we generate. It is assumed in this script that we don't need to do any calculations with these.

Calculations needed are to generate, in this case, a total of 16 pairs of values (the original tile which is now 16 segments in a 4x4 grid, the actual number of rows and columns being passed in as the counter parameter on the command line). The grid then has 4 rows and 4 columns. The columns are numbered from 0 to 3 and the rows are also numbered from 0 to 3. For each segment, the X coordinate of the top-left corner is calculated by this formula, iterating through values of colNum from 0 to 3:

segmentX = ((rightX-baseX)/counter)*colNum)+baseX

Breaking that down:
rightX-baseX gives us the width of the base tile
divide by counter to give us the width of one segment
multiply by the column number to get the offset for a particular column
add to baseX so that we have the absolute coordinate, that is base plus offset.

Likewise segmentY is calculated in a similar way from rowNum in 0-3:

segmentY = ((downY-baseY)/counter)*rowNum)+baseY

So a double loop is employed to calculate 16 pairs of values:

for colNum in range(4);
       for rowNum in range(4):
             segmentX = (((rightX - baseX) / counter) * colNum) + baseX
             segmentY = (((downY - baseY) / counter) * rowNum) + baseY
             gridX = "x4." + str(colNum + 1)
             gridY = "x4." + str(rowNum + 1)
             print(gridX + " " + gridY + ":" + str(segmentX) + "," + str(segmentY))

That prints out 16 lines of data. gridX and gridY are interesting as they are the segment descriptors which are added to the end of the row and column descriptors of the original file name (basename in the script). For example 92XJ7-92MKF becomes 16 segments whose names run in the sequence from 92XJ7x4.1-92MKFx4.1 through to 92XJ7x4.4-92MKFx4.4

The next step is to write out the  .jgw files. Here we have a choice of just writing 16 files with the 16 pairs of values in them, which would be really easy to do here, or with a few more lines of code we can look for just the files we need to find by creating the filename pattern to search for and checking to see if that file exists. In other words, the user has put the segment tile (jpg file) into the segments directory for us to look up, and we shall write out a corresponding jgw file.

The actual spec of the file's contents would be as follows (using our existing variable names):

• Line 1: pixelSize
• Line 2: baseSkewX
• Line 3: baseSkewY
• Line 4: -pixelSize (in other words the pixelSize string with a - in front of it)
• Line 5: segmentX
• Line 6: segmentY  

The steps needed (in English rather than Python):

• strip the extension off the base file (92XJ7-92MKF.jgw) so that we get 92XJ7-92MKF
• split the filename at the hyphen so we get two pieces, baseColDescriptor and baseRowDescriptor respectively being 92XJ7 and 92MKF in this example.
• set gridRowDescriptor to be a concatenation of baseRowDescriptor and gridY e.g. 92XJ7x4.1
• set gridColDescriptor to be a concatenation of baseColDescriptor and gridX e.g. 92MKFx4.3
• set gridFilename to be gridColDescriptor + "-" + gridRowDescriptor + ".jpg"
• search for a filename that ends with gridFileName (it will start with something else like K1977full-)
• If the file name exists then change the full name into ".jgw" extension and write the 6 lines mentioned just above.
• There may be more than one filename that ends with gridFileName so we need to go through all the filenames in the directory.

The code needed to do that will appear in the next part in this series.

It would be fair to say this script is rather more complex than the first one I did (the files copying one) but the benefits of it are great because of the number of manual steps and potential errors eliminated. But it definitely has taxed my brain at times. I guess just as if we don't handwrite these days because of computers we lose the skills, if we don't do much manual calculations with our brains we lose that as well.

 

Python Scripting [3B]: Layer Fractional Segments for NZ Rail Maps 2

Yesterday we looked at how to collect command line parameters easily in a Python script. Today we need to turn the parameters into filenames and then read their contents to memory.

First thing is to get the arguments into variables we can refer to and manipulate. This is pretty simple as the arguments are stored in the args object which as we can see argparse provides us with.

basename = args.base

rightname = args.right

downname = args.down

counter = args.counter

pixelsize = args.pixelsize

We are using a fixed directory for all the files so it's defined here:

rootpath = "/home/patrick/Sources/Segments/"

Then we test the source files exist. I changed my filespec to assume the user has typed in the full file name. They only need to put the name because the directory is fixed and defined above.

basefilename = rootpath + basename

rightfilename = rootpath + rightname

downfilename = rootpath + downname

Then for each file we can read all the contents straight into an object using readlines

basefile = open(basefilename,"r")

basedata = basefile.readlines()

basefile.close()

repeating the same pattern for the other two files. 

In this example basedata is now a list object containing the lines (6 in total) read from the jgw file. The spec of this file is as follows:

• Line 1: A: x-component of the pixel width (x-scale)
• Line 2: D: y-component of the pixel width (y-skew)
• Line 3: B: x-component of the pixel height (x-skew)
• Line 4: E: y-component of the pixel height (y-scale), typically negative
• Line 5: C: x-coordinate of the center of the original image's upper left pixel transformed to the map
• Line 6: F: y-coordinate of the center of the original image's upper left pixel transformed to the map

We write out our target jgw files in the same way. Lines 2 and 3 are copied straight from the base file. Lines 1 and 4 are the pixelsize parameter (which in line 4 has to have a - in front of it). Lines 5 and 6 are generated by our calculations in the script (in part 3 we will look at these calculations).

The result of the readlines() call is to put all the lines in the file into a type of object called a list. A list object is referenced like an array, so I can get the first line in basedata by referencing it like this:

x = basedata[0]

Zero is the number of the first item in the list, so we look at 6 items numbered 0 to 5 in this case.

So far the complete script looks like this:

import argparse

parser = argparse.ArgumentParser(prog='segments')
parser.add_argument('-b', '--base', required=True)
parser.add_argument('-r', '--right', required=True)
parser.add_argument('-d', '--down', required=True)
parser.add_argument('-c', '--counter', type=int, default=4)
parser.add_argument('-p', '--pixelsize', type=float, default=0.1)

args = parser.parse_args()
basename = args.base
rightname = args.right
downname = args.down
counter = args.counter
pixelsize = args.pixelsize

rootpath = "/home/patrick/Sources/Segments/"
basefilename = rootpath + basename
rightfilename = rootpath + rightname
downfilename = rootpath + downname

basefile = open(basefilename, "r")
basedata = basefile.readlines()
basefile.close()
rightfile = open(rightfilename, "r")
rightdata = rightfile.readlines()
rightfile.close()
downfile = open(downfilename, "r")
downdata = downfile.readlines()
downfile.close() 

Part 3 will look at the calculations we need to do in order to get the coordinates of each segment.