Debian Buster Release due soon

Debian Buster (Debian 10) has been in development for the last few years, and it is expected to be released about the middle of this year.

As was described in some earlier posts, I have had Buster running on some of my systems in the past, but not in the last year or so. However I am now slowly upgrading my systems one by one to it. First up is pc4, and from there, mediapc or the bedroom pc will probably be next. This depends on the result for pc4. Since Qgis releases above 3.4 now require Buster, there is an impetus for me to want to move to Buster as soon as it is available, which is the reason I have started evaluating the impending release. The release cycle hard freeze is currently about a week away, so it is getting nearer to the release being ready.

mainpc and serverpc won't be upgraded until the release is out, and probably serverpc before mainpc is the order in which the upgrade will happen. Upgrade in all cases is by using the inbuilt upgrade system, which is fairly simple to use.

• Switch to a command window (ctrl-alt-F1) and log in as root. 
• Change /etc/apt/sources.list package sources from stretch to buster and save. 
• Then type in apt update and wait for the packages list to update.
• Type in apt upgrade to install the list of new packages (over 1000 for the first computer I upgraded). 
• Once it has finished that process then type apt dist-upgrade to update from stretch to buster.
• There will be a list of packages that are no longer needed so type in apt autoremove to discard them.

I am currently expecting to have all my computers running buster within a couple of days as all the upgrades have gone smoothly, provided the above spec is followed exactly. Also pc4 has been upgraded from XFCE to LXQt, meaning none of my computers are running XFCE now. LXQt on Buster looks better than the previous versions I evaluated on Stretch, probably because it is now version 0.14, up from 0.11.

Python Scripting [3F]: Layer Fractional Segments for NZ Rail Maps 6

Today's little bit of fun and games has been to change the script so that it can read in a list file and process a list of source layers and produce the world and auxiliary files for them. Last night I discovered there was some additional aerial photos available for OtiraNorth area, for which I opened up the existing xcf project in Gimp that covered four existing areas with a total of 10 base tiles, I then added two more 0.4m base tiles rescaled to 0.1m, added the historical aerial photo, and as a result had six segments to export from the project. The result was a 17.3 GB Gimp file which it was able to handle OK with the extra swap space that it now has. If I can get a bigger swap disk for the computer then it should be able to handle very large files in future.

This means as there were three segments from each of two base tiles, I could put the parameters for the two base tiles into a file and save that, and then pass it to the script with a few modifications. These turned out to be rather more complex than I expected due to the different data types involved. 

Basically when you get arguments off the command line from sys.argv this is not a string containing arguments. It is a list of arguments. parse_args expects to be passed a list. But if you are reading from a file, with readlines, you get a list of strings, and when you loop through them, you have a string to pass in, which isn't what parse_args expects. So you have to turn that into a list, which typically you'd do by calling the string's split method.

So I have had to do some extra work to make sure parse_args is getting the list it expects to receive, because  otherwise it doesn't work as expected.

Here is the first part of the script up to the point where it reads the world files, showing the extra code needed to handle the extra and different parsing:

# declarations
import argparse
import os
import shutil
import sys

rootPath = "/home/patrick/Sources/Segments/"

# set up command line argument parser
parser = argparse.ArgumentParser(prog='segments')
parser.add_argument('-l', '--listfile')
parser.add_argument('-b', '--base')
parser.add_argument('-r', '--right')
parser.add_argument('-d', '--down')
parser.add_argument('-c', '--counter', type=int, default=4)
parser.add_argument('-p', '--pixelsize', type=float, default=0.1)

# check first for list file and handle if found otherwise assume single line input
argList = sys.argv[1:]                              # drop the script name parameter
args = parser.parse_args(argList)
if args.listfile == None:                           # single line input from command line
    listData = [" ".join(argList)]
else:                                               # multi line input from file
    listDataFileName = rootPath + args.listfile
    listDataFile = open(listDataFileName, "r")
    listData = listDataFile.readlines()
    listDataFile.close()

for listLine in listData:

    # parse arguments
    listLine = listLine.strip("\n")
    listItems = listLine.split(" ")
    args = parser.parse_args(listItems)
    #save the parameters
    baseName = args.base
    rightName = args.right
    downName = args.down
    counter = args.counter
    pixelSize = args.pixelsize

The major differences in this script therefore are:

• import sys is needed in order to use sys.argv which is the parameters typed on a command line.
• parser.add_argument calls are different. -b -d and -r are no longer mandatory and we put a new one in which is -l for the listfile.
• The next block is to parse the arguments to look for the list file parameter (-l). This time instead of using the default to parse_args which gets sys.argv itself, I have saved this into a string. We used sys.argv[1:] in order to drop the script name which is actually passed in as the first parameter; parse_args itself uses the same syntax to achieve the same thing.
• If a list file name was passed in then we read the list file into a list of strings. Each string contains one set of parameters. Otherwise we create a list containing one set of parameters from the command line. This means we have to first turn the command line parameter list into a string, with spaces between each parameter, and then turn that string into a list, which in this case will only have one string in it.
• We then enter a loop which loops through our list of parameter lines. We get each list item into a string.
• The first thing is to strip any newline characters (which is what we get as part of the input when we use readlines() to read multiple lines from a file, there will be a newline at the end of each line).
• Next is to split the string into a list whose items are the parameters themselves, using the split function with a space as input.
• Then finally we can call parse_args with this list as input.

From there the rest of the script is the same as before.

So we have about 20 more lines of code to handle the differences, which include discovering if there is a list file specified, and reading it, and then handling the conversions needed between different data formats. 

So I tested both types of input and it has worked as expected.

Python Scripting [3E]: Layer Fractional Segments for NZ Rail Maps 5

I made a lot of progress on this on Wednesday, mainly due to dropping nearly everything else and pushing on to finish it. After testing it, which so far has worked well, I decided to add an extra step to the workflow. The source layer (base layer) is a jpg file and we are making a jgw file for each of its segments. It also has a .xml file and a .jpg.aux.xml file with it, which we want to copy automatically.

So the extra code for this is

auxFileNameSource = rootPath + baseNameBase + ".jpg.aux.xml"

auxFileNameDest = rootPath + rootNameBase + ".jpg.aux.xml"

if os.path.isfile(auxFileNameSource):

    shutil.copyfile(auxFileNameSource,auxFileNameDest)

auxFileNameSource = rootPath + baseNameBase + ".xml"

auxFileNameDest = rootPath + rootNameBase + ".xml"

if os.path.isfile(auxFileNameSource):

    shutil.copyfile(auxFileNameSource,auxFileNameDest)

Again testing this has worked out. The script has now been tested and worked with all 10 segments that I had produced. One small change made near the top of the script is to reduce the amount of typing and have the script automatically add the .jgw extensions to the filenames that were typed in on the command line. So we are now only specifying layer names, not layer file names. This means changing another line further down in the script (now 82 lines) to eliminate the splitting of the extension off the input parameter. A few more lines now print out status messages, and comments have been added as well as whitespace. One thing I have done differently from my previous effort is to start camelCasing variable names. I don't know what convention the Python people recommend, but that's my personal preference, compared to underlines and other things that people sometimes use.

If I wanted to make the script even more useful I could have it work out the right and down layers automagically and it's possible I could do that but I would have to put in code to work out Linz's rules for naming sequences and that could be a lot more work so at the moment I will leave it here for the script and just put in those parameters but it is taking a bit of work to remember to put in -b and -d and -r before typing those layer names.

The other idea I am having is to be able to feed in a parameter list in a file and have it process all  source tiles at the same time, in a case where more than one or two source tiles are ready to be processed together. So if I modify my original script then I can have a different command line parameter and that will be -l or --layerlist that will read a list that is basically the command line parameters and the script runs through the whole file and then processes the entire list. It turns out that argparse can process a list of arguments that you pass to the parse_args function call, instead of the command line which it processes by default.

Here is the full script as it is for now. I expect that I will pursue the idea of adding an option to feed in an input file, and will look at it next time I need to use the script, so there may be an additional part added to this series then.

import argparse
import os
import shutil

#parse command line parameters
parser = argparse.ArgumentParser(prog='segments')
parser.add_argument('-b', '--base', required=True)
parser.add_argument('-r', '--right', required=True)
parser.add_argument('-d', '--down', required=True)
parser.add_argument('-c', '--counter', type=int, default=4)
parser.add_argument('-p', '--pixelsize', type=float, default=0.1)
args = parser.parse_args()

#save the parameters
baseName = args.base
rightName = args.right
downName = args.down
counter = args.counter
pixelSize = args.pixelsize

#Input file names
rootPath = "/home/patrick/Sources/Segments/"
baseFileName = rootPath + baseName + ".jgw"
rightFileName = rootPath + rightName + ".jgw"
downFileName = rootPath + downName + ".jgw"

#read input files
baseFile = open(baseFileName, "r")
baseData = baseFile.readlines()
baseFile.close()
rightFile = open(rightFileName, "r")
rightData = rightFile.readlines()
rightFile.close()
downFile = open(downFileName, "r")
downData = downFile.readlines()
downFile.close()

#save input file data
baseX = float(baseData[4])
baseY = float(baseData[5])
baseSkewX = float(baseData[1])
baseSkewY = float(baseData[2])
rightX = float(rightData[4])
rightY = float(rightData[5])
downX = float(downData[4])
downY = float(downData[5])

#Main calculation and processing section
#Initialisation
for colNum in range(counter):
    for rowNum in range(counter):
        segmentX = (((rightX-baseX)/counter)*colNum)+baseX
        segmentY = (((downY-baseY)/counter)*rowNum)+baseY
        gridX = "x" + str(counter) + "." + str(colNum + 1)
        gridY = "x" + str(counter) + "." + str(rowNum + 1)
       
        #Generate segment tile filename
        baseNameBase = baseName
        baseNameSplit = baseNameBase.split("-")
        baseColDescriptor = baseNameSplit[0]
        baseRowDescriptor = baseNameSplit[1]
        gridRowDescriptor = baseRowDescriptor + gridY
        gridColDescriptor = baseColDescriptor + gridX
        gridFileName = gridColDescriptor + "-" + gridRowDescriptor + ".jpg"
        # Next line for debugging output only
        #print(gridX + " " + gridY + " : " + str(segmentX) + " , " + str(segmentY))
       
        #Look for segment tiles that match current grid position
        rootFilesList = os.listdir(rootPath)
        for rootFile in rootFilesList:
            if rootFile.endswith(gridFileName):
               
                #Generate world file name
                rootNameBase = os.path.splitext(rootFile)[0]
                segmentName = rootNameBase + ".jgw"
                segmentFileName = rootPath + segmentName
                print(rootFile + " -> " + segmentName)
               
                # Write world file
                segmentFile = open(segmentFileName, "w+")
                segmentFile.write(str(pixelSize) + "\n")
                segmentFile.write(str(baseSkewX) + "\n")
                segmentFile.write(str(baseSkewY) + "\n")
                segmentFile.write("-" + str(pixelSize) + "\n")
                segmentFile.write(str(segmentX) + "\n")
                segmentFile.write(str(segmentY) + "\n")
                segmentFile.close()
               
                # find xml files for base layer and copy to segment
                auxNameSource = baseNameBase + ".jpg.aux.xml"
                auxFileNameSource = rootPath + auxNameSource
                auxNameDest = rootNameBase + ".jpg.aux.xml"
                auxFileNameDest = rootPath + auxNameDest
                if os.path.isfile(auxFileNameSource):
                    print("   " + auxNameSource + " -> " + auxNameDest)
                    shutil.copyfile(auxFileNameSource,auxFileNameDest)
                auxNameSource = baseNameBase + ".xml"
                auxFileNameSource = rootPath + auxNameSource
                auxNameDest = rootNameBase + ".xml"
                auxFileNameDest = rootPath + auxNameDest
                if os.path.isfile(auxFileNameSource):
                    print("   " + auxNameSource + " -> " + auxNameDest)
                    shutil.copyfile(auxFileNameSource,auxFileNameDest)