PDF Editors For Linux

This post is brought to you by the new Blogger interface Google has forced on everyone! Which is dreadful! The only way I keep my sanity editing posts on Blogger is to create the post in a separate editor (LibreOffice Writer in this case) and then paste the completed article into Blogger. There is much angst in the Blogger users’ community over the broken functionality in the new interface and the fact it has been forced into use with what they believe to be insufficient testing. My experiences with it exemplify those experiences, but this is the only one of my regular blogs that is still hosted on Blogger, and the workarounds for me are sufficient until such time as they fix all the bugs.

Today’s post is about editing a PDF using free/open source software. The PDF format as we are generally aware has historically been an Adobe thing, and so has the main editor software package. Everyone knows and uses Adobe Reader on various platforms, but relatively few people use Acrobat, the expensive commercial package that can edit PDF documents. Hence, a few alternative solutions have been developed, and the abilities of these are improving all the time. Here are my takes on a few of them.

My particular requirement here is filling out a PDF form and inserting my signature. It’s fine to be able to fill out the form in Okular (KDE’s in house PDF viewer) but inserting a graphic is impossible. So I looked at some of these alternatives:

LibreOffice Draw is part of the LibreOffice suite and can read and edit PDFs as files made up of individual elements. In my brief examination of Draw, the main concern I had was that it would be able to output the document looking like the original after editing; it seemed to have difficulty converting all of the text to typefaces that would fit cleanly into the original format. Because of this, I have not explored Draw further for my particular requirement at this stage.

Inkscape is a well known graphics editor that has a lot of features and is one of a few favourite graphical editors I have installed on my computer. I haven’t looked very deeply into its capabilities because the major limitation I have observed so far is that it can only handle a single page PDF; there is no obvious way of working with multi page documents.

Most of the full editors that are available are paid only. PDFSam and MS Word 2019 are examples that are Windows only. I have no desire at all to spend money on any type of Windows computer, or even a virtual machine, just to run these solutions. PDFstudio is an alternative that is available on Linux. The Pro edition that is capable of PDF editing costs $129 to buy and is licensed for 2 computers. It would be interesting to evaluiate this product at some stage to see if it is worth purchasing in future. Master PDF Editor is another product I might evaluate, it just puts a watermark on each page but it might be possible to remove that with one of the free editors.

Scribus is a FOSS desktop publishing package that also can open PDF files. Version 1.5 which is currently a development edition and only supported on most distros as an AppImage. I found however it has the same issue as some other packages of being unable to render fonts in the previously filled out PDF form.

Ultimately for this particular situation, needing a quick and easy solution to create my PDF and get it useful for my requirement, I have used Gimp which will import each page as either a layer or a separate image according to a selection choice when opening the document. It imports the pages as graphics, but you can fill in a form in something like Okular, save it to a new document, and then inserting a signature as a graphic can be done in Gimp, then export each image to a new file and paste them into a new document and export it back to PDF. A complex process for just one form but it lets me send my document completely filled out complete with signature because Okular cannot do the insertion of a graphic into the appropriate place on a PDF. I think this Gimp solution will be the best short term but I will still be interested in evaluating other possibilities in future.

Reinstall KDE system with LXQt [3]

This is the third article in a series about problems with KDE leading to a reinstallation with LXQt. Both running on top of Debian 10.

The computer concerned is running two NICs because it needs to access both networks that I have here. I have the main one based on a corporate wireless link from next door, which is filtered and throttled, and a second networtk based on a Skinny cellular broadband network modem, which is not filtered or throttled, although speed is moot since there are limitations inherent with 4G cellular data. The Skinny modem is connected to the local devices through a Ubiquiti AirRouter, which includes a DHCP server and network switch. It also provides the wireless network for phones or tablets. The capabilities of the AirRouter are much better than those built into the Huawei Skinny modem for internal networks and that is why I am using it.

There are some issues with LXQt mainly in panels but apart from that the reinstallation of software has gone well on the platform. One issue with this computer in general is that it needs more installed RAM and I am planning to address that in the next week or two, since I need it to be able to run a virtual machine at the same time as the range of stuff I normally use it for. One issue was reinstalling Gimp, I had difficulty in working out where Gimp keeps its config data as I wanted to copy over the config from my existing computer. It turns out that Gimp uses ~/.config/Gimp as a path, but apparently it also used a Flatpak specific path in ~/.var/ to store some data and there is considerable confusion over why it maintains these two paths especially as the user interface of the Preferences dialog doesn't actually point to the ~/.config path when it appears it should.

One of the interesting experiments this week has been to work with the mounting arms that I use to hold some displays. I have used Brateck wall mount arms with VESA adapters on them for this for some years. The key has been to modify these to allow fine control of the height so that when displays are stacked vertically, it is easy to put them close together. 

This photo shows how the attachment of the arm to the "wall" (in this case a vertical post attached to the desk) has been modified to allow its height to be adjustable. This is done with two right angle brackets which are bolted together using 80 mm length M6 gutter bolts, which gives theoretically an adjustment range of 80 mm.

I have to work around issues with the backup system from one of my computers at the moment. rynsc has not been able to connect over the network between the computers and running the backup locally on a computer seems to be the way to go at the moment.

NZ Rail Maps: Using Gimp To Georeference Retrolens Aerial Photos [7]: Extracting Mosaic Tiles

So I have been trialling the alternative overlay method with more files and have decided to redo all the mosaics for Auckland so far. This is a few days' work but as there are already some issues with accuracy in some of the more hilly areas or where the embankment is raised, it is going to bring improvements. However, file size savings are harder to nail down, and in general I think I am not seeing consistent meaningful reductions across a range of files. What I have arrived at to maximise file size saving, is to do a bulk crop to 50% of the original size, and then custom crop each layer to just the amount that is needed, that will vary from layer to layer depending on what is needed to align with the surrounding terrain (roads etc) to get smaller still. Currently this technique is being used on a big mosaic project, Auckland-Westfield NIMT, which covers 14 km of corridor and has around 180 layers. OK so there I broke my rule, by cropping the historical layers so much we can save a lot on disk usage and have more of them, but I am still having to test the water with so many layers in this project using just a small part of a very large canvas of 35 gigapixels which is probably the biggest canvas ever, as well, and that aggressive cropping is necessary to ensure the file size doesn't balloon unmanageably. Here's a screenshot of the canvas, with all the base tiles in place. The Port of Auckland can be seen upper left. This corridor was opened in 1930, the original route of the NIMT being the western line that became the North Auckland Line and Newmarket Branch dating from 1873.

This post outlines the last stage in this georeferencing process, which is to extract tiles from the mosaic project that can be imported into the GIS and used to produce historical maps that are accurate in positioning of features compared to present day maps. This being, of course, the major point of the whole exercise. There are some important considerations, the first being whether to use the same tile size, 4800x7200 in this case, as the base imagery, or combine a number of tile spaces into a larger sized tile. I tend to go for this latter option since it is just less work in extraction. The important issue here is to use the grid to get the top left corner to be the same as an existing background tile so that I can copy the coordinates from that existing tile when I need to make the sidecar files for the mosaic tiles. Sometimes there isn't one available and we will need to calculate the coordinates off the nearest available tile and here using EPSG:3857 with its coordinates expressed in metres we get a distinct advantage over a CRS that has its coordinates expressed in degrees because all you have to do, knowing the pixel resolution in metres, is make a simple calculation. In this case with this layer, at 0.1 metre pixel resolution of the base tile, each of those tiles covers an area that is 480 metres wide and 720 metres high. It then is straighforward to alter the top/left coordinates in the world file using these numbers.

The second consideration is how to name the new tiles. I always base my names on the original base tiles with a prefix. If the base tile gets scaled up or down, I also put a suffix on the original name to show that it has been resized. Let's say I have a base tile 93Y35-92S5Z that has been scaled double in each direction, I will have renamed it 93Y35x2-92S5Zx2. Then the prefix will be a single letter representing the station and followed by four digits representing the year. Then with a tile area that covers multiple original tiles I am going to use a filename that shows the first tile and the last tile in the range covered. So here is what my large mosaic tile's filename might look like: 

X1984-93Y35x2-92S5Zx2+93Y3Bx2-92S63x2.jpg

So in there we have station X in 1984, and then the range using layers that have been scaled up by 2 a side. Note the use of the + character which may not be permitted in Windows, this file name is legal in Linux which I use and which has far fewer filename letter restrictions than Windows. This is just an example - I don't know if I can cover all that large area in one file, as Gimp does have an export size limit, it may not be an efficient space to cover in one single file, and in this example we are actually not scaling 0.1 metre tiles up by two times anyway. 

My sidecar files are .jgw, .xml and .jpg.aux.xml and they all have to be given the same base name with the correct extension and we just copy the ones from the original base tile for the top left corner (93Y35-92S5Z in this example). The only change we need to make is in the jgw file - if we have scaled then the pixel resolution measurements need to be changed e.g. from 0.3 to 0.15 m. This is well described in previous posts. I described earlier in this post how I might change the top-left coordinates in the same file if I didn't have a set of sidecar files already downloaded to match the top left base tile in the area I am covering. Other posts on this blog describe the world file format in general and what those six numbers mean.

So hopefully in this series of posts I have adequately covered how to georeference the historical aerial layers in Gimp and how to get them into the GIS. Along the way I have learned some stuff as well, which is why I like to write these articles.

NZ Rail Maps: Using Gimp To Georeference Retrolens Aerial Photos [6]: Alternate Overlaying Technique

Today I am going to look at an alternative way of stitching together the source layers. I have to do a set of mosaics for the station of Kamo, which is the northernmost part of Whangarei. From a rail perspective there were two stations there. Just a little south of Kamo there was a coal mine which appears to have had a siding going off on the west side of the line, and although I don't know any more than that, I will include the NZR aerial images from Kamo right down to this area, since they are for continuous corridor. The idea that I am working on today is to reduce the amount of overlap between aerial layers. With every other project to date I skipped every second layer in the sequence but still found there was a lot of overlap between layers, enough to create various challenges I mentioned with overlapping, but not enough to be able to skip to every third layer. I want to look at reducing those overlaps, the potential benefits being a lot faster to overlay them in sequence, smaller file sizes and more accurate representation of the ground below with reduced perspective distortion. The way I did it before with every second layer was still overlapping quite a percentage of each layer and it being overlapped at one side, which means one side of the image will be more distorted.

In the layer sequence for Kamo down to the colliery siding the images are Survey 2788, Run A 1 to 3 and 2788 B 1 to 8. A total of 11 source layers. But I am going to use only the middle of each one of those. In addition because there is overlap between the A run and the B run, as is fairly common when the same rail corridor is covered by more than one run, because it goes around a curve and goes out of the side of the straight line of the existing run, I might be able to drop one or two layers. So what we need to do is to run Gimp with no projects open and open the source layers up using the Open as Layer command. Gimp creates a new canvas the size of the first layer and loads all the layers into it, centering them within that canvas. As all the source images are more or less the same size, they all line up with each other more or less exactly, or at least good enough for this exercise.

The obvious approach is to select a percentage of all of the source layers and apply that percentage to all of them. Using the centremost part of each layer is a good rule of thumb but you will have to check if this works with the source images you are using. But because you are using all of them, there is so much overlap everywhere that it's highly likely it will. The way to do this with Gimp is to use the Guides that you can set up from the Image menu. First of all, go to the View menu and select Show Guides. Then go to the Image menu and come down to the Guides submenu and if there are any existing guides visible and you don't know what they are then Remove All Guides. Then on the same submenu, New Guide (by percent). In the dialog select vertical or horizontal (you need guides that are at right angles to the rail corridor, but I set up both orientations to cater for both possibilities). Start with two guides at 33.33% and 66.66%.  The next step is to go through the layers one at a time and look at where the guides fall on them for example from A 1 to A 2 and see if there is enough overlap. In my case I decided to go for the middle 40% and therefore I went back, removed the guides and set them to 30% and 70% then there was enough overlap.

The next stage is to crop the layers and save them to be imported into the mosaic project. First thing to do is to create a selection that covers that middle 40%, which I do firstly by going to the View menu and turning on Snap to Guides, then I create a rectangle selection and use the guides to get the long edges into the right place. The short edges go up to the border, which we need to trim off in any case. This is why I didn't mention cropping the frame off each source layer as I did when I imported the full size layers directly into Gimp as I mentioned in a previous post. The way I am going to draw this selection is that it is a rectangle where the short edges parallel the rail corridor and the long edges are at right angles. This is how I get only the middle 40% of the rail corridor on each layer.

There are a couple of techniques for cropping in Gimp, just as a by-and-by, with different results in terms of their impact on the source layers. The obvious one to use is the Crop to Selection option on the Image menu. This will take the canvas down to the size of the selection and at the same time it will crop every layer to fit that canvas. As it happens this is the right thing to do with this project. The alternative which is useful to know if you want to keep the source layers at their full size, is to choose the option on the Image menu to Fit Canvas to Selection. That doesn't have an application to this particular project at all, but there are times when it's useful to have the full size layer available even if part of it is outside the visible canvas area and I just mention that in passing in case one day that is the situation you have, and there have been times I have used this alternative, for example when selecting a few layers from off a bigger canvas and saving them in a small canvas. Gimp will save layers that are outside the visible canvas, obviously using up more disk space than if all the layers are cropped within the canvas. Keeping the canvas just to the size you need is important because I discovered a bug in Gimp with a large canvas with only one small corner occupied that it will crash when saving. So my rule of thumb is only create the canvas size you will use and expand it as needed.

So use this Crop Image to Selection option to crop down all the layers at once. Then save to a new Gimp project file. I have 11 layers (A 1 to 3 and B 1 to 8), and the new project file has saved all of the middle 40% of 11 layers at less than 1 GB total, which looks really good so far. Now I am going to import additional base layers and my 11 segments of overlays into my mosaic project. In this case my base layers go from 93Y35-92S5Z to 93Y3B-92S63, which is a total of six columns (Y35-36-37-38-39-3B) and five rows (S5Z-60-61-62-63), so the first task is to increase canvas size by adding 36000 pixels to the top, the canvas being already wide enough for the six columns. My new canvas will therefore be 151200 x 52800 pixels since I added an extra blank row to ensure a one-row gap in the canvas between this geographical area and the one appearing on the canvas below, as I have multiple areas (different railway stations) shown on the same canvas. One thing when I started doing large projects is it doesn't matter how much of the canvas is actually used. What uses the disk space is each layer, regardless of where it sits in the canvas. I have had some very large canvases that only have an L shaped area along two sides occupied, maybe only 15-20% of the entire canvas, and there are no issues like wasting disk space because Gimp only saves the actual layers, not the canvas itself. It just saves the measurements of the canvas and the position of each layer, not every single pixel of the canvas itself. 

After resizing the canvas, use Open as Layers to bring in the base layers, being sure to first select in the Layer list the layer that you want the new layers to be inserted above, and if you want the new layers to appear in the right order then in the file dialog box select a sort order that is the reverse of the one you want. Then once the base layers have been imported and positioned in the right places in the canvas, bring in the overlays. You do this using the Open as Layers command again, but instead of selecting image files to bring in as one layer each, you select your project file that you created from cropping the overlays.  Gimp brings in all the layers from that project file into your current project individually, inserting them above the current position in the layer list in the same order as you had them in their project file.

Then you can overlay them the same as usual. That turned out for me to be a lot easier working with smaller layers, and the result was they came together in the mosaic project quite rapidly. I did change the layer order in one place, putting layer A 3 above A 2, and was able to dispense with layer B  1 because B 2 had enough overlap onto A 3, and I also found I didn't need layer B 8, so that got me down to 9 layers out of the 11 I thought I would originally need. After putting in the first layer, I can speed up the subsequent layers by just dragging them into the same size as each previous one and then do the underlap straight away, then line up the bottom edge. Even with the potential challenges of surrounding hilly terrain not lining up and significant changes in the general area - there is now a lot of residential development on the east side of the railway at the first Kamo station it was very straightforward. (Kamo had two stations both of them named Kamo, the earlier one being further south than the later one, and as both were open at the same time, the more northern one was originally named Ruatangata or Ruatangata Springs.) The amount of overlap (and therefore underlap) is much less and the narrower layer segments, for various reasons, are simply a lot easier to work with - because they are a closer representation of the actual terrain. I believe the result has fewer overlapping errors and is a more accurate representation of what is on the ground overall but this will have to be tested out in practice, and that is much more likely to be tested in mosaics covering large rail yards than with a station with a few tracks or a single track main. It's hard to work out any possible improvement in file size but as an exercise I went back to my original project, removed every second layer and the redundant layers and then saved it, the result was a file 50% larger than the original. When I saved my mosaic project in Gimp, it already had the full size versions of all the layers in Run A and Run B imported in, so it's hard to pick up the actual saving in disk space that might have resulted. I've taken out 17 full size layers and replaced them with 9 that are less than half the original size, but in their place I imported another 16 base layers. But anyway the new version of the mosaic project file is about half a gigabyte smaller than the previous version. So the whole I think there are going to be some physical benefits in resource usage as well.

So the net result of this exercise is that is how I will be doing my mosaics in future. Taking the middle 40% slice out of consecutive layers because it works out better all round. The overlaying process at this point can be described as:

1. Align and size an overlay along the long edges of the previous overlay (does not apply to first overlay)
2. Find the underlap point with the previous overlay (overlap of background for first overlay) and roughly align to it.
3. Line up the bottom edge to roughly the right place
4. Do Step 2 again more precisely
5. Do Step 3 again more precisely
6. Transform.

There is one more part to write and that is how to extract the mosaic tiles for the GIS. That will be Part 7 next time.

NZ Rail Maps: Using Gimp To Georeference Retrolens Aerial Photos [5]: Overlaying Multiple Layers

So in my last post I showed how to get started in georeferencing with the first overlay. In that post I put the second layer in to show how it would join onto the first layer. The aim of this part is to show how to add additional overlays and how to deal with the overlap between one layer and the next. 

Right now from the last post this is what I can see on my computer screen of the Whangarei rail yards and sidings. Whangarei has a main station yard and a separate port area with its own sidings. The port used to handle container traffic until it was all moved to Marsden Point (which is near the heads of the Whangarei Harbour, whereas Whangarei is at the top) around 20 years ago. There has been huge debate ever since over the lack of a rail link to that port, and the rail-friendly Clark Government started the process to build the spur line that would come off the North Auckland Line at Oakleigh, only 14 km south of Whangarei. But then there was an election and the anti-rail Key administration came into power and stopped all the rail work. It is now being resumed by the Ardern government and probably the construction will be approved to start in the next 12 months. Anyway those port sidings are still in place to some extent and some ships still come into the port so the maps of Whangarei will document that but we need seven overlays in the D series and two in the F series, a total of nine. I am also mapping Kamo which is to the north of Whangarei as the same series of aerial photos Survey 2788 has two runs that cover that part of the Whangarei urban area and the Northland 0.1 metre urban area aerials are also available for there. 

But for now let's have a look at overlaying D 5 onto D 3. As you can see I am not getting across much distance with each overlay. I can do a rough measurement from the grid because I know there are 4800 pixels wide and 7200 pixels high and each pixel measures 0.1 metres on a side, so with the two overlays so far covering roughly 2 high and 2 wide, so far we have covered an area 960 metres by 1440 metres, or 0.96 km by 1.44 km. It's easy from that to understand how the disk space gets gobbled up because I have to be able to work at this high resolution to get the detail visible down on the ground that I need and that means a lot of layers. This particular project file covers Oakleigh, Portland, Whangarei, Kamo and possibly Hikurangi / Waro in one canvas which is not continuous but broken into separate areas for each station, and it's currently on a 52800x108000 canvas with 69 layers and is taking up 10 GB on the disk. As I mentioned previously my rules are not to exceed 100 layers and 20 GB disk size although I do have a small number of projects that are around 30 GB but Gimp doesn't seem to be able to handle much larger than this without crashing a lot. But when covering a long urban corridor and having to split it across multiple files, then an extra complexity is to deal with the overlaps between the edges of each file. Of which more later.

One of the reasons progress is slow is that even with skipping to every second image in the NZR survey, there is still up to 50% of each image overlapped and that is a lot as you can understand. It lends weight to the idea of taking all the images in the survey and just chopping the middle out of them. That is something I might try with Kamo because I could save a lot of disk space that way because the overlapped pixels are just wasted duplication that uses up storage.

So now in Gimp I've selected D 5, put it into unified transform, and started lining the top of it up with the bottom of D 3. Now, it pays to be smart about the order in which Gimp draws layers, because there is this overlap in the case of these overlays. The base aerial photos are all nicely broken up into tiles that don't overlap, they all just nicely fit against each other in the grid, but we don't have that convenience with the overlays. So if I have three layers in the layer list the highest up one will be drawn over the top of the ones lower down in the list. And that calls for some careful thinking when doing the unified transform, because what you see in the preview is the overlay that is selected for the transform, drawn over the top of all the other layers and contradicting the order in which it will actually be drawn when the transform is completed. Here's a screenshot that illustrates that pretty neatly.

So what you can see happening in this screenshot is I've done the first stage of overlay transform. I've put the pivot right up the top of D 5 at a point that I am confident I can line up on that is also present in D 3, and then I've gone down to the bottom of D 5 and dragged a size handle to size the overlay to match the surrounding terrain pretty well. So we start by overlapping somewhere near the top of our current overlay, because that is the easiest until we get the overlay to the right size. Then once we have that sorted, we now have to "underlap" as I call it. We have to align the overlay at the bottom of D 3 because that part of D 3 will be drawn over the top of D 5 all the way to the bottom edge of D 3. You can see where the overlay transform's frame is, the handles and the pivot at the top of the frame. But down near the bottom of this screenshot, what's underneath changes to colour instead of black and white. That's where the bottom edge of D 3 actually reaches to and that's where it is most critical to actually line up D 5 because that is where the join between the two layers will actually be drawn. 

When you come to do overlays on rail tracks, these old aerial images have marks that were specially placed on the ground by NZR staff. They would paint a sleeper completely white over its whole top and then add white paint on the centre sections of the sleepers on either side. The frog rails and guard rails on turnouts are usually also painted white, but this is more to highlight them as a safety hazard when crossing over the rails in the yard, as someone could get their shoe stuck in the gap. This is what the result looked like. The purpose of this was to make it possible to overlay the aerial photos into a mosaic that was printed out on a big piece of paper that was used by the engineering staff. I have seen these big printed plans which were over a metre by a metre when printed out and some of them are in storage in archives around the country.

It is a bit harder to get the join between two layers exactly right when it isn't actually on one of the overlay edges, partly because you have to take the preview opacity up and down to check the fit, and because the overlay at 100% opacity will not show the join at all, it will look like the join is at the top of D 5 instead of at the bottom of D 3. Before Gimp was updated to 2.10.18 with the Readjust feature, it was a lot harder to do a proper alignment at these joins and one trick I used was to do the overlays starting from the bottom of the Gimp layer list instead of the top because the preview would be correct for the layer order, another trick was to reorder the layers in the layer list if the join ended up being rough. But especially since Readjust became available it is now much easier to do these joins really smoothly and that has made a big difference to this task.

So now I am going to move the pivot down to the bottom of D3 and line things up down there. At this stage I still have the transform frame at the full size of the overlay and haven't used Readjust, which moves the frame and changes its size. By bringing the preview opacity up and down and finding the exact point of overlap, I can get a very smooth join. Here the shortcut keys I have set up to step opacity up and down 10% per step are very useful to preview what the join will look like. Basically you are comparing the overlay at 100% to the overlay at 0% and looking at how smoothly it overlaps close to the join. You can get a pretty good feel for how that join will look and in the case of a rail yard or some other situation where the join is wider than a track or two, you want to be checking all the way across. Bring the pivot down to that join, put it on some key feature like a main line, and if necessary use the Readjust to change the size of the overlay at the join so you can get it looking really smooth. Just remember Readjust moves the pivot to the centre of the new transform frame, and that you can only ever reposition the overlay by dragging inside the transform frame regardless of its size.

Once you have that pivot in place and have the join looking good you need to check at the edges of the overlay again for the underlying base imagery you are lining up on, if necessary using Readjust to make a really good smooth alignment, again being aware with Gimp 2.10.18 you need to move the pivot back to the join when you press the Readjust button, then finalise by checking the join is still correct, and then we are ready to transform. In this case the most important alignment for the bottom of the overlay is the main rail line heading south, and for the sides, it is the surrounding streets and houses, because with no clear features near the bottom to ensure we have the bottom of the overlay in the right place, we have to use features out to the sides to help check our alignment. Here's the result as we have it now, with a really smooth join that I am not going to zoom in but I got it really neat again across the rail yard, but with varying results further out, as we'd expect.

And so I will carry on with the rest of the overlays to get all the way down, all nine of them.

There is one more thing to mention and that is when you are doing a continous corridor, like in an urban area. In reality, NZR had corridor surveys of various types done, with the major main line corridors completely covered, but it's impractical for me to do that myself and use all those aerial photos. So typically in rural areas I am just doing a few stations by themselves, but in the major urban areas, such as the five main centres, continuous corridor is what I do. Now it's inevitable that I will hit the layer or file size limits that I work to and such a segment of corridor will end up being split over more than one file. For example earlier on I did a continuous corridor from Newmarket to Waitakere in Auckland which is a total of 36 km that ended up going across three files in total. When you get to the edge of the current file what you do is copy the overlay that is at the edge and the underlying base layers to a new file. There are a few ways of doing this such as creating a layer group, putting the overlay and base layers relevant into it and then dragging it to a new canvas and then saving that to a new file. Another is to crop the original image down to the base layers that cover the area occupied by the edge overlay and then save to a copy of the original file. The reason for base layers being saved with the overlay is in the new file you can drag all the layers together (including the overlay) by using the layer linking in Gimp's layer list, into the grid in the new file so that the base layers are lined up properly in that grid, and then add the rest of the base layers and overlays as required to the file. This ensures you don't have to realign the overlay and also that the overlay is still exactly in the same position as it was in the original file, so that new overlays you add to the new file can be aligned to it and therefore the extracted tiles will line up perfectly in the GIS.

Part 6 is going to look at how to extract tiles for a GIS from the Gimp mosaic projects I have been creating, because that is the ultimate aim of this process.

NZ Rail Maps: Using Gimp To Georeference Retrolens Aerial Photos [4]: Getting Started With Overlaying

So now we can look at how we actually do the georeferencing. This happens as a multi part process, and explaining the steps gives you an understanding of how it all works.

The basic of georeferencing a historic aerial photo, is that by aligning it to a current aerial photo, of which we know the area that it covers, we can make the historic photo cover the same area as that current aerial photo. So this makes it easy to trace or otherwise reference features from the historic aerial photo and see a comparison with the current aerial photo. In NZ Rail Maps I use the historic photos to trace, for example, old closed branch lines, bypassed sections, and old yards and buildings at stations. This gives me a documentary record of historical features of a rail corridor and these features are displayed on the resulting map.

We do the georeferencing by using Gimp's transform tools to overlay the historical image over the top of the current image, or a grid of images (tiles), that cover the same area as the historical image. In practice we usually have multiple historic images in each Gimp project file, that cover multiple stations or yards, or a continuous length of corridor (most commonly in an urban area, where there can be lots of points of interest between stations, such as sidings). For example, in the five main centres, I have created Gimp projects that cover continuous corridors for a section each. The Auckland corridors are at present up to 10 km in each Gimp project each because they only have one historic layer. Christchurch corridors created so far are only a few km each because they have multiple historic layers from different eras. It doesn't take much distance working with originals of resolution 0.075 metres and NZR survey images of 1:4350 scale to start adding up to gigabytes of disk space in a Gimp project, when just one 4800x7200 background tile has 34 million pixels in it.

The overlaying is the tricky and slow part. We have to try to make as many features on both the current background image and the historical overlay image line up as we possibly can. For a variety of reasons this can be difficult. The major one is perspective distortion. When a photo is taken, with any type of camera, the subject is most accurately rendered where it is directly aligned to the centre of the camera's lens. At any other point of the image, the camera is effectively looking at that part of the subject from the side, and the resulting distortion gets worse the further you go out to the side of the lens. On a typical photo taken on the ground, there isn't really an issue with that. But when the image is taken for the purposes that an aerial photo is taken for, where we want to get an accurate representation of what is on the ground below, and given the distances incorporated in each photo, the distortion makes that difficult. That's a key reason why adjacent photos in a survey run are given a big overlap. You could get a really accurate aerial view by taking the centre most piece of each photo and joining it to the centre most piece of the next photo which would be easy with the amount of overlap on, for example, the NZR station surveys. I could actually do this, but so far, I have just used every second image. Because of the distortion, overlapping features that go above the ground like buildings are difficult. So are features that are on hillsides. The most challenging ones from a rail perspective are where a railway line on a raised embankment runs next to tracks that are on the flat, such as in the Dunedin railway yard where the main line is raised, and the technique for that is to have two copies of the historical image, one for the raised and one for the flat area, overlay each one just for the raised or flat features, and then overlap them using masking.

OK let's get started. Typically the layer list will show your Retrolens aerial photos at the top of the list, and the base imagery from Linz down the bottom of the list. So that your historic aerial photos are always drawn on top of the base images. Here is a typical Retrolens image that you've downloaded and imported into Gimp. First thing is to rectangle select it, and crop off the edges (shown here with the rectangle selection in place). 

Using Layer menu, Crop to Selection. Repeat this for every layer in the stack that is the same size. Change the selection rectangle if you have layers that are different sizes or have different borders. The borders being of no use to us.

Next step is to do the overlaying, one layer at a time. First, check that Snap To Grid is turned off in the View menu. It gets annoying when you are trying to precisely position a layer and Gimp is snapping it to a side of the grid. Also, there will still be a selection active from the cropping stage. Go to the Select menu and choose None to make sure there is no current selection.

Back in the day, with earlier versions of Gimp, you would have to use different tools to perform the overlay task. When I started using Gimp, I used the Scale tool to size the overlay, which doesn't have any preview capability, so I would just try scaling by trial and error until I got the correct size. As you can imagine this was very time consuming. Next step after that would be the Rotate tool, to get everything to line up. After a few months I discovered the Unified Transform tool. This is a new feature in Gimp 2.10 and it combines the function of both these tools and it offers a full preview, which lets you do the lining up. We recommend to make things easier, set up the keyboard shortcuts mentioned in Part 3 to enable you to quickly change preview opacity, canvas rotation, and zoom level.

Select the Unified Transform tool in the toolbox. Then select the first layer in the layer list and click on it. Your layer will be displayed as a preview frame, with a pivot (a circle with crosshairs inside it) in the centre and various handles at the corners and along the sides. Familiarise yourself in the documentation about how to use these handles. Now click somewhere within the layer preview and drag it to somewhere over the base imagery. Look for a feature that is common to both layers. Then drag the pivot from the centre of the preview to the common point on it. My favourites for common alignments are the ends of bridges, or a set of points, but often I use road features as well. It only has to be approximate at this initial stage. Once you have the pivot in place on the preview frame, set the opacity to zero and drag the entire preview frame so that the pivot sits over the same place on the base imagery. Then increase the opacity and confirm the pivot is properly located.

One caveat that is important when doing a unified transform. If you press the wrong key on the keyboard, it may start the transform before you are ready. This is an annoying feature in the current version of Gimp. Many's the time I have pressed the wrong key on the keyboard and the transform starts before I am ready.

In the above photo, we have positioned the preview of 2788-D-1 over the appropriate place on the base imagery, with the pivot very near the top centre, lined up on the end of a railway bridge.

Once the pivot is in place then decrease the preview opacity to about 50% and use the most convenient resize or skew handles to drag the preview to fit the underlying base imagery. To make this work off the pivot, go to the Tool Options dock and ensure that under "From pivot" the Scale option is checked. This makes sure everything else scales centered around the pivot when you drag those handles. Then resize the preview and try lining some other features up in both images. Use the opacity adjustments to compare the preview and base as much as needed and adjust the preview position and size to suit.

Youll see there are stretching (square) handles at the corners of the preview as well as in the middle of each side. But what if you need to have one in another place closer to where you are working with the canvas zoomed right up and the nearest one is out of the view? This is where the Readjust feature comes in. Press that button and you'll get a new frame right where you are viewing at the moment, instead of one around the full preview. Just note that the pivot moves too, to the centre of the new frame, and most likely you'll want to drag it back to where it was before.

 Somewhere along the way you'll want to recheck the pivot position to make sure the preview is properly lined up there as well. If you need to adjust the preview position at the pivot, then check if you need to redrag a side or corner handle to resize the preview and line the rest of it up again. I usually try having the pivot as close as possible to the top of the preview, while I am resizing at the bottom-most side. For the Retrolens layer 2788-D-1 I initially chose to line up on the railway bridge over Walton St. The bridge has been lengthened since 1975, because the road it crosses has become a  roundabout, which makes it tricky to line up on. I really need to check against something else on the ground that it will line up. And because it's raised above the ground, the tracks lower down might not line up so well. After a lot of checking, I realised I needed to move my pivot to a different location, because the southern end of the bridge was just not the right place, because the bridge was extended at that end. But the north end of the bridge was not right, either. Eventually I decided the best option was to line up on the next bridge further north (Water St) which meant adding an extra base tile at the top of the canvas and increasing canvas height.

Once you are sure you have the preview lined up as best you can over the base image, check that interpolation is set to the right setting in the Tool Options dock (I use Cubic), then click the Transform button and away it goes. When the Transform has completed, the next thing that is important to do is to lock the position of the transformed historic layer, so that it can't be accidentally dragged out of place on the canvas, because unlike the base tiles, it isn't aligned to the grid in any way.

The finished result after overlaying the first historical tile for Whangarei railway station, which is Survey 2788, run D, photo 1, over the Linz 2014 0.1m base imagery for the area. The main railway yard is about half covered by this historical image. Meaning that the join onto the next historic layer, D 3, would be highly visible as it would be across multiple tracks. One way to avoid this would be to use a different layer to start with (like maybe D 2 instead of D 1) that would push the edge out, and another way is to exploit the overlap itself and use a bit of masking to move the join to a different part of the overlap.

In this case I went ahead with D 3 without any masking and managed to get a nearly perfect overlap, as you can see below. But there have been plenty of times when it hasn't been quite so smooth.

An overlap is going to be less than smooth when there are rail wagons or service buildings across it, for the reasons I mentioned near the top of this post. And sometimes, there is simply too much distortion to make it ever work. In this case, it looks perfect across the rail yards, but further out, across suburbs and houses, there are bits of buildings cut off and misaligned, as shown below.

 

In this case, because the rail yard ended up near perfect, I didn't bother checking the result further out. The reality, as I mentioned above, is that quite often it is impossible to get a perfect alignment all the way across an aerial photo. I could have worked on improving the alignment towards the edges and then the rail yard would have been misaligned.

 

Gimp has got better over time, with the readjust feature (a result of a feature request I made on the Gimp forum) making it much easier to get perfect alignments across joins. I'll look at the challenges of doing this in Part 5.

NZ Rail Maps: Using Gimp To Georeference Retrolens Aerial Photos [3]: Getting Imagery Into Gimp

So let's get this rolling. First thing I need is to download the layers from Linz Data Service. Which in this case is at the following URL: https://data.linz.govt.nz/layer/53399-northland-01m-urban-aerial-photos-2014-2015/

The layer is called Northland 0.1m Urban Aerial Photos (2014-2015) and covers the stations of Whangarei, Portland and Kamo. It also covers the port area of Whangarei, which is good, because the NZR station surveys cover all the extra bits. I chose to download the entire layer, which is 4.8 GB. I find it preferable to use a download manager when handling downloads of this size. Since I use the Firefox Developer Edition browser, the best download manager is uGet. Although a bit fiddly to set up, it works well, especially since I have my computer set up to save my profile whenever I need to reinstall the operating system.

I'll also note where I get the NZR survey images from. I'll go to the Retrolens site and zoom it in on the Whangarei rail yards. Just typing in "Whangarei" and zooming in on that word on the Retrolens interface is enough in this case, because it conveniently brings us right to the railway station. Often though this is not the case and you'll find Retrolens thinks the centre of an area is way out in the countryside in the case of smaller townships and localities. It pays basically to have a look at another map, find a street name close to the railway station, and put that into Retrolens to make it easier to find the exact part of the map that I want to look up the aerial photos for.

The NZR survey is No.2788, with six different runs, from A to F, covering all the way down from Kamo to the Whangarei Port, and all of it was captured in 1975 at a scale of 1:4325. If I start from the rail yard area, on the right hand side of the Retrolens web site, I can see the aerial photos for an area, and I'll be looking for the ones that have a date of 5/01/1975. Clicking on one of these (I only need one of them at the moment, and it doesn't matter which one) will come up with more details, at which time I can confirm the survey number SN2788, the run number and photo number (the run number is going to be between A and F in this case but it doesn't really matter that much) and the scale is 4325. This scale in particular, or 4350, is virtually a unique signature of the NZR station surveys that can help you to identify them on sight, but in case you are unsure, view the labels in the margin to see if it says "NZR" or "RAIL" anywhere. Some of the aerials I have downloaded around Auckland and Wellington of the rail yards there are an even larger scale than 4325 or 4350.

Now here's a couple of little tricks to download all the images I need from this survey. First of all I'll click the Large button and bring up this image in the Large size, which is the highest resolution I can download from Retrolens. Now in this case the URL of this image is as below (so you can save yourself a bit of work by just skipping the above steps and going straight to this web page: 

https://files.interpret.co.nz/Retrolens/Imagery/SN2788/Crown_2788_D_3/High.jpg

Now, the fact is all NZR surveys start at Run A and photo 1. So I could go through the Retrolens site to find every single one of the images and get them. Or I could just change the letter D in the above URL in the address bar on the browser to the letter A and the number 3 to the number 1 and then press enter with the new URL to get 2788 A 1. And that is exactly what I do. This is a much faster way of downloading all the images from Retrolens. And when I hit a 404 error at the end of run A, I can start with run B photo 1, knowing in this case they go up to run F.

The other trick is usually you won't need every image from the survey, because they overlap enough that in 99.9% of cases with an NZR station or corridor survey, there is so much overlap that you only need to use every second image. Generally, I download every image, and then decide which ones to import into Gimp, which may be A 1, A 3, etc but sometimes it can be A 2, A 4 etc. Obviously choosing to use all the images just makes a lot more work for you in Gimp, but it also is going to increase significantly the amount of data that is stored in the file and make it bigger than it needs to be. 

The amount of disk space I have used to create mosaics is such a challenge for me to manage that I am sooner or later going to have to implement an economising strategy of actually deleting project files, simply because I can recreate them quickly in future with the extracted mosaic tiles, at the cost of a small loss of quality. At the moment my computer has more than 1.5 TB of Gimp mosaic originals stored on it, and I have decided I am simply not going to upgrade it to 4 TB disks, so managing the use of the 2 TB disks is what I am actually doing in practice. The extracted mosaic tiles in Jpeg format use much less space than Gimp does. That's understandable because the Jpeg format is lossy one-way compression, whereas Gimp is using lossless two-way compression that can recover all of the original data. If Gimp had some way of saving an archive copy of a project so that it was lossy compressed that would save a lot of disk space, and since that isn't practicable, the alternative is to split the files into lossy extracted mosaic tiles, which save a lot of disk space, and delete the project files. So far I haven't deleted any project files but it's inevitable I will have to do this sooner or later.

Anyway back to the maps. So we want to lay out our background tiles downloaded from LDS in the Gimp canvas. First thing is to set up the Gimp grid to 4800 pixels wide x 7200 pixels high. Next thing is to view the grid, and then turn on Snap to Grid. Then with a blank canvas, or no canvas because it will create one if there is nothing open, use Open as Layers to load the layers JPEG files from LDS. If you want them to appear in the "correct" order in the layer list, set the filename sort order in the dialog to Filename descending, this is because of the order Gimp adds the filenames to the layer list. They will all be displayed stacked on top of each other in the middle of the canvas and each one will be a separate layer. 

Then you need to increase the size of the canvas, to a multiple of 4800 pixels width and a multiple of 7200 pixels high. Once you have that, then start dragging the tiles into their place in the grid. Select the first layer in the list (right at the top) and then drag them one at a time. This is where Snap to Grid comes in handy. The filenames are based on a column reference followed by a row reference. The filenames use the digits 0 to 9 in sequence, followed by capital letters, then increment the next column of digits and back to 0 again. E.g. 50, 59, 5B, 5Z, 60 etc. But Gimp's filename dialog gets this order wrong for some reason (at least in Linux it does). There are a small number of letters missing from filenames, notably the five vowels. The tiles are usually all 4800x7200 wide, which is why I set up the grid to these settings. It pays to download a bigger area than you actually need, because LDS has this nasty habit of serving you fractional tiles around the edges of your crop area, which can be tricky to work with.

Next step: load the Retrolens images using the same Open as Layers command. They will also be stacked in the middle of the canvas. You don't need to do anything with them just yet so don't drag them anywhere as the next step (which will be in the next post) is most conveniently done to all of them whilst they are still stacked together. Right now is probably a great time to save your Gimp project. Have a cup of tea or a coffee and then start reading Part 4 of this blog series. Here we'll actually prepare the images and start overlaying them.

NZ Rail Maps: Using Gimp To Georeference Retrolens Aerial Photos [2]: Preliminary Considerations

So I aim to write one post every day about this topic (this post was supposed to be finished last evening but has stretched through until this morning) until all the parts are completed, there are probably three or four parts. This is easy because I am doing georeferencing at the moment on a particular volume of the NZ Rail Maps project.

Firstly, the choice of GIS project CRS. This relates more particularly to the GIS and strictly speaking is not relevant to the georeferencing task but is worth considering if your GIS project intends to use the Linz WMTS layers at any stage.

When you are creating maps in the GIS you need to consider the CRS of the project and this is very important if you are using Linz's WMTS live aerial photo layers. I use these layers for tracing all the current features of the maps and have been doing so for only the past year. Prior to that, I downloaded the individual map tiles for every rail corridor from the LDS website, but now I only need to download what I actually use for georeferencing.

Linz only supplies its WMTS layers in two CRSs, which are EPSG:2193 (NZTM2000) and EPSG:3857 (WGS84). If your project is not in one of these CRSs, using WMTS is only possible if the GIS can transform the CRS on the fly, and this can be fraught with challenges. In my case, I selected EPSG:3857 for my maps some years ago because of its compatibility with Google Earth KML. Even though I no longer import KML files into my maps in order to ensure there will be no licensing challenges in the project, I have stayed with EPSG:3857 as the project CRS.

If your project is not able to use one of the available CRS that is supported directly by Linz WMTS then downloading all of the tiles you need to cover your corridors and importing them into the GIS to provide the contemporary base imagery for the area being mapped, if this is what you are using, is the alternative. You will need some of this imagery as the georeference base for Gimp in any case. The difference with using it in a GIS and using it in Gimp is that the GIS will be able to load JPEG or TIFF files directly without expanding them out of their compressed state. However, when you import the tiles into Gimp, it breaks everything into separate pixels without the benefit of JPEG compression, and as far as I can tell, the resulting file size does not have the same level of compression that is possible in JPEG. I am not totally sure of this, and I know Gimp has its own compression built in, but I have to be able to account for a small volume of tiles becoming a big volume of Gimp project data.

Secondly it is very important to ensure Gimp is configured with adequate tile cache as the transform tools for handling large images will use up a lot of swap space. My computer has 32 GB of system RAM and 168 GB of SSD based swap space that Gimp can use. At the OS level, the SSD is configured as a Linux swap partition, and then in Gimp, I set the tile cache to 200 GB and it uses the 32 GB of RAM and 168 GB of the swap partition (which in practice is larger than this to allow for other system or application use). As a rule of thumb you could expect to be able to handle a Gimp project with a filesize up to 30 GB with this amount of tile cache. My experience has been not to push larger file sizes (generally I go with no more than 100 layers in my projects as another rule of thumb) since Gimp is more likely to crash during a save operation resulting in data loss. I have also experimented with another computer that has just as much SSD swap available but only 8 GB of RAM and this computer is far less able to handle 20 GB or larger project files and will often crash trying to load them, even though it is nowhere near using up all the swap space. It seems that both the system RAM and SSD swap are important but the amount of RAM makes a bigger difference to performance and stability than the amount of swap.

Thirdly, when you are doing transforms in Gimp you'll want to consider the best interpolation algorithm. Interpolation is what happens when you increase or decrease the size of a graphical layer. This is because the existing pixels are either being increased or decreased in number, and information from the existing pixels has to be sampled to make the new pixels. If you do a scale that is a whole number multiple of the existing size then each pixel just gets cut into the appropriate number of pieces needed, for example scaling by 2x each size means each pixel becomes four new pixels. But in real life we work with all sorts of scale multipliers and this process becomes a lot more complex. The interpolation algorithms are designed for various capabilities and Gimp has a few different ones supplied. When I started doing this stuff I tried with the most intensive and therefore slowest interpolations, LoHalo and NoHalo, but after a while I found that using Cubic gave just as good a quality with a much faster transformation speed and that has been what I have used ever since. 

It's worth noting that I found the scaling process counter-intuitive until I worked it out in my head after a while. To go from a pixel size of 0.3 metres to 0.15 metres, you scale the layer up 200% each side. I had always thought you would shrink each pixel down to half its previous axis by scaling at 50%. However, what you are actually doing at that point is creating new pixels with 50% of the previous information in them and so actually losing resolution rather than increasing it. When you are scaling up 200%, you are doubling the number of pixels so that each pixel contains less information and this is doubling the resolution. You end up with a layer that contains four times as many pixels, but when you tell the GIS to draw them at half the original size, the layer occupies the same area of the screen as the original.

The key reason we want to scale 0.3 m to 0.15 m, or perhaps 0.4 m to 0.1 m, or 0.75 m to 0.15 m, is that we want to avoid downsampling our historical layers too much. I found this out early in the piece with one project having to use 0.3 m background, putting a 1:5500 corridor survey image onto it, scaling that down in the transform too much, and losing too much detail. So these considerations have been worked out by trial and error. I now decide that base imagery where the size of a pixel is more than 0.15 m is a bit too big. I am still working with a lot of 0.3 m base imagery around NZ and scaling it to 0.15 m is the best way to deal with it. The result is a larger tile size and it is easier just to go with those larger tiles than to try dividing up the new tile into four tiles of the original size, because they have to be given a new naming scheme, and because you have to work out the coordinates of the top left corner of something that is spaced halfway along one side of the original tile, which makes for extra work. I tried this for a while but eventually I came around to understanding that a bigger tile, that is actually drawn by the GIS at the same physical area as the original by telling it the pixels are half the original size, is not enough of a deal to warrant getting pedantic about the size of the new tiles.

There is one more thing to think about and that is the merits, if your station doesn't fit within one tile, of making the mosaic tiles that you import into the GIS, cover more than one of the original tiles footprint. This is just done for the sake of convenience reducing the number of files you have to copy or store. Each tile (JPEG is the format I use) apart from the graphical file which takes up megabytes on the disk, has three sidecar files, which each take at most a few kB. You are not really saving a lot of disk space by reducing the number of files, but if you are creating the sidecar files for your mosaic tiles by hand, you'll appreciate having a smaller number of steps (I wrote a script to copy the sidecars). The main downside is the GIS will take longer to refresh larger tile sizes. Gimp has some sort of limit to its export functionality or plugin that will limit the actual size of your exported tile. When you combine multiple tiles into one bigger one, you do have to create each new tile as a rectangle and look at how a bigger area can be divided into multiple pieces. You also have to know the top left corner coordinates of the new big tile and the easiest way to do that is to put the top left corner of the new tile in the same place as one of the background tiles you got from LDS. Well, I'll try to explain these little details more as we work through the practical steps.

One more thing is to make Gimp easy for you to use. For me, that is by setting up the right shortcut keys so that I can make things happen really quickly. The key ones for me are:

• Zoom level - this is on the View menu, "Zoom In" and "Zoom Out" in the keyboard shortcuts interface. I have these assigned to + and - on the numeric keypad, which just allows a single key to be used to zoom. I have also experimented with using the Ctrl key with the - and = keys on the main keyboard.
• Canvas rotation - this comes from the Flip and Rotate submenu on the View menu. In the keyboard shortcuts interface, within the View menu section, the actions that I find useful are "Rotate 15 degrees clockwise", "Rotate 15 degrees counter-clockwise" and "Reset Flip & Rotate". These are assigned to Ctrl+., Ctrl+, and Ctrl+/ in my case.
• Preview opacity - the transform tools we are going to use all have previews which we use to line up the overlays with, before pushing a button to carry out the actual transform. We have to be able to compare the background and historical images at various levels of opacity and it's convenient to be able to go quickly up or down the levels. This is found in the Tools section of the keyboard shortcuts interface where it is programmed with the actions "Tool's Opacity: Increase by 1" and "Tool's Opacity: Decrease By 1". In practice, the steps are actually 10%, which is actually quite satisfactory, and there are also actions for 10, which I would use if it hadn't turned out that the 1 action actually turns out to be 10. I set these to be the , and . keys. The reason for using these keys is they look like left and right arrows because they are the < and > keys when shifted.

So next time we will look at actually doing the steps with Gimp, based on a real situation, this is creating a historical mosaic for Whangarei that is being done right now for NZ Rail Maps. The blog post will be written alongside the actual steps that are being done to create the historical mosaic to cover this station. This is part of Volume 1 of NZ Rail Maps that is being updated at the present time. The next post will appear later today because the schedule is actually one post a day and this post is being finished half a day late but the schedule will be caught up.

NZ Rail Maps: Using Gimp To Georeference Retrolens Aerial Photos [1]: Introduction

OK so here is another post in the NZ Rail Maps category and this post will explain how I carry out the georeferencing of the historical Retrolens aerial photos to produce the maps for the NZ Rail Maps project. This is a very rewarding aspect of the project but it does take a lot of work to complete, and whilst it is my desire to have the maps produced for the majority of stations in NZ being documented historically, it may not be practical to achieve this for more than a few hundred locations in practice.

This is the first of several posts that will outline the various steps needed for this particular process and this particular post introduces the process,

The first question to be resolved is whether to use this system at all. There have been in QGIS software and possibly other GIS packages, tools or plugins provided that enable georeferencing directly within the GIS. The plugin for QGIS is poorly designed in my view and I had considerable difficulty in making it work. It is not fully visual in that it is not possible to preview the final result, so it does have limitations. I had previously experimented with the image overlay capability in Google Earth but found that it was limited in its design.

So using Gimp has been arrived at over a period of time, due to its being best supported. The main downside of using Gimp is the large files that are generated by it. These vary, but for example to cover about 10 km of track distance with one generation of historical aerial imagery can result in about a 20 GB disk file in order to get a desirable resolution for picking up all the details needed for trackwork. This has largely been arrived at due to the source resolutions available. NZR had their own surveys done for major stations and also major corridors, and these have scales varying from  1:3000 up to 1:5500, which gives enough detail to be able to count individual sleepers. Most station surveys are at 1:4350 approximately but there are a few around Auckland and Wellington that go as high as 1:3000. Corridor surveys to date have always been 1:5500 but there are some continuous station surveys covering 10 or 20 km of a corridor around Auckland and Wellington at the larger scales mentioned.

Where NZR surveys are unavailable then the next best choice generally will be either for some larger urban areas, urban surveys done for a local authority or some other government department at various scales, or where the rail line passes close to a highway corridor, then the highway surveys generally around 1:8000 will be adequate. Generally we are looking for a scale of 1:10000 minimum to get enough detail to pick out individual tracks in a station yard. Sometimes I use smaller scale surveys if there is nothing else available but they are very fuzzy looking to print as a map background and therefore have limited value.

In order to get the best out of surveys that are 1:10000 or larger scale we must also use base aerials that are able to match this scale. Base aerials are downloaded from the Linz data service and generally I am using the latest release for any particular area but I might also have the ability to use earlier versions through WMTS when drawing the maps in the GIS. However as we are talking about static imagery as a background in Gimp, I am going to download the latest version and for an area where I have historical imagery with a scale of 1:10000 or greater, I want to be using backgrounds that have a pixel size not exceeding 0.15 metres a side. But if there is only 0.3 metre imagery available for the area then scaling this to the size needed to get the smaller sized pixels is what has to be done. For most major urban areas there is generally imagery at 0.125, 0.1 or 0.075 metres resolution available and no scaling is needed, but generally outside these regions, 0.3 metre imagery will have to be scaled up to double in each dimension, which Gimp handles by doubling the pixels in each direction, effectively cutting each existing pixel into four pieces. It is then necessary when importing these new larger tiles into Qgis to change the settings in the world file to reflect the new pixel size correctly so that the tiles fit into the original tile grid. This will be covered in a future part of the series.

So there are basically two important steps to be determined first of all when embarking on this process. The most significant of these is to ascertain what is available from Retrolens or any other source for the historical imagery, and then from there, the second step is to get suitable base imagery from Linz Data Services that can provide the georeference base for the historical imagery. The next step covered in Part 2 is to prepare both the base and historical imagery for georeferencing.

New "Readjust" feature in Gimp Unified Transform (NZRM Aerial Mosaics)

Unified Transform is a very useful capability in Gimp since about 2.10 that combines a number of transformation tools together, such as Scale, Rotate and Shear. I have used this tool extensively when creating aerial map mosaics for the NZ Rail Maps project.

To create an aerial map mosaic, I start with downloading the current aerial imagery layer for the part that I want to create a mosaic from, and then extract the map tiles from it (each is 4800x7200 although if at the edges of the layer they can be smaller) and lay them out in Gimp. Using a grid size of 4700x7200 and snapping to grid makes this quick and easy. If the tiles need to be scaled to a multiple (usually this is x2 in each direction) then use the Scale Layer function to achieve this with Cubic interpolation.

Then import the historical tiles, and the first thing you need to do with downloads from Retrolens is to crop out the borders. Apart from the copyright banner along one border there is also the negative mounting frames, which I presume are cardboard like slides. Sometimes I need to get so close to an edge that I'll be forced to include part of the border and then use masking to hide it. When I first started doing these mosaics, I didn't bother cropping the borders and instead I just masked them off every time.

The next step is to overlay the historical tiles on to the current background tiles. This is where the Unified Transform tool comes into play. The first thing to do is to find a common point near one of the overlay's borders. Then drag the pivot from the centre of the overlay to that point on the overlay, and then set the overlay's opacity to zero. Then drag the overlay until the pivot is in the same place on the base layer.

Then ensuring that the Constrain and From Pivot options are set properly (Scale option turned on), and then scale and rotate the overlay until the rest of it lines up with the base. I have set keyboard shortcuts to control the opacity without having to set this with the mouse (this is Tool Opacity in the keyboard shortcuts section) and also canvas rotation with more shortcuts so I can quickly turn the canvas to make the view work better for me.

The new "Readjust" feature addresses a fairly common issue I have with lining up the overlay with the base where the part I want to match up is nowhere near the edge or the scale handles. It can be somewhere near the middle, for example, and I need to be able to rotate or scale without having to zoom out to get the edge or scale handle in view and then lose sight of what I am trying to line up. Clicking the "Readjust" button creates a new transform control frame that fits in the current view in the window instead of being aligned to the edges of the image, which means you can get a set of edges and adjustment handles anywhere on the overlay. This is particularly useful with some of the quite big images I am working with for ensuring there is perfect alignment especially where two overlays overlap each other and will make a big difference in creating these mosaics. The only issue I have found with Readjust is that it moves the pivot when used, and the option for locking the pivot doesn't seem to achieve this, so I have to drag the pivot back to where it was before but it still seems to be OK. This is likely to make a huge difference in creating more accurate alignments with mosaics than I have been able to achieve in the past.

Normally for Unified Transform these days I will use Cubic interpolation for everything. I used to use LoHalo but I doubt that any extra quality it might have produced was significant so Cubic seems to have little effect on the final quality.

NZ Rail Maps Technical Review 2020-04-18

So we are now at Qgis version 3.10.4 for the NZ Rail Maps map production task. Back a way we had a lot of trouble with Qgis with WMTS layers which were unusable at that time. Since then however we haven't been able to replicate the problem but we suspect that their code has bugs in WMTS handling and possibly wasn't dealing with a temporary issue with the WMTS feed. We had to go back to 3.4.4 running on a Stretch VM to be able to continue editing our projects until we decided to have another go with the latest LTR and it seems to be OK for now.

Also looking back a few posts we see a reference to saving mosaic tiles as a group to reduce the overall number of them. This is one of several possible solutions to a number of situations. One of these situations is when we have to upscale Linz imagery that is at 0.3 m or 0.4 m resolution in order to match the kind of scale that we get in images we download from Retrolens. From experience, a pixel height/width no greater than 0.15 m is needed to be able to match 1:4300 or 1:5500 scale and get the full resolution of the Retrolens images when we export out the mosaic tiles to be used in Qgis.

The question then to be solved was how to change that into 4800x7200 tiles because each of the original base images is now several times larger in each dimension. For example these days we will commonly scale a 0.3 m resolution tile by 200% in each direction which makes the new pixel size 0.1 m. We did scale 0.4 m tiles to 0.1 m at one time but for the rare times we use them will try to get away with 0.2 m by again scaling the base 200% both ways. Obviously the original 4800x7200 tile now becomes four if we have doubled in both directions.

One way we attempted to resolve this was to create a special grid of 4800x7200 tiles by adding a part suffix to the original filename so that XXXX1-YYYY2 would become XXXX1.2-YYYY1.3 where the .1 or .3 suffix indicated a position in a particular grid. Special grid templates would be used in Gimp to help split the image into the correct grid segments. This was tried with Northland aerial mosaics in particular but has now been ditched in favour of simply creating a larger mosaic tile made up of multiple 4800x7200 tiles. This is much easier than running the special script to create all the special world files for each sub segment that would need to calculate the top left coordinates to tell Qgis where to draw the tile.

The great advantage of creating one big mosaic tile is that only one world file is needed and it only needs to specify the top left coordinates of the big tile. So you can make that tile cover as big an area as you need as long as you can replicate the top left coordinates and the pixel size in the world file. Typically we are just taking the world file for the top left tile which is based on the original top left tile even though it has been scaled up to a new size, and just changing the pixel resolution parameters in there to reflect the new resolution.

We have been looking at the Northland mosaics again this week and have just been removing the grids and exporting the mosaics again as one big mosaic tile for each station because we never got around to creating the grid segment world files which was very complex and hard to do so this is getting these mosaics back into our Volume 1 Qgis project so they can be put to work.

Rolling back a Flatpak application to a previous version

Today on my main graphics editing PC, I decided to update Gimp from 2.10.16 to 2.10.18. Which should have been OK. Except that it wasn't. I have spent all day dealing with Gimp crashing during saves and trying to work around with ever smaller files to be saved. It did crash last night as well but crashes during saves like that are normally relatively infrequent, and in this case I observed that the system resources were almost completely exhausted, which hasn't been the case at any time today - one save was only half of the file, another was only 3%, and in neither case were the system resources anywhere near excluded.

As the evening approached I have decided enough is enough, let's try to roll back Gimp to the previous version. Well, as it turns out, provided a previous version is available on the server, this is certainly possible.

Flatpak can interrogate a repository for you to see what versions are available by using the following command:

flatpak remote-info --log flathub org.gimp.GIMP

Obviously the last parameter is the path to the repository which for Gimp is the one shown in the example above.

What will come out is a list of commits, for example here is some of the output from a command issued today:

Commit: 62578b6f3b1262ceab9fc5d8a93d1b034cda51636a88bce9de6036ae6c5f038c
   Subject: Release GIMP 2.10.18. (bbbf1b4e)
      Date: 2020-02-25 00:53:47 +0000

    Commit: 57036c515dd653f828b8c6574e4b56fca6c29b6ac04f5e84c328cc6fc5e7987f
   Subject: Release GIMP 2.10.16. (796cb329)
      Date: 2020-02-18 22:43:45 +0000

    Commit: 798d8cd8a6a2ccf1bf21c0f5e9749dd4d97ecd511a3af1922d6fedf6c26e3a7c
   Subject: Release GIMP 2.10.12! (1366fa63)
      Date: 2019-06-13 02:25:52 +0000

In this case I have decided to roll back to 2.10.12 because of being uncertain about whether I had 2.10.12 or 2.10.16 previously installed (there was no release for 2.10.14 for some reason).

To achieve this the following command is required:

flatpak update --commit=798d8cd8a6a2ccf1bf21c0f5e9749dd4d97ecd511a3af1922d6fedf6c26e3a7c 
  org.gimp.GIMP

Note the slightly erroneous command "update" even though you are downgrading in this instance.

One blessing at least is that flatpak, at least on Debian, will not automatically update when you use apt upgrade. So we can use apt to update everything else and flatpak applications will not be affected. I suppose Synaptic or KDE's software manager might one day try to update flatpaks itself - I have no idea if that is the plan the developers have in mind. Every time I see the message there are updates ready I just go to the command window and use apt to install the updates.

However, my other computer has 2.10.18 installed and hasn't been having crashes when I edited large Gimp projects on it so I am not sure what is going on with my graphics computer but I have wasted too much time in the last couple of days to want to continue with 2.10.18 at present. Both computers are fairly up to date with 15 new packages released, most of them being a new Qgis build, so it isn't really obvious right now why the systems appear to behave differently, unless it is this particular file.

Subsequent testing however has shown there is nothing wrong with Gimp and the issue is down to the particular characteristics of this file. Gimp does seem to have internal limits as to the amount of file data it can deal with and it is rare for me to be able to edit a project larger than about 30 GB because they tend to crash on save, probably due to the additional resources needed to compress the file before saving it. There doesn't seem to be any issue with canvas size as such - as one would expect a canvas does not require every pixel to be saved. The software simply needs to save the data in each layer along with its coordinates within the canvas. So I have been able to work with very large canvases of over 10 Gpx without difficulty - as long as they are sparsely populated with graphics layers.

As part of my testing I reinstalled the graphics pc completely which was very easy with only a few applications installed on this special purpose computer, but especially because of being able to reuse the previous home drive which saves a great deal on reinstallation because all of the user settings are kept. The reality is a new installation can be up and running in an hour. I also noted that Debian 10 ships with its own Gimp 2.10 package pre installed (2.10.8) but for some reason this version could not open anything so I have uninstalled it and only use the Flatpak version.