As recent posts have outlined I am using a new grid system for the NZ Rail Maps mosaics to standardise on the use of 4800x7200 tiles for historical aerial images, because Qgis limitations means it tends to run out of memory exponentially more rapidly and be much slower to update the screen if larger sizes are used. I have standardised on using 0.1 metre resolution aerial imagery when making use of official New Zealand Railways survey scans from Retrolens, in order to take advantage of the large scale (high resolution) of these survey images, but this means a lot more work to get the best out of them by ensuring we use high resolution Linz imagery, because Gimp uses a lot more resources obviously working with the higher resolution.
This means if I have 0.4 metre resolution Linz tiles they have to be increased to 400% of their original size to give me a tile that matches native 0.1 metre resolution tiles. And then the next step is to make up new tile names using a grid of 4x4, so that that scaled up tile now gets divided into 16 tiles at 4800x7200 pixels. The earlier posts talked about how to number the grids and to begin with I was actually splitting the large tile into 16 smaller ones using the Guides and Guillotine tools and then positioning them all correctly. I have now got to the point where instead of making 16 layers I will keep one layer and only split them at extraction (export) time. Instead I will have background grid tiles that are named and can be saved in a template, and when I export the tile it will get named according to the grid reference which will be visible in the layers list.
The last post on this blog, which was not tagged to NZ Rail Maps, talked about optimising Gimp to use memory and Linux swap space together which you do by treating them as a single resource, in this case totalling 132 GB, so that I can set Gimp to use 66 GB as tile cache (half the total) before it goes to using the Home drive for its own swap, and there is still memory/system swap space left for other applications. I have been monitoring the usage of these resources to see how well this new optimisation works with the largest mosaic ever. It ended up having to be a very large file because for one station it took four of the original tiles to cover the station yard due to a skew alignment, being north-east to south-west roughly centred at the intersection of the four tiles, which doesn't usually happen and most yards need at most two tiles at that resolution and some only one. So I was left with unavoidably having to provide for four 0.4 metre resolution tiles in the layout which once they are scaled up uses a lot of disk and memory space to work with.
So I have tracked the resource usage while working on this very large file and whilst I did eventually exceed the 66 GB cache setting for Gimp, it was almost not happening until I had finished creating the image. As usual the main memory resource usage is the ten original tiles that have been scaled up to 400% in both directions. The grid base tiles and the Retrolens images only added relatively small amounts of resource usage. The final version of the file saved to 14.8 GiB of disk space and the system resource usage at that time was the entire Gimp Cache being used and a small amount of Gimp Swap. On the system resource meter the usage overall for Debian is 32 GiB of memory and around 50 GiB of swap. Gimp is the main application running but it also has 8 GiB allocated to the undo buffer. Since there is still swap available, setting Gimp to have access to 100 GiB for its Cache setting is possible for handling still larger images. The image size is 38400x57600 pixels (2.2 GP). The system has been far easier to use than has ever been the case before with an image of that size without massive paging to the hard disk which is very slow and noisy. Definitely the noise I hate the most with my computer.
Given that about two months ago I spent quite a lot on speeding up this system with a new board, CPU and extra 16 GB RAM, which cost around $500, I have come to the conclusion that the most cost effective solution for increasing performance in future is to put in a larger SSD, which will be around $60 for 240 GB (about 200 GiB) or maybe $120 for 480 GB (about 400 GiB) that will double or quadruple the Linux swap partition size for really quite a low price. Admittedly SSD is a bit slower than main RAM, but silent in operation and faster than the regular HDD. In the meantime I will set Gimp to use more of the Linux swap partition which is currently 100 GB but will have to carefully watch it to make sure there is no risk of a system crash if other applications are running and use the swap. It will be interesting to see how that works out with different images and map drawing tasks this year.
