General update 2019-02-15

Last week I had some issues with mainpc resulting in it running out of /tmp space which of course is caused or impacted by running out of space on the install volume and caused a number of user files in /home I was working on to become corrupted. I haven't looked at the bigger question of how we can prevent issues like this from bringing down the whole system or how it actually happened. Instead I just chose to reinstall the system from scratch.

There was a delay to getting this done because issues with my internet here created problems for the Debian installer trying to connect to the local mirrors to download install files so in fact mainpc was down for something like 5 days but with the other computers available I was able to keep working to an extent with the main constraint being unable to access any files over the network shares from mainpc.

Key objectives of the reinstall include:

• Recover the entire free space on the SSD for swap file. Previously an attempt was made to allocate half of the swap to Gimp specific temporary/cache allocation. This proved difficult to implement so I decided to revert the full space (99.9 GB) back to swap as Gimp does use this space to cache as needed.
• Allocate /mnt to a separate partitition on the SSD like done to mediapc to ensure there is no possibility of a mount failure allowing files to be written to the SSD instead of a mounted volume on another disk filling up the install volume.
• Allocate /tmp to a separate partition to ensure there is no possibility of a filling up of the rest of the root volume causing a disk space crisis for /tmp or vice versa.

With the system back up and running having rejoined to /home on the RAID-1 array and therefore reusing my existing user profile I gave attention to software reinstallation. The previous installation used the Debian packages for Thunderbird which installed the Daily channel build and caused problems with extensions like Lightning being incompatible and therefore unavailable. I chose this time to download Thunderbird directly from their site and install it directly into my user profile to save space on future reinstalls and ensure it is the Release channel which allows me to use Lightning again.

mediapc has had a new R/W share created in samba to give access to the Video Archive folder which contains a volume of material not in the media share and therefore I can work on the material in that folder from mainpc for convenience. mediapc is about to get various video editors installed as mentioned in a previous post some time ago.

So questions still remain over managing disk space in some of the areas of a typical Linux install and maybe I can address these.

I had a look at rearranging screens because the arrangement was less than optimum for mediapc. Basically at eye level I have four screens: two for mainpc and one each for serverpc and mediapc. Three of those are vertical to save space and keep them as close to me as possible without having to turn around too much. The problem is for video editing on mediapc you want it to be a horizontal aspect ratio for best use of space. However because mainpc and serverpc are used the most the screen layout suits them best and is not really set up for anything else so after trialling an alternative layout I decided to leave things as they are, except for the two 3 metre DVI cables I purchased which will now not be needed but still have to be paid for. At the end of the day the layout really only can be optimised for the two most used computers so video editing will just have to be improvised or I remote to the computer doing the editing from mainpc.

NZ Rail Maps: Scaling up map mosaics & creating new tile grids

One of the issues that comes up with doing the map mosaics for NZ Rail Maps is that the resolution of Retrolens aerial imagery can often be very high, when using the official NZR survey images at the scale that they appear on Retrolens. A typical scale is only 1:4300 which gives us a lot of detail on the ground. However to mosaic that over the Linz stuff can be problematic when some of the latter comes in at a much smaller scale. This week I was working with source imagery of 0.5 metres (pretty unusual for Linz aerials from the last five years) and the downscaling of the NZR stuff to match this would result in way too much detail being lost.

So the obvious solution is to go the other way and upscale the Linz stuff but the problem is you end up with a massive tile size that the current version of Qgis can't handle very well. As an example, the railway station at Tokoroa is covered by one single tile at 4800x7200 pixels with each pixel measuring 0.5 metres on each side. This is called 92VJD-92MCL in the EPSG:3857 grid that Linz supply the imagery in. Scaling that up in Gimp to 0.1 metre (500%) gives us a tile that is now 24kx36k pixels which means 25 times in total area and even the smallest export will be over 200 MB which Qgis finds very hard to handle. 

So the solution arrived at here was to split the scaled up tile into 25 tiles at the original dimensions. This has all been done by hand this first time, laboriously cutting out the grid sections and pasting them one at a time as individual layers into a new image, but apparently it should be possible to set up some guides in Gimp and use the guillotine tool to automatically slice the original into pieces (hopefully with each piece as a separate layer).

The next step is to come up with a new grid where a set of numbers get added to the original column and row references of the original tile. 92VJD refers to a column name and 92MCL refers to a row name. So I cobbled together a spreadsheet and obtained the top left coordinates from various world files in a table in the sheet so that I could work out the height and width of the original tile and then divide these numbers into fives so that I can work out the top left coordinates to put into our new world files for the new grid. So in accordance with the way I have done it in the past, 92VJD-92MCL will become 92VJDx5.1-92MCLx5.1 and this decipers as, x5 means that we have scaled the original tile up five times, and .1 means this tile is a piece of the full tile. As we have a 5x5 grid of tiles then the column and row numbers range from x5.1 to x5.5. In other words the part that is actually a row or column number is the part after the decimal point.

Then with the spreadsheet, it did the calculations to work out the top left coordinates to put in the copies of the world files that were produced for each new tile, and after a bit of trial and error, it worked just fine. Qgis is now using a much more reasonable amount of memory because it only needs to load five tiles (1/5 of the original tile size) of around 15 MB each instead of the very large original of over 200 MB and the refresh is much faster and Qgis takes much longer to run out of memory. So that has been worthwhile but it took ages to set up so I need to work out ways to speed it up and that guillotine tool, if I can make it work, could speed up the slicing a lot, and a Python script or the like to work out the new coordinates and maybe even write the new world files directly so as to save a lot of manual editing and copying files.

I have a similar thing happening in respect of the map tiles for Waipara where I have simply added new tiles that were not in the original supplied imagery grid because of covering some original areas to the side of the supplied grid and so I have to again calculate manually the correct coordinates for the new tiles. But this case is not one of having scaled the original images and I did not have to create the grid like I did in the example mentioned above. So that one will not be anything like as much work.

UPDATE: For the time being I have created a Gimp template that creates a grid of up to 5x5 4800x7200 layer tiles. When a new mosaic project is created these tiles are in the background and enable the grid numbering system to be followed for exported tiles out of the full image when the full image is cropped to individual tiles. This does not directly address the purpose of scripting which is to auto generate World files for individual tiles that are created out of a full mosaic. The scripting task is still being considered at this stage, meanwhile the World Slice Calculations spreadsheet has been in use to calculate the coordinates of each tile.

How to auto start x11vnc on LXQt or LXDE (including Raspbian)

LXQt is the desktop environment of the current edition of Lubuntu. Previous editions of Lubuntu used LXDE as the desktop environment. My personal preference is to use LXQt on Debian instead of XFCE on Debian, due to XFCE development being too slow (the same reason I switched most of my more powerful computers to KDE). I would only use LXQt on a computer like my Toshiab R500 which only has 2 GB of RAM, or in the case of the Raspberry Pi, they ship Raspbian using LXDE. I installed LXQt over the top of XFCE after installing Debian onto this laptop, and it works great.

LXQt and LXDE (and Raspbian) feature an easy autostart capability which can be configured from the GUI and can also be set up by placing a file called x11vnc.desktop into the ~/.config/autostart directory in the user's profile. If you use the GUI it creates the file with default settings. 

We want to use x11vnc to share the default desktop so that we can remote into it with a VNC client (I use Remmina) to remote control the laptop. This means I can use the laptop as my tethered desktop. The laptop sits on a docking station and can be quickly docked and undocked so the docking station can be left in place with all cables connected and all I need to do is bring out the laptop, dock it, turn it on and away we go.

The settings to be put into the desktop file (e.g. x11vnc.desktop) in the folder mentioned above are as follows. Note we have omitted -usepw as we are running it here without a password on a home network where there is no other user. If you want to use it with a password there are additional steps on first installing it to set the password and the extra switch to cause it to use the password which I have left out here.

[Desktop Entry]

Encoding=UTF-8

Type=Application

Name=X11VNC

Comment=

Exec=x11vnc -forever -display :0

StartupNotify=false

Terminal=false

Hidden=false