Networking / Wireless with Debian LXQt

There are some issues possible with LXQt if you have Wifi because of limitations in the current Buster install. Some of these may be resolved by the time Buster is released...but not all, for reasons explained below.

First issue for Wifi is if the drivers are non-free, as is fairly likely. You'll notice this at setup if you get asked for a driver file to allow the wireless networking connection to be accessed (more likely if using the network based installation). This occurs because Debian is configured by default to have non-free stuff in separate repositories and access to those has to be specifically enabled in the sources.list file. Basically you have to change the first set of repository statements in the file that point to main, to point to main contrib non-free. This is due to the Debian licensing model and philosophy.

After that you can, for example, install firmware-iwlwifi which is the package for Intel wireless. 

The next issue is that there is no network connection manager installed by default in LXQt with Buster. Installing network-manager-gnome is the way to get this, along with nm-applet. Then you have to edit /etc/NetworkManager/NetworkManager.conf and change the following:

[ifupdown]
managed=false

change the false into true. This is necessary to ensure that you can manage the wired interface as by default you will not be able to manage this.

After that you should be able to get the wireless interface working on your computer. If you don't have wireless, having Network Manager installed is still useful if you need to set up anything non default with the wired networking such as a static IP address. 

UPDATE: I had a lot of trouble on this computer getting Network Manager to be able to set the IP address and parameters for the Ethernet cable port properly. The settings in the /etc/network/interfaces file read

iface enp3s0 inet dhcp

whereas in Network Manager I had changed to manual configuration which should have been shown in the file. So there is an ability to directly edit the parameters in that file so in our case we write something like

iface enp3s0 inet static
    address 192.168.1.222
    netmask 255.255.255.0

with no gateway or DNS servers or anything because this particular interface is only being used on the intranet and not to access the internet. But the question remains why Network Manager is not setting those parameters in the file. In other words it seems Network Manager was only able to configure the current session when it was used and not the configuration settings for next time. Network Manager was apparently able to handle the wireless connection. Possibly the issue of what is being managed in the Network Manager settings mentioned above.
 

NZ Rail Maps: Two different ways to cover a large area in Gimp [4]

Last time I wrote in this series I had been experimenting with the linear and segmented methods of covering a large geographical area in Gimp, such as I do with map mosaics for NZ Rail Maps. In summary the linear method consists of creating a large canvas that can lay out all the tiles in a continuous sequence with the canvas area used once, while the segmented method consists of creating a small canvas that can lay out tiles for each area over the top of each other re-using the canvas area multiple times.

The previous posts commented on the segmentation method using maps for Dunedin to Mosgiel, a distance of around 17 km, and the canvas size used was 7x7, where each tile is 4800 pixels wide and 7200 pixels high. About 200 layers were used in total for a file size of around 60 GB. I am now testing the linear method for Horotiu-Hamilton-Claudelands. The canvas size used is 20x13, a total of 260 tiles, but many of them are not used because of the L shaped corridor area. There are 96 layers at present and the file size is 12 GB. The base aerial photography resolution is 0.1 metres per pixel, which means that the canvas covers an area of 9.6 x 9.36 km.

It is therefore technically possible that a long linear area could possibly be laid out (for example like the MSL corridor in Christchurch in some of the sections used) which was up to 65x4 giving a linear length of 31 km. However since there would be a much higher tile occupancy ratio, it is unlikely we could get to 31 km before running out of system resources. This is not really that big a deal since 8 km or so is plenty and long straight sections of corridor that are perfectly aligned to latitude or longitude are pretty rare anyway.

Nevertheless it does look like a linear canvas usage method is achievable as an alternative to a segmented canvas usage method and with similar resource usage considering that I did split my original Dunedin file into two and therefore each individual file around 30 GB covers about an 8 km section and has 100 layers. Also, the Hamilton file will be able to have some base layer tiles removed once I have finished putting in the mosaics as some of the tiles are superfluous. Linear has a considerable speed advantage for corridors covering a continuous section because overlaps are much easier to work with, not having to be duplicated across both ends of a segmented section.

NZ Rail Maps: Mosaic overlays and masking

One of the useful things a good graphics package such as Gimp can do is use masks to combine layers when they overlap, so that instead of by default they overlap at their physical edges, you can mask off a part of an image so that the overlap actually is visible at the edge of the mask, which can be any area within the image. When I first started developing the NZ Rail Maps mosaics, I used to mask the historical imagery so that just the rail corridor was visible in the historic imagery and the rest of a mosaic tile showed the current aerial background imagery. There was also a version of each tile created by reversing the order of the historical and current imagery layers, so that the full historical tile was displayed, rather than just a masked off section of it.

There was a second reason I used to mask for, and that was for the historical images, which generally come with a border that is scanned from the contact print, and a copyright notice from Retrolens/LGGA. Since we don't want any of the border displayed on maps, I used to mask out the border, which enabled me to use the entire content area. These days I now crop the images to remove the border, which is useful for 99.9% of the time.

As time has progressed the requirements of the project have been changed so that masking is eliminated for routine mosaic tiles, which are now just the full historical imagery with the layer order never being changed. Hence as existing projects are updated, the layer orders are changed and the masks of this type on the layers are deleted.

Masking is now becoming very useful primarily when overlapping different historical layers and primarily for two purposes:

• Where two different layers overlap, we can use masking to control where the join between two different layers is positioned. The main reason is where there are features that are a height off the ground, such as buildings, which can be affected by perspective distortion, causing a join across the roof of a building to be discontinuous because one layer may have more distortion than the other. By masking, we can wherever possible move the join to a less conspicuous location (for example on the ground in front of the building) where there is less likely to be visible discontinuity between layers.
• Where two different parts of one layer are at different heights. An example is where there are railway lines on an embankment of 5 metres or more height going through the middle of a railway yard. Dunedin is an example of this because around 100 years ago the main line was raised up on an embankment in the middle of the railway yard, whereas originally it was on the ground level with the yard tracks. The yard has not been raised, so the issue is that perspective distortion with the embankment can make it difficult to get good alignment of both the embankment and yard tracks. By creating a duplicate of the layer and having the two layers labelled "BANK" and "FLAT respectively we can mask them over each other and then align each part independently of the other. The BANK layer would sit over the FLAT layer in the layer draw order and have just the embankment masked off to appear, the rest of the BANK layer would be transparent to allow the FLAT layer to be shown through.

I actually thought I had dispensed with masking until this week when I have started applying both of the above two situations to get some nice clean alignments and joins in the Dunedin-Mosgiel project.  The result is more masks than ever before, because most of this project had been done without the form of masking I used historically. There were some aerial images already in the project that did not have the border cropped off, which meant some extra work to crop them, requiring them to be rotated to orthogonality within Gimp because the software crop tools only work on orthogonal borders, and then rotated back after cropping. There is one set of two layers in the project that still uses border masking over full uncropped borders in order to maximise the displayed information. But as I noted above most of the time cropping the borders is sufficient.

Anyway I have just about finished masking off and layering everything I possibly can, so at least for the maps for Dunedin to Burnside, the project has now reached 109 layers and around 33 GiB in size and that is where it will be left for now.

 

Gimp resource limits

Right now I am completing a set of maps covering from Dunedin to Mosgiel, some 17 km of rail corridor continuously mapped. The result is a Gimp file of over 60 GB in size and Gimp itself is using about 150 GB of storage (memory plus swap plus home drive cache) to allow this file to be worked on.

Previously I have written about how we can use bigger files by having sufficient SSD storage capacity, and pitched for bigger SSDs to enable larger projects to be handled. However the key problem with a big project is how long it takes for it to be saved to disk - over an hour with a file of this size.

So I am looking for a balance where I can have somewhat smaller project files that don't take such a long time to save, and yet can still contain several station sites in one file instead of just one. This has been really an experiment, and only possible since Gimp 2.10 was released, because previous versions could not write more than 4 GiB to a disk file.

Prior to this Dunedin to Mosgiel project, the largest one I had on file is the OtiriaNorth file that covers some stations on the Opua and Okaihau branches. There is not as much stuff in that file by any means and it last saved at 17.8 GiB. There is not really a lot of merit in allowing this Dunedin-Mosgiel project file with 178 layers to have grown as large as it has. The long save time is really tedious, as is the resource usage in the computer. However I was interested in seeing just what Gimp could handle.

In a previous post I compared longitudinal canvas with a sectioned canvas with multiple overlays. I am starting to think a longitudinal canvas may work as long as the number of layers is kept low, so I will be experimenting with this format in Christchurch and see how long I can make the canvas before it gobbles up too much resources. In future at any rate I will work to 100 or fewer layers for files as a rough rule of thumb. I don't think I need a bigger SSD anymore. But having the 95 GB of SSD available for Gimp has been very good.

The Dunedin-Mosgiel project has now been cut into two files as Gimp is having difficulty working with the large project size and has crashed a couple of times so as I still have a little work needed on the project then making it work properly with Gimp is a priority.

NZ Rail Maps: Two different ways to cover a large area in Gimp [3]

Last time in this series I talked about my experiments in testing whether Gimp could handle a large canvas size successfully, since this would tend to suggest a linear project for covering a long section of rail corridor (17 km in a current project) could work. If successful this might be a better way of covering maps for a long suburban track area compared to overlaying the sections in a smaller canvas.

Well I took Gimp up to 11x7 and then up to a much larger 17x10 size to test this idea. However this is the point where Gimp couldn't handle the large canvas size. A save operation having repositioned layers but not added any new ones, saw the file saved to the exact same size as its predecessor, and then when refreshing the project after save, as generally happens, Gimp crashed after having exhausted the tile cache. This should have been taken up by its swap onto the home drive, but for some reason that didn't occur.

So I currently have my project in the last stable save at 11x7 and will be taking it back to 7x7. Therefore the overlaying of different layers is the only way to produce maps for multiple different areas in a single file. The organisation of these files means you are scrolling through 149 layers in the layer list. Gimp has layer groups that should do the job, but in fact consume additional resources that slow everything down.

At a guess the canvas size aspect is because Gimp has to resource the complete canvas, regardless of how much of its area is used or not. I am picking the resource use will be a complete canvas, and then individual layers referenced to an area of the canvas. Hence using a smaller canvas is more efficient where there are multiple layers that can share the same area of the canvas. This is a design aspect of the software in that it has to maintain the resources to draw every part of the canvas even those parts that aren't being used.

I'm still rapt with what I can do with Gimp and how I can make this all work for my maps projects, yet also see some limitations of Gimp design. Gimp is a great piece of open source software and hopefully it might be possible to address the layer groups issue in a future revision.

NZ Rail Maps: Two different ways to cover a large area in Gimp [2]

Following on from my previous post, in the case of the Dunedin-Wingatui map mosaic Gimp project, I chose to extend the size of the map canvas from 7x7 to 11x7 and bring the tiles for the Wingatui station alongside those of Green Island-Abbotsford. The reason for this was to be able to bring some smaller scale aerial images of the Walton Park branch into the project and easily overlaying them over the large area that stretches from Green Island to Wingatui.

This means this project now has a significantly larger canvas size than before and after adding two new Retrolens aerial photos covering the area in question and saving into a new file, the size of this file has naturally increased as would be expected with adding new content. In this case it went up by 5 GiB. I am not so sure how much of this is increased canvas size and how much is new content, as the two new retrolens images cover a significant area - a corridor length of more than 3 km. In order to test this further I am going to increase canvas size again just to lay the Mosgiel LDS tiles (which are already in the project) linearly alongside Green Island-Wingatui, and then save again and see how much increase if any in file size there is.

It is of course also opportune to ask about the merits of very large projects combining multiple locations into one file compared with single location files. One of the challenges with Gimp is the save time for a 30 GB project can be more than an hour. This, along with the time it takes for Gimp to run a unified transform over a large layer, is the key reason I have two computers working on the maps, as whilst this delay is occurring, I can use the other computer to work on something else. Having one of the computers almost solely for the purpose of running Gimp is not such a far fetched concept after all. 

Having a lot of stuff in one file is meritous mainly for efficiency, both in reduced number of disk files, and in combining and overlapping aerial images together. However, the overlapping is really only relevant where you have two or more stations that are fairly close together like in the main centres. For some areas where stations are further apart that I have combined them in one file, it is mostly for efficiency and organisation purposes.