NZ Rail Maps: Large Area Raster Tiles At Reduced Quality Setting

Using Qgis to draw the maps, we've run against a few times some sort of limitation on the number of raster layers that it seems to have, which runs out a long time before hitting the actual resource limits of the computer it's running on. This in turn limits the number of raster layers that can be included in a project which has created quite a few challenges in creating map volumes. So a few times we have looked at ways of making bigger raster layers to reduce the physical number of them used. At the same time we have had a play with JPEG quality, reducing it from 100 to 50, which drastically cuts file size. 

For example, over Lyttelton, we have a mosaic with a resolution of 57600x28800 (1,658,880,000 pixels in total) and there are in total 48 tiles (12x4) at 4800x7200 resolution. The 48 tiles downloaded from LINZ are 600 MiB in total. For the 1967 era layers we produced a total of 32 exported tiles which also reached about 600 MiB in total at quality of 100. However the large area tile that we produced for this article covering the entire 48 tile space with quality 50 only uses 156 MB which is quite a lot smaller overall, for only an imperceptible loss of quality. We may even try lower quality settings but for now, 50 is where we will leave this, as it will have to be tested over a wide range of mosaics.

Creating the full size mosaics is also much faster as there is only one tile output for each era and this means we will be doing all the exports for Greater Christchurch Maps in future at full size for each mosaic project in order to speed up the exporting process. Some additional work may be needed to make the areas covered be full rectangles without gaps as these will generally be exported as black areas that we don't need to be in large tiles. As it happens, with the Lyttelton examples, a number of large tiles will have the Linz 2015 contemporary areas included around the edges by default as the historical areas often do not cover the full canvas.

The main challenges are where a base area is not a perfect rectangle, meaning it will have to be broken down into two or more areas that are, and the time that it takes Qgis to refresh its canvas with the larger layer size. At the moment these issues are not too challenging and we expect them to be able to be worked around satisfactorily. Making the sidecar files is extremely straightforward as all that is needed is to copy the sidecars for the top left tile of the original tile grid and rename it to suit the new file name specification, due to the fact that the only coordinates in the world file are at the top left corner and the software assumes it is just meant to keep drawing the tile until all pixels have been rendered.

We have also observed that Qgis doesn't seem to have an issue loading raster tiles from a WMTS server in our projects that also have local raster layers, so the option is there of perhaps working out how to have a WMTS server on our local network to serve the raster tiles, instead of being file based, but also to query the difference in architecture of the software, as it is evidently handling WMTS layers in a better way than local file based layers.

Change Monitor Configurations Permanently in LXQt for Screen Recording

As we have discussed in the past, I use LXQt in two different configurations with my various computers. Lubuntu, which now uses LXQt as its desktop environment, is what most of my non-desktop computers use. For example a media playback computer and a laptop are normally set up with Lubuntu which is quick and easy to install as it has all the drivers included in the setup.

For my desktops I am using Debian, with LXQt on one computer in particular because this computer needs stuff like hibernation that is better supported in Debian, but also with LXQt it gets around some of the bugs in KDE which I use on other desktops, as well as ensuring the resources that KDE uses aren't wasted in its lower spec. So I am testing these settings on that desktop because it has two screens and it is in fact the main computer that I have at the moment that these settings are important for.

Because I use this computer to do screen recording (using SimpleScreenRecorder), the screens have to be nearly at diagonal placement to each other. That is, the left hand screen and the right hand screen are to the left and right of each other, but in a vertical sense there is only a tiny overlap at the top right corner of the left screen and the bottom left corner of the right screen. This means the mouse can only be moved from one screen to the other by passing through a very small area of overlap that is in the top right corner of the left screen and the bottom left corner of the right screen, and also windows needing to be dragged from one screen to the other can be passed through this small area.

The reason for this configuration is that for screen recording, we have the software that is doing the recording running on the left (control) screen, recording the right (target) screen. We want to ensure what we need to do on the left screen does not cause anything unwanted to be displayed on the right screen. Some examples of this are:

• accidentally moving the mouse from the control screen onto the target screen
• any action on the control screen which causes pop up notifications, such as operating keyboard volume up and down. Even if the popup is on the control screen, if it is adjacent to an area of the target screen it can cause unwanted display of information on the target screen.

We ensure this by making sure as described above, minimum overlap between the two screens, just enough to get the mouse from one to the other and drag windows from one to the other. We still have to be careful not to run the control screen applications maximised to ensure the mouse is kept out of the top right corner of the control screen at all times other than actually going from one screen to the other.

The LXQt monitor settings GUI doesn't actually position the screens exactly as we would like to. They can be aligned vertically and horizontally close to the suggested outcome, but with a small amount of unwanted superimposition of the two screens. Fortunately you aren't limited to using that GUI to set positions. After changing the positions in the GUI and closing applying or saving settings, you can further tweak these with manual editing in the file ~/.config/lxqt/lxqt-config-monitor.conf. These seem to be just the same sort of parameter specs as running xrandr will spit out for you. Under the [currentConfig] section are the ones presently being used for the computer and with a few adjustments we can eliminate that superimposition and ensure the overlap is as small as possible. For example, with the control screen being screen 1 on HDMI-1 (settings\1) at a resolution of 1440x900, xPos=0 yPos=745 are the settings and for the target screen being screen 2 on HDMI-2 (settings\2) resolution 1360x768, xPos=1440 ypos=0. 

The names of the displays xrandr generates for this system are interesting as it has one HDMI port, one DVI port and one VGA port, yet the DVI port appears as an HDMI port in the list of outputs. It seems this initial configuration of the displays happens just after logon. Also, the settings are saved in the user's profile (~/.config folder) which means when reinstalling, these settings should be reused into the new installation of LXQt.

Using a Raspberry Pi as a livestream player [1]

When I wrote the first post of this series I fully expected using my former Windows 10 PC as a home theater PC would work out. That didn't last long and the Windows 10 part was soon moved to another computer so the Mini-ITX system could be reinstalled with Lubuntu for the specific role that it was needed for. That hasn't worked out either, in both cases the older low-spec hardware is the issue, not specifically because it is low-spec but because it is older. The AMD E350, while good for desktop use, is not recent enough to be able to decode modern video codecs fast enough to be able to give good video playback on Youtube and other live streams.

So as the updates to that post attest, I had moved to using the Antec chassis system with Lubuntu, but that has not worked out after about a month of use. The only reasonably cheap solution, therefore, is to move to using a Raspberry Pi for this application. Although my one and only Pi is being used at my desk for a particular application where it gets its internet connection off my phone's mobile data, I can use an old laptop for that particular purpose in the meantime while I use the Pi for the bedside PC application (for night prayer/intercession use), until I can get another Pi.

The model 3 B Pi is just slightly too underspec to keep up with all livestreams. It actually still does work very well with Youtube on 240p or lower, but at higher rates tends to drop out a little. That is OK for now, and far better than the E350. The Pi 4 which has just been released is available with 2 GB of onboard RAM at $85 for the board ($105 with 4 GB of RAM) and is much faster than the 3 B and has many enhancements such as dual mini-HDMI ports, USB C power , etc. To that price you have to add a case and power supply, which can be done in stages as funds permit. There is no real hurry on that as I can get by for now with the 3 B, which is currently powered off a spare Ipad power adapter. I could continue a similar arrangement with the Pi 4 and save on the cost of a power adapter, but it will be a toss up whether to go with 2 GB or 4 GB of RAM, and also which distro to run on it if I want to use Firefox, which is apparently unavailable for Raspbian.

So the Antec chassis will go back to being my Windows 10 PC, which in turn can be moved out of a bulky spare desktop chassis again.

Power Tools: Best brand of home power tools?

If you are working at the cheap end of power tools there are a few brands to choose from. I have bought Ryobi stuff a few times, weedeaters have been good, also a good 1200 watt electric drill (a knockoff of a Bosch or Hitachi drill) with variable speed, hammer, 2 speeds, reverse etc. I used this some years ago to drill a lot of holes in desks to run cables through and it did a great job. 

Then there is Makita, I have owned a few Makita products but not anything that I bought myself. Orbital sanders, mains powered and cordless drills have been some of their products that I have had, some of which are still in use today.

Then there is Bosch. Their home stuff has been quite good, from a range of cordless drills, saws, heat guns and sanders.

The problem seems to be these days that all three of these brands, plus Black and Decker, have gone mad and are now producing cheap rubbish that only lasts a short time. Well, B&D stuff was always cheap and nasty but now there is a race to the bottom. Some examples:

• Mouse sanders (triangular detail sanders for sanding into corners): Bosch have two models that are of very poor quality, the base plate is made of foam onto which the sanding pad is held with velcro, the pad being a soft material disintegrates and is then expensive to replace. A ludicrously poor design with also the dust collector box being either difficult to remove and empty, or falling off all the time. 
• 1/4 sheet detail sander: Makita have done a very poor job with their BO4555/4556 models. Numerous reviews worldwide attest that the paper clamps on these will break off after only a few uses, whilst there have also been a number of people who have found the whole base plate has broken off completely.
• Any sander that uses Velcro fastening: there are too many of these where the hooks and loops wear down quite quickly and will no longer hold the sanding sheet into place. This also includes the sanding pads that can be bought for multitools.

Seeing these situations for these brands of tools which have previously been fairly well regarded is a big deal for me. It seems these manufacturers are just producing throwaway junk with a life that can be measured in hours.

These situations are hard to understand because I have here an older Bosch random orbital sander that has a velcro pad for attaching sandpaper and not only is the pad made of good quality material (hard rubber) but also the velcro works very well and holds the sheets in place properly and they don't fall off. Generally the older Bosch tools are well made and don't break. The Makita detail sander I have is well made and has given a lot of use (except for the dust bag which has a flimsy internal support that broke). 

So I have to look at some other choices:

• Metabo (which now has taken over Hitachi)
• Bosch Professional
• Hikoki ?
• DeWalt

Metabo is the only one of these that doesn't seem to be specifically geared at the professional trade (but I could be mistaken in that assumption). Their gear is going to cost twice as much as Bosch but about the same as Makita, yet it looks to be of better quality. They have a mouse sander that Mitre 10 carries in stock for about $175 that appears to be quite well made although reviews are hard to find.

Dewalt don't have a mouse sander but they do have a 1/4 sheet sander and also a random orbital sander which price around $175-225 in NZ and appear to be well made. Bosch have some professional products which can be hard to find here.

Mostly what I see in reviews for certain home user Bosch and Makita products is people questioning why these manufacturers now turn out cheap junk when they used to produce good quality stuff. At least everyone already thought Black and Decker was low end but it seems Bosch and Makita are falling over themselves to emulate B&D and see who can produce the cheapest flimsiest tools that don't last much longer than the ones you can buy from The Warehouse.

I don't need to have any more sanding gear at the moment so this is an interesting discussion / conversation for future reference.

Python Scripting: Planned Scripting Projects Update

Back in February this year I set out a list of planned scripting projects following my acquisition of Python programming skills. The sequence of planned or implemented projects to date is:

1. NZ Rail Maps project script to copy a set of GIS raster layer files based on reading a QLR file produced by Qgis. Currently used occasionally.
2. Auto sync script for creating an audio-only clone of a collection of video files. Not yet started.
3. A script to produce layer fractional segment sidecar files for GIS raster layers. Not used at present due to this feature not currently being used for mosaic tile generation.
4. EXIF based image rename script for digital camera photos / movies. Used almost daily.
5. I'm not sure what scripting project was going to have the number 5 because I went straight to 6 instead, and it's a mystery at the moment why I did this. It would have made sense at the time, because project 5 may not have been written down, or it appeared in one of the previous articles that I haven't re-read thoroughly enough to determine.
6. Taking the script from (3) above and changing it into a straight duplication script for simply copying sidecars where layers are duplicated. This is currently used quite regularly when mosaic tiles are exported for use with Qgis.

It is now time to pick up item (2) and start work on it, this week hopefully.  I now have a situation where I need to be able to play back music from a phone and base it on automating the audio extraction from video files to produce a music-only tree clone of a video file collection.

OverGrive seems to work again...for now.

OverGrive is a Google Drive client for Linux. It has been around for several years. I used (and licensed) an earlier version of the product, which I was forced to stop using last year because it stopped working (this quite often happens when Google updates their API to a new version which requires software changes). At that point with no apparent communication or updates from the developer, I switched to the free "grive" package, which however is command line only and lacks the GUI configuration interface, system tray icon and other features of OverGrive. However that also stopped working earlier this year, probably for similar reasons.

It so happened I was reading a Linux website today and saw an article on Tech Republic showcasing OverGrive, so I decided to try reinstalling it, and was surprised to find that it works. The version listed on their website is still the same version as I had installed before. It seems to work OK on Buster despite this (the free grive package had to have specific versions for each release of Ubuntu, which was a trial and error installation onto Debian because of course, Ubuntu uses a different release schedule).

I currently have two licenses for two different versions, on two different computers. One of these is for the NZ Rail Maps project and provides for a daily backup of the core project files and layers. It doesn't, of course, include all the aerial photo layers which would not fit into a 15 GB Google Drive cloud storage space. However at present these layers are stored on two different computers as well as being backed up on removable disk.

So with very minimal documentation on OverGrive it's hard to understand why it stopped working on my computer and whether there really has been an update to it. The homepage still links to the highly out of date GitHub and Launchpad repositories, without any explanation of why these areas have not been updated for several years. If this outfit actually still maintains their product and take an active interest in it, they haven't exactly made it obvious.