Computing resources optimisation [2A]

So after a couple of months ago when I did some resource optimisation with my existing computers, it's time to look at the next stage. That stage will be upgrades to enable all four computers to be used for the most stuff they can.

The plan I am considering at the moment (subject to resourcing) is:

• serverpc gets a new mainboard with 4 RAM slots and new CPU and keeps the 16 GB of memory and we add another 8 GB to it, so the new motherboard would have to be compatible with the memory currently in serverpc as of now. serverpc gets renamed as graphicspc or mapspc as well.
• mediapc gets serverpc's old mainboard and 4 GB of RAM that I will have to add specifically to mediapc. One advantage is mediapc will have a spec that can play Google VP8/VP9 codec stuff which is not currently the case.
• pc4 gets mediapc's old mainboard with the 4 GB of RAM that it has on it. pc4 will be faster but with less RAM so it ends up doing pretty much what it does now, just better.

So I'm looking at the costs etc of that. All four computers will probably use Intel Graphics without issues with the NVidia cards and their drivers. They all have enough ports to be able to do two displays for the three computers that have a dual display setup.

I also need to look at whether I have enough backup disks at the moment to back up all three computers. The disks in all of the computers will remain the same as at present. I don't have any desire to increase any of the disk capacities in the near future, mainpc and serverpc have 2 TB each and mediapc has 1 TB and cleaning stuff up will be first call. The maps are growing but the stuff is really going onto serverpc more than it is onto mainpc and there is a lot of space that can be freed up on serverpc if I need to.

The plan is to give serverpc ability to go up to 32 GB RAM eventually and the numbers wouldn't have worked if we couldn't have reused the existing memory in it. Because even 16 GB of RAM would have cost $300 if I had to buy new RAM for the mainboard which compares with putting the board into another computer and putting 4 GB of RAM into it at $85. So the saving from being able to reuse the existing RAM is enough to be able to add another 8 GB to this computer.

The existing board in pc4 will be a spare for the Win10PC which is using the same type of Mini-ITX board. I have no other use for these boards although they will fit any standard ATX chassis so if the Antec Mini-ITX chassis had a power supply failure or whatever the board could go in another chassis.

What's interesting about computer hardware at the moment is that Moore's Law has slowed down a bit. Intel has hit the limit of 14 nanometres for five generations of CPUs. The four computers will have the following specs:

• mediapc: Ivy Bridge, 22nm, Intel DB75EN, Pentium G2120 
• mainpc:  Haswell,  22 nm, Gigabyte H97, Pentium G
• serverpc: Skylake, 14 nm, Gigabyte H110M-S2H, Pentium G4400
• new: one of the following depending on what I can get
• Kaby Lake, 14 nm, Gigabyte H270M-D3H, Pentium G4560
• Coffee Lake, 14 nm, Gigabyte B360 D3H, Pentium G5400

 But the new Coffee Lake and Whiskey Lake which are the next two generations of CPUs will still be 14 nm. And the 14 nm architecture started with Broadwell which is between Haswell and Skylake.

 So at 14 nm there is Broadwell,  Skylake, Kaby Lake, Coffee Lake, Whiskey Lake and then finally there will be the new 10 nm architecture on Cannon Lake, Ice Lake and Tiger Lake, etc, etc, etc. Usually with Intel you will only have two generations (tick and tock) on the same architecture so they must have had some difficulties getting to 10 nm.

With this optimisation (assuming it happens) I am also getting away from the idea of being able to use three or four computers for maps. The idea that mediapc could do some stuff has been dropped in favour of adding one of its screens to serverpc and using four screens across two computers and doing all the map work on those four screens. So the case is there to pump serverpc up to more memory.

Graphics issues with map mosaics [3]

Continuing from the previous post in this series. Once you have made up some mosaic tiles you need to know how to edit the sidecar files. In this case with the map tiles that come from LDS there are three sidecar files alongside the actual image. We only need to concern ourselves here with the .jgw file:

• jgw file: this is called the World File and contains six lines of text. The values of these in order are:
• Line 1: A: x-component of the pixel width (x-scale)
• Line 2: D: y-component of the pixel width (y-skew)
• Line 3: B: x-component of the pixel height (x-skew)
• Line 4: E: y-component of the pixel height (y-scale), typically negative
• Line 5: C: x-coordinate of the center of the upper left pixel
• Line 6: F: y-coordinate of the center of the upper left pixel
• If we have tiles that are double the resolution in both directions but otherwise identical we just need to change the first four to match the pixel size.
• In this case the world file for the upper-left most tile out of the four we have in the file contains the following values:
• 0.400000000000000
• 0.000000000000000
• 0.000000000000000
• -0.400000000000000
• 18835200.199999999254942
• -5627520.200000000186265
• We just need to change the first line to 0.2 and the 4th line to -0.2 to say that each pixel is half its previous size.

 When you open that tile as a raster in Qgis it looks for all the sidecar files. The other two files amongst other things will tell it the CRS (EPSG:3857 is the one I use for all NZ Rail Maps as it is compatible with Google Earth KML files) which it needs to know in order to interpret the world file coordinates etc. The files all need to be renamed to the same file name part but leaving the extensions correctly and they do need extensions because Qgis is multi platform so the extensions are required by the Windows versions (even though in theory they could read the magic numbers from the file headers the way Linux and MacOS do). So I use Thunar's bulk renamer to rename groups of files and I have installed it under KDE (it is part of XFCE but will work under KDE no problems).

Well it came together exactly as planned with perfectly aligned tiles so that is the technique I will be using to do a lot more maps whenever necessary if I need to make best use of higher quality scan imagery.

Graphics issues with map mosaics [2]

As I wrote in my last post there are problems when you have to scale down imagery significantly. So I am working on a new map mosaic experiment to see what I can do with it.

Starting from four tiles at 0.4 metre resolution i.e. 1 pixel measures 0.4 metres on one side, I am scaling each of these tiles to double in each direction so they have four times as many pixels in total.

The key will be the assumption that Qgis will be able to load these much larger tiles and display them as well as the originals, we just have to tweak the jgw file (world file) to say the pixel size is half of what it was previously and then they should draw in the right place

I have to recreate the mosaic from scratch i.e. I can't use my previous mosaic in any shape or form unless I scale up all the historic aerials as well and that will lose significant quality but as they are only background since I have these completely new ones that are the most significant that could be an option (starting from the old mosaic and arranging the new tiles around it) but I have decided to start from scratch and build a completely new one with only Retrolens coverage.

So far the results look really good in Gimp and I am banking on the larger tiles looking really good in Qgis because this has been my experience with Qgis and map mosaics to date, basing on experience of using Christchurch City 0.075 metre LDS contemporary coverage and the higher resolution Christchurch retrolens stuff as Qgis handles this well.

Graphics issues with map mosaics [1]

Well this is a tricky little number for people like me who are doing things with high resolution images. I am frequently combining high resolution scanned Retrolens stuff with other aerial images and running into some scaling issues. Basically if I have got a Retrolens image at a certain resolution and I have the aerial image from LDS at quite a different resolution then scaling the Retrolens image can have quite tricky outcomes if the scaling is significant.

I have of today been working with a high resolution image of a particular area and it's been scaled down to something like 15% of the original dimensions in each direction (resulting reduction in pixels is 2.2% of the original) and this results in losing a lot of resolution. Well the truth is there is not a lot that can be done about this because regardless of how many pixels there were in the original, the size of the pixels in the final image is preset by the original aerial image that I am overlaying onto, and therefore there is a considerable loss of detail. This resampling is hard to understand because the detail on the screen preview is so sharp from just scaling the original pixels to a smaller size and then we can see after doing the transformation that so much detail has been lost in the scale down.

To be able to preserve all of the original detail I would have to have the original layer much larger with a pixel size set in the world file for the Jpeg image as this is how a GIS works out what area to display a map tile over - it looks up the world file and that tells it where the upper left corner is and also the size of each pixel and it scales the tile to cover a certain amount of space on the canvas. 

I am going to experiment with that as it does seem desirable to be able to scale the maps up to see all of that detail on the ground at certain levels so this means greatly scaling up the original image and then not having to scale down the retro image so much. Up until now it has been the reverse with scaling most retro images up in size, which is understandably a lot easier for the software to deal with and results in much less quality loss. However there is a lot more work in that and in part it entails redrawing the whole mosaic from scratch which is not something that is easy or straightforward but it looks like that is the way to go but it will be experimental and not a high priority to carry out.

Limits.conf settings to allow a lot of file handles

One of the things I always have to remember to do for Qgis is for each time I install a computer or VM is to change the default settings in /etc/security/limits.conf to allow a larger number of file handles. Otherwise Qgis can't open a lot of layers at once.

Open this file and put the following at the end

*  hard  nofile  10000
*  soft   nofile   10000

Then after doing this you need to reboot the system. This fixes the crashes.

Setting a custom content width on Blogger's Simple template

Most of my Blogger blogs use the Simple template; I haven't got much time or inclination to scroll through masses of templates and experiment.

You can use the theme designer or you can directly edit the HTML to change some settings.

On some of my blogs I have directly edited the HTML to set a custom width of 1900 pixels so as to use the full width of my 1920x1080 screen. I prefer this width because I like to make my images 1600 pixels wide for display on the blog. 

In the Simple blog template HTML edit this section from line 583-586:

    <b:template-skin>
      <b:variable default='960px' name='content.width' type='length' value='1900px'/>
      <b:variable default='0' name='main.column.left.width' type='length' value='330px'/>
      <b:variable default='310px' name='main.column.right.width' type='length' value='0px'/>

The line you are changing is name='content.width' and as you can see I have put in 1900px which I can only set to somewhat less if I use the theme designer interface.

NZ Rail Maps is the first blog to use this wide layout. Some of the other blogs will be adjusted as necessary.

Apps, apps, apps

I don't write much about mobile so it's time to fill in this gap by blogging about some of the apps I have on my various phones. The main phone I have is a Nexus 5X, and I have a second phone, a Galaxy J2, which has a tablet share SIM (off the Nexus' account) and is capable of making calls, but not receiving them (although it can receive texts). I also have a Galaxy Tab A 8" tablet which also has a tablet share SIM and also can make calls but not receive. 

Apart from those devices which all get a reasonable amount of use and can connect to the cellular network I have two more older phones that are Wifi only and have been used as media players. The Moto E that was my first Android phone and cost $99 new isn't really flavour of the month as a media player now because it has trouble running apps reliably. Not sure if this is old version of Android, CPU/RAM or lack of storage (it only has 4 GB). So that leaves me with my Lumia 635 running Windows 10 Phone (it had WP8 when I got it) which has become the default media player. I always have a separate (low value) device as the media player so you can leave it playing music wherever and still use your phone, and not running down the phone's battery as these smartphones tend to go down too quickly for my liking, it seems that cellular services do drain the battery faster.

This is about some of the useful apps I use every day on the Nexus and also on the Lumia because those two get the most use everyday. In the below listings, everything is Android unless it says otherwise.

• Media Players:  The best media players are the ones that can browse folders on your device. If you are like me and have converted your video clip collection into MP3s for your phone, the tags that most media players need to sort them by artists, albums and etc aren't automatically put into an MP3 so players like the default ones (Google Play Music and Groove) can't pick them up. And of course videos don't have tags like MP3s. Here are some great media players:
• Pulsar music player - easily browse folders, no ads.
• PlayerXtreme video player - folder view (not exactly browsing all folders but just lists those that have videos in them).
• Still trying to find something as good as Pulsar for W10P. Have looked at so far:
• Loco Delight - corny name, a bit hard to navigate, stable and plays well.
• Next-Player - a little tricky to navigate. Plays well when it isn't freezing the phone. I am looking at whether it conflicts with another app to see if I can work out where the problem is.
• One thing to be aware of is that like videos, you need to put a number at the beginning of the name to ensure playback in a certain order when playing from folders e.g. 01, 02 etc. At the moment most of my music collection isn't converted from videos so they don't have these numbers, but as soon as I get around to changing over to converted clips and number-naming everything else, things will be better.
  
• Do Not Disturb: The default DND app in the Nexus (built in Google/Android app) is very annoying because of bugs, the biggest of which is if you go in to check what is turned on (e.g. a rule that may have turned on), just looking at the settings overrides any rule currently in place, so you can't go back to the rule being the higher priority. That's a little hard to explain, but that simple fact means there are many times I have accidentally turned DND on permanently without realising it and only noticed some time (days later often) that it was still on.
• Auto Do Not Disturb is a great replacement for the Nexus default DND app. The profiles give you an option to absolutely be able to make sure the DND is turned off at a certain time every day. I use an overnight profile which turns off alert sounds but leaves me able to receive phone calls and alarms. So no more annoying texts and app alerts in the middle of the night, but DND is forced off every morning. Apart from creating your own profiles for custom days and time slots, there is also a preset coundown  profile you can just click straight onto that will turn on DND for a preset amount of time. Premium version is $5.49 and well worth it.
  
• Mapping and GPS: A simple GPS logger that works a bit like my old cheap eTrex for logging a journey you are taking is a good app to have on your phone.
• GPS Logger by BasicAirData is a FOSS app that does a great job of logging and tracking a journey, with real time display of data including altitude, coordinates and speed. The only thing I found difficult was exporting the data because its data folder isn't accessible on my PC because of the way data is stored on my phone's built in memory. Uploading to Google Drive fixed that.
  
• Realtime Metro Timetabling: 
• Christchurch Transit is the latest app that can display RTTI information for bus services in Christchurch. I found the display cluttered and a little hard to use, and the older Chch Metro app is still my preference for its simplicity, although it is no longer in the Google Play store.
  
• Scanning: These apps can let you scan a photo or document from your phone. A killer feature some have is the ability to automatically correct geometric alignment issues.
• Office Lens is a neat little app from Microsoft that can attempt to correct the shape of photos taken from your phone that are not perfectly aligned. It works quite well a lot of the time, but in practice, I found it easier to turn off the correction for certain tasks (like copying stuff from bound files in Archives New Zealand, where it was hard to delineate the edges of an original). However, it was good for copying family photos, except that if the lighting is insufficient, the gainup adds a lot of noise or resolution reduction.
  
• Data Sync: these apps can be used to synchronise data between a phone and PC for example.
• KDE Connect is an app that can be used to transfer data between a phone and a PC that is running KDE. I haven't used it much, and one consideration is that like Bluetooth it can send maybe only one file at a time instead of a whole folder of files. A great substitute for Bluetooth with desktop computers which mostly lack the hardware for BT.
  
• Camera apps can do more than the built in app of your phone in many cases
• Camera FV-5 has a good reputation and I bought the Premium version of it on the Nexus. However it can have a bit of a learning curve, and has this annoying feature of rotating the display to landscape mode after taking a picture in portrait mode. I do use it when I want to do swish stuff, but for everyday simple things the built in app still gets a lot of use.

So there we have it, a pile of different apps that I find useful, and hopefully you will as well.

HOWTO: Enable root login in KDE / Debian

UPDATE: Since I discovered Ctrl-Alt-F1 and found I could root login there and have no problems, I have not bothered modifying any KDE computer to enable the root login.

Last time I wrote that I needed to try a new user profile on serverpc because of issues resuming from hibernation. This has proved to be more involved than I expected because sddm automagically disables root login, and you can't login as yourself, go into a root shell and delete your account, because processes have locked your user login, preventing it from being modified.

After a lot of investigation I found the following instructions which enable root to appear on the KDE logon screen:

https://blog.libero.it/Laoblog2/13228972.html

• Basically, you need to login to a root shell and edit /etc/sddm.conf file. This file may not yet exist on your system, in which case you must create it. You then need to add or change these sections:

[Autologin]
Relogin=false 

[Users]
HideShells=/usr/sbin/nologin,/bin/false,/bin/sync
MaximumUid=65000
MinimumUid=0

It's possible the [Autologin] section may not be needed. The [Users] section is needed in full. The first line hides every system user except root because there are a whole pile of other system users you don't need to appear on the graphical login. The other two lines set the range of user accounts that can appear. Normally the default for MinimumUid is 1000 which of course is the first general user account in the system. Change that to 0 and it will take in root (and other system users, which get blocked by HideShells).

• Also in your root shell run passwd root to ensure a password has been created for root.

• You also need to edit /etc/pam.d/sddm and make the following changes:

Comment out the following line by putting a # character at the beginning of it

auth    required        pam_succeed_if.so user != root nopasswdlogin

If the following line is commented, uncomment it (remove any # character in front of it)

auth    sufficient      pam_succeed_if.so user ingroup nopasswdlogin 

Reboot the system and it should come up with Root enabled on the login screen (it will be highlighted in red instead of black).

• In addition the following step has been found to be necessary:

Comment out a line in /etc/pam.d/sddm that reads as follows

auth      required     pam_succeed_if.so user != root quiet_success