General computing update - week 34, 2020.

 Since writing early July about about the need to replace the disks in mainpc I have finally progressed towards this stage by purchasing the first of two disks. As noted in the earlier post, this was prompted by an I/O failure of one of the disks that resulted in it dropping out of the RAID array. However I was eventually able to add it back into the array and it has worked properly ever since. The first disk has been added to the existing array which is currently a three disk mirror. As soon as mdadm has synced this disk completely, I can then remove one of the existing disks off the array and then take it out of the computer, as I need backup disks right now. As noted at previous article the disk array will be the same size as current. I am expecting to buy the second disk in about another month and install it in the same way so that when it is also installed, the old disks can be removed completely, and all without actually copying any files. Last time when I did an installation I manually copied the files, but that was mainly because I doubled the disks from 1 TB to 2 TB; this time I will be staying at 2 TB.

Another techical task I have been working on around the house is erecting a TV aerial on the outside of the house. This has meant installing a cable duct and two faceplates between the inside wall and the underside of the soffit to take the aerial cable. That was the hard part as running the cable through the duct was very simple as was putting the aerial itself up. The TV itself is wall mounted and the cable comes through directly behind it. As I am in a prime reception area, the aerial does not need actual line of sight to the transmitter – there is actually another house in the direct line, but the signal is so strong that there is enough signal to get a good picture on every channel.

A few weeks ago I had some problems with a new package-based version of Qgis that would crash every time I tried to run it, on more than one computer. I had to install a flatpak-based version of Qgis, although I could have also run it in a virtual machine. The project developers built a new release after several weeks and it started working again, as did the next edition. Due to the disruption, I have implemented an extra precaution of testing every new release on a VM before installing it onto a working computer.

Google has again heaped glory all over themselves by implementing an arbitrary and dictatorial change to the Youtube platform. As of last week, they stopped making it possible for Youtube to send out notification emails whenever a new video is uploaded or streamed onto a channel. This has left a lot of people, including me, very annoyed by the decision. It is similar to the nonsense of the new Blogger user interface that is badly broken and which has forced me to copy and paste all my posts from another editor like LibreOffice. As I am doing right now with this post. Contrast that with the approach taken by Wordpress.com with a new editor system introduced a few months back; it was pushed through, but they did a lot of consultation with their users. In this case, Google has just announced the changes, but if there are any issues, it’s a one way conversation filing a problem report, just like the way you cannot actually have a conversation with Facebook about bugs or problems on their platform most of the time.

General update week 30, 2020

I wrote about a lot of fun getting a system to work with static IP addressing (it is running Debian 10 with LXQt). Since it was able to be made to statically configure its IP with no issues, I configured the other Debian10-LXQt system with a static configuration also. The second computer only has one network adapter, but the benefit is that since the network router and wireless broadband router are only switched on when needed, I do not need to wait for the network router to come online each time and be ready to supply DHCP addresses to the client computers when they start up. Everything is working well now.

At the present time the backups are all being done locally on each computer as previously referenced using rsync since I have yet to do further work to make an rsync daemon function as planned for trial for backups. Another option I could look at for backups is the BackInTime application, however I have been generally wary of using anything that does not do straight file by file backups to the destination but instead creates its own archive format. This is not such an issue with commercially supported proprietary software, but I am very insistent that I do not get locked into archive formats with open source software that may not be supported forever. Probably this is the reason why lots of people prefer just to use rsync.

mainpc had an issue a couple of weeks ago when some sort of timeout on one of the RAID array disks caused mdadm to push it out of the array. It was then coming on and off line seemingly randomly for a time, but then stabilised and stayed online. Smartctl testing showed there were no errors on it. After a week it was readded to the array which is now functioning at 2 disks again seemingly with no problems. However as these disks are 4 years old (last replaced 2016) I am currently planning to buy 2 new disks for the array as soon as I can, this will be at the rate of 1 disk per month with the old ones being available for backups although it would still be preferred to buy completely new backup disks. 

The disks in the array currently are WD Black and that is the proposed replacement at $250 each. SSD is considerably more expensive but by the time I next have to replace any of these disks the cost of SSD could well have come down and be more affordable. WD Black SSDs in the size needed are  very expensive at around $1000 regular pricing each (four times the HDD price) and are only M.2 format which is problematic since it is rare for a board to have more than one M.2 slot at least in the price range I have. I have checked and the H-97 and B250M-D3H boards do have one M.2 slot each but not two. There are adapters available but another option is simply to switch to another SSD brand such as Intel which is considerably cheaper although with lower performance, if the performance at least equals standard HDD it would be a big reduction in cost to around $400 each. As I think, in a few more years getting a pair of SSDs for the array will be within the same price range as the regular HDD. I am still keeping the lid on the amount of storage on this computer resisting the temptation to buy bigger disks as in spite of growing photo and general data storage, so far it has proved possible to manage existing storage sufficiently well to avoid running out of space.

Rsync Backup System [6]: Latest Progress / Using The Rsync Daemon

Because of issues with Rsync over ssh I am looking into using the Rsync daemon instead. The commentary on that is below. Until I can get that working properly I am backing up each computer individually using its own removable drive bay. This can be achieved by changing the ownership of the backup folder on the removable backup drive to backupuser and then  logging into a virtual terminal as backupuser and then running the backup under that user account. We use that account rather than root or our regular user account on the system being backed up in order to ensure the account has only read and execute permissions on the backed up computer so that it cannot in any way risk deleting files from the computer. This is very important because I have used the --delete flag on rsync to ensure files removed from the source are also removed from the backup. When doing an incremental backup that flag is important to ensure the disk does not end up filling up with removed files and running out of space. But it is also useful to have the safeguard that you can't accidentally copy files from the backup disk over the source files, especially with incremental where your backup disk is already full of files.

The simple means for an incremental backup that rsync supports is using an existing backup disk that has previously had full backup onto it. This enables rsync to compare the source and backup without any additional system to record file details that it can then work out if they have changed. In other words with each source file it compares to the existing backup then only backs up incremental changes. This is quite different from other backup solutions I have used that have a separate incremental backup disk.

To implement this system I will need at least 3 disks for each computer if I want to keep one full backup and two incrementals. In other words each disk will start off as a full backup, then two of them will be alternating incrementals, and a full backup from scratch will be done every few months and kept separately from the incrementals.

I would still have preferred separate incrementals, and in fact it is possible with two removable drive bays, one contains a full backup disk and one contains an incremental disk, then the incremental can be written to the incremental disk instead of the full backup. However the problem with this solution is that rsync cannot produce progressive incrementals since it is always comparing with the last full backup, that means the incrementals will not be a snapshot at a particular point since the last incremental, but will always be incremented from the date of the last full backup. It is also messy working with the extra paths to the different disks in the backup command, and therefore, more likely to risk mistakes like backing up to the wrong disk.

So in summary the best solution is using a separate disk for each backup, which will be an incremented full backup in the case of 2 disks and a separate full-only backup in the case of one disk.

At the moment I am down to 1 backup disk for each computer and so I need to buy some more disks along with the removable bay caddies and storage cases. I am also working on secure storage for the disks in the storage location which is away from the house. At the moment I have only been doing one backup about every 3 months because the RAID-1 array system in the computer is so reliable for day to day backup that I don't need backups too often. But I will attempt to schedule a monthly backup cycle in future.

Since I have been using Rsync as a backup solution, it has been very reliable, but there are occasional difficulties, and one I am having at the moment is the backup terminating with rsync error 12 partway through. I have done some fault finding but haven't been able to narrow down what is occurring that causes this termination but it is probably something to do with the ssh client in the source computer, which rsync connects to. This means I have to either try and find out what is happening with ssh on the source, or try the alternative, which is using the rsync daemon (service) on the source and connecting rsync to that from the backup server.

I'll be starting with this guide and applying the steps in it: https://linuxconfig.org/how-to-setup-the-rsync-daemon-on-linux
Also https://www.atlantic.net/vps-hosting/how-to-setup-rsync-daemon-linux-server/

Firstly as I am on Debian, rsyncd is already installed with rsync. Next step is to configure the /etc/rsyncd.conf file which I will do from this sample provided in that article:

uid = 1001

gid = 1001

pid file = /var/run/rsyncd.pid
lock file = /var/run/rsync.lock
log file = /var/log/rsync.log

[backupuser]
path = /home/patrick/
comment = backup patrick
read only = true
timeout = 300

There are some changes I am doing from their sample, the most notable being the uid and gid, which are set to the values for backupuser, which is the user that has permissions to do the backup on the source computer. Assuming rsyncd is run as root, this will ensure the permissions are correct. Also I am specifying the module name as backupuser, and it will automatically connect to the backup path for my home folder.

Start rsync with systemctl start rsync and set it for startup with systemctl enable rsync .

So next time I will list the result of testing with the daemon and whether it worked successfully.

Rsync Backup System [5]: Ongoing Backup Journey

Since writing the first four articles on this topic, I have not yet figured out how to implement an incremental backup strategy, and so I am still doing only the full backup every few months. Hence right now I am doing my first full backup of mainpc for three months.

The first issue that has come up is zfs saying there are no pools available on the server that mounts the backup disks and runs the backup. This apparently is because zpool is not designed with removable disks in mind, even though they have been unmounted prior to removal. After a lot of head scratching trying to figure out how to get zfs to recognise the previously created pool on the removable disk I had inserted for the backup, I eventually stumbled across the zpool import command. Once that was issued the pool was said to be online and the zfs set mountpoint command could be issued:

zfs set mountpoint=/mnt/backup/fullbackup mpcbkup2

At that point I could go to the mount path and work with the contents of the disk.

One possibility for this issue could be that the disk is physically in a different path on the computer than the one it was created in. A key factor is that zfs appears to be designed primarily to work with /dev/sdx device paths. We already know that for regular disk mounting we can use UUIDs to get around the problem of the device path /dev/sdx changing at the whim of the operating system at boot time, which actually does happen. For some reason or other the backup disk in this instance was at /dev/sdh when it may have originally been in a zpool mounted at /dev/sdb according to the earlier articles I wrote. At any rate, using the import command can bring the zpool back to life. Perhaps there is a need to have a command that suspends a zpool when the disk is taken offline, since it appears to be necessary to do more than unmount the disk.

I looked into this further and after considering the options, the best command to use is
zpool export mpcbkup2

This command basically closes down the pool for it to be removed from the system. The system will then report that no pools are available. The next time we need to use it, it can be imported back into the system and then mounted as above. 

I also need to buy another backup disk for ensuring each major server has two full backup volumes. At the moment I don't have enough disks to assure this. It will only cost about $100 to get another 2 TB disk.

The lingering question of course is how to detect which files were backed up on which date and therefore which files need to be backed up incrementally. To make any progress with this, the very first step is to find out if rsync can log the files that have been successfully backed up, and find some way of automatically scanning the log for the names of files, and then store them somewhere (for example, an SQLite database). Another option is to find some way of having rsync only back up files that have been modified after the date of the last full backup. So far I actually haven't spent nearly any time at all thinking too much about these options, because the simplest solution by far would be to have some sort of command or script that handles this completely automatically. Linux lacks the file modification flag that is implemented in Windows (the archive bit). People explain this away by saying the FS has superior capabilities and that a file modification bit is a very crude capability, but it still isn't possible to get around the fact that being able to reset that flag after each backup, and then being able to scan for files where it has been set again, are very easy to implement. In Linux there is no easy way of being able to record that information unless you store a date somewhere for each file and then scan against those dates. I explored the possibility of writing an extended attribute for a file, the question there is where this data is stored as we do not want to modify the file itself. So I hope to spend more time over the next month or so exploring these issues further.

I am currently trialling having a log file produced and using this form of the rsync command:

rsync -arXvz --progress --delete backupuser@192.168.x.y:/home/patrick/ /mnt/backup/fullbackup/patrick --log-file=/home/patrick/rsync.log --log-file-format "mainpc|%f|%M|%l|%b|%o|%U"

The only issue to date with it is there is supposed to be a %a option but that is not being recognised so having the remote IP address logged is not available so the script has been customised to output the actual remote computer name as a literal.  The rest of the information is logged in the format and the pipe characters can be used as delimiters to separate the parameters. So I have made some progress on this issue and now the question is how to use the information to analyse what is needed for future backups.

With the backup of mainpc I wiped the disk first, but for serverpc I chose to send the data to the existing backup which means it can run a lot faster because it is not transferring every single piece of data to the disk. So that option should speed up the backup and using the --delete option will remove files on the disk that are no longer present in the source directory.

I have to set up mediapc to be able to back itself up, which will be implemented as a "backupuser" account that is logged on to in the terminal virtual console, and then runs the backup for mediapc locally with read permissions to the source files/folders.

An option for progressing the backups is simply to use the full backup disks to create the incrementals as well, but this is possibly going to need bigger disks. The advantage is that rsync can handle this by default with the incrementals pushed into a separate directory. I would want to have possibly three backup disks for each comptuer, which brings the need to buy more disks.

At any rate this will take some time to devise. Another alternative is separate incremental disks, with a second removable disk caddy installed in this computer, so that it can backup to a completely separate disk. This does have the advantage of not requiring new disks, and keeping the full and incremental backups separate.

Debugging Linux startup issues with JournalCtl

My computers running Linux are very reliable but every so often a spanner will be thrown in the works and the computer will refuse to come up beyond a command prompt. When this happens you have to look at the logs to try to work out what is happening in the system and how to get it going again.

This happened to me a couple of days ago when I turned on the computer and instead of resuming after hibernation, it threw up a number of error messages and then asked for the root password for maintenance (or there was the option to press Ctrl-D to continue). The Ctrl-D option simply put the computer into a loop where it ran a FSCK only of the boot drive, and then failed to resolve any issues, and came back to the same message. Or the option was there to enter the password which took it to a terminal to see what could be done.

Since the advent of systemd, its companion journalctl (with the -xb parameter) is what is recommended to look through to try to determine what the problem is. This took quite a while and it wasn't particularly obvious what was actually happening. The main issue I noticed immediately was no networking; the wired network adapter in the computer appeared not to be functioning, indicating that part of the startup of getting the adapter up and running had not happened. I tried a whole lot of things but eventually concluded I would have to try a live CD to see if this was a hardware issue or if it was something with a driver. 

The network turned out to be a red herring once it was realised that there was no home path in the file tree (i.e. /home was completely missing) and fairly soon running fsck over /dev/md1 revealed some errors. Once these had been fixed, rebooting the system brought up KDE normally and suddenly everything was working again like usual.

It isn't the first time this has occurred and I think last time we just threw in the towel and reinstalled. At the time of trying to resolve this issue I was downloading netinst and live images to burn to a pen drive and try to reinstall, but finding this relatively simple solution sorted the issue without having to take that complex, lengthy and very inconvenient route of putting everything back together. And what seems an obvious issue may actually not be the problem at all. The network adapter not working turned out to be a non issue as soon as the disk issue was taken care of and a reboot initiated.

Rsync Backup System [4]: Linux File and Directory Security Using ACLs

As per our series on Rsync backups, we desire to use a different user from the one that owns the home directory in order to ensure they have only the permissions they need when logging in remotely. At this point whilst by default the backup user could access many things on the computer, there were some files that they received permission denied errors for.

Whilst there is a simple file and directory permission scheme built into Linux, the acl extensions are very worthwhile as they give a lot more control, especially with the all-important ability to have inheritable permissions on a parent folder. In this case we'd want to give the backup user read-only permissions on the home directory that are inherited by all subdirectories and files as they are created.

Getting ACL is pretty simple, firstly you need to install the acl package using apt. The second requirement is to see if a volume is mounted with acl support. Use a command called tune2fs -l /dev/xxx (part of e2fsprogs package) and look for "Default mount options". If it says acl in there then the volume will be mounted by default with ACLs enabled. If this is not a default then you'd want to go into the volume mount entry in /etc/fstab and you can add acl to the mount options there. In this case because in fstab I used "defaults" as the mount options, acl is automatically turned on at mount time. If I needed to specify acl then I could change the word "defaults" to "defaults,acl" in the fstab entry to enable ACLs.

ZFS also has its own (more extensive) ACL support which is not covered by this post as we are referring to the existing filesystems which are ext4 on the computers being backed up in this case. I am not planning to change the existing RAID arrays from MDADM to ZFS but it could be an interesting consideration for the future should a disk array be replaced in any system.

Now how to configure ACLs for our requirement? Commands used to set and get ACLs are setfacl and getfacl respectively. To set up read access for backupuser on /home/patrick we need to use a command that looks like this:

setfacl -dR -m u:backupuser:rx /home/patrick

If we then use a ls -l (or ls -al) command in /home/patrick we can see next to the standard permissions a + sign which tells us that ACLs are set. In this case I can see whilst all directories have ACLs set, the files that are in each directory are not set with ACLs. So I would need to run another command to set these:

setfacl -R -m u:backupuser:rx /home/patrick/* /home/patrick/.* /home/patrick/*.*

The two different commands are as follows:

The first one using 

-d setting the default ACL for each directory (the default ACL to be applied to any new file or directory)

-R operating recursively on all subdirectories

-m modifying the ACLs

u:backupuser:r is user backupuser granting read permission.

The second one is practically the same except there is no default applied because it is for existing files and the paths following it are the different filespecs to match. 

Whenever a new file or directory is created in future it will inherit the default ACLs that have been defined using the first command which means backupuser will get the permissions it needs. Multiple paths to apply the ACLs to can be specified but only one user specification can be put in at a time.

ACLs override the default permissions system when they come into effect and so you need to have good knowledge of how Linux permissions work. In particular you need to ensure your user has x (execute) as well as r (read) permissions specified in ACLs for directories in order to be able to traverse directories, otherwise you will keep getting permissions issues. It took me a bit of work to get the ACLs set up but once they are done properly, the advantage is that inheritable permissions are possible, so for my backups, the backup user will have automatic rights to new files and folders that are created in future. In fact for any computer the way to make it happen is to set a default ACL on /home that gives the backup user rx permissions on everything below in future.

With the ACLs properly set the test backup was able to access all the files it needed to and complete the backup. Since I knew there were roughly 114,000 items waiting for backup, it was interesting to watch rsync at work discovering new files (a big increase from the roughly 7000 it had access to the first time). 

The means of doing the incrementals is still being worked out as it is not as straightforward as I had hoped. The log file is not consistent in its format (at least not so far) that would make it easy to determine which files successfully transferred. The option I am tending towards would probably consist of getting a list of files that have changed since the last backup, then getting rsync to copy one at a time, looking at the result code that it returns, and if there is no error then writing an extended attribute of the source file to indicate a successful backup result. The extended attributes will probably be backup.full = <date> and backup.incr = <date> and these two extended attributes will be written by the appropriate backup script. A full backup simply updates the attribute at backup completion without checking it. Whereas for an incremental, we compare the last modified date in the file with the date stored (backup.incr if set, otherwise backup.full) and do the backup if the file has been modified since the last backup date. So I expect to start more testing reasonably soon with developing scripts to perform these tasks.

But so far as the full backups go, it's doing pretty good. I have just set up another full backup and it's working very well. The second time setting it all up was so much smoother. It also looks like ZFS disk compression works very well and has achieved some useful saving but I will check out exactly what sort of improvement I am getting on these disks.

Rsync Backup System [3]: Using Rsync For Full Backup

So last time we talked about how to set up a single disk with ZFS to use compression. Having got our backup disks sorted, the next step is to work out how to use rsync to do the actual backups.

rsync is written by the same people that devised samba and is a very powerful piece of software. What we need to do is to first of all set up a dedicated backup user on the system that is to be backed up, that has only read access to the filesystem. Having this is extremely important in case we make a mistake and accidentally overwrite the files on the source filesystem (which is quite possible with such a powerful system).

Using useradd we can set up backupuser in this case with a very simple password consisting of four consecutive digits and then by default it will have read only access to other users' home drives.  This means it can read my home directory. backupuser in this case is used with SSH to access the home drive over the network for creating the backup.

The next step is to look at the required form of the rsync command. There are certainly a lot of options for this. Assuming the backup destination zpool has been mounted to /mnt/backup/fullbackup and the source is on 192.168.x.y at /home/patrick then a good starting point for the backup would be along the following lines:

rsync -arXvz --progress --delete backupuser@192.168.x.y:/home/patrick/ /mnt/backup/fullbackup/patrick --log-file=/home/patrick/rsync.log

This looks to cover all bases needed. The options are -a which means copy in archive mode (preserving all source file attributes. -r which means recurse into source directories, -v which means verbose and -z which means compress during transfers (which can speed things up). --progress means during each file transfer you will see the bytes transferred, speed, percentage and ETA which is useful for large files. --delete will remove extraneous files from destination directories (which means files that aren't in the source). This option is obviously useful when a file gets deleted or moved in the source. -X means to preserve extended file attributes if any exist.

At the moment I am testing this with a full backup of serverpc with this running in a terminal window. There are some issues with some directories not having permission for backupuser which so far has only affected a few hidden folders but will have to be looked into further. Previous backups always used patrick as the user but it is pretty important to have a special backup user with restricted permissions which is really a best practice for any kind of professional computer setup. An example is because a mistake with the rsync command could wipe out the source directory if there was read-write access to it.

After looking at log file options there is this option that we can set the log file format for rsync using an extra parameter and this is what I came up with in deciding what format of information would be useful in each line of the rsync log:

rsync -arXvz --progress --delete backupuser@192.168.x.y:/home/patrick/ /mnt/backup/fullbackup/patrick --log-file=/home/patrick/rsync.log --log-file-format "%a|%f|%M|%l|%b|%o|%U"

Using in particular the log file format option, it seems to be usefully logging the information we need for each file. However the %a logfile option does not actually seem to be recognised by rsync. 

There is one thing that rsync does not do by itself and that is incremental backups to a separate disk. rsync is designed to be able to by default create incremental backups in a separate directory from the one where the full backup is stored, but the full backup directory must be online at the time when the incremental is done so that it knows which files have changed. This is a problem when, as I intend, you want to be able to use separate disks for the incrementals. Here you see the fundamental issue with the Linux file system (at least ext4) that does not have the separate archive flag for a file that NTFS has. The argument goes that it isn't necessary but that archive bit is the way you can tell that a file has been modified since the backup last ran, and increment it. 

There are several possible solutions to this and one of them is to create an extended attribute for each source file. This is possible using the setfattr command. So we could create this on each of the source files following the full backup (it would have to be a separate script process following the execution of rsync). Maybe the script would be part of a verify process that we run after the backup is carried out, to verify that the source and destination files both exist, and then write the extended attribute to the source file. The issue is that it may well prove difficult to be sure the source file was backed up unless we can have a look at a log file and prove that it lists the source files and then feed that into a script that produces the extended attribute. Anyway, the point of this is to write an extended attribute to each source file that says when that file was last backed up. This could be a part of an incremental script that uses find to get all files that were last backed up since a certain date. Or we can just use a range of dates for the find command to get the incremental file list using the last modified file time. This will all be looked at in more detail when I start working on incrementals, because for now I am just doing a full backup like I did before with rdiff-backup.

Rsync Backup System [2]: Using ZFS For Compressed Backup Disks

So last time I talked briefly about ideas for using Rsync to do my backups. Over time this will gel into a whole lot of stuff like specific scripts and so on. Right now there will be a few different setup steps to go through. The first stage is to come up with a filesystem for backup disks, which are removable, and this time around I am having a look at ZFS.

Basing on some stuff I found on the web, the first step is to install zfs on the computer that does the backups. Firstly the contrib repository needs to be enabled in Debian, and then after that we need to install the following packages:

apt install dpkg-dev linux-headers-$(uname -r) linux-image-amd64

apt-get install zfs-dkms zfsutils-linux

After completing these steps I had to run modprobe zfs as well to get it running for systemd (this was flagged by an error message from systemd). This only works on the current running session. To ensure there are no "ZFS modules are not loaded" errors after rebooting we need to edit /etc/modules and add zfs to the list of modules to be loaded at startup.

Since Debian Buster we have to run a lot of commands in a virtual console by pressing Ctrl-Alt-F1, that previously could be run in a terminal window. This is certainly the case for the zfsutils commands so the following steps need to be run in such a console.

The next step after installation is to look at the steps needed to create a filesystem on a disk and set up compression on it. In this case the existing disk is called spcbkup1 and its ext4 partition currently set up on it appears as /dev/sdb1 which means the actual disk is /dev/sdb.

After deleting the existing filesystem from /dev/sdb the following command creates the storage pool:
zpool create -m /mnt/backup/spcbkup1 spcbkup1 /dev/sdb

whereby the new storage "pool" (just a single disk which is not really what ZFS is made for but it can be done) is set up to mount to /mnt/backup/spcbkup1, is called spcbkup1 and is using the physical disk /dev/sdb.

Issuing the blkid command at this point tells me I have two new partitions. Apart from /dev/sdb1 which is of type "zfs_member" there is also another partition called /dev/sdb9 for some other purpose known to zfs. /dev/sdb1 is mounted up at this point and can be used as a normal disk.

To turn on compression we then use this command:
zfs set compression=lz4 spcbkup1

There is one more question and that is automounting. When you put an entry into /etc/fstab then the default for mounting is automatic. For a removable disk we don't want this, so the entries in /etc/fstab end up looking like this:
UUID=....     /mnt/backup/spcbkup1 ext4 noauto 0 2
whereby noauto as the options means it will not automount. Instead you have to manually mount it before use. Obviously you do this after inserting the disk. This requires me to hibernate or turn off the computer before inserting or removing the disk.

ZFS is a little different. We use zfs set mountpoint=none spcbkup1 to remove the mount point before removing the disk. Then later on when we want to put the disk back in we would use zfs set  mountpoint=/mnt/backup/spcbkup1 spcbkup1 to reset the mountpoint. As we can change mountpoints on the fly without a config file or needing to know a long UUID, this immediately and obviously lends itself to being able to mount more than one disk to the same mountpoint. In other words, I can dispense with a different mountpoint for each of the disks and mount them all to the same path. So for this backup scheme I will have two mountpoints for spcbkup. One mountpoint is for the backup disk(s) for the full backup, and the other is for the backup disk(s) for the incremental backup. These will look like:
/mnt/backup/spcbkup-full
/mnt/backup/spcbkup-incr

where the suffix is self explanatory hopefully
and as long as I swap the incremental disks in and out then the backups proceed automagically.

For this scheme we just have one backup disk for the full backup and we are sharing one backup disk for incrementals for all sources so the paths will look like
/mnt/backup/fullbackup
/mnt/backup/incrbackup

and the two incremental disk pools will be called incrbackup-odd and incrbackup-even which refers to the week number. The backup script will work out the week number itself and work accordingly. The plan is each incremental disk will contain two generations of backups and with two disks, there will accordingly be four incremental generations at any one time. In practice we take the week number and get the modulus after dividing by 4, which will give a number from 0 to 3, then the backup for that week is done into a folder on the disk called mainpc-x or serverpc-x or mediapc-x where x is the modulus. rsync will be set to ensure that any redundant files are deleted from the target at each sync and the week number is also used to calculate the date range for the command so that it knows how to handle file modification times to identify the files that have changed within that date range. The script automatically runs the same day and same time each week for the incrementals. At about every three months a full backup is done on a different day from the scripted day so the script is not disturbed, this amounts with three computers to one full backup each month.

It is important to unmount using the above commands before removing the disk otherwise the system will behave badly at next startup (although the Raidon disk caddies are theoretically hotswappable, in practice I am using them as non-hotswap for various reasons, so the system is shutdown or hibernated before changing disks).

So having worked out how to set up disks, the next step is to work out how to use rsync and I will be running a full backup of serverpc first of all.

Rsync Backup System [1]: Introduction

In February of last year I was looking for a new backup solution (I had used rsync to that point but found some issues with it) and tried a few different things. I have been using rdiff-backups since then (it comes as a part of Debian) but this has its issues too.

At the moment I can't recall what the issues were with rsync and it's looking more attractive as a new option. I think the reason I couldn't make it work satisfactorily before was because I used it over a network (samba) share which it doesn't work well over. But Linux has SSH as a much better option for network syncing which rsync has built in support for and should work a lot better than over a Samba network share. 

The reason I want to ditch rdiff-backup is that I want a system that will do incremental backups by copying the whole files and on a different disk volume. rdiff-backups works on the basis that you are backing up both full and incremental backups to the same volume and an incremental will only be the actual changed parts of files. Unfortunately this creates an incremental that is dependent on the full backup; if you lose the full version of the file then the incremental is of no use to you since you only have the file fragments that have changed, not the whole files that have changed.

The reason I chose rdiff-backup in the first place (over some options like borg, which I could not make work easily) was that it would copy a file structure (directories and files) instead of storing the files and folders inside an archive file. If for some reason the archive became corrupted it could be very difficult to recover the files inside it. There is an advantage that the archives are compressed but another option is to set up your backup disk to compress at disk level.  I tried doing this by formatting my backup disks as BTRFS but then the support for this changed in Buster and I am unsure if there are still issues with it. I will have another look at this especially if there is any option with compression support in ext4.

The idea at the moment is I would use rsync over SSH to back up each computer using a dedicated backup computer with a full backup about every three months and incremental backups about weekly in between. I have 2 TB disks for the full backups and 1 TB disks for the incrementals. Currently a WD Blue 2 TB disk suitable for a backup costs $105, while a 1 TB is $78. I'd need to get some more of the Raidon drive caddies for extra disks and the leather bags that they are stored in.

Rsync or anything that copies files and folders as is is a better more transparent option than using something that uses its own structure and archives that leaves you dependent on it being supported and developed as well. Rsync is very well optimised for syncing and the idea is it can do incrementals on top of existing full backups. rdiff-backup does the same sort of thing except that unlike rsync it will leave the original file unchanged, which rsync will overwrite (when resyncing to the same backup destination). There are a few possible schemes for backup. One possible is to put the backup computer in a remote location (not really possible in a house) and have it sync unattended a month's worth of data at a time then change the disk for another one and let it do another month, it would be copying all three computers' data on the same disk. Another option is two disks to do the incrementals with the disks exchanged weekly. This is the more likely scenario. Probably each disk will store 6 incrementals each at 2 weeks (the full backups happen each 3 months).

Anyway there is a bit of detail to work out and probably some more disks and trays etc to purchase to get this scenario all set up and hopefully it will come together soon.

Setting up a second swap partition for Linux

In my last post I waxed lyrical about the merits of the Linux file system and how it is possible to have two swap partitions. Today I am setting up just that in my computer. It has an existing SSD with 100 GB available for swap, and due to an upgrade on the other computer, another 120 GB SSD has become available to install into it to add a second swap partition thereby more than doubling the swap space. This means the system should be able to edit very large Gimp files up to 50 GB whereas at the moment it can only handle ones up to about 20 GB because of all the other stuff I usually have running on it.

The first step to achieve this, having installed the disk and found it is in the system as /dev/sdc, is to make life easier and install GPartEd, the graphical partition editor (preferable to using command line in this instance). We can then use GPartEd to remove any existing partitions on the disk, and then allocate the entire space on it as a swap partition.

The next step is to run the mkswap command on our new partition (although it looks like GPartEd in fact already did this step)

mkswap /dev/sdc1

And then we can use the swapon command to activate it:

swapon /dev/sdc1

after which running swapon -s will list the swaps currently in use which we find lists both of them and also conveniently the KDE system load monitor widget tells me the swap is now 198 GiB in size (it was 87 GiB and the new SSD added 111 GiB).

To permanently add the second swap partition it needs to be put into /etc/fstab:

UUID=83c0e622-bcdc-41fd-a1ce-ec3f6c88bfcc none            swap    sw              0       0which is basically copying the previous swap file entry except for the UUID which we get from running blkid

Then I tested by two Gimp projects at the same time that are around 40 GB in total. Originally the first one would not even load in the system, while the second one would use the entire swap space. Loading both together only used less than half of the new swap space, making it possible to work on them without any concerns about the system running out of swap and crashing Gimp.

 

Linux has the best filesystem ever :)

I've been in the IT industry for something like 30 years (longer if I was to include the time in my secondary education) and in that time I have used a few operating systems, starting with Apple DOS 3.3, UCSD P-System, CP/M, Acorn MOS, Macintosh System 6/7, Netware, MS-DOS, Windows (since 3.1), macOS and Linux. Obviously with the evolution of computer hardware, operating systems have evolved with better functionality, so we can't really compare anything current with the old Apple ][ or BBC Micro Model B. However, Windows and Linux can be compared as they are both current, and Linux has definitely stood out as having the superior architecture to Windows, which reflects the former's well designed structure and its roots in the high end mainframe computing, compared to Windows' basic architecture derived from exclusively PC basis.

So what is important and relevant on the Linux filesystem structure are the following things:

• Linux is engineered from the ground up to allow different parts of the core filesystem to be mapped to different drives or partitions. On Windows part of the core filesystem has to be on the same partition, C drive. Hacks to move, for example, the home drive to another partition, whilst provided for in a registry key setting, will actually break the service pack or version upgrade process, and therefore can only be used by technically savvy people who can manually engineer their own updates while backing up their home partition and migrating their data to the updated version. Since Windows 10 has frequent version upgrades, it is pretty well impossible for an end user to migrate the home drive.
  
  Linux, on the other hand, makes it very easy to map /home to a different partition. This is a real advantage when it comes to reinstalling, because you can keep the home partition intact and just reinstall the operating system and re-use your profile. That's another thing too - Windows will not allow you to re-use your existing profile when you reinstall. It will spew if you try to do that.
  
  I give my systems a SSD for OS install and put /home onto a RAID-1 hard disk array.
  
• In Windows your paging or virtual memory goes into a file. In Linux, swap goes into a dedicated partition by default. I suppose you can tell Windows to use only a special partition for its swap file, but the default is to put it on C drive. The issue with using a file is disk fragmentation, and potentially running out of disk space to store it, further exacerbated by being unable to relocate your home folder (see above).
  
• In Linux, you can spread your swap across more than one partition. This means I can simply expand my swap by adding another SSD into the system. That's about to happen in fact, so that I can have more virtual memory available for editing large Gimp projects. Windows has no such capability; the paging file all has to be on the same partition.

And there's a whole lot more after that, including the ability to support a whole pile of different disk formats; NTFS is pretty good but MS made it proprietary making it harder to exchange data between different types of computers.

•  
•  

NZ Rail Maps: Gimp optimisation with SSD works well

A few posts back in label NZ Rail Maps, I commented on how to optimise Gimp to use the swap partition on the computer effectively. I have since been able to prove using a computer with only 8 GB of RAM that it is able to use the SSD to load much bigger images without much of a performance hit and this validates my ideas about having SSD based swap to make Gimp perform well with very large projects that contain a lot of aerial photography data because this is how the mosaics get made in NZ Rail Maps.

The reason a large SSD based swap partition is becoming so important is that it speeds up the process of creating map mosaics from historical aerial photography. It also works well with Qgis when I am loading these large numbers of aerial photo layers into the GIS to draw the maps. However it is mainly Gimp that benefits, especially as I am taking advantage of the optimisation of resources to be able to create projects of increasing size.

Generally these large projects will cover a relatively large area. For example my largest ever project, currently around 24 GB, has 128 layers at present and this covers around 17 km of physical track distance (Dunedin to Mosgiel) with aerial layers present for the entire distance as well as other areas like the Fernhill and Walton Park branches and the first section (to Mosgiel Township) of the Outram Branch. There are still more historical aerial images to add to this map and so far the computer is coping quite well with working on it with about half the SSD in use for swap. So I expect this project will be able to grow quite a bit more yet.

Compare that with the previous largest project on this computer (although it's possible it may have had less RAM then) around 12 GB, but without optimising Gimp. The issue is simply if you leave that tile cache setting at the default then the Linux swap partition will not get used at all. Instead, Gimp will use its own swap space, which on my system is in the home volume. This means a lot of noisy and slow hard disk swapping. 

This essentially helps me to build a case for buying a larger SSD specifically for swap use. At the moment there is one SSD in the system which is mainly used for the root partition and some other OS stuff and most of it is a swap partition about 100 GB. A 240 GB SSD would create a large amount of swap space. The other option is to buy a 480 GB SSD for the NUC and put its disk into the computer to be used as a 240 GB swap volume. Anyway the cost of a SSD is around $80-120, so it is on the list of priorities to hopefully achieve soon.

Computing resources optimisation [2F]: Managing Disk Space Usage

The completion of this computer upgrade project is close to finished. I do have to complete mounting the fourth computer, which is inching along at a glacial pace at the moment, and various small tasks. One of the important ones is that mainpc's disks which have a total capacity of 1.8 TiB or 2 TB are nearly full. The main increase in capacity use is to do with mapping. This is to some extent the Maps folder and to a lesser extent Oracle virtual machines that have been used to run different software versions. Each time I download more aerial photography for maps it takes up additional space. I am satisfied the usage is appropriate so I now have to look at whether other resources can be released to make more space for various uses.

In the past I did look at moving some stuff like photos or other media to mediapc. I am not sure this is going to be easy or actually possible because right now mediapc's disks are nearly full and there doesn't seem to be an easy way of freeing up capacity. It will probably be a case of cleaning up a few folders here and there on mainpc especially where duplicated on mediapc. The Downloads and DownloadArchive folders on mainpc together total more than 300 GB and I think working through them regardless of how tedious or slow it is, is going to be worthwhile. At the very least there are large volumes of LDS downloads that can be cleaned up. It shouldn't be too hard to free up at least 100 GB relatively quickly and take some of the pressure off the fact I only have 132 GB free at the moment.

serverpc is the other major disk space gobbler with for the most part the aerial mosaics using nearly half of the total disk. With more and more mosaics being put together as the maps progress, I have to take a hard look at whether I need to keep so many files. Cleaning up downloads will also yeild space savings on that computer. With it now having 32 GB of RAM there is less demand for swap space in the past and it does have the SSD available for swap, but still it is currently down to less than 100 GB free space available. It is actually so simple and quick to make mosaics that I don't really need to keep the last three files of each one and possibly I can cut back to just the most recent file.

Cleaning all these up will also speed up backups. With mediapc becoming the backup host, I have to get it set up to run some backups which are very overdue of late.

Well as it turned out  I was able to free up 400 GB on mainpc with only a little effort and easily get that on serverpc as well but it will soon be gobbled up with more mosaics so I will have to get creative with deciding how many mosaic files I need to keep on serverpc. The backups are underway again but I am only keeping one generation of backup for serverpc and two each of the other two computers, it can be slow as a full backup can take more than a day which is why I have a dedicated computer, in this case mediapc, which is running the backup while I can keep using the other computers that are being backed up as normal. Once mediapc has finished doing the backup of mainpc it will be reinstalled with Debian 9.6 with KDE on top as I only need one computer running XFCE and that will be playerpc. In the process mediapc gets its swap put back onto using all the SSD space as there have been issues that it can't hibernate at the moment and it needs to have that swap available if it is doing big tasks with only 8 GB of RAM. I am meanwhile pushing along finishing the installation of playerpc under the desk.

Backup software / recap

About eight years ago I thought I was building a really special PC with a Intel DG41RQ mainboard, Celeron E3300 CPU and 2 GB of RAM. The article series started here: https://enzedtech.blogspot.com/2010/10/rebuilding-my-pc-1.html

Unfortunately I outgrew that computer pretty quickly because it could only take 4 GB of RAM in total and therefore became pretty slow with later versions of Windows. However it would work fairly well under Linux. I often wondered why I bought a LGA775 system that was right at the end of that generation of CPU, and it looks like cost was an issue. The Celeron E3300 did support hardware virtualisation though (that was something that would have been useful if the computer had enough RAM, as 4 GB was barely enough just for Windows).

These days I try to do a better job but at times there can still be a price/performance tradeoff to consider, the main reason why I have a serverpc that only has 16 GB of RAM in it because the board only has two memory slots. Well we could put 16 GB DIMMs into the two slots but would have to throw away the existing RAM. Most of the time I try to buy something with four slots, and that's why the computer this is being written on has 32 GB in it.

Now to focus on good free backup software for Windows. I may have written about this before but that particular post isn't easy to find. I get asked this question from time to time so here is an answer. The two best free packages in my view are NCH FileFort and Cobian Backup. FileFort is a commercial package that has a free edition for home use, while Cobian is just free. I used to use Cobian to backup my systems when everything was running Windows, but of course now I don't have a need for it.

FileFort is perhaps a little more user friendly than Cobian however whereas Cobian will simply stop reporting error messages if there are too many, FileFort's problem is it will stop running if there are too many errors. This is annoying because you should be able to just check the errors later (they may happen for reasons like being in use as not all software packages can handle this well). Cobian also has the ability to use VSS which means it can copy in use files. So Cobian is probably the better of the two, however FileFort is probably more up to date (Cobian does work with Win10 so it's OK for a few more years, anyhow).

NFS on Linux

When it comes to file sharing on Linux, you have two main choices. NFS is the one that is intrinsic to Unix environments, that have been carried down into Linux by convention. It has been around for a long time and is quite stable and reliable. Since the development of Samba, you also have SMB as an option both for Windows and Linux clients. I have used Samba quite a lot because I still have one Windows client in my network, but I am interested in experimenting with NFS because I have seen some issues on my Linux desktops with Samba and I want to know if NFS is more reliable.

So I will try to sum up the process of installing NFS and making it work on a Debian 9 system. As with Samba, mainpc will be both a client and a server for NFS, sharing the maps share.

There is a guide on the Debian wiki which I accessed here:

• https://wiki.debian.org/NFSServerSetup

I also referenced this guide on the HowToForge website:

• https://www.howtoforge.com/install_nfs_server_and_client_on_debian_wheezy

Here is another site I also read:

• http://bencane.com/2012/11/23/nfs-setting-up-a-basic-nfs-file-system-share/

So the first step is to install the NFS server (kernel mode version)

apt install nfs-kernel-server

Note that you may see errors about the nfs-server services. This seems to be a common issue with systemd. I checked with systemctl and found that nfs-server.service had the status code active (exited). This seems to be indicating there were no issues with this as a service. However, the (exited) bit means the service is not running at present because it is not fully configured, mainly there are no exports (another name for shares) set up. When you have fully configured it, then restarting the service is necessary to get the exports available for clients to connect to.

The next step is to install the rpcbind package (this replaces portmap which may be referred to in older documentation). In my case this was already installed as part of the install of the server package.

I didn't initially config rpcbind to anything even though the instructions for portmap specify some configuration settings for it. 

I also installed the client on another computer and created a mount point for mounting the maps share to. This re-used /mnt/share/mainpc/maps because this is the common network access path for software, even though this one has been used for Samba, I need to keep it as a common path for file access from Qgis. Therefore I edited the fstab to remove the Samba mount using this same location. To install the nfs client you just need this command:

apt install nfs-client

Next is to set up the exports on the server, by using nano to edit the exports file. To allow my local subnet to access the maps share on the NFS server I need this line in the file:

/home/xxx/yyy 192.168.1.0/24(rw,sync)

Note there is not a space between the IP address specification and the options in brackets.

After entering that I need to run a command called exportfs to refresh the exports. 

exportfs -a

On the client the entry in /etc/fstab looks like

192.168.x.y:home/xxx/yyy /mnt/share/mainpc/maps    nfs    defaults     0    0

In this case I did a manual mount first, then on finding it has worked, I used mount-a to test further. This also worked.

The main restriction I have noted at the moment is in creating directories and files at the root of the share. There may well be some sort of configuration option I need to set up to allow this because there hasn't been a problem up until now with this option. However, it's not a deal breaker. At the moment there is no problem at all caused by this issue.

The main issue has been with virtualbox VM clients that are using different IP address ranges by default and have to be configured to use the same IP address range so the server will allow client connections from the virtual machines. I could either address this by allowing extra address ranges in the server, or, as I chose to do, configuring the virtual machines to use a bridged adapter instead of a NAT adapter, so that it gets an address from the DHCP server in my network and is in the right address range.

This was going to be a multipart series but it hasn't turned that way, because NFS just works, and there have been no issues. Obviously there is no password by default, but that isn't an issue on my network.

Backup setup commands (btrfs, rdiff-backup, ssh)

Once btrfs-tools is installed these are the commands to create a single partition volume and add a filesystem to it. First step is to wipe the existing partition table with wipefs, then use mkfs.btrfs to create the filesystem. This is the best way to create a backup disk, even if it is for more than one PC. A single partition shared by multiple repos doesn't waste space like a separate partition for each repo would.

• wipefs -a <device>
• mkfs.btrfs -L <label> <device>

After creating the filesystem, use blkid to find out the UUID of the new filesystem and add a line like this to /etc/fstab 

• UUID=... /local/mount/path btrfs compress,noauto 0 2

I use paths off /mnt/media to mount my removable disks. If you want to backup as something other than root then be sure to give permissions to the user account concerned, chown is one simple way of doing this.

With the above line in fstab the volume will not automount which is handy for a removable disk that might not be present at startup. Just type mount /local/mount/path to mount the disk when it is inserted. The mount line should ensure files are compressed when they are stored or updated but it leaves the type of compression to be chosen by the default setting of btrfs tools.

---

The next step we want to cover is how to set up ssh for passwordless connections to the remote computer. For my setup, serverpc is the computer that will host every backup. When I need to change disks, I can quickly shut down serverpc and then bring it back up again (the disks are supposed to hotswap but they seem to fail quite quickly so I have been given to coldswapping).

This means serverpc needs the openssh-client package installed and each target computer (mainpc and mediapc) needs openssh-server installed.

My current backup scheme assigns a pair of 2 TB disks to backup mainpc and a combination of a 2 TB disk and two 1 TB disks to backup mediapc and serverpc so there are two sets of disks for each. And that's without making use of rdiff's differential backup capability for multiple generations per disk.

Using the instructions from my previous post I need to follow these steps after installing the respective packages on each computer.

• On serverpc create the key pair for SSH:
• ssh-keygen -t rsa
• The file will be saved in .ssh/id_rsa
• Press enter to put in an empty passphrase when prompted (twice)
• Use serverpc to ssh into the target (mainpc in this example) and create the directory there for the key to be copied to.
• ssh user@a.b.c.d mkdir -p ssh
• Enter yes when asked to continue connecting
• Enter the password for that user on the remote system when prompted
• Copy the public key from serverpc to target
• cat .ssh/id_rsa.pub | ssh user@a.b.c.d 'cat >> .ssh/authorized_keys'
• Enter the password for the user on the remote system when prompted
• Ensure the correct permissions are set on the remote filesystem
• ssh user@a.b.c.d "chmod 700 .ssh; chmod 640 .ssh/authorized_keys"
• Enter the password for the user on the remote system when prompted
• Test the login to see that no password is needed
• ssh user@a.b.c.d

---

Finally I can do my backups with rdiff-backup. All computers need to have the rdiff-backup package installed and the same versions preferably.

Usually what I do is mount the disk and then change into the directory I want the backup to go into. Then I can use these types of commands to do the backup:

• To backup one of the remote systems using SSH to connect to that system
• rdiff-backup -v5 user@a.b.c.d::/path/to/files .
• This is telling rdiff-backup to use verbosity level 5, and the files will be backed up to the current directory, assuming I changed to that directory as mentioned above.
• To backup serverpc's local files:
• rdiff-backup -v5 /path/to/files .
• Again we assume the current directory is the backup path.

rdiff-backup has been generally good to use but there has been one instance (on the PC doing the backups) of for some reason the backup disk not being mounted properly. We mount disks to a path off /mnt which is hosted on the installation volume. The problem is when no mount exists, this just looks like an ordinary directory and stuff being copied to there will fill up the install disk instead of the disk that is supposed to be mounted. As I couldn't identify any files to be deleted in the path, I was forced to reinstall the OS.

As a precaution, when I reinstalled Debian, I set up a separate partition for /mnt on the backup host computer,  with a size of 100 MB.

New bits for "old" computers [12]

The Win10PC has got itself a bluetooth keyboard now as well as the wireless mouse it already had. The Logitech K380 bluetooth keyboard works quite well and is nice and compact, it can be used with up to three devices and has a battery life up to 24 months. I had an existing Promate bluetooth receiver that is plugged into one of the USB ports on the back of the computer and which works well with the built in Windows 10 Bluetooth drivers. 

After looking at a lot of backup options I have settled on rdiff-backup as being the best solution for what I want to do, since it will actually produce something that looks more like a mirror, unlike all these deduplicating solutions the weakest feature of which is losing one file chunk can result in losing a lot more than one file. I am now trialling rdiff-backup with a backup set from mainpc to see where I can go with it.

I have also installed Btrfs on my computer so as to be able to compress the backup volume as this is the simplest way of getting more backup data onto a disk. rdiff-backup can do multiple generations of backups with its differencing system, so this is worth looking at, even with two main backup disks. rdiff-backup supports SSH, so I might also have that option to be able to make a backup happen via serverpc as originally I had considered, and even multiple backups per disk.

Other options for transparent compression include OpenZfs, but it is really a bit too complex for me to get my head around, so I will just be using Btrfs for now. After following some tutorials online I got SSH working with passwordless login and immediately realised why I never could get it working with borg. Simply because the borg examples failed to clarify that the command was to be issued on the target rather than the source whereas I thought it would be the other way round. However, as I have determined since, borg isn't the preferred backup solution for my needs and I am happy with using rdiff-backup. At the moment serverpc is backing up the data from mediapc and after that I will test it doing its own backups.

• https://www.tecmint.com/rdiff-backup-remote-incremental-backup-for-linux/

New bits for "old" computers [10]

As I detailed recently I have changed my desk layout to allow four keyboards to be used easily and have also changed the screen layout to suit. I had four wireless keyboards to make this happen. Unfortunately the MS multimedia keyboard I use with the win10pc has gone wonky and having go slows and keys randomly sticking and repeating so that has gone in the rubbish.

Since it is hard to get enough different wireless keyboards and especially multimedia keyboards for my layout I have decided not to bother with replacing it. Instead the left hand keyboard slide (the one where I often have drinking cups/glasses) will be adapted to become wider and deeper so it can double as a storage/usage location for the Genius mini USB keyboard I have. This will be used with a regular wireless mouse for the relatively small amount of time that I need to use this setup, since I can always switch one of the two main keyboards to this computer. 

The regular backups will be started this weekend. serverpc will in fact act as a server for all the computers but they will each have their own repo, which for mainpc will be on one particular 2 TB backup disk, while serverpc and mediapc will have a repo each on a shared 2 TB disk. Alternatively I may stick with a disk for each computer, so as to have spares, the main issue being not enough disks at the moment. This arrangement is preferred because serverpc can be easily shut down to change disks without disrupting things as can happen with disks having to be inserted in the removable bay and brought online.

The amount of disk space on mainpc for a complete set of aerial photos for a rail corridor 235 km long has turned out to be about 2.4 GB. This is very good so the required files can be kept on mainpc with the rest of the map project files, serverpc will be used to store the bulk downloads before the images actually needed are selected. So I am budgeting on having 100 GB of disk space used on mainpc for all the maps which is quite reasonable with its disk capacity at present.

The living room is getting a makeover at present to optimise all the things I do in here and a lot of stuff that doesn't need to be in here will get stored "temporarily" in the bedroom until I get that properly tidied up, as I need to make the living room the daytime space for everything to save on winter heating costs, using the aircon to make the room temp 16-17 degrees and wearing a few layers of clothes to avoid having to heat the room more than that.

KDE vs XFCE

Whilst I have been generally happy with both Xubuntu and Debian with XFCE as the GUI, there have been a number of issues running programs under XFCE that have caused me to doubt its capabilities. Mostly these surface with Qgis because of the nature and extensive use of the software and are in three two areas:

• The right click or shortcut menu is almost impossible to open without accidentally clicking the menu item closest to the mouse pointer. This has been mostly under Qgis but I have lately been seeing it happen under other applications as well. This has been confirmed not to be a XFCE specific issue and is possibly application implementation specific.
• Drag and drop in many cases is poorly implemented, that in many cases you will be dragging an item even when you aren't holding the mouse button down. This causes a lot of problems with the Qgis layer list which supports drag and drop to reorder items. Frequently layers are inadvertently dragged and dropped to wrong places in the list.
• Double click also seems to be poorly handled and in many cases single click is taking the action where a double would normally be needed.

Up until now apart from filing a Qgis bug report concerning the right click issue I haven't done much, but seeing the right click affecting another piece of software in the last couple of weeks has spurred me forward into creating a Kubuntu virtual machine with Qgis installed for testing. So far the drag and drop issue seems to be resolved, but I need to spend a lot more time using it to be really sure. Understandably I will be testing the other issues on it as well.

This Kubuntu VM will be used on mainpc instead of the natively installed Qgis on that computer so that I can get as much testing as possible done with it over the coming weeks and that will give me time to look at whether to permanently adapt my Debian computers, probably just mainpc and serverpc at this stage, to KDE. This would probably happen when Buster comes out. There is no pressing need to upgrade all the computers at this stage.

In regard to Qgis, I have previously looked into making the raster aerials usage more efficient by scripting the copying of only the images used by extracting from the QLR file in XML. I have instead gone for a much simpler process of a simple command script that copies the manually entered filenames (with as much abbreviation as possible) running on the Windows 10 PC, to copy only the required imagery from serverpc to mainpc for each section of rail corridor that I have aerial imagery for. Hopefully this will soon progress to all the aerial imagery I have for the various corridors, and without gobbling hundreds of gigs of disk space as was happening before.

The need to come up with a backup solution is becoming more pressing and will, I think, simply consist of local Borg backups on each computer. The idea of a backup server is not well enough supported by Borg at present to work well with multiple PCs. In addition there are now problems with OverGrive which is a Linux sync client for Google Drive, which I rely on to do backups of map projects and data files. Google has changed its API yet again and typically they will drop support for an old API at the same time so I have been shocked to discover OG has no mechanism for flagging upload errors which are being reported in its log. It turns out the OG developers have abandoned it.

UPDATE: Because buster is probably as much as year to 18 months away I am looking at putting KDE onto mainpc as a replacement UI install, rather than doing a completely new install. It takes a bit of work to make it the default replacement for XFCE. I am also going to test out LXQT in a virtual machine, this is like LXDE but updated to use QT components and therefore has a more modern appearance than LXDE. Both environments will be tested in VMs before installing them, I am about to build a Debian 9.4 VM for LXQT testing.

I have decided to continue the "backup server" model with Borg but there will be separate disks for two different backups. One 2 TB disk will be mainpc by itself and the other will be for mediapc and serverpc combined on one 2 TB disk. It may be that I need to buy some more disks, but at the moment there isn't really any money to spend on these. I will begin testing out Borg for this very soon. There is a free replacement for OverGrive but without the automatic detection for uploading so I have to think about how to install it.

New bits for "old" computers [9]

Unfortunately I have not attended to any of the tasks in my last post yet because some aerial photos I was working on last week for mapping have to be redone. This time I have just closed off all web browsers and other applications in mainpc and used that one for Gimp so its full memory can be taken advantage of. This has sped things up a bit and the mediapc has been pressed into use for email reading and Facebook etc. Being able to copy layers between different Gimp projects has helped roll the aerial photo refactoring along and although I am not doing any imaging on serverpc as yet it will get pressed into use in a day or two when I am ready to re-render some of these aerials.

So while that is happening only a little map work has been taking place, mostly finding and marking km pegs from the Otago Central rail trail onto a map and also a special layer in Qgis, getting these aerials finished is a high priority and using mainpc to work them will speed things up a lot.

So getting the backups and stuff started has been put off yet again but it is getting more urgent by the day and as soon as the bulk of this refactoring has been pushed through and things slow down a little then the delayed weekend project combining tidying up a lot of things will actually come to pass.