Setting up a second swap partition for Linux

In my last post I waxed lyrical about the merits of the Linux file system and how it is possible to have two swap partitions. Today I am setting up just that in my computer. It has an existing SSD with 100 GB available for swap, and due to an upgrade on the other computer, another 120 GB SSD has become available to install into it to add a second swap partition thereby more than doubling the swap space. This means the system should be able to edit very large Gimp files up to 50 GB whereas at the moment it can only handle ones up to about 20 GB because of all the other stuff I usually have running on it.

The first step to achieve this, having installed the disk and found it is in the system as /dev/sdc, is to make life easier and install GPartEd, the graphical partition editor (preferable to using command line in this instance). We can then use GPartEd to remove any existing partitions on the disk, and then allocate the entire space on it as a swap partition.

The next step is to run the mkswap command on our new partition (although it looks like GPartEd in fact already did this step)

mkswap /dev/sdc1

And then we can use the swapon command to activate it:

swapon /dev/sdc1

after which running swapon -s will list the swaps currently in use which we find lists both of them and also conveniently the KDE system load monitor widget tells me the swap is now 198 GiB in size (it was 87 GiB and the new SSD added 111 GiB).

To permanently add the second swap partition it needs to be put into /etc/fstab:

UUID=83c0e622-bcdc-41fd-a1ce-ec3f6c88bfcc none            swap    sw              0       0which is basically copying the previous swap file entry except for the UUID which we get from running blkid

Then I tested by two Gimp projects at the same time that are around 40 GB in total. Originally the first one would not even load in the system, while the second one would use the entire swap space. Loading both together only used less than half of the new swap space, making it possible to work on them without any concerns about the system running out of swap and crashing Gimp.

 

Linux has the best filesystem ever :)

I've been in the IT industry for something like 30 years (longer if I was to include the time in my secondary education) and in that time I have used a few operating systems, starting with Apple DOS 3.3, UCSD P-System, CP/M, Acorn MOS, Macintosh System 6/7, Netware, MS-DOS, Windows (since 3.1), macOS and Linux. Obviously with the evolution of computer hardware, operating systems have evolved with better functionality, so we can't really compare anything current with the old Apple ][ or BBC Micro Model B. However, Windows and Linux can be compared as they are both current, and Linux has definitely stood out as having the superior architecture to Windows, which reflects the former's well designed structure and its roots in the high end mainframe computing, compared to Windows' basic architecture derived from exclusively PC basis.

So what is important and relevant on the Linux filesystem structure are the following things:

• Linux is engineered from the ground up to allow different parts of the core filesystem to be mapped to different drives or partitions. On Windows part of the core filesystem has to be on the same partition, C drive. Hacks to move, for example, the home drive to another partition, whilst provided for in a registry key setting, will actually break the service pack or version upgrade process, and therefore can only be used by technically savvy people who can manually engineer their own updates while backing up their home partition and migrating their data to the updated version. Since Windows 10 has frequent version upgrades, it is pretty well impossible for an end user to migrate the home drive.
  
  Linux, on the other hand, makes it very easy to map /home to a different partition. This is a real advantage when it comes to reinstalling, because you can keep the home partition intact and just reinstall the operating system and re-use your profile. That's another thing too - Windows will not allow you to re-use your existing profile when you reinstall. It will spew if you try to do that.
  
  I give my systems a SSD for OS install and put /home onto a RAID-1 hard disk array.
  
• In Windows your paging or virtual memory goes into a file. In Linux, swap goes into a dedicated partition by default. I suppose you can tell Windows to use only a special partition for its swap file, but the default is to put it on C drive. The issue with using a file is disk fragmentation, and potentially running out of disk space to store it, further exacerbated by being unable to relocate your home folder (see above).
  
• In Linux, you can spread your swap across more than one partition. This means I can simply expand my swap by adding another SSD into the system. That's about to happen in fact, so that I can have more virtual memory available for editing large Gimp projects. Windows has no such capability; the paging file all has to be on the same partition.

And there's a whole lot more after that, including the ability to support a whole pile of different disk formats; NTFS is pretty good but MS made it proprietary making it harder to exchange data between different types of computers.

•  
•  

Tamron AF 70-300 Di LD telephoto lens for Canon EF-M?

UPDATE: Since first writing this I discovered I can get the EF-EOS M adapter from Canon for $199. Although having to add this to the camera won't really help to make adding lenses to the EOS-M affordable, it would give me the option to use the telephoto lens from my existing EOS 600D with the M100. Assuming it actually is available at that price, it would have fallen sharply in cost from the original pricing which was around $500-600 for the adapter.

One of the obvious factors about owning an interchangeable lens camera is that we expect to be able to change lenses when we need something like a telephoto or wide angle instead of a standard lens. Since I own a Canon EOS M100 camera, I'd hope to be able to fit it with a tele zoom one day and be able to take photos at considerably greater distance than the lens that comes with it, which is a Canon 15-45 mm collapsible zoom. The Canon zoom lenses for the EF-M mount, however, are very expensive, priced around $600, which is almost as much as I paid for the camera itself, making it inevitable I'd want to take a look at a third party lens. 

Industry commentators have long remarked on Canon's extreme reluctance to threaten their profitable D-SLR market segment with mirrorless cameras, given the less than inspiring design and high pricing of the early EOS M models. Canon has insisted this is not the case and have brought out a much wider range of EOS M models with lower pricing in recent years. However, there is only a very limited range of extremely expensive Canon lenses and accessories produced for the EOS M models, compared to what is available for the EOS D-SLR in the standard EF mount. The cheapest Canon EF-M zoom lens costs around $550 retail. It's very clear that Canon is only prepared to position the EOS-M series as high-end compacts with deliberately strategisation to discourage end users from looking for an interchangeable lens camera at a cheaper price than their DSLR range.

Current pricing on the Tamron AF 70-300 Di LD lens is around $287 which is a very low price for a lens (the actual range of the zoom may be somewhat different as it will vary depending on the crop factor of the body it is fitted to, this is unclear at the time of writing). At that price it is made of plastic like the Canon kit lenses. There are a few other limitations that you would expect with a cheap lens like a front element that rotates (tricky with some types of filters that are angle dependent), slow autofocus, no image stabiliser, sticky zooming, blurry corners, some distortion, telephoto fringing / softness. The results will depend on the type of camera it is attached to, with a smaller APS-C sensor the edges of the lens are missed which gives better quality in many respects, but does not overcome the mechanical / design limitations.

The main issue is that I am finding it difficult to discover if this lens is actually available for the EOS M. If it should turn out that Canon has placed restrictive licensing on third party lenses for the M mount it will achieve their strategy of deliberately strangling the market for EOS M lenses. So far it looks like Tamron's only offering is the much more expensive 18-200mm F/3.5-6.3 Di Ⅲ VC, which whilst being a better quality lens, ends up costing practically the same as Canon's lenses. It must be apparent that Canon has placed severe restrictions on the production of affordable third party lenses for its EOS M cameras.

Python Scripting [6D]: Layer Sidecar Duplication & Renaming for NZ Rail Maps 4

Since we last looked at this topic a couple of months have gone by with everything working as expected when the duplicate script is used, both for duplicating sidecar files for layers that are the same size as the original and ones where the destination layer has been increased in size. The former is the most common usage although the latter case was the original reason the script was devised, but being able to carry out simple duplication for layers that are the same size as the original has proved extremely useful as this was originally a manual task requiring copying the files and renaming them which was tedious even allowing for Thunar's built in bulk renaming.

There has now been one refinement added to the original script which is an additional command line parameter -o or --overwrite which is for the purpose of specifying that existing files can be overwritten. If this parameter is included in one of the forms shown (no value required) then overwriting will be allowed, otherwise it will not be allowed.

To implement this in Python we have to add an extra line to the argument parser initialisation code block:

parser = argparse.ArgumentParser(prog='duplicate')
parser.add_argument('-s', '--source', required=True)
parser.add_argument('-d', '--dest', required=True)
parser.add_argument('-o','--overwrite', action = 'store_true')
parser.add_argument('-m', '--multisuffix', type=str, default="")
parser.add_argument('-p', '--pixelsize', type=float, default=None)

(the new line being the one in italics, the rest are existing)

The difference being that as no parameter value is passed with the command line parameter, an action must be specified, which is 'store_true'. The way this works is that if the parameter is specified then its value is set to True, otherwise it is set to False. (I am not sure how it comes to be False but this must be some sort of default or something ???)

Then we collect the parameter value with all of the others:

overWrite = args.overwrite

Since there is a block of a number of lines of code used to write the world file that would have to be duplicated, we put this block into a function and then call it as required by function name. So at the top of the script there is this function definition:

 # function to write new world file
def writeWorldFile(wFPName,wFLines,pSize):
    # write new world file
    wFile = open(wFPName, "w+")
    if  pSize is None:
        wFile.write(wFLines[0] + "\n")
    else:
        wFile.write(str(pSize) + "\n")
    wFile.write(wFLines[1] + "\n")
    wFile.write(wFLines[2] + "\n")
    if pSize is None:
        wFile.write(wFLines[3] + "\n")
    else:
        wFile.write("-" + str(pSize) + "\n")
    wFile.write(wFLines[4] + "\n")
    wFile.write(wFLines[5] + "\n")
    wFile.close()

Then when we are going to write the world file it looks like this:

           # copy world file
           #  first read existing world file
            worldFileName = fileNameBase + ".jgw"
            worldFilePath = os.path.normpath(rootPath + "/" + worldFileName)
            worldFile = open(worldFilePath, "r")
            worldFileLines = worldFile.readlines()
            worldFile.close()
            WFNNew = fileNameNew + ".jgw"
            WFPNew = os.path.normpath(rootPath + "/" + WFNNew)
            if overWrite:
                    print worldFileName + " -> " + WFNNew
                    writeWorldFile(WFPNew,worldFileLines,pixelSize)
            elif not os.path.exists(WFPNew):
                print worldFileName + " -> " + WFNNew
                writeWorldFile(WFPNew,worldFileLines,pixelSize)
            else:
                print WFNNew + " exists --------------------"

That could be simplified and made more efficient by putting the read operation into another function and only calling it when actually needed. At the moment the read is performed even if the write is not performed. 

The code for duplicating the two xml files is basically the same for both and the code block looks like this:

            # copy xml files
            auxFileName = fileNameBase + ".jpg.aux.xml"
            auxFilePath = os.path.normpath(rootPath + "/" + auxFileName)
            AFNNew = fileNameNew + ".jpg.aux.xml"
            AFPNew = os.path.normpath(rootPath + "/" + AFNNew)
            if overWrite:
                print auxFileName + " -> " + AFNNew
                shutil.copyfile(auxFilePath,AFPNew)
            elif not os.path.exists(AFPNew):
                print auxFileName + " -> " + AFNNew
                shutil.copyfile(auxFilePath,AFPNew)
            else:
                print AFNNew + " exists --------------------"

 

The same block is duplicated in the script with the .xml extension instead of .jpg.aux.xml

This could be made more efficient by putting this entire block of code into a function and simply passing the two different file extensions to it in two separate function calls.

Probably after doing more testing I will tidy the code up further along the suggested lines.

There is also an issue with some of the aux files possibly not being copied because every so often I will see a "CRS was undefined" message from Qgis when I load a duplicated layer. I have not yet tried to work out what is happening but the information about the CRS is stored in one of the xml files and there must be an issue happening with the file copy operation or something else happening that I haven't worked out yet.

Python Scripting [4D]: Exif Based Image Renaming 4

After testing the Exif renaming script for the past couple of weeks I have added a piece of code to check for collisions and rename any file that collides with an existing filename. With Exif data, the most likely cause of a filename collision is when a camera is set to shoot continuously and it takes several pictures a second. If the camera doesn't set the subsec time field then the exif string for the time the photo was taken will be identical for several photos and if used as the basis of a filename the result can be identical filenames. Another possibility is you own two cameras of the same type and you let someone else use the second one at the same time as you are using the first one (for example there are two of you covering a big event) so you can possibly get duplicates that way.

When I used to use IrfanView it had some sort of collision handling function built into it based on the one that Windows Explorer uses, which as some will be aware handles copying files to a new destination when it puts an extra number in brackets on the end of a filename. Apart from the fact that on Linux we obviously don't have the use of Windows Explorer, this wasn't able to be accessed for exif string renaming, so I wasn't able to use it and had to manually rename colliding filenames. There are a few older directories in my archive of photos on the computer that do have duplicates which in that case came about by the files being copied to somewhere else and then being copied back to the original folder, in this case the duplicate has an extra string of numerical digits on the end of the filename, which was what happened after a collision had occurred with the first time of copying some files, a second copy operation was run using the IrfanView capability to add a sequence number to the end of the filename.

So for this script I planned on adding a filename collision handling functionality to the code to deal with these possibilities. This has consisted of writing a function and calling the function from within the main script because it is needed in two places. Functions are fairly easy to do in Python and like any other programming language they are for more than just where you need to reuse a block of code. There is in fact a strong case to write all of your code in functions and then the main execution block just consists of a series of function calls and is very neat and tidy. I haven't done this yet with any of my scripts but I will start doing it with the next project.

The function is fairly straightforward and looks like this:

def fixCollision(fn): # handle filename collisions. fn is full filename with path
    p = os.path.splitext(fn)
    f = p[0]
    e = p[1]
    x = 1
    ff = fn
    while os.path.exists(ff):
        ff = f + "-" + str(x) + e
        x += 1
    return ff  

The first line is the file definition and an explanatory comment. The subsequent lines of code are focused on splitting the filename into its base and extension, and setting the initial value of the collision counter x. We then set the initial value of ff, the complete filename string that is tested in the while loop, to the filename that was passed into the function, and then enter the loop where the while clause checks to see if the filename exists. If it does then it makes a new filename by inserting the value of x with a dash separator between the base and extension, increments x for the next time around, and then goes back to the top of the loop. As soon as a filename is found which doesn't collide (which may include the filename that was passed into the function) then it exits the loop and it returns the filename to the calling code.

This is called within the script in two places (the code blocks that handle images and non images) as follows:

                    destFile = os.path.normpath(destPath + "/" + destName)
                    if not os.path.exists(destPath):
                        print "Create destination path " + destPath
                        os.mkdir(destPath)
                    destFile = fixCollision(destFile)

This code block is just before the move operation that moves the source file to its destination and renames it at the same time. In other words where we have a destination to move the file to and we move it to the new location and give it its new name as well. The first four lines in that block are creating the destination file path from the new direction and the new exif-based file name string (or a different process for non-exif files), checking for and if necessary creating the destination directory, and calling the collision resolution function. Essentially this results in ensuring the destination filename is unique within the destination folder.

The only other issue which has come about from the script is with the file permissions for the originals off the camera. As far as the computer and me are concerned, I only have read permissions to the photos and that is unchanged by the script. I am currently considering whether the script should change the permissions on each file as it is renamed. However there is an advantage in preventing the files from being changed in the Photos folder as they should be copied before any alterations are made and it may be that I will just leave things as they are.