Firefox Developer exhausts system resources

One week after reinstalling mainpc I had a repeat of the situation that caused the reinstallation. Firefox Developer, which is the main browser I use, somehow got into a situation where it was using full CPU, RAM and swap for a period. I use the KDE system load monitor widget on the panel to monitor what the system is doing on my computers that have KDE. The system slowed right down and I noticed all three bars on the widget were all the way up. After a period of the system being bogged down and almost impossible to use due to being maxed out in resources, Firefox Developer crashed (or at least the instance that has the most tabs open, the other instances stayed open) and then the bargraphs all dropped to almost nothing.

Unlike the previous occasion the /tmp space was not all consumed so everything is still working properly. Possibly this is because I put /tmp onto its own partition and possibly also because the system had 100 GB of swap available after the reinstall. All the disks had the usual amount of free space available.

When Firefox was reopened there were no obvious issues and the resource use stayed low. However it is worth noting that the open tabs won't all be reloaded at startup, they are reloaded when they are clicked on. I did go through them later to reload and saw no issues with resource use.

It looks like there is some sort of scenario where Firefox can "run away" and crash after using up all the free resources. The whole thing is a bit bizarre but it has happened twice in one week and so I will have to see what I can do to try to find out exactly what is going on. But this could be quite hard to achieve. However I can guess that it probably leaks memory. It probably doesn't release or reuse memory used by a tab when the tab is closed, so it eventually uses up all the memory in the system, crashes and releases it all, then when it reopens, of course it doesn't need all that memory that it had.

The thing you see in KDE's system activity monitor if you open it up (press Ctrl and Esc together) is a few processes called firefox-bin which are the individual browser windows, and a number of processes called Web which don't mean a lot but are using significant amounts of memory, CPU or whatever. When I dig further in, Web  turns out to be using Firefox library files, so it must have something to do with Firefox. The common factor in most cases is using libxul.so, a library that is installed as part of Firefox Developer. 

At any rate this seems to be yet another example of the enormous resource hog that web browsers just seem to all be by default. All the tabbed ones seem to just use ridiculous amounts of memory in the computer.

UPDATE: The process that crashes is called Web Content (as top shows it) and it may only crash a tab rather than the whole browser. I currently have a bug report lodged into the Bugzilla database to see if it can be addressed as a bug in the browser, as I have been able to replicate this with one particular web site which crashes Firefox Developer every time.

General update 2019-02-15

Last week I had some issues with mainpc resulting in it running out of /tmp space which of course is caused or impacted by running out of space on the install volume and caused a number of user files in /home I was working on to become corrupted. I haven't looked at the bigger question of how we can prevent issues like this from bringing down the whole system or how it actually happened. Instead I just chose to reinstall the system from scratch.

There was a delay to getting this done because issues with my internet here created problems for the Debian installer trying to connect to the local mirrors to download install files so in fact mainpc was down for something like 5 days but with the other computers available I was able to keep working to an extent with the main constraint being unable to access any files over the network shares from mainpc.

Key objectives of the reinstall include:

• Recover the entire free space on the SSD for swap file. Previously an attempt was made to allocate half of the swap to Gimp specific temporary/cache allocation. This proved difficult to implement so I decided to revert the full space (99.9 GB) back to swap as Gimp does use this space to cache as needed.
• Allocate /mnt to a separate partitition on the SSD like done to mediapc to ensure there is no possibility of a mount failure allowing files to be written to the SSD instead of a mounted volume on another disk filling up the install volume.
• Allocate /tmp to a separate partition to ensure there is no possibility of a filling up of the rest of the root volume causing a disk space crisis for /tmp or vice versa.

With the system back up and running having rejoined to /home on the RAID-1 array and therefore reusing my existing user profile I gave attention to software reinstallation. The previous installation used the Debian packages for Thunderbird which installed the Daily channel build and caused problems with extensions like Lightning being incompatible and therefore unavailable. I chose this time to download Thunderbird directly from their site and install it directly into my user profile to save space on future reinstalls and ensure it is the Release channel which allows me to use Lightning again.

mediapc has had a new R/W share created in samba to give access to the Video Archive folder which contains a volume of material not in the media share and therefore I can work on the material in that folder from mainpc for convenience. mediapc is about to get various video editors installed as mentioned in a previous post some time ago.

So questions still remain over managing disk space in some of the areas of a typical Linux install and maybe I can address these.

I had a look at rearranging screens because the arrangement was less than optimum for mediapc. Basically at eye level I have four screens: two for mainpc and one each for serverpc and mediapc. Three of those are vertical to save space and keep them as close to me as possible without having to turn around too much. The problem is for video editing on mediapc you want it to be a horizontal aspect ratio for best use of space. However because mainpc and serverpc are used the most the screen layout suits them best and is not really set up for anything else so after trialling an alternative layout I decided to leave things as they are, except for the two 3 metre DVI cables I purchased which will now not be needed but still have to be paid for. At the end of the day the layout really only can be optimised for the two most used computers so video editing will just have to be improvised or I remote to the computer doing the editing from mainpc.

Firefox Multi Account Containers vs User Profiles [3]

After a bit of playing with a few settings I am returning to my expectation of multiple FF profiles to handle container isolation-like capabilities. The problem with containers is when you do need tracking information to be exchanged between Facebook and another site. With my Wordpress blogs they are set to auto publish to a Facebook page and without being able to selectively disable the containers they will not be able to authenticate with Facebook. I have found that the settings keep getting lost or turned off probably because the Facebook cookies are getting deleted or being isolated in the Facebook container which the Wordpress tab can't get access to.

Firefox has always supported multiple user profiles. There is a built in user profile manager which you can bring up by starting Firefox with the switch -P. This then gives you a list of current profiles and the ability to create delete or rename profiles, and to select a default. In this case since I have both standard Firefox and Quantum, there are profiles for both editions (Quantum's profile is called dev-edition-default). 

After completing the process of creating profiles via this screen, you'll want to make sure you have the default one selected as the one that FF will start up when it opens, because you want to start a specific profile with a custom shortcut (and you'll probably want to have custom themes, home tabs and bookmarks in it so you can tell it apart from the others, too).

To start a browser with a specific profile, you need to ensure you have the correct profile name to pass into the shortcut. You can type in "about:profiles" in the address bar of the browser to get a list of the profiles the browser knows about. However, the list is only updated each time you open the browser and that means if necessary you have to close all windows of that browser to get it to reload. In this case I had created my profile list with regular Firefox and because Quantum windows were still open I had to close them all to get Quantum to show me the profile list.

The command to start Firefox with a specific profile is firefox -P "ProfileName" and it opens that specific profile. Plain old FF with no parameters keeps on using the default. Quantum has some way of seeing there is a dev-edition-default profile and using that instead of the one regular FF is using, which I think is the result of a setting in Quantum's preferences, that tells it to start up with that specific path. If you are using Quantum alongside Firefox the command to start Quantum will be different. I have installed Quantum on some of my computers inside my home directory, instead of a system path, so that I don't have to reinstall it each time I reinstall the system. This is also impacted by the fact we install Quantum from a TGZ file we download, rather than from our distro's repositories as a DEB package. It also ensures that Quantum is able to update itself in our user account's name (i.e with the right permissions) when Mozilla tells the browser to update, as if it was installed in the system path e.g. /usr, only root would have rights to that path.

So in this case to ensure I am starting Quantum I need pass the whole path in the shortcut. With the command line as above it indeed starts up Quantum with a completely different window that has nothing: no shortcuts, theme or extensions. I can now completely customise that profile independently of any other. What I really do need is a custom colour, or maybe changing the string in the title bar. And that I will look at next time. But for now I don't need the multi containers because that window is only being used very specifically for that blog's Wordpress account. So that is how it will happen from here. The next thing is to add a custom shortcut in KDE Menu Editor and I will create a new submenu for browser profiles as well. Well I have been setting these profiles and shortcuts up and creating icons for them out of the profile pictures I use in FB so it is going pretty well and I need to install the Tab Session Manager extension into each profile to save the tabs and the NZ Rail Maps one will need the uget extension as well to handle the downloads. And so it goes.

Firefox Multi Account Containers and Equivalents [2]

OK I have managed to simplify this from yesterday.

I now expect to have just single user profile in FF and do the following:

• Window 1 will be the one on the main screen of my PC and it will be the one that mainly I am using MACs. The MACs will be geared around Google, Wordpress and Disqus which are the multiple accounts I mostly have.
• Window 2 will be on the second display and it will use MACs specific to NZ Rail Maps. Currently I use Opera to access this stuff but I need to use FF for this.
• Both Window 1 and Window 2 are Firefox Developer Edition (aka Firefox Quantum).
• Window 3 is specific to certain accounts and purposes and won't need to have the MAC extension installed. It will maintain separation by using the standard version of Firefox which has its own separate user profile from FFQ.

So that makes it relatively easy to set up the MACs and use them in FFQ. I want to use Firefox for everything now especially what I used to use Chrome for in the past and also Opera. Both are still installed but won't need to be used.

Firefox Multi Account Containers and Equivalents

Since these days I now use Firefox as my main browser (partly to give the finger to Google for the data thieving Chrome browser), I need to have a look at how to access all my Gmail accounts in the same browser without having to log in and out of whichever account I am using at a particular time. The same goes for, say, Wordpress (I now have four Wordpress blogs and each has a different account associated with it), Instagram and some other services. At the moment one workaround is to use several different browsers at the same time. For example I can use Firefox Developer, regular Firefox, Opera and Chrome.

Mozilla provide the Multi Account Containers extension to help you set up the capability to handle multiples of the same website account (for example, Google accounts) in the same browser window. You can set up a container group (for example Home, Work) and then add individual tabs to that group. Each group can operate independently of each other.

As good as this is, I am still considering an alternative of setting up multiple Firefox user profiles and starting a different Firefox session on a different user profile. This would still require the container groups to be used, but each group type would be unique to that user profile and there would probably only need to be one container group type per user profile.

I have been thinking about this for months (MACs have been available for a while and were first installed on Firefox some time ago for me) but now perhaps is the time I will start testing in earnest to see if I can make these ideas work.

The multi window thing addresses mainly that I don't want a lot of tabs open in a window as would be the case with MACs all opening in one window. Being able to label the window and/or colour its title bar independently of other windows would also be a help. However, it could just be a lot simpler to classify everything using the MAC container categories sufficiently descriptively bridging what the different accounts are used for. I may still keep Firefox standard and Developer both in use with MACs.

Firefox quality still too variable

If you had seen my series of earlier posts on browser choices, you'd have seen that at one stage I was recommending Chrome over Firefox. Then later, I changed to recommending Firefox. Stability in Firefox for me has been in two areas:

1. Crashes of individual tabs

2. Websites failing to load completely or being very slow to load.

Mozilla seems to have trouble making a release of Firefox that doesn't have either of these problems. A new release will come out and every tab will start crashing on just about every website. Then another new release will come out and tab crashes stop, but now it keeps freezing when loading websites. This is hard to understand because I have tested the non-developer edition of Firefox with literally hundreds of tabs open, and it never missed a beat. Yet the current edition of Firefox Developer, which is supposed to be better (which in a lot of cases it is) than the regular edition of Firefox, struggles at times with only a dozen tabs, and that's when you know it's time to restart the browser.

When I adopted Firefox Developer as my core browser it was so I could run it across multiple PCs, some of which are resource constrained, and know there would be a reasonable user experience on the slower computers. Like a lot of people I am finding Chrome is a huge resource hog (and this is written on my gruntiest PC running Xubuntu with 24 Gigs of RAM). There have been a couple of times when I have wanted to switch back to Chrome because of the variable experience Firefox has offered. At the moment it is just hanging in there. Best thing at the moment is on this PC it can update (being the version from Ubuntu Make). Worst thing is not being able to install it on anything else running Linux at the moment and get an updateable version, because Ubuntu Make people haven't bothered updating their software for months.

UPDATE 01/2019: Firefox Quantum has become a lot more stable and is now my main browser. If I install it into my home directory, there is no problem with permissions for updating itself, the best part is I don't need to reinstall each time I reinstall Linux. On mainpc I'm using it with multiple user profiles.

Mozilla Thunderbird

So I am trialling out Tbird again. For some reason I disliked it for quite a long time even though I used it a lot, years ago. I suspect the interface was kind of clunky, but of course there are all these styles available now. Really easy to set up with Google and Hotmail accounts, and with the calendar extension which is integrated, you just need a Gdata provider extension to connect to Google calendars.

Yesterday I was wrong when I wrote that you had to pay for the Google calendar extension, the one you need to pay for is for the extension to connect MS Exchange accounts. I can get by using Windows 10 Mail for just that one Ex2003 account that I use regularly, since it looks like Tbird will do everything else. If it does the Google calendars well then it can replace EmClient that I currently use for my timesheet calendars. It is not quite as tidy or featured as the EmClient calendars but the functionality appears to be there.

It will be interesting to see if Tbird also handles some of the other Google stuff, or Live calendars, although I can get by without the Live calendars since this is mainly for work computers with Windows 10 that I will be using Thunderbird with to start.

Well since writing the above this morning I have quickly moved to put Tbird on every computer I own or use. The Google calendar functionality is great, it works just as well as emClient although the information is not quite displayed in the same way. There is another extension that synchronises contacts which I am also trialling. Basically, Tbird has superior performance over Outlook for IMAP accounts, although in fact just about everything does, including Outlook Express/Windows Mail. It is the fact it works well across Outlook.com accounts and Gmail as well that makes it highly suitable to my requirements. What it doesn't have is the ability to sync Outlook.com calendars readily and possibly their contacts, but that really isn't much of an issue for me at the present time.