Linux Mint / LinuxCNC 2.8.0 freezing - SOLVED?

I don't know how to check for sure but it's a vanilla 2.8.0 install which presumably means it's running RTAI from what I can see. I haven't done anything unconventional unless by mistake. I simply installed the default setup, then got Probe Basic Lathe running. I've since (this week) got gmoccapy running as PB still seems to have some distance to go before it's finished but the issue was there before I got that far.

I don't actually need to start LinuxCNC to get it to freeze up. Does the kernel only get run when LinuxCNC starts?

None of the crash logs seem to tell me much of use, possibly because there is so much in them. I'm trying to figure out how to provoke the problem consistently so I stand a better chance of knowing if I've influenced it at all. I'm planning to remove the PC from the cabinet and run it standalone until I figure this out.

I had a job getting it to boot up initially, which seems to have been something to do with sleep states. However, now it simply freezes up, screen, mouse, kbd and all at random times. I can be running almost any kind of operation at the time. Time to freeze seems to be anything from 5 minutes to 30 hours.

Intermittent faults are the worst sort, as you can never be 100% certain you've fixed them....

If it’s a conventional Mint install then it is probably premmpt-rt

uname -a

Will tell you.

Looks as if you were right:
muzzer@LinuxCNC:~$ uname -a
Linux LinuxCNC 4.19.106-rt46-lcnc #1 SMP PREEMPT RT Mon Sep 14 12:23:06 AEST 2020 x86_64 x86_64 x86_64 GNU/Linux

Does the kernel only run when you start LinuxCNC?


I was reading this from the downloads page which seemed to suggest RTAI:LinuxCNC 2.8 Downloads

• LinuxCNC 2.8.2  Debian 10 Buster PREEMPT-RT ISO

The Debian 10 Buster ISO uses a PREEMPT-RT patch which is close to mainstream Linux but does not, in some cases, give quite such good realtime performance as the previous RTAI kernel. It is very often more than good enough. It should probably be the first version tried even if using a parallel port. This is compatible with all Mesa and Pico interface boards.

• LinuxCNC 2.8.0 Debian 7 Wheezy RTAI

Users requiring a known-stable RTAI installation can install the  Debian 7 Wheezy ISO  and then upgrade as described in  2.8 documents .

• LinuxCNC 2.8.2 Debian 10 Buster RTAI

The more adventurous can install the Buster ISO and then install the experimental RTAI kernel as described in  2.8 documents

• LinuxCNC 2.8.1  Raspberry Pi OS based on Debian 10 Buster

Raspberry Pi 4 Uspace compatible with Mesa Ethernet and SPI interface boards.

Muzzer wrote:

Does the kernel only run when you start LinuxCNC?

Nope, it runs pretty much from boot time.
LinuxCNC will use the real-time part of the kernel when active, so that will set the processor to higher frequency and do some other stuff that will end up heating the processor much more.
Try adding a fan, a used PC one or whatever you can find, see if that makes the time to lock longer or get rid of it altogether.

The last freezes were almost 2 days ago during and following an update using Update Manager. Since then it's been running continuously without issue. It's too early to say but it's possible that fixed it. Dangerous talk, I know.

I've got Axis Sim running in a continuous loop plus Youtube streaming endless music video, Visual Studio code sitting in the background, Thunderbird email and VNC server running.

It has a Sunon fan in the case running all the time and I've not been able to get the core temperature above 59C no matter what I do.

About to go for a few hours walk in the hills. If it's still running on our return, I'll be tempted to consider itself fixed. Fingers crossed....

I hope this isn't premature but I'm becoming more confident that I've fixed the issue here. After an unplanned excursion into Linux crash logs and bug fixes, I appear to have a solution. If so, it's possible someone else may find this useful.

The error messages in the crash log seemed to feature 2 main themes:

• PAM adding faulty module: pam_kwallet5.so
• [drm:intel_pipe_update_end [i915}} *ERROR* Atomic update failure on pipe B .........etc.

The first seems to refer to a password wallet (KWallet), so was easy to disable.  forums.linuxmint.com/viewtopic.php?t=286950

The second seems to be caused by an "open bug" in the Intel HD graphics device. Something about a "PSR" feature that doesn't appear to be critically required yet can cause an "Atomic failure" (aka the processor won't continue until it's finished).  www.dedoimedo.com/computers/intel-microcode-atomic-update.html . This sounded like a strong candidate.

My first step to fixing the i915 error was to run Update Manager. I was 99% certain I'd done this previously but when I ran it late last week, I recall it reported that it planned to install the "Intel microcode", which is the firmware for the chipset to run Linux. Perhaps I hadn't run it previously after all. This almost fixed it - but not quite. I got a dark screen lockup shortly after the update which at least represented a change from the previous freeze (a change is as good as a rest?). The second and hopefully final step was to disable the PSR feature and after something like 4 days of continuous stress testing, I haven't seen any issues. That could change at any moment of course but I seem to have found a plausible root case and a credible corrective action.

If I've spoken too soon, I'll report back....