random fails on latency tests

I'm getting random fails with latency-histogram. Even on a fresh install --using RTAI-- it can take quite a bunch of seconds to start; then it shows min servo latencies of -24,7 usecs (base) and -999,7 usecs (servo). It doesn't updates for about other 40-50 seconds, the it starts to update and to throw messages

can't use non-numeric floating point value as operand of "*" (variance=-1521835223)

But sometimes it starts without any problem. Please see screenshot; there you can see how it's working fine just after killing a previous fail.

Tried latency-test. It started fine at first attempt then on another try got stuck signaling 0 usec latency. I tried latency-plot just once and it worked fine.

I noticed the same latency-histogram behavior in a previous custom install using Devuan Beowulf. There I added linuxcnc repos to sources.list.d/linuxcnc.list and installed kernel 4.19.195-rtai-amd64 and linuxcnc. On the live install of linuxcnc, I uninstalled linuxcnc-uspace, then installed kernel 4.19.195-rtai-amd64, linuxcnc and linuxcnc-dev

In both cases system was running the RTAI kernel

I'm attaching the output of halcmd show and linuxcnc_info  

And here you can see the previously mentioned screenshot:
https://justpaste.it/827de

Upload the pictures here, please.
Did you disable hyperthreading and power saving features in BIOS?
RTAI 4.19 will refuse to work on some older hardware, it boots and everything works except the real time stuff, here are some examples:
forum.linuxcnc.org/18-computer/38838-har...ts-used-pc-s?start=0

Not really much to do in this BIOS, but processor.max_cstate=0 and nosmt have worked wonders. No latency problems here, but latency test software problems. Since I got the same behavior twice in a row with two different distros, I tougt it could probably be worth to report it.

Not really sure about how to upload pictures to this forum, do you mean to attach them? Well I'm going to try ...   

Here's the picture from yesterday's post


Here it is dmesg output, it turns out that there is an overrun reported, but latency-histogram did say nothing about it. It correspond to the first picture in this post, that test is still running with reported max latency 3.6 usecs
 

The error message is from somewhere around here:
github.com/LinuxCNC/linuxcnc/blob/98a847...ts/latency-histogram

I don't know Tcl at all, but it seems that the "hal getp" is returning a non-numeric string, so perhaps add a "puts" command in that area to see what is being returned into the calculation of variance.
(As Tcl is interpreted rather than compiled, you can just edit and run)

Though, after typing this, I found forum.linuxcnc.org/38-general-linuxcnc-q...-gets-error-messages

It seems I was getting negative variance values. Not sure why it looked random to me. After quite a bunch of tests, I drew these conclusions:

a) 2.8-rtpreempt seems to be more than good enough to get a Mesa card working perfectly. So I can fully understand why this is the chosen way to go ahead, and will probably take it, if/when I go with servos/Mesa cards in the future

b) rt-preempt isn't even near to be enough to work with the parallel port, unless a quad-core computer is available. In that case, some 15-20 usecs latency could be expected using isolcpus=1,2,3 and a boot line longer than the police record of the entire Capone gang. Without isolcpus= on the boot line, better to forget about it.

That's what I was getting much more easily from a dual core and 2.7-rt. With 2.8-rtpreempt, I was anyway getting overruns for a 50000 ns base thread as soon as latency-histogram gets over 30 usecs, give or take a bit.

c) Maybe cpuset is the way to go, maybe not. I don't know, but since it seems that the rt-preempt kernel still has more latency than I want for my stepper drives, I'm not going to explore it

d)2.8-rt is much better latency-wise, and I got some 3-4 usecs latency after a 24-hours test with it, using isolcpus. However it looked to me as if cpuset and cpu hotplug were disabled, so no chance to play with cpuset. Also, it seems to be more fragile than its predecessor, hence the messages gotten from latency-histogram, be it about variance, be it about waiting for "thread" component, etcaetera

e) 2.7-rt is the way to go for me. I'm getting 1-2 usec base latency very easily on the very same computer that gives 3-4 usec with 2.8-rt and 30-50 with 2.8-rtpreempt. Without any bizarre behavior. I don't care about Debian Wheezy being unmaintained, all I want is a computer able to manage those pesky sherline drivers that need a 22 usec signal to make a step. I'm not planning to do anything else with it but to cheaply control steppers.

Anyway, kudos to the linuxcnc community for the good work done, and thanks to you all who answered my messages.

tatel wrote:

b) rt-preempt isn't even near to be enough to work with the parallel port, unless a quad-core computer is available. In that case, some 15-20 usecs latency could be expected using isolcpus=1,2,3 and a boot line longer than the police record of the entire Capone gang. Without isolcpus= on the boot line, better to forget about it.

Not always, Dell Optiplex 980 work perfectly well with RT kernel and nothing added to grub.
With RT kernel 4.9 a base period of 40000 and with 5.18 a base period of 60000 never fail. Just needs some stuff disabled in BIOS.