Cannot seem to get good latency numbers!

Latency with Mesa hardware does not have much effect on performance.
As long as you can run a 1 KHz or so servo thread without timeouts
it should be fine.

I have seen quite a bunch of posts from you, there are some things that I've noticed

1- Kernel boot parameters

Even if your BIOS/UEFI doesn't allow you a great deal of options, you can try some boot line kernel parameters to disable, say, hyperthreading and c-states. Those boot parameters have to be written into /etc/default/grub, line GRUB_CMDLINE_LINUX_DEFAULT=.

After saving changes, you must run update-grub and reboot the machine

"nosmt" will disable hyperthreading even if your BIOS doesn't offer that option to you.

"intel_idle.max_cstate=0" will disable intel_idle kernel module. This kernel module deals with C-states. You'll probably need to use also "processor.max_cstate=0" to effectively disable C-states now that intel_idle kernel module isn't working anymore.

"processor.max_cstate=0" will disable any c-states on non-intel machines. No need to worry about intel_idle module on non-intel systems

"idle=poll" can be tested after "intel_idle.max_cstate=0" and "processor.max_cstate=0" have been proven not to work for you. This parameter makes the processor unable to get into any c-state; it will not be idling but polling instead. Processor will run hotter, however. So, use it as last resource.

2- Now, some other boot parameter that have nothing to do with the CPU idling but are/could be useful. I've seen that you are not using isolcpus parameter.

isolcpus=a,b,c,...x,y,z will isolate cores a,b,c,...x,y,z thus avoiding the linux scheduler to send any tasks to them. Well, allmost. So, in a dual-core machine, you would use typically "Isolcpus=1" and you could use "isolcpus=2,3" or even "isolcpus=1,2,3" on a quad-core machine. Linuxcnc and the latency tests know about isolcpus= parameter, as you can see in latency-histogram window. So you can usually start any latency test or linuxcnc in the usual way, even while using this parameter. I know that the newest advice is not to use it, but in my experience it works like a charm. Your latency could drop by orders of magnitude and it could even be the only way to get a usable system. Why not to use it?

"acpi_irq_nobalance" instructs acpi not to balance the system load by changing the cores where IRQs are addressed to. It may or may not work for you, but you can give it a try.

"nowatchdog" will disable both hard and soft kernel watchdogs, which will be triggered if you are overloading your system while testing latency. So, if your system load is high while testing, you'll get probably worse latency because a) system is overloaded in the first place, and b) if the watchdog detects there are processes waiting too much, it will throw a warning, that by itself will make latency perhaps some tens of microseconds worse. This is not the reason you are getting so bad latency numbers, however. But I find useful to have this parameter in force while testing. and usually forget about deleting it while in production. No harm here, it seems to me.

You can read about 4.19 kernel parameters here:

www.kernel.org/doc/html/v4.19/admin-guid...rnel-parameters.html

Please note that both the RT and RTAI kernels may disable many things at build time so things like cpufreq governor, p-states, et al, could have no viable application. You can read about, say, nohz_full paremeter but you should check, before testing, if those things were disabled in the kernel at build time. Kernel configuration is under /boot directory. So, if you want to know if your running kernel have, say, no_hz capabilities, you could run:

grep -i no_hz /boot/config-`uname -r`

On Debian Wheezy with linuxcnc 2.7 it will return:

CONFIG_RCU_FAST_NO_HZ=y
CONFIG_NO_HZ=y

The "y" meaning that capability is built into the kernel and used by default
If that command doesn't give you any answer, it means that the parameter/capability doesn't exist as you named/typed it. You could also get an answer" XXXX is not set", meaning it's disabled

By the way, the nohz parameter is on by default. It's about local timer interrupts. It could be that, by disabling those interrpts, that core would be idling, and to get that core up again, some microseconds would be lost, making latency worse. So, you can give "nohz=off" a try, to see if it makes your latency worse or better.

3- smp_affinity

smp-affinity decides which cpu/core your tasks are addressed to. If you are isolating a core, you don't want any tasks other than your realtime task sent to that core. It can't be fully done (yet) but you want as few as possible processes/threads going on that core. Also, generally speaking, you don't what that core dealing with any system interrupts.

isolcpus= may not be enough to get an isolated core, actually I have seen IRQs sent to supposedly isolated cores. I don't know exactly how it happens. I mean, I didn't sent anything to them and, since isolcpus is in force, the kernel shouldn't have done it, either. But there they are, and you want them out of the isolated core where your realtime task is to run.

You could run this command in a terminal window while testing:

watch -n 1 -d cat /proc/interrupts

This could be the output:

CPU0 CPU1
0: 39 0 IO-APIC-edge timer
1: 68 9 IO-APIC-edge i8042
7: 1 0 IO-APIC-edge parport0
8: 0 1 IO-APIC-edge rtc0
9: 0 0 IO-APIC-fasteoi acpi
12: 14041 135 IO-APIC-edge i8042
16: 0 422 IO-APIC-fasteoi ohci_hcd:usb2, pata_atiixp, snd_h
da_intel
17: 219362733 304 IO-APIC-fasteoi ohci_hcd:usb3, ohci_hcd:usb5, rad
eon
18: 0 0 IO-APIC-fasteoi ohci_hcd:usb4, ohci_hcd:usb6
19: 117464 38 IO-APIC-fasteoi ehci_hcd:usb1, eth0
21: 56 11 IO-APIC-fasteoi
22: 200833 3229 IO-APIC-fasteoi ahci
NMI: 0 0 Non-maskable interrupts
LOC: 32572366 31318 Local timer interrupts
SPU: 0 0 Spurious interrupts
PMI: 0 0 Performance monitoring interrupts
IWI: 0 0 IRQ work interrupts
RTR: 0 0 APIC ICR read retries
RES: 2173 18 Rescheduling interrupts
CAL: 0 683 Function call interrupts
TLB: 0 0 TLB shootdowns
TRM: 0 0 Thermal event interrupts
THR: 0 0 Threshold APIC interrupts
MCE: 0 0 Machine check exceptions
MCP: 261 261 Machine check polls
ERR: 1
MIS: 0

As you can see, now you have a detailed, updating list about where your interrupts are going. So, 1, is your keyboard, and if you press any keys, you should see how the number changes under CPU 0 or CPU 1, or whatever it would be in your case. On this example, you can see that the radeon graphic card on IRQ 17 has sent 219362733 interrupts to CPU 0, but just 304 to CPU 1 (and that was while booting).

Of course, if you are using isolcpus=1, probably you don't want any system IRQ going to CPU 1, but to CPU 0 where they doesn't mess the realtime task. The output of this command will show you what's happening in you CPU's cores and will give you clues about what could be the cause of your bad latency.

Of course, if you use isolcpus=, commands like top or htop will show the isolated core has 0% load. This is because neither top nor htop show kernel threads unless instructed to do so. But if you do instruct them to do so, you'll see threads created by your latency test or linuxcnc in the isolated core. Not that realtime thread is going to overload that core, however. Far from it.

Now, as I understand it, the trend about not isolating cores is about not wasting those cores' processing power. But, hey, what I care about is latency. So, if I can get better latency by isolating a core, sign me up.

Actually, isolcpus is deprecated by linux kernel developers. The advice is to use cpuset capabilities. But, isolcpus still works, is easier, specially for newbies, and I'm willing to have it in use as long as it remains useful. YMMV.

You can have your smp_affinity set to 1 automatically at the end of the boot process by using this in your /etc/rc.local file. However it could perhaps not work for you. So it will probably be better to use it as an script launched manually while testing; then, in case it's proven useful to have the smp_affinity set to 1, you can put it in /etc/rc.local to have it in force after booting the system

###################################################
#!/bin/sh -e
#
# rc.local
#
# This script is executed at the end of each multiuser runlevel.
# Make sure that the script will "exit 0" on success or any other
# value on error.
#
# In order to enable or disable this script just change the execution
# bits.
#
# By default this script does nothing.

for i in /proc/irq/*;do
if [ -d $i ];then
/bin/echo 1 > $i/smp_affinity || true
else
/bin/echo 1 > $i || true
fi
done

exit 0
############################################

This can be used on Debian Wheezy with no problems. It works also on Buster where systemd is there to mess things if they are not of his liking. So on Buster after putting it in your /etc/rc.local, you need to do:

systemctl daemon-reload
systemctl start rc-local # yes rc-local not rc.local. Ask systemd about it

After that it should always be in force after booting the system. You can check it with

systemctl status rc-local

4- Your graphics card.

It's usually said that integrated graphics are bad, however I could give some examples where this doesn't hold true. Which remains as true to me, however, is: nvidia graphics are bad. May be this will change in the future, now that a free driver has appeared. We will see. Anyway, you could get an used, 30 bucks radeon card and have some testing. I always test latency with and without a discrete radeon card.

5- Latency-histogram and variance errors

This happens to me with HP-Compaq dc5750 and dc5850. When this happens, I can see a call trace in dmesg output about MSI messaging and smp_affinity. Sometimes latency-histogram works fine, sometimes i got those pesky variance error messages. You should check your dmesg output looking for traces.

6- Machines with known good latency.

To me, a couple dual-core, Intel-based, MEDION-branded machines gave me good results, in the 15-20 usecs range.

Recently I got a bunch of HP-Compaq machines, dc5750 MY and dc5850 MT, with radeon integrated graphics, that give me usec latency with 2.7 Wheezy, about 10 usecs with 2.8-RTAI (launching with taskset 2 latency-histogram - taskset 2 linuxcnc to work around the MSI quirk already mentioned) and about 60 usecs with 2.8-RT (using isolcpus=1) Those are athlon X2 dual-core, bussines systems and can be found dirty cheap in used but good condition

And, last year, I got from Ebay a quad-core Dell Optiplex 960 for about 100 bucks giving about 10,15 usec latency using isolcpus=1,2,3. These Optiplex are also bussines systems and AFAIK there is quite a bunch of them on Ebay.

Systems that I have heard of recently but not tested personally:

Dell Optiplex 980, which is said to be giving about 40 usecs latency without any kernel parameters whatsoever, just disabling things on BIOS, and judging by the kernel mentioned, I guess it was using 2.8-RT. You can see it on this thread that, BTW, is about variance errors in latency-histogram output:

forum.linuxcnc.org/38-general-linuxcnc-q...ils-on-latency-tests

Celeron J4125, about 150-200 bucks (new) in ebay/amazon. See:

forum.linuxcnc.org/18-computer/44740-celeron-j4125

7- No base thread needed

As already posted, if you are going to use a MESA/Pico card, you don't need the base thread and should run your tests with something like this

latency-test 1000000 -

or

latency-histogram --nobase -sbinsize 1000

8- Stick to the stable release until you gain experience with linuxcnc, or else you'll get probably frustrated. Leave the new, experimental stuff to the developers. Until now, you are just dealing with linux latency. You thought this would be an easy ride? Welcome to the club. Please remember that dealing with hal is another can of worms. LinuxCNC is very powerful, flexible software and has a learning curve. I'm glad I got a Sherline mill that has a working linuxcnc conf out of the box, or else I would probably commited seppuku some years ago. Even if I was used to build rt-preempt kernels when it bas just an out-of-mainline set of patches a couple of decades ago.

Best wishes

WOW! Thanks tatel for this great info. I am gonna need to find some time to go through it all and run some tests. I can already tell this is exactly what I was wanting to know. I will post my testing results.

THANKS!

I tried an old Radeon video card I had but the fan was locked up and the screen was all messed up. I found this card online for $14. Would it work well?

Dell AMD Radeon R5 240 1GB DDR3 DVI/ D-Port Video Card F9P1R 0F9P1R

Not sure exactly what to look for.

You can bet your life you are gonna need some time. I'm still learning ten years after I installed linuxcnc for the first time, and, as already said, I was then already used to patch the vanilla sources and build rt-preempt linux kernels. So, take it easy and avoid the samurai's fate.

I'm afraid that latency tuning is still an art, not yet a science. So you'll need to gain some experience. To watch the /proc/interrupts thing seems to be giving the best clues, to me, about what's going on. Do I have many rescheduling interrupts on my isolated core? Since that core does not have so much load, perhaps that shouldn't be happening. TLB shotdowns? They are EVIL. Does cat /proc/cpuinfo say my processor has constant_tsc, yet I'm seeing an HPET local timer interrupt in each core? Time to put hpet=disable on the boot line. And so on and so on and so on.

So, again, take it easy. Get your system working by hook or by crook --this is about machining, after all, it isn't?-- then you'll get as much time as you need to learn more.

I think the aim is for latency tuning to become a science, and that going the RT way, even if it has far worse latency than RTAI, is to make it easier both for linuxcnc developers and users.

But, I fear RT-Preempt still has too much latency. Being able to record a punk band that are friends of you, with a salvaged-from-junk PC and a cheap sound card, without latency issues is one thing. That <5 miliseconds latency can perhaps be good to do trading, too. But if you need to have <50 usec latency or else have a probability to see your machined part ruined, well, things are not so easy. This is why I'm in love with RTAI and this is why is much easier to have good enough latency by using a card that does hardware stepping

Anyway, as said, welcome to the club

Yes, probably it will work quite well if you fix the fan issue. It happens I have been just looking for some radeon card on aliexpress. Seen that card, seems to be PCIe 3 16 x if memory serves me well. I think it should be good enough to see if your integrated card - or discrete nvidia, perhaps intel graphic card are messing your latency numbers.

Not an expert on graphic cards, but I think you should look at your motherboard specs and see what you can get for little money. PCIe 3x cards seem to work on PCIe 2 slots without losing performance noticeably, but I have not already tested it. Of course, you want both the motherboard and the graphic card having PCIe 16x, better than 8x.

Usually, the more VRAM, the better but you shouldn't run amok to get the more VRAM-loaded graphic card.

The wider the graphics car memory bus, the better, so 128 bit is better than 64, 256 is better than 128, and so on, but this is not the only thing having to do with the graphic card performance. What you want is total throughput. I just purchased an HD 6750 because it is PCIe 2 16x x as the destinated motherboard, has 128bit-wide VRAM bus and a GB of RAM. So it seemed to me it was cheap enough and better than the HD 6450 which is the better radeon I have lying around (64-bit) and decided to give it a try. I'll be getting another, PCIe 3 16x card in the future.

Also, you should look at the power output from the PSU, high-end graphic cars eat a lot of power and, should the PSU be unable to cope with that demand, your system would become unstable. If the card is 2 slots high, it could get in the way of another card and that could be annoying in some cases.

I think a good place to know about GPUs is

www.techpowerup.com/gpu-specs/

There you can filter your search by a whole bunch of criteria.

This is the moment to give you a little stabbing by telling that, even if you have no overruns, you can still have "latency excursions" which are visible on terminal window if you launch linuxcnc from one of those windows. They mean, I think, that you graphics systems is unable to do its work --updating the screen-- in 10x the time allocated to do so.

However, don't worry about it. If you are using axis simulator, the time allocates is 1 millisecond. So, if your graphics system needs a hundredth of second, you'll get an "excursion". Now, with the Sherline standard stepper drivers that need a 22 usec signal, base thread is 50 usec, not 25 as "usual". So, when using the sherline mill config instead of axis sim, the time allocated is greater, and latency excursions don't appear unless they took 0,1 secs. So I'm not seeign any while milling, but while using axis sim, y can see them appearing after each press of a button in the GUI.

So, latency excursions are not overruns and having them doesn't mean your system is unable to do the intended work. What matters is the tool moving as expected. You can give a blind eye to "latency excursions" if your tool moves along the path as expected. AFAI, yoll have to get probably thousands of excursions before need to get rid of them arises. There are other here, with more experience, that surely could give to you more info than I could. But, having a good graphics card surely will help with that.