Can't eek out that last bit of jitter

How can I track down the remaining sources of jitter? 

I have reduced the jitter a lot, but there are still some odd areas of jitter in my base histogram that I can't track down.  On my base thread, you can see a few spikes out at -20us and 20us and also some spikes around -10us.  (I'm going to use the parallel port for driving my milling machine.)
 
I'm using an AMD FX-8350 cpus on a biostar A880GZ motherboard.  The AMD FX-8350 has 4 cores each with 2 threads, but I set nosmt, so linuxcnc is only using 2 cores.  I'm using the integrated graphics card.  It is ATI Radeon HD4250 Graphics .

I am running from a live usb drive with the following boot command.

    linux    /live/vmlinuz-6.1.0-30-rt-amd64 boot=live components quiet splash findiso=${iso_path} skew_tick=1 nosmt=force kthread_cpus=0-1 irqaffinity=0-1 rcu_nocb_poll rcu_nocbs=2-7 nohz=on nohz_full=2-7 isolcpus=2-7 intel_pstate=disable amd_pstate=disable idle=poll cpufreq.off=1 cpuidle.off=1 intel_idle.max_cstate=1 amd_idle.max_cstate=1 nowatchdog nosoftlockup
    initrd    /live/initrd.img-6.1.0-30-rt-amd64

When booted I also set

1. Code:

   sudo sysctl kernel.timer_migration=0
2. Code:

   sudo sysctl kernel.sched_rt_runtime_us=-1

(All these setings based on:  dantalion.nl/2024/09/29/linuxcnc-latency...arameter-tuning.html )

When I run 

watch -n 1 -d cat /proc/interrupts

I only see LOC (Local timer interrupts) on CPU2, CPU4, and CPU6, until I start the latency-histogram.  All other interrupts are on CPU0.  After starting the latency histogram I also see IWI (IRQ work interrupts) on CPU6.  (Note: I tried not isolating CPU6. Then the IWI interrupts stayed on CPU6, but the histogram looked worse.

I think I've disabled everything I can in the bios, but maybe I misunderstood some bios setting, and mistakenly turned something on.  Here's all my settings I think apply:
Advanced>PCI Subsystem Settings>
  PCI ROM Priority = Legacy ROM
  PCI Latency Timer = 32 PCI Bus Clocks
  VGA Palette Snoop = Disabled
  PERR# Generation = Disabled
  SERR# Generation = Disabled
Advanced>PCI Subsystem Settings>PCI Express Settings>
  Relaxed Ordering = Disabled
  Extended Tag = Disabled
  No Snoop = Enabled
  Maximum Payload = Auto
  Maximum Read Request = Auto
  ASPM Support = Disabled
  Extended Synch = Disabled
  Link Training Retry = 5
  Link Traing Timeout (uS) = 100
  Unpopulated Links = Keep Link ON
Advanced>ACPI Settings
  Enable Hibernation = Disabled
  ACPI Sleep State = Suspend Disabled
  Lock Legacy Resources = Disabled
  Wake System With Fixed Time = Disabled
  USB Device Wake Up From S3/S4 = Enabled
  PME Wake Up From S5 = Enabled
  PS2 Keyboard PowerOn = Any Key
  Stroke Key = Wake Key
  PS2 Mouse PowerOn = Enabled
Advanced>CPU Configuration
  Limit CPUID Maximum = Disabled
  PowerNow = Disabled
  C1E = Disabled
  SVM = Disabled
  Core C6 State = Disabled
  HPC Mode = Disabled
  core performance boost = Disabled
  HTC temperature limit = 70
  Core Leveling = Auto
       (I tried disabling, but it just reduces the amount of cores that I can use.)
Advanced>SATA Configuration
  All ports = Not Present
Advanced>USB Configuration
  USB Devices: = 1 Drive, 1 Mouse
  Legacy USB Support = Auto  (I tried Disabled, but then I can't boot from USB at all)
  EHCI Hand-off = Disabled
  USB transfer time-out = 20 sec
  Device reset time-out = 20 sec
  Device power-up delay = Auto
  Mass Storage Devices: Samsung Flash Drive FIT 1100 = Auto
Advanced>SMART FAN Control
  CPU Smart FAN = Disabled
Advanced>Super IO Configuration
  Super IO Chip = IT8728
  Restore AC Power Loss = Power Off
Advanced>Super IO Configuration> Serial Port 0 Configuration
  Serial Port = Disabled
Advanced>Super IO Configuration>Parallel Port Configuration
  Parallel Port = Enabled
  Device Settings =  I0=378h; IRQ=5;
  Change Settings = Auto
  Device Mode = Standard Parallel ...
Advanced>Super IO Configuration>CIR Controller Configuration
  CIR Controller = Disabled
Advanced>Chipset
  NB Power Management Features = Disabled
Advanced>Chipset>Internal Graphics Configuration
  Internal Graphics Mode = UMA
    UMA Frame Buffer Size = 1024MB
  GFX Engine Clock Override = Disabled
  Surround View = Auto
  FB Location = Above 4G
  AMD 880 HD Audo = Enabled
  Vbios Type = HDMI
Advanced>Chipset>PCI Express Configuration
  GPP Slots Power Limit, W = 25
Advanced>Chipset>PCI Express Configuration>Port #02 Features
  Gen2 High Speed Mode = Auto
  Link ASPM = Disabled
  Link Width = Auto
  Slot Power Limit, W = 75
Advanced>Chipset>PCI Express Configuration>Port #06 Features
  Gen2 High Speed Mode = Auto
  Link ASPM = Disabled
Advanced>Chipset>PCI Express Configuration>NB-SB Port Features
  NB-SB Link ASPM = L1
  NP NB-SB VC1 Traffic Support = Disabled
  Link Width = Auto
Advanced>Chipset>South Bridge Configuration
  SB CIM Version:  1.1.0.9
  SB Clock Spread Spectrum = Disabled
Advanced>Chipset>South Bridge Configuration>SB SATA Configuration
  OnChip SATA Channel = Disabled
  OnChip IDE mode = Legacy mode
  SATA-III Mode = Auto
Advanced>Chipset>South Bridge Configuration>SB USB Configuration
  (All enabled)
Advanced>Chipset>South Bridge Configuration>SB HD Azalia Configuration
  HD Audio Azalia Device = Disabled
  Azalia Snoop = Disabled
Advanced>Chipset>OnBoard Peripherals Configuration
  Realtek PCIE NIC = Disabled
  MAC ID not find

Under the Performance section I left everything default.

Any ideas where I can further reduce jitter?

Do you need a parallel port?
If not:
latency-histogram --nobase --sbinsize 1000 --show

Yes. I'm using the parallel port to drive my milling machine.

See the downloads page on how to install the RTAI kernel and RTAI LinuxCNC version, they should be much better for latency on older hardware.
Also, always use --show for latency histogram, in your screenshot you can see excursions on both sides of the base period plot.

Thanks tommylight,

I'm trying to install the RTAI version of the kernel, but I'm having no luck.  I'm not sure where I'm going wrong.

I installed from the linuxcnc_2.9.4-amd64.hybrid.iso .

So I think I covered Step 1 of 7.2 in the instructions at linuxcnc.org/docs/html/getting-started/g..._installing_linuxcnc

From there I did step 4 of 7.1, but that causes an issue with "Conflicting values".

I tried ( forum.linuxcnc.org/9-installing-linuxcnc...-kernel-on-debian-12 ), but with 2.9.4:

Try just installing the Linuxcnc package, after removing the docs package

wget linuxcnc.org/dists/bookworm/2.9-rt/binar...xcnc_2.9.2_amd64.deb

sudo apt install ./linuxcnc_2.9.2_amd64.deb

The issue is with the docs package, which isn't really required for a basic install.

But, it appears there is no 2.9.4 version.  So I tried 2.9.3, but that is giving me all kinds of errors.

First is there a 2.9.4 version of the RTAI kernel?
If not, should I have tried to install from the 2.9.3 iso instead?
 

See section 7.2 on the link you provided, did you try step 3 and 4?

Thank you for the reply. I had done 7.2.3 and 7.2.4.

I'll start over and try again., but let me see if I have the correct steps, because I think I'm either doing something extra, or leaving something out.

linuxcnc.org/docs/html/getting-started/g..._installing_linuxcnc

I do the following:
*All of step 1 (Download the image)
*All of step 2 (Write the image to a bootable device)
*All of step 4 (Installing LinuxCNC)
(This all worked to here, as I can boot from my hard drive.)
* Do I do 7.1.3? (There are 2 7.1.3, so this is the first one with apt update and dist-upgrade)
*I assume I skip 7.1.1, as I don't want the rt kernel anyway. (sudo apt-get install linux-image-rt-amd64) Correct?
*uname -v shows the preempt-rt at this point.
*Do 7.1.3. Remove the original non-rt linux-image. (Or should this come after installing linuxcnc from 7.2.3 and 7.2.4?)
*Do 7.1.4 (Linux Archive Signing Key) - This seems to create problems with apt later on for me.
*I assume I go to 7.2.2 next, but I'm not sure what it is telling me to do. I think I haven't installed the RTAI kernel yet, but here it says if RTAI is detected it will stop before installing any packages. Thus, I'm not sure what I'm doing with this step.
*Do 7.2.3. (apt-get update)
*Do 7.2.4 (install linuxcnc) This is where i was getting "conflicting values" if I had run the linuxcnc-install.sh script and not removed its changes from the sources list.

I would try up to 4 as usual then skip directly to 7.2.3 and 7.2.4.

Thank you again.

I'm on 7.1.3. There is no "non-rt" "linux-image". I have
linux_image-6.1.0-30-rt-amd6 6.1.124-1
linux_image-6.1.0-30-rt-amd6 6.1.128-1
linux-image-rt-amd64 6.1.128-1

Should I remove one of these, or just go to 7.1.4 (linuxcnc-install.sh)?

I tried without removing any of the "rt" "linux_image" s . Again, there was no "non-rt" linux-images.

Also, I tried without linuxcnc-install.sh first (you'll see why later).

When I ran sudo apt-get update, it ran fine, but when I ran sudo apt-get install linuxcnc, it gives the error:

Package linuxcnc is not available, but is referred to by another pacakge. This may mean that the package is missing, has been obsoleted, or is only availabe from another source.

E: Package 'linuxcnc has no installation candidate.

So then I went back and did sudo ./linuxcnc-install.sh

I got the error:

E: Conflicting values set for option Signed-by regarding source httyps://www.linuxcnc.or/ bookworm: /etc/apt/trusted.gpg.d/linuxcnc.gpg.key.binary.gpg != /usr/share/keyrings/linuxcnc.gpg
E: The list of sources could not be read.
Installing LinuxCNC
E: Conflicting values set for option Signed-by regarding source httyps://www.linuxcnc.or/ bookworm: /etc/apt/trusted.gpg.d/linuxcnc.gpg.key.binary.gpg != /usr/share/keyrings/linuxcnc.gpg
E: The list of sources could not be read.

You might want to consider installating the docs package for your language
And also the linux-cnc-uspace-dev package
sudo apt-get install linuxcnc-doc-[en/'fr/de/ja/es/ru/da/hu/s1]
sudo apt-get install linux-dev

Next I ran sudo apt-get update and I get the following error again:

E: Conflicting values set for option Signed-by regarding source httyps://www.linuxcnc.or/ bookworm: /etc/apt/trusted.gpg.d/linuxcnc.gpg.key.binary.gpg != /usr/share/keyrings/linuxcnc.gpg

I can clear this error by commenting out both lines in /etc/apt/sources.list.d/linuxcnc.list . (I think this basically partially undoes the linuxcnc-install.sh . )

After that I'm back to having no install canidate for linuxcnc.

So, I did skip 7.1.3, which may be my problem, but none of the packages I have installed are "non-rt", so I don't know which to remove.

Also, I don't know if there is a step that tells the system where to get "linuxcnc". I suspect the linuxcnc-install.sh is supposed to provide it, but it just causes problems with apt.