Debian Stretch 2.7-rtpreempt latency issues.

More
19 Nov 2016 16:42 - 19 Nov 2016 16:47 #82991 by jCandlish
Hello Forum

After getting disappointing latency test results from the Jessie uspace buildbot package, I built linuxcnc-2.7.8.7.g8a172f6.1 by the 'apt-get source linuxcnc' method.

I see the same high latencies with my locally built package, and question if latency-test is running with realtime priority. (see attachment).
latheoperator@125cnc:~$ ps -ef | grep laten
latheop+  4154  3348  0 17:18 pts/0    00:00:00 /bin/bash /usr/bin/latency-test 1000000 1000000
latheop+  4442  3489  0 17:18 pts/1    00:00:00 grep laten
latheoperator@125cnc:~$ chrt -p 4154
pid 4154's current scheduling policy: SCHED_OTHER
pid 4154's current scheduling priority: 0
latheoperator@125cnc:~$ pstree 4154
latency-test───halrun───halcmd─┬─pyvcp───{pyvcp}
                               └─rtapi_app───{rtapi_app}
latheoperator@125cnc:~$ ps -ef | grep rtapi
root      4180  4179  0 17:18 ?        00:00:00 /usr/bin/rtapi_app load threads name1=slow period1=1000000
latheop+  4630  3489  0 17:19 pts/1    00:00:00 grep rtapi
latheoperator@125cnc:~$ chrt -p 4180
pid 4180's current scheduling policy: SCHED_OTHER
pid 4180's current scheduling priority: 0
latheoperator@125cnc:~$ ls -l /usr/bin/rtapi_app 
-rwsr-xr-x 1 root root 102224 Nov 19 15:38 /usr/bin/rtapi_app
latheoperator@125cnc:~$ uname -a
Linux 125cnc 4.8.0-1-rt-amd64 #1 SMP PREEMPT RT Debian 4.8.5-1 (2016-10-28) x86_64 GNU/Linux

What scheduling policy and priority should latency-test have, and how is this priority established?


Thanks
.
Attachments:
Last edit: 19 Nov 2016 16:47 by jCandlish.

Please Log in or Create an account to join the conversation.

More
19 Nov 2016 18:37 - 19 Nov 2016 18:38 #82998 by PCW
What hardware do you have?

Here are a few latency plots of Preempt-RT

3.18 on a slow machine:

freeby.mesanet.com/atom-330-preempt-rt.png

3.18 on a 3.16 GHz Core Duo

freeby.mesanet.com/e8500-preemt-rt-dc7800-1day.png

3.18 on fairly fast machine

freeby.mesanet.com/h97-g3258-preemt-rt.png

3.18 on a laptop

freeby.mesanet.com/e6420.png

(you cannot change power modes or backlight brightness on this laptop without
large latency spikes, but is fine if you dont mess with those things)
Last edit: 19 Nov 2016 18:38 by PCW.

Please Log in or Create an account to join the conversation.

More
19 Nov 2016 19:00 - 19 Nov 2016 19:02 #83000 by jCandlish
The computer is a Shuttle X50V5
root@125cnc:~# cat /proc/cpuinfo 
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 78
model name	: Intel(R) Celeron(R) CPU 3855U @ 1.60GHz
stepping	: 3
microcode	: 0x9e
cpu MHz		: 1600.000
cache size	: 2048 KB
physical id	: 0
siblings	: 2
core id		: 0
cpu cores	: 2
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 22
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave rdrand lahf_lm abm 3dnowprefetch epb intel_pt tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust erms invpcid rdseed smap clflushopt xsaveopt xsavec xgetbv1 xsaves dtherm arat pln pts hwp hwp_notify hwp_act_window hwp_epp
bugs		:
bogomips	: 3216.00
clflush size	: 64
cache_alignment	: 64
address sizes	: 39 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 78
model name	: Intel(R) Celeron(R) CPU 3855U @ 1.60GHz
stepping	: 3
microcode	: 0x9e
cpu MHz		: 1600.000
cache size	: 2048 KB
physical id	: 0
siblings	: 2
core id		: 1
cpu cores	: 2
apicid		: 2
initial apicid	: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 22
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave rdrand lahf_lm abm 3dnowprefetch epb intel_pt tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust erms invpcid rdseed smap clflushopt xsaveopt xsavec xgetbv1 xsaves dtherm arat pln pts hwp hwp_notify hwp_act_window hwp_epp
bugs		:
bogomips	: 3217.14
clflush size	: 64
cache_alignment	: 64
address sizes	: 39 bits physical, 48 bits virtual
power management:

root@125cnc:~# free
              total        used        free      shared  buff/cache   available
Mem:        3928112      634176     2112840      109388     1181096     3118296
Swap:       6369276           0     6369276
root@125cnc:~# lspci
00:00.0 Host bridge: Intel Corporation Sky Lake Host Bridge/DRAM Registers (rev 08)
00:02.0 VGA compatible controller: Intel Corporation HD Graphics 510 (rev 07)
00:14.0 USB controller: Intel Corporation Sunrise Point-LP USB 3.0 xHCI Controller (rev 21)
00:14.2 Signal processing controller: Intel Corporation Sunrise Point-LP Thermal subsystem (rev 21)
00:16.0 Communication controller: Intel Corporation Sunrise Point-LP CSME HECI (rev 21)
00:17.0 SATA controller: Intel Corporation Sunrise Point-LP SATA Controller [AHCI mode] (rev 21)
00:1d.0 PCI bridge: Intel Corporation Sunrise Point-LP PCI Express Root Port (rev f1)
00:1f.0 ISA bridge: Intel Corporation Sunrise Point-LP LPC Controller (rev 21)
00:1f.2 Memory controller: Intel Corporation Sunrise Point-LP PMC (rev 21)
00:1f.3 Audio device: Intel Corporation Sunrise Point-LP HD Audio (rev 21)
00:1f.4 SMBus: Intel Corporation Sunrise Point-LP SMBus (rev 21)
00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection I219-LM (rev 21)
01:00.0 Network controller: Realtek Semiconductor Co., Ltd. RTL8821AE 802.11ac PCIe Wireless Network Adapter

The machine interface is an 7I80HD-16, from order # 2771 on its way from you to me.

I have always used RTAI in the past. My understanding is that the hm2_eth requires an rt-preempt kernel.

I have a little experience with rt-preempt real time audio. There the JACK setup creates an /etc/security/limits.d/audio.conf file, where members of the 'audio' group are granted "rtprio 95" and "memlock unlimited".

I don't see where linuxcnc is setting the realtime parameters for its userspace programs, and checking with 'chrt' seems to indicate latency-test is not running with any elevated real-time privileges.
Last edit: 19 Nov 2016 19:02 by jCandlish.

Please Log in or Create an account to join the conversation.

More
19 Nov 2016 19:53 #83002 by PCW
At least on the latency-histogram, the GUI part does not operate at elevated priority
but must spawn a task that does (if you set the test GUI to RT priority you will get dreadful latency)

Please Log in or Create an account to join the conversation.

More
19 Nov 2016 20:05 #83005 by jCandlish
I don't see any tasks started with elevated priority, or the documentation that explains the proper configuration to achieve such.
latheoperator@125cnc:~$ pstree 4154
latency-test───halrun───halcmd─┬─pyvcp───{pyvcp}
                               └─rtapi_app───{rtapi_app}
latheoperator@125cnc:~$ ps -ef | grep rtapi
root      4180  4179  0 17:18 ?        00:00:00 /usr/bin/rtapi_app load threads name1=slow period1=1000000
latheop+  4630  3489  0 17:19 pts/1    00:00:00 grep rtapi
latheoperator@125cnc:~$ chrt -p 4180
pid 4180's current scheduling policy: SCHED_OTHER
pid 4180's current scheduling priority: 0

What scheduling policy and priority should rtapi_app have?

Please Log in or Create an account to join the conversation.

More
19 Nov 2016 20:27 #83007 by PCW
If have not seen that it make any difference in the latency (at least with the latency-histogram)

Please Log in or Create an account to join the conversation.

Time to create page: 0.087 seconds
Powered by Kunena Forum