error finishing read
01 May 2023 10:08 #270400
by rodw
Replied by rodw on topic error finishing read
I updated my 6.3 kernel following the final release
drive.google.com/drive/folders/1NzQIHnf9...chOOxaER?usp=sharing
Performance seems to be much the same as my previous build
The kernel has advanced to 6.3.1 but there are no RT patches for it yet.
Note this only installs on Bookworm becasue it was compiled on that platform. I tried it on Bullseye and it failed
I may try building it on Bullseye as I have one machine I'm not keen on fully upgrading to bookworm
drive.google.com/drive/folders/1NzQIHnf9...chOOxaER?usp=sharing
Performance seems to be much the same as my previous build
The kernel has advanced to 6.3.1 but there are no RT patches for it yet.
Note this only installs on Bookworm becasue it was compiled on that platform. I tried it on Bullseye and it failed
I may try building it on Bullseye as I have one machine I'm not keen on fully upgrading to bookworm
The following user(s) said Thank You: tommylight
Please Log in or Create an account to join the conversation.
16 May 2023 12:10 #271452
by rodw
They gave me two methods. I thought trace-cmd sound the easiest way (it is in the debian repos). We need to issue this command:
Either with a tool like trace-cmd or manual.
- trace-cmd
- Enable events:
trace-cmd start -e sched:* -e irq:* -e irq_vectors:*
- Start the test
- Once the error happens, stop the trace
trace-cmd stop
- Extract the trace
trace-cmd extract
- The command produces a trace.dat file. It can viewed in text via
"trace-cmd report" or kernelshark.
- Disable the events
trace-cmd reset
- Manual
- Enable events
cd /sys/kernel/debug/tracing/
echo 1 > events/sched/enable
echo 1 > events/irq/enable
echo 1 > events/irq_vectors/enable
- Start the test
- Once the error happens, stop the trace (assume you are still in the
previous folder)
echo 0 > tracing_on
- Extract the trace
cat trace > ~/trace.txt
- The average editor can be used.
- Disable the events (and reset).
echo 0 > events/enable
echo 1 > tracing_on
And I have to create a test environment that is known to fail! Man that will hurt after al the effort I've put into optimising!
Replied by rodw on topic error finishing read
So I just received the tracing methodology. I need to issue a terminal command when the packet error happens. Can someone help me with this?If you have tracing enabled you would want to stop it on a packet error,
not EFR. This is available hal pin so you could launch a script
to stop the trace when it becomes true (or the packet error total is > 0)
They gave me two methods. I thought trace-cmd sound the easiest way (it is in the debian repos). We need to issue this command:
trace-cmd stop
Warning: Spoiler!
Either with a tool like trace-cmd or manual.
- trace-cmd
- Enable events:
trace-cmd start -e sched:* -e irq:* -e irq_vectors:*
- Start the test
- Once the error happens, stop the trace
trace-cmd stop
- Extract the trace
trace-cmd extract
- The command produces a trace.dat file. It can viewed in text via
"trace-cmd report" or kernelshark.
- Disable the events
trace-cmd reset
- Manual
- Enable events
cd /sys/kernel/debug/tracing/
echo 1 > events/sched/enable
echo 1 > events/irq/enable
echo 1 > events/irq_vectors/enable
- Start the test
- Once the error happens, stop the trace (assume you are still in the
previous folder)
echo 0 > tracing_on
- Extract the trace
cat trace > ~/trace.txt
- The average editor can be used.
- Disable the events (and reset).
echo 0 > events/enable
echo 1 > tracing_on
And I have to create a test environment that is known to fail! Man that will hurt after al the effort I've put into optimising!
Please Log in or Create an account to join the conversation.
16 May 2023 12:12 #271453
by rodw
Replied by rodw on topic error finishing read
I'm assuming we need a Python script to do this but I am way outside my comfort zone with Python, let alone adding Linuxcnc into the mix!
Please Log in or Create an account to join the conversation.
16 May 2023 14:36 - 16 May 2023 14:41 #271459
by PCW
Replied by PCW on topic error finishing read
Python is not needed, a simple bash script should do.
Something like:
#bash
stat=0
while (($stat < 1))
do
stat=`halcmd getp hm2_7i96s.0.packet-error-total`
done
trace-cmd stop
Something like:
#bash
stat=0
while (($stat < 1))
do
stat=`halcmd getp hm2_7i96s.0.packet-error-total`
done
trace-cmd stop
Last edit: 16 May 2023 14:41 by PCW.
The following user(s) said Thank You: rodw
Please Log in or Create an account to join the conversation.
16 May 2023 20:11 #271466
by rodw
Replied by rodw on topic error finishing read
Thanks Peter. Nothing is simple about bash for me
So I think If I install Bullseye on known problematic hardware, the 5.10 kernel should trigger the EFR
Then maybe try on Debian Bookworm 6.1 kernel
So I think If I install Bullseye on known problematic hardware, the 5.10 kernel should trigger the EFR
Then maybe try on Debian Bookworm 6.1 kernel
Please Log in or Create an account to join the conversation.
18 May 2023 20:42 - 18 May 2023 20:42 #271623
by PCW
Replied by PCW on topic error finishing read
I have been running the 6.3.1-rt13 kernel for about a week now
and it seems to be the best so far of 6.X kernels in terms of
network latency (3 systems, 0 packet errors.at 1 KHz for a week
one running on my normal desktop so plenty of other things going on)
and it seems to be the best so far of 6.X kernels in terms of
network latency (3 systems, 0 packet errors.at 1 KHz for a week
one running on my normal desktop so plenty of other things going on)
Last edit: 18 May 2023 20:42 by PCW.
Please Log in or Create an account to join the conversation.
18 May 2023 20:55 #271624
by rodw
Replied by rodw on topic error finishing read
Great Peter,
I have not built the latest 6.3.1 kernel. I was hanging out for 6.4 which apparently is better again!
We've made one attempt with a trace last night and just got some feedback. We are just sorting out the right syntax to keep our trace away from the isolated cores before retesting.
I have a couple of other helpers now willing to test and a Kernel Developer at Red Hat has also thrown his hat in the ring to help us too!
I have not built the latest 6.3.1 kernel. I was hanging out for 6.4 which apparently is better again!
We've made one attempt with a trace last night and just got some feedback. We are just sorting out the right syntax to keep our trace away from the isolated cores before retesting.
I have a couple of other helpers now willing to test and a Kernel Developer at Red Hat has also thrown his hat in the ring to help us too!
Please Log in or Create an account to join the conversation.
19 May 2023 02:51 #271642
by blazini36
Replied by blazini36 on topic error finishing read
I'm working on getting my 7i92 project going and I'm having a pretty rough time using an Odroid H3. I used to be pretty fond of H2's but I haven't gotten anything running LinuxCNC on an H3 and I'm just now realizing the H3 itself is the culprit.
I initially thought there was something wrong with the 7i92 since no IO was working but it was connecting until I realized I was getting "error finishing read" immediately after starting LinuxCNC. I tried separate 5v power supplies on the 7i92 and several different ethernet cables.
I started with 2.9 from the apt repo on Bookworm with the 6.1 kernel. the NICs were using the r8169 module. (they are r8125s) and I was getting over 1ms pings. R8125-dkms is in the non-free repo so I installed that....no better. I then followed all the most of the suggestions in Rodw's notes including installing the 6.3 kernel and nothing has changed.
I posted on the Odroid forums but nothing there yet. Anyone got LinuxCNC actually running reasonably on an H3 or something similar?
I initially thought there was something wrong with the 7i92 since no IO was working but it was connecting until I realized I was getting "error finishing read" immediately after starting LinuxCNC. I tried separate 5v power supplies on the 7i92 and several different ethernet cables.
I started with 2.9 from the apt repo on Bookworm with the 6.1 kernel. the NICs were using the r8169 module. (they are r8125s) and I was getting over 1ms pings. R8125-dkms is in the non-free repo so I installed that....no better. I then followed all the most of the suggestions in Rodw's notes including installing the 6.3 kernel and nothing has changed.
I posted on the Odroid forums but nothing there yet. Anyone got LinuxCNC actually running reasonably on an H3 or something similar?
Please Log in or Create an account to join the conversation.
19 May 2023 04:11 #271644
by rodw
Replied by rodw on topic error finishing read
I'm sorry to hear of your issues but if you have such a persistently bad kernel, I'd love it if you could join our kernel testing team!
You can contact me here to join www.vmn.com.au/contact
Somewhere on the Odroid forum it is mentioned that building the R8125 driver from the realtek sources gives 15-30% more performance.
You can enrol that driver in the dkms system so it updates if the kernel has changed. See its docs.
I do know one member told me he did that and had no issues. Not sure if it was H3 or H2. I have a H2 in service but its using ethercat and has not had an issue.
But I would also be looking at all the ususal BIOS settings and/or setting cstates and isolcpus etc in grub.
You can contact me here to join www.vmn.com.au/contact
Somewhere on the Odroid forum it is mentioned that building the R8125 driver from the realtek sources gives 15-30% more performance.
You can enrol that driver in the dkms system so it updates if the kernel has changed. See its docs.
I do know one member told me he did that and had no issues. Not sure if it was H3 or H2. I have a H2 in service but its using ethercat and has not had an issue.
But I would also be looking at all the ususal BIOS settings and/or setting cstates and isolcpus etc in grub.
Please Log in or Create an account to join the conversation.
19 May 2023 04:15 #271645
by blazini36
Replied by blazini36 on topic error finishing read
I looked back through the beginning of the thread and noticed PCW suggested running a 2ms servo thread. Not sure why that didn't dawn on me just to get going but it nonetheless sounds like a horrible thing to have to do.
I haven't used the 4.19 kernel in a long time. I know 6.x added some GPU stuff that these newer intel IGPU's need. Assuming that theres 6.x kernels really are this horrible for RT performance, have the older LTS kernels been back ported sufficiently?
I haven't used the 4.19 kernel in a long time. I know 6.x added some GPU stuff that these newer intel IGPU's need. Assuming that theres 6.x kernels really are this horrible for RT performance, have the older LTS kernels been back ported sufficiently?
Please Log in or Create an account to join the conversation.
Time to create page: 0.115 seconds