ODroid as raplacement for Raspberry Pi

More
21 Feb 2023 00:32 #264926 by Bari
Yes we use cyclictest to compare to other systems than run cyclictest. However, FF + youtubes gives us the worst latency test scores vs any other tests, cyclictest included..
The following user(s) said Thank You: rodw

Please Log in or Create an account to join the conversation.

More
21 Feb 2023 08:10 #264934 by alangibson
> The real test for latency has been starting Firefox and then playing a youtube.

You have to be careful when doing stress testing. The test needs to look like the load you expect to have for the numbers to be relevant. The workload of this board is only LinuxCNC, no browsing or video decoding ever. And AFAIK the workload of decoding video is nothing like the workload of running LinuxCNC.

If you go by playing video, then you're getting absolute worst case numbers of the board as a video decoder, not the board as a LinuxCNC host. Now if you were going to use the board to watch video while LinuxCNC was running, then I agree that playing a video as a load test would be good. But this board will run LinuxCNC in kiosk mode.
The following user(s) said Thank You: rodw

Please Log in or Create an account to join the conversation.

More
21 Feb 2023 08:31 #264935 by rodw
I agree with this Alan. Lets face it. If you are doing things with the PC, you are not (or shouldn't be) running a machine at the same time...
I've been scouring the net for kernel config settings for arm and preempt_rt.
Interesting there is a new preempt mode that lets you select PREEMPT_RT and other modes as you would isolcpus. But not in the 5.10 kernel you use.

Please Log in or Create an account to join the conversation.

More
21 Feb 2023 08:39 #264938 by alangibson
> If you are doing things with the PC, you are not (or shouldn't be) running a machine at the same time...

Agreed. I'm planning to run LinuxCNC in kiosk mode so you literally can't do anything else. I won't even have a desktop installed.

> I've been scouring the net for kernel config settings for arm and preempt_rt.

I did the same and came up with the ones you see in that build script. Nothing else seemed to do any good, or actually made the numbers worse.

This is probably a good time to mention that it's critical to start processes with schedtool (like `schedtool -a 2-5 -e nice -n99 latency-histogram`). Giving the process a high priority and binding it to the isolated cpus has a big positive effect on latency.

> Interesting there is a new preempt mode that lets you select PREEMPT_RT

That's interesting. I only knew about the CONFIG_PREEMPT_RT kernel build config setting that gets you an RT kernel.

Please Log in or Create an account to join the conversation.

More
21 Feb 2023 09:08 #264943 by alangibson
> Everything looks right but something goes awry with threading

I had about every failure imaginable. I tried Armbian, but the build would fail I think because one of the official RT patches wouldn't apply cleanly.

I also had lots of boot failures with tobetter's Ubuntu 22.04 image. It u-boot appears to be very sensitive to wrong kernel commandline in boot.scr.

The compiler you use also seems to matter a lot. The kernel wouldn't boot when built with Linaro ARM compiler v7 (as HardKernel recommends), but it works perfectly when built with Linaro ARM compiler v12.

And LinuxCNC won't build with Linaro v12. I have to switch to using the cross compiler in the crossbuild-essential-arm64 package.

> Do you have an image you can share somewhere?

Give this a try: drive.google.com/file/d/1wbU988tXvZyPX5V.../view?usp=share_link

(Warning to future readers: that link will go dead once we have a better build option)

> Has Tobetter's image already got it in there?

They maintain Odroid and PREEMPT_RT patches in a separate repo: github.com/tobetter/linux

> If we can migrate the process to Armbian's build system, the process will be a lot easier.

Indeed. It looks like Armbian just pulls from HardKernel's kernel repo. In theory you could just swap in tobetter's repo url. I'm pretty sure there are some other build differences though.

> It would be good to see the output from cyclictest as it speaks a language other RT guys understand.

I'll post those numbers today.

Please Log in or Create an account to join the conversation.

More
21 Feb 2023 13:00 #264953 by rodw
I've forked the Armbian Build process here
github.com/rodw-au/armbianbuild
I'm currently building for Bookworm edge with./compile.sh EXPERT=yes
build an image
don't change the kernel
add a graphical environment xfce
add browser, editor internet
build and burn to an SD (not emmc yet)
I'm not sure how much can be stored in git yet....

I think Armbian just pull the pettitboot (and maybe uboot) from HardKernel.
You can select the image used by editing this file so you can use a matching RT build to your kernel
github.com/armbian/build/blob/master/con...e/meson64_common.inc

These are some of the kernel settings I wanted to review


CONFIG_HAVE_PREEMPT_LAZY=y
CONFIG_PREEMPT_LAZY=y
# CONFIG_PREEMPT_NONE is not set
# CONFIG_PREEMPT_VOLUNTARY is not set
# CONFIG_PREEMPT is not set
CONFIG_PREEMPT_RT=y
CONFIG_PREEMPT_COUNT=y
CONFIG_PREEMPTION=y
CONFIG_KVM=n
CONFIG_VIRTUALIZATION=n
CONFIG_LOCALVERSION_AUTO=y
CONFIG_ARCH_SUPPORTS_RT=y
CONFIG_HZ_1000=y
CONFIG_HZ=1000
CONFIG_HZ_PERIODIC=y
CPU_IDLE_GOV_LADDER =y
EXPERT=y ?????????
CONFIG_NO_HZ_FULL=y

CONFIG_CPU_FREQ=y
CONFIG_CPU_FREQ_GOV_PERFORMANCE=y
CONFIG_CPU_FREQ_GOV_ONDEMAND=y
CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE=y
CONFIG_CPU_IDLE=n
CONFIG_EMBEDDED=y
I also wanted to review toebetters commits to see what is changing in his kernel.

Please Log in or Create an account to join the conversation.

More
21 Feb 2023 13:20 #264957 by alangibson
To the best of my knowledge,

Master switch to build RT kernel
CONFIG_PREEMPT_RT=y

Needed to enable CONFIG_PREEMPT_RT (on >= 5.15 ?)
CONFIG_VIRTUALIZATION=n
CONFIG_ARCH_SUPPORTS_RT=y

Doesn't matter due to CONFIG_PREEMPT_RT
CONFIG_HAVE_PREEMPT_LAZY=y
CONFIG_PREEMPT_LAZY=y
# CONFIG_PREEMPT_NONE is not set
# CONFIG_PREEMPT_VOLUNTARY is not set
# CONFIG_PREEMPT is not set
CONFIG_PREEMPT_COUNT=y
CONFIG_PREEMPTION=y

Alias for CONFIG_EXPERT (lwn.net/Articles/421304/)
CONFIG_EMBEDDED=y

Off because we don't need virtualized guests
CONFIG_KVM=n

Automatically append version information to the version string
CONFIG_LOCALVERSION_AUTO=y

Mutually exclusive. We want PERFORMANCE so we're always at max frequency
CONFIG_CPU_FREQ_GOV_PERFORMANCE=y
CONFIG_CPU_FREQ_GOV_ONDEMAND=y

menuconfig Menu only
CONFIG_CPU_FREQ=y

menuconfig Menu only (github.com/torvalds/linux/blob/master/drivers/cpuidle/Kconfig)
CONFIG_CPU_IDLE=n
CPU_IDLE_GOV_LADDER =y

Please Log in or Create an account to join the conversation.

More
22 Feb 2023 00:12 #264995 by rodw
Thanks, I will try and set some things tonight.
Currently am using kernel 6.2 rc 3  with Armbian.edge.
Thinking about it I will revert to 6.1.12 and Armbian Current. This should boot on an eMMC and align with the bookworm kernel (and htere is a RT patch)
In my PREEMPT_RT branch, there is a file called command.txt which is a single command that the build envionment gives me at the end which should replicate what I have run. I'll keep it current. It will take a couple of days to get this sorted out before my repo will do much.

It will be something like
git clone https://github.com/rodw-au/armbianbuild
git checkout PREEMPT_RT
#run contents of command.txt.

Give it a go and let me know how it goes (burn on SD card not eMMC and boot with switch to the right). Image will be in the output folder.
I use Balena Etcher on Linux whjich is super cool but eventually, we can make it burn as part of the script if we want.






 
The following user(s) said Thank You: CarterKraft

Please Log in or Create an account to join the conversation.

More
22 Feb 2023 00:43 #264998 by Bari
> You have to be careful when doing stress testing. The test needs to look like the load you expect to have for the numbers to be relevant.

We have little control over how the applications and drivers are developed so we look for worst case in testing. If you wish to gamble on what the lowest latency might be to run a base thread for software stepping on your hobby machine that is up to you. There is no latency police enforcing anything involved here :)

If any of the ODroids end up testing reliable enough to use in production machines they will be be running servo threads anyway with external FPGAs for steppers, encoders and servos.

Please Log in or Create an account to join the conversation.

More
22 Feb 2023 10:34 #265025 by rodw

If any of the ODroids end up testing reliable enough to use in production machines they will be be running servo threads anyway with external FPGAs for steppers, encoders and servos.

I don't think there is any doubt Odroid's  are reliable as I've used several of their  x86 devices and they work fine once you get the NIC drivers installed. (Hint: the 5.10 kernel -  Bullseye and above includes the driver). One of them runs Bullseye and Ethercat. Its just their user base is not really worried about Real time so we are not catered for.

You see ignorant comments like this one whxih was well retorted!
forum.armbian.com/topic/13250-preempt-rt...omment&comment=96666
 
Curious as to why one wants to apply the patches for preempt-rt in the first place? Do you have a reason/application that would benefit? Most times, one can see an overall decrease in application performance, and to take advantage of preempt-rt, the application must call pthreads in a very specific way - which means that many times, the app must also be rebuilt.

My reason for asking - if preempt-rt was so awesome, it would already be in the upstream linux kernel mainline and not require the patch. Avoiding politics around preempt-rt - many of the benefits here can be done without the patches, by selecting an appropriate scheduler 
Some people have no idea. 20 ms is real time what crap!
Nice reply
Curious as to why you waste your time if you don't want to help?

With the Odroid N2, you get a 6 core CPU (4 BIG, 2 little for power saving)

I think if you look at kernel development both normal and real time, there is a lot of effort happening to support the ARM environment, some of it is directed at the N2. You can't use this stuff on the Buster 4.x kernel......

Please Log in or Create an account to join the conversation.

Time to create page: 0.090 seconds
Powered by Kunena Forum