Remora - Rpi Software Stepping Using External Microcontroller via SPI

More
22 Apr 2022 10:07 #240874 by Rafal
Hi folks. I'm attempting to integrate Remora with my CNC minilathe and for the most part it's been trouble free. I'm new to LinuxCNC - so apologies for any obvious mistakes I make.

The setup is an rpi4 alongside STM32F106RCT6 based SKR mini 1.1 running a lightly modified version of develop. The modifications were just to setup SPI for this board (port A -> B, CS -> PA15) on the SKR_MINI_E3 target. The board is flashed by an STLINKV2.

On the pi side I've modified the xyz hal and ini files to suit my setup (2 joints, XZ config) and commented out spindle and other modules to ensure no interference.

I've run through and fixed a whole host of issues including SPI packet errors and configuration problems. Most of the solutions were already in this topic.

The long standing issue crops up enough to make the lathe unusable. You can move the system on either axis and when it stops or nears stopping, it (sometimes) suddenly jumps into motion (step change) and starts slowing down. This can be seen below. The position feedback value changes and the system compensates.



Some interesting observations:
  • During the 'tail' end of movement, if you press F2, it will stop and continue the same movement when re-enabled with F2
  • This behavior seems to be related to the DDS accumulator overflowing - this is consistent. Slowing movement near the edge of the int32 will cause this issue (when it overflows).
  • The issue still occurs if you significantly reduce the DDS step bit (I had it at 10 in both fw and sw). Although it is less consistent, and occurs well within the int32 boundaries.

Has anyone seen or fixed this behavior? I'd appreciate any insight!

-Rafal
Attachments:

Please Log in or Create an account to join the conversation.

More
22 Apr 2022 22:04 #240923 by scotta

made some progress 
void loadModules()
{
    //Stepgen
    int joint;
    ptrInputs = &txData.inputs;
    ptrOutputs = &rxData.outputs;
    joint = 0;
    printf("\nCreate step generator for Joint %d\n", joint);
    ptrJointFreqCmd[joint] = &rxData.jointFreqCmd[joint];
    ptrJointFeedback[joint] = &txData.jointFeedback[joint];
    ptrJointEnable = &rxData.jointEnable;
    Module* joint0 = new Stepgen(PRU_BASEFREQ, joint, "PB_12", "PB_13", "PB_14", STEPBIT, *ptrJointFreqCmd[joint], *ptrJointFeedback[joint], *ptrJointEnable);
    baseThread->registerModule(joint0);
    
    //Blink
    Module* blinkServo = new Blink("PB_0", PRU_SERVOFREQ, 1);
    servoThread->registerModule(blinkServo);
    //Input
    Module* IN1 = new DigitalPin(*ptrInputs, 0, "PD_12", 0, true, NONE);
    servoThread->registerModule(IN1);
    // OUTPUTS
    Module* OUT1 = new DigitalPin(*ptrOutputs, 1, "PC_3", 0, false, NONE);
    servoThread->registerModule(OUT1);
    // PWM (not working)
    Module* pwm = new PWM(*ptrSetPoint, "PC_4");
    servoThread->registerModule(pwm);

@ Scott : everything works except PWM if you can help me

Hi, which board are your setting this up on?

The setPoint is an array of floats so you need an array index
int sp = 0; //or which ever SP[i] you wish to use
ptrSetPoint[sp] = &rxData.setPoint[sp];
Module* pwm = new PWM(*ptrSetPoint[sp], pin, pwmMax);
servoThread->registerModule(pwm);

Please Log in or Create an account to join the conversation.

More
22 Apr 2022 22:11 #240927 by scotta

Hi folks. I'm attempting to integrate Remora with my CNC minilathe and for the most part it's been trouble free. I'm new to LinuxCNC - so apologies for any obvious mistakes I make.

The setup is an rpi4 alongside STM32F106RCT6 based SKR mini 1.1 running a lightly modified version of develop. The modifications were just to setup SPI for this board (port A -> B, CS -> PA15) on the SKR_MINI_E3 target. The board is flashed by an STLINKV2.

On the pi side I've modified the xyz hal and ini files to suit my setup (2 joints, XZ config) and commented out spindle and other modules to ensure no interference.

I've run through and fixed a whole host of issues including SPI packet errors and configuration problems. Most of the solutions were already in this topic.

The long standing issue crops up enough to make the lathe unusable. You can move the system on either axis and when it stops or nears stopping, it (sometimes) suddenly jumps into motion (step change) and starts slowing down. This can be seen below. The position feedback value changes and the system compensates.



Some interesting observations:
  • During the 'tail' end of movement, if you press F2, it will stop and continue the same movement when re-enabled with F2
  • This behavior seems to be related to the DDS accumulator overflowing - this is consistent. Slowing movement near the edge of the int32 will cause this issue (when it overflows).
  • The issue still occurs if you significantly reduce the DDS step bit (I had it at 10 in both fw and sw). Although it is less consistent, and occurs well within the int32 boundaries.


    Has anyone seen or fixed this behavior? I'd appreciate any insight!

    -Rafal

Hi Rafal. 

It looks like you have really understood the Remora firmware and the Step Generator implementation. And a great pick-up. How did you narrow this down to the DDS accumulator overflow? I'd like to check this out as following error spikes are something I'm looking into and to date have not had a clue where they were coming from. You may have found it!

In the mean time you can increase the pgain to bring the jump back down quickly.

Please Log in or Create an account to join the conversation.

More
23 Apr 2022 00:02 - 23 Apr 2022 00:08 #240945 by Rafal
Hi scotta,

I did some more testing and cannot reproduce the error on overflow consistently anymore - although it still does occur. It took a little bit of prodding this morning before it would produce the error again. Perhaps it's marginal and related to the micro coming up to temperature?

I've just placed a printf in Stepgen::makePulses that's slow enough to not trip estop (not pretty, but it gives insight):
static uint32_t count = 0;
if (this->isEnabled == true){
 this->enablePin->set(false);
  if(jointNumber == 0){
   if((count % 2000) == 0){
    printf("\n\t%d", this->DDSaccumulator);
   }
   count++;
  }

 

It could be that the feedback is being modified which causes an error in the LinuxCNC module which in turn compensates and adjusts the freq command. Or the other way around, where the error is generated in LinuxCNC. Perhaps something is overwriting memory - frequencyCommand, DDSaddValue, DDSaccumulator? I'd suspect the micro, but haven't looked at the hal module in any detail.

Thanks for your pgain increase suggestion. I tried it and it does help, although the issue still crops up and is quite large in magnitude (700 steps ~ 1 rotation over a few seconds), so it overshadows any pgain gains.

-Rafal
Attachments:
Last edit: 23 Apr 2022 00:08 by Rafal.

Please Log in or Create an account to join the conversation.

More
14 May 2022 00:15 #242926 by Rafal
After a bit more digging, I think I found the issue. The HAL module (remora.c) has two places where it calls
spi_transfer();
- line 882 & 997 (develop version). This means you get two back to back SPI transactions with ~50us between them when CS is high.

As the F103 starts copying data (from the first transaction) on the rising edge of CS, it isn't complete before the second transaction starts and DMA overwrites the original data. Since the header is first, and doesn't change, it's difficult to detect an error in the modified data.

As a temporary fix, I've commented out line 882 & changed the code around 997 to:
if( *(data->SPIstatus) )
{
  // Transfer to and from the PRU
  //spi_transfer();
}
spi_transfer();

A more robust solution is needed, but for the moment, my lathe is working well. No more jittering and no random 360 degree revolutions!

-Rafal

 

Please Log in or Create an account to join the conversation.

More
15 May 2022 01:19 #242994 by scotta
Hi Rafal, 

I'm surprised that things are working with this change. There should be two transfers, one to read the status of the board, then we calculate the new frequency commands based on the read, and then we do another transfer to write to the board with the new commands, set points and output values. The 50us between them is when the update_freq() function is run by LinuxCNC.

The header does change with each transfer, lines 863 and 955, so that the firmware in RemoraComms::processPacket() can determine what to do with the packet data. In a read the received buffer is not moved to the rxBuffer. So how is the board getting commands.... hmm

If the CS is not going low then we need to look at the spi_transfer() function, which is calling bcm2835_spi_transfernb().

 

Please Log in or Create an account to join the conversation.

More
16 May 2022 08:56 #243070 by Rafal
Hi scotta,

Yep, it was one of those 'deal with the consequences later' types of changes. I'm probably lagging a cycle. I can see the headers changing as you noted.. There is definite feedback from the board. Not sure how it's working!

In any case, I was trying to find the root of the issue. Narrowed it down to the SPI link, but not the physical interface. CS is working fine - it's a DMA issue from what I can see. Correct me if I'm wrong, but there's no double buffering. When the micro starts reading data on CS rising edge, it can take some time to finish and a new SPI transaction (the second one with the write) begins. There is potentially data being read from DMA, while the DMA is writing new SPI data.

To find the issue, I put a simple data sum checksum into the spare bytes in the read and write data structures. If either the HAL component or micro notice an error in the sum, they discard the packet.

This immediately changed the behavior from occasional glitches, to no glitches but very sparse update rate (only occasional packets were getting through). After seeing the back to back SPI transactions, I thought it likely the above DMA was causing this. It still could be something else that can be 'fixed' by increasing the delay between SPI transactions.

The changes I have made to make the lathe run smoothly include keeping the checksum. There is no more noticeable data intermittency / hesitance as I am running it.

Ultimately, it'd be great to come up with a fix that retains normal functionality. Are you able to reproduce the issue I'm seeing?

-Rafal

Please Log in or Create an account to join the conversation.

More
20 May 2022 22:40 - 20 May 2022 22:41 #243440 by scotta
Hi Rafal, 

A really interesting update. SPI has always been susceptible to noise on the bus and has caused issues for many people including myself. I get around this by using custom made pcb's to connect the board to the RPi or using very short ribbon cables.

Can you share your checksum implementation as this would be a great addition to Remora to make the comms more resilient to these types of issues.

For the F103, the packet processing is only done when the Chip Select rises (see line 36 in github.com/scottalford75/Remora/blob/dev...omms/RemoraComms.cpp) and the next DMA transfer is only enabled (line 182) after the data is moved from the temporary buffer to the rxBuffer. So maybe not related to the DMA SPI.
Last edit: 20 May 2022 22:41 by scotta.

Please Log in or Create an account to join the conversation.

More
20 May 2022 22:48 #243441 by scotta
Hi Rafal, 

Thinking more on your post "only occasional packets were getting through". What is your hardware setup? Maybe you are getting lots of noise on your SPI wiring if you are dropping lots of packets with your checksum implementation.

Please Log in or Create an account to join the conversation.

More
31 May 2022 23:56 - 01 Jun 2022 00:45 #244327 by cakeslob
sorry to interrupt with cakeslob bullshit but.... more cakeslob bullshit incoming....

Trying to setup an STM32h743, the new SKR3 uses this chip, a pretty big jump from f407 at 480hz (it looks pretty slick)

github.com/cakeslob/Remora/tree/stm32h74...areSource/Remora-OS5

I made the TARGET_STM32H7 based on the f4, with just a quick replace from f4 to h7
I made a custom target based on nucleo h743 called SKRV3, its not configured yet, but that I can mash my way through later,

github.com/cakeslob/Remora/tree/stm32h74...ARGET_STM32H7/thread

This part....I think I will need to actually change things...like.....timers and interrupts....or something....

Im using these as reference, Migration from Stm32F42x to Stm32F7, and then Migration from Stm32F7 to Stm32H743

www.st.com/resource/en/application_note/...microelectronics.pdf

www.st.com/resource/en/application_note/...microelectronics.pdf


Here is my interpretation so far

I dont really know what Im doing most of the time already, but from the looks of it, a lot of things are the same but better, but there are some different in the NVIC.

24 TIM1_BRK_TIM9 becomes TIM1_BRK
25 TIM1_UP_TIM10 becomes TIM1_UP
26 TIM1_TRG_COM_TIM11 becomes TIM1_TRG_COM


I dont know what it means, but I know its different and its in the createThreads.h so it will probably need to be changed. Am I on the right path here for porting to h7 or no?
Your patience is appreciated
Last edit: 01 Jun 2022 00:45 by cakeslob.
The following user(s) said Thank You: satsuma

Please Log in or Create an account to join the conversation.

Time to create page: 0.506 seconds
Powered by Kunena Forum