Test LinuxCNC without motors and encoders connected

Peterdeleu wrote:

But I don't know what the difference is between incremental and differential....

It seems likely that your encoder is both incremental _and_ differential.

An incremental encoder outputs a quadrature signal
AB
aB
ab
Ab
AB
and so on.

A differential encoder sends the signals as +ve and -ve differential voltages on two wires per phase

A+ B+
A- B+
A- B-
A+ B-

A single ended encoder sends the signal as 0V or 5V relative to ground on just one wire per phase

A5V B5V
A0V B5V
A0V B0V
A5V B0V

Depending on exactly where you stop turning the encoder you can see any of these states. So you would expect to not be able to predict the output when turning by hand.