Encoder Spindle Help

I have a Masso Optical Encoder (signals A,B and Index) hooked to my Mesa 7i96 board. I can see the signals when I look at the Hal show and watch, so the signals are coming across. My speed is not reading constant though. I am not sure what settings I need to put in the configuration to have it work properly. I am attaching a picture of the encoder decal on my spindle.

 

That's pretty low resolution (20 counts/turn) so I would not expect very good velocity signals.

Can you adjust the A-B phasing? (by repositioning one of the sensors)

Lack of 90 degree phase between A an B will cause large velocity
estimation errors (90 degrees means A changes right on the middle
of the B high or low state and vice versa)

No, I can't reposition the sensors as they are on a pcb. I am not sure how Masso does it, they say it is really accurate, but I am not really sure.

What I can do is make a decal that would have more divisions on it, assuming all the encoder does is pick up signals and send them to the card, I am wondering if that may be a good route to take.

I used the Halscope to see the inputs and attached that. It seems it is reading correctly, but my knowledge of using the Halscope is really limited. I just can tell something is coming across. 

You can see that there is substantial quadrature error
that is, the time between edges varies by a factor of about 2:1

This will result in about a 2:1 variation in the velocity calculation
depending on the timing of the last set of edges.

Note that you cannot in general change the number of divisions
without changing the spacing between the A,B sensors or you
will cause quadrature errors.

If the distance from black to black = N, the spacing from A sensor
to B sensor must be 1/4N + M*N, where M is an integer.
 

So, is what I have worth pursuing? or should I go in a different direction? It seems it has the potential to work, and the Masso system uses it to do threading, which is what I'd like to be able to accomplish eventually if possible.

Before, when I tried someone told me that with what I had there would be errors at low speed, if that is the case, what is the minimum RPM I should expect to get a decent reading?

Not really sure but 20 counts per turn is pretty crude
It also depends on how stable spindle speed is.