Tool Lengths Offsets and Touch off

Getting myself in a mess wrt to a machining job on a three axis mill and three tools that get manually changed.
Spindle has a quick change system that accepts ISO type holders. Each tool has its own backend.
I have been setting the three tools by calling T1M06, T2M06 etc  in the MDI, putting the appropiate tool into the spindle and then touching it down on a setting gauge, until it registers zero, clicking the Tool touch off button. Repeat for the other two tools after calling for the tool change.
Then I take one of these tools, and use the Z touch off button to touch off onto the work piece datum.  
Can then check the tools individually, by changing to the tool, running G43 in the MDI, and Issuing G0 Z0.
The Z figure in the DRO, is zero and the tool is at the workpiece daum height. So far so understandable.
At this point things get a bit shakey if say I want to pull T1 tool out of its holder another 10mm because I've not allowed enough clearance between the collet nut and the  workpiece.
If I adjust the tool, and put it back over the setting gauge, which is at the same fixed point on the machine it was at the start, and re tool touch it off and call G43, the z height reading on the DRO is zero again, but the actual z position wrt to the workpiece  is around 12mm off from being where I expected it to be.

I think I am missing a bit of the puzzle somewhere, as I cannot see mistake I am making.

Thanks

I don't know if this is needed or not (my M6 macro does it for me) but are you calling G43 after each tool change where you don't touch off, to load that tool's offsets?