Please specify what you mean by idle. Is this with LinuxCNC running or one of the other programs polling the pendant?
I left the pendant connected last night directly to the xhc-whb04b-6 component, and now some 15 hours later there still hasn't been a single disconnect.