How can i use the function of linuxcnc by a custom remote windows UI?

If you can get Windows to open a network socket, and get the UI to send messages, then it ought to work.

There is documentation here: linuxcnc.org/docs/2.8/html/man/man1/linuxcncrsh.1.html

I have only experimented with it very briefly.
I think that you can assume that it works, but would need to check whether all that you want to do can be done through it.

If you require more then possibly you should look at Machinekit, they have "Machinetalk" which seems to be aimed at this sort of remote-GUI task.