Under normal ladder logic creation there is no way for you to make the custom function label name a contact bit in the ladder program because that's not the way the program works.
You probably had used the the "edit label" name to force this to happen and that't why the compiler did not detect the error.
The normal way is for the ladder program to trigger the custom function written in TBASIC. The custom function can change the ladder logic element's logic state by using SETIO, CLRIO, TOGGLEIO or by changing the system variables representing the I/O such as INPUT[n], OUTPUT[n], etc.
You can take a look at many examples in the
"c:TRiLOGI\TL5\usr\samples"