With DP cable, it works better, but still has massed up screen.
The card you tested is GTX, not RTX; and GTX is q quite old graphic card.
I have another laptop with GTX1060, it works smoothly never give an issue.
So there must be something between card driver and the java version you use.