We know what the problem is now - the V-Ray extension. I could sensed it through the internet and @mchandler verified it from your bugsplats. Mchandler works for Trimble. He is an intermediary. The Chaos Group developed and owns V-Ray copyright. V-Ray 3, which runs on SU17, hasn’t been out for very long. So there may be some issues with this new release. From what you shared here about you computer, you have a very nice system. It is right for you to expect more. Yet there seems to be some issues. Your computer “broke” V-Ray. So it turns out your computer could be a nice machine to help sort out problems with V-Ray - which, in the end, will make V-Ray, and you the V-Ray user, very happy. But that means things in the V-Ray extension code needs tweaking. That’s something the Chaos Group will deal with. Your role is rather passive - but very valuable. You break things that someone else will fix.
Go to the V-ray forum and intermingle with fellow V-Ray users. There even is a dedicated V-Ray forum on SketchUcation.