Like @bmike accurately pointed out, this sounds pretty much like a situation that BIMShell could handle well. Please give it a try (particularly the Outer Shell option, if you want to keep the texture, but reduce the polygon count). It basically removes all internal geometries, and it can get coarser to finer according to the grid resolution that you set.
I would be curious to learn to which extent it can help solve your issue.