I only just recently picked up SubD and Vertex tools and I think they’re probably the right tool for this job, but I haven’t learned them yet.
This is what I remember from school as a Boundary Value Problem: two regions of space where both the value and the derivative of the functions must equal each other at the boundary for the sake of smooth continuity. Here I have one region with waves and the other with flat water. I think Vertex Tools could scale the wave at the boundary to zero amplitude to match the flat water, and it’s selection tool could taper that effect off into the rest of the waves, perhaps? Am I heading in the right direction?