# Concavity problem and need help

You can compare the cross product of the face normals with the edge’s “actual” direction, with whether the edge is considered reversed in one of the faces.

``````# Assuming the faces are oriented the same way, the faces are not co-planar and
# the edge binds exactly two faces.
def concave?(edge)
faces = edge.faces
vector = faces[0].normal * faces[1].normal

vector.samedirection?(edge.line[1]) == edge.reversed_in?(faces[0])
end
``````

If the faces are correctly oriented any shared edge must be considered reversed in one of the faces but no the other, compared to the faces’ winding order. You can also check the “reversity” in relation the the second face, but then you must also swap the two faces when calculating the cross product (which reverses the product vector). What is the first and second face makes no difference, as long as it is internally consistent in the method.

In a real implementation you should also confirm there is only 2 faces bound by the edge, they are correctly oriented (the “reversity” of the edge differs between them) and the faces aren’t co-planar (the cross product of the normals bot being zero length).

2 Likes