Medeek: It is always good to brainstorm about some base algorithm.
The problem with intersection at an edge is that the intersection can be counted either as ZERO or ONE, as illustrated on the picture below. The result is given by using a small variation of the ray and checking how many faces of the edge it intersects.
So the algorithm can be adapted to make this test.
When the intersection is at a vertex, then we consider one arbitrary edge at this vertex and make the test on this edge.
Note also that it is possible to use a Raytest algorithm. The benefit would be performance in the case the solid has a huge number of faces (I assume that raytest() is therefore much faster)
This imposes however that you are given the transformation of the solid at top level, because model#raytest operates at top level of the model. Another complication is that raytest() does not return the start of the ray when it is on a face. There is a way to circumvent the two issues, as in the code below. Indeed, the extra test for edge and vertex intersection should be added.
#Check if a point is within a solid <grouponent>
# Returns :inside, :outside or :on_face
# Point <pt> must be in top model coordinates system
def G6.point_within_grouponent(pt, grouponent)
model = Sketchup.active_model
vec = Geom::Vector3d.new(rand, rand, rand)
nb_inter = 0
ptback = false
bbox = grouponent.bounds
trinv = nil
#Get the next intersection point
ptinter, chain_comp = model.raytest([pt, vec], true)
#Special treatment with backward raytest (done only once) to check if the point is on a face
ptback = (ptinter)? ptinter : pt.offset(vec, 0.1)
ptinter2, chain_comp = model.raytest([ptback, vec.reverse], true)
return :on_face if ptinter2 && ptinter2 == pt && chain_comp.include?(grouponent)
#Stop if no more intersection point
break unless ptinter
#Top level transformation for the grouponent (compute it once only)
if !trinv && chain_comp.include?(grouponent)
tr = Geom::Transformation.new
chain_comp.each do |c|
break if c == grouponent
tr = tr * c.transformation
trinv = tr.inverse
#Check if outside the bounding box of the grouponent
break unless trinv && bbox.contains?(trinv * ptinter)
#Counting the intersection
nb_inter += 1 if chain_comp.include?(grouponent)
pt = ptinter
#Return the status
(nb_inter.modulo(2) == 0) ? :outside : :inside