You can use the Entities #add_arc method to create an arc curve segment.
But definitely, you need to do your own maths to calculate the parameters for that method.

Here is what I came up with for future users. I am new to Ruby so I am sure this isnâ€™t very Ruby-ish.

# construct circular arc of given radius between two intersecting lines
# lines intersect at the vertex. Points pA and pB are any points on those lines
def fillet (pA, pVertex, pB, radius)
vA = (pA - pVertex).normalize!()
vB = (pB - pVertex).normalize!()
vNormal = vA * vB # orientation vector
theta = vA.angle_between vB # angle between sides
arc_angle = Math::PI - theta # included angle of fillet
# vector from vertex to arc center
vCenter = (vA + vB).normalize().transform( radius / Math.sin(theta/2) )
#vector from vertex to intercept on side A
vInterceptA = vA.transform(radius/Math.tan(theta/2))
pInterceptA = pVertex + vInterceptA
#vector from vertex to intercept on side B
vInterceptB = vB.transform(radius/Math.tan(theta/2))
pInterceptB = pVertex + vInterceptB
ents = Sketchup.active_model.entities
fillet_arc = ents.add_arc(pVertex + vCenter, vInterceptB, vNormal, radius, Math::PI/2, Math::PI/2+arc_angle)
return [pInterceptA, pInterceptB]
end

Took me a while to understand that 0 angle for arc_start wasnâ€™t on the x-axis required as a calling parameter. Seems silly.