You are right I was not selecting the group, instead, I was selecting a face.
My confusion was because I didn’t know that variable g was short for ‘group’. I should have figured it out when I saw g.entities but I didn’t. I would recommend to @javalitterboy to not name variables with a single letter next time so the code is understood better.
I would use the following to get a selected group.
model = Sketchup.active_model
selection = model.selection
group = selection.grep(Sketchup::Group)
msg = 'Please select a group before using this tool'
UI.messagebox(msg, MB_OK) if group.nil?
When using the code @DanRathbun posted and I select the group it works, but if you have a circle inside a group it transforms the vertex in the wrong direction. Seems it is using another axis that is different than the group axis.
However when I try running the code on the model that @javalitterboy shared it gives an error.