It does sound as if you are clicking too quickly. A Triple click with Select opens the object for editing and selects all connected geometry. Either slow down and give a break between the double click and single click or right click on the group and choose Edit Group. Then single click on the face. You could also set up a keyboard shortcut for Edit Group but you’d have to single click to select the group first.
It shouldn’t take a full second delay between clicks.