Your rectangles are components separate from the building. While you are editing the window component you cannot make changes to the building geometry. You need to be editing the building, not the window.
Yoiu should go through the tutorials at learn.sketchup.com. This is a fundmental thing in SketchUp and you have to get it sorted out before you can make any good progress.
This sounds a little more complicated than it needs to be.
Here is a collection of 3 rectangles. One is just geometry. The second is a group. The third is a component. For one, geometry, you just need to draw your shape (window). For groups and components you need to ‘click into’ the context. Here I did that and erased the rectangle.
Here is a ‘wall’ of geometry; a group, and a component. I clicked into the context of the component and the group. For the geometry I, no need to click inside. I used Push/Pull. Make sure to push to the face to the other side (you can see when that happens if you look at the cursor).
you are in the right (= editing) context when you use the ‘Push/Pull’ tool. But you can’t inference on an edge or vertex on the back side of your slab from this camera position. Orbit a bit to see at least an edge.
Or hit [k] to see all the back edges of that slab to use for inferencing.
You can only cut one face with a ‘Glue to’ > ‘Cut opening’ component. So even after deleting the inner rectangle after push/pulling to the back you’ll still see the entire large back face. A 'Intersect Faces > with model from within the highest context (the face context) will help to create the small back rectangle and to delete its face.
Don’t create the component. Just draw the window on the geometry of the wall and push/pull it.
Some little errors have crept into the model you shared. Go to Window → Model Info and remove the checkmark next to, “Enable length snapping”. That will help prevent accidentally making measurement errors. You should save a template with that turned off.
Why did you created a component inside the group? Just draw the rectangle, don’t make it a group or component and then use the push pull tool to make a hole. If you want yo create a window component do it outside the group.
I’m sure that all this stuff is explained on the different courses from the sketchup website.