This is because when you click on a face, you select that face not its material property. Then pressing delete erases that face because it is what is selected. There is no way in the view to directly select the material on a face. There are, however, other routes.
You can open Entity Info while the face is selected. Entity Info then displays and lets you alter various properties of the face. The applied material is one of these, and shows up in the sample swatch at the top of entity info:
If you double-click that swatch, the Materials (labelled “Colors” on Mac) Window will open and you can drag a different material onto the swatch to change it. The default material is among the choices on “Colors in Model” in the Materials WIndow.
Alternatively, you can activate the paint bucket tool, select a material in the Materials Window, and click on the face to paint it with that material.
To set up this animation, I made a simple component with two rectangular faces inside it. I painted the component with a pink color. I then opened the component for edit and painted one of the faces with a blue color. In the animation I click first the component to show how it is painted. Notice that the blue painted face overrides the component color - it shows blue. But the other face shows as having the component’s color. Then I open the component for edit and click each face to show how they are painted. This reveals that the one that appears pink actually has the default material applied (which is really “no material”).