Perhaps there is some natural confusion about what Diffusion really is. This is not a rendering engine, it does not calculate physical light off of objects in 3Dspace like a rendering engine nor does it create specific materials for any given surface.
This is more like a game you play with an alien artist who lives on the moon and watches the earth through a telescope. The AI has some vague ideas about what this planet is and how things work around here, but it doesn’t really know the “why” of what it has observed. You send off a letter (your prompt and input geometry) to the moon and describe a picture and then the artist paints it according to their own ideas and sends it back. The better you describe the painting you want the closer it can get but being an alien and not living in this world it’s also bound to get some fundamental things wrong from time to time. With these expectations, and properly used, diffusion is capable of some extraordinary things.