Alas, that’s not how layers/tags are implemented. Each entity can point to a single tag and thereby use its settings, but tags have no pointer the other way; they have no record of what entities are using them. So, traversing the entire model is the only way to find what uses a particular tag.
This one-way relationship is a deliberate aspect of how layers in SketchUp were originally conceived: they are consulted by the rendering engine as each entity is considered for drawing. Drawing inherently involves walking through all the entities. There has been a lot of discussion of other/additional semantics for tags. It seems likely that some other facility could be implemented at some point to serve these purposes.