There exist probably questions to worry more about. Partly you have answered yourself, partly does it depend on SketchUp internals that we cannot know.
Unused components are in the “definitions list”, but not in the scene graph (entities hooked into the hierarchy below model.entities). It is more likely performance suffers from the used components that your create rather than from unused components.
Rendering: probably no impact, since only the scene graph of used entities is traversed, not necessarily all definitions.
Picking and inferencing: no
Drawing geometry and Intersection operations: no
Creating new definitions or deleting definitions: depends very much on internal data structures used to manage components and possible other dependent datastructures that need updating.
Worst: linear (or worse) slow down with number of components (O(n) insertion in array)
Best: no impact (O(1) set operations) apart from memory allocations.
(However it is known that long material lists affected performance of the materials browser.)
Thumbnailing all definitions in component browser: yes, takes linearly more time the more (unused) components you have, but usually thumbnails are cached in the component and only generated once after creation or editing.