Just ran into the same problem, and this thread helped resolve it for me, but I thought I’d add some clarification.
Like the OP, I am using a Mac with OS X El Capitan (10.11.6). I was using SU a while ago (several years) on my older Mac and I definitely assigned shortcuts, but I don’t remember running into this issue. Now I have a new(er) Mac and just downloaded SU Make '17 and started playing around with it and went to start assigning my old shortcuts and ran into the same “Key cannot be used as a shortcut.” message WHEN I HIT RETURN (which is the same as enter).
This is generally true, but in this case hitting Return seems to be causing the problem and you have to “click out of” to make it register. I also tested “Tabbing” out and got the same error as when I hit Return.
I think that hit the nail on the head.
David