If you need to use a gem with your extension I really recommend that you make a copy of your own and wrap it into your extension namespace. Otherwise you’re likely to run into clashes with other extensions that would try to use the same gem.
I’ve done this myself before. It’s a bit clunky at times. You’d need a version of each target Ruby versions installed as a standalone installation on your system. For Window you need DevKit installed. Then install the gem to your standalone Ruby installation. Afterwards you can find it in that Ruby’s gem directory where you can copy it into your project and made modifications so that’s it’s wrapped in your own namespace.
For Mac it’d be the same, except as you say RubyInstaller is only for Windows. Mac comes with a compiler out of the box so you might not need anything extra. However, to get the right Ruby versions I recommend you use RVM to manage your Ruby installations.
Depending on what you need, you might find that it’s easier to use a C/C++ library and instead write your own Ruby C Extension wrapping that library.