SQLite3 gem in SU2019 and 2020 (Ruby 2.5.5)

I have been successfully using compiled gems with SU 2015 to 2018 on macos. A couple of examples are SQLite3 (1.3.13) and ruby_clipper 6.2.1.5.2.

This is my workflow on macos (A different workflow works on Win)

  • I use rbenv to manage local Ruby installations
  • SU 2015 - 2.0.0-p247
  • SU 2017 & 2018 - 2.2.4
  • SU 2019 & 2020 - 2.5.5p157

Most of the tools I need are installed via brew.

In order to use compiled gems I first install them on my local ruby installation and then require the gem_extension.bundle file in my extension. It is a bit more involved than that, but you get the point.

Everything works great for SU2015/16/17/18. Unfortunately this is not working anymore for SU2019 or SU2020.

When I require the gem_extension.bundle file I receive the error

Error: #<LoadError: incompatible library version - /Users/xxxxxxx/dev/MCTools/src/mct-tools/ext/gems/ruby/2.5.0/extensions/x86_64-darwin-19/2.5.0/sqlite3-1.4.2/sqlite3/sqlite3_native.bundle>
/Applications/SketchUp 2020/SketchUp.app/Contents/Frameworks/Ruby.framework/Versions/2.5.5/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require'
/Applications/SketchUp 2020/SketchUp.app/Contents/Frameworks/Ruby.framework/Versions/2.5.5/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require'
<main>:in `<main>'
SketchUp:1:in `eval'

Every gem gives exactly the same error. If I require them in pry for example they are loaded without any issue and they work as expected.

I know that Gems are not officially supported, but I was hoping anyway that someone more knowledgeable than me could help.

I don’t know if this helps, but if you run otool -L on the two bundles these are the 2 different outputs

sqlite 1.3.13 - Working bundles for SU < 2019

/usr/lib/libsqlite3.dylib (compatibility version 9.0.0, current version 308.4.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1281.100.1)
/usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0)

sqlite 1.3.13 - Not working for SU >= 2019

/Users/xxxx/.rbenv/versions/2.5.5/lib/libruby.2.5.dylib (compatibility version 2.5.0, current version 2.5.5)
/usr/lib/libsqlite3.dylib (compatibility version 9.0.0, current version 308.5.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1281.100.1)
/usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0)

The first thing that you notice is that the dependence on libruby.2.5.dylib is not present in the previous versions. From some posts that I read on this forum I think I understood that the SU Ruby Framework has changed the files that it includes and the way they are named. For example here and here. Honestly this is a bit over my head.

I have also tried with the latest sqlite3 1.4.2 and with various compilation options (using sqlite from brew for example), but with no success.

Any idea? Has someone managed to run sqlite3 successfully on macos SU2019/20?

Thanks for looking.

@Rojj

I’m a Windows guy, so…

You mentioned rvm. Too confirm, that has no effect on your SU installs, it only affects your stand-alone Rubies?

You’re compiling with a stand-alone, then using it in SU. Normally, SU would resolve gems in Gem.Path, are you placing them there?

For that matter, what is Gem.path on macOS SU? Note that it may be different in SU Rubies, vs stand-alone Rubies, vs rvm Rubies.

Finally, most extension gems require with an *.rb file, and that file handles loading additional files, some of which may be compiled.

Note that SU’s Ruby sets ENV['GEM_HOME'] & ENV['GEM_PATH'], at least on Windows.

I use rbenv that is similar to rvm. It does not affect the SU installation.

I use bundler to deploy the exact gems versions within my plugin folder. This includes all the .rb files required. Then in my loader I do:

gem_libs = File.expand_path("ext/gems/ruby/2.2.0/gems/**/lib", __dir__)
$LOAD_PATH.unshift *Dir.glob(gem_libs)

This allows my to call the gem (for example sqlite3) with

require "sqlite3"

and avoids changing the user’s SU gems. Plus I can control the gems with my installer rather than relying on SU.

It works fine with

numo-narray-0.9.1.6
numo-linalg-0.1.5
ruby-prof-0.18.0
ruby_clipper-6.2.1.5.2
sqlite3-1.3.13

on SU 2015 to 2018.

BTW I use the same approach for the Windows version, but Windows requires some additional libraries copied to

"%ProgramFiles%\SketchUp\SketchUp 2017\Tools\RubyStdLib\platform_specific\"

and this also is handled by the installer.

Sorry for mixing up rbenv & rvm. Thanks for the better explanation.

Rather than:

gem_libs = File.expand_path("ext/gems/ruby/2.2.0/gems/**/lib", __dir__)

you may need to use RbConfig::CONFIG['ruby_version'] in place of 2.2.0:

gem_libs = File.expand_path("ext/gems/ruby/#{RbConfig::CONFIG['ruby_version']}/gems/**/lib", __dir__)

That will duplicate the *.rb files for every gem for every version of SU Ruby you’re targeting. Otherwise, you’d need to make all the gems ‘fat-binary’ style, as sqlite3 does with:

begin
  RUBY_VERSION =~ /(\d+\.\d+)/
  require "sqlite3/#{$1}/sqlite3_native"
rescue LoadError
  require 'sqlite3/sqlite3_native'
end

but Windows requires some additional libraries copied to

Any additional libraries required by the extension gems that aren’t in ENV['Path'] would need that.

Now that I understand what/how, am I correct that the issue of sqlite3 compiled with stand-alone 2.5.5 doesn’t run in SU 2019/20 on macOS?

Also, re fat-binary, I just noticed in num-narray, commit Drop fat gem support, so you’d have a bit of hacking to assemble them in your plugin…

That’s correct. The error happens when i require sqlite3_native.bundle

Error: #<LoadError: incompatible library version - /Users/xxxx/dev/MCTools/src/mct-tools/ext/gems/ruby/2.5.0/gems/sqlite3-1.4.2/lib/sqlite3/sqlite3_native.bundle>
/Applications/SketchUp 2020/SketchUp.app/Contents/Frameworks/Ruby.framework/Versions/2.5.5/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require'
/Applications/SketchUp 2020/SketchUp.app/Contents/Frameworks/Ruby.framework/Versions/2.5.5/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require'
/Users/xxxx/dev/MCTools/src/mct-tools/ext/gems/ruby/2.5.0/gems/sqlite3-1.4.2/lib/sqlite3.rb:6:in `rescue in <top (required)>'
/Users/xxxx/dev/MCTools/src/mct-tools/ext/gems/ruby/2.5.0/gems/sqlite3-1.4.2/lib/sqlite3.rb:2:in `<top (required)>'
/Applications/SketchUp 2020/SketchUp.app/Contents/Frameworks/Ruby.framework/Versions/2.5.5/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require'
/Applications/SketchUp 2020/SketchUp.app/Contents/Frameworks/Ruby.framework/Versions/2.5.5/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require'
/Users/xxxx/dev/MCTools/src/mct-tools/loader.rb:65:in `<top (required)>'
.....

Notice LoadError: incompatible library version. This happens for every other compiled gem in SU 2019/20 only.

I can live without numo-narray. The two key gems that I need are sqlite3 and clipper.

Just to confirm, sqlite3_native.bundle was compiled with stand-alone Ruby 2.5.5, and runs correctly using it? Kind of makes no sense.

Re numo-narray and other gems, you don’t need to drop them, it’s just more code in your plugin.

Re the ‘fat-binary’ issue, if you want be compatible with SU back to 2015, you’ll need three sets of gems, located in:

/Users/xxxx/dev/MCTools/src/mct-tools/ext/gems/ruby/2.0.0/gems/
/Users/xxxx/dev/MCTools/src/mct-tools/ext/gems/ruby/2.2.0/gems/
/Users/xxxx/dev/MCTools/src/mct-tools/ext/gems/ruby/2.5.0/gems/

That places the ‘version switching’ outside of the gems.

If you want to make all the gems ‘fat-binary’, you would just need one folder, as the ‘version switching’ is done within the gem. All it takes is placing the *.bundle files within the correct folder, and adding the switching code to the require for it. SQLite3 already has the code.

Indeed, it does not make sense … at least to me :slight_smile:

Yes, sqlite3_native.bundle was compiled with the stand-alone Ruby 2.5.5-p157 installed with rbenv. Using pry I can simply run require "sqlite3" and it works. Just for testing I have also tried to require the bundle file directly and it gives no error. Of course it cannot be used as all the other .rb support files are missing.

When I move the gem in my gem folder /Users/xxxx/dev/MCTools/src/mct-tools/ext/gems/ruby/2.5.0/gems/ all the pure Ruby gems are loaded normally, but the compiled ones give the LoadError: incompatible library version error.

Honesty, it does not seem a problem specific to sqlite3.

Interesting. One thing you might try is opening sqlite3_native.bundle in a text editor and look for libruby.2.5.dylib or similar, and see if the path is hard coded. In Windows Ruby, almost all gem extension *.so files refer to x64-msvcrt-ruby2*0.dll, with no path info. It’s found by normal dll resolution.

Maybe macOS and/or rbenv uses something different?

Isn’t that what otool -L does? See my original message for the output. It looks hardcoded.

Another interesting thing is that the bundle for 2.2.4 does not have any reference to libruby.

Remember, I’m a Windows guy (hope to be Ubuntu soon). I don’t know if otool does the resolution or not. Can you run it with backticks in the SU Ruby console? Windows has issues with that, not sure about macOS…

Yes, you can

> `otool -L /Users/xxxx/dev/MCTools/src/mct-tools/ext/gems/ruby/2.5.0/extensions/x86_64-darwin-19/2.5.0/sqlite3-1.4.2/sqlite3/sqlite3_native.bundle`
/Users/xxxx/dev/MCTools/src/mct-tools/ext/gems/ruby/2.5.0/extensions/x86_64-darwin-19/2.5.0/sqlite3-1.4.2/sqlite3/sqlite3_native.bundle:
	/Users/xxxx/.rbenv/versions/2.5.5/lib/libruby.2.5.dylib (compatibility version 2.5.0, current version 2.5.5)
	/usr/local/opt/sqlite/lib/libsqlite3.0.dylib (compatibility version 9.0.0, current version 9.6.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1281.100.1)
	/usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0)

I think otool is the equivalent of objump. For example in Windows I use

objdump.exe -p C:/path/to/clipper.so | Select-String -Pattern 'DLL Name:'

I don’t know. Some thing macOS specific…

JFYI,

The path below is the path/file used to compile::

/Users/xxxx/dev/MCTools/src/mct-tools/ext/gems/ruby/2.5.0/extensions/x86_64-darwin-19/2.5.0/sqlite3-1.4.2/sqlite3/sqlite3_native.bundle

Below is the file the gem actually loads:

/Users/xxxx/dev/MCTools/src/mct-tools/ext/gems/ruby/2.5.0/gems/sqlite3-1.4.2/lib/sqlite3/sqlite3_native.bundle`

In your extension, I believe you could delete the extensions folder`…

@Rojj

So the following appeared when running otool from within the SU Ruby console:

/Users/xxxx/.rbenv/versions/2.5.5/lib/libruby.2.5.dylib (compatibility version 2.5.0, current version 2.5.5)

Shouldn’t that be pointing to a libruby.2.5.dylib file (or symlinked to libruby.2.5.5.dylib) that’s within the SketchUp folder? Again, don’t know about macOS…

@Rojj

I screwed around with this, and the sqlite3_native.bundle has the following embedded in it:

/usr/local/opt/sqlite/lib/libsqlite3.0.dylib

and

/Users/runner/.rubies/ruby-2.6.6/lib/libruby.2.6.dylib

So it would seem that the path is hard coded.

I also tried setting ENV['CONFIGURE_ARGS'] = '--disable-rpath' before the script ran and it made no difference.

There must be a way, as this means that compiled gems aren’t portable, as they are on Windows (with a few assumptions).

The same type of path existed in the ruby exe file for its link to libruby.2.x.dylib

JFYI, I’m MSP-Greg on GitHub, I did all this on GitHub Actions macOS runners…

I found this one. It is using the brew sqlite. That’s an easy fix. I can recompile with the system sqlite in /usr/lib/libsqlite3.0.dylib.

I cannot find this one though. It says .rubies. Is this RVM?

Sorry, but I don’t understand what you mean with this…

@Rojj

Sorry. I guess I wasn’t clear.

I was investigating portability of extension gems on macOS vs Windows.

GitHub Actions is a cloud based testing service that has Ubuntu, macOS, and Windows systems available. Since I don’t run macOS locally, I use it for whatever when needed.

You mentioned using brew to compile SQLite3. If you uninstall it, does the gem still function? On Windows SQLite3 compiles with a dependency on libsqlite3-0.dll, so I assume you include that in your plugin and load it with fiddle?

Extension gems and SU are a messy subject, with version collisions being the main issue…

@MSP_Greg

Honestly I would give up if it wasn’t for a customer that needs to upgrade to SU 2020!! :expressionless:

Another test with ruby_clipper. This library does not have any dependency other than libruby and system libraries.

otool -L /Users/xxxx/dev/MCTools/src/mct-tools/ext/gems/ruby/2.5.0/gems/ruby_clipper-6.2.1.5.2/ext/clipper/clipper.bundle
/Users/xxxx/dev/MCTools/src/mct-tools/ext/gems/ruby/2.5.0/gems/ruby_clipper-6.2.1.5.2/ext/clipper/clipper.bundle:
        /Users/xxxx/.rbenv/versions/2.5.5/lib/libruby.2.5.dylib (compatibility version 2.5.0, current version 2.5.5)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1281.100.1)
        /usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0)
        /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 902.1.0) 

Then I do

install_name_tool -change /Users/xxxx/.rbenv/versions/2.5.5/lib/libruby.2.5.dylib "/Applications/SketchUp 2020/SketchUp.app/Contents/Frameworks/Ruby.framework/Versions/2.5.5/libruby.2.5.dylib" /Users/xxxx/dev/MCTools/src/mct-tools/ext/gems/ruby/2.5.0/gems/ruby_clipper-6.2.1.5.2/ext/clipper/clipper.bundle

Now otool -L gives

otool -L /Users/xxxx/dev/MCTools/src/mct-tools/ext/gems/ruby/2.5.0/gems/ruby_clipper-6.2.1.5.2/ext/clipper/clipper.bundle/Users/ruggiero/dev/MCTools/src/mct-tools/ext/gems/ruby/2.5.0/gems/ruby_clipper-6.2.1.5.2/ext/clipper/clipper.bundle:
        /Applications/SketchUp 2020/SketchUp.app/Contents/Frameworks/Ruby.framework/Versions/2.5.5/libruby.2.5.dylib (compatibility version 2.5.0, current version 2.5.5)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1281.100.1)
        /usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0)
        /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 902.1.0)

Unfortunately

require '/Users/xxxx/dev/MCTools/src/mct-tools/ext/gems/ruby/2.5.0/gems/ruby_clipper-6.2.1.5.2/ext/clipper/clipper.bundle'

still returns

Error: #<LoadError: incompatible library version - /Users/xxxx/dev/MCTools/src/mct-tools/ext/gems/ruby/2.5.0/gems/ruby_clipper-6.2.1.5.2/ext/clipper/clipper.bundle>
/Applications/SketchUp 2020/SketchUp.app/Contents/Frameworks/Ruby.framework/Versions/2.5.5/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require'
/Applications/SketchUp 2020/SketchUp.app/Contents/Frameworks/Ruby.framework/Versions/2.5.5/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require'
<main>:in `<main>'
SketchUp:1:in `eval'

Digging a bit deeper…

/Users/xxxx/.rbenv/versions/2.5.5/lib/libruby.2.5.dylib is a link to /Users/xxxx/.rbenv/versions/2.5.5/lib/libruby.2.5.5.dylib

Also /Users/xxxx/.rbenv/versions/2.5.5/lib/libruby.2.5.5.dylib is 3.4MB, while /Applications/SketchUp 2020/SketchUp.app/Contents/Frameworks/Ruby.framework/Versions/2.5.5/libruby.2.5.dylib is 8.7MB. So they are not the same thing.

(BTW I have also tested install_name with 2.5.5 in SU Framework with the same results)

Also otool -L gives different results

otool -L /Users/xxxx/.rbenv/versions/2.5.5/lib/libruby.2.5.dylib 
/Users/xxxx/.rbenv/versions/2.5.5/lib/libruby.2.5.dylib:
        /Users/xxxx/.rbenv/versions/2.5.5/lib/libruby.2.5.dylib (compatibility version 2.5.0, current version 2.5.5)
        /System/Library/Frameworks/Foundation.framework/Versions/C/Foundation (compatibility version 300.0.0, current version 1677.104.0)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1281.100.1)
        /usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0)
        /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1677.104.0)

and

otool -L "/Applications/SketchUp 2020/SketchUp.app/Contents/Frameworks/Ruby.framework/Versions/2.5.5/libruby.2.5.dylib"
/Applications/SketchUp 2020/SketchUp.app/Contents/Frameworks/Ruby.framework/Versions/2.5.5/libruby.2.5.dylib:
        @executable_path/../Frameworks/Ruby.framework/Versions/2.5.5/Ruby (compatibility version 2.5.0, current version 2.5.5)
        /System/Library/Frameworks/Foundation.framework/Versions/C/Foundation (compatibility version 300.0.0, current version 1153.20.0)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1213.0.0)
        /usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0)
        /usr/lib/libutil.dylib (compatibility version 1.0.0, current version 1.0.0)
        @rpath/libssl.1.1.dylib (compatibility version 1.1.0, current version 1.1.0)
        @rpath/libcrypto.1.1.dylib (compatibility version 1.1.0, current version 1.1.0)
        /usr/lib/libffi.dylib (compatibility version 1.0.0, current version 1.0.0)
        /usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.5)
        /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1153.18.0)

Sorry for the delay. I actually just saw a GitHub issue, and the main problem was that non Windows Ruby builds aren’t portable due to the hard coded paths.

I suppose if one dug far enough, one could add all the symlinks needed, but that may involve access issues, etc.

Until I get running on either macOS and/or Ubuntu, I can’t work much more with it. But thanks to you, I know of the problem…