Did you consider that SketchUp stores in the last matrix element (m[15]) a uniform scale factor (scales in all directions)? To normalize the matrix I believe one has to do:
[ 0, 1, 2].each{ |i| m[i] /= m[15] }
[ 4, 5, 6].each{ |i| m[i] /= m[15] }
[ 8, 9,10].each{ |i| m[i] /= m[15] }
# [12,13,14] store the translation which is unaffected.
m[15] = 1.0
I observed another inconsistency. Does the problem already occur in the Transformation.axes factory method or only in inverse? I recreated the skew transformation with the normal constructor and a matrix array:
skew2 = Geom::Transformation.new([ 1,0,0,0, Math.sqrt(0.5),Math.sqrt(0.5),0,0, 0,0,1,0, 0,0,0,1 ])
which prints just like skew
puts tmatrix_bycolumn(skew2)
1.000, 0.707, 0.000, 0.000
0.000, 0.707, 0.000, 0.000
0.000, 0.000, 1.000, 0.000
0.000, 0.000, 0.000, 1.000
but has a different inverse:
puts tmatrix_bycolumn(skew2.inverse)
1.000, -1.000, 0.000, -0.000
-0.000, 1.414, -0.000, -0.000
0.000, -0.000, 1.000, -0.000
0.000, 0.000, 0.000, 1.000
which leads to the correct identity:
puts tmatrix_bycolumn(skew2 * skew2.inverse)
1.000, 0.000, 0.000, 0.000
0.000, 1.000, 0.000, 0.000
0.000, 0.000, 1.000, 0.000
0.000, 0.000, 0.000, 1.000
So a transformation created from axes must have a certain private attribute since it behaves different from a transformation with identical elements.