Using pipes on windows does not work on 64-bit Sketchup, works on 32-bit SketchUp

No, the goal is to create communication between ruby and an external c application not a C extention.

If you look at the docs for Kernel.spawn() you see that the actual arguments allow an (optional) hash at position 1, setting / changing environment variables. This turns out to be like block “changer” methods that reset back to the prior state after the block. (See Dir::chdir, Dir::mktmpdir for examples.)

Ie, you can set or change environment vars in the ENV hash, and they’ll reset after the spawn call.

So, there happens to be a "RUBYSHELL" environment var that points at the command processor to use for shell commands, ie, %x strings, which use Kernel.`() [the global backtick method.] The backtick method (on the C-side) checks the "RUBYSHELL" environment var.

In the past, with regard to running shell commands via %x or Kernel.`() strings without the shell window popping up, … I played with setting

ENV["RUBYSHELL"]= "WScript.exe"

… but it seems the C-side Ruby code still appends a “/c” parameter to ALL strings passed to command processors. The issue is that the “Windows Scripting Host” (either via “WScript.exe” or “CScript.exe”) chokes on a “/c” parameter, instead of ignoring it. (It bails out and produces an "unknown parameter" error, which may be a messagebox if not run silently.)

Yesterday, I tried again,… with respect to Kernel.spawn(),… but it seemed to not have any effect. Ie, the spawn() method C code just ignores the command processor path set in ENV["RUBYSHELL"].

What I am describing really should be considered Ruby Core Issues (Bugs.)

Yes, I was going to recommend a workaround using the WIN32OLE class.

I checked the Open3 module code. It is making a call to the global spawn method, inherited via Object mixing in the Kernel module. Even though the documentation refers to “Process::spawn”, introspection also reveals the Process module just aliases Kernel::spawn within it’s singleton proxy class block.

Not exactly nitpicking here. Because the standard method of changing the way core things work for your extension, would be a refinement.

This prevents your monkey patch from effecting any other of your extensions, but more importantly, other author’s or company’s extensions.

Problem is we can only refine classes, not modules. And Kernel, Open3 and Process are all modules. So you normally cannot refine them to redefine the spawn method.

But when Ruby loads, it loads the definition of the Kernel module, and then immediately mixes it into class Object. And [ just about ] everything in Ruby is an object and is derived from class Object, inheriting all of the instance methods from class Object and module Kernel by inclusion.

So (as I explained above,) it is notable that Open3 uses the global spawn method, because your code can refine that method, in several ways at a class level.

Refinements (in Ruby v 2.0,) only work within the current file, so you can separate this little part of your extension into it’s own .rb file, and it will not effect anything else.


EXAMPLE 1: Refining class Object

module Jeroen::SomePlugin::RefinedObject
  refine ::Object do

    def spawn(*args)
      # WIN32OLE implementation
    end

  end
end

using Jeroen::SomePlugin::RefinedObject

require "Open3"

module Jeroen::SomePlugin

  test, s = Open3.capture2e('ipconfig /all')

end

EXAMPLE 2: Refining a custom class inside your plugin modules:

Open3 is actually a combination mixin / library module. If you look at the code, you’ll see all it’s methods are defined as instance methods. This is for mixing into other objects via include, prepend and extend. Then it makes itself a library module by creating singleton copies of each of these methods by calling module_function() upon each (just after defining them.) In this way it’s methods can be called from outside like a library, or mixed into any module or class.

require "Open3"

module Jeroen::SomePlugin

  class Listener
    include Open3 # mixin all Open3's methods
  end

  module RefinedSpawn
    refine Listener do

      def spawn(*args)
        # WIN32OLE implementation
      end

    end
  end

end

using Jeroen::SomePlugin::RefinedSpawn

module Jeroen::SomePlugin

  listen = Listener::new
  test, s = listen.capture2e('ipconfig /all')

end

But obviously, (looking at example 2,) since it’s your custom wrapped class, including Open3’s functionality, you don’t even need to mess with a refinement.

EXAMPLE 3: Just use a platform conditional to decide when to override your internal inherited spawn method:

require "Open3"

module Jeroen::SomePlugin

  class Listener
    include Open3 # mixin all Open3's methods
    if Sketchup::platform == :platform_win rescue RUBY_PLATFORM !~ /(darwin)/i
    # ... otherwise just leave it as is, if running on OSX.
      def spawn(*args)
        # WIN32OLE implementation
      end
    end # conditional method redefinition
  end # class Listener

  listen = Listener::new
  test, s = listen.capture2e('ipconfig /all')

end # plugin module

Thanks Dan! This is a great help!
We will surely be using this.

Just missing one more thing with the WIN32OLE solution now:

So calling Open3::capture2e(‘some_command’) results in popen_run calling Kernel.spawn passing in some crucial options; bindings for :in, :out, :err to anonymous pipes known in the local process. In this case in Hash form like you mentioned, like so:
{:in=>#<IO:fd 8>, [:out, :err]=>#<IO:fd 11>}
The Kernel.spawn() documentation you pointed me to also clarifies this.

My last remaining issue: how can I realize an equivalent to this in the WIN32OLE implementation (as mentioned in my post above)?
The example I posted uses the Create method of the Win32_Process class.
No idea how I can get the standard input, output and error on the newly created process connected to the IO:fd objects passed into from popen_run.

cmd = 'dir' #just an example here, might be 'our_app.exe -some_arguments'
require 'win32ole'
objStartup = WIN32OLE.connect("winmgmts:\\\\.\\root\\cimv2:Win32_ProcessStartup")
objConfig = objStartup.SpawnInstance_
objConfig.ShowWindow = 0 #HIDDEN_WINDOW
objProcess = WIN32OLE.connect("winmgmts:root\\cimv2:Win32_Process")
errReturn = objProcess.Create(cmd, nil, objConfig, nil)

Just to give some use-case context: we are using an application/separate process using SketchUp and Layout’s c-api’s and need it to communicate in both ways with our Skalp extension running on a live Sketchup instance. So being able to use Open3 for this on both mac and windows would really help things a lot for us in the future.

You have no idea how much I appreciate all your efforts so far. Thank you.

Jeroen

I am not sure you can (yet.) Everything I have played with, even using

shell = WIN32OLE::new("WScript.Shell")
pid = shell.Exec(cmd)

still shows the console window, if the cmd is a console io application.

But does not if cmd is a GUI application (like “calc.exe”, etc.)


Actually your ultimate goal, is to use Open3 functionality, by “tweaking” spawn.

I think you’ll get it done faster (knowing C,) by copying and “tweaking” the Ruby core C, code for your custom “Listener” class (as in Example 3 above.)

Do you have the Ruby C source for “2.0.0-p576” ?

In “io.c”, line 5605 gives:

#ifdef _WIN32
#define HAVE_SPAWNV 1
#define spawnv(mode, cmd, args) rb_w32_aspawn((mode), (cmd), (args))
#define spawn(mode, cmd) rb_w32_spawn((mode), (cmd), 0)
#endif

The aliased functions are in “win32/win32.c”:

rb_win32_spawn() line 1180

rb_win32_aspawn() line 1357
which calls:
rb_w32_aspawn_flags() line 1287

which use:
child_result() line 1058
CreatChild() line 1077
etc.


All of these functions are preceded by the comment:

/* License: Artistic or GPL */

… which indicates this code is all originally from Perl.

I would have helped had the original Perl programmers commented it. (Maybe they have done so since the fork to Ruby ?)

For the record, here’s my test script, which is Open3.popen_run() modified:

###
    def wsh(cmd, opts = {}) # :nodoc:
      #
      # replace:
      #   pid = spawn(*cmd, opts)
      # with:
      ###
        #
        UI.messagebox("Starting WScript Shell...")
        shell = WIN32OLE::new("WScript.Shell")
        UI.messagebox("#{shell.inspect}\n\nStarting WshScriptExec...")
        #puts shell.inspect
        prog  = shell.Exec(cmd)
        pid   = prog.ProcessID
        UI.messagebox("The exec's PID: #{pid}")
        #
      ###
      wait_thr = Process.detach(pid)
      #child_io.each {|io| io.close }
      #
      parent_io = [ prog.StdIn, prog.StdOut, prog.StdErr ]
      result = [ *parent_io, wait_thr ]
      #
      if block_given?
        begin
          return yield(*result)
        ensure
          parent_io.each{|io| io.Close }
          wait_thr.join
        end
      end
      #
      result
      #
    end ###

###

However the WScript objects have differing method names than the standard Ruby IO class’ method names, so is not a direct drop in replacement.

Notice I had to change io.close to io.Close inside the ensure clauses iterator block.

ref:

I tried this example also, but keep getting Type mismatch errors for the ObjProcess.Create() call.
Ie:

Error: #<WIN32OLERuntimeError: (in OLE method `Create': )
    OLE error code:80041005 in SWbemObjectEx
      Type mismatch 
    HRESULT error code:0x80020009
      Exception occurred.>
<main>:41:in `method_missing'
<main>:41:in `obj'
<main>:in `<main>'
SketchUp:1:in `eval'

EDIT: Ok it was some wrong types passed to Create().

BUT whatever, or however I try there is no Process ID returned. And there should be.
The return is 0 for success. It is not worth it. (Better to use Fiddle.)

###
    def obj(*args)

      #{# Create Pipes: from Open3.popen3()

        if Hash === args.last
          opts = args.pop.dup
        else
          opts = {}
        end

        in_r, in_w = IO.pipe
        opts[:in] = in_r
        in_w.sync = true

        out_r, out_w = IO.pipe
        opts[:out] = out_w

        err_r, err_w = IO.pipe
        opts[:err] = err_w

        child_io  = [in_r, out_w, err_w]
        parent_io = [in_w, out_r, err_r]

      #}#
      
      #{# Handle Environment Var hash as first argument:
        if Hash === args.first
          env = args.shift
        else
          env = {}
        end
        if !env.empty?
          prev = {}
          env.each_pair {|key,val|
            if (val.nil? && !ENV[key].nil? ) || val != ENV[key]
              prev[key]= ENV[key] # save to restore
            end
            ENV[key]= val # temporary settings
          }
        end

      #}#

      #{# Convert args array to a command string
        cmd = args.join(' ')
      #}#

      #{# Replace call to global Kernel.spawn()
      
        ### replacing:
        #
        #   pid = spawn(*cmd, opts)
        #
        ### with:
        #
        require 'win32ole'
        #
        # https://msdn.microsoft.com/en-us/library/aa392758(v=vs.85).aspx
        privylist = "ProfileSingleProcess,SystemProfile,Tcb"
        privilege = "{impersonationLevel=impersonate, (#{privylist})}!"
        #
        puts "\nStarting Process Object ..."
        objStartup = WIN32OLE.connect(
          "winmgmts:#{privilege}\\\\.\\root\\cimv2:Win32_ProcessStartup"
        )
          #"winmgmts:#{privilege}root\\cimv2:Win32_ProcessStartup"
        #)
        puts "objStartup.inspect: #{objStartup.inspect}"
        puts "\nConfiguring Process..."
        # Configuring Process:
        objConfig = objStartup.SpawnInstance_
        #objConfig.CreateFlags = 0x200 # New Process group
        objConfig.ShowWindow = 0 # HIDDEN_WINDOW
        #
        objProcess = WIN32OLE.connect("winmgmts:#{privilege}root\\cimv2:Win32_Process")
        #errReturn  = objProcess.Create(cmd, nil, objConfig, nil) # gives error
        #
        # Create a reference for the returned Process ID number:
        # (Create wants a uint32 on C-side. Assuming VT_UI4 is 4byte Unsigned Int.)
        #@pid = WIN32OLE_VARIANT::new(0,WIN32OLE::VARIANT::VT_UI4)
        #@pid = 0 #WIN32OLE_VARIANT::new(0,WIN32OLE::VARIANT::VT_UINT)
        #
        @buf = [0].pack("V")
        #
        if opts[:chdir]
          process_dir = WIN32OLE_VARIANT::new(opts[:chdir])
        else
          process_dir = nil #WIN32OLE::VARIANT::VT_NULL --> causes unknown error 8
        end
        #
        ole_string = WIN32OLE_VARIANT::new( cmd ) # WIN32OLE::VARIANT::VT_BSTR
        #
        errReturn = objProcess.Create(
          ole_string, 
          process_dir,
          objConfig,
          @buf # returned Process ID number always 0 !! WHY ??
        )
        #
        pid = objProcess.ProcessId() # returning nil !! Why ?
        #
        if errReturn == 0
          puts "The Process.Create call successfully returned : #{errReturn.to_s}"
        else
          puts "The Process.Create call returned error code : #{errReturn.to_s}"
        end
        # Successful completion (0)
        # Access denied (2)
        # Insufficient privilege (3)
        # Unknown failure (8)
        # Path not found (9)
        # Invalid parameter (21)
        # Other (22–4294967295)

        @pid = @buf.unpack("V")[0]
        puts "The Process Object's PID: #{@pid}"
        
        puts "The Process Object's PID Information:"
        puts "  inspect: #{objProcess.ProcessId().inspect}"
        puts "    class: #{objProcess.ProcessId().class}"
        puts "      pid: #{objProcess.ProcessId()}"

      #}#
      
      #{# From here, down to rescue, is the same as Open3.popen_run():
      
        wait_thr = Process.detach(@pid)
        child_io.each {|io| io.close }
        #
        result = [ *parent_io, wait_thr ]
        #
        if block_given?
          begin
            return yield(*result)
          ensure
            parent_io.each{|io| io.close unless io.closed? }
            wait_thr.join
          end
        end
        #
        return result

      #}#

      #{#
    rescue => e
      #
      puts e.inspect
      raise
      #
    else
      #
      #
      #
    ensure # that the ENV hash is restored:
      #
      if !env.empty? && !prev.empty?
        prev.each_pair {|key,val|
          ENV[key]= val # restore variables
        }
      end
      #}#
    end ###

###

Came across a gem called Open4 that explicitly keeps a PID reference to the created child process:

It is available at Rubygems

Gem::Install("open4")

All very interesting stuff Dan.

I was out with my family these last few days. I’ll be looking into your feedback now.

"Actually your ultimate goal, is to use Open3 functionality, by “tweaking” spawn."
Indeed!

Last week I was fiddling with Fiddle…
And I was looking into ‘childprocess’ https://github.com/jarib/childprocess.git
But that one uses ‘ffi’ (foreign function interface) https://github.com/ffi/ffi.git
I thought maybe I could adapt that to use Fiddle instead.
And then there is one last thing:
a posix compatible spawn implementation.
https://github.com/rtomayko/posix-spawn.git
Maybe I could just have a look at part of its implementation on windows…

Not sure about the open4 gem, I’ll have a look though.

Jeroen

Fiddle is a Ruby wrapper around “libffi-6.dll”. (Fiddle is a replacement for the Ruby FFI gem, which is no longer maintained.)

So it should be relatively easy to translate to using Fiddle.

After trying almost everything we discovered the popen3 function on ruby on windows works perfect, but when we use the popen3 function on SketchUp the stdout and stderr are lost. Probably this have something to do with how the SketchUp ruby console works on windows. Can anybody help us how we can fix this?

The following example works perfect on standard ruby, but gives no result we you run this on SketchUp:

require 'open3'

command1 = 'dir /B'
command2 = 'sort /R'

reader,writer = IO.pipe
Open3.popen3(command1) {|stdin, stdout, stderr, wait_thr|
  writer.write stdout.read
}
writer.close

stdout, stderr, status = Open3.capture3(command2, :stdin_data => reader.read)
reader.close

puts "status: #{status}"   #pid and exit code
puts "stderr: #{stderr}"   #use this to debug command2 errors
puts "stdout: #{stdout}"


A small other test showing that stdout have an other behavior on SketchUp for window:

SketchUp 2016 mac version:
%{ifconfig}

SketchUp 2016 windows version:
%{ipconfig}

Ruby for windows:
%{ipconfig}

The ipconfig is just a test, we don’t need this result. We need the stdout send from an external application.

However %x strings work fine for me, Win7 64-bit, SketchUp 2016 Pro 64-bit.

The Kernel backquote method was bugged early in SketchUp with Ruby 2.0, I think either v2014 or v2015 initial release. But I thought it was fixed in a later maintenance release. (%x strings use the Kernel backquote method.)

See my post: [code] Backquote method patch as Mixin module

Works fine for me (with the Ruby Console open,) on the latest versions of both SU2015 and SU2016 (both 64-bit.)

The exit code for staus is 0, and the stdout var contains the complete listing of my ENV["HOME"] folder.

However the listing is not sorted. EDIT: I take this back. It IS sorted in REVERSE order.

And I see the two command shell windows popping up when the script runs.

Dan,

The output itself is not important we are just looking to get output back to ruby. I tested this code on 3 different computer and none gives any feedback.

computer 1 (VMWare)
SketchUp Pro 2014 32bit
SketchUp Pro 2015 64bit
SketchUp Pro 2016 32bit

computer 2 Windows 10
SketchUp Pro 2014 32bit
SketchUp Pro 2015 64bit
SketchUp Make 2016 64bit

computer 3 Windows 7
SketchUp Pro 2016 32 bit (clean install)

What is different on your system that this seams to work on your system and not at any of my systems?

Besides that I’m running 64-bit on Win7 for v14,15 and 16, …

I cannot know unless you run ThomThom’s diagnostics tool:

Dan,

This are two diagnostic files from two SU versions where we don’t get any data back on the stdout.
Thanks for your help!

SketchUp-Diagnostic win10 SU2016Make64bit.txt (19.8 KB)
SketchUp-Diagnostic win7(vmware)SU2016Pro32bit.txt (23.3 KB)

I see something weird with Ruby installs…

Your Win10 64-bit machine has a path to C:\Ruby200\bin which might normally be a 32-bit Ruby install. That is OK on a 64-bit machine running either 32-bit or 64-bit Windows. (I actually have both installed as well as both the 32-bit and 64-bit DevKits, but only one is “active” at a time.)

However, on your Win7 32-bit machine, it has a path to C:\Ruby200-x64\bin which normally would be a 64-bit Ruby install.

I also notice that your user name has a space in it. Any Unicode characters ?

You start SketchUp from the shortcut correct ? And it is set to “Run as administrator” ?
IE, My report has:
__COMPAT_LAYER : RunAsAdmin
just below the output for the %windir% environment variable. Neither of your reports do.

Another issue. Your Windows 10 machine is not running the latest v16.
You should update it to 16M1.

I am not running this tracing:

windows_tracing_flags         : 3
windows_tracing_logfile       : C:\BVTBin\Tests\installpackage\csilogfile.log

I am not running these extensions:

Ruby Code Editor               (3.2) LOADED
Skalp                          (1.5.0031_beta) LOADED

I am running these extension in common with you:

Ruby Console+                  (2.1.7) 
Advanced Camera Tools          (1.3.0) LOADED
SketchUp Diagnostics Tools     (1.2.0) LOADED
Dynamic Components             (1.4.0) LOADED
Sandbox Tools                  (2.3.0) LOADED
Trimble Connect                (1.1.1) LOADED
Photo Textures                 (1.1.2) LOADED

I am not loading these Standard Ruby Library files:

e2mmap.rb
open-uri.rb
Date.rb
Matrix.rb
SecureRandom.rb
rubygems/path_support.rb
win32/registry.rb