How to run testsup2 from command line on mac

anuragjoshi · September 15, 2021, 8:38am

Hi there,

I have written some tests for an extension I am working on. I am able to run the tests using the testup2 UI. I want to run the tests from a command line. I have a tasks.json file like following

  {
		// See https://go.microsoft.com/fwlink/?LinkId=733558
  // for the documentation about the tasks.json format
  "version": "2.0.0",
  "tasks": [
    {
      "label": "Debug SketchUp 2021",
      "type": "shell",
      "command": "open -a '/Applications/SketchUp 2021/SketchUp.app' --args -rdebug 'ide port=7000'"
    },
    {
      "label": "Run tests",
      "type": "shell",
      "command":     "open -a '/Applications/SketchUp 2021/SketchUp.app' --args 'TestUp:CI:Path: /Users/anuragjoshi/Bonito/Bonito_Sketchup_Tools/Product Smoke Tests'"
    }

  ]
}

But when I run my command the tests won’t run(See the command in Run Tests in above tasks.
Q: How to run invoke the tests to run from the command line on a mac?(I am asking mac because I haven’t tested this on a windows machine).

tt_su · September 15, 2021, 9:01am

Have a look at the example arguments from the documentation again:

You are using --args, but to pass arguments to SketchUp’s Ruby interpreter you need to use -RubyStartupArg

-RubyStartupArg "TestUp:CI:Config: \Full\Path\To\Config.yml"

anuragjoshi · September 15, 2021, 9:03am

using -RubyStartupArg gives the following error

open: unrecognized option `--RubyStartupArg'`

I have tried both --RubyStartupArg and -RubyStartupArg

tt_su · September 15, 2021, 9:11am

Hm… I’ll have to look closer at that. (But all my macs are busy updating atm, so I cannot do that right now.)

tt_su · September 15, 2021, 9:21am

Wait, that says that open isn’t recognising --RubyStartupArg. Meaning SketchUp isn’t getting the arguments.

This is probably an issue with how your command is set up in tasks.json.
I see you have the whole command in "command"'s value, but tasks.json have an "args" field to fill in the arguments passed to the command. Tasks in Visual Studio Code

Maybe something like this? (I’ve not been able to test this.)

      "program": "/Applications/SketchUp 2021/SketchUp.app/Contents/MacOS/SketchUp",
      "args": [
        "-RubyStartupArg",
        "TestUp:CI:Path: /Users/anuragjoshi/Bonito/Bonito_Sketchup_Tools/Product Smoke Tests"
      ],

tt_su · September 15, 2021, 9:24am

Oh, I see now, --args is a parameter for open, and that is used to pass any trailing arguments to the app open is set to lunch.

This will probably also work:

open -a '/Applications/SketchUp 2021/SketchUp.app' --args -RubyStartupArg 'TestUp:CI:Path: /Users/anuragjoshi/Bonito/Bonito_Sketchup_Tools/Product Smoke Tests'

open -a '/Applications/SketchUp 2021/SketchUp.app' says to use open to launch SketchUp 2021, and --args -RubyStartupArg 'TestUp:CI:Path: /Users/anuragjoshi/Bonito/Bonito_Sketchup_Tools/Product Smoke Tests' says to pass all arguments after --args to the application being opened.

anuragjoshi · September 22, 2021, 8:25am

Just out of curiosity has anyone written GitHub action for testup. Just like this one

DanRathbun · September 22, 2021, 1:50pm

“This one” is for running rubocop on pure ruby or rails code.

TestUp 2 must run within the SketchUp Ruby subprocess if the Ruby code uses the SketchUp APIs.

So, how could GitHub run an instance of SketchUp on it’s servers ? And how would it know which SketchUp version to test under ? (It cannot as running on servers is prohibited by the EULA.)

anuragjoshi · September 22, 2021, 2:29pm

Yeah you right. I was thinking of having a docker image with Shectchtp installed and take from there. Anyway right now it’s not a high priority so I haven’t explored any other options as well like Jenvins

tt_su · September 23, 2021, 11:52am

Yea, I really wish something like that was easily available. Unfortunately, we cannot use any of the standard agents CI services offer because the build machine needs an activated SketchUp version.

Personally I have plans to try to create my own self-hosted runner for GitHub Actions. I got a tiny x64 Windows 10 machine, poor graphic card, but enough to run SketchUp. And it doesn’t need to be fast, just need to run the tests.

DanRathbun · September 23, 2021, 12:28pm

Yeah, I was going to say perhaps Trimble could host tests on a server that runs the desired SU test version in a VM.

anuragjoshi · September 25, 2021, 10:16am

can you elaborate on this?. I can’t set up a physical server(With SketchUp installed and licensed) and run tests there(I mean I will have to use tools like Jenkins(TBH I am not sure if this is possible)).

anuragjoshi · September 25, 2021, 10:30am

above command when directly run from CLI doesn’t throw any errors but it doesn’t run the tests.

I also modified this file. Just open and write to a file. This is also not being trigger after the above command. This means that the testup can not activated to parse and run tests

However if I open a file manually and bring up the testup UI, my modified code gets run and the file I am expecting gets created.

DanRathbun · September 25, 2021, 11:25am

Read the license. Your license does not apply to others.
Ie, other people cannot use SketchUp under your license.

Now if it is a private server and only you have access to it, this might be a different scenario.

tt_su · September 28, 2021, 10:39am

The command won’t open the UI to display the test results. It’s supposed to print the results in JSON format to STDOUT. But there appear to be a bug on macOS where this isn’t happening.

Can you check TestUp’s log folder? /Users/thomas/Library/Application Support/TestUp/Logs/

While the STDOUT output is missing I see new logs being created when I invoke the command.

tt_su · September 28, 2021, 11:36am

I had a closer look. I’m still not sure why STDOUT doesn’t work on macOS. But even on Windows I experienced that other extensions could interfere if they used STDOUT (which a couple of extensions I had installed did.)

A more reliable approach is probably to configure an output filepath. I got a proof of concept working:
https://github.com/thomthom/testup-2/tree/dev/ci-options#running-with-custom-config

If you have TestUp setup to load from source you can checkout the ci-options branch. Then there is a new option Output to the config file alternative (TestUp:CI:Config). The “simpler” TestUp:CI:Path alternative would need further work to allow more options.

tt_su · September 28, 2021, 1:45pm

This appear to be related to using open -a to launch SketchUp.

For instance, this will not yield any output to the file when using STDOUT.puts "Hello World":

open -a /Applications/SketchUp\ 2021/SketchUp.app > sustdout.txt

But this will:

/Applications/SketchUp\ 2021/SketchUp.app/Contents/MacOS/sketchup > sustdout.txt

anuragjoshi · October 11, 2021, 8:17am

I am still not able to get the tests run, I am using the config file as you suggested.

This is my config file

# Config.yml
# Required:
Path: /Users/anuragjoshi/Bonito/Bonito_Sketchup_Tools/Product Smoke Tests

# Optional: (By default, don't include a fixed seed!)
Seed: 123 # The seed number for the random order of execution of the tests

# Optional:
# By default the results will be output to STDOUT. This can be redirected to a
# file.
Output: /Users/anuragjoshi/Bonito/Bonito_Sketchup_Tools/Product Smoke Tests/output.json

# Optional:
# Set to true to prevent SketchUp from closing. Useful for debugging purposes.
# Note that if `Output` is used the results won't be written until SketchUp is
# closed.
KeepOpen: false

# Optional:
# List the set of sets you want to run.
#   Run all tests in test case: TC_TestCaseName#
#   Run specific test: TC_TestCaseName#test_testname
Tests:
  - TC_all_products_are_created_with_default_configuration

and this is the command I am running

"command": "'/Applications/SketchUp 2021/SketchUp.app/Contents/MacOS/sketchup' -RubyStartupArg 'TestUp:CI:Config: /Users/anuragjoshi/Bonito/Bonito_Sketchup_Tools/Product Smoke Tests/config.yml'"

The TC_all_products_are_created_with_default_configuration has only one test which create a file a writes to it:

def test_at_least_tests_are_running()
    File.open("/Users/anuragjoshi/Bonito/Bonito_Sketchup_Tools/Product Smoke Tests/out.txt", "w") do |f|     
      f.write("Tests are running. Yahooooooooo")   
    end
  end

And when I run the command this is output I get in the console:

2021-10-11 13:36:09.570 sketchup[27921:13609527] Initializing Bugsplat
objc[27921]: Class SkpTimerCaller is implemented in both /Applications/SketchUp 2021/SketchUp.app/Contents/MacOS/sketchup (0x102eec2b8) and /Applications/SketchUp 2021/SketchUp.app/Contents/Frameworks/LayOutAPI.framework/Versions/A/Libraries/libSketchUpViewerAPI.dylib (0x118daedf8). One of the two will be used. Which one is undefined.
objc[27921]: Class SKTextPath is implemented in both /Applications/SketchUp 2021/SketchUp.app/Contents/MacOS/sketchup (0x102eec330) and /Applications/SketchUp 2021/SketchUp.app/Contents/Frameworks/LayOutAPI.framework/Versions/A/Libraries/libSketchUpViewerAPI.dylib (0x118daee70). One of the two will be used. Which one is undefined.
objc[27921]: Class SKTextPath is implemented in both /Applications/SketchUp 2021/SketchUp.app/Contents/MacOS/sketchup (0x102eec330) and /Applications/SketchUp 2021/SketchUp.app/Contents/Frameworks/SketchUpAPI.framework/Versions/A/SketchUpAPI (0x118099380). One of the two will be used. Which one is undefined.
2021-10-11 13:36:10.554 sketchup[27921:13609527] INFO: Start CrashManager processing
2021-10-11 13:36:37.934 sketchup[27921:13609527] [HockeySDK] -[BITCrashManager invokeDelayedProcessing]/974 [HockeySDK] WARNING: Another exception handler was added. If this invokes any kind exit() after processing the exception, which causes any subsequent error handler not to be invoked, these crashes will NOT be reported to HockeyApp!
[1011/133640.977812:INFO:CONSOLE(7)] "Error accessing file:///Applications/SketchUp%202021/SketchUp.app/Contents/Resources/welcomescreen/../en.lproj//welcomescreen/en.json--[object ProgressEvent]", source: file:///Applications/SketchUp%202021/SketchUp.app/Contents/Resources/welcomescreen/regularlaunchpage.bundle.js (7)
[1011/133640.978742:INFO:CONSOLE(7)] "Unable to find resource file for language en", source: file:///Applications/SketchUp%202021/SketchUp.app/Contents/Resources/welcomescreen/regularlaunchpage.bundle.js (7)
[1011/133640.978972:INFO:CONSOLE(7)] "In location file:///Applications/SketchUp%202021/SketchUp.app/Contents/Resources/welcomescreen/../en.lproj//welcomescreen/en.json", source: file:///Applications/SketchUp%202021/SketchUp.app/Contents/Resources/welcomescreen/regularlaunchpage.bundle.js (7)
[1011/133641.226827:INFO:CONSOLE(27)] "Uncaught (in promise) NavigationDuplicated: Navigating to current location ("/home") is not allowed", source: file:///Applications/SketchUp%202021/SketchUp.app/Contents/Resources/welcomescreen/regularlaunchpage.bundle.js (27)

[1011/134435.316597:ERROR:shared_image_manager.cc(143)] SharedImageManager::ProduceGLTexture: Trying to produce a representation from a non-existent mailbox. 08:19:D0:F5:11:BC:96:E6:14:1A:6C:F7:A9:DB:66:DF
[1011/134435.329562:ERROR:gles2_cmd_decoder.cc(19021)] [.DisplayCompositor]GL ERROR :GL_INVALID_OPERATION : DoCreateAndTexStorage2DSharedImageINTERNAL: invalid mailbox name
[1011/134435.329877:ERROR:gles2_cmd_decoder.cc(19042)] [.DisplayCompositor]GL ERROR :GL_INVALID_OPERATION : DoBeginSharedImageAccessCHROMIUM: bound texture is not a shared image
[1011/134435.329977:ERROR:gles2_cmd_decoder.cc(13907)] [.DisplayCompositor]GL ERROR :GL_INVALID_VALUE : glScheduleCALayerCHROMIUM: unsupported texture format
[1011/134435.330025:ERROR:shared_image_manager.cc(143)] SharedImageManager::ProduceGLTexture: Trying to produce a representation from a non-existent mailbox. CA:E2:29:03:B4:AC:7A:86:86:DE:2C:30:BB:88:01:6B
[1011/134435.330080:ERROR:gles2_cmd_decoder.cc(19021)] [.DisplayCompositor]GL ERROR :GL_INVALID_OPERATION : DoCreateAndTexStorage2DSharedImageINTERNAL: invalid mailbox name
[1011/134435.330120:ERROR:gles2_cmd_decoder.cc(19042)] [.DisplayCompositor]GL ERROR :GL_INVALID_OPERATION : DoBeginSharedImageAccessCHROMIUM: bound texture is not a shared image
[1011/134435.342086:ERROR:gles2_cmd_decoder.cc(13907)] [.DisplayCompositor]GL ERROR :GL_INVALID_VALUE : glScheduleCALayerCHROMIUM: unsupported texture format
[1011/134435.515626:ERROR:gles2_cmd_decoder.cc(19071)] [.DisplayCompositor]GL ERROR :GL_INVALID_OPERATION : DoEndSharedImageAccessCHROMIUM: bound texture is not a shared image
[1011/134435.515947:ERROR:gles2_cmd_decoder.cc(19071)] [.DisplayCompositor]GL ERROR :GL_INVALID_OPERATION : DoEndSharedImageAccessCHROMIUM: bound texture is not a shared image
[1011/134724.792894:ERROR:shared_image_manager.cc(143)] SharedImageManager::ProduceGLTexture: Trying to produce a representation from a non-existent mailbox. CB:46:F9:0D:CD:2F:FF:6B:6F:FE:5D:45:DE:0C:55:97
[1011/134724.794105:ERROR:gles2_cmd_decoder.cc(19021)] [.DisplayCompositor]GL ERROR :GL_INVALID_OPERATION : DoCreateAndTexStorage2DSharedImageINTERNAL: invalid mailbox name
[1011/134724.794208:ERROR:gles2_cmd_decoder.cc(19042)] [.DisplayCompositor]GL ERROR :GL_INVALID_OPERATION : DoBeginSharedImageAccessCHROMIUM: bound texture is not a shared image
[1011/134724.794253:ERROR:gles2_cmd_decoder.cc(13907)] [.DisplayCompositor]GL ERROR :GL_INVALID_VALUE : glScheduleCALayerCHROMIUM: unsupported texture format
[1011/134724.794283:ERROR:shared_image_manager.cc(143)] SharedImageManager::ProduceGLTexture: Trying to produce a representation from a non-existent mailbox. B8:D9:F9:B4:2E:69:F3:25:49:47:32:7E:4F:68:7D:D1
[1011/134724.794317:ERROR:gles2_cmd_decoder.cc(19021)] [.DisplayCompositor]GL ERROR :GL_INVALID_OPERATION : DoCreateAndTexStorage2DSharedImageINTERNAL: invalid mailbox name
[1011/134724.794345:ERROR:gles2_cmd_decoder.cc(19042)] [.DisplayCompositor]GL ERROR :GL_INVALID_OPERATION : DoBeginSharedImageAccessCHROMIUM: bound texture is not a shared image
[1011/134724.794890:ERROR:gles2_cmd_decoder.cc(13907)] [.DisplayCompositor]GL ERROR :GL_INVALID_VALUE : glScheduleCALayerCHROMIUM: unsupported texture format
[1011/134724.923024:ERROR:gles2_cmd_decoder.cc(19071)] [.DisplayCompositor]GL ERROR :GL_INVALID_OPERATION : DoEndSharedImageAccessCHROMIUM: bound texture is not a shared image
[1011/134724.923304:ERROR:gles2_cmd_decoder.cc(19071)] [.DisplayCompositor]GL ERROR :GL_INVALID_OPERATION : DoEndSharedImageAccessCHROMIUM: bound texture is not a shared image

tt_su · October 11, 2021, 9:43am

There won’t be output in STDOUT to the system console. Are the tests not running? No output in /Users/anuragjoshi/Bonito/Bonito_Sketchup_Tools/Product Smoke Tests/output.json?

anuragjoshi · October 11, 2021, 10:46am

No there is no output in the json. And the test is not run for sure (because in the test as I mentioned above I create a file and write).
Q: Should the josn file be already present or the test will creat it?