Unit-testing automation (TestUp2)

So we’re running the TestUp2 unit-test extension inside SketchUp in order to ensure the health of our extension-in-development.

Now I’m planning to take it to the next step by having a build server (listening to Github) run SketchUp & the test suites automatically on every commit to the source code repo.

Any experiences in setting this up using scripting (and receiving the results in shell) ?

Running SketchUp API specific functionality is of importance to us.

Might as well ad a second question: How do I ensure a test using TestUp2 (and by extension: minitest) completes after doing an async execution?

I.e.

def test_myfunction do
   do_something_async(->(result) do #this is a callback block

           print result
           assert(result)
           #ideally I would call a Done function here.

       end #end callback block
   )

  #how do I prevent test from being regarded as done at this point?
}

We want to set up something like, but have not started yet. So we’ll definitely be watching this thread, and add in our experience once we dive into it.

  • Have a unit testing framework with asynchronous support. This requires not only TestUp2 but also the underlying MiniTest testrunners to be event-based and complete the test results when triggered by events. Unfortunately MiniTest is the defacto standard and I have neither found a simple way to patch it nor another library (maybe there exists something for Rails). See also how jQuery’s Qunit does it (with done() callback).
  • Adapt your software so that it can be toggled between asynchronous and synchronous mode, and only ( :confounded: ) test the synchronous part in an automated manner. Design interfaces so that essential work is only done in easy-to-test synchronous methods.
  • I have developed an async helper library that imitates ES6’s await (using sleep) and stops waiting when the asynchronous callback triggers an event. Unfortunately it can lockup (e.g. it works in CLI, but not in SketchUp).

There is however a general problem with testing asynchronous code: The test run will never complete if a callback is not called (due to a bug). We need a timeout that is long enough to exclude the possibility of a long-lasting operation being incorrectly classified as a failure. But if many tests do time out, the test run can take a hundred times longer than it normally does.

Actually I would prefer to run all my tests in a pure command line without starting SketchUp. It would also be cool to have SketchUp in a docker container (can we have that???).

Thanks for a thorough reply.

We might just only run the core-business-logic unit-tests on our automated build-machine if no one drops info on how they have successfully run SketchUp TestUp2 tests in an automated fashion.
With this I mean the logic that doesn’t directly touch Sketch-Up.

And then we will treat all the logic touching the SketchUp API directly as an integration and have our developers run these tests manually for now inside the real SketchUp environment (we have a small dev team of 3-4 so it should be manageable).

TestUp must be run inside of SketchUp anyway.

What made you think you could run automated tests outside of a SketchUp application process ?

from the TestUp-2 Git README:

TestUp is a wrapper on top of the minitest gem. It allow SketchUp Extension developers to write minitest tests that runs within SketchUp.

I didn’t. I was looking for a way to automate the SketchUp integration tests (with the build machine firing up SketchUp, running the tests and reporting back to the external build-management environment, everything using scripts).

I saw something recently on GitHub called Thin::AsyncResponce

www.rubydoc.info/gems/thin_async/Thin/AsyncResponse

It is sort of a mod/plugin to a so called “Thin” web server framework, by the same author it seems:

2 Likes

I’m just “bustin’ your stones” :wink:

FYI

Yes, you can specify a script to run at SketchUp startup via a command line parameter:

sketchup.exe -rubyStartup "path/to/somefile.rb"

… I think, if memory serves.

And you can set an environment variable (via outside script) and read it from within
SketchUp’s Ruby process via the hash-like object ENV

test_to_run = ENV["minitest"]
1 Like

… I should add that you need to set the environment variable before you start the SketchUp process, or SketchUp’s copy of the environment will not contain the variable (or have any values set after the SketchUp process starts.)

Why not just let TestUp run its course - and push a JSON with the results back to your server?
(I can dig up how to make TestUp run a JSON reporter. By default the reporter used is to feed the results back to the webdialog.)

Just beware, if you attempt to use anything that uses threads within Ruby that will most likely not work very well inside of SketchUp. For some reason, still unknown, Ruby threads aren’t behaving well when embedded into a desktop application like SU.

This topic was automatically closed 91 days after the last reply. New replies are no longer allowed.