Sign a .rbz—requiring Trimble ID log in—from a script

#1

Hi everyone,

I’ve almost got this, but figured I might as well ask, since I still have a few steps left to figure out. (Also because identity.trimble.com is down right now, so I’m twiddling my thumbs…)

I’m trying to automate the extension signing process for our plugin, so I can integrate it into other scripts we use to build plugins for several platforms. Manual extension signing represents a break in the chain before we build our multiple-platform installers, so I’m trying to get a script for it that we can integrate.

We’re using Python and the Requests library, hoping to avoid having to do something like using Selenium (headless browser) to fill out forms.

Dealing with the .zip/.rbz/.hash file operations is no problem. The tough thing is making the http requests and getting through all the redirects and forms successfully, through the Trimble authorization service.

The process starts with https://extensions.sketchup.com/en/user/login, which also takes a destination param: https://extensions.sketchup.com/developer_center/extension_signature (where we ultimately want to end up). We use a GET request to get a hidden token from the page, and then make a POST with this token and our login data. This all seems ok so far, and we successfully follow a few redirects from here, but we don’t seem to get all the way through to being signed in—checking log in status by another GET request to the destination page (from a saved session object that stores cookies etc.) indicates that we aren’t signed in there.

I have a feeling we are missing something, indicated by the action tag associated with the sign in form. On that page, the action for the sign in form is given as ../../commonauth. Modeling from a successful manual sign in indicates that this expands to https://identity.trimble.com/commonauth. One confusing thing is that normally, using Requests for sign ins via POST means you feed the action as part of the request url, but in this case we already have a url, https://extensions.sketchup.com/en/user/login, so I’m not sure how to handle this.

At any rate, at this point I wondered whether anyone out there has tried something similar with the Trimble authorization. Thanks in advance for any thoughts you might have about this!

Jesse

2 Likes

#2

Not so much myself, but being able to automate the process would be very helpful, especially in a continuous integration setting. For example bumping the version number by clicking in a form field is superfluous as it could be specified in the extension itself (an extension manifest including all description, keywords & image references would save me also copy-pasting).

The current Extension Warehouse has not been built with a developer API (which now appears odd since it is oriented towards developers).

Ask @thomthom, he can either help figure it out, or take your input for what a developer API could look like.

1 Like

#3

Thanks. Likewise, if others are interested in the resulting scripting for logging in to Trimble and posting/downloading .rbz files, I’ll post some relevant parts here.

In our case, we’re signing for an installer, so we have to do the whole extract-the-hash thing.

0 Likes

#4

We don’t support automation of signing right now. But it is on our radar. We need for perform some infrastructure changes first to support this.

3 Likes

#5

I was eventually able to sign in to Trimble ID from our script. Will post some steps once I get a little further along.

3 Likes

#6

Great to hear. Sorry for slow response on our end. We’ve been pretty busy these last days.

0 Likes

#7

Yes please do!

1 Like

#8

The Trimble sign in is working, but I’m still working out how to get my .rbz post request working with that extension signing form. Will post some stuff once it’s further along.

1 Like

#9

In particular, if anyone has any insight perhaps, I am trying to get through the file uploading process before I am able to submit the form for signing the extension. It is looking an awful lot like you’re forced to POST your file to the plupload uploader, which means you need to get a hidden token from the page. This is normally something you can do easily with a GET request, but in this case, all of pluploader is only rendered by JS after the page loads, meaning a GET request won’t catch it because it’s not there yet.

Anyone have any idea how I can get that token so I can POST to plupload, without resorting to a headless browser? I’ve been digging through the plupload JS, but haven’t turned anything up yet.

0 Likes

#10

have you tried from a HtmlDialog inside SU…

then you can use :execute_script for all sort of things…

john

0 Likes

#11

Interesting idea, but in this case this is one python script being called from another python script. Having it open SU might not be practical.

0 Likes

#12

An update:
In the end, I couldn’t find a way without a headless browser to get the JS-rendered plupload form for uploading .rbz files. I am currently using Selenium with geckodriver to run a headless Firefox browser for the purpose of the upload form. However, to minimize the time/overhead this adds, I’m not using that for anything else—so my Requests session passes its cookies over to the webdriver, which fills out the form and then hands control back over to my Requests session.

The sign in process is still entirely doable without the headless browser. Below is a method for signing in. (This is in Python, using the Requests library, as mentioned earlier…)

def cas_sign_in():
    """Use a POST request to sign in to the Trimble Developer Center
    using their CAS (Central Authentication Service).
    """
    destination = 'https://extensions.sketchup.com/developer_center/extension_signature'
    params = {'destination': destination}
    login_url = 'https://extensions.sketchup.com/en/user/login'
    action_url = 'https://identity.trimble.com/commonauth'
    username = <our actual username>
    password = <our actual password>

    # Initialize a requests session so we retain cookies and stay signed in.
    session = requests.session()

    # Use GET request to get the unique token from the sign in page.
    login = session.get(login_url, params=params)
    # Follow through 2 additional redirects to get the page with the complete form rendered on it.
    login = login.history[2]

    # Find the hidden fields on the page (including the token).
    login_html = lxml.html.fromstring(login.text)
    hidden_inputs = login_html.xpath(r'//form//input[@type="hidden"]')

    # Initialize the form_data dict with the hidden form values.
    form_data = {x.attrib["name"]: x.value for x in hidden_inputs}

    # Add our login data to the dict.
    form_data['username'] = username
    form_data['user'] = username
    form_data['password'] = password
    form_data['action'] = action_url

    # Log in
    session.post(
        action_url,
        data=form_data,
        params={'sessionDataKey': form_data['sessionDataKey']}
        )

    # # DEBUG: check the page using the logged in session to see if the url indicates we're signed in.
    # conf = session.get('https://extensions.sketchup.com/developer_center/extension_signature')
    # print conf.text

    return session
4 Likes

#13

Well, that didn’t take long—as far as I can tell, Trimble ID has changed their login process, rendering this obsolete. I’ll update it if I’m able to find a solution. The form here now also seems to be rendered by JS in the browser. That means that a normal request won’t get the form token, since the form won’t exist yet in the response.

0 Likes

#14

Ah, bummer. Yea the Trimble ID is out of the control of the SketchUp team.

Mind you, I have made a point of highlighting the need for Signing API internally. It would simplify our workflow as well. Kenny logged a formal request in the tracker: https://github.com/SketchUp/api-issue-tracker/issues/193 (might be worth following that thread if you are interested)

1 Like

#15

Oh, that’s very good to know about, thanks.
Meanwhile, the solution here didn’t turn out to be complicated—the session’s request response turns out to have a history attribute, and the last element in this history array is the response after all the redirects. So by using the last element in the history, I’m able to get the form token and sign in again. I’ll update the code above.

I should add this is all still work in progress…

2 Likes

#16

OK, as promised, this is the current method I’ve built for posting and downloading from the extension signing form, as part of an automated build process we use to put our plugins together for all our platforms at once. In the end, I figured out how to get rid of Selenium or any other headless browser and do it all with Requests.

I haven’t removed our logging from this method—you should definitely remove the g_logger calls from the try/except blocks and add whatever you want there, or you’ll get errors on the undeclared g_logger.

Also, you’ll need to pass in your logged in session from the previous method (posted above) in order to make this work.

(And, of course, this is somewhat subject to break if Trimble makes changes to the signing form!)

In the future I’ll be updating this to work with urllib2 instead of Requests, if an API doesn’t make the whole thing obsolete.

This assumes you’ve already imported a few modules like re, requests, lxml, and so on.

    def post_rbz(rbz_file, session):
        """Use our signed-in session to send POST requests to SketchUp's extension
        signature form. There are two POST requests required to to make this work:
        POST 1: upload the .rbz file to the plupload upload handler, using a token.
        POST 2: submit the .rbz signing form, using the form tokens and references
        to the uploaded file.
        After posting, get the download link, and then pass it back to our session
        to download it.
        Return the signed file.
        """
        url = 'https://extensions.sketchup.com/en/developer_center/extension_signature'
        
        try:
            # Get the page so we can extract the plupload_token
            form_page = session.get(url)

            # Use a regex to find the plupload token value embedded in the plupload script
            plupload_token = re.search('plupload_token=(.+?)"',
                                    form_page.content).group(1)

            files = {('file', open(rbz_file, 'rb'))}
            # We must provide a name for the tempfile once it's uploaded, and then
            # refer to it by name when we submit the signing form.
            data = {'name': 'tempfile.rbz'}

            # # DEBUG: use this hook callback function to check the POST response.
            # # To use this, add this line to the POST request:
            # # hooks={'response': print_content}
            # def print_content(r, *args, **kwargs):
            #     print(r.content)

            # The file uploaded must be posted to the pluploader url.
            plup_url = 'https://extensions.sketchup.com/en/plupload-handle-uploads'

            g_logger.debug('Posting the .rbz file for upload with plupload token {}...'.format(plupload_token))

            res = session.post(
                plup_url,
                files=files,
                data=data,
                params={"plupload_token": plupload_token}
            )
        except Exception as e:
            g_logger.error('Unable to post to uploader: {}'.format(e))
            raise e

        try:
            # Submit the signing form, with references to our uploaded file.
            # First, get the signing page so we can grab required tokens and form data values:
            sign = session.get('https://extensions.sketchup.com/en/developer_center/extension_signature')
            sign_html = lxml.html.fromstring(sign.text)
            hidden_inputs = sign_html.xpath(r'//form[@id="ruby-certificate-upload-form"]//input[@type="hidden"]')

            # Add data values relating to the uploaded file
            form_data = {x.attrib["name"]: x.value for x in hidden_inputs}
            form_data['op'] = 'Sign+the+Extension'
            form_data['edit-file_count'] = '1'
            form_data['edit-file_0_status'] = 'done'
            # This sets the file name for the download file:
            form_data['edit-file_0_name'] = '<our actual plugin name>.rbz'
            # This sets the reference to the temp file we uploaded:
            form_data['edit-file_0_tmpname'] = 'tempfile.rbz'

            g_logger.debug('Posting to submit the .rbz signing form with form data {}...'.format(form_data.items()))

            # Submit the form with the data.
            res = session.post(
                url,
                data=form_data
            )
        except Exception as e:
            g_logger.error('Unable to post to .rbz signing form: {}'.format(e))
            raise e

        # Find the download URL for the signed extension by parsing the response html
        # Use xpath to find the message download link containing our extension file name.
        try:
            g_logger.debug('Searching for the download link in the response html...')
            res_html = lxml.html.fromstring(res.content)
            dl_msg_el = res_html.xpath('//*[@class="messages status"]/a[@href[contains(., "<our actual plugin name>.rbz")]]')[0]
            dl_url = dl_msg_el.get('href')
            g_logger.debug('Download link found: {}.'.format(dl_url))
        except Exception as e:
            g_logger.error('Unable to get a signed extension: {}'.format(e))
            raise e

        try:
            # Use the session to download the signed .rbz extension.
            temp_dir = os.path.dirname(rbz_file)
            dl_rbz = os.path.expanduser(os.path.join(temp_dir, '<our actual plugin name>.rbz'))
            g_logger.info('Downloading the signed .rbz to {}...'.format(dl_rbz))
            res = session.get(dl_url, stream=True)
            with open(dl_rbz, 'wb') as f:
                shutil.copyfileobj(res.raw, f)
            g_logger.info('Signed .rbz downloaded successfully.')
            return dl_rbz
        except Exception as e:
            g_logger.error('Unable to download signed extension: {}'.format(e))
            raise e
5 Likes

#17

If that is python, you can use the ```python to lex that code. :bulb: ( :white_check_mark: Done )

1 Like