Async download a file?

Developers Ruby API
1

Can the Sketchup::Http module be used to asynchronously download files?
It looks like so considering the set_download_progress_callback method.

But where is the file stored once downloaded?

Is there more information somewhere?

2

I think you get the response as a String. You should then be able to save it out anywhere you want it to go.

4

@kengey

when dealing with large files

How large? The below seemed to work several times on a 10.3 MB file. Code isnā€™t quite what Iā€™d use in an appā€¦

# frozen_string_literal: true

module SUFileDownload
  URI = 'https://github.com/MSP-Greg/ruby-loco/releases/download/ruby-master/ruby-mingw.7z'
  FN = "#{__dir__}/ruby-mingw.7z"

  class << self
    def run
      t_st = Time.now.to_f
      req = Sketchup::Http::Request.new URI
      req.start do |req, resp|
        if req.status == Sketchup::Http::STATUS_SUCCESS
          File.write FN, resp.body, mode: 'wb'
          puts "Download time: %5.2f sec" % (Time.now.to_f - t_st)
        else
          puts "#{req.status} can't download"
        end
      end
    end
  end
end

SUFileDownload.run
6

As shown above, it is loaded as the response.body.

The block parameters for this method have a bug (at least on Windows.)
I logged it privately, but I should create a public issue for it.

Basically the total parameter is nil or 0 (cannot remember which,) ā€¦ making any comparison with the current value (as to percent done downloading) a no-op.

So your code would need to know how big the file to be downloaded really is, before the download begins.

7

@kengey

I played around with this a bit more. Used a 127MB file, and it downloaded ok with both Net::HTTP and Sketchup::Http. Both took about the same amount of time, and which was fastest varied. The file is hosted on AWS from a GitHub redirect, so the connection is solid (and also https).

But, Net::HTTP showed an hourglass, while Sketchup::Http did not.

Hence, if youā€™re having issues with Sketchup::Http, it may be the manner in which youā€™re calling it, as you mentioned.

I did find that https using Net::HTTP with OpenSSL <= 1.0.2 / Ruby < 2.5 / SU <= 2018 is a mess. I had a large model (30 MB) loaded, and the first connection took 22 sec, and that was downloading a small file. Thomas has mentioned this, and tracked the problem down a while ago.

Iā€™m not sure how it could be done, but some way to run ā€˜safeā€™ Ruby code in a manner that didnā€™t ā€˜blockā€™ SU would be helpful.

Off topic, but I ran the downloads with both mingw & mswin Ruby builds, no difference.

2 Likes
8

I guess so. I am not always having trouble with it, not at all. One of my larger extensions is having lots of issues with it. There is an update checker that does an http request. In the callback, if there is an update, it shows a UI::Notification, and, in the callback of that notification, which gets called if the user does press the agree button, , it does a second request to actually download the update. This chain gave me many headaches, which I probably assigned falsely to the http module.

9

I set a separate script to download the 130 MB file and then exit, and in SU spawned a process running it with rubyw.exe.

Then using a UI timer and Process.waitpid(pid, Process::WNOHANG) in the timer block, waited for the download to finish. SU seemed to stay totally responsive during the download.

One might consider the spawned ruby process similar to a web worker, in that it canā€™t use any SU objects.

Regardless, having ruby executables included with SU would allow devs to expand on what can currently be done.

1 Like
10

Love the idea to have these async web-worker-like ruby executables, which you could send a script to. I currently do more or less the same, but with .net executables that I spawn and check their pid in a timer. You can then also define a pool size, of lets say 5, and continously check which subprocesses are done and execute new tasks in parallel.
Maybe start another thread about this? Looks cool

If only communication using stdin stdout with a subprocess was not broken in SUā€¦

Edit: I also removed my comments regarding the issues I experience with Sketchup:HTTP, it must be because I use it in some edge case, not really adding to the topic.

11

Thank you all for your input!

I donā€™t know how I didnā€™t think about that ^^

12

See Include Ruby executables with SU?

1 Like
13

Actually the total parameter is always -1.

I looked up my previous report and see that it was internally logged on 1 OCT 2018 by Hilllard.

Iā€™ve relogged in the public Issue tracker for all to see and track. Also updated the test script to v3.

14

Thatā€™s not the experience Iā€™m having.
The total is correct on SketchUp 2017 up to 2020.

In your case, does the response header contain the content size?

15

I just tested it this morning on 2020, and had tested it back to 2017 in the past.
Did you test with the test script I posted ?

Do you have another URL that I can run the test script on?

The response is not available in the block for the method that the issue is posted for.

I think you need to reread the posted issue. The current parameter (which is a cumulative value) is correct, and will be the total value downloaded on the last call of the block, BUT ā€¦ there is no way to know which block call is the last, because (as I wrote in the issue report) the request.status is still STATUS_PENDING.

16

Are you suggesting a workaround to firstly make a HEAD request, get the ā€œContent-Lengthā€ and then make the actual GET request and use the size from response to the HEAD request ?

Tried this. See discussion:

Unfortunately, servers MAY (and often do) omit the ā€œContent-Lengthā€ header in response to HEAD requests.

My test happens to request a zip from GitHub which isnā€™t usually returning a ā€œContent-Lengthā€ for a HEAD request. (I did once get it to do so, but I donā€™t know what and why it did in that case. It was a bogus mistaken request header I think that triggered it.)

17

Thatā€™s my point. If the server is not returning the Content-Length, how could SketchUp know what itā€™s supposed to be?

1 Like
18

Well, I am operating under the principle (which Trimble employees keep stating) that the API documentation states ā€œcontractsā€ for feature behavior.

In the case of the Sketchup::Http::Request#set_download_progress_callback method, the docs state without any variance, that the total block parameter is ā€¦

  • total (Integer) ā€” Total bytes to transfer.

So, if the documentation is incorrect it needs to be corrected to say for example ā€¦

When the request is a GET, and this method registers a callback block, the API will first send a HEAD request for an attempt to retrieve the ā€œContent-Lengthā€ header for the total block parameter.
If the server at the request URL does not return this header per RFC2616:9.4 or RFC2616:14.13, then the value of the total block parameter will be -1. Coders should write their callback code in an agile manner to account for situations where the total length of the download is unknown.

Iā€™ve updated the tracker Issue with comments to this effect.

Still seeking a test URL for a server that will return a "Content-Length" header in response to a HEAD request, with a "Access-Control-Request-Method": "GET" header.

19

If you want a progress indicator and can live with a sync (blocking download), the attached file shows an example using Net::HTTP. It just outputs the percent and ā€˜stepā€™ to the console approx every secondā€¦

EDIT: first copy of the file I uploaded had an error in it, as I mistakenly left io.write chunk inside a conditional it shouldnā€™t have been inside. I added the conditional to ā€˜lightenā€™ the logging of the download progressā€¦

download-test_su-http.rb (3.0 KB)

20

Well, yes, Iā€™ve used Net::HTTP in the past (especially before the SketchUp API added Sketchup::Http classes.)

But ā€¦ this, (knowing the download size in order to calculate percentage,) is kind of more about the SketchUp API and itā€™s documentation.

But to humor you, how would the Standard libraries deal with the same issue ?
Ie, a server that does not return a "Content-Length" header to a HEAD request ?

21

But to humor you

Really?

Well, yes, Iā€™ve used Net::HTTP in the past

Apparently not enough.

Ie, a server that does not return a "Content-Length" header to a HEAD request

Look at the code I attached. With Sketchup::Http, the headers are only available after the full response body has been retrieved. So you thought doing a HEAD request was a possible solution.

HTTP doesnā€™t work that way, thatā€™s only the SU implementation. With Net::HTTP, the headers are available immediately, and most downloads of any size are chunked. Hence, one can calculate the percent completed. There are too many chunks to have SU respond for each, so thatā€™s why the code only outputs approx every secondā€¦

22

Ouch! Iā€™ll admit itā€™s been about a year since. Thanks for ā€œrubbinā€™ it inā€.

I am (and am about to test it out.)

I know this, and perhaps this is part of the problem with the API download and upload callback block form methods. (Ie, ā€¦ they ā€œdumbed these classes downā€ too much. It might be beneficial if the chunked response headers were available as a block parameter for the these methods.)

A workaround actually. I think that a GET Sketchup::Http::Request might actually do this ā€œunder the hoodā€ so this is why I tried it. It has helped us to understand what is going on and that code (in these API callback blocks) needs to be a bit more agile.

Again, I realize this (after spending more than 3 days playing and reading RFCs.)

The SketchUp API has wrapped up a bunch of implementation, but poorly documented what to expect when using these methods. I feel that the coders of this implementation must have known that the download total parameter would remain as initialized at -1 if the "Content-Length" header was not present, but did not document this in the API dictionary.
One simple short line of explanation in the docs could have saved me from wasting at least 3 days (probably more) on this issue.