# Async download a file?

**URL:** <https://forums.sketchup.com/t/async-download-a-file/123421>\
**Category:** Ruby API\
**Created:** [April 27, 2020, 9:59am UTC](https://forums.sketchup.com/t/async-download-a-file/123421 "2020-04-27T09:59:10Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![jiminybillybob](https://sea2.discourse-cdn.com/sketchup/user_avatar/forums.sketchup.com/jiminybillybob/32/197139_2.png) [@jiminybillybob](https://forums.sketchup.com/u/jiminybillybob)\
**Post date:** [April 27, 2020, 9:59am UTC](https://forums.sketchup.com/t/async-download-a-file/123421/1 "2020-04-27T09:59:10Z")

</div>

Can the `Sketchup::Http` module be used to asynchronously download files?  
It looks like so considering the `set_download_progress_callback` method.

But where is the file stored once downloaded?

Is there more information somewhere?

---

<div class="post-metadata">

**Author:** ![eneroth3](https://sea2.discourse-cdn.com/sketchup/user_avatar/forums.sketchup.com/eneroth3/32/1222_2.png) [@eneroth3](https://forums.sketchup.com/u/eneroth3)\
**Post date:** [April 27, 2020, 10:00am UTC](https://forums.sketchup.com/t/async-download-a-file/123421/2 "2020-04-27T10:00:56Z")

</div>

I think you get the response as a String. You should then be able to save it out anywhere you want it to go.

---

<div class="post-metadata">

**Author:** ![MSP\_Greg](https://sea2.discourse-cdn.com/sketchup/user_avatar/forums.sketchup.com/msp_greg/32/206066_2.png) [@MSP\_Greg](https://forums.sketchup.com/u/MSP_Greg)\
**Post date:** [April 27, 2020, 4:40pm UTC](https://forums.sketchup.com/t/async-download-a-file/123421/4 "2020-04-27T16:40:23Z")

</div>

@kengey

> when dealing with large files

How large? The below seemed to work several times on a 10.3 MB file. Code isn’t quite what I’d use in an app…

```ruby
# frozen_string_literal: true

module SUFileDownload
  URI = 'https://github.com/MSP-Greg/ruby-loco/releases/download/ruby-master/ruby-mingw.7z'
  FN = "#{ __dir__ }/ruby-mingw.7z"

  class << self
    def run
      t_st = Time.now.to_f
      req = Sketchup::Http::Request.new URI
      req.start do |req, resp|
        if req.status == Sketchup::Http::STATUS_SUCCESS
          File.write FN, resp.body, mode: 'wb'
          puts "Download time: %5.2f sec" % (Time.now.to_f - t_st)
        else
          puts "#{req.status} can't download"
        end
      end
    end
  end
end

SUFileDownload.run

```

---

<div class="post-metadata">

**Author:** ![DanRathbun](https://sea2.discourse-cdn.com/sketchup/user_avatar/forums.sketchup.com/danrathbun/32/500_2.png) [@DanRathbun](https://forums.sketchup.com/u/DanRathbun)\
**Post date:** [April 27, 2020, 5:50pm UTC](https://forums.sketchup.com/t/async-download-a-file/123421/6 "2020-04-27T17:50:26Z")

</div>

> [@jiminybillybob](#):
>
> But where is the file stored once downloaded?

As shown [above](https://forums.sketchup.com/t/async-download-a-file/123421/4), it is loaded as the `response.body`.

> [@jiminybillybob](#):
>
> It looks like so considering the `set_download_progress_callback` method.

The block parameters for [this method](https://ruby.sketchup.com/Sketchup/Http/Request.html#set_download_progress_callback-instance_method) have a bug (at least on Windows.)  
I logged it privately, but I should create a public issue for it.

Basically the `total` parameter is `nil` or `0` (cannot remember which,) … making any comparison with the `current` value (as to percent done downloading) a no-op.

So your code would need to know how big the file to be downloaded really is, **before** the download begins.

---

<div class="post-metadata">

**Author:** ![MSP\_Greg](https://sea2.discourse-cdn.com/sketchup/user_avatar/forums.sketchup.com/msp_greg/32/206066_2.png) [@MSP\_Greg](https://forums.sketchup.com/u/MSP_Greg)\
**Post date:** [April 27, 2020, 7:53pm UTC](https://forums.sketchup.com/t/async-download-a-file/123421/7 "2020-04-27T19:53:33Z")

</div>

@kengey

I played around with this a bit more. Used a 127MB file, and it downloaded ok with both Net::HTTP and Sketchup::Http. Both took about the same amount of time, and which was fastest varied. The file is hosted on AWS from a GitHub redirect, so the connection is solid (and also https).

But, Net::HTTP showed an hourglass, while Sketchup::Http did not.

Hence, if you’re having issues with Sketchup::Http, it may be the manner in which you’re calling it, as you mentioned.

I did find that https using Net::HTTP with OpenSSL \<= 1.0.2 / Ruby \< 2.5 / SU \<= 2018 is a mess. I had a large model (30 MB) loaded, and the first connection took 22 sec, and that was downloading a small file. Thomas has mentioned this, and tracked the problem down a while ago.

I’m not sure how it could be done, but some way to run ‘safe’ Ruby code in a manner that didn’t ‘block’ SU would be helpful.

Off topic, but I ran the downloads with both mingw & mswin Ruby builds, no difference.

---

<div class="post-metadata">

**Author:** ![kengey](https://sea2.discourse-cdn.com/sketchup/user_avatar/forums.sketchup.com/kengey/32/234545_2.png) [@kengey](https://forums.sketchup.com/u/kengey)\
**Post date:** [April 27, 2020, 8:00pm UTC](https://forums.sketchup.com/t/async-download-a-file/123421/8 "2020-04-27T20:00:48Z")

</div>

> [@MSP\_Greg](#):
>
> Hence, if you’re having issues with Sketchup::Http, it may be the manner in which you’re calling it, as you mentioned.

I guess so. I am not always having trouble with it, not at all. One of my larger extensions is having lots of issues with it. There is an update checker that does an http request. In the callback, if there is an update, it shows a UI::Notification, and, in the callback of that notification, which gets called if the user does press the agree button, , it does a second request to actually download the update. This chain gave me many headaches, which I probably assigned falsely to the http module.

---

<div class="post-metadata">

**Author:** ![MSP\_Greg](https://sea2.discourse-cdn.com/sketchup/user_avatar/forums.sketchup.com/msp_greg/32/206066_2.png) [@MSP\_Greg](https://forums.sketchup.com/u/MSP_Greg)\
**Post date:** [April 28, 2020, 1:50am UTC](https://forums.sketchup.com/t/async-download-a-file/123421/9 "2020-04-28T01:50:54Z")

</div>

I set a separate script to download the 130 MB file and then exit, and in SU spawned a process running it with rubyw.exe.

Then using a UI timer and `Process.waitpid(pid, Process::WNOHANG)` in the timer block, waited for the download to finish. SU seemed to stay totally responsive during the download.

One might consider the spawned ruby process similar to a web worker, in that it can’t use any SU objects.

Regardless, having ruby executables included with SU would allow devs to expand on what can currently be done.

---

<div class="post-metadata">

**Author:** ![kengey](https://sea2.discourse-cdn.com/sketchup/user_avatar/forums.sketchup.com/kengey/32/234545_2.png) [@kengey](https://forums.sketchup.com/u/kengey)\
**Post date:** [April 28, 2020, 5:26am UTC](https://forums.sketchup.com/t/async-download-a-file/123421/10 "2020-04-28T05:26:24Z")

</div>

> [@MSP\_Greg](#):
>
> Regardless, having ruby executables included with SU would allow devs to expand on what can currently be done.

Love the idea to have these async web-worker-like ruby executables, which you could send a script to. I currently do more or less the same, but with .net executables that I spawn and check their pid in a timer. You can then also define a pool size, of lets say 5, and continously check which subprocesses are done and execute new tasks in parallel.  
Maybe start another thread about this? Looks cool

If only communication using stdin stdout with a subprocess was not broken in SU…

Edit: I also removed my comments regarding the issues I experience with Sketchup:HTTP, it must be because I use it in some edge case, not really adding to the topic.

---

<div class="post-metadata">

**Author:** ![jiminybillybob](https://sea2.discourse-cdn.com/sketchup/user_avatar/forums.sketchup.com/jiminybillybob/32/197139_2.png) [@jiminybillybob](https://forums.sketchup.com/u/jiminybillybob)\
**Post date:** [April 28, 2020, 12:36pm UTC](https://forums.sketchup.com/t/async-download-a-file/123421/11 "2020-04-28T12:36:42Z")

</div>

Thank you all for your input!

I don’t know how I didn’t think about that ^^

---

<div class="post-metadata">

**Author:** ![MSP\_Greg](https://sea2.discourse-cdn.com/sketchup/user_avatar/forums.sketchup.com/msp_greg/32/206066_2.png) [@MSP\_Greg](https://forums.sketchup.com/u/MSP_Greg)\
**Post date:** [April 28, 2020, 2:46pm UTC](https://forums.sketchup.com/t/async-download-a-file/123421/12 "2020-04-28T14:46:50Z")

</div>

> [@kengey](#):
>
> Maybe start another thread about this?

See [Include Ruby executables with SU?](https://forums.sketchup.com/t/include-ruby-executables-with-su/123590)

---

<div class="post-metadata">

**Author:** ![DanRathbun](https://sea2.discourse-cdn.com/sketchup/user_avatar/forums.sketchup.com/danrathbun/32/500_2.png) [@DanRathbun](https://forums.sketchup.com/u/DanRathbun)\
**Post date:** [April 28, 2020, 3:35pm UTC](https://forums.sketchup.com/t/async-download-a-file/123421/13 "2020-04-28T15:35:30Z")

</div>

> [@DanRathbun](#):
>
> The block parameters for [this method](https://ruby.sketchup.com/Sketchup/Http/Request.html#set_download_progress_callback-instance_method) have a bug (at least on Windows.)  
> I logged it privately, but I should create a public issue for it.
> 
> Basically the `total` parameter is `nil` or `0` (cannot remember which,) … making any comparison with the `current` value (as to percent done downloading) a no-op.

Actually the `total` parameter is always `-1`.

I looked up my previous report and see that it was internally logged on 1 OCT 2018 by Hilllard.

I’ve relogged in the public Issue tracker for all to see and track. Also updated the test script to v3.

> <https://github.com/SketchUp/api-issue-tracker/issues/463>
>
> \### SketchUp Ruby API Issue
> 
> \*\*\[\`Sketchup::Http::Request.set\_download\_progress…\_callback\`\](https://ruby.sketchup.com/Sketchup/Http/Request.html#set\_download\_progress\_callback-instance\_method) is near worthless.\*\*
> 
> The \*\*\`total\`\*\* block parameter is always \*\*\`-1\`\*\* (making comparison to the \*\*\`current\`\*\* block parameter worthless.)
> 
> \>\*\*EDIT\*\*: \[TH believes\](https://forums.sketchup.com/t/async-download-a-file/123421/17) that the \*\*\`total\`\*\* value being \*\*\`-1\`\*\* is a documentation issue.
> \> Ie, that \[\`RFC2616 sec14.13\`\](https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.13) states that \_"Any \*\*Content-Length\*\* greater than or equal to zero is a valid value."\_ and so \*\*\`-1\`\*\* indicates that the server at the URL did not return a \*\*\`"Content-Length"\`\*\* header that could be passed into the block as the 2nd parameter.
> 
> In addition, the block runs the final time after download is complete ... \*\*BUT,\*\* the status is \*\*erroneously\*\* still \*\*\`STATUS\_PENDING\`\*\* in the final call even though the \`current\` parameter is equal to the full download size (of the \`response.body\`.)
> 
> \> \*\*EDIT\*\*: This "might" also be happening because the test URL uses GitHub, which is not returning a \*\*\`"Content-Length"\`\*\* header for a HEAD request.
> We need another URL to test with. Perhaps a EW model URL ?
> 
> We \*\*can\*\* get the full download from the \`response\` body in the \`request#start\` callback block.
> 
> In order for a workaround, we'd need to find some way to figure out ourselves what the download size is in order to have something to compare the current parameter to within the progress callback in order to make it useful and display a progress bar.
> 
> Platform: Windows 10
> Bugged in SU2017, SU2018, SU2019 and SU2020.
> 
> \*\*Reproduce by\*\*: run \`DLTEST.go\` and observe console output.
> \* \[DLTEST\_v3.zip\](https://github.com/SketchUp/api-issue-tracker/files/4546734/DLTEST\_v3.zip)
> 
> \----
> 
> \*\*NOTE: Jira issue created: SU-40832\*\* by @hilliard-sketchup on 1 OCT 2018

---

<div class="post-metadata">

**Author:** ![jiminybillybob](https://sea2.discourse-cdn.com/sketchup/user_avatar/forums.sketchup.com/jiminybillybob/32/197139_2.png) [@jiminybillybob](https://forums.sketchup.com/u/jiminybillybob)\
**Post date:** [April 28, 2020, 6:55pm UTC](https://forums.sketchup.com/t/async-download-a-file/123421/14 "2020-04-28T18:55:52Z")

</div>

> [@DanRathbun](#):
>
> Actually the `total` parameter is always `-1` .

That’s not the experience I’m having.  
The total is correct on SketchUp 2017 up to 2020.

In your case, does the response header contain the content size?

---

<div class="post-metadata">

**Author:** ![DanRathbun](https://sea2.discourse-cdn.com/sketchup/user_avatar/forums.sketchup.com/danrathbun/32/500_2.png) [@DanRathbun](https://forums.sketchup.com/u/DanRathbun)\
**Post date:** [April 28, 2020, 7:18pm UTC](https://forums.sketchup.com/t/async-download-a-file/123421/15 "2020-04-28T19:18:46Z")

</div>

> [@jiminybillybob](#):
>
> The total is correct on SketchUp 2017 up to 2020.

I just tested it this morning on 2020, and had tested it back to 2017 in the past.  
Did you test with the test script I posted ?

Do you have **another URL** that I can run the test script on?

> [@jiminybillybob](#):
>
> In your case, does the response header contain the content size?

The **response** is **not available** in the block for the [method](https://ruby.sketchup.com/Sketchup/Http/Request.html#set_download_progress_callback-instance_method) that the [issue](https://github.com/SketchUp/api-issue-tracker/issues/463) is posted for.

I think you need to **reread** the posted issue. The **`current`** parameter (which is a cumulative value) is correct, and **will be** the total value downloaded on the **last call** of the block, BUT … there is no way to know which block call is the last, because (as I wrote in the issue report) the `request.status` is still `STATUS_PENDING`.

---

<div class="post-metadata">

**Author:** ![DanRathbun](https://sea2.discourse-cdn.com/sketchup/user_avatar/forums.sketchup.com/danrathbun/32/500_2.png) [@DanRathbun](https://forums.sketchup.com/u/DanRathbun)\
**Post date:** [April 29, 2020, 12:49am UTC](https://forums.sketchup.com/t/async-download-a-file/123421/16 "2020-04-29T00:49:28Z")

</div>

> [@jiminybillybob](#):
>
> In your case, does the response header contain the content size?

Are you suggesting a **workaround** to **firstly** make a HEAD request, get the “Content-Length” and then make the actual GET request and use the size from response to the HEAD request ?

Tried this. _See discussion:_

> <https://stackoverflow.com/questions/3854842/content-length-header-with-head-requests>

**Unfortunately** , servers MAY (and often do) omit the “Content-Length” header in response to HEAD requests.

My test happens to request a zip from GitHub which isn’t **usually** returning a “Content-Length” for a HEAD request. (I did once get it to do so, but I don’t know what and why it did in that case. It was a bogus mistaken request header _I think_ that triggered it.)

---

<div class="post-metadata">

**Author:** ![jiminybillybob](https://sea2.discourse-cdn.com/sketchup/user_avatar/forums.sketchup.com/jiminybillybob/32/197139_2.png) [@jiminybillybob](https://forums.sketchup.com/u/jiminybillybob)\
**Post date:** [April 29, 2020, 7:53am UTC](https://forums.sketchup.com/t/async-download-a-file/123421/17 "2020-04-29T07:53:44Z")

</div>

> [@DanRathbun](#):
>
> My test happens to request a zip from GitHub which isn’t **usually** returning a “Content-Length” for a HEAD request

That’s my point. If the server is not returning the `Content-Length`, how could SketchUp know what it’s supposed to be?

---

<div class="post-metadata">

**Author:** ![DanRathbun](https://sea2.discourse-cdn.com/sketchup/user_avatar/forums.sketchup.com/danrathbun/32/500_2.png) [@DanRathbun](https://forums.sketchup.com/u/DanRathbun)\
**Post date:** [April 30, 2020, 12:52am UTC](https://forums.sketchup.com/t/async-download-a-file/123421/18 "2020-04-30T00:52:22Z")

</div>

> [@jiminybillybob](#):
>
> That’s my point. If the server is not returning the `Content-Length` , how could SketchUp know what it’s supposed to be?

Well, I am operating under the principle (which Trimble employees keep stating) that the API documentation states “contracts” for feature behavior.

In the case of the [`Sketchup::Http::Request#set_download_progress_callback`](https://ruby.sketchup.com/Sketchup/Http/Request.html#set_download_progress_callback-instance_method) method, the docs state without any variance, that the **`total`** block parameter **is** …

> - _`total` (`Integer`) — Total bytes to transfer._

So, if the **documentation** is incorrect it needs to be corrected to say _for example_ …

> When the request is a GET, and this method registers a callback block, the API will first send a HEAD request for an attempt to retrieve the **“Content-Length”** header for the **`total`** block parameter.  
> If the server at the request URL does not return this header per [**`RFC2616:9.4`**](https://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html#sec9.4) or [**`RFC2616:14.13`**](https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.13), then the value of the **`total`** block parameter will be **`-1`**. Coders should write their callback code in an agile manner to account for situations where the total length of the download is unknown.

I’ve updated the [tracker Issue](https://github.com/SketchUp/api-issue-tracker/issues/463) with comments to this effect.

Still seeking a test URL for a server that will return a `"Content-Length"` header in response to a HEAD request, with a `"Access-Control-Request-Method": "GET"` header.

---

<div class="post-metadata">

**Author:** ![MSP\_Greg](https://sea2.discourse-cdn.com/sketchup/user_avatar/forums.sketchup.com/msp_greg/32/206066_2.png) [@MSP\_Greg](https://forums.sketchup.com/u/MSP_Greg)\
**Post date:** [April 30, 2020, 12:54am UTC](https://forums.sketchup.com/t/async-download-a-file/123421/19 "2020-04-30T00:54:08Z")

</div>

If you want a progress indicator and can live with a sync (blocking download), the attached file shows an example using Net::HTTP. It just outputs the percent and ‘step’ to the console approx every second…

**EDIT:** first copy of the file I uploaded had an error in it, as I mistakenly left `io.write chunk` inside a conditional it shouldn’t have been inside. I added the conditional to ‘lighten’ the logging of the download progress…

[download-test\_su-http.rb](https://forums.sketchup.com/uploads/short-url/Io9Eh7XTnhVlfuoLhhpqCN49y1.rb) (3.0 KB)

---

<div class="post-metadata">

**Author:** ![DanRathbun](https://sea2.discourse-cdn.com/sketchup/user_avatar/forums.sketchup.com/danrathbun/32/500_2.png) [@DanRathbun](https://forums.sketchup.com/u/DanRathbun)\
**Post date:** [April 30, 2020, 12:59am UTC](https://forums.sketchup.com/t/async-download-a-file/123421/20 "2020-04-30T00:59:17Z")

</div>

Well, yes, I’ve used [`Net::HTTP`](https://ruby-doc.org/stdlib-2.5.5/libdoc/net/http/rdoc/index.html) in the past (especially before the SketchUp API added [`Sketchup::Http`](https://ruby.sketchup.com/Sketchup/Http.html) classes.)

**But** … this, (knowing the download size in order to calculate percentage,) is kind of more about the SketchUp API and it’s documentation.

But to humor you, how would the Standard libraries deal with the same issue ?  
Ie, a server that does **not** return a `"Content-Length"` header to a HEAD request ?

---

<div class="post-metadata">

**Author:** ![MSP\_Greg](https://sea2.discourse-cdn.com/sketchup/user_avatar/forums.sketchup.com/msp_greg/32/206066_2.png) [@MSP\_Greg](https://forums.sketchup.com/u/MSP_Greg)\
**Post date:** [April 30, 2020, 2:34am UTC](https://forums.sketchup.com/t/async-download-a-file/123421/21 "2020-04-30T02:34:31Z")

</div>

> But to humor you

Really?

> Well, yes, I’ve used [`Net::HTTP`](https://ruby-doc.org/stdlib-2.5.5/libdoc/net/http/rdoc/index.html) in the past

Apparently not enough.

> Ie, a server that does **not** return a `"Content-Length"` header to a HEAD request

Look at the code I attached. With `Sketchup::Http`, the headers are only available after the full response body has been retrieved. So you thought doing a HEAD request was a possible solution.

HTTP doesn’t work that way, that’s only the SU implementation. With `Net::HTTP`, the headers are available immediately, and most downloads of any size are chunked. Hence, one can calculate the percent completed. There are too many chunks to have SU respond for each, so that’s why the code only outputs approx every second…

---

<div class="post-metadata">

**Author:** ![DanRathbun](https://sea2.discourse-cdn.com/sketchup/user_avatar/forums.sketchup.com/danrathbun/32/500_2.png) [@DanRathbun](https://forums.sketchup.com/u/DanRathbun)\
**Post date:** [April 30, 2020, 3:45am UTC](https://forums.sketchup.com/t/async-download-a-file/123421/22 "2020-04-30T03:45:42Z")

</div>

> [@MSP\_Greg](#):
>
> Apparently not enough.

Ouch! I’ll admit it’s been about a year since. Thanks for “rubbin’ it in”.

> [@MSP\_Greg](#):
>
> Look at the code I attached.

I am (and am about to test it out.)

> [@MSP\_Greg](#):
>
> With `Sketchup::Http`, the headers are only available after the full response body has been retrieved.

I know this, and perhaps this is part of the problem with the API download and upload callback block form methods. (Ie, … they “dumbed these classes down” too much. It might be beneficial if the chunked response headers were available as a block parameter for the these methods.)

> [@MSP\_Greg](#):
>
> So you thought doing a HEAD request was a possible solution.

A workaround actually. I think that a GET `Sketchup::Http::Request` might actually do this “under the hood” so this is why I tried it. It has helped us to understand what is going on and that code (in these API callback blocks) needs to be a bit more agile.

> [@MSP\_Greg](#):
>
> HTTP doesn’t work that way, that’s only the SU implementation.

Again, I realize this (after spending more than 3 days playing and reading RFCs.)

The SketchUp API has wrapped up a bunch of implementation, but poorly documented what to expect when using these methods. I feel that the coders of this implementation must have known that the download `total` parameter would remain as initialized at `-1` if the `"Content-Length"` header was not present, but **did not document this in the API dictionary**.  
One simple short line of explanation in the docs could have saved me from wasting at least 3 days (probably more) on this issue.

[Next page](https://forums.sketchup.com/t/async-download-a-file/123421.md?page=2)
