Async download a file?

[MSP_Greg]

download total parameter would remain as initialized at -1 if the "Content-Length" header was not present

You might check that. I think I did, but I don’t recall. Even with the Content-Length header present, I don’t think it works correctly. It is present with the GitHub downloads. Also, I think the download used in the example does not have a redirect, but the larger one (that’s commented out) does. That may also affect it, not sure.

Regardless, Sketchup::Http was probably created due to the issues with OpenSSL 1.0.2/Ruby 2.2. Those issues aren’t a problem with Ruby 2.5 and later.

With most downloads, the user is probably performing some action to start them, so having an async method for something that will probably freeze the UI when it’s done is kind of pointless for most downloads. Freezing a high capacity server is one thing, but a second or two for a UI isn’t as critical.

Over time, I’ve seen enough comments about Net::HTTP not working that I thought that, given using Ruby or later, that a) it really isn’t a problem, and b) it is a mature library. Hence, some code to demo how it works.

Also, async/promises was a big thing with node.js for a while, but now it’s moving somewhat back to sync for many things, as socket based API’s don’t run async very well. Remember also that async can really strain a call stack. Using a Ruby term, think of all the binding that have to be saved with a large async call chain…

[DanRathbun]

@MSP_Greg

With most downloads, the user is probably performing some action to start them, so having an async method for something that will probably freeze the UI when it’s done is kind of pointless for most downloads.

I think the main thing was to try to keep the SketchUp app from going into “Not Responding” mode.
Users tend to think that the application has locked up and try to end the task w/ the Task Manager.

@MSP_Greg

You might check that. I think I did, but I don’t recall. Even with the Content-Length header present, I don’t think it works correctly.

I’d like to check it. Thomas (above) says it works for him. But I don’t yet have a test URL outside of GitHub.

@MSP_Greg

Also, I think the download used in the example does not have a redirect, but the larger one (that’s commented out) does.

It took awhile but I got your example to work.

Made some changes and additions. Both URIs I tried at GitHub had redirects.

It looks like small zip files will never have ‘Content-Length’ but big 7z files will.
But the zip file was still off of GitHub, and the big Ruby 7z off of "amazonaws.com"

download-test_su-http_v2.1.rb (6.7 KB)

[eneroth3]

I’m not sure why an initial head request would be needed. SketchUp could do the Get request and report back the Content-Size from that. Since every web browser manages to show the download progression even if Head requests lack the Content-Size value, this can’t be the problem.

@DanRathbun

So, if the documentation is incorrect it needs to be corrected to say for example …

When the request is a GET, and this method registers a callback block, the API will first send a HEAD request for an attempt to retrieve the “Content-Length” header for the total block parameter.
If the server at the request URL does not return this header per RFC2616:9.4 or RFC2616:14.13 , then the value of the total block parameter will be -1 . Coders should write their callback code in an agile manner to account for situations where the total length of the download is unknown.

The documentation could just state:

total ( Integer ) — Total bytes to transfer. -1 if unknown.

There could be an additional paragraph explaining why the value might be unknown, but you should absolutely not have to go through such an explanation just to get to the fact that -1 can be returned.

[MSP_Greg]

It took awhile but I got your example to work.

The code did run with SU 2019 with both Ruby 2.8 & 2.5… (?)

My sample posts aren’t production code, and I normally test in SU with a relatively recent Ruby master version.

If some of the code will run in stand-alone Ruby, I’ll set it up so it will run both from stand-alone and the SU Ruby console. I almost never hook things to menu items, as one can load the file or call the method(s) from the console.

You might check you code though, I didn’t check everything. Line 43 and 53 (?, and maybe more) have frozen string errors, and line 145 needs quotes:

puts TRYING REDIRECT:

[DanRathbun]

@eneroth3

I’m not sure why an initial head request would be needed.

Perhaps not, as it seems if a server does not return a "Content-Length" in the response, then it seems these servers usually do not return one for a HEAD request.

I myself tried it as an attempted workaround because the total length was -1 (in every call to the block) and I didn’t know why as the docs were no help.

Ie … if (as ThomThom is always saying) the docs from a API contract of functionality, and they state unambiguously without variance, “Total bytes to transfer.” So, based upon this, I had to assume that there was a bug, and filed a bug report (twice.)

@eneroth3

The documentation could just state:

total ( Integer ) — Total bytes to transfer. -1 if unknown.

True. And even that little extra bit of info would have been better, and I would then not have assumed a bug.

@eneroth3

There could be an additional paragraph explaining why the value might be unknown, but you should absolutely not have to go through such an explanation just to get to the fact that -1 can be returned.

I’ll agree. I often tend to be too verbose. And the suggested “blurb” above is spectulating about internal implementation which ThomThom does not care to put into the docs.

Likely a single short sentence in the method’s YARD docstring could replace my proposed 2 paragraphs. Something to the effect that sometimes the server will not return the length of the download.

@MSP_Greg

The code did run with SU 2019 with both Ruby 2.8 & 2.5… (?)

I ran it on 2020 under Ruby 2.5.5 as it comes “out of the box”.

I don’t know why but at first I was getting errors with no download and no feedback, so I inserted some puts to spit out some values (the statements inside “comment brackets”, which can be removed when no longer needed.) Might have been a server error, because all of a sudden it started working.

The only major error (causing the download to “crash”) was that when the "Content-Length" header is nil (not sent,) the length variable becomes 0 and the percent downloaded format string causes a divide by zero error. So I associated conditionals with those statements to check if length > 0.
Really though, this should be adjusted to just display the bytes downloaded without a percentage in these cases, instead of wholly omitting the bytes downloaded.
Same for the general use of the Sketchup::Http::Request#set_download_progress_callback block.

@MSP_Greg

If some of the code will run in stand-alone Ruby, I’ll set it up so it will run both from stand-alone and the SU Ruby console.

I think it’s already set up this way.

@MSP_Greg

I almost never hook things to menu items, as one can load the file or call the method(s) from the console.

I have arthritis in my hands, so every little bit helps, especially if the file needs to be tweaked and reloaded again, and again, etc. Rerun, and rerun …

Also, very often within SketchUp we need to test how things work without the console having the focus. Ie the main application window having focus, and clicking a menu item does this.

@MSP_Greg

You might check you code though, I didn’t check everything. Line 43 and 53 (?, and maybe more) have frozen string errors, and line 145 needs quotes:

It was 3am. Sorry. I threw that in at the last and didn’t run it afterward. “My bad!”

EDIT: Fixed the file posted above. Bumped to ver “2.1.0”

[DanRathbun]

@MSP_Greg

Line 43 and 53 (?, and maybe more) have frozen string errors …

• Why frozen string literals?

[tt_su]

@MSP_Greg

You might check that. I think I did, but I don’t recall. Even with the Content-Length header present, I don’t think it works correctly. It is present with the GitHub downloads. Also, I think the download used in the example does not have a redirect, but the larger one (that’s commented out) does. That may also affect it, not sure.

No, the Content-Length header is missing from GitHub;

When making a request to another server that do provide it, like BitBucket, the you see the Content-Length header and Sketchup::Http will provide a positive total_size parameter.

Ref: Sketchup::Http::Request.set_download_progress_callback pending issue · Issue #463 · SketchUp/api-issue-tracker · GitHub

For comparison, here’s the output I see from the GitHub download link:

Http::Request(#set_download_progress_callback block)
  Download Pending: 2048 of -1
  Download Pending: 6128 of -1
  Download Pending: 10206 of -1
  [...]
  Download Pending: 319872 of -1
  Download Pending: 323959 of -1
  Download Pending: 327444 of -1
Http::Request(#start callback block)
  Response body size is: 327444
  File Download is Complete.

And here is a download link from BitBucket:

Http::Request(#set_download_progress_callback block)
  Download Pending: 4096 of 1294654
  Download Pending: 8192 of 1294654
  Download Pending: 12288 of 1294654
  [...]
  Download Pending: 1285652 of 1294654
  Download Pending: 1289748 of 1294654
  Download Pending: 1293844 of 1294654
  Downloaded Size Equals Total Size.
  Download Pending: 1294654 of 1294654
Http::Request(#start callback block)
  Response body size is: 1294654
  File Download is Complete.

Btw, when working with HTTP traffic, using a tool like PostMan is useful to easily see what is being transmitted from the server, as well as making quick experiments to the requests. Also Fiddler is really useful when you want to see what other clients (software) are doing.

[MSP_Greg]

JFYI, if Transfer-Encoding header exists, I believe the Content-Length header should be ignored…