Cannot upload .skp file to DB via HTTP POST request - Ruby

Developers
, , , ,
1

I am trying to upload a .skp file to a DB using a HTTP POST request. Since the request includes a query parameter, the body has to be of multipart/form-data type. The request also includes a header with some authorization info.

file = ‘test.skp’
url = ‘https://my_files_host’
auth_token = ‘my_authorization_token’
folder_id = ‘my_folder_id’

Since .skp is a non-standard MIME Type, the Content-Type value was previoulsy unkwown, and it was treated as application/octet-stream. However, checking this link, we see that now we can treat it as application/x-koan. Actually this can solve the question raised here.

I have tried three different solutions in Ruby code:

headers = { Authorization: auth_token, params: { folderId: folder_id } }
## query params are taken out of the headers hash according to rest-client docs
body = { filename: File.new(file, 'rb') }
response = RestClient.post(url_auth, body, headers)

Here the response obtained is nil. The problem is the .skp IO stream is not being read correctly, probably it is not possible to upload .skp files with this solution. It does work with a standard MIME type file, like a .txt.

uri = URI.parse(url)
File.open(file) do |skp|
	request = Net::HTTP::Post::Multipart.new(uri.path, { 'filename' => UploadIO.new(skp, 'application/x-koan', file), 'folderId' => folder_id})
	## Also tried in the line above with 'application/octet-stream' and MIME::Types.type_for(file). Does not work either
	request.add_field('Authorization', auth_token)
	http = Net::HTTP.new(uri.host, uri.port)
	http.use_ssl = true
	response = http.request(request)
end

Here I obtain a Bad Request response, with message: “The ‘folderId’ parameter is missing”. According to the doc, I am passing the parameter correctly. Maybe I am missing something.

BOUNDARY = "AaB03x"

headers = { 'Authorization' => auth_token, 'Content-Type' => "multipart/form-data; boundary=#{BOUNDARY}" }
params = { 'folderId' => folder_id }

body = []
body << "--#{BOUNDARY}\r\n"
body << "Content-Disposition: form-data; name=\"filename\"; filename=\"#{File.basename(file)}\"\r\n"
body << "Content-Type: #{MIME::Types.type_for(file)}\r\n\r\n"
body << File.read(file)
body << "--#{BOUNDARY}\r\n"
body << "Content-Disposition: form-data; name=\"folderId\"\r\n\r\n"
body << params.to_json
body << "\r\n\r\n--#{BOUNDARY}--\r\n"

http = Net::HTTP.new(uri.host, uri.port)
http.use_ssl = true
request = Net::HTTP::Post.new(uri.request_uri, headers)
request.body = body.join
response = http.request(request)

Here I obtain a Bad Request response as well, with the same message than the previous case. I have observed tha the IO stream of the .skp file is not read correctly again, so when the body is joint, the query parameter is missing.

2

Have you tried comparing a working POST request with what you are crafting via Ruby?

You could use POSTman (https://www.getpostman.com/) to play around with the requests and then inspect the data being sent compared to what is being sent from your Ruby script.

3

@tt_su I have tested the request in POSTMAN and it’s working correctly. The environment parameters are set as in the Ruby script, that is why I don’t understand the bad request.

My only guess is that the reading of the .skp file is not being performed correctly in Ruby, probably the UploadIO class is not valid for .skp files. In POSTMAN you can directly upload the file from a dialog, so probably they use another uploading method that works.

4

Beware of NULL characters in binary strings. Also, ensure you are using ASCII-8BIT encoding when reading binary files with Ruby.

Have you compared the actual HTTP data? There must be some difference there. Maybe some missing headers.

5

@tt_su I have double checked that I’m using ASCII-8BIT encoding, and precisely it is here where the problem is. I have found it debugging, but I don’t know how to fix it.

Both REST Client and multipart-post use the net/http library so I am going to focus in the last solution. When I create the request body StringIO:

body = []
body << "--#{BOUNDARY}\r\n"
body << "Content-Disposition: form-data; name=\"filename\"; filename=\"#{File.basename(file)}\"\r\n"
body << "Content-Type: #{MIME::Types.type_for(file)}\r\n\r\n"
body << File.binread(file)
body << "--#{BOUNDARY}\r\n"
body << "Content-Disposition: form-data; name=\"folderId\"\r\n\r\n"
body << params.to_json
body << "\r\n\r\n--#{BOUNDARY}--\r\n"

The body is something like:

body = [“–AaB03x\r\n”, “Content-Disposition: form-data; name="filename"; filename="test.skp"\r\n”, “Content-Type: application/x-koan\r\n\r\n”, “\xFF\xFE\xFF…(continues)”, “–AaB03x\r\n”, “Content-Disposition: form-data; name="folderId"\r\n\r\n”, “{"folderId":"folder_id"}”, “\r\n\r\n–AaB03x–\r\n”]

which is correct. But when I join the body to convert it from array to string:

body_join = body.join
body_join = “–AaB03x Content-Disposition: form-data; name=“filename”; filename=“test.skp” Content-Type: application/x-koan ���S”

It turns back to unicode and everything after the file info is missing, that is the reason of the Bad Request missing folderId. In net/http, the request body has to be a string (there is a bytesize error if you pass an array). I tried also streaming first the folderId parameter and then the file, but still the last BOUNDARY is missing, and you get the same error.

The question is, how to add info to a string after a UTF-8 encoding, or how to keep the file stream in ASCII-8BIT when joining the body, in order to keep appending info after it?

6

Yea, the default encoding for Ruby strings in SketchUp will be UTF-8. So you need to be careful when concatenating strings. I’m guessing that you have a binary text string which you are injecting into a UTF-8 and get a UTF-8 string back. That array of strings you have, try iterating over it and ensuring it’s encoding is ASCII-8BIT. From the example posted it looks like you could use .force_encoding. But if you actually might have UTF-8 characters in your string array you might want to transpose with .encode!.

body.each { |item| item.force_encoding("ASCII-8BIT") }

7

@tt_su That does not work for me either. Can’t find a way to concatenate .skp file info with a text string, even with ASCII-8BIT encoding:

file_path = "test.skp"
f = File.binread(file_path)     ## f: "���S"
enc = f.encoding     ## enc: ASCII-8BIT
f.force_encoding('ASCII-8BIT')     ## even when the encoding is ASCII-8BIT I force it
test_string = "AAA" + f + "BBB"     ## test_string: "AAA���S"

The “BBB” string is not concatenated. The result is I cannot concatenate text after the .skp info, but there is no problem in doing it before. I have tried with different .skp files and different ways to read the .skp file info like:

  f2 = File.open(file_path, 'rb'){|f| f.read }
  f3 = []
  f3 << File.binread(file_path)

And the result is the same. That is why the ‘folderId’ parameter is missing in the HTTP POST request.

8 
test_string = "AAA" + f + "BBB"

You can’t do that because the "AAA" string and the f string are not the same encoding, and the String#+ method will just assume all strings are using the default script encoding, (which is UTF-8, unless you’ve changed it with a “magic comment” at the beginning of your script file.)

So, you’d need to do something like:

test_string = "AAA".encode('ASCII-8BIT') + f + "BBB".encode('ASCII-8BIT')

…or… :

test_string =( "AAA" + f.encode('UTF-8') + "BBB" ).encode('ASCII-8BIT')

But if you want automatic transcoding to and from UTF-8, for external encoded files, …

Class: Encoding (Ruby 2.2.0)

To process the data of an IO object which has an encoding different from its external encoding, you can set its internal encoding. Ruby will use this internal encoding to transcode the data when it is read from the IO object.

Conversely, when data is written to the IO object it is transcoded from the internal encoding to the external encoding of the IO object.

The internal encoding of an IO object can be set with IO#set_encoding or at IO object creation
(see IO::new options).

File::open is a form of File::new, which is inherited from IO::new (it’s superclass.)

Hence, the blurb in the docs for these methods:

See IO.new for a description of the mode and opt parameters.

…, then you might be able to do something like:

f2 = File.open(file_path, 'rb:ASCII-8BIT:UTF-8'){|f| f.read }
1 Like
9

As Dan described, string literals will be marked with UTF-8 encoding. So when you append an ASCII-8BIT to a UTF-8 string you will get a UTF-8 string back. Make sure all strings you use to compose your HTTP request are ASCII-8BIT before concatenating them.

1 Like
10

@DanRathbun It is not working even with same encoding:

file_path = "test.skp"
f = File.binread(file_path)     ## f: "���S"
enc = f.encoding     ## enc: ASCII-8BIT
test_string = "AAA".encode('ASCII-8BIT') + f + "BBB".encode('ASCII-8BIT')     ## test_string: "AAA���S"
test_string2 = ("AAA" + f.encode('UTF-8') + "BBB").encode('ASCII-8BIT')     ## Code crashes

And trying the automatic transcoding:

f2 = File.open(file_path, 'rb:ASCII-8BIT:UTF-8'){|f| f.read }     ## f2: "���S"
enc = f2.encoding     ## enc: ASCII-8BIT (Transcoding not working)
f2.force_encoding('UTF-8')     ## forcing to UTF-8
enc = f2.encoding     ## enc: UTF-8 (It works)
test_string = "AAA"+ f2 + "BBB"     ## test_string: "AAA���S" (UTF-8)
test_string2 = "AAA".encode('ASCII-8BIT') + f2.encode('ASCII-8BIT') + "BBB".encode('ASCII-8BIT')     ## test_string2: "AAA���S" (ASCII-8BIT)

provides the same result, the “BBB” string is never concatenated.

The question is why you can concatenate a string before the file info, but not after, even using same encoding. I am wondering if it is correct to binary read a .skp file with f = File.binread(file_path). The only thing I’m sure, as @tt_su said, it is that all the strings have to be ASCII-8BIT in order to HTTP POST them.

11

Quick side question, what SketchUp versions are you planning to support?
If you are ok with SU2017 and up you could try Sketchup::Http as an alternative: Module: Sketchup::Http — SketchUp Ruby API Documentation

(It’s async non-blocking.)

1 Like
12

Well, my idea was to support from SU2016 onwards, but I think I could be fine with >=SU2017.

Didn’t know about that new Sketchup::Http module. I saw you opened an issue to add a tutorial for it here, which I think it’s a great idea, at least for .skp file POSTing, since there is no doc about it. Apart from that, I only see this thread regarding the module.

I’ll give it a try and hope it solves my problem. Let you know about it.

13

@tt_su I have tried the Sketchup::Http module and it’s incomplete so far. It does not allow updating files in a form-data fashion. According to my Postman environment (that works correctly):

image
image713×400 20.7 KB

The file has to be included in the body, which in Sketchup::Http::Request has to be added as a string. Besides, there is no method to include query parameters, so what I am doing is to encode the folderId param as a www_form and append it to the url. The only straightforward thing is adding headers, since it already includes a method. My code is the following:

query = URI.encode_www_form(params)     # query = "folderId=dPzrTsBAtJQ"
url_query = url + '?' + query     # url_query = https://my_files_host?folderId="dPzrTsBAtJQ"

request = Sketchup::Http::Request.new( url_query, Sketchup::Http::POST )
request.headers = headers
request.body = File.binread(file_path)     # Try to include the file as a binary string

request.start do |req, res|
  # Not entering
end

The problem is you can only add strings to the request body, and I need to add a file (test.skp) associated to a parameter (filename).

14

Both files and strings are represented as a string of bytes.

Can you provide a complete code snippet of a code example? It’s hard to tell what’s going on without seeing exactly the HTTP request with headers and everything.

15

I tried to extract some of the code we use in Trimble Connect to upload SKP files to the Trimble Connect service:

3 Likes
16

@tt_su Is that code working for you? It never gets into the request.start block, right? Actually it looks pretty similar to my net/http solution, but just using Sketchup::Http instead. I know it’s incomplete, because you need to add the query parameters to the payload, and also the Authorization token to the header.

In any case, I have tried your approach and I am having the same issue, nothing changed. This is a snippet from my code:

 # I try with Chris 
model_file = 'Chris.skp'
filename = File.basename(model_file)

BOUNDARY = "----RubyMultipartClient#{rand(1000000)}ZZZZZ"

 # The folderId param in the API contains the ID of the folder for the upload
param = { folderId: dPzrTsBAtJQ }

data = []
data << "Content-Type: multipart/form-data; boundary=#{BOUNDARY}\r\n\r\n"
data << "--#{BOUNDARY}\r\n"
 # The field in the API to upload the file is called filename
data << "Content-Disposition: form-data; name=\"filename\"; filename=\"#{filename}\"\r\n\r\n"     

File.open(model_file, "rb:BINARY") { |file|
   contents = file.read
   data << contents
}

data << "--#{BOUNDARY}\r\n"
 # The folderId param is added to the payload (see my POSTMAN working solution)
data << "Content-Disposition: form-data; name=\"folderId\"\r\n\r\n"
data << param.to_json

data << "\r\n\r\n--#{BOUNDARY}--\r\n"

data.each { |line|
   line.force_encoding(Encoding::BINARY)
}

payload = data.join

 # The field Authorization in the header contains the authorization token for the request
headers = { 'Authorization' => auth_token,
           'Content-Type' => "multipart/form-data; boundary=#{BOUNDARY}" }

 # connect_host is the host of my DB
url = 'https://' + connect_host + '/files'

request = Sketchup::Http::Request.new(url, Sketchup::Http::POST)
request.headers = headers
request.body = payload

request.start do |req, res|
   # Not entering
   p res
   p res.status_code
   puts res.body
end

As I said, the problem is the same. When you run the line payload = data.join, everything in data after the reading of the file:

File.open(model_file, "rb:BINARY") { |file|
   contents = file.read
   data << contents
}

is missing in payload.

Maybe you can give it a try changing the parameters for the ones you use in Trimble.

17

Yes, that gist is working by copy+paste for me:

image
image694×135 3.13 KB

How are you checking that? By what you see in the Console? The console will not be able to render everything - running into null characteers might terminate the rendering.
You’d have to inspect the bytes of the string using .bytes to verify.

18

Finally I could make it work! :slightly_smiling_face:

The solution is using your data string in the body but then including the folderId parameter directly in the query of the URL when doing the request:

url = url + '?' + URI.encode_www_form(param)
request = Sketchup::Http::Request.new( url, Sketchup::Http::POST )

I had tried this solution before (using my data string) with the other three methods that I implemented but never worked, happily with Sketchup::Http it does work. I guess my data string didn’t work before because I forgot to add the following line:

data << "Content-Type: multipart/form-data; boundary=#{BOUNDARY}\r\n\r\n"

On the other hand, using your data string but including the folderId parameter in the body together with the file as multipart/form-data type gives me the same issue than before. The strings are fine compared with .bytes, but then the message in the response body is always the same:

"{"message":"Missing required parameter : folderId","errorcode":"MISSING_REQUIRED_PARAMS"}"

In any case, now I know I can use the Http method of the API for my purposes. Thank you very much for your help.

2 Likes
19

This topic was automatically closed 91 days after the last reply. New replies are no longer allowed.