Need help to debug on MAC CSV.parse, work perfect on windows

simjoubert · November 18, 2024, 2:41pm

Hello, I wrote a plugin for a client who is unfortunately on MAC. And I don’t have one on hand to conduct the tests and understand where the incompatibility is.

My plugin allows you to model an item from its product range by taking into account a lot of options.

The plugin has, as a resource, 2 CSV files. The first, lists the RAL colors with their Hexadecimal equivalent. The second, lists the custom materials, with the parameters if it is a color or a textured material, the name of the thumbnail file and the name of the .skp material.

I have an observer who rewrites the HTML file of the dialog box from an .ERB file in order to update the list of available materials with a preview thumbnail.

The plugin is fully functional on Windows, the client and I work on Sketchup 2024.

On MAC the console displays the following error:

Error:#<CSV::MalformedCSVError: Any value after quoted field isn't allowed in line 43.>

when it executes the following method.

# Rewrite the Dialog HTML file 
def self.rewrite_dialogue_html_file()
  csv_materials()
  erb_file = File.join(DIRECTORY[:html], 'dialog.erb')
   renderer = ERB.new(File.read(erb_file))
  html_file = File.join(DIRECTORY[:html], 'dialog.html')      
   File.write(html_file, renderer. Result(binding))
  puts "Freetness FTMS Rewrite HTML Dialog"
end

# Rewrite the materials list from the CSV files
def self.csv_materials()
  csv_ral = File.join(DIRECTORY[:csv],"FTMS_RAL.csv")
  csv_mat = File.join(DIRECTORY[:csv],"FTMS_MATERIAL.csv")
  SESSION[:materials][:ral] = CSV.parse(File.read(csv_ral, liberal_parsing: true, headers: :first_row))[1..-1]
  SESSION[:materials][:materials] = CSV.parse(File.read(csv_mat, liberal_parsing: true, headers: :first_row))[1..-1]
  puts "Freetness FTMS Rewrite CSV Materials"
end

Line 43 is the end of the csv file FTMS_MATERIALS.csv

Anyone who is very happy, would you have suggestions and explanations on this difference in treatment between the two platforms?

I added the reading parameter liberal_parsing: true
Thanks
Simon

The FTMS_MATERIALS.csv

"name","type","thumbnail","r","g","b","a","skp","height","width"

"Anneau Filet Basket","color","","255","127","0","255","Anneau Filet Basket.skm","0","0"

"Cordage","texture","Cordage.jpg","","","","","Cordage.skm","33.1216","24.8158"

"FREETNESS NOIR","color","","27","29","27","255","FREETNESS NOIR.skm","0","0"

"FREETNESS VERT CITRON","color","","170","255","47","255","FREETNESS VERT CITRON.skm","0","0"

"FTMS_FilletBasket","texture","FTMS_FilletBasket.jpg","","","","","FTMS_FilletBasket.skm","33.1216","24.8158"

"FTMS_Marquage_Blanc","color","","255","255","255","255","FTMS_Marquage_Blanc.skm","0","0"

"FTMS_Marquage_Bleu","color","","50","131","194","255","FTMS_Marquage_Bleu.skm","0","0"

"FTMS_Marquage_Jaune","color","","255","255","0","255","FTMS_Marquage_Jaune.skm","0","0"

"FTMS_Marquage_Orange","color","","229","131","16","255","FTMS_Marquage_Orange.skm","0","0"

"FTMS_Marquage_Rose","color","","229","131","194","255","FTMS_Marquage_Rose.skm","0","0"

"FTMS_Marquage_Rouge","color","","255","0","0","255","FTMS_Marquage_Rouge.skm","0","0"

"FTMS_PanneauInfos","texture","FTMS_PanneauInfos.jpg","","","","","FTMS_PanneauInfos.skm","18.1356","15.24"

"FTMS_Plastique_Noir","color","","0","0","0","255","FTMS_Plastique_Noir.skm","0","0"

"FTM_PlotBeton","texture","FTM_PlotBeton.png","","","","","FTM_PlotBeton.skm","20.32","20.32"

"Filet 10x10 noir","texture","Filet 10x10 noir.png","","","","","Filet 10x10 noir.skm","1.651","1.651"

"Filet 10x10 vert","texture","Filet 10x10 vert.png","","","","","Filet 10x10 vert.skm","1.651","1.651"

"Filet 10x10","texture","Filet 10x10.png","","","","","Filet 10x10.skm","1.651","1.651"

"Filet 20x20 noir","texture","Filet 20x20 noir.png","","","","","Filet 20x20 noir.skm","1.651","1.651"

"Filet 20x20 vert","texture","Filet 20x20 vert.png","","","","","Filet 20x20 vert.skm","1.651","1.651"

"Filet 20x20","texture","Filet 20x20.png","","","","","Filet 20x20.skm","1.651","1.651"

"Filet 5x5 noir","texture","Filet 5x5 noir.png","","","","","Filet 5x5 noir.skm","1.651","1.651"

"Filet 5x5 vert","texture","Filet 5x5 vert.png","","","","","Filet 5x5 vert.skm","1.651","1.651"

"Filet 5x5","texture","Filet 5x5.png","","","","","Filet 5x5.skm","1.651","1.651"

"FiletCordage","texture","FiletCordage.jpg","","","","","FiletCordage.skm","33.1216","24.8158"

"Galva Brut","texture","Galva Brut.jpg","","","","","Galva Brut.skm","9.652","15.7226"

"Gazon Coloré","texture","Gazon Coloré.jpg","","","","","Gazon Coloré.skm","15.24","15.24"

"Gazon","texture","Gazon.jpg","","","","","Gazon.skm","15.24","15.24"

"Gazon_Bande","texture","Gazon_Bande.jpg","","","","","Gazon_Bande.skm","15.24","15.24"

"Green","color","","0","128","0","255","Green.skm","0","0"

"Logo FREETNESS","texture","Logo FREETNESS.png","","","","","Logo FREETNESS.skm","24.104599999999998","23.9776"

"Logo Freetness 2","texture","Logo Freetness 2.png","","","","","Logo Freetness 2.skm","8.9916","41.6814"

"Red","color","","255","0","0","255","Red.skm","0","0"

"Sol Asphalte neuf","texture","Sol Asphalte neuf.jpg","","","","","Sol Asphalte neuf.skm","6.5024","6.5024"

"Sol Bitume Ancien","texture","Sol Bitume Ancien.jpg","","","","","Sol Bitume Ancien.skm","12.7","12.7"

"Sol Granit","texture","Sol Granit.jpg","","","","","Sol Granit.skm","13.0048","13.0048"

"Sol Marbre Brun Clair","texture","Sol Marbre Brun Clair.png","","","","","Sol Marbre Brun Clair.skm","15.24","15.24"

"Sol Marbre Brun","texture","Sol Marbre Brun.png","","","","","Sol Marbre Brun.skm","15.24","15.24"

"Sol Marbre Gris Clair","texture","Sol Marbre Gris Clair.jpg","","","","","Sol Marbre Gris Clair.skm","13.0048","13.0048"

"Sol Marbre Gris Fonce","texture","Sol Marbre Gris Fonce.jpg","","","","","Sol Marbre Gris Fonce.skm","13.0048","13.0048"

"Sol Marbre Gris","texture","Sol Marbre Gris.png","","","","","Sol Marbre Gris.skm","15.24","15.24"

"Sol Quartz marron clair","texture","Sol Quartz marron clair.jpg","","","","","Sol Quartz marron clair.skm","13.9954","12.7"

sWilliams · November 18, 2024, 7:10pm

You’re passing liberal_parsing: and headers: as an option to File.open() instead of CSV.parse()

Another potential problem is that every other line in the CSV file is blank, and the last line in the file is not terminated with a carriage return or a line feed.

Hopefully one of these ideas will fix the problem.

original:
SESSION[:materials][:ral] = CSV.parse(File.read(csv_ral, liberal_parsing: true, headers: :first_row))[1..-1]

corrected:
SESSION[:materials][:ral] = CSV.parse(File.read(csv_ral), liberal_parsing: true, headers: :first_row)[1..-1]

simjoubert · November 18, 2024, 7:32pm

Thanks,
I will apply this fix and pass it on to the client. Because I don’t have any mac to test it on.
It’s really annoying, this difference in interpretation of identical data.
I hope is will be enough to patch the code.
It will be useful to know all the different between the two platform !

sWilliams · November 18, 2024, 8:18pm

The error message is originating at this line in the CSV parser code if you need to do further debugging.

github.com

ruby/ruby/blob/ruby_3_2/lib/csv/parser.rb#L1064


      
            quoted_fields << @quoted_column_value
            emit_row(row, quoted_fields, &block)
            break
          else
            if @quoted_column_value
              if liberal_parsing? and (new_line = @scanner.check(@line_end))
                message =
                  "Illegal end-of-line sequence outside of a quoted field " +
                  "<#{new_line.inspect}>"
              else
                message = "Any value after quoted field isn't allowed"
              end
              ignore_broken_line
              raise MalformedCSVError.new(message, @lineno)
            elsif @unquoted_column_value and
                  (new_line = @scanner.scan(@line_end))
              ignore_broken_line
              message = "Unquoted fields do not allow new line " +
                        "<#{new_line.inspect}>"
              raise MalformedCSVError.new(message, @lineno)
            elsif @scanner.rest.start_with?(@quote_character)

simjoubert · November 18, 2024, 9:34pm

I wrote a method to add the current color to the end of the CSV FTMS_MATERIAL.csv
maybe the problem start in this method :

# Add the current material to the material list
def self.add_CurrentMaterial()
  csv_mat = File.join(DIRECTORY[:csv],"FTMS_MATERIAL.csv")
  materials_folder = DIRECTORY[:materials]
  thumbnail_folder = File.join(DIRECTORY[:html],"thumbnail")

  model = Sketchup.active_model
  materials = model.materials   
  mat = materials.current

  UI.messagebox("Aucune matière n'est sélectionnée !", MB_OK) unless mat
  raise  "Error No current material" unless mat
  
  mat_name = mat.name

  q1 = UI.messagebox("Souhaitez vous ajouter la matière nommée #{mat_name} ?", MB_YESNO)
  raise "Abandon" if q1 == IDNO
  
  mat_file_basename = mat_name + ".skm"
  
  # Test type of material
  if mat.materialType == 0

    r,g,b,a = mat.color.to_a 
    mat_color = "rgba(#{r},#{g},#{b},#{(a.to_f / 255.0).round(2)})"
    mat_type = "color"
    thumbnail_file = ""
    mat_height = 0
    mat_width = 0

  else

    mat_type = "texture"
    mat_color = ""
    texture = mat.texture
    mat_height = texture.image_height.to_m
    mat_width = texture.image_width.to_m
    basename = File.basename(mat.texture.filename)
    extension = basename.split(".")[1]
    thumbnail_file = mat_name +"."+ extension  
    path = File.join(thumbnail_folder,thumbnail_file)

    #Save the texture colorized in the thumbnail folder
    status = mat.texture.write(path,true)

   end
  
  #Write a new row
   new_csv_row = [mat_name, mat_type, thumbnail_file, r, g, b, a, mat_file_basename, mat_height, mat_width]
   
  #Save the SKP in the materials folder
   mat.save_as(File.join(materials_folder, mat_file_basename))
  
  #Write the line at the end of the CSV FTMS_MATERIALS.csv
   CSV.open(csv_mat,'ab',
    :quote_char=>'"',
    :col_sep =>",", 
    :force_quotes => true) do |csv|

       csv << new_csv_row

   end
   # Rewrite the Dialog Html file from the ERB file template
   Dialog.rewrite_dialogue_html_file()

   return "Success"

end

simjoubert · November 19, 2024, 11:47am

Thank sWilliams
I have a bug in the end line for the new row
my CSV wrote in windows use ’ \r\n’ and MAC by default ‘\n’
I force the row_sep as ‘\r\n’

#Prepare the new row to append at the end of the csv
         new_csv_row = [mat_name, mat_type, thumbnail_file, r, g, b, a, mat_file_basename, mat_height, mat_width]
         
         #Write the new row
         CSV.open(csv_mat,'ab',
         :headers =>:first_row,
         :quote_char=>'"',
         :col_sep =>",", 
         :row_sep =>"\r\n",
         :force_quotes => true,
         :liberal_parsing => true
         ) do |csv|
           csv << new_csv_row
         end

Topic		Replies	Views
Error when using require "csv" in SketchUp 2024 Ruby API	8	230	May 12, 2024
CSV file open on mac SketchUp mac , csv	17	1321	April 10, 2021
Looking for a simple plugin developed Commercial and Collaborative Work	23	2285	September 1, 2018
Export component attribute list in csv Ruby API attribute , export , csv	0	1106	April 26, 2021
Ruby script to extract instance name to csv Ruby API advice , ruby , export , components	7	989	November 1, 2022

Need help to debug on MAC CSV.parse, work perfect on windows

Related topics