SketchUp & AI - StableDiffusion, Automatic1111, Controlnet and segmentation colors

Off Topic
6

Hi @petitclercj The steps are fairly easy;

  1. In SketchUp assign specific colors to all the faces. The RGB values for these segmentation colors can be found at GoogleDocs for instance. Each color is a category (wall, grass, street, etc).
  2. Export an image from SketchUp while shadows are disabled. You will get an image with weird colors like in the pic below.
  3. Download the segmentation model file from Huggingface
  4. Open your StableDiffusion app (Automatic1111 / InvokeAI / ComfyUI). They all can work with controlnet as long as you don’t use the SDXL model (at this time).
  5. Enable controlnet, open the image in the controlnet-section. In controlnet, keep the preprocessor at ‘none’ because you already have an image with segmentation colors but set the model to ‘seg…’
  6. In the prompt describe your image. SD will generate a new image using the seg-colored image from SketchUp and the corresponding categories from the seg-colors-table and your prompt.

You can also have controlnet generate the seg colors from a regular colored image using a seg-model in the preprocessor but I prefer to create it myself to have total control.

See this nice explanation at Reddit.

segControlnet
segControlnet821×1055 121 KB