How do I get rid of a little rectangle at beginning and end of every line I draw?

If by “Outliner” you mean the heavier dark lines around the model, you can turn off Profile just above Endpoints… if you mean the Profiler window, just click the X to close as @slbaumgartner said.