How to control data after mining?

Hi. after accessing data such as mining objects dimension, is it better save it then evaluate some code for it? or do immediately? Due to the crashing weak system and memory.tank you

It’s probably better to investigate and understand why you would need this.

  • Is the crash due to bugs in the code? Then debug the code with faster feed back cycles (don’t run with all the big data, but a minimal subset as a test case so that you can quickly reach the step where the error happens).
  • Is the crash due to resource limits (CPU/RAM)? Maybe you aren’t using the right algorithms or data structures. Check for nested loops and whether they can be simplified or replaced by search datastructures (hash, indices, spatial trees). If necessary, keep data on disk and only load those chunks into memory on which you are working on. Check that you don’t have memory leaks (like adding objects to a static hash/array, but never removing them).

If you really think about saving interim results immediately to disk (as backup), it will only be valuable to you if you can also restart again from the interim results, skipping the previous steps.

1 Like

Do you have some sample code that illustrates what you’re working with?