Remove train/val/test hickle generation, X_*.hkl to tfrecords directly

Since compress to train/val/test will causes memory issue. We remove this process, and generate rfrecords from X_*.hkl directly.

  • Optimise the code and parallel preprocessing for hickle file
  • statistics.json has to re-relocate to upper level
  • testing
Edited by Ghost User