Skip to content
Snippets Groups Projects
Commit e302a6ee authored by Amirpasha Mozaffari's avatar Amirpasha Mozaffari
Browse files

removing Data concurrency test

parent 72da2fc4
No related branches found
No related tags found
No related merge requests found
Show whitespace changes
Inline Side-by-side
DataExtraction/concurrency_test/distribution_job_100.log deleted 100644 → 0
+ 0
− 90
View file @ 72da2fc4
2020-06-03 16:20:09,092:INFO:The destination exist
2020-06-03 16:20:09,092:CRITICAL:Destination : /home/a.mozaffari/data_dest/ exist -> Remove and Re-Cereate
2020-06-03 16:20:09,092:INFO:The source path is : //home/a.mozaffari/data_era5/2017/
2020-06-03 16:20:09,092:INFO:The destination path is : /home/a.mozaffari/data_dest/
2020-06-03 16:20:09,092:INFO:==== Directory scanner : start ====
2020-06-03 16:20:09,105:INFO:=== Directory Scanner output ===
2020-06-03 16:20:09,106:INFO:Total size of the source directory is:18.975132Gb.
2020-06-03 16:20:09,106:INFO:Total number of the files in the source directory is: 95
2020-06-03 16:20:09,106:INFO:Total number of the directories in the source directory is: 3
2020-06-03 16:20:09,106:INFO:==== Directory scanner : end ====
2020-06-03 16:20:09,106:INFO:==== Data Structure Builder : start ====
2020-06-03 16:20:09,106:INFO:The destination path exists
2020-06-03 16:20:09,106:INFO:Load Level = 0 : Data Sctructure will be build
2020-06-03 16:20:09,112:INFO:01 is created
2020-06-03 16:20:09,117:INFO:02 is created
2020-06-03 16:20:09,124:INFO:03 is created
2020-06-03 16:20:09,125:INFO:==== Data Structure Builder : end ====
2020-06-03 16:20:09,125:INFO:MA0: Next to be processed is 01 loacted in //home/a.mozaffari/data_era5/2017/01
2020-06-03 16:20:09,126:INFO:=== Directory Scanner output ===
2020-06-03 16:20:09,126:INFO:Total size of the source directory is:5522.964184Gb.
2020-06-03 16:20:09,126:INFO:Total number of the files in the source directory is: 27
2020-06-03 16:20:09,126:INFO:Total number of the directories in the source directory is: 0
2020-06-03 16:20:09,126:INFO:The follwoing is in the load Balancer
2020-06-03 16:20:09,126:INFO:{1: None, 2: None, 3: None, 4: None, 5: None, 6: None, 7: None, 8: None, 9: None, 10: None, 11: None, 12: None}
2020-06-03 16:20:09,126:INFO:['ecmwf_era5_17010115.nc', 'ecmwf_era5_17010204.nc', 'ecmwf_era5_17010121.nc', 'ecmwf_era5_17010117.nc', 'ecmwf_era5_17010113.nc', 'ecmwf_era5_17010201.nc', 'ecmwf_era5_17010120.nc', 'ecmwf_era5_17010205.nc', 'ecmwf_era5_17010118.nc', 'ecmwf_era5_17010106.nc', 'ecmwf_era5_17010103.nc', 'ecmwf_era5_17010119.nc', 'ecmwf_era5_17010116.nc', 'ecmwf_era5_17010122.nc', 'ecmwf_era5_17010105.nc', 'ecmwf_era5_17010110.nc', 'ecmwf_era5_17010112.nc', 'ecmwf_era5_17010203.nc', 'ecmwf_era5_17010114.nc', 'ecmwf_era5_17010108.nc', 'ecmwf_era5_17010100.nc', 'ecmwf_era5_17010102.nc', 'ecmwf_era5_17010111.nc', 'ecmwf_era5_17010123.nc', 'ecmwf_era5_17010104.nc', 'ecmwf_era5_17010101.nc', 'ecmwf_era5_17010202.nc']
2020-06-03 16:20:09,126:INFO:0
2020-06-03 16:20:09,127:INFO:27
2020-06-03 16:20:09,127:INFO:{1: 'ecmwf_era5_17010115.nc;ecmwf_era5_17010116.nc;ecmwf_era5_17010104.nc', 2: 'ecmwf_era5_17010204.nc;ecmwf_era5_17010122.nc;ecmwf_era5_17010101.nc', 3: 'ecmwf_era5_17010121.nc;ecmwf_era5_17010105.nc;ecmwf_era5_17010202.nc', 4: 'ecmwf_era5_17010117.nc;ecmwf_era5_17010110.nc', 5: 'ecmwf_era5_17010113.nc;ecmwf_era5_17010112.nc', 6: 'ecmwf_era5_17010201.nc;ecmwf_era5_17010203.nc', 7: 'ecmwf_era5_17010120.nc;ecmwf_era5_17010114.nc', 8: 'ecmwf_era5_17010205.nc;ecmwf_era5_17010108.nc', 9: 'ecmwf_era5_17010118.nc;ecmwf_era5_17010100.nc', 10: 'ecmwf_era5_17010106.nc;ecmwf_era5_17010102.nc', 11: 'ecmwf_era5_17010103.nc;ecmwf_era5_17010111.nc', 12: 'ecmwf_era5_17010119.nc;ecmwf_era5_17010123.nc'}
2020-06-03 16:20:09,127:INFO:{1: 'ecmwf_era5_17010115.nc;ecmwf_era5_17010116.nc;ecmwf_era5_17010104.nc', 2: 'ecmwf_era5_17010204.nc;ecmwf_era5_17010122.nc;ecmwf_era5_17010101.nc', 3: 'ecmwf_era5_17010121.nc;ecmwf_era5_17010105.nc;ecmwf_era5_17010202.nc', 4: 'ecmwf_era5_17010117.nc;ecmwf_era5_17010110.nc', 5: 'ecmwf_era5_17010113.nc;ecmwf_era5_17010112.nc', 6: 'ecmwf_era5_17010201.nc;ecmwf_era5_17010203.nc', 7: 'ecmwf_era5_17010120.nc;ecmwf_era5_17010114.nc', 8: 'ecmwf_era5_17010205.nc;ecmwf_era5_17010108.nc', 9: 'ecmwf_era5_17010118.nc;ecmwf_era5_17010100.nc', 10: 'ecmwf_era5_17010106.nc;ecmwf_era5_17010102.nc', 11: 'ecmwf_era5_17010103.nc;ecmwf_era5_17010111.nc', 12: 'ecmwf_era5_17010119.nc;ecmwf_era5_17010123.nc'}
2020-06-03 16:20:09,127:INFO:MA0: Next to be processed is 02 loacted in //home/a.mozaffari/data_era5/2017/02
2020-06-03 16:20:09,127:INFO:=== Directory Scanner output ===
2020-06-03 16:20:09,128:INFO:Total size of the source directory is:6953.662487Gb.
2020-06-03 16:20:09,128:INFO:Total number of the files in the source directory is: 34
2020-06-03 16:20:09,128:INFO:Total number of the directories in the source directory is: 0
2020-06-03 16:20:09,128:INFO:The follwoing is in the load Balancer
2020-06-03 16:20:09,128:INFO:{1: None, 2: None, 3: None, 4: None, 5: None, 6: None, 7: None, 8: None, 9: None, 10: None, 11: None, 12: None}
2020-06-03 16:20:09,128:INFO:['ecmwf_era5_17020206.nc', 'ecmwf_era5_17020201.nc', 'ecmwf_era5_17020111.nc', 'ecmwf_era5_17020114.nc', 'ecmwf_era5_17020108.nc', 'ecmwf_era5_17020104.nc', 'ecmwf_era5_17020200.nc', 'ecmwf_era5_17020123.nc', 'ecmwf_era5_17020105.nc', 'ecmwf_era5_17020116.nc', 'ecmwf_era5_17020204.nc', 'ecmwf_era5_17020113.nc', 'ecmwf_era5_17020205.nc', 'ecmwf_era5_17020101.nc', 'ecmwf_era5_17020115.nc', 'ecmwf_era5_17020121.nc', 'ecmwf_era5_17020112.nc', 'ecmwf_era5_17020119.nc', 'ecmwf_era5_17020110.nc', 'ecmwf_era5_17020203.nc', 'ecmwf_era5_17020109.nc', 'ecmwf_era5_17020117.nc', 'ecmwf_era5_17020207.nc', 'ecmwf_era5_17020209.nc', 'ecmwf_era5_17020120.nc', 'ecmwf_era5_17020100.nc', 'ecmwf_era5_17020118.nc', 'ecmwf_era5_17020208.nc', 'ecmwf_era5_17020103.nc', 'ecmwf_era5_17020202.nc', 'ecmwf_era5_17020107.nc', 'ecmwf_era5_17020106.nc', 'ecmwf_era5_17020122.nc', 'ecmwf_era5_17020102.nc']
2020-06-03 16:20:09,128:INFO:0
2020-06-03 16:20:09,128:INFO:34
2020-06-03 16:20:09,128:INFO:{1: 'ecmwf_era5_17020206.nc;ecmwf_era5_17020205.nc;ecmwf_era5_17020120.nc', 2: 'ecmwf_era5_17020201.nc;ecmwf_era5_17020101.nc;ecmwf_era5_17020100.nc', 3: 'ecmwf_era5_17020111.nc;ecmwf_era5_17020115.nc;ecmwf_era5_17020118.nc', 4: 'ecmwf_era5_17020114.nc;ecmwf_era5_17020121.nc;ecmwf_era5_17020208.nc', 5: 'ecmwf_era5_17020108.nc;ecmwf_era5_17020112.nc;ecmwf_era5_17020103.nc', 6: 'ecmwf_era5_17020104.nc;ecmwf_era5_17020119.nc;ecmwf_era5_17020202.nc', 7: 'ecmwf_era5_17020200.nc;ecmwf_era5_17020110.nc;ecmwf_era5_17020107.nc', 8: 'ecmwf_era5_17020123.nc;ecmwf_era5_17020203.nc;ecmwf_era5_17020106.nc', 9: 'ecmwf_era5_17020105.nc;ecmwf_era5_17020109.nc;ecmwf_era5_17020122.nc', 10: 'ecmwf_era5_17020116.nc;ecmwf_era5_17020117.nc;ecmwf_era5_17020102.nc', 11: 'ecmwf_era5_17020204.nc;ecmwf_era5_17020207.nc', 12: 'ecmwf_era5_17020113.nc;ecmwf_era5_17020209.nc'}
2020-06-03 16:20:09,128:INFO:{1: 'ecmwf_era5_17020206.nc;ecmwf_era5_17020205.nc;ecmwf_era5_17020120.nc', 2: 'ecmwf_era5_17020201.nc;ecmwf_era5_17020101.nc;ecmwf_era5_17020100.nc', 3: 'ecmwf_era5_17020111.nc;ecmwf_era5_17020115.nc;ecmwf_era5_17020118.nc', 4: 'ecmwf_era5_17020114.nc;ecmwf_era5_17020121.nc;ecmwf_era5_17020208.nc', 5: 'ecmwf_era5_17020108.nc;ecmwf_era5_17020112.nc;ecmwf_era5_17020103.nc', 6: 'ecmwf_era5_17020104.nc;ecmwf_era5_17020119.nc;ecmwf_era5_17020202.nc', 7: 'ecmwf_era5_17020200.nc;ecmwf_era5_17020110.nc;ecmwf_era5_17020107.nc', 8: 'ecmwf_era5_17020123.nc;ecmwf_era5_17020203.nc;ecmwf_era5_17020106.nc', 9: 'ecmwf_era5_17020105.nc;ecmwf_era5_17020109.nc;ecmwf_era5_17020122.nc', 10: 'ecmwf_era5_17020116.nc;ecmwf_era5_17020117.nc;ecmwf_era5_17020102.nc', 11: 'ecmwf_era5_17020204.nc;ecmwf_era5_17020207.nc', 12: 'ecmwf_era5_17020113.nc;ecmwf_era5_17020209.nc'}
2020-06-03 16:20:09,129:INFO:MA0: Next to be processed is 03 loacted in //home/a.mozaffari/data_era5/2017/03
2020-06-03 16:20:09,129:INFO:=== Directory Scanner output ===
2020-06-03 16:20:09,129:INFO:Total size of the source directory is:6953.662487Gb.
2020-06-03 16:20:09,129:INFO:Total number of the files in the source directory is: 34
2020-06-03 16:20:09,129:INFO:Total number of the directories in the source directory is: 0
2020-06-03 16:20:09,129:INFO:The follwoing is in the load Balancer
2020-06-03 16:20:09,129:INFO:{1: None, 2: None, 3: None, 4: None, 5: None, 6: None, 7: None, 8: None, 9: None, 10: None, 11: None, 12: None}
2020-06-03 16:20:09,129:INFO:['ecmwf_era5_17020206.nc', 'ecmwf_era5_17020201.nc', 'ecmwf_era5_17020111.nc', 'ecmwf_era5_17020114.nc', 'ecmwf_era5_17020108.nc', 'ecmwf_era5_17020104.nc', 'ecmwf_era5_17020200.nc', 'ecmwf_era5_17020123.nc', 'ecmwf_era5_17020105.nc', 'ecmwf_era5_17020116.nc', 'ecmwf_era5_17020204.nc', 'ecmwf_era5_17020113.nc', 'ecmwf_era5_17020205.nc', 'ecmwf_era5_17020101.nc', 'ecmwf_era5_17020115.nc', 'ecmwf_era5_17020121.nc', 'ecmwf_era5_17020112.nc', 'ecmwf_era5_17020119.nc', 'ecmwf_era5_17020110.nc', 'ecmwf_era5_17020203.nc', 'ecmwf_era5_17020109.nc', 'ecmwf_era5_17020117.nc', 'ecmwf_era5_17020207.nc', 'ecmwf_era5_17020209.nc', 'ecmwf_era5_17020120.nc', 'ecmwf_era5_17020100.nc', 'ecmwf_era5_17020118.nc', 'ecmwf_era5_17020208.nc', 'ecmwf_era5_17020103.nc', 'ecmwf_era5_17020202.nc', 'ecmwf_era5_17020107.nc', 'ecmwf_era5_17020106.nc', 'ecmwf_era5_17020122.nc', 'ecmwf_era5_17020102.nc']
2020-06-03 16:20:09,130:INFO:0
2020-06-03 16:20:09,130:INFO:34
2020-06-03 16:20:09,130:INFO:{1: 'ecmwf_era5_17020206.nc;ecmwf_era5_17020205.nc;ecmwf_era5_17020120.nc', 2: 'ecmwf_era5_17020201.nc;ecmwf_era5_17020101.nc;ecmwf_era5_17020100.nc', 3: 'ecmwf_era5_17020111.nc;ecmwf_era5_17020115.nc;ecmwf_era5_17020118.nc', 4: 'ecmwf_era5_17020114.nc;ecmwf_era5_17020121.nc;ecmwf_era5_17020208.nc', 5: 'ecmwf_era5_17020108.nc;ecmwf_era5_17020112.nc;ecmwf_era5_17020103.nc', 6: 'ecmwf_era5_17020104.nc;ecmwf_era5_17020119.nc;ecmwf_era5_17020202.nc', 7: 'ecmwf_era5_17020200.nc;ecmwf_era5_17020110.nc;ecmwf_era5_17020107.nc', 8: 'ecmwf_era5_17020123.nc;ecmwf_era5_17020203.nc;ecmwf_era5_17020106.nc', 9: 'ecmwf_era5_17020105.nc;ecmwf_era5_17020109.nc;ecmwf_era5_17020122.nc', 10: 'ecmwf_era5_17020116.nc;ecmwf_era5_17020117.nc;ecmwf_era5_17020102.nc', 11: 'ecmwf_era5_17020204.nc;ecmwf_era5_17020207.nc', 12: 'ecmwf_era5_17020113.nc;ecmwf_era5_17020209.nc'}
2020-06-03 16:20:09,130:INFO:{1: 'ecmwf_era5_17020206.nc;ecmwf_era5_17020205.nc;ecmwf_era5_17020120.nc', 2: 'ecmwf_era5_17020201.nc;ecmwf_era5_17020101.nc;ecmwf_era5_17020100.nc', 3: 'ecmwf_era5_17020111.nc;ecmwf_era5_17020115.nc;ecmwf_era5_17020118.nc', 4: 'ecmwf_era5_17020114.nc;ecmwf_era5_17020121.nc;ecmwf_era5_17020208.nc', 5: 'ecmwf_era5_17020108.nc;ecmwf_era5_17020112.nc;ecmwf_era5_17020103.nc', 6: 'ecmwf_era5_17020104.nc;ecmwf_era5_17020119.nc;ecmwf_era5_17020202.nc', 7: 'ecmwf_era5_17020200.nc;ecmwf_era5_17020110.nc;ecmwf_era5_17020107.nc', 8: 'ecmwf_era5_17020123.nc;ecmwf_era5_17020203.nc;ecmwf_era5_17020106.nc', 9: 'ecmwf_era5_17020105.nc;ecmwf_era5_17020109.nc;ecmwf_era5_17020122.nc', 10: 'ecmwf_era5_17020116.nc;ecmwf_era5_17020117.nc;ecmwf_era5_17020102.nc', 11: 'ecmwf_era5_17020204.nc;ecmwf_era5_17020207.nc', 12: 'ecmwf_era5_17020113.nc;ecmwf_era5_17020209.nc'}
2020-06-03 16:21:20,780:INFO:MA0: S7: is finished the 01 .
2020-06-03 16:21:20,781:INFO:MA0: S10: is finished the 01 .
2020-06-03 16:21:21,017:INFO:MA0: S4: is finished the 01 .
2020-06-03 16:21:21,084:INFO:MA0: S9: is finished the 01 .
2020-06-03 16:21:21,088:INFO:MA0: S6: is finished the 01 .
2020-06-03 16:21:21,102:INFO:MA0: S5: is finished the 01 .
2020-06-03 16:21:21,104:INFO:MA0: S11: is finished the 01 .
2020-06-03 16:21:21,230:INFO:MA0: S12: is finished the 01 .
2020-06-03 16:21:21,410:INFO:MA0: S8: is finished the 01 .
2020-06-03 16:21:54,059:INFO:MA0: S3: is finished the 01 .
2020-06-03 16:21:54,095:INFO:MA0: S2: is finished the 01 .
2020-06-03 16:21:54,256:INFO:MA0: S1: is finished the 01 .
2020-06-03 16:22:26,617:INFO:MA0: S12: is finished the 02 .
2020-06-03 16:22:26,747:INFO:MA0: S11: is finished the 02 .
2020-06-03 16:22:58,655:INFO:MA0: S10: is finished the 02 .
2020-06-03 16:22:59,255:INFO:MA0: S6: is finished the 02 .
2020-06-03 16:22:59,522:INFO:MA0: S4: is finished the 02 .
2020-06-03 16:22:59,585:INFO:MA0: S7: is finished the 02 .
2020-06-03 16:22:59,591:INFO:MA0: S8: is finished the 02 .
2020-06-03 16:22:59,722:INFO:MA0: S9: is finished the 02 .
2020-06-03 16:22:59,780:INFO:MA0: S5: is finished the 02 .
2020-06-03 16:23:32,178:INFO:MA0: S2: is finished the 02 .
2020-06-03 16:23:32,470:INFO:MA0: S12: is finished the 03 .
2020-06-03 16:23:32,507:INFO:MA0: S3: is finished the 02 .
2020-06-03 16:23:32,743:INFO:MA0: S11: is finished the 03 .
2020-06-03 16:23:33,204:INFO:MA0: S1: is finished the 02 .
2020-06-03 16:24:33,815:INFO:MA0: S10: is finished the 03 .
2020-06-03 16:24:34,418:INFO:MA0: S6: is finished the 03 .
2020-06-03 16:24:35,163:INFO:MA0: S8: is finished the 03 .
2020-06-03 16:24:35,181:INFO:MA0: S4: is finished the 03 .
2020-06-03 16:24:35,525:INFO:MA0: S5: is finished the 03 .
2020-06-03 16:24:35,625:INFO:MA0: S9: is finished the 03 .
2020-06-03 16:24:35,743:INFO:MA0: S7: is finished the 03 .
2020-06-03 16:25:05,500:INFO:MA0: S2: is finished the 03 .
2020-06-03 16:25:06,343:INFO:MA0: S3: is finished the 03 .
2020-06-03 16:25:06,744:INFO:MA0: S1: is finished the 03 .
2020-06-03 16:25:06,745:INFO:MA0: Sucssfully terminated with total time : 297.6529154777527
DataExtraction/concurrency_test/stager.log deleted 100644 → 0
+ 0
− 9
View file @ 72da2fc4
2020-06-03 16:54:21,422:DEBUG: === PyStager is started ===
2020-06-03 16:54:21,437:WARNING:('Node', '4', 'is idle')
2020-06-03 16:54:21,437:WARNING:('Node', '5', 'is idle')
2020-06-03 17:07:59,395:INFO:('Node:', '1', 'finished :', 'rsync -r //home/a.mozaffari/data_era5/2017/01/ /home/a.mozaffari/data_dest/01', '\r\n')
2020-06-03 17:11:27,336:INFO:('Node:', '3', 'finished :', 'rsync -r //home/a.mozaffari/data_era5/2017/03/ /home/a.mozaffari/data_dest/03', '\r\n')
2020-06-03 17:11:30,417:INFO:('Node:', '2', 'finished :', 'rsync -r //home/a.mozaffari/data_era5/2017/02/ /home/a.mozaffari/data_dest/02', '\r\n')
2020-06-03 17:11:30,417:DEBUG:1028.9953315258026
2020-06-03 17:11:30,418:INFO:== PyStager is done ==
2020-06-03 17:11:30,418:INFO:exit status : 0
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment