Meeting freeing memory issue after the training stage when using Intel-MLlib to run PCA and K-means algorithms.

[haojinIntel]

We try to use Intel-mllib to run K-means and PCA algorithms on AWS EMR. The training stages can successfully finish while we find some failed tasks about freeing memory which can influence performance. The error messages are showed below:

2021-08-16T06:21:49.897+0000: [GC (Allocation Failure) [PSYoungGen: 980480K->39485K(1119744K)] 1012870K->71899K(3132416K), 0.0244501 secs] [Times: user=0.77 sys=0.00, real=0.02 secs]
2021-08-16T06:21:50.063+0000: [GC (Metadata GC Threshold) [PSYoungGen: 78684K->24007K(1119744K)] 111098K->56430K(3132416K), 0.0182626 secs] [Times: user=0.39 sys=0.02, real=0.02 secs]
2021-08-16T06:21:50.082+0000: [Full GC (Metadata GC Threshold) [PSYoungGen: 24007K->0K(1119744K)] [ParOldGen: 32422K->41994K(2475520K)] 56430K->41994K(3595264K), [Metaspace: 83342K->75668K(100352K)], 0.1082003 secs] [Times: user=1.59 sys=0.25, real=0.11 secs]
2021-08-16T06:21:51.350+0000: [GC (GCLocker Initiated GC) [PSYoungGen: 1032704K->25774K(1636864K)] 1074709K->67795K(4112384K), 0.0201072 secs] [Times: user=0.24 sys=0.03, real=0.02 secs]
2021-08-16T06:21:51.673+0000: [GC (Allocation Failure) [PSYoungGen: 1627822K->20614K(1635840K)] 1669845K->62645K(4111360K), 0.0148229 secs] [Times: user=0.30 sys=0.02, real=0.01 secs]
2021-08-16T06:21:51.938+0000: [GC (Allocation Failure) [PSYoungGen: 1622662K->21027K(2599424K)] 1664693K->63066K(5074944K), 0.0128648 secs] [Times: user=0.28 sys=0.01, real=0.01 secs]
2021-08-16T06:21:52.380+0000: [GC (Allocation Failure) [PSYoungGen: 2587171K->4015K(2598912K)] 2629210K->62097K(5074432K), 0.0714582 secs] [Times: user=2.49 sys=0.08, real=0.07 secs]
2021-08-16T06:21:53.050+0000: [GC (Allocation Failure) [PSYoungGen: 2570159K->21441K(3652096K)] 2628241K->79700K(6127616K), 0.0080649 secs] [Times: user=0.18 sys=0.00, real=0.01 secs]
2021-08-16T06:21:53.598+0000: [GC (Allocation Failure) [PSYoungGen: 3652033K->5604K(3722240K)] 3710292K->81310K(6197760K), 0.0096613 secs] [Times: user=0.19 sys=0.01, real=0.00 secs]
2021-08-16T06:21:54.159+0000: [GC (GCLocker Initiated GC) [PSYoungGen: 3693540K->14913K(5032960K)] 3769285K->91450K(7508480K), 0.0066516 secs] [Times: user=0.06 sys=0.11, real=0.00 secs]

[haojinIntel]

@xwu99 @zhixingheyi-tian Please help to track the issue.