Parsing parameter file failed at force-level2 #102

cxxuuuuuuu · 2021-05-27T17:29:27Z

cxxuuuuuuu
May 27, 2021

Hi all,

I got an error when I run ''force-level2 /data/force/param/l2ps.prm'':

Computer:jobs running/jobs completed/%of started jobs/Average seconds to complete
ETA: 1907s Left: 587 AVG: 3.25s local:1/279/100%/3.3s No valid parameter file!
Parsing parameter file failed.
ETA: 1904s Left: 586 AVG: 3.25s local:1/280/100%/3.3s No valid parameter file!
Parsing parameter file failed.
ETA: 1900s Left: 585 AVG: 3.25s local:1/281/100%/3.3s No valid parameter file!
Parsing parameter file failed.
ETA: 1896s Left: 584 AVG: 3.25s local:1/282/100%/3.3s No valid parameter file!
Parsing parameter file failed.
ETA: 1893s Left: 583 AVG: 3.25s local:1/283/100%/3.3s No valid parameter file!
Parsing parameter file failed.
ETA: 1890s Left: 582 AVG: 3.25s local:1/284/100%/3.3s No valid parameter file!
Parsing parameter file failed.
ETA: 1887s Left: 581 AVG: 3.25s local:1/285/100%/3.3s No valid parameter file!
Parsing parameter file failed.
ETA: 1883s Left: 580 AVG: 3.25s local:1/286/100%/3.3s No valid parameter file!
Parsing parameter file failed.
ETA: 1880s Left: 579 AVG: 3.25s local:1/287/100%/3.3s No valid parameter file!
Parsing parameter file failed.
ETA: 1877s Left: 578 AVG: 3.25s local:1/288/100%/3.3s No valid parameter file!
Parsing parameter file failed.
ETA: 1874s Left: 577 AVG: 3.25s local:1/289/100%/3.3s No valid parameter file!
Parsing parameter file failed.
ETA: 1871s Left: 576 AVG: 3.26s local:1/290/100%/3.3s No valid parameter file!
Parsing parameter file failed.
ETA: 1868s Left: 575 AVG: 3.25s local:1/291/100%/3.3s No valid parameter file!
Parsing parameter file failed.
ETA: 1865s Left: 574 AVG: 3.25s local:1/292/100%/3.3s No valid parameter file!
Parsing parameter file failed.

...

I have 866 Sentinel 2 level-1C files. It worked for some files at the beginning. Then the other files could not be processed as showed in the error.

Here is my parameter file:

++PARAM_LEVEL2_START++

INPUT/OUTPUT DIRECTORIES

------------------------------------------------------------------------

FILE_QUEUE = /home/eouser/level-1-datapool/l1.pool
DIR_LEVEL2 = /home/eouser/src/force/level2
DIR_LOG = /home/eouser/src/force/log
DIR_TEMP = /home/eouser/src/force/temp

DIGITAL ELEVATION MODEL

------------------------------------------------------------------------

FILE_DEM = NULL
DEM_NODATA = -32767

DATA CUBES

------------------------------------------------------------------------

DO_REPROJ = TRUE
DO_TILE = TRUE
FILE_TILE = /home/eouser/src/force/misc/bw.txt
TILE_SIZE = 30000
BLOCK_SIZE = 3000
RESOLUTION_LANDSAT = 30
RESOLUTION_SENTINEL2 = 10

ORIGIN_LON = 7.92
ORIGIN_LAT = 51.51
PROJECTION = PROJCS["ETRS89 / LAEA Europe",GEOGCS["ETRS89",DATUM["European_Terrestrial_Reference_System_1989",SPHEROID["GRS 1980",6378137,298.257222101,
AUTHORITY["EPSG","7019"]],TOWGS84[0,0,0,0,0,0,0],AUTHORITY["EPSG","6258"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],
UNIT["degree",0.0174532925199433,AUTHORITY["EPSG","9122"]],AUTHORITY["EPSG","4258"]],PROJECTION["Lambert_Azimuthal_Equal_Area"],
PARAMETER["latitude_of_center",52],PARAMETER["longitude_of_center",10],PARAMETER["false_easting",4321000],PARAMETER["false_northing",3210000],
UNIT["metre",1,AUTHORITY["EPSG","9001"]],AUTHORITY["EPSG","3035"]]

RESAMPLING = CC

RADIOMETRIC CORRECTION OPTIONS

------------------------------------------------------------------------

DO_ATMO = TRUE
DO_TOPO = TRUE
DO_BRDF = TRUE
ADJACENCY_EFFECT = TRUE
MULTI_SCATTERING = TRUE

WATER VAPOR CORRECTION OPTIONS

------------------------------------------------------------------------

DIR_WVPLUT = NULL
WATER_VAPOR = NULL

AEROSOL OPTICAL DEPTH OPTIONS

------------------------------------------------------------------------

DO_AOD = TRUE
DIR_AOD = NULL

CLOUD DETECTION OPTIONS

------------------------------------------------------------------------

ERASE_CLOUDS = FALSE
MAX_CLOUD_COVER_FRAME = 90
MAX_CLOUD_COVER_TILE = 100
CLOUD_BUFFER = 300
SHADOW_BUFFER = 90
SNOW_BUFFER = 30
CLOUD_THRESHOLD = 0.225
SHADOW_THRESHOLD = 0.02

RESOLUTION MERGING

------------------------------------------------------------------------

RES_MERGE = IMPROPHE

CO-REGISTRATION OPTIONS

------------------------------------------------------------------------

DIR_COREG_BASE = NULL
OREG_BASE_NODATA = -9999

MISCELLANEOUS OPTIONS

------------------------------------------------------------------------

MPULSE_NOISE = TRUE
BUFFER_NODATA = FALSE

TIER LEVEL

------------------------------------------------------------------------

TIER = 1

PARALLEL PROCESSING

------------------------------------------------------------------------

NPROC = 32
NTHREAD = 2
PARALLEL_READS = FALSE
DELAY = 3

OUTPUT OPTIONS

------------------------------------------------------------------------

OUTPUT_FORMAT = GTiff
OUTPUT_DST = FALSE
OUTPUT_AOD = FALSE
OUTPUT_WVP = FALSE
OUTPUT_VZN = FALSE
OUTPUT_HOT = FALSE
OUTPUT_OVV = TRUE

++PARAM_LEVEL2_END++

Does anybody know a solution? Thanks a lot.

Best regards,
Chunyan

Answered by davidfrantz

May 31, 2021

Hi Chunyan,
I believe you can simply drag'n'drop a file when writing a message.
Cheers,
David

View full answer

JonasardoLundUniv · 2021-05-30T14:24:38Z

JonasardoLundUniv
May 30, 2021

Hi

I don't think DO_TOPO = TRUE work with FILE_DEM = NULL
DEM needed to do topo-correction.

Regards
/Jonas

1 reply

cxxuuuuuuu May 31, 2021
Author

Hi @JonasardoLundUniv ,

thank you! I have changed DO_TOPO = TRUE to DO_TOPO = FALSE.

Regards,
Chunyan

davidfrantz · 2021-05-31T06:38:36Z

davidfrantz
May 31, 2021
Maintainer

Hi,

yes, @JonasardoLundUniv is right. However, the error message here is because of some other error that appears before the DEM error message would have been emitted.

Probably very much related to this thread here:
#84

This is a strange error that keeps popping up, and I am not able to reproduce so far.

What FORCE version are you using?
What data are you processing? Landsat / S2?
What System are you running on? Do you have enough RAM etc.?
Are you using Docker?

Can you please attach the parameter file (when you post the content only, I cannot see if something is wrong with the file itself).

Cheers,
David

1 reply

cxxuuuuuuu May 31, 2021
Author

Hi David,

thank you for the answer. I am using FORCE 3.6.5, and only processing Sentinel 2 Level 1C now.
I run on Windows 10 with a virtual machine to Ubuntu 18.04.5 LTS (GNU/Linux 4.15.0-124-generic x86_64) where the RAM is 32 GB.
And I am not using Docker.

Could you please tell me how to attach a file here? Because I couldn't find the option to upload my parameter file. Thanks!

Cheers,
Chunyan

davidfrantz · 2021-05-31T08:39:22Z

davidfrantz
May 31, 2021
Maintainer

Hi Chunyan,
I believe you can simply drag'n'drop a file when writing a message.
Cheers,
David

9 replies

cxxuuuuuuu Jun 8, 2021
Author

Try with 1 CPU instead of 32.

Hi @JonasardoLundUniv ,

"Start processing with 32 CPUs" is default here. Could you please tell me how to try 1 CPU? How can I change 32 CPUs to 1 CPU?

Thanks!

Best regards,
Chunyan

davidfrantz Jun 9, 2021
Maintainer

there is the parameter NPROC in the parameter file...
Best,
David

cxxuuuuuuu Jun 23, 2021
Author

there is the parameter NPROC in the parameter file...
Best,
David

Hi David,

I changed NPROC to 3 and the ''new errors'' that I mentioned above have been solved. During these two weeks I work very intensively at modifying the parameter file, where I use SRTM as DEM to rectify the Sentinel-2 Level 1 images. But it really doesn't work no matter how many times that I try.

I have communicated a lot with @skoenig-eo Simon König who also got the similar error at parsing parameter file. But we really couldn't find a solution. #84

The error is still the same:

eouser@force:~$ force-level2 /home/eouser/src/force/param/12ps.prm
13 images enqueued. Start processing with 3 CPUs
No valid parameter file!
Parsing parameter file failed.

Computers / CPU cores / Max jobs to run
1:local / 8 / 3

Computer:jobs running/jobs completed/%of started jobs/Average seconds to complete
ETA: 0s Left: 13 AVG: 0.00s local:1/0/100%/0.0s No valid parameter file!
Parsing parameter file failed.
ETA: 36s Left: 12 AVG: 3.00s local:1/1/100%/4.0s No valid parameter file!
Parsing parameter file failed.
ETA: 30s Left: 11 AVG: 3.00s local:1/2/100%/3.5s No valid parameter file!
Parsing parameter file failed.
ETA: 27s Left: 10 AVG: 3.00s local:1/3/100%/3.3s No valid parameter file!
Parsing parameter file failed.
ETA: 24s Left: 9 AVG: 3.00s local:1/4/100%/3.2s No valid parameter file!
Parsing parameter file failed.
ETA: 21s Left: 8 AVG: 3.00s local:1/5/100%/3.2s No valid parameter file!
Parsing parameter file failed.
ETA: 20s Left: 7 AVG: 3.17s local:1/6/100%/3.3s No valid parameter file!
Parsing parameter file failed.
ETA: 17s Left: 6 AVG: 3.14s local:1/7/100%/3.3s No valid parameter file!
Parsing parameter file failed.
ETA: 15s Left: 5 AVG: 3.12s local:1/8/100%/3.2s No valid parameter file!
Parsing parameter file failed.
ETA: 12s Left: 4 AVG: 3.11s local:1/9/100%/3.2s No valid parameter file!
Parsing parameter file failed.
ETA: 9s Left: 3 AVG: 3.10s local:1/10/100%/3.2s No valid parameter file!
Parsing parameter file failed.
ETA: 6s Left: 2 AVG: 3.18s local:1/11/100%/3.3s No valid parameter file!
Parsing parameter file failed.
ETA: 3s Left: 1 AVG: 3.17s local:1/12/100%/3.2s No valid parameter file!
Parsing parameter file failed.
ETA: 0s Left: 0 AVG: 2.92s local:0/13/100%/3.0s

12ps.prm.txt

Would you like to check and run the 12ps.prm again and see whether there are errors of parameters? Since not just I got this error, it is very meaningful for helping other FORCE users if there is a solution. Thanks again.

Best regards,
Chunyan

davidfrantz Jun 24, 2021
Maintainer

Hi @cxxuuuuuuu ,

there are several issues intermingled in this thread.

I am glad that you fixed your initial issues with the resource allocation. which caused many of the images to fail. (both because of overutilization of threads and RAM limitations). This is not the same issue as No valid parameter file + directory not existing/readable #84
I would be happy if you could mark my answer above as solution as this was the main issue here.
Your current issue is because your parameter file is indeed invalid, probably because you did a lot of testing and somwthing went wrong on the way.

tail ~/temp/cxuuuuu/12ps.prm.txt
# Type: Logical. Valid values: {TRUE,FALSE}
OUTPUT_VZN = FALSE
# Output the  Haze Optimzed Transformation output? This product can be
# used in force-level3; no other higher-level FORCE module is using this.
# Type: Logical. Valid values: {TRUE,FALSE}
OUTPUT_HOT = FALSE
# Output overview thumbnails? These are jpegs at reduced spatial resolution,
# which feature an RGB overview + quality information overlayed (pink: cloud,
# red: cirrus, cyan: cloud shadow, yellow: snow, orange: saturated, green:
# subzero reflectance). No higher-leve /codede/auxdata/CopDEM/COP-DEM_GLO-90-DGED/DEM1_SAR_DGE_90_20110408T171540_20130501T171033_ADS_000000_6951.DEM/Copernicus_DSM_30_N51_00_E008_00/DEM/Copernicus_DSM_30_N51_00_E008_00_DEM.ti+

should look like this:

tail ~/temp/cxuuuuu/LEVEL2-skeleton.prm
# Type: Logical. Valid values: {TRUE,FALSE}
OUTPUT_HOT = FALSE
# Output overview thumbnails? These are jpegs at reduced spatial resolution,
# which feature an RGB overview + quality information overlayed (pink: cloud,
# red: cirrus, cyan: cloud shadow, yellow: snow, orange: saturated, green:
# subzero reflectance). No higher-level FORCE module is using this.
# Type: Logical. Valid values: {TRUE,FALSE}
OUTPUT_OVV = TRUE

++PARAM_LEVEL2_END++

Note the last line. After fixing this, it worked as expected on my end.

Regarding No valid parameter file + directory not existing/readable #84, I accidentally was able to reproduce this strange behaviour when testing your parameter file. I will answer in No valid parameter file + directory not existing/readable #84.

Cheers,
David

cxxuuuuuuu Jul 7, 2021
Author

Hi David,

thanks a lot!! It works now on my end!! Although I got the same error ''Parsing parameter file failed'', but I found that the BOA results were somehow still generated under the /level2 folder, what a incredible thing!!

Thank you for your kind explanation again.

Best regards from Berlin,
Chunyan

davidfrantz · 2021-07-19T08:09:38Z

davidfrantz
Jul 19, 2021
Maintainer

Hi @cxxuuuuuuu ,

I am happy to hear that it works on your end now.

Can you please mark this thread as answered?

For the strange warning, please see my response here: #84

Cheers,
David

1 reply

cxxuuuuuuu Jul 19, 2021
Author

Hey David,

yes you can mark it as answered.

Best regards,
Chunyan

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Parsing parameter file failed at force-level2 #102

{{title}}

Replies: 4 comments 12 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Parsing parameter file failed at force-level2 #102

cxxuuuuuuu May 27, 2021

INPUT/OUTPUT DIRECTORIES

------------------------------------------------------------------------

DIGITAL ELEVATION MODEL

------------------------------------------------------------------------

DATA CUBES

------------------------------------------------------------------------

RADIOMETRIC CORRECTION OPTIONS

------------------------------------------------------------------------

WATER VAPOR CORRECTION OPTIONS

------------------------------------------------------------------------

AEROSOL OPTICAL DEPTH OPTIONS

------------------------------------------------------------------------

CLOUD DETECTION OPTIONS

------------------------------------------------------------------------

RESOLUTION MERGING

------------------------------------------------------------------------

CO-REGISTRATION OPTIONS

------------------------------------------------------------------------

MISCELLANEOUS OPTIONS

------------------------------------------------------------------------

TIER LEVEL

------------------------------------------------------------------------

PARALLEL PROCESSING

------------------------------------------------------------------------

OUTPUT OPTIONS

------------------------------------------------------------------------

Replies: 4 comments · 12 replies

JonasardoLundUniv May 30, 2021

cxxuuuuuuu May 31, 2021 Author

davidfrantz May 31, 2021 Maintainer

cxxuuuuuuu May 31, 2021 Author

davidfrantz May 31, 2021 Maintainer

cxxuuuuuuu Jun 8, 2021 Author

davidfrantz Jun 9, 2021 Maintainer

cxxuuuuuuu Jun 23, 2021 Author

davidfrantz Jun 24, 2021 Maintainer

cxxuuuuuuu Jul 7, 2021 Author

davidfrantz Jul 19, 2021 Maintainer

cxxuuuuuuu Jul 19, 2021 Author

cxxuuuuuuu
May 27, 2021

Replies: 4 comments 12 replies

JonasardoLundUniv
May 30, 2021

cxxuuuuuuu May 31, 2021
Author

davidfrantz
May 31, 2021
Maintainer

cxxuuuuuuu May 31, 2021
Author

davidfrantz
May 31, 2021
Maintainer

cxxuuuuuuu Jun 8, 2021
Author

davidfrantz Jun 9, 2021
Maintainer

cxxuuuuuuu Jun 23, 2021
Author

davidfrantz Jun 24, 2021
Maintainer

cxxuuuuuuu Jul 7, 2021
Author

davidfrantz
Jul 19, 2021
Maintainer

cxxuuuuuuu Jul 19, 2021
Author