Rescaled refined particles extraction error

[GalkinVitold]

Describe your problem

After initial extraction we obtained 4,030,812 particles (4.068A/px). This was followed by a 2D-classification to eliminate "bad" particles. The Select job using "good" classes from the 2D classification yielded 3,574,004 particles. Extraction of refined particles using a star file from the previously used Selection job yielded 4,757,248 particles (2.712A/px) - way more that even is contained in the input file for the Select job. The only difference that this particular set has from the others that never possessed this error is that 4 subsets of images were combined at the level of MotioCorrection and each set was placed into the parent relion folder. The technical details are appended below:

FILES LOCATION IS IN PARENT FOLDER. MOTION CORRECTED FILES WERE DONE SEPARATELY ELSEWHERE USING THE SAME VERSION OF RELION AND MOTIONCORRECTION FOLDERS WERE COPIED INTO THE PARENT DIRECTORY. THEIR LOCATION IN THE PARENT RELION FOLDER IS AS FOLLOWS:
MotionCorr/job002
MotionCorr/job004
MotionCorr/job020
MotionCorr/job200

1. EXTRACT JOB Extract/job015 yields 4,030,812 particles

++++ Executing new job on Tue Apr 7 13:05:12 2020
++++ with the following command(s):
which relion_preprocess --i Select/job010/micrographs.star --coord_dir AutoPick/job013/ --coord_suffix _autopick.star --part_star Extract/job015/particles.star --part_dir Extract/job015/ --extract --extract_size 324 --scale 108 --norm --bg_radius 50 --white_dust -1 --black_dust -1 --invert_contrast --helix --helical_outer_diameter 110 --helical_bimodal_angular_priors
++++

run.out file info:

• Using pixel size from input STAR file of 1.356 Angstroms
  Extracting particles from 5260 micrographs ...
  000/??? sec (,,"> [oo]
  0.06/3.74 hrs ~~(,,">
  0.13/3.88 hrs .(,,">
  0.19/3.86 hrs ..~~(,,">
  0.25/3.82 hrs ...(,_,">
  0.31/3.78 hrs ....(,,">
  0.38/3.80 hrs .....~~(,,">
  0.44/3.79 hrs ......(,_,">
  0.50/3.80 hrs .......(,,">
  0.56/3.78 hrs ........~~(,,">
  0.63/3.79 hrs .........(,_,">
  0.69/3.81 hrs ..........(,,">
  0.75/3.80 hrs ...........~~(,,">
  0.83/3.84 hrs ............(,_,">
  0.89/3.86 hrs .............(,,">
  0.97/3.91 hrs ..............~~(,,">
  1.03/3.89 hrs ...............(,_,">
  1.10/3.90 hrs ................(,,">
  1.17/3.93 hrs .................~~(,,">
  1.25/3.96 hrs ..................(,_,">
  1.32/3.98 hrs ...................(,,">
  1.38/3.98 hrs ....................~~(,,">
  1.46/4.01 hrs .....................(,_,">
  1.54/4.04 hrs ......................(,,">
  1.62/4.08 hrs .......................~~(,,">
  1.69/4.10 hrs ........................(,_,">
  1.77/4.11 hrs .........................(,,">
  1.84/4.12 hrs ..........................~~(,,">
  1.91/4.13 hrs ...........................(,_,">
  1.98/4.13 hrs ............................(,,">
  2.05/4.12 hrs .............................~~(,,">
  2.12/4.13 hrs ..............................(,_,">
  2.19/4.15 hrs ...............................(,,">
  2.26/4.13 hrs ................................~~(,,">
  2.32/4.13 hrs .................................(,_,">
  2.38/4.11 hrs ..................................(,,">
  2.44/4.10 hrs ...................................~~(,,">
  2.50/4.08 hrs ....................................(,_,">
  2.56/4.07 hrs .....................................(,,">
  2.62/4.06 hrs ......................................~~(,,">
  2.69/4.06 hrs .......................................(,_,">
  2.75/4.05 hrs ........................................(,,">
  2.81/4.05 hrs .........................................~~(,,">
  2.88/4.04 hrs ..........................................(,_,">
  2.94/4.04 hrs ...........................................(,,">
  3.00/4.03 hrs ............................................~~(,,">
  3.07/4.03 hrs .............................................(,_,">
  3.13/4.03 hrs ..............................................(,,">
  3.20/4.03 hrs ...............................................~~(,,">
  3.26/4.03 hrs ................................................(,_,">
  3.33/4.03 hrs .................................................(,,">
  3.41/4.04 hrs ..................................................~~(,,">
  3.48/4.04 hrs ...................................................(,_,">
  3.54/4.04 hrs ....................................................(,,">
  3.61/4.04 hrs .....................................................~~(,,">
  3.68/4.05 hrs ......................................................(,_,">
  3.74/4.04 hrs .......................................................(,,">
  3.81/4.04 hrs ........................................................~~(,,">
  3.89/4.05 hrs .........................................................(,_,">
  3.96/4.05 hrs ..........................................................(,,">
  4.02/4.05 hrs ...........................................................~~(,,"> Joining metadata of all particles from 5257 micrographs in one STAR file...
  Written out STAR file with 4030812 particles in Extract/job015/particles.star
  The new pixel size of the extracted particles are 4.068 Angstrom/pixel.
  Done preprocessing!

2. SELECT JOB Select/job019 after 2D classification yields 3574004 particles out of 4,030,812

++++ Executing new job on Mon Apr 13 20:16:47 2020
++++ with the following command(s):
which relion_display --gui --i Class2D/job017/run_ct1_it012_model.star --allow_save --fn_parts Select/job019/particles.star --fn_imgs Select/job019/class_averages.star --recenter
++++
run.out file info:

Written out Select/job019/backup_selection.star
Saved Select/job019/class_averages.star with 77 selected images.
Saved Select/job019/particles.star with 3574004 selected particles.
run.job
job_type == 7
is_continue == false
Metadata label for images: == rlnImageName
Sigma-value for discarding images: == 4
OR: select on image statistics? == No
Submit to queue? == No
Randomise order before making subsets?: == No
Re-center the class averages? == Yes
Regroup the particles? == No
OR: remove duplicates? == No
Select based on metadata values? == No
OR: split into subsets? == No
Minimum inter-particle distance (A) == 30
OR select from picked coords: ==
OR select from particles.star: ==
OR select from micrographs.star: ==
Select classes from model.star: == Class2D/job017/run_ct1_it012_model.star
Pixel size before extraction (A) == -1
Minimum dedicated cores per node: == 1
Approximate nr of groups: == 1
OR: number of subsets: == -1
Additional arguments: ==
Queue submit command: == qsub
Standard submission script: == /public/EM/RELION/relion/bin/relion_qsub.csh
Queue name: == openmpi
Metadata label for subset selection: == rlnCtfFigureOfMerit
Maximum metadata value: == 9999.
Minimum metadata value: == -9999.
Subset size: == 100

particles.star file info:

RELION; version 3.0.8

data_

loop_
_rlnCoordinateX #1
_rlnCoordinateY #2
_rlnClassNumber #3
_rlnAutopickFigureOfMerit #4
_rlnHelicalTubeID #5
_rlnAngleTiltPrior #6
_rlnAnglePsiPrior #7
_rlnHelicalTrackLength #8
_rlnAnglePsiFlipRatio #9
_rlnImageName #10
_rlnMicrographName #11
_rlnMagnification #12
_rlnDetectorPixelSize #13
_rlnCtfMaxResolution #14
_rlnCtfFigureOfMerit #15
_rlnVoltage #16
_rlnDefocusU #17
_rlnDefocusV #18
_rlnDefocusAngle #19
_rlnSphericalAberration #20
_rlnCtfBfactor #21
_rlnCtfScalefactor #22
_rlnPhaseShift #23
_rlnAmplitudeContrast #24
_rlnOriginX #25
_rlnOriginY #26
_rlnGroupNumber #27
_rlnAngleRot #28
_rlnAngleTilt #29
_rlnAnglePsi #30
_rlnNormCorrection #31
_rlnLogLikeliContribution #32
_rlnMaxValueProbDistribution #33
_rlnNrOfSignificantSamples #34
294.279645 2018.394332 1 0.467900 2 90.000000 74.403066 120.817969 0.500000 000007@Extract/job015/i_0006.mrcs MotionCorr/job002/i_0006.mrc 10000.000000 4.068000 4.329398 0.087316 300.000000 23195.082031 22096.390625 80.364815 2.700000 0.000000 1.000000 0.000000 0.070000 5.962610 3.962610 1 0.000000 0.000000 78.539680 0.939714 37588.169490 0.785995 60

3. RE-EXTRACT JOB using Select/job019/particles.star yields 4,757,248 particles!

++++ Executing new job on Sun May 3 21:00:36 2020
++++ with the following command(s):
which relion_preprocess --i Select/job010/micrographs.star --reextract_data_star Select/job019/particles.star --part_star Extract/job111/particles.star --part_dir Extract/job111/ --extract --extract_size 324 --scale 162 --norm --bg_radius 75 --white_dust -1 --black_dust -1 --invert_contrast --helix --helical_outer_diameter 110 --helical_bimodal_angular_priors
echo Select/job010/micrographs.star > Extract/job111/coords_suffix_extract.star

run.out file info:

++++

• Using pixel size from input STAR file of 1.356 Angstroms
• Re-extracting particles based on coordinates from input _data.star file
• Select/job019/particles.star
  Extracting particles from 5260 micrographs ...
  000/??? sec (,,"> [oo]
  0.18/11.03 hrs ~~(,,">
  0.38/11.45 hrs .(,,">
  0.56/11.30 hrs ..~~(,,">
  0.74/11.14 hrs ...(,_,">
  0.90/10.89 hrs ....(,,">
  1.07/10.83 hrs .....~~(,,">
  1.22/10.55 hrs ......(,_,">
  1.38/10.40 hrs .......(,,">
  1.53/10.29 hrs ........~~(,,">
  1.74/10.53 hrs .........(,_,">
  1.93/10.63 hrs ..........(,,">
  2.11/10.61 hrs ...........~~(,,">
  2.32/10.78 hrs ............(,_,">
  2.52/10.90 hrs .............(,,">
  2.76/11.12 hrs ..............~~(,,">
  2.94/11.12 hrs ...............(,_,">
  3.12/11.09 hrs ................(,,">
  3.29/11.05 hrs .................~~(,,">
  3.48/11.08 hrs ..................(,_,">
  3.66/11.07 hrs ...................(,,">
  3.86/11.11 hrs ....................~~(,,">
  4.10/11.26 hrs .....................(,_,">
  4.34/11.41 hrs ......................(,,">
  4.57/11.52 hrs .......................~~(,,">
  4.77/11.53 hrs ........................(,_,">
  4.95/11.51 hrs .........................(,,">
  5.13/11.48 hrs ..........................~~(,,">
  5.30/11.43 hrs ...........................(,_,">
  5.44/11.34 hrs ............................(,,">
  5.58/11.24 hrs .............................~~(,,">
  5.72/11.16 hrs ..............................(,_,">
  5.89/11.12 hrs ...............................(,,">
  6.02/11.03 hrs ................................~~(,,">
  6.16/10.95 hrs .................................(,_,">
  6.28/10.85 hrs ..................................(,,">
  6.41/10.77 hrs ...................................~~(,,">
  6.54/10.69 hrs ....................................(,_,">
  6.67/10.61 hrs .....................................(,,">
  6.81/10.55 hrs ......................................~~(,,">
  6.95/10.50 hrs .......................................(,_,">
  7.09/10.45 hrs ........................................(,,">
  7.23/10.41 hrs .........................................~~(,,">
  7.37/10.36 hrs ..........................................(,_,">
  7.50/10.31 hrs ...........................................(,,">
  7.63/10.26 hrs ............................................~~(,,">
  7.77/10.22 hrs .............................................(,_,">
  7.91/10.18 hrs ..............................................(,,">
  8.05/10.14 hrs ...............................................~~(,,">
  8.18/10.10 hrs ................................................(,_,">
  8.33/10.07 hrs .................................................(,,">
  8.48/10.06 hrs ..................................................~~(,,">
  8.63/10.04 hrs ...................................................(,_,">
  8.77/10.00 hrs ....................................................(,,">
  8.92/9.98 hrs .....................................................~~(,,">
  9.06/9.96 hrs ......................................................(,_,">
  9.20/9.93 hrs .......................................................(,,">
  9.34/9.90 hrs ........................................................~~(,,">
  9.49/9.89 hrs .........................................................(,_,">
  9.63/9.87 hrs ..........................................................(,,">
  9.78/9.85 hrs ...........................................................~~(,,"> Joining metadata of all particles from 5256 micrographs in one STAR file...
  Written out STAR file with 4757248 particles in Extract/job111/particles.star
  The new pixel size of the extracted particles are 2.712 Angstrom/pixel.
  Done preprocessing!

particles.star file info:

RELION; version 3.0.8

data_

loop_
_rlnCoordinateX #1
_rlnCoordinateY #2
_rlnClassNumber #3
_rlnAutopickFigureOfMerit #4
_rlnHelicalTubeID #5
_rlnAngleTiltPrior #6
_rlnAnglePsiPrior #7
_rlnHelicalTrackLength #8
_rlnAnglePsiFlipRatio #9
_rlnImageName #10
_rlnMicrographName #11
_rlnMagnification #12
_rlnDetectorPixelSize #13
_rlnCtfMaxResolution #14
_rlnCtfFigureOfMerit #15
_rlnVoltage #16
_rlnDefocusU #17
_rlnDefocusV #18
_rlnDefocusAngle #19
_rlnSphericalAberration #20
_rlnCtfBfactor #21
_rlnCtfScalefactor #22
_rlnPhaseShift #23
_rlnAmplitudeContrast #24
_rlnOriginX #25
_rlnOriginY #26
_rlnGroupNumber #27
_rlnAngleRot #28
_rlnAngleTilt #29
_rlnAnglePsi #30
_rlnNormCorrection #31
_rlnLogLikeliContribution #32
_rlnMaxValueProbDistribution #33
_rlnNrOfSignificantSamples #34
5401.593396 421.744784 1 -0.18798 25 90.000000 175.395505 140.954295 0.500000 000001@Extract/job111/i_1208.mrcs MotionCorr/job200/i_1208.mrc 10000.000000 2.712000 3.611457 0.144146 300.000000 29607.343750 28674.744141 32.229267 2.700000 0.000000 1.000000 0.000000 0.070000 -7.55608 -0.05608 4341 0.000000 0.000000 -180.21032 0.860866 37553.244330 0.714844 3

Please write a clear description of what the problem is.

Environment:

• OS: CentOS 7.7
• MPI runtime: OpenMPI 2.0.1
• RELION version [RELION-3.08]
• Memory: [512 GB]
• GPU: [RTX 2080Ti - two cards]

Dataset:

• Box size: [324 px]
• Pixel size: [1.356 Å/px]
• Number of particles: [4,030,812]
• Description: [actin filaments]

Job options:
see above
Error message:

no error message - just wrong number of particles

[biochem-fan]

Select/job010/micrographs.star

How was this produced? Did you first combine four MotionCorr jobs and then run CTFFIND?

Do any of them contain micrographs with a same name (e.g. i_0006.mrc in MotionCorr/job002 and job004)? If so, this error is understandable. Recall that RELION strips off JobType/jobXXX internally to get the file name used for the next task. For example, particles from both MotionCorr/job002/001.mrc and MotionCorr/job004/001.mrc would result in Extract/jobXXX/001.mrcs and get mixed up. In this case, you should have put movies in different folders (e.g. Dataset1/001.tif, Dataset2/001.tif, ...) before running motion correction.
Then motion corrected micrographs would be named like MotionCorr/jobXXX/Dataset1/001.mrc and MotionCorr/jobXXX/Dataset2/001.mrc and will never be mixed up.

[GalkinVitold]

Thanks Bioche-fan for your input! Yes, we combined four MotionCorr jobs and then run CTFFIND on all of those followed by elimination of images with bad Thong rings. Yes, the files in each job have same names. We hoped that since the files are located in different Job folders, they would not get mixed. Is there any way we can correct this problem without re-running everything from the scratch?

[biochem-fan]

Is there any way we can correct this problem without re-running everything from the scratch?

You can change movie names and replace names in all relevant STAR files.

But this is extremely error prone. Probably running motion correction again overnight is easier.

elimination of images with bad Thong rings

In my opinion, manual inspection of Thon rings is not worth the effort. I only eliminate those with very bad rlnCtfMaxResolution.