RELEASE_NOTES

-------------------------------------------------------------------------

v0.64

1) Fixed a serious bug in multishear that caused all galaxies to fail.

2) Fixed a bug in the multishear bounds checking where an assert could 
   fail because two different bounds, which are conceptually basically
   the same thing, could be numerically different, since they are 
   defined by different routines that do slightly different things.

3) Fixed a bug in PixelList where it was leaking memory out of the 
   Pool allocation into the general memory allocation.  This isn't 
   a memory leak per se, but it allows the memory to get fragmented
   when the PixelLists are deleted at the end of working on that section.
   In fact, this was exactly the problem that the Pool allocator was
   written to solve.  Now the PixelLists use shared memory semantics on
   copy which means they are both more efficient when used in a 
   std::vector<> and also keep all the allocation in the Pool where
   fragmentation isn't a problem.

4) Added a feature to multishear where it checks how much memory is 
   being used periodically throughout the program.  If a particular
   section exceeds this amount (parameter max_vmem) then it now
   splits that section into 4 quadrants and tries again.  It will do
   this at most multishear_max_resplits times.

5) Split the functionality of output_dots into output_dots and output_info.
   output_dots now really is just about dots.  And output_info says 
   whether to output all the other information that you might want.

6) Used non-physical values like -999 for the shear, cov, and nu values
   calculated by multishear.  It was an oversight that we hadn't 
   done this before, since the regular measureshear program does this.

7) Implemented the new bnl queue system in the python scripts.

8) A bunch of work on the new way of interfacing with the database.
   Especially as pertains to the multishear files.

9) Added wl_desdm.config to the install list.  The idea here is that
   wl.config will always keep up to date on any algorithmic parameters.
   Anything that desdm might want to change for a particular run
   will also be copied to wl_desdm.config.  They will maintain that 
   file with the defaults they need for running on their system.
   So they would typically run programs with: (e.g.)
   measureshear $etc/wl.config +$etc/wl_desdm.config image_file=....

10) Changed units of psf_aperture to be in arcsec rather than the 
    psf radius.  This way the aperture doesn't shrink for good seeing
    data to where we don't have enough pixels for a good measurement.

-------------------------------------------------------------------------

v0.63

1) Some more work on the low S/N galaxy measurements:
   - Use higher aperture size
   - Use higher f_psf
   - Automatically reduce shapelet order for low S/N objects.

2) Fixed calculation of the shear covariance matrix.

3) Added Bob's new s/g algorithm.

4) Changed output galorder to be the order actually used in the 
   measurement process, since it may be less than the requested value.

5) Fixed some swig/gcc-4.3 bugs.

6) Switch from pbs to condor.  

-------------------------------------------------------------------------

v0.62  (2011-04-29) 

1) Shear average now dowe directly in gamma space, rather than by 
   translating to and from delta per definition in BJ02.

2) A number of algorithmic changes to the shear measurement process, 
   mostly to fix some biases at low S/N.
   - When shear measurement fails, reduce order of shapelet decomposition 
     and try again, rather than failing.
   - Made the shear measurement routine work for either single or multiple
     epoch estimates, so no need to have two different versions anymore.
   - Iterate multiple times to estimate shear, using a weighted average of
     the 'round' frame and the 'shear' frame for doing the measurement.
   - Also iterate for the native measurement a couple of times, which helps
     for highly elliptical galaxies.

3) Added Bob's improvements to the star-galaxy separation algorithm.

4) Some other minor bug fixes.

-------------------------------------------------------------------------

v0.61  (2011-01-08) 

The biggest development is to reconfigure the basic shear measurement 
algorithm to address a bug we had seen where the shears come out up to 10%
wrong when the psf has high order terms in it.  

It turns out that this is prone to significant errors:

First, for highly elliptical galaxies, this requires shearing the PSF 
a great deal, which is not very accurate with the finite order of the
decomposition.

Second, the decomposition has the highest signal-to-noise in the frame where
the _observed_ galaxy is round, not where the pre-seeing galaxy is round.
So for small galaxies where these two frames are potentially very different,
the S/N can be much worse and the algorithm had trouble converging.

So the new method does a single deconvolution measurement in the frame
where the observed galaxy is round.  Then it analytically determines what
frame that shapelet vector _would_ be observed to be round.
I don't remeasure in that frame to check though because of the measurement
issues mentioned above.

On the Great08 simulations, I'm getting:
Q = 260 for LowNoise_Known
Q = 668 for LowNoise_Blind
(I'm not sure why there is such a big difference.)

The new algorithm is also much faster than the old one, since we aren't 
inverting a large matrix (with all the pixel information) at every step 
in the non-linear solver.  It takes about 0.1 sec / galaxy / cpu.

OK, now the list of other changes in this update:

-- Updated stars_maxmag and stars_maxoutmag for latest GSN 
   (to 17 and 16 respectively).

-- Updated shear_gal_order2 for new algorithm.  The new meaning is that
   gal_order = the measured order of the shapelet decomposition.
   gal_order2 = the order for the intermediate matrices during the 
                various steps of the calculation.
   I think good values to use are 8 and 20.

-- Switched over the Image class to using CCFits for reading the image.

-- Added some better exception throwing withn the Image can't be read 
   correctly.

-- Added debug statements whenever an exception is caught to help follow
   what the code is doing when an exception happens.

-- Ignore any exceptions that happen when writing the Log.  This happens
   after our normal exception handling stuff, so it tends to just be a 
   problem if an exception is thrown then.

-- Added a workaround for a bug in gcc 4.4 where it wouldn't catch
   exceptions properly.  Need -lpthread before -lgcc_eh in lib list.

-- Got rid of no-longer-used PSF_TRAINING and PSF_VALIDATION flags

-- Added SHAPELET_NOT_DECONV flag to indicate when the returned shapelet
   vector has not been deconvolved by the PSF.  This can happen if there
   is some other error before we get to the point in the code where the
   deconvolution happens.

-- Removed some of the timing stuff that wasn't really useful anymore.

-- Various non-substantive code cleanups.

-------------------------------------------------------------------------

v0.60  (2010-11-15)  Quite a few bug fixes and some structural changes
  in the python framework.  I'll do these by topic rather than by file,
  since there are a lot of them, and many span multiple files.

--  Overwrite the psf catalog after running the interpolation module so 
    we can access the new flags indicating which objects were removed
    as outliers.

--  Changed teh default shear_gal_order2 from 2 to 6.  This seems to 
    be less susceptible to biases in the shear from model errors, but
    we are still testing this.
    
--  Erin added a whole bunch of stuff for the python bindings.

--  Added ability to make a shared library which is required when doing the 
    python stuff.  Also need TMV compiled with a shared library.

--  Updated the SConstruct script to work with the tmv-link file 
    produced by the TMV installation.

--  Fixed a bug where the centroids would end up biased in the direction
    of the initial estimate.  Also fixed a discrepancy between the 
    definition of the lower left pixel.  (Use cat_x_offset = 1 and
    cat_y_offset = 1 for agreement with the SExtractor catalog.)
    And, added a check to see if the centroids are consistently
    needing to be adjusted in the same direction to help find similar
    errors in the future.

--  Fixed bug to not try to write the Log into the fits header when
    the output is not a fits file.  

--  Fixed bug where some galaxies were spuriously flagged with the 
    SHEAR_LOCAL_MINIMUM flag.

--  Flag objects that are thought to be stars with the TOO_SMALL flag,
    and skip trying to compute a shear for them.

--  Increased the shear_aperture from 3 to 4 and shear_max_aperture from
    10 to 12 to try to reduce biases that occur with the smaller apertures.

--  Added ability to split the StarCatalog into two random samples
    for use with training and validation tests.

--  Added ability in scons to explicitly set the locations of required
    libraries: TMV, cfitsio, CCFits.
    
--  Added some functionality and some optional parameters that are 
    useful for testing.  (Like psf_force_sigma_p, etc.)

--  Changed the name of the isotropic signal-to-noise in the shear output 
    files from nu to shear_signal_to_noise.


-------------------------------------------------------------------------
v0.55  (2010-04-23)  One bug fix regarding centroiding, and added a couple
  features that are useful for testing.  Features in framework to simplify
  configurations and identify particilar wl.config with each run.

python/files.Runconfig:  Require wl_config as input to run configurations.
	the wl_config are now being kept under SVN in $DESFILES_DIR/wl.config
python/wlpipe: Run config through os.path.expandvars() to expand things
	environment variables like $DESFILES_DIR and $DESDATA.

src/MeasureShearAlgo.cpp: Fixed a subtle bug in the centroiding where
  the centroid used has a slight memory of the direction the centroid came 
  from.  If the input positions are consistently biased in the same 
  direction (e.g. by having a different definition of pixel (1,1), then
  this leads to a bias in the final shear values.

src/ShearCatalog_omp.cpp: Check whether there is such a bias in the input
  positions, and output a warning if the bias is more than 0.5 pixels,
  since this is something that should probably be corrected using the 
  cat_x_offset and cat_y_offset values.

example_config/wl.config:
  I (MJ) discovered that there is a 1 pixel offset in the input catalogs
  (as ES had thought there was, and I wrongly thought there wasn't).
  So I changed cat_x_offset and cat_y_offset to 1 in the config file.

src/Pixel.cpp: Moved the xOffset and yOffset correction back to getPixList
  rather than in ReadCatalog.  This way all the catalogs will maintain
  whatever convention is established in the original input catalog, and 
  we just correct for our different convention when we load the pixels.
  This is also needed for the WCS to work correctly and still get the
  right pixels.

src/EllipseSolver2.cpp: Fixed getB() to return the vector with the correct
  conjugation.  Nothing uses this value, but for testing it is useful to 
  have it give the right answer.

src/ShearCatalog.cpp: Added an parameter option, shear_output_psf,
  which adds the interpolated PSF at the location of the galaxy to the 
  output file.  Useful for testing.

src/InputCatalog.cpp, src/PsfCatalog.cpp, src/ShearCatalog.cpp, etc.:
  Removed the read() statement from constructors that just take a params
  argument.  Now it requires an explicit read() statement following the
  constructor.  This is really just to make the style consistent, since 
  FittedPsf already worked that way.  It makes it easier to write test 
  programs that read in a particular file without having to go through the
  makeName stuff with root, etc.

-------------------------------------------------------------------------
v0.54.1  (2010-04-04)  Small updates to framework on top of v0.54

python/wlpipe.py: 
	Added ability to send wl_config as a keyword to run_shear or
	run_multishear, or get it from the runconfig, with default
	falling back to $WL_DIR/etc/wl.config.  I'm putting these 
	config files under $DESFILES_DIR/wl.config

	Added && between all setup commands: if one fails all fail.

-------------------------------------------------------------------------
v0.54   (2010-3-14)  Mostly feature additions, rather than bug fixes.
        Especially features that are helpful in testing.

src/PsfCatalog_omp.cpp, src/ShearCatalog_omp.cpp, etc.: Added option to
  fix the centroid at the value given by the input catalog.
  Use psf_fix_centroid=true or shear_fix_centroid=true respectively.

  Also, when the centroid is not fixed, I now update the position in the 
  PsfCatalog or ShearCatalog object, so the new values are written out
  in the output file.

src/PsfCatalog_omp.cpp: Added option to force a particular value of sigma_p,
  rather than have the code calculate a value from the measured native-fit
  value of sigma for each star.
  Use psf_force_sigma_p=<value>

src/FittedPsf.cpp: Changed FittedPsf constructor to just take params and set
  up the sizes of the objects, but then have separate read and calculate 
  commands to either read from disk or calculate as appropriate.

src/Ellipse_omp.cpp: Switched deconvolving solver to use numeric J, rather
  than calculated J, since it turns out to be faster.

example_config/wl.config,
src/PsfCatalog_omp.cpp: Added option of specifying sigma_p directly,
  rather than calculating it from the data.  This is useful for 
  testing with simulation images that have a known sigma to test that
  the PSF measures the same thing that you put in.

Updated to work with TMV v0.63.

Added option of using Eigen rather than TMV 
  For lsst compatibility.
  Not recommended for DES, since TMV is faster (with or without BLAS).

-------------------------------------------------------------------------
v0.53   (2009-12-18)  Found some bugs when the DM team staged the DC5
        data to try out the multishear code.

src/ShearCatalog.cpp (and others): Make skyBounds from only good objects.
  Also, keep ra, dec in degrees rather than arcsec.

src/*Catalog.cpp: Changed CCfits::FITS construction to not load everything
  until it is needed.  (4th parameter true->false)

src/Pixel.cpp: added explicit copy constructor and op=, since implicit
  versions did the wrong thing.

src/BVec.cpp: added explicit op=, since implicit version did the wrong thing.

src/*Catalog.cpp: Improved precision of ASCII output.

src/NLSolver.cpp: Improved accuracy of numeric J calculation.

src/CrudeMeasure.cpp: Sped up estimate of sizes.

src/StarCatalog.cpp: Fixed bug in size estimation where it relied on a 
  return flag from ell.measure being set, but it isn't always set on
  failure.  So now explicitly set NATIVE_FAILED on failures.

example_config/wl.config,
src/MultiShearCatalog.cpp: Added support for explicitly listing 
  background image file in the input file list, rather than forming it
  from the main image name.

example_config/wl.config,
src/*Catalog.cpp: Added support for using a different hdu for each
  possible extension when multiple extensions are provided for in 
  input file.  Also, looks for the hdu here even if the filename is
  explicitly given.  Also, if explicit filename is not found, looks for 
  other names constructed from the extension list.

example_config/wl.config: Set to good values for DC5.

Updated style to match lsst style (mostly) to make it easier to keep the
two versions of the code in sync.

... other miscellaneous small bug fixes.

-------------------------------------------------------------------------
v0.52   (2009-11-25)  We have made a number of bug fixes since v0.51, and
        the DM team is about to start their multishear tests, so it seemed
	worth tagging the current stable version.  

python/*:  Converted to py3k.  Move to using JSON files instead of XML.

src/MultiShearCatalog.cpp: Enhanced fits writer to be more robust to 
  adding/removing columns.

src/ShearCatalog.cpp: Output ra/dec in degrees.  E.S.S. 2009-11-05

src/Ellipse*: Fixed bug in calls to Measure that still included an 
  obsolete desqa parameter, which was being confused with other bool 
  parameters.

src/FittedPSF.cpp: Fixed a bug in how the code deals with distortions in
  spherical coordinates. 
  Also added outlier rejection to PSF interpolation.

src/Transformation*: Fixed bug in coordinate transformation from pixels
  to radec.  Mike Jarvis, 2009-11

src/EllipseSolver* (and other): Corrected the calculation of the covariance
  matrix of the shear values.  See log entry for r4720 for more details.

src/Ellipse* (and others): Added a more robust S/N calculation, which is 
  output as the nu value.  This value had been output, but not ever set in
  the code, so they were all 0.  Now these are set to good significance 
  values that are independent of the PSF.  (Important for WL selection.)

src/dbg.h: Fixed a bug in mixing g++ with openmp.

src/StarCatalog.cpp: Added optional slower size measurement prior to running 
  findstars algorithm.  The parameter stars_use_shapelet_sigma governs
  whether to use the new algorith.  (The default is currently true.)

-------------------------------------------------------------------------
v0.51   (2009-10-16)  Incremental changes. The main reason for a new tag
	is that shear_dc4_input_format=False now; there was no easy keyword
	for this from the command line tool.

src/ConfigFile.  Added feature to allow +filename include an extra config
	file.  This also works on the command line.

python/wlpipe.py:  
	make_me_commandlist():  shear_dc4_input_format=False.  This will 
		require tagging a new version for the ME run based on wlse0001.
	check_shear(): new function to check the outputs of the SE shear
		measurements
	check_shear_qa(): Print the qa file for the input file info list.
	check_shear_input_images(): Try to read the image file.  Used for
		testing the fitserr messages which are the most common 
		failure mode.
	
	Moved to using the new wlse_path()/wlme_path() code everywhere.

	find_collated_coaddfiles()  work with more standardized inputs:  
		dataset, band, localid, serun=, rootdir=
	
	pbs creation routines, make sure tmv and esutil setups come after wl,
	in case they are not current.  Also allow dots by default for the 
	multishear code.
	
	Extensive additions to the doc string.

-------------------------------------------------------------------------
v0.50  (2009-10-10)  This tag represents substantive changes in the
	python framework and the C++ code.  Hence the jump from 0.43 to 0.50

src/MultiShearCatalog.cpp:  record maximum memory usage in header.

src/FittedPSF.cpp,PSFCatalog.cpp,ShearCatalog.cpp,StarCatalog.cpp: 
  Write serun if sent as a parameteer

src/fitparams.config:  Added merun,serun support.

src/FullPipeline.cpp: Print short status messages.

python/shear-run:  New executable for running SE code.

python/SConscript: Install shear-run

python/files.py: Simplified directory structure.  File routines for
	dealing with SE image and file lists.  PBS files and directories.

python/wlpipe.py: Rewrote SE shear pipelines to use standardized
	file routines and execution framework.  Made ME also conform
	to this.  Moved lots of file routines to files.py.  stdout,stderr
    now not redirected, QA stuff is redirected using writelog keyword.
	Write PBS files for ME and SE.  SE can be by ccd or exposurename.


Fixed memory leak when using regions in multi-epoch code combined with openmp.
The problem was basically that the memory was getting fragmented.  So 
the program over time was allocating much more memory than it was actually 
using.  The rest of the memory was presumably is pieces that were too small
for the allocator to use.  So top reported that the program was using much
more memory than valgrind (for example) said the program acutally had 
allocated.  In other words, not a normal memory leak, but it still caused
the computer to run out of memory.

The fix required two changes: 1) using boost::shared_ptr to minimize the 
number of reallocations, and 2) using a pool allocator to sequester
the memory used for the PixelList objects.  (I tried using the boost
pool_allocator, but it turns out to be not very good for our purpose, so 
I ended up writing my own.)

See the log entry for r4465 for more details.

Anyway, this fix means that the memory usage seems to scale more or less as
const + size^2.  The const is for some memory allocations that are needed
regargless of what size is used.  But the bulk of the memory usage is
proportionaly to the largest number of galaxies in one of the sections, which
scales approximately as size^2.


-------------------------------------------------------------------------
v0.43   (2009-09-30)  DC5 development version:

Added ability to use background maps for the sky values.
This was needed for the multishear program, since the objects detected
on the coadd image are not necessarily detected on the single-epoch
images.  Thus, there is no catalog that lists the sky values for these
objects, which is needed for doing any measurements on those objects.

Updated metadata file handling to keep track of more data and make it 
more robust to errors.

-------------------------------------------------------------------------
v0.42   (2009-09-23)  DC5 development version:

New framework for running multishear code.  Ready for full run on dc4
data at BNL.

-------------------------------------------------------------------------
v0.41   (2009-09-22)  DC5 development version:

Bug fixes since v0.40 from tests of rerunning single-epoch pipeline on 
reprocessed DC4 and GSN data.  Also, some work in fixing bugs in multishear.


-------------------------------------------------------------------------
v0.40   (2009-09-01)  DC5 development version:

This is the first version to include the multishear program.

See:
https://desweb.cosmology.uiuc.edu/confluence/display/WLPipe/WL+Code+and+Data+Products#WLCodeandDataProducts-Multishearprogram
for information on how to run multishear.

Current status:

Works with g++ versions 4.1, 4.3, 4.4.

Works with icpc version 10.1.

Works with icpc version 9 only WITH_OPENMP=false.
  - I am getting segmentation faults with openmp turned on.
  - I suspect this was a bug in icpc 9 that was fixed in version 10.

In all cases, I recommend compiling both tmv and wl with the scons option
WITH_LAPACK=false, especially if you are going to run withopenmp.  
Many lapack distributions are not properly thread-safe, and the native TMV
code is about as fast as lapack, so there isn't much advantage to using
lapack anyway.  So the safest thing to do is compile without it.

The default configuration uses a lot of memory (about 13 GB for a DC4
    i-band tile).

To use less memory, use command line option: "multishear_section_size=SIZE"
- SIZE=10 uses about 700 MB
- SIZE=20 uses about 2 GB
- SIZE=30 uses about 4.5 GB
- In general memory scales roughly as SIZE^2
The trade off, of course, is that using a larger SIZE value is more efficient,
so you should use as large a size as your machine can handle.
NB: The memory usage is dependent on the number of component single-epoch
images.  The numbers above are for i-band.  Less memory would be used for
g,r and more for z.



Known problem:

There seems to be a memory leak when running with openmp when using the 
above multishear_section_size option.
- It's a strage leak, since valgrind doesn't detect it.
- The symptom is that with each section, top reports more memory usage, 
  but valgrind reports that the memory usage is the same as for the 
  single-threaded run (ie. more or less constant as it progresses through 
  the sections).
- So it seems to be not a normal memory leak, but rather some kind of 
  problem where the OS is fragmenting the memory or something so that 
  the amount it blocks out for the program keeps growing.
- The leak didn't occur on my MacBook with either g++ 4.3 or 4.4,
  so this seems to have something to do with the OS.  Maybe the way 
  the OS allocates and releases memory.
- The leak doesn't seem to happen (or at least was much less severe)
  with icpc 10.1 with a block size of 5.  
  Not sure if this is true for icpc 10.1 with larger block sizes, since
  the computer I tested with 10.1 only has 1 GB of memory.  Also, it could
  be something about this computers OS, rather than icpc 10.1.  Hard
  to say.

Recommended solutions:
- Use a computer with enough memory to run with one section.
- Use openmp and hope that your OS doesn't show this symptom. (And then
  watch the memory usage in top to make sure you don't run out.)
- Use openmp, but set the section size to 5 or lower.
  (And still watch to make sure you don't run out of memory.)
- Don't use openmp.

I'm still trying to track down what in the code might be causing it, so
hopefully I'll find a way to fix it by the next tagged version.


Current To-do List:

- Implement baackground map method for calculating sky values.  I need to 
  see what kind of format the background map comes in, so this has
  to wait until the DM team has implemented the creation of the maps.
- I suspect it would be faster to calculate the first crude estimate of the 
  size and shape from the coadd image, rather than the se images.  
- Need to calculate the robust singal-to-noise estimate
- I don't think the shear covariance matrix is accurate, so I want to 
  implement a calculation that surveys the local likelihood to calculate
  the covariance.


-------------------------------------------------------------------------
v0.30.1   (2009-05-01)  DC5 development version:

Minor bugfixes since v0.30.

-------------------------------------------------------------------------
v0.30   (2009-04-14)  DC5 development version:

Added the program fullpipe, which runs all three of the programs
(findstars, measurepsf, measureshear) in one execution to save on 
disk I/O.  The input image and catalog are only read once.  Also, the
intermediate catalogs are written, but the unneccessary read step is skipped.

Also, changed all (well, most) of the fitsio stuff to CCFits for better
stability with the type names.  We had been having a number of problems
with long vs. int on different machines.  This seems to have fixed that.

Also enabled the code to more easily keep track of what versions each of the
components are, which can be put into the headers.

Other bug fixes.  Mostly related to errors on specific platforms.


-------------------------------------------------------------------------
v0.21   (2009-01-14)  This version was the final version used in DC4 testing.

It includes the programs:

1) findstars -- Measure a robust size of each object in the input
   catalog, and then use the resulting size-magnitude diagram
   to determine which objects are stars.

2) measurepsf -- Measure a shapelet-based description of the
   the PSF for each star, and do a polynomial interpolation 
   within the bounds of a single chip.

3) measureshear -- Measure the shear of each detected galaxy
   based on a single detection image.
   It uses the distortion given by the WCS information in the FITS
   header, and the single-image PSF interpolation found by measurepsf.

All input and output are done with binary fits files for the catalogs.

The parameter file is given as the first command line parameter for each
program.  See example_config/wl.config for a sample file.  Many aspects
of the programs are configurable, especially including various ways of
specifying the names of the files to use.  

All subsequent command line parameters are expected to be of the form:
param=value.  These are used to supersede the values found in the parameter
file.  For example, one might specify the names of the extensions and 
file directories in the configuration file, and just specify root=[whatever]
on the command line.  Or one might specify all of the filenames directly
on the command line and not use root at all.  (This is the mode that the 
DESDM team seems to prefer.)

The DES QA outputs are made if the parameter desqa=true.  These are output
either to stdout or to the logfile, if the logfile name is defined.

For more information about running the code and other details, see:
https://desweb.cosmology.uiuc.edu/confluence/display/WLPipe/WL+Code+and+Data+Products

-------------------------------------------------------------------------
v0.20   (2008-12-18)  A working version during DC4 development.


-------------------------------------------------------------------------
v0.10   (2008-12-08)  A working version during DC4 development.