match() live calculation and cross linking

[Martin-Rey]

Hi,

I am having trouble with the match() live calculation command. I am trying to match halos between different simulations that have been cross linked.

The links physically exist and are stored in the database (In [2] in the following python code). The match() command works as expected to find an earlier or later halo within the same simulation (In [3] in the code). When trying to match() across simulations, the live calculation returns no result (In [4] and [5]).

I am also seeing this behaviour in the web server, for which the "Find in another simulation feature also fails".

Any ideas on what is causing this?

Martin

In [1]: halo = tangos.get_halo("Halo1459_DMO_lowres/%101/halo_1")

In [2]: halo.all_links
Out[2]: 
[<HaloLink ptcls_in_common Halo1459_DMO_lowres/output_00101/halo_1 to Halo1459_DMO_lowres/output_00100/halo_1 weight=0.99>,
 <HaloLink ptcls_in_common Halo1459_DMO_lowres/output_00101/halo_1 to Halo1459_DMO_lowres_variancex09/output_00101/halo_1 weight=1.00>,
 <HaloLink ptcls_in_common Halo1459_DMO_lowres/output_00101/halo_1 to Halo1459_DMO_lowres_variancex11/output_00101/halo_1 weight=1.00>]

In [3]: halo.calculate('match("Halo1459_DMO_lowres/output_00100")')
Out[3]: <Halo 'Halo1459_DMO_lowres/output_00100/halo_1' | NDM=997808 Nstar=0 Ngas=0>

In [4]: halo.calculate('match("Halo1459_DMO_lowres_variancex11/output_00101")')
---------------------------------------------------------------------------
NoResultsError                            Traceback (most recent call last)
<ipython-input-4-d65a1a9e3b96> in <module>()
----> 1 halo.calculate('match("Halo1459_DMO_lowres_variancex11/output_00101")')

~/Documents/tangos/tangos/core/halo.py in calculate(self, calculation, return_description)
    143         from .. import live_calculation
    144         calculation = live_calculation.parser.parse_property_name_if_required(calculation)
--> 145         (value,), description = calculation.values_sanitized_and_description([self], Session.object_session(self))
    146         if len(value)==1:
    147             retval = value[0]

~/Documents/tangos/tangos/live_calculation/__init__.py in values_sanitized_and_description(self, halos, load_into_session)
    130         See values_sanitized for the definition of sanitized"""
    131         values, desc = self.values_and_description(halos)
--> 132         return self._sanitize_values(values, load_into_session), desc
    133 
    134     def values(self, halos):

~/Documents/tangos/tangos/live_calculation/__init__.py in _sanitize_values(self, unsanitized_values, load_into_session, no_results_raises)
    175         for x in output_values:
    176             if len(x)==0 and no_results_raises:
--> 177                 raise NoResultsError("Calculation %s returned no results" % self)
    178 
    179         if load_into_session is not None:

NoResultsError: Calculation match("Halo1459_DMO_lowres_variancex11/output_00101") returned no results


In [5]: halo.calculate('match("Halo1459_DMO_lowres_variancex11")')
---------------------------------------------------------------------------
NoResultsError                            Traceback (most recent call last)
<ipython-input-6-f201d26cb9ac> in <module>()
----> 1 halo.calculate('match("Halo1459_DMO_lowres_variancex11")')

~/Documents/tangos/tangos/core/halo.py in calculate(self, calculation, return_description)
    143         from .. import live_calculation
    144         calculation = live_calculation.parser.parse_property_name_if_required(calculation)
--> 145         (value,), description = calculation.values_sanitized_and_description([self], Session.object_session(self))
    146         if len(value)==1:
    147             retval = value[0]

~/Documents/tangos/tangos/live_calculation/__init__.py in values_sanitized_and_description(self, halos, load_into_session)
    130         See values_sanitized for the definition of sanitized"""
    131         values, desc = self.values_and_description(halos)
--> 132         return self._sanitize_values(values, load_into_session), desc
    133 
    134     def values(self, halos):

~/Documents/tangos/tangos/live_calculation/__init__.py in _sanitize_values(self, unsanitized_values, load_into_session, no_results_raises)
    175         for x in output_values:
    176             if len(x)==0 and no_results_raises:
--> 177                 raise NoResultsError("Calculation %s returned no results" % self)
    178 
    179         if load_into_session is not None:

NoResultsError: Calculation match("Halo1459_DMO_lowres_variancex11") returned no results

[apontzen]

Looks like there is a mismatch between the times of the two steps which are linked? To match across different simulations, the times must match to within a small tolerance.

[Martin-Rey]

There is indeed a small time mismatch between the two simulation time steps, at the level of 0.02 Gyr, though I am not sure how you can see that?

Where is the time tolerance set in the code to see if that fixes this issue?

[apontzen]

The step numbers mismatch, which is why I thought of this.

The tolerance is specified by SMALL_FRACTION in multi_hop.py (it should probably be moved to config.py)

[Martin-Rey]

Increasing SMALL_FRACTION does indeed solves this problem and the match() calculation returns the expected result.

I am still puzzled by the behaviour though. I add to increase SMALL_FRACTION to 1e-1 to obtain this result, even though my cross linked timesteps are displaced in time at the 0.1% fractional level, but 2e-2 absolute level.

I think there is a missing factor in the "across" direction keyword:

tangos/tangos/relation_finding/multi_hop.py

Lines 154 to 166 in 7f36946

	if self.directed is not None:
	directed = self.directed.lower()
	if self._one_simulation:
	recursion_filter &= (timestep_new.simulation_id == timestep_old.simulation_id)
	if directed == 'backwards':
	recursion_filter &= (timestep_new.time_gyr < timestep_old.time_gyr*(1.0-SMALL_FRACTION))
	elif directed == 'forwards':
	recursion_filter &= (timestep_new.time_gyr > timestep_old.time_gyr*(1.0+SMALL_FRACTION))
	elif directed == 'across':
	existing_timestep_ids = self.session.query(core.Halo.timestep_id).\
	select_from(self._link_orm_class).join(self._link_orm_class.halo_to).distinct()
	recursion_filter &= ~timestep_new.id.in_(existing_timestep_ids)
	recursion_filter &= sqlalchemy.func.abs(timestep_new.time_gyr - timestep_old.time_gyr) < SMALL_FRACTION

to make the comparison of time steps time relative rather than absolute, as for the "backwards" and "forwards" keywords.

Setting this parameter probably belongs to the config file.

[apontzen]

Yes, I agree the across mode should be fractional too and that the parameter belongs in config.py

[Martin-Rey]

Fixed by #94