Refactor and simplify how TF adjustments are made in _find_new_matches_mode and _compare_two_records_mode

[RobinL]

This is the first step in a PR to remove the need for:

• linker._find_new_matches_mode
• linker._compare_two_records_mode

Once I began working on this, I realised that to remove these flags, we first need to simplify the tf logic.

It also resolves a longstanding bug with linker.compare_two_records whereby it would only work if term frequency tables were present, see here

#802 is therefore solved in Splink 4.

What does this do?

• In _find_new_matches_mode and _compare_two_records_mode new records are submitted by the user. Suppose term frequency adjustments are requested for the first_name column.
• We want to compute tf columns such as tf_first_name for these records
• If the relevant tf table (__splink__df_tf_first_name) is registered, then we can simply left join the table of new records
• If not, where do tf adjustments come from? We can derive them using:

select 
  __splink__compare_two_records_left.*, 
  nodes_tf__0."tf_first_name" 
from 
  __splink__compare_two_records_left 
  left join (
    select 
      distinct "first_name", 
      "tf_first_name" 
    from 
      __splink__df_concat_with_tf
  ) as nodes_tf__0 on __splink__compare_two_records_left."first_name" = nodes_tf__0."first_name"
)

• If __splink__df_concat_with_tf is not available, we simply create te table with no adjustments

select __splink__compare_two_records_left.*, null as "tf_first_name"
    from __splink__compare_two_records_left

What do we have to be careful about?

The compare_two_records function is more fiddly than it looks because...

... if the user just wants to compare two records, there's no guarantee they've provided Splink with a full input datasets. They might have just loaded in two records. If there is no full input dataset, we can't compute the tf tables. Instead, the user would have to provide them.

(See this comment)

This PR makes this possible:

linker = Linker([], settings, DuckDBAPI())
linker.compare_two_records(recs[0], recs[1]).as_pandas_dataframe()

i.e. we can use a trained model without needing to provide a big input dataset and computing __splink__df_concat_with_tf.

This makes real time scoring easier: It would be possible to do this in a live API:

tf_data_in = # data provided by caller of http API, e.g. grabbing tfs for the specific records from a prod postgres database
records in = # data provided by caller of http API
linker = Linker([], settings, DuckDBAPI())
linker.register_term_frequency_lookup(tf_data_in)
linker.compare_two_records(recs[0], recs[1]).as_pandas_dataframe()

Other notes

(You now get a warning as follows if the tf table doesn't exist. You still get prediction results, just without any tf adjustments)

No term frequencies found for column "first_name".
To apply term frequency adjustments, you need to register a lookup using `linker.register_term_frequency_lookup`.

A previous attempt at this was made here:
#1604

[RobinL]

I've moved the logic that derives tf columns from __splink__df_concat_with_tf to here

This logic is no longer used in linker.compute_tf_table because it's rare to need to 'backwards induce' tf tables in this way, and it adds complexity for little computational gain (tf tables are fast to compute)

[RobinL]

Previously, we were reusing logic in _join_tf_to_input_df_sql by performing string replacement operations on its outputs.

This was confusing. Also these two functions have diverged more with this PR, so it's no longer easy to have a single function

[RobinL]

This is a separate bug I found while doing this PR

[ADBond]

All seems sensible - looks good to me 👍