FeaturizeText outputTokens uses a magical string to name a new column

[rogancarr]

When using OutputTokens=true, FeaturizeText creates a new column called ${OutputColumnName}_TransformedText. This isn't really well documented anywhere, and it's odd behavior. I suggest that we make the tokenized text column name explicit in the API.

My suggestion would be the following:

• Change OutputTokens = [bool] to OutputTokensColumn = [string], and a string.NullOrWhitespace(OutputTokensColumn) signifies that this column will not be created.

What do you all think?

[rogancarr]

@justinormont @TomFinley @sfilipi @Ivanidzo4ka Your thoughts?

[Ivanidzo4ka]

I would vote to hide that option in general and switch our examples to chain of estimators like we do here:

machinelearning/docs/samples/Microsoft.ML.Samples/Dynamic/WordEmbeddingTransform.cs

Line 28 in 9f87099

var wordsPipeline = ml.Transforms.Text.NormalizeText("NormalizedText", "SentimentText", keepDiacritics: false, keepPunctuations: false)

[rogancarr]

@Ivanidzo4ka I agree with what you're saying, but FeaturizeText is a "grab-bag" transform that does a bunch of stuff in one go, so unless we drop it from the APIs, we should fix this directly.

Update: I see what you're saying. You are saying to keep the transform, but turn off the "introspection" pieces.

[justinormont]

Exploding the text transform in to it's subparts will make the WordEmbedding quite error prone to use. For instance, users will often forget to lowercase their text, causing a silent degradation as it will still semi-work. I strongly prefer the explorability and simplicity of using a single transform instead of the seven or so sub-components.

On the original topic of creating an explicit tokens column:
Does the API allow for multiple columns to be handled independently? Eg: Title => NgramsTitle and Description => NgramsTitle? If so we also produce NgramsTitle_TransformedText & NgramsTitle_TransformedText; reducing to just token means we can only handle one input column.

[Ivanidzo4ka]

So we have this chunky transform which do lot of stuff, and have plenty of knobs. One of this knobs is OutputTokens. It's involves half of other knobs, and independent from extractors and normalization part.

I see main purpose of FeaturizeText transform to, you know, take text and produce something we can consume in trainers, which is vector of floats. And I like then transform or function in general does one thing.

OutputTokens for me feels like some dangling peace of functionality which we added because we could add it, but unnecessary because we should add it.

I understand it can be error prone to convert text into tokens, but no one stops us from introducing another transform which would do that (and inside would be same just chain of already existing transformers) and only that, rather than having one transform which does two things.

To answer @justinormont question regarding current behavior for multiple columns -> We just concat them together into one column and process it as one column.

[justinormont]

Another reason it's nice for the FeaturizeText to produce the tokens: the winning combo is generally keeping both the ngrams & the word embedding. Not just the word embedding by itself.

[abgoswam]

As suggested by @Ivanidzo4ka if users want to extract tokens as part of the pipeline , we can point them to use the TokenizeIntoWords featurizer estimator (which is an API designed specifically for that task)

machinelearning/src/Microsoft.ML.Transforms/Text/TextCatalog.cs

Lines 154 to 165 in 91a8703

	/// <summary>
	/// Tokenizes incoming text in <paramref name="inputColumnName"/>, using <paramref name="separators"/> as separators,
	/// and outputs the tokens as <paramref name="outputColumnName"/>.
	/// </summary>
	/// <param name="catalog">The text-related transform's catalog.</param>
	/// <param name="outputColumnName">Name of the column resulting from the transformation of <paramref name="inputColumnName"/>.</param>
	/// <param name="inputColumnName">Name of the column to transform. If set to <see langword="null"/>, the value of the <paramref name="outputColumnName"/> will be used as source.</param>
	/// <param name="separators">The separators to use (uses space character by default).</param>
	public static WordTokenizingEstimator TokenizeIntoWords(this TransformsCatalog.TextTransforms catalog,
	string outputColumnName,
	string inputColumnName = null,
	char[] separators = null)

For V1.0 I feel it makes sense to hide the OutputTokens option from the FeaturizeText API.

p.s. @rogancarr @Ivanidzo4ka @justinormont let me know if we have consensus on this -- i can get going on this issue

[rogancarr]

If we add a separate step for outputting tokens, then that is actually a completely different pass through the data to create tokens that we already have. I would recommend keeping the output option, but letting the user specify a name.

[abgoswam]

In other places of the API we have moved away from adding "smarts" (no-auto normalization, no-auto calibration). Should we adhere to this motto for FeaturizeText as well ?

If we expose OutputTokens in V1.0 we are fixing the behavior of FeaturizeText to always tokenize without giving us the flexibility to modify this behavior in the future.

@TomFinley

[justinormont]

@abgoswam in addition, there's various more steps needed than just TokenizeIntoWords to get the ngrams to the form the pretrained word embeddings need (diacritics, lowercasing, sometimes stopwords).