Remove tests for preprocessing inside a functional model #1175
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This removes all testing for using string dtypes inside the
call
graph of a functional model. This will never work in a multi-backend fashion becausetorch
andjax
deliberately lackstring
dtypes.There is really only one case where adding tokenization into the call graph of a model makes sense--inference in tensorflow. And for that use case, we should switch to recommending an export flow (like
tf.keras.export.ExportArchive()
). Ditching these tests will simplify the code base, speed up testing, and allow us to focus on workflows that apply to all backends.Instead we, can focus on the follow options for applying tokenization:
Task
andtf.data
.tf.data
.tf.function
.For the export flow, it would look something like:
Plenty of planning will still need to be done on
export()
flows forkeras_core
, but that's probably out of scope for this PR.