SkimLit positional embeddings

[ItayCoifman]

Hey, after walking through the SkimLit milestone project, I thought that implementing the positional encoding using "one-hot", is not the most effective way to do so.
The sentence position is a non-nominal variable, which means that line = 7 > line =4, if you divide the line number by the total lines feature you get a Ratio variable.
Treating these variables using "one-hot" is a bit wasteful (curse of dimensionality) and also truncates some of the information.
I suggest the next solution:
train_relative_line_pos =tf.constant(train_df['line_number'].to_numpy()/train_df['total_lines'].to_numpy()) val_relative_line_pos =tf.constant(val_df['line_number'].to_numpy()/val_df['total_lines'].to_numpy()) train_relative_line_pos[:10], train_relative_line_pos.shape

#3. Posistion_model relative_pos_inputs = layers.Input(shape = [1],dtype = tf.float32, name = 'Relative_pos_input') outputs = layers.Dense(32, activation = 'relu', name = 'Relative_pos_dense')(relative_pos_inputs) relative_pos_model = tf.keras.Model(inputs = relative_pos_inputs, outputs = outputs, name = "Relative_pos_model" )

...continue the same as did at "model_5"

Hope my idea makes some sense!
Love your curse!