Add support for FLOAT to Python RowCoder #22626

TheNeuralBit · 2022-08-08T20:55:01Z

Part of #19815

This adds support for encoding the schema type FLOAT to Python's RowCoder. This is added in a new coder, SinglePrecisionFloatCoder, designed to be compatible with Java's FloatCoder.

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

TheNeuralBit · 2022-08-08T21:06:02Z

@lostluck it looks like Go might be using the same coder for single-precision floats as for double-precision, is that right? From the Go PreCommit:

13:58:13 --- FAIL: TestStandardCoders (0.03s)
13:58:13     --- FAIL: TestStandardCoders/beam:coder:row:v1#06 (0.00s)
13:58:13         fromyaml_test.go:42: Failed "{beam:coder:row:v1 
13:58:13             
13:58:13             �f_float�����$8c97b6c5-69e5-4733-907b-26cd8edae612 [] false}": err decoding "\x01\x00\x00\x00\x00\x00": decoding a struct { F_float float32 "beam:\"f_float\"" }
13:58:13             	caused by:
13:58:13             error decoding double field
13:58:13             	caused by:
13:58:13             EOF
13:58:13 FAIL

TheNeuralBit · 2022-08-08T21:21:26Z

Yes that does seem to be the case:

beam/sdks/go/pkg/beam/core/graph/coder/row_encoder.go

Lines 211 to 214 in cc0b446

    
           case reflect.Float32, reflect.Float64: 
        
           	return typeEncoderFieldReflect{encode: func(rv reflect.Value, w io.Writer) error { 
        
           		return EncodeDouble(rv.Float(), w) 
        
           	}}, nil

Would you be opposed to me sending a change to encode as single-precision in that case?

codecov · 2022-08-08T21:23:44Z

Codecov Report

Merging #22626 (bcb3822) into master (7a9bb76) will decrease coverage by 0.00%.
The diff coverage is 80.00%.

@@            Coverage Diff             @@
##           master   #22626      +/-   ##
==========================================
- Coverage   74.19%   74.19%   -0.01%     
==========================================
  Files         709      709              
  Lines       93499    93524      +25     
==========================================
+ Hits        69370    69387      +17     
- Misses      22852    22860       +8     
  Partials     1277     1277

Flag	Coverage Δ
python	`83.59% <80.00%> (-0.01%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...ks/python/apache_beam/coders/coders_test_common.py	`98.54% <ø> (ø)`
sdks/python/apache_beam/coders/coders.py	`88.00% <63.63%> (-0.35%)`	⬇️
sdks/python/apache_beam/coders/coder_impl.py	`93.91% <85.71%> (-0.06%)`	⬇️
sdks/python/apache_beam/coders/row_coder.py	`94.64% <100.00%> (+0.14%)`	⬆️
sdks/python/apache_beam/coders/slow_stream.py	`94.69% <100.00%> (+0.19%)`	⬆️
sdks/python/apache_beam/internal/metrics/metric.py	`93.00% <0.00%> (-1.00%)`	⬇️
...hon/apache_beam/runners/direct/test_stream_impl.py	`93.28% <0.00%> (-0.75%)`	⬇️
...eam/runners/portability/fn_api_runner/execution.py	`92.44% <0.00%> (-0.65%)`	⬇️
...ks/python/apache_beam/runners/worker/sdk_worker.py	`88.94% <0.00%> (-0.16%)`	⬇️
...on/apache_beam/runners/dataflow/dataflow_runner.py	`82.87% <0.00%> (-0.14%)`	⬇️
... and 3 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

github-actions · 2022-08-08T22:37:16Z

Assigning reviewers. If you would like to opt out of this review, comment assign to next reviewer:

R: @AnandInguva for label python.

Available commands:

stop reviewer notifications - opt out of the automated review tooling
remind me after tests pass - tag the comment author after tests pass
waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

TheNeuralBit · 2022-08-08T23:13:33Z

Well I tried to modify Go's RowCoder but couldn't manage to get the test working right. Instead I've just skipped this case, and filed #22629 to track.

TheNeuralBit · 2022-08-09T00:18:16Z

Run Java PreCommit

TheNeuralBit · 2022-08-10T16:27:39Z

@AnandInguva do you have time to review this?

AnandInguva · 2022-08-10T17:09:46Z

@TheNeuralBit I am on vacation. I will be back on Aug 22nd.

TheNeuralBit · 2022-08-10T18:04:30Z

assign to next reviewer

TheNeuralBit · 2022-08-10T18:46:26Z

R: @tvalentyn

github-actions · 2022-08-10T18:47:28Z

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control

TheNeuralBit · 2022-08-10T22:53:58Z

Oops Valentyn is also out

R: @ryanthompson591

TheNeuralBit · 2022-08-10T22:55:34Z

sdks/go/test/regression/coders/fromyaml/fromyaml.go

@@ -51,6 +51,7 @@ var unimplementedCoders = map[string]bool{
 var filteredCases = []struct{ filter, reason string }{
 	{"logical", "BEAM-9615: Support logical types"},
 	{"30ea5a25-dcd8-4cdb-abeb-5332d15ab4b9", "https://github.com/apache/beam/issues/21206: Support encoding position."},
+	{"8c97b6c5-69e5-4733-907b-26cd8edae612", "https://github.com/apache/beam/issues/22629: Support single-precision float."},


Noting there's a race with #22664. Whichever is merged later should remove this exception.

@jrmccluskey this test case is still failing in Go. It looks like it may just be a test infra issue, but I'm not able to debug on my own. Could you take a look?

I'm re-opening #22629 to track

Can you share the output from the test failure?

Sorry I should have shared it to begin with but it had scrolled out of my shell history and I was lazy 😬

I dropped the log in #22629 (comment)

TheNeuralBit · 2022-08-12T18:15:58Z

Run Go PreCommit

ryanthompson591 · 2022-08-12T18:22:13Z

model/fn-execution/src/main/resources/org/apache/beam/model/fnexecution/v1/standard_coders.yaml

@@ -455,6 +455,17 @@ examples:

 ---

+coder:
+  urn: "beam:coder:row:v1"
+  # f_float: float32


why f_float instead of float or float32?

f_float here is the name of a field in the schema. We could call it "float" or "float32" instead, but it's not that important. I just followed the f_ convention used elsewhere.

ryanthompson591 · 2022-08-12T18:34:42Z

model/fn-execution/src/main/resources/org/apache/beam/model/fnexecution/v1/standard_coders.yaml

@@ -455,6 +455,17 @@ examples:

 ---

+coder:
+  urn: "beam:coder:row:v1"


can you double check this is not a typo. For example with double this is:
beam:coder:double:v1

I would suspect this should be
beam:coder:float:v1

This is a reference to row coder:

beam/model/pipeline/src/main/proto/org/apache/beam/model/pipeline/v1/beam_runner_api.proto

Line 1062 in 3173b50

ROW = 13 [(beam_urn) = "beam:coder:row:v1"];

Row Coder is a complex coder that is parameterized by a Beam Schema. It encodes different types of values by deferring to other coders, like beam:coder:double:v1 for DOUBLE

* Add support for FLOAT to Python RowCoder * Skip the new test case in Go

github-actions bot added model python labels Aug 8, 2022

github-actions bot added the Next Action: Reviewers label Aug 8, 2022

TheNeuralBit mentioned this pull request Aug 8, 2022

[Bug]: Mismatch in single-precision float encoding between Java and Go RowCoder implementations #22629

Closed

github-actions bot added the go label Aug 8, 2022

github-actions bot added go and removed go labels Aug 8, 2022

TheNeuralBit mentioned this pull request Aug 9, 2022

Add BatchConverter implementations for pandas types, use Batched DoFns in DataFrame convert utilities #22575

Merged

jrmccluskey assigned jrmccluskey and unassigned jrmccluskey Aug 10, 2022

TheNeuralBit commented Aug 10, 2022

View reviewed changes

TheNeuralBit added 2 commits August 12, 2022 10:47

Add support for FLOAT to Python RowCoder

699ecf4

Skip the new test case in Go

bcb3822

TheNeuralBit force-pushed the python-float branch from df729fa to bcb3822 Compare August 12, 2022 17:48

ryanthompson591 approved these changes Aug 12, 2022

View reviewed changes

TheNeuralBit merged commit 63ba9c7 into apache:master Aug 12, 2022

MarcoRob pushed a commit to MarcoRob/beam that referenced this pull request Sep 5, 2022

Add support for FLOAT to Python RowCoder (apache#22626)

e016fc4

* Add support for FLOAT to Python RowCoder * Skip the new test case in Go

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for FLOAT to Python RowCoder #22626

Add support for FLOAT to Python RowCoder #22626

TheNeuralBit commented Aug 8, 2022

TheNeuralBit commented Aug 8, 2022

TheNeuralBit commented Aug 8, 2022 •

edited

Loading

codecov bot commented Aug 8, 2022 •

edited

Loading

github-actions bot commented Aug 8, 2022

TheNeuralBit commented Aug 8, 2022

TheNeuralBit commented Aug 9, 2022

TheNeuralBit commented Aug 10, 2022

AnandInguva commented Aug 10, 2022

TheNeuralBit commented Aug 10, 2022

TheNeuralBit commented Aug 10, 2022

github-actions bot commented Aug 10, 2022

TheNeuralBit commented Aug 10, 2022

TheNeuralBit Aug 10, 2022

TheNeuralBit Aug 12, 2022

jrmccluskey Aug 12, 2022

TheNeuralBit Aug 12, 2022

TheNeuralBit commented Aug 12, 2022

ryanthompson591 Aug 12, 2022

TheNeuralBit Aug 12, 2022

ryanthompson591 Aug 12, 2022

TheNeuralBit Aug 12, 2022

Add support for FLOAT to Python RowCoder #22626

Add support for FLOAT to Python RowCoder #22626

Conversation

TheNeuralBit commented Aug 8, 2022

GitHub Actions Tests Status (on master branch)

TheNeuralBit commented Aug 8, 2022

TheNeuralBit commented Aug 8, 2022 • edited Loading

codecov bot commented Aug 8, 2022 • edited Loading

Codecov Report

github-actions bot commented Aug 8, 2022

TheNeuralBit commented Aug 8, 2022

TheNeuralBit commented Aug 9, 2022

TheNeuralBit commented Aug 10, 2022

AnandInguva commented Aug 10, 2022

TheNeuralBit commented Aug 10, 2022

TheNeuralBit commented Aug 10, 2022

github-actions bot commented Aug 10, 2022

TheNeuralBit commented Aug 10, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TheNeuralBit commented Aug 12, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TheNeuralBit commented Aug 8, 2022 •

edited

Loading

codecov bot commented Aug 8, 2022 •

edited

Loading