-
Notifications
You must be signed in to change notification settings - Fork 3.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
refactor search index builder to store urn parts efficiently (#1937) #1972
Conversation
Maybe send out two commits for easier review: one for revert of the revert and the other for the actual fix after revert? |
dddc0fa
to
e9a0bab
Compare
Please wait for #1971. You can patch and rebase on top of it if you want to do the work now. |
Done. The first commit is revert of the revert (i.e. cherry-pick) and the second is the actual fix. |
...ta-builders/src/main/java/com/linkedin/metadata/builders/search/DataProcessIndexBuilder.java
Show resolved
Hide resolved
Neato. That'll make it easier to review :) |
Neato the robot? Didn't get the reference :) |
…-project#1937) Co-authored-by: Jyoti Wadhwani <jywadhwa@linkedin.com>
e9a0bab
to
1d00aff
Compare
https://en.wiktionary.org/wiki/neato#:~:text=(file)-,Interjection,Oh%2C%20neato! Just another way of saying "neat" lol |
Haha TIL |
...ta-builders/src/test/java/com/linkedin/metadata/builders/search/DatasetIndexBuilderTest.java
Outdated
Show resolved
Hide resolved
It's easier to write, has more specific asserts, and has better error messages. We did talk about this a few weeks ago :) |
1d00aff
to
29175df
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Tested the following way
curl 'http://localhost:8080/datasets?action=ingest' -X POST -H 'X-RestLi-Protocol-Version:2.0.0' --data '{"snapshot": {"aspects":[{"com.linkedin.common.Ownership":{"owners":[{"owner":"urn:li:corpuser:fbar","type":"DATAOWNER"}],"lastModified":{"time":0,"actor":"urn:li:corpuser:fbar"}}},{"com.linkedin.dataset.UpstreamLineage":{"upstreams":[{"auditStamp":{"time":0,"actor":"urn:li:corpuser:fbar"},"dataset":"urn:li:dataset:(urn:li:dataPlatform:foo,barUp,PROD)","type":"TRANSFORMED"}]}},{"com.linkedin.common.InstitutionalMemory":{"elements":[{"url":"https://www.linkedin.com","description":"Sample doc","createStamp":{"time":0,"actor":"urn:li:corpuser:fbar"}}]}},{"com.linkedin.schema.SchemaMetadata":{"schemaName":"FooEvent","platform":"urn:li:dataPlatform:foo","version":0,"created":{"time":0,"actor":"urn:li:corpuser:fbar"},"lastModified":{"time":0,"actor":"urn:li:corpuser:fbar"},"hash":"","platformSchema":{"com.linkedin.schema.KafkaSchema":{"documentSchema":"{\"type\":\"record\",\"name\":\"MetadataChangeEvent\",\"namespace\":\"com.linkedin.mxe\",\"doc\":\"Kafka event for proposing a metadata change for an entity.\",\"fields\":[{\"name\":\"auditHeader\",\"type\":{\"type\":\"record\",\"name\":\"KafkaAuditHeader\",\"namespace\":\"com.linkedin.avro2pegasus.events\",\"doc\":\"Header\"}}]}"}},"fields":[{"fieldPath":"foo","description":"Bar","nativeDataType":"string","type":{"type":{"com.linkedin.schema.StringType":{}}}}]}}],"urn":"urn:li:dataset:(urn:li:dataPlatform:foo,mBar,PROD)"}}'
Checklist