Add Support For Postgres Full Text Search tsvector Data Type #2727

anshap1719 · 2023-09-01T16:45:04Z

Relates to #729

abonander

Since it appears that supporting full text search does not require enabling any optional dependencies, it doesn't need a feature flag. It's perfectly fine for it to be unconditionally compiled, like with the support for ltree and lquery.

sqlx-core/Cargo.toml

anshap1719 · 2023-09-20T07:43:46Z

Hi @abonander, please approve the workflow run, it should now be fixed.

anshap1719 · 2023-10-04T11:50:32Z

Hi @abonander, Did you get a chance to look at this again yet?

abonander · 2023-10-12T01:11:43Z

I review PRs as I get to them. SQLx is only a small part of my full-time job. Pinging me just fills up my notifications, it won't get your PR reviewed any faster.

anshap1719 · 2023-10-12T12:13:17Z

@abonander I understand. Apologies for bothering you with those.

abonander · 2024-03-30T22:59:05Z

@anshap1719 can you please rebase and fix conflicts one last time? We're ready to merge this.

Note that the type mappings for the macros have moved to their corresponding database drivers:

sqlx/sqlx-postgres/src/type_checking.rs

Line 10 in 02c68a4

impl_type_checking!(

anshap1719 · 2024-03-31T07:41:32Z

Thanks @abonander, I have updated this PR.

abonander · 2024-03-31T10:03:18Z

Looks like cargo fmt needs to be run.

anshap1719 · 2024-03-31T11:02:11Z

@abonander Right, my bad. Done now.

sqlx-postgres/src/types/ts_vector.rs

anshap1719 · 2024-04-02T06:44:12Z

@wyhaya Thanks for the review. Will make the changes over the weekend and will also use all the cases you described as test cases.

anshap1719 · 2024-04-08T12:54:25Z

@wyhaya Updated everything. Thanks for the code as well.

sqlx-postgres/src/types/ts_vector.rs

sqlx-postgres/src/types/ts_query.rs

anshap1719 · 2024-04-21T08:49:56Z

Hi @abonander Just FYI, this PR no longer covers tsquery. I removed it since I realized it's not ready and needs a lot of work, and the tsvector is already done and we want to avoid blocking that.

sqlx-postgres/src/types/ts_vector.rs

…dge#2705

…2705

This reverts commit 279868a.

…adge#2705

…dge#729,launchbadge#2705

…#2705

…nchbadge#2705

…2705

…ctor Words launchbadge#729,launchbadge#2705

…aunchbadge#2705

KirDontsov · 2024-05-13T23:01:43Z

Oh, guys sorry, but I really need tsvector support, can you check and merge it? @abonander

abonander

Review comments plus a couple nice-to-haves:

At least a line or two of documentation on each public type.
Encode+decode test for Vec<TsVector>.

abonander · 2024-07-16T00:37:53Z

sqlx-postgres/src/types/ts_vector.rs

+}
+
+#[derive(Debug)]
+pub struct TsVector {


Nit: structs should be sorted in order of importance, it's weird for TsVector to be buried way down in the file. I would do TsVector, then Lexeme, then LexemeMeta.

Group the structs together, then the impls in the same order (this isn't a hard rule, but it's what I'm starting to prefer).

abonander · 2024-07-16T00:38:48Z

sqlx-postgres/src/types/ts_vector.rs

+    }
+}
+
+impl TryFrom<&[u8]> for TsVector {


This shouldn't be a public trait impl. A static method is fine.

abonander · 2024-07-16T00:39:07Z

sqlx-postgres/src/types/ts_vector.rs

+    }
+}
+
+impl TryInto<Vec<u8>> for &TsVector {


Same here. This should be a private method that takes &mut Vec<u8> and writes to it.

abonander · 2024-07-16T00:45:19Z

sqlx-postgres/src/types/ts_vector.rs

+#[derive(Debug)]
+pub struct ParseLexemeMetaError {
+    kind: IntErrorKind,
+}
+
+impl From<ParseIntError> for ParseLexemeMetaError {
+    fn from(value: ParseIntError) -> Self {
+        Self {
+            kind: value.kind().clone(),
+        }
+    }
+}
+
+#[allow(deprecated)]
+impl Display for ParseLexemeMetaError {
+    fn fmt(&self, f: &mut Formatter<'_>) -> fmt::Result {
+        self.description().fmt(f)
+    }
+}
+
+impl Error for ParseLexemeMetaError {
+    fn description(&self) -> &str {
+        match self.kind {
+            IntErrorKind::Empty => "cannot parse integer from empty string",
+            IntErrorKind::InvalidDigit => "invalid digit found in string",
+            IntErrorKind::PosOverflow => "number too large to fit in target type",
+            IntErrorKind::NegOverflow => "number too small to fit in target type",
+            IntErrorKind::Zero => "number would be zero for non-zero type",
+            _ => "unknown",
+        }
+    }
+}


All of this can be replaced by the following:

Suggested change

#[derive(Debug)]

pub struct ParseLexemeMetaError {

kind: IntErrorKind,

}

impl From<ParseIntError> for ParseLexemeMetaError {

fn from(value: ParseIntError) -> Self {

Self {

kind: value.kind().clone(),

}

}

}

#[allow(deprecated)]

impl Display for ParseLexemeMetaError {

fn fmt(&self, f: &mut Formatter<'_>) -> fmt::Result {

self.description().fmt(f)

}

}

impl Error for ParseLexemeMetaError {

fn description(&self) -> &str {

match self.kind {

IntErrorKind::Empty => "cannot parse integer from empty string",

IntErrorKind::InvalidDigit => "invalid digit found in string",

IntErrorKind::PosOverflow => "number too large to fit in target type",

IntErrorKind::NegOverflow => "number too small to fit in target type",

IntErrorKind::Zero => "number would be zero for non-zero type",

_ => "unknown",

}

}

}

#[derive(Debug, thiserror::Error)]

#[error("error parsing lexeme metadata: {inner}")]

pub struct ParseLexemeMetaError {

#[error(from)]

inner: ParseIntError,

}

abonander · 2024-07-16T00:48:15Z

sqlx-postgres/src/types/mod.rs

@@ -234,6 +234,8 @@ mod mac_address;
 #[cfg(feature = "bit-vec")]
 mod bit_vec;

+mod ts_vector;


If Lexeme and LexemeMeta are meant to be visible, this needs to be pub so their documentation gets rendered (I wouldn't reexport them here since it would be out of context).

abonander · 2024-07-16T00:48:35Z

sqlx-postgres/src/types/ts_vector.rs

+use std::str::FromStr;
+
+#[derive(Debug, Copy, Clone)]
+pub struct LexemeMeta {


Doesn't need to be pub since it's not used outside this module. Or if it's meant to be accessible then it needs some sort of getter from Lexeme.

It's also rather useless without at least getters for position and weight.

abonander · 2024-07-16T00:49:00Z

sqlx-postgres/src/types/ts_vector.rs

+}
+
+impl TsVector {
+    pub fn words(&self) -> &Vec<Lexeme> {


Suggested change

pub fn words(&self) -> &Vec<Lexeme> {

pub fn lexemes(&self) -> &[Lexeme] {

abonander · 2024-07-16T00:49:20Z

sqlx-postgres/src/types/ts_vector.rs

+}
+
+impl Lexeme {
+    pub fn word(&self) -> &str {


Suggested change

pub fn word(&self) -> &str {

pub fn as_str(&self) -> &str {

abonander · 2024-07-16T00:59:41Z

sqlx-postgres/src/types/ts_vector.rs

+impl From<u16> for LexemeMeta {
+    fn from(value: u16) -> Self {
+        let weight = (value >> 14) & 0b11;
+        let position = value & 0x3fff;
+
+        Self { weight, position }
+    }
+}
+
+impl From<&LexemeMeta> for u16 {
+    fn from(LexemeMeta { weight, position }: &LexemeMeta) -> Self {
+        let mut lexeme_meta = 0u16;
+        lexeme_meta = (weight << 14) | (position & 0x3fff);
+        lexeme_meta = (position & 0xc00) | (weight & 0x3fff);
+
+        lexeme_meta
+    }
+}


These impls lack context. Maybe from_packed() and to_packed() as private methods.

abonander · 2024-07-16T01:08:48Z

sqlx-postgres/src/types/mod.rs

@@ -251,6 +253,8 @@ pub use range::PgRange;
 #[cfg(any(feature = "chrono", feature = "time"))]
 pub use time_tz::PgTimeTz;

+pub use ts_vector::TsVector;


A listing should also be added to the table at line 23 above.

abonander · 2024-07-22T22:25:08Z

Closing due to inactivity. Feel free to reopen as a new PR, but please address my review comments.

anshap1719 · 2024-07-23T04:40:59Z

@abonander I don't think I will. Thanks.

anshap1719 marked this pull request as ready for review September 4, 2023 12:33

anshap1719 changed the title ~~Add Support For Postgres Full Text Search~~ Add Support For Postgres Full Text Search Data Types Sep 4, 2023

abonander requested changes Sep 12, 2023

View reviewed changes

sqlx-core/Cargo.toml Outdated Show resolved Hide resolved

anshap1719 requested a review from abonander September 17, 2023 08:36

anshap1719 closed this Nov 23, 2023

anshap1719 reopened this Feb 1, 2024

anshap1719 force-pushed the main branch from eeee2fd to dcec1dd Compare February 1, 2024 14:01

anshap1719 force-pushed the main branch from dcec1dd to 0c357cb Compare March 31, 2024 07:24

wyhaya reviewed Mar 31, 2024

View reviewed changes

sqlx-postgres/src/types/ts_vector.rs Show resolved Hide resolved

sqlx-postgres/src/types/ts_vector.rs Outdated Show resolved Hide resolved

sqlx-postgres/src/types/ts_vector.rs Outdated Show resolved Hide resolved

sqlx-postgres/src/types/ts_vector.rs Outdated Show resolved Hide resolved

anshap1719 force-pushed the main branch from f2ce3da to af22cd6 Compare April 8, 2024 12:51

anshap1719 requested a review from wyhaya April 8, 2024 12:54

wyhaya reviewed Apr 8, 2024

View reviewed changes

sqlx-postgres/src/types/ts_vector.rs Outdated Show resolved Hide resolved

sqlx-postgres/src/types/ts_query.rs Outdated Show resolved Hide resolved

anshap1719 requested a review from wyhaya April 21, 2024 08:50

anshap1719 changed the title ~~Add Support For Postgres Full Text Search Data Types~~ Add Support For Postgres Full Text Search tsvector Data Type Apr 21, 2024

wyhaya suggested changes Apr 23, 2024

View reviewed changes

sqlx-postgres/src/types/ts_vector.rs Show resolved Hide resolved

sqlx-postgres/src/types/ts_vector.rs Show resolved Hide resolved

anshap1719 added 4 commits April 28, 2024 15:29

WIP: Add Postgres TsVector And TsQuery Types launchbadge#729,launchba…

7cc0566

…dge#2705

WIP: Add Initial Decoding For TsVector launchbadge#729,launchbadge#2705

b6a93ca

WIP: Add Link To TsVector Binary Data Format Information launchbadge#…

a2578ed

…2705

WIP: Add Initial Encoding For TsVector launchbadge#729,launchbadge#2705

b603365

anshap1719 added 13 commits April 28, 2024 15:29

WIP: Remove Unimplemented TsQuery Definitions launchbadge#2705

b98a3eb

Revert "WIP: Remove Unimplemented TsQuery Definitions launchbadge#2705"

c4b2274

This reverts commit 279868a.

WIP: Add Support For Decoding And Encoding TsQuery launchbadge#2705

6abc12f

WIP: Add Missing Exports For TsQuery launchbadge#2705

612a452

WIP: Fix Build Issues, Add Tests For TsVector launchbadge#729,launchb…

8071cac

…adge#2705

Remove Feature Flag For Postgres Full Text Search Data Types launchba…

032f8ba

…dge#729,launchbadge#2705

Update Type For Pg Argument Buffer launchbadge#729,launchbadge#2705

8166072

Run cargo fmt

1ce2f7d

Add Support For Weights In Word Positions launchbadge#729,launchbadge…

bf67dfa

…#2705

Fix Clippy Lint Warning launchbadge#729,launchbadge#2705

65a8267

Add Handling For Whitespace Inside TSVector Words launchbadge#729,lau…

5ac77d0

…nchbadge#2705

Remove Incomplete TSQuery Implementation launchbadge#729,launchbadge#…

a85c27f

…2705

Add Proper Handling For Escaped And Non-Escaped Single Quotes In TSVe…

570216b

…ctor Words launchbadge#729,launchbadge#2705

anshap1719 force-pushed the main branch from 4163aac to 570216b Compare April 28, 2024 10:45

anshap1719 requested a review from wyhaya April 28, 2024 10:45

Remove All TSQuery Related DB Type Info Declarations launchbadge#729,l…

8e8c0f1

…aunchbadge#2705

abonander requested changes Jul 16, 2024

View reviewed changes

abonander closed this Jul 22, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Support For Postgres Full Text Search tsvector Data Type #2727

Add Support For Postgres Full Text Search tsvector Data Type #2727

anshap1719 commented Sep 1, 2023 •

edited

Loading

abonander left a comment

anshap1719 commented Sep 20, 2023

anshap1719 commented Oct 4, 2023

abonander commented Oct 12, 2023

anshap1719 commented Oct 12, 2023

abonander commented Mar 30, 2024

anshap1719 commented Mar 31, 2024

abonander commented Mar 31, 2024

anshap1719 commented Mar 31, 2024

anshap1719 commented Apr 2, 2024

anshap1719 commented Apr 8, 2024

anshap1719 commented Apr 21, 2024

KirDontsov commented May 13, 2024

abonander left a comment

abonander Jul 16, 2024

abonander Jul 16, 2024

abonander Jul 16, 2024

abonander Jul 16, 2024

abonander Jul 16, 2024 •

edited

Loading

abonander Jul 16, 2024

abonander Jul 16, 2024

abonander Jul 16, 2024

abonander Jul 16, 2024

abonander Jul 16, 2024

abonander Jul 16, 2024

abonander commented Jul 22, 2024

anshap1719 commented Jul 23, 2024

	pub fn words(&self) -> &Vec<Lexeme> {
	pub fn lexemes(&self) -> &[Lexeme] {

Add Support For Postgres Full Text Search tsvector Data Type #2727

Add Support For Postgres Full Text Search tsvector Data Type #2727

Conversation

anshap1719 commented Sep 1, 2023 • edited Loading

abonander left a comment

Choose a reason for hiding this comment

anshap1719 commented Sep 20, 2023

anshap1719 commented Oct 4, 2023

abonander commented Oct 12, 2023

anshap1719 commented Oct 12, 2023

abonander commented Mar 30, 2024

anshap1719 commented Mar 31, 2024

abonander commented Mar 31, 2024

anshap1719 commented Mar 31, 2024

anshap1719 commented Apr 2, 2024

anshap1719 commented Apr 8, 2024

anshap1719 commented Apr 21, 2024

KirDontsov commented May 13, 2024

abonander left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

abonander Jul 16, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

abonander commented Jul 22, 2024

anshap1719 commented Jul 23, 2024

anshap1719 commented Sep 1, 2023 •

edited

Loading

abonander Jul 16, 2024 •

edited

Loading