Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix(cli): import data errors with strings, nulls #290

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

fix(cli): import data errors with strings, nulls #290

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
6030925
fix(cli): import data errors with strings, nulls
dtbuchholz Jun 18, 2024
2639b0c
fix(cli): remove hardcoded chain ID
dtbuchholz Jul 10, 2024
58661c7
refactor: wip parsing fixes
dtbuchholz Jul 19, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion packages/cli/src/utils.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -630,7 +630,13 @@ export const logger = {

export const parseCsvFile = async function (file: string): Promise<string[][]> {
return await new Promise(function (resolve, reject) {
const parser = parse();
// custom parsing
const parserOptions = {
skipEmptyLines: true,
trim: true,
};

const parser = parse(parserOptions);
const rows: any[] = [];

parser.on("readable", function () {
Expand Down
31 changes: 29 additions & 2 deletions packages/cli/test/import-data.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ const defaultArgs = [
TEST_PROJECT_ID,
];

describe("commands/import-data", function () {
describe.only("commands/import-data", function () {
this.timeout(30000 * TEST_TIMEOUT_FACTOR);

let table1: string;
Expand Down Expand Up @@ -74,7 +74,7 @@ describe("commands/import-data", function () {
restore();
});

test("can import with all values included", async function () {
test("can import a single row", async function () {
const csvStr = `id,val\n1,test_value`;
const csvFile = await temporaryWrite(csvStr, { extension: "csv" });

Expand All @@ -93,6 +93,33 @@ describe("commands/import-data", function () {
equal(successRes, `successfully inserted 1 row into ${defName1}`);
});

test.only("can import with quotes, escape chars, and multi line", async function () {
/* eslint-disable no-useless-escape */
const csvStr = `id,val
1,"I'll work"
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@joewagner up to you! i've trying to mess with the csv parser to get things to work with these different test cases. lmk if this is what you were thinking.

(I haven't been able to get the csv-parse to work properly tbh, and I'll probably need to add a SQLite-specific parser on top of it)

1,And I'll work
4,This ends with a backslash \\
7,"Escapes \'single quotes\'"
8,This "quote" works
`;
console.log(csvStr);
const csvFile = await temporaryWrite(csvStr, { extension: "csv" });

const consoleLog = spy(logger, "error");
const stdin = mockStd.stdin();
setTimeout(() => {
stdin.send("y\n");
}, 1000);
await yargs(["import-data", defName1, csvFile, ...defaultArgs])
.command<GlobalOptions>(mod)
.parse();

const res = consoleLog.getCall(0).firstArg;
const successRes = res.match(/^(.*)$/m)[1];

equal(successRes, `successfully inserted 7 rows into ${defName1}`);
});

test("can import with empty row values", async function () {
const csvStr = `id,val\n1,\n,test_value\n`;
const csvFile = await temporaryWrite(csvStr, { extension: "csv" });
Expand Down
Loading