Add support for `-f` command to run commands from files #120

alamb · 2024-09-08T17:21:03Z

This is my (very tardy) downpayment to help dft

It adds a simple "run commands from a file" mode via the -f argument and adds some basic tests

Here is an example:

andrewlamb@Andrews-MacBook-Pro-2:~/Software/datafusion-tui$ cat ~/Downloads/q.sql
show all
andrewlamb@Andrews-MacBook-Pro-2:~/Software/datafusion-tui$ dft -f ~/Downloads/q.sql  | head
+-------------------------------------------------------------------------+---------------------------+
| name                                                                    | value                     |
+-------------------------------------------------------------------------+---------------------------+
| datafusion.catalog.create_default_catalog_and_schema                    | true                      |
| datafusion.catalog.default_catalog                                      | datafusion                |
| datafusion.catalog.default_schema                                       | public                    |
| datafusion.catalog.format                                               |                           |
| datafusion.catalog.has_header                                           | false                     |
| datafusion.catalog.information_schema                                   | false                     |
| datafusion.catalog.location                                             |                           |

Things I hope to do next:

Add support for -c <sql string>
Move the execution machinery into its own module (out of app) and start adding other integrations

alamb · 2024-09-08T17:21:37Z

README.md


-DataFusion-tui provides an extensible terminal based data analysis tool that uses [DataFusion](https://github.com/apache/arrow-datafusion) as query execution engines. It has drawn inspiration and several features from `datafusion-cli`. In contrast to `datafusion-cli` a focus of `dft` is to provide an interface for leveraging DataFusions extensibility (for example connecting to `ObjectStore`s or querying custom `TableProvider`s).
+`dft` provides two interfaces to the [DataFusion](https://github.com/apache/arrow-datafusion) query execution engine:


I also took a shot at updating the description here to describe the two different UIs. Let me know what you think

Looks good, and I like the scripting option. Only comment from the updated description is that right now there isnt any MySQL integration.

alamb · 2024-09-08T17:22:02Z

src/app/mod.rs

@@ -300,32 +301,75 @@ pub async fn run_app(cli: cli::DftCli, state: state::AppState<'_>) -> Result<()>
    info!("Running app with state: {:?}", state);
    let mut app = App::new(state, cli.clone());

-    match &cli.command {


this just removes a seemingly unecessary level of wrapping of Clap argument parsing

alamb · 2024-09-08T17:22:28Z

tests/cli.rs

@@ -0,0 +1,236 @@
+// Licensed to the Apache Software Foundation (ASF) under one


here are a bunch of tests for the CLI

We can probably move some of these to a common or something similar test since it covers both the CLI and TUI

At the moment I think it only covers the "CLI" as I have defined it (aka running dft without any interaction)

In terms of testing the TUI I do think that would be valuable too (like fire up the tui and send key strokes like q and x to it). I think there are some frameworks that could help here, but I haven't looked into them

What I was thinking is that we could use the "CLI" mode (aka batch mode) to test the various integrations like delta lake and other table providers and focus the UI testing on just the user interaction (rather than the details of execution)

I think that makes sense. we'll need to do some testing of the execution triggered by the UI but it definitely doesnt need to be concerned with details - more just making sure the results make it into app state.

alamb · 2024-09-08T17:22:53Z

src/app/mod.rs

+
+/// run and execute SQL statements and commands from a file, against a context
+/// with the given print options
+pub async fn exec_from_file(ctx: &ExecutionContext, file: &Path) -> Result<()> {


This is the "process commands from a file" code -- it is based on the code in datafusion-cli

matthewmturner · 2024-09-09T11:53:29Z

README.md

+## `dft` CLI
+
+The `dft` CLI is a scriptable engine for executing queries from files.  It is used in a similar manner to `datafusion-cli` but with the added benefit of being able to query from multiple data sources.
+
+For example you can run the contents of `query.sql` with 
+
+```shell
+$ dft -f query.sql
+```


Very cool, love this addition

matthewmturner · 2024-09-09T11:54:20Z

README.md

+- "Catalog File" support
+  - Save table definitions *and* data
+  - Save parquet metadata from remote object stores


I think it would be useful if we had an issue that provided more details on the aspirations here

Filed #122 and will link here

matthewmturner · 2024-09-09T11:56:01Z

src/app/execution.rs

@@ -121,7 +121,8 @@ impl ExecutionContext {
        &self.session_ctx
    }

-    pub async fn execute_stream_sql(&mut self, query: &str) -> Result<()> {
+    /// Executes the specified query and prints the result to stdout
+    pub async fn execute_stream_sql(&self, query: &str) -> Result<()> {
        let df = self.session_ctx.sql(query).await.unwrap();
        let physical_plan = df.create_physical_plan().await.unwrap();
        // We use small batch size because web socket stream comes in small increments (each


I need to remove this comment

matthewmturner · 2024-09-09T12:13:42Z

src/cli/mod.rs

+        help = "Execute commands from file(s), then exit",
+        value_parser(parse_valid_file)
+    )]
+    pub file: Vec<PathBuf>,


Suggested change

pub file: Vec<PathBuf>,

pub files: Vec<PathBuf>,

good call -- this was just copied from datafusion-cli. files is better

matthewmturner · 2024-09-09T12:19:01Z

src/cli/mod.rs

+dft provides a rich terminal UI as well as a broad array of pre-integrated
+data sources, formats to make querying and analyzing data from
+various sources.


Suggested change

dft provides a rich terminal UI as well as a broad array of pre-integrated

data sources, formats to make querying and analyzing data from

various sources.

dft provides a rich terminal UI as well as a broad array of pre-integrated

data sources and formats for querying and analyzing data from

various sources.

matthewmturner · 2024-09-09T12:45:24Z

tests/cli.rs

@@ -0,0 +1,236 @@
+// Licensed to the Apache Software Foundation (ASF) under one


We can probably move some of these to a common or something similar test since it covers both the CLI and TUI

matthewmturner · 2024-09-09T12:48:02Z

Thanks for the contribution Andrew! I think this is a great addition. I added some comments / feedback, none of which need to addressed now. I also tested the app locally with all features (with and without a config file) to ensure it still worked and it still ran as expected. Either one of us can decide to work on the comments in follow on PRs.

alamb · 2024-09-09T18:24:12Z

Implemented follow ons in #123

@matthewmturner

Implement suggestions from @matthewmturner in #120

alamb added 5 commits September 8, 2024 10:35

Improve application description

d6e6efa

simplify CLI argument handling

21938a9

Implement rudimentary execution from a file

40e3cbf

Add basic CLI testing

7f6cc8e

update README

c547879

alamb commented Sep 8, 2024

View reviewed changes

alamb requested a review from matthewmturner September 8, 2024 17:23

matthewmturner reviewed Sep 9, 2024

View reviewed changes

matthewmturner merged commit 40b0901 into datafusion-contrib:main Sep 9, 2024
3 checks passed

alamb deleted the alamb/file branch September 9, 2024 12:57

alamb mentioned this pull request Sep 9, 2024

Start setting up tui tests #121

Merged

alamb added a commit to alamb/datafusion-dft that referenced this pull request Sep 9, 2024

Minor: apply comments from datafusion-contrib#120 code review

37bfd73

alamb mentioned this pull request Sep 9, 2024

Minor: apply comments from #120 code review #123

Merged

matthewmturner pushed a commit that referenced this pull request Sep 9, 2024

Minor: apply comments from #120 code review (#123)

f27bf6d

Implement suggestions from @matthewmturner in #120

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for `-f` command to run commands from files #120

Add support for `-f` command to run commands from files #120

alamb commented Sep 8, 2024

alamb Sep 8, 2024

matthewmturner Sep 9, 2024

alamb Sep 8, 2024

alamb Sep 8, 2024

matthewmturner Sep 9, 2024

alamb Sep 9, 2024

matthewmturner Sep 9, 2024

alamb Sep 8, 2024 •

edited

Loading

matthewmturner Sep 9, 2024

matthewmturner Sep 9, 2024 •

edited

Loading

alamb Sep 9, 2024 •

edited

Loading

matthewmturner Sep 9, 2024

matthewmturner Sep 9, 2024

alamb Sep 9, 2024

matthewmturner Sep 9, 2024

matthewmturner Sep 9, 2024

matthewmturner commented Sep 9, 2024 •

edited

Loading

alamb commented Sep 9, 2024


		DataFusion-tui provides an extensible terminal based data analysis tool that uses [DataFusion](https://github.com/apache/arrow-datafusion) as query execution engines. It has drawn inspiration and several features from `datafusion-cli`. In contrast to `datafusion-cli` a focus of `dft` is to provide an interface for leveraging DataFusions extensibility (for example connecting to `ObjectStore`s or querying custom `TableProvider`s).
		`dft` provides two interfaces to the [DataFusion](https://github.com/apache/arrow-datafusion) query execution engine:

		@@ -0,0 +1,236 @@
		// Licensed to the Apache Software Foundation (ASF) under one

Add support for -f command to run commands from files #120

Add support for -f command to run commands from files #120

Conversation

alamb commented Sep 8, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alamb Sep 8, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

matthewmturner Sep 9, 2024 • edited Loading

Choose a reason for hiding this comment

alamb Sep 9, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

matthewmturner commented Sep 9, 2024 • edited Loading

alamb commented Sep 9, 2024

Add support for `-f` command to run commands from files #120

Add support for `-f` command to run commands from files #120

alamb Sep 8, 2024 •

edited

Loading

matthewmturner Sep 9, 2024 •

edited

Loading

alamb Sep 9, 2024 •

edited

Loading

matthewmturner commented Sep 9, 2024 •

edited

Loading