Add ignorePrometheus Flag for integTest and docTest

DEVELOPER_GUIDE.rst

@@ -261,6 +261,8 @@ For faster local iterations, skip integration tests. ``./gradlew build -x integT
 
 For integration test, you can use ``-Dtests.class`` "UT full path" to run a task individually. For example ``./gradlew :integ-test:integTest -Dtests.class="*QueryIT"``.
 
+If Prometheus isn't available in your environment, you can skip downloading and starting it by adding ``-DignorePrometheus`` (or setting it to any value other than ``false``) to the command. For example ``./gradlew :integ-test:integTest -DignorePrometheus`` bypasses Prometheus setup and excludes Prometheus-specific integration tests, and ``./gradlew :doctest:doctest -DignorePrometheus`` skips the Prometheus-dependent doctest cases.
+
 To run the task above for specific module, you can do ``./gradlew :<module_name>:task``. For example, only build core module by ``./gradlew :core:build``.
 
 Troubleshooting

docs/dev/testing-doctest.md

@@ -52,15 +52,15 @@ For actually testing the code, the goal is to thoroughly test every case, rather
 
 ## 1.4 How to use doctest?
 ### 1.4.2 How to run existing doctest?
-Doctest runs with project build by `./gradlew build`. You can also only run doctest by `./gradlew doctest`
+Doctest runs with project build by `./gradlew build`. You can also only run doctest by `./gradlew doctest`. If a Prometheus instance isn't available locally, add `-DignorePrometheus` (or set the property to any value other than `false`) to skip Prometheus setup and the Prometheus-specific doctest scenarios.
 
 Make sure you don't have any OpenSearch instance running at `http://localhost:9200`
 
 ### 1.4.2 How to write documentation with doctest?
 1. If you want to add a new doc, you can add it to `docs` folder, under correct sub-folder, in `.rst` format. 
 > **Attention**: For code examples in documentation, a Mixing usage of `cli` and `bash` in one doc is not supported yet.
 2. Add your new doc file path to `docs/category.json` by its category
-3. Run doctest `./gradlew doctest` to see if your tests can pass
+3. Run doctest `./gradlew doctest` (optionally with `-DignorePrometheus`) to see if your tests can pass
 
 Currently, there is a `sample` folder under `docs` module to help you get started.
 

docs/user/ppl/admin/datasources.rst

@@ -261,6 +261,28 @@ PPL query for searching PROMETHEUS TABLES::
     +---------------+--------------+--------------------------------------------+------------+------+----------------------------------------------------+
 
 
+.. _datasources-prometheus-metadata:
+
+Fetch metadata for table in Prometheus datasource
+=================================================
+
+After a Prometheus datasource is configured, you can inspect the schema of any metric by running the ``describe`` command against the fully qualified table name. For example::
+
+PPL query::
+
+    PPL> describe my_prometheus.prometheus_http_requests_total;
+    fetched rows / total rows = 6/6
+    +---------------+--------------+--------------------------------+-------------+-----------+
+    | TABLE_CATALOG | TABLE_SCHEMA | TABLE_NAME                     | COLUMN_NAME | DATA_TYPE |
+    |---------------+--------------+--------------------------------+-------------+-----------|
+    | my_prometheus | default      | prometheus_http_requests_total | handler     | string    |
+    | my_prometheus | default      | prometheus_http_requests_total | code        | string    |
+    | my_prometheus | default      | prometheus_http_requests_total | instance    | string    |
+    | my_prometheus | default      | prometheus_http_requests_total | @timestamp  | timestamp |
+    | my_prometheus | default      | prometheus_http_requests_total | @value      | double    |
+    | my_prometheus | default      | prometheus_http_requests_total | job         | string    |
+    +---------------+--------------+--------------------------------+-------------+-----------+
+
 Limitations
 ===========
 

docs/user/ppl/cmd/describe.rst

@@ -67,22 +67,7 @@ PPL query::
     +----------------+
 
 
-Example 3: Fetch metadata for table in prometheus dataSource
-=========================================================
+Example 3: Fetch metadata for table in Prometheus datasource
+============================================================
 
-The example retrieves table info for ``prometheus_http_requests_total`` metric in prometheus dataSource.
-
-PPL query::
-
-    os> describe my_prometheus.prometheus_http_requests_total;
-    fetched rows / total rows = 6/6
-    +---------------+--------------+--------------------------------+-------------+-----------+
-    | TABLE_CATALOG | TABLE_SCHEMA | TABLE_NAME                     | COLUMN_NAME | DATA_TYPE |
-    |---------------+--------------+--------------------------------+-------------+-----------|
-    | my_prometheus | default      | prometheus_http_requests_total | handler     | string    |
-    | my_prometheus | default      | prometheus_http_requests_total | code        | string    |
-    | my_prometheus | default      | prometheus_http_requests_total | instance    | string    |
-    | my_prometheus | default      | prometheus_http_requests_total | @timestamp  | timestamp |
-    | my_prometheus | default      | prometheus_http_requests_total | @value      | double    |
-    | my_prometheus | default      | prometheus_http_requests_total | job         | string    |
-    +---------------+--------------+--------------------------------+-------------+-----------+
+See `Fetch metadata for table in Prometheus datasource <../admin/datasources.rst>`_ for more context.

doctest/build.gradle

@@ -19,6 +19,8 @@ def path = project(':').projectDir
 // temporary fix, because currently we are under migration to new architecture. Need to run ./gradlew run from
 // plugin module, and will only build ppl in it.
 def plugin_path = project(':doctest').projectDir
+String ignorePrometheusProp = System.getProperty("ignorePrometheus")
+boolean ignorePrometheus = ignorePrometheusProp != null && !ignorePrometheusProp.equalsIgnoreCase("false")
 
 task cloneSqlCli(type: Exec) {
     def repoDir = new File("${project.projectDir}/sql-cli")
@@ -61,6 +63,7 @@ task startPrometheus(type: SpawnProcessTask) {
     command "$projectDir/bin/prometheus/prometheus --storage.tsdb.path=$projectDir/bin/prometheus/data --config.file=$projectDir/bin/prometheus/prometheus.yml"
     ready 'TSDB started'
     pidLockFileName ".prom.pid.lock"
+    onlyIf { !ignorePrometheus && getOSFamilyType() != "windows" }
 }
 
 //evaluationDependsOn(':')
@@ -88,6 +91,7 @@ task doctest(type: Exec, dependsOn: ['bootstrap']) {
     if (debug == 'true') {
         environment 'DOCTEST_DEBUG', 'true'
     }
+    environment 'IGNORE_PROMETHEUS_DOCS', ignorePrometheus ? 'true' : 'false'
 
     if (docs) {
         def args = ['.venv/bin/python', 'test_docs.py']
@@ -121,14 +125,17 @@ task stopPrometheus(type: KillProcessTask) {
         file("$projectDir/bin/prometheus").deleteDir()
         file("$projectDir/bin/prometheus.tar.gz").delete()
     }
+    onlyIf { !ignorePrometheus && getOSFamilyType() != "windows" }
 }
 
 // Stop Prom AFTER Start Prom...
 if(getOSFamilyType() != "windows") {
     stopPrometheus.mustRunAfter startPrometheus
-    startOpenSearch.dependsOn startPrometheus
-    stopOpenSearch.finalizedBy stopPrometheus
-    startOpenSearch.finalizedBy stopPrometheus
+    if (!ignorePrometheus) {
+        startOpenSearch.dependsOn startPrometheus
+        stopOpenSearch.finalizedBy stopPrometheus
+        startOpenSearch.finalizedBy stopPrometheus
+    }
 }
 doctest.dependsOn startOpenSearch
 doctest.finalizedBy stopOpenSearch

doctest/test_docs.py

@@ -46,6 +46,10 @@
 }
 
 DEBUG_MODE = os.environ.get('DOCTEST_DEBUG', 'false').lower() == 'true'
+IGNORE_PROMETHEUS_DOCS = os.environ.get('IGNORE_PROMETHEUS_DOCS', 'false').lower() == 'true'
+PROMETHEUS_DOC_FILES = {
+    'user/ppl/cmd/showdatasources.rst'
+}
 
 
 def debug(message):
@@ -100,6 +104,13 @@ def load_categories(self, file_path):
         try:
             with open(file_path) as json_file:
                 categories = json.load(json_file)
+            if IGNORE_PROMETHEUS_DOCS:
+                categories = {
+                    category: [
+                        doc for doc in docs if doc not in PROMETHEUS_DOC_FILES
+                    ]
+                    for category, docs in categories.items()
+                }
             debug(f"Loaded {len(categories)} categories from {file_path}")
             return categories
         except Exception as e:

integ-test/build.gradle

@@ -53,6 +53,8 @@ String bwcVersion = baseVersion + ".0";
 String baseName = "sqlBwcCluster"
 String bwcFilePath = "src/test/resources/bwc/"
 String calciteCodegen = "$projectDir/src/test/java/codegen/"
+String ignorePrometheusProp = System.getProperty("ignorePrometheus")
+boolean ignorePrometheus = ignorePrometheusProp != null && !ignorePrometheusProp.equalsIgnoreCase("false")
 
 repositories {
     mavenCentral()
@@ -317,6 +319,7 @@ task startPrometheus(type: SpawnProcessTask) {
     }
     command "$projectDir/bin/prometheus/prometheus --storage.tsdb.path=$projectDir/bin/prometheus/data --config.file=$projectDir/bin/prometheus/prometheus.yml"
     ready 'TSDB started'
+    onlyIf { !ignorePrometheus }
 }
 
 task stopPrometheus(type: KillProcessTask) {
@@ -459,7 +462,7 @@ integTest {
     }
 
     dependsOn ':opensearch-sql-plugin:bundlePlugin'
-    if(getOSFamilyType() != "windows") {
+    if(!ignorePrometheus && getOSFamilyType() != "windows") {
         dependsOn startPrometheus
         finalizedBy stopPrometheus
     }
@@ -499,7 +502,7 @@ integTest {
         }
     }
 
-    if(getOSFamilyType() == "windows") {
+    if(getOSFamilyType() == "windows" || ignorePrometheus) {
         exclude 'org/opensearch/sql/ppl/PrometheusDataSourceCommandsIT.class'
         exclude 'org/opensearch/sql/ppl/ShowDataSourcesCommandIT.class'
         exclude 'org/opensearch/sql/ppl/InformationSchemaCommandIT.class'