Spec Insert

A program that insert API Components generated from the OpenSearch OpenAPI Spec into markdown files

Signed-off-by: Theo Truong <theotr@amazon.com>

spec_insert/.gitignore

@@ -0,0 +1,2 @@
+opensearch-openapi.yaml
+rspec_examples.txt

spec_insert/.rspec

@@ -0,0 +1 @@
+--require spec_helper

spec_insert/.rubocop.yml

@@ -0,0 +1,25 @@
+require: rubocop-rake
+AllCops:
+  Include:
+    - 'lib/**/*.rb'
+    - 'Rakefile'
+  NewCops: enable
+
+Metrics/CyclomaticComplexity:
+  Enabled: false
+Metrics/MethodLength:
+  Enabled: false
+Metrics/ParameterLists:
+  Enabled: false
+Metrics/AbcSize:
+  Enabled: false
+Metrics/PerceivedComplexity:
+  Enabled: false
+
+Layout/EmptyLineAfterGuardClause:
+  Enabled: false
+
+Style/MultilineBlockChain:
+  Enabled: false
+Style/SingleLineMethods:
+  Enabled: false

spec_insert/.ruby-version

@@ -0,0 +1 @@
+3.1.0

spec_insert/README.md

@@ -0,0 +1,91 @@
+# README: Spec Insert
+ - [What is this?](#what-is-this)
+ - [Installation](#Installation)
+ - [How to use](#how-to-use)
+   - [Insert Query Parameters](#insert-query-parameters)
+   - [Insert Path Parameters](#insert-path-parameters)
+   - [Insert Paths and HTTP Methods](#insert-paths-and-http-methods)
+ - [Ignored files and folders](#ignored-files-and-folders) 
+
+## What is this?
+This program allows you to insert API components generated from the OpenSearch Specification into this repository's markdown files. It's still underdevelopment, and many features are not yet implemented. This document will be updated as the program evolves.
+
+## Installation
+1. Clone this repository.
+2. Change to the `spec_insert` directory.
+3. Install Ruby 3.1.0 or later.
+4. Install the required gems by running `bundle install`.
+
+## How to use
+Edit your markdown file and insert the following snippet where you want the API components to be inserted:
+```markdown
+<!-- spec_insert_start 
+api: <API_NAME>
+component: <COMPONENT_NAME>
+other_param: <OTHER_PARAM>
+-->
+
+This is where the API component will be inserted.
+Everything between the `spec_insert_start` and `spec_insert_end` tags will be overwritten.
+
+<!-- spec_insert_end -->
+```
+
+Then run the following Rake commands to download the latest OpenSearch Specification and insert the API components into the markdown files:
+```shell
+rake download_spec
+rake insert_spec
+```
+
+### Insert Query Parameters
+To insert query parameters table of the `cat.indices` API, use the following snippet:
+```markdown
+<!-- spec_insert_start
+api: cat.indices
+component: query_parameters
+-->
+<!-- spec_insert_end -->
+```
+
+- This will insert the query parameters of the `cat.indices` API into the markdown file 3 default columns: `Parameter`, `Type`, and `Description`. There are 5 columns that can be inserted: `Parameter`, `Type`, `Description`, `Required`, and `Default`. When `Required`/`Default` is not chosen, the info will be written in the `Description` column.
+- This component accepts `include_global` (boolean, default to `false`) argument to include global query parameters in the table.
+- This component accepts `include_deprecated` (boolean, default to `true`) argument to include deprecated parameters in the table.
+- This component accepts `pretty` (boolean, default to `false`) argument to render the table in the pretty format instead of the compact format.
+
+```markdown
+<!-- spec_insert_start
+api: cat.indices
+component: query_parameters
+include_global: true
+include_deprecated: false
+pretty: true
+-->
+<!-- spec_insert_end -->
+```
+
+### Insert Path Parameters
+
+To insert path parameters table of the `indices.create` API, use the following snippet:
+```markdown
+<!-- spec_insert_start
+api: indices.create
+component: path_parameters
+-->
+<!-- spec_insert_end -->
+```
+
+This table behaves the same as the query parameters table except that it does not accept the `include_global` argument.
+
+### Insert Paths and HTTP Methods
+
+To insert paths and HTTP methods of the `search` API, use the following snippet:
+```markdown
+<!-- spec_insert_start
+api: search
+component: paths_and_http_methods
+-->
+<!-- spec_insert_end -->
+```
+
+### Ignored files and folders
+The program will ignore all markdown files whose names are in ALL CAPS. On top of that, you can also add files and folders you want to the [ignored.txt](./ignored.txt) file. Each line in the file should be the name of a file or folder you want to ignore.

spec_insert/Rakefile

@@ -0,0 +1,23 @@
+# frozen_string_literal: true
+
+require 'rake'
+
+desc 'Download the OpenSearch API specification'
+task :download_spec do
+  sh 'curl -L -X GET ' \
+     'https://github.com/opensearch-project/opensearch-api-specification' \
+     '/releases/download/main-latest/opensearch-openapi.yaml ' \
+     '-o opensearch-openapi.yaml'
+end
+
+desc 'Insert the OpenSearch API specification info into the documentation'
+task :insert_spec do
+  require_relative 'lib/spec_inserter'
+  require_relative 'lib/doc_processor'
+
+  SpecInserter.new(
+    root_folder: '../',
+    spec_file: './opensearch-openapi.yaml',
+    ignored: './ignored.txt'
+  ).insert_spec
+end

spec_insert/gemfile

@@ -0,0 +1,19 @@
+# SPDX-License-Identifier: Apache-2.0
+#
+# The OpenSearch Contributors require contributions made to
+# this file be licensed under the Apache-2.0 license or a
+# compatible open source license.
+
+# frozen_string_literal: true
+
+source 'https://rubygems.org'
+
+gem 'rake', '~> 13'
+gem 'activesupport', '~> 7'
+gem 'mustache', '~> 1'
+
+group :development, :test do
+  gem 'rspec'
+  gem 'rubocop', '~> 1.44', require: false
+  gem 'rubocop-rake', require: false
+end

spec_insert/ignored.txt

@@ -0,0 +1,5 @@
+# All files and folders listed below are ignored by the spec_insert program.
+# MD files whose names are in ALL CAPS are always ignored.
+
+spec_insert/
+release-notes/

spec_insert/lib/components/action.rb

@@ -0,0 +1,68 @@
+# SPDX-License-Identifier: Apache-2.0
+#
+# The OpenSearch Contributors require contributions made to
+# this file be licensed under the Apache-2.0 license or a
+# compatible open source license.
+
+# frozen_string_literal: true
+
+require_relative 'parameter'
+require_relative 'operation'
+
+# A collection of operations that comprise a single API Action
+# AKA operation-group
+class Action
+  # @param [SpecHash] spec Parsed OpenAPI spec
+  def self.actions=(spec)
+    operations = spec.paths.flat_map do |url, ops|
+      ops.filter_map { |verb, op| Operation.new(op, url, verb) unless op['x-ignorable'] }
+    end
+    @actions = operations.group_by(&:group).values.map { |ops| Action.new(ops) }.index_by(&:full_name)
+  end
+
+  # @return [Hash<String, Action>] API Actions indexed by operation-group
+  def self.actions
+    raise 'Actions not set' unless @actions
+    @actions
+  end
+
+  # @return [Array<Operation>] Operations in the action
+  attr_reader :operations
+
+  # @param [Array<Operation>] operations
+  def initialize(operations)
+    @operations = operations
+    @operation = operations.first
+    @spec = @operation&.spec
+  end
+
+  # @return [Array<Parameter>] Input arguments.
+  def arguments; @arguments ||= Parameter.from_operations(@operations.map(&:spec)); end
+
+  # @return [String] Full name of the action (i.e. namespace.action)
+  def full_name; @operation&.group; end
+
+  # return [String] Name of the action
+  def name; @operation&.action; end
+
+  # @return [String] Namespace of the action
+  def namespace; @operation&.namespace; end
+
+  # @return [Array<String>] Sorted unique HTTP verbs
+  def http_verbs; @operations.map(&:http_verb).uniq.sort; end
+
+  # @return [Array<String>] Unique URLs
+  def urls; @operations.map(&:url).uniq; end
+
+  # @return [String] Description of the action
+  def description; @spec&.description; end
+
+  # @return [Boolean] Whether the action is deprecated
+  def deprecated; @spec&.deprecated; end
+
+  # @return [String] Deprecation message
+  def deprecation_message; @spec['x-deprecation-message']; end
+
+  # @return [String] API reference
+  def api_reference; @operation&.external_docs&.url; end
+end

spec_insert/lib/components/operation.rb

@@ -0,0 +1,34 @@
+# SPDX-License-Identifier: Apache-2.0
+#
+# The OpenSearch Contributors require contributions made to
+# this file be licensed under the Apache-2.0 license or a
+# compatible open source license.
+
+# frozen_string_literal: true
+
+# An API Operation
+class Operation
+  # @return [Openapi3Parser::Node::Operation] Operation Spec
+  attr_reader :spec
+  # @return [String] URL
+  attr_reader :url
+  # @return [String] HTTP Verb
+  attr_reader :http_verb
+  # @return [String] Operation Group
+  attr_reader :group
+  # @return [String] API Action
+  attr_reader :action
+  # @return [String] API Namespace
+  attr_reader :namespace
+
+  # @param [Openapi3Parser::Node::Operation] spec Operation Spec
+  # @param [String] url
+  # @param [String] http_verb
+  def initialize(spec, url, http_verb)
+    @spec = spec
+    @url = url
+    @http_verb = http_verb.upcase
+    @group = spec['x-operation-group']
+    @action, @namespace = @group.split('.').reverse
+  end
+end

spec_insert/lib/components/parameter.rb

@@ -0,0 +1,94 @@
+# frozen_string_literal: true
+
+module ArgLocation
+  PATH = :path
+  QUERY = :query
+end
+
+# Represents a parameter of an API action
+class Parameter
+  # @return [String] The name of the parameter
+  attr_reader :name
+  # @return [String] The description of the parameter
+  attr_reader :description
+  # @return [Boolean] Whether the parameter is required
+  attr_reader :required
+  # @return [SpecHash] The JSON schema of the parameter
+  attr_reader :schema
+  # @return [String] Argument type in documentation
+  attr_reader :doc_type
+  # @return [String] The default value of the parameter
+  attr_reader :default
+  # @return [Boolean] Whether the parameter is deprecated
+  attr_reader :deprecated
+  # @return [String] The deprecation message
+  attr_reader :deprecation_message
+  # @return [String] The OpenSearch version when the parameter was deprecated
+  attr_reader :version_deprecated
+  # @return [ArgLocation] The location of the parameter
+  attr_reader :location
+
+  def initialize(name:, description:, required:, schema:, default:, deprecated:, deprecation_message:,
+                 version_deprecated:, location:)
+    @name = name
+    @description = description
+    @required = required
+    @schema = schema
+    @doc_type = get_doc_type(schema).gsub('String / List', 'List').gsub('List / String', 'List')
+    @default = default
+    @deprecated = deprecated
+    @deprecation_message = deprecation_message
+    @version_deprecated = version_deprecated
+    @location = location
+  end
+
+  # @param [SpecHash | nil] schema
+  # @return [String | nil] Documentation type
+  def get_doc_type(schema)
+    return nil if schema.nil?
+    union = schema.anyOf || schema.oneOf
+    return union.map { |sch| get_doc_type(sch) }.join(' / ') unless union.nil?
+    return 'Integer' if schema.type == 'integer'
+    return 'Float' if schema.type == 'number'
+    return 'Boolean' if schema.type == 'boolean'
+    return 'String' if schema.type == 'string'
+    return 'NULL' if schema.type == 'null'
+    return 'List' if schema.type == 'array'
+    'Object'
+  end
+
+  # @param [SpecHash] Full OpenAPI spec
+  def self.global=(spec)
+    @global = spec.components.parameters.filter { |_, p| p['x-global'] }.map { |_, p| from_parameters([p], 1) }
+  end
+
+  # @return [Array<Parameter>] Global parameters
+  def self.global
+    raise 'Global parameters not set' unless @global
+    @global
+  end
+
+  # @param [Array<SpecHash>] operations List of operations of the same group
+  # @return [Array<Parameter>] List of parameters of the operation group
+  def self.from_operations(operations)
+    operations.flat_map(&:parameters).filter { |param| !param['x-global'] }
+              .group_by(&:name).values.map { |params| from_parameters(params, operations.size) }
+  end
+
+  # @param [Array<SpecHash>] params List of parameters of the same name
+  # @param [Integer] opts_count Number of operations involved
+  # @return [Parameter] Single parameter distilled from the list
+  def self.from_parameters(params, opts_count)
+    param = params.first || SpecHash.new
+    schema = param&.schema || SpecHash.new
+    Parameter.new(name: param.name,
+                  description: param.description || schema.description,
+                  required: params.filter(&:required).size >= opts_count,
+                  schema: schema,
+                  default: param.default || schema.default,
+                  deprecated: param.deprecated || schema.deprecated,
+                  deprecation_message: param['x-deprecation-message'] || schema['x-deprecation-message'],
+                  version_deprecated: param['x-version-deprecated'] || schema['x-version-deprecated'],
+                  location: params.any? { |p| p.in == 'path' } ? ArgLocation::PATH : ArgLocation::QUERY)
+  end
+end