Skip to content
This repository has been archived by the owner on Nov 29, 2023. It is now read-only.

feat: include package parser developed in Semantics lab #409

Merged
merged 68 commits into from
Feb 10, 2022
Merged

feat: include package parser developed in Semantics lab #409

merged 68 commits into from
Feb 10, 2022

Conversation

lars-reimann
Copy link
Member

@lars-reimann lars-reimann commented Feb 10, 2022 •
edited
Loading

Summary of Changes

  • Include package-parser from sem21 repository
  • New top-level folders for different components

lars-reimann and others added 30 commits September 30, 2021 15:39
@lars-reimann
Initial commit
ee37bb3
@lars-reimann
chore: configure template
68ac8ab
@lars-reimann
chore: add starter code
5f360a4
@lars-reimann
build: add numpydoc as dependency
559a0f6
@lars-reimann
build: add spacy as dependency
2f0314b
@lars-reimann
docs: installation of poetry
e3f286b
@lars-reimann
docs: use Python 3.9
3bf71f9
@lars-reimann
feat: store source code of classes/functions and docstrings of classe…
d1b4055 
…s/functions/parameters
@lars-reimann
feat: store whether parameters as assigned by position or name or both
9391c2a
@dependabot
build(deps): bump scikit-learn from 1.0 to 1.0.1 in /package_parser (#12
abd215e 
)

Bumps [scikit-learn](https://github.com/scikit-learn/scikit-learn) from 1.0 to 1.0.1.
- [Release notes](https://github.com/scikit-learn/scikit-learn/releases)
- [Commits](scikit-learn/scikit-learn@1.0...1.0.1)

---
updated-dependencies:
- dependency-name: scikit-learn
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@dependabot
build(deps): bump spacy from 3.1.3 to 3.1.4 in /package_parser (#11)
75eccbd 
Bumps [spacy](https://github.com/explosion/spaCy) from 3.1.3 to 3.1.4.
- [Release notes](https://github.com/explosion/spaCy/releases)
- [Commits](explosion/spaCy@v3.1.3...v3.1.4)

---
updated-dependencies:
- dependency-name: spacy
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@dependabot
build(deps): bump amannn/action-semantic-pull-request (#8)
d1ad13f 
Bumps [amannn/action-semantic-pull-request](https://github.com/amannn/action-semantic-pull-request) from 3.4.2 to 3.4.6.
- [Release notes](https://github.com/amannn/action-semantic-pull-request/releases)
- [Changelog](https://github.com/amannn/action-semantic-pull-request/blob/master/CHANGELOG.md)
- [Commits](amannn/action-semantic-pull-request@v3.4.2...v3.4.6)

---
updated-dependencies:
- dependency-name: amannn/action-semantic-pull-request
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@dependabot
build(deps): bump astroid from 2.8.2 to 2.8.4 in /package_parser (#10)
daae4a5 
Bumps [astroid](https://github.com/PyCQA/astroid) from 2.8.2 to 2.8.4.
- [Release notes](https://github.com/PyCQA/astroid/releases)
- [Changelog](https://github.com/PyCQA/astroid/blob/main/ChangeLog)
- [Commits](pylint-dev/astroid@v2.8.2...v2.8.4)

---
updated-dependencies:
- dependency-name: astroid
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@lars-reimann
docs: remove community files that have a default
75e9858
@lars-reimann
ci: use reusable workflow to check format of PRs
87d09b7
@lars-reimann
ci: update mega-linter
dc6616f
@lars-reimann
ci: master -> main in mega-linter config
cbad180
@lars-reimann
feat: enhance extracted data for usage with API editor (#14)
a19b551 
* feat: add new fields to model

* fix: get everything running again

* feat: clean docstrings

* feat: extract imports and from-imports for modules

* fix: use qualified name of functions as key

* fix: some linter errors

* fix: remaining linter errors
@lars-reimann
fix: duplicate imports in extracted API data
4fa2a05
@lars-reimann
fix: skip nested classes for now
0f5964c
@lars-reimann
chore: add starting point for ontology development
92fcd4f
@lars-reimann
feat: handle properties correctly
6039f36 
Multiple functions with the same name can exist if they are marked with @Property, @x.setter, or @x.deleter. We, hence, need to refer to them by something other than their qualified name.
@aserge16
feat: Python model classes for dependencies (#15)
130eb9c 
* initial ontology commit

* 2 basic property subclass ideas

* added two more property subclasses

* moved ontology to predefined directory, fixed imports

* renamed property to dependency, added arithmetic dependency

* rework ontology model in code

* updated gitignore to ignore VSCode settings

* moved python-related changes to package_parser

* a stronger method for creating class instances instead of using a static method

* fixed potential lint error

* style: apply automatic fixes of linters

Co-authored-by: Aleksandr Sergeev <aleksandr.sergeev@tngtech.com>
Co-authored-by: aserge16 <aserge16@users.noreply.github.com>
@lars-reimann
ci: disable bandit, flake8, pylint
02d32d2
@dependabot
build(deps): bump spacy from 3.1.4 to 3.2.0 in /package_parser (#18)
c1f79cb 
Bumps [spacy](https://github.com/explosion/spaCy) from 3.1.4 to 3.2.0.
- [Release notes](https://github.com/explosion/spaCy/releases)
- [Commits](explosion/spaCy@v3.1.4...v3.2.0)

---
updated-dependencies:
- dependency-name: spacy
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@dependabot
build(deps): bump importlib-metadata in /package_parser (#19)
9c8a592 
Bumps [importlib-metadata](https://github.com/python/importlib_metadata) from 4.8.1 to 4.8.2.
- [Release notes](https://github.com/python/importlib_metadata/releases)
- [Changelog](https://github.com/python/importlib_metadata/blob/main/CHANGES.rst)
- [Commits](python/importlib_metadata@v4.8.1...v4.8.2)

---
updated-dependencies:
- dependency-name: importlib-metadata
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@dependabot
build(deps): bump megalinter/megalinter from 5.0.5 to 5.2.0 (#16)
6ea3e92 
Bumps [megalinter/megalinter](https://github.com/megalinter/megalinter) from 5.0.5 to 5.2.0.
- [Release notes](https://github.com/megalinter/megalinter/releases)
- [Changelog](https://github.com/megalinter/megalinter/blob/main/CHANGELOG.md)
- [Commits](oxsecurity/megalinter@v5.0.5...v5.2.0)

---
updated-dependencies:
- dependency-name: megalinter/megalinter
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@lars-reimann
ci: reusable workflow for MegaLinter
650df1d
@lars-reimann
ci: rename workflow
3f086fa
@dependabot
build(deps): bump astroid from 2.8.4 to 2.9.0 in /package_parser (#17)
b436e7b 
Bumps [astroid](https://github.com/PyCQA/astroid) from 2.8.4 to 2.9.0.
- [Release notes](https://github.com/PyCQA/astroid/releases)
- [Changelog](https://github.com/PyCQA/astroid/blob/main/ChangeLog)
- [Commits](pylint-dev/astroid@v2.8.4...v2.9.0)

---
updated-dependencies:
- dependency-name: astroid
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
dependabot bot and others added 18 commits January 1, 2022 15:57
@dependabot
build(deps): bump importlib-metadata in /package_parser (#38)
e79f549 
Bumps [importlib-metadata](https://github.com/python/importlib_metadata) from 4.8.2 to 4.10.0.
- [Release notes](https://github.com/python/importlib_metadata/releases)
- [Changelog](https://github.com/python/importlib_metadata/blob/main/CHANGES.rst)
- [Commits](python/importlib_metadata@v4.8.2...v4.10.0)

---
updated-dependencies:
- dependency-name: importlib-metadata
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@aserge16 @Prajakta46 @lars-reimann
feat: Dependencies (#42)
c743914 
* initial ontology commit

* 2 basic property subclass ideas

* added two more property subclasses

* moved ontology to predefined directory, fixed imports

* renamed property to dependency, added arithmetic dependency

* rework ontology model in code

* updated gitignore to ignore VSCode settings

* moved python-related changes to package_parser

* a stronger method for creating class instances instead of using a static method

* fixed potential lint error

* added ontology file for dependencies

* initial commit dependency fetcher

* initial code commit

* playing code for spacy rules

* some extra ontology properties for depenencies

* Dependencies extract first commit

* removed playground code

* fixed check in preprocessing

* oops, check is correct now

* removed name==main at bottom of file

* Added new instances

* Added a case of !=

* reworked preprocessing and added test

* using regex for preprocessing, cleaner, thanks Lars

* removed unused import and added type hint for dict

* type hint fixed again

* Modified pattern for "nominal subject (passive)" and "prepositional modifier"

* #35 used pattern created

* #36 first extraction function implemented for the first pattern

* #35 ignored condition pattern added

* #36 dependency extraction function for ignored condition pattern

* added second case of action word having uppercase first letter

* added pattern parameter applies

* fix: mypy error

* style: apply automatic fixes of linters

Co-authored-by: Aleksandr Sergeev <aleksandr.sergeev@tngtech.com>
Co-authored-by: Prajakta Bhujbal <prajaktabhujbal7@gmail.com>
Co-authored-by: Lars Reimann <mail@larsreimann.com>
Co-authored-by: lars-reimann <lars-reimann@users.noreply.github.com>
@duklin
feat: Model refined types as dataclasses, introduce only one property…
4c2198b 
… in th… (#45)

* Model refined types as dataclasses, introduce only one property in the json output for the refined types

* style: apply automatic fixes of linters

Co-authored-by: Dushko Klincharov <klincarov@halicea.com>
Co-authored-by: duklin <duklin@users.noreply.github.com>
@dependabot @lars-reimann
build(deps): bump astroid from 2.9.0 to 2.9.1 in /package_parser (#40)
2af8763 
Bumps [astroid](https://github.com/PyCQA/astroid) from 2.9.0 to 2.9.1.
- [Release notes](https://github.com/PyCQA/astroid/releases)
- [Changelog](https://github.com/PyCQA/astroid/blob/main/ChangeLog)
- [Commits](pylint-dev/astroid@v2.9.0...v2.9.1)

---
updated-dependencies:
- dependency-name: astroid
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Lars Reimann <mail@larsreimann.com>
@duklin @mmdoja
refactor: manually extracted refined types (#46)
3efa424 
* Refactor the manually extracted refined types for classes KMeans, PCA, TruncatedSVD, CountVectorizer and TfIdfVectorizer

* refactor refinedtypes for DecisionTreeClassifier

* refactor json for model_selection,neighbors,preprocessing,svm,tree

* refactoring for metrics,linearmodel,impute

* refactored json for feature_extraction

* updated BoundaryTypes and NamedTypes and the remaining classes

* Refactor manually extracted refined types for cluster, decomposition, ensemble

* Refactor manually extracted refined types for feature_extraction, impute and linear_model

* refactored manually extracted refined types for metrics,model_selection,neighbors,preprocessing,svm,tree

* Format the json files from the manual extraction of refined types

* Extract the boundary types in another json and the docstring part that is related to the boundary

* style: apply automatic fixes of linters

Co-authored-by: Dushko Klincharov <klincarov@halicea.com>
Co-authored-by: mmdoja <mmdoja@gmail.com>
Co-authored-by: duklin <duklin@users.noreply.github.com>
@aserge16 @Prajakta46
feat: Action/Condition Classification (#48)
13c789c 
* #43 basic condtion and action extraction functions

* #43 corrected extractions, now matching occurs per sentence, removes many false positives

* #43 fixed recursive action extraction function

* Added new pattern

* #43 classifying of ignored parameter actions

* #43 conditions and actions are now dataclasses, renamed one condition

* #43 phrases to categories actions/conditions

* #43 mypy fixes

* style: apply automatic fixes of linters

* #43 renamed IDs in general SCONJ pattern

* #43 more descriptive pattern name

Co-authored-by: Aleksandr Sergeev <aleksandr.sergeev@tngtech.com>
Co-authored-by: prajakta <prajaktabhujbal7@gmail.com>
Co-authored-by: aserge16 <aserge16@users.noreply.github.com>
@lars-reimann
ci: smoke test (#53)
3278c4d 
* Update pr.yml

* test: add smoke test to main workflow

* ci: use poetry run instead
@aserge16 @lars-reimann
test: Dependencies unit tests (#50)
60fc290 
* correctly formatted models to data classes

* updated model Names and added new phrases

* #49 dependency extraction function unit tests

* style: apply automatic fixes of linters

* fix: possible fix for build error

* added json dump of dependencies to CLI flow

* mypy fix

* style: apply automatic fixes of linters

* do the versions make a difference on dependency understanding?

* updated url for spacy langauge module

* quick fix, need to work on punctuation extraction

* removed hanging punctuation when extracting actions

* style: apply automatic fixes of linters

* test for dependency extraction class subordinating_conjuction

* style: apply automatic fixes of linters

* more documentation for functions

* style: apply automatic fixes of linters

* updated ontology to represent current code state

* removed pattern without implemented function for now

* added dependency README

Co-authored-by: Aleksandr Sergeev <aleksandr.sergeev@tngtech.com>
Co-authored-by: aserge16 <aserge16@users.noreply.github.com>
Co-authored-by: Lars Reimann <mail@larsreimann.com>
@duklin
feat: Boundary extraction (#54)
d689de7 
* Extract boundaries with regex

* Write tests for extracting boundaries

* Categorize boundaries

* style: apply automatic fixes of linters

Co-authored-by: Dushko Klincharov <klincarov@halicea.com>
Co-authored-by: duklin <duklin@users.noreply.github.com>
@dependabot
build(deps): bump astroid from 2.9.1 to 2.9.3 in /package_parser (#58)
28d9feb 
Bumps [astroid](https://github.com/PyCQA/astroid) from 2.9.1 to 2.9.3.
- [Release notes](https://github.com/PyCQA/astroid/releases)
- [Changelog](https://github.com/PyCQA/astroid/blob/main/ChangeLog)
- [Commits](pylint-dev/astroid@v2.9.1...v2.9.3)

---
updated-dependencies:
- dependency-name: astroid
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@dependabot
build(deps): bump importlib-metadata in /package_parser (#56)
df89cba 
Bumps [importlib-metadata](https://github.com/python/importlib_metadata) from 4.10.0 to 4.10.1.
- [Release notes](https://github.com/python/importlib_metadata/releases)
- [Changelog](https://github.com/python/importlib_metadata/blob/main/CHANGES.rst)
- [Commits](python/importlib_metadata@v4.10.0...v4.10.1)

---
updated-dependencies:
- dependency-name: importlib-metadata
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@dependabot
build(deps-dev): bump mypy from 0.930 to 0.931 in /package_parser (#55)
ec703c9 
Bumps [mypy](https://github.com/python/mypy) from 0.930 to 0.931.
- [Release notes](https://github.com/python/mypy/releases)
- [Commits](python/mypy@v0.930...v0.931)

---
updated-dependencies:
- dependency-name: mypy
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@dependabot
build(deps): bump numpydoc from 1.1.0 to 1.2 in /package_parser (#57)
e7049fd 
Bumps [numpydoc](https://github.com/numpy/numpydoc) from 1.1.0 to 1.2.
- [Release notes](https://github.com/numpy/numpydoc/releases)
- [Changelog](https://github.com/numpy/numpydoc/blob/main/doc/release_notes.rst)
- [Commits](numpy/numpydoc@v1.1.0...numpydoc-1.2)

---
updated-dependencies:
- dependency-name: numpydoc
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@mmdoja
feat: updated ontology to represent the current model (#65)
135c61e 
* updated ontology to represent the current model

* fix: updated ontology

* removed subclasses for UnionType
@aserge16 @lars-reimann
feat: Dependencies updated README (#59)
e6e9b0e 
* corrected pattern name

* updated readme with future work

* updated readme with instructions on how to add further patterns

* expected type correction

* updated test

* refactor: possible fix for mypy error

* refactor: another possible fix for mypy error

Co-authored-by: Aleksandr Sergeev <aleksandr.sergeev@tngtech.com>
Co-authored-by: Lars Reimann <mail@larsreimann.com>
@AdrianBajraktari @BitaAfshar @lars-reimann
feat: integrate purity parser (#68)
c210d05 
* Initial prototype written with astroid as shown in last meeting (10.12.21)

* Update to ast file & node classes file

* Add files via upload

* Delete ast_example2.py

* Add files via upload

* State of ast visitor as discussed in meeting

* Add files via upload

* Some fixes to handlers of many ast nodes

* Added saving properties to lists for ontology integration

* Add files via upload

* Add files via upload

* Added comments, some fixes to certain nodes

* Delete PurityV1.owl

* Delete PurityV2.owl

* Add files via upload

* style: minor formatting changes

* style: apply automatic fixes of linters

Co-authored-by: Adrian Bajraktari <s6adbajr@uni-bonn.de>
Co-authored-by: BitaAfshar <93251213+BitaAfshar@users.noreply.github.com>
Co-authored-by: Lars Reimann <mail@larsreimann.com>
Co-authored-by: lars-reimann <lars-reimann@users.noreply.github.com>
@lars-reimann
refactor: move everything into top-level folder
35a68e3
@lars-reimann
Merge remote-tracking branch 'api-editor/main'
a7daa4e
@github-actions
Copy link

github-actions bot commented Feb 10, 2022 •
edited
Loading

MegaLinter status: ⚠️ WARNING

Descriptor Linter Files Fixed Errors Elapsed time
✅ CREDENTIALS secretlint yes no 3.2s
✅ GIT git_diff yes no 0.01s
✅ HTML htmlhint 1 0 0.28s
✅ JAVASCRIPT prettier 1 0 0 1.01s
✅ JSON eslint-plugin-jsonc 60 0 0 9.91s
✅ JSON jsonlint 60 0 16.54s
✅ JSON prettier 60 0 0 8.44s
✅ JSON v8r 60 0 37.53s
✅ KOTLIN ktlint 45 0 0 103.61s
⚠️ MARKDOWN markdownlint 4 0 11 0.5s
✅ MARKDOWN markdown-table-formatter 4 0 0 0.36s
✅ PYTHON black 32 0 0 1.06s
✅ PYTHON isort 32 0 0 0.2s
✅ PYTHON mypy 32 0 20.43s
✅ TYPESCRIPT prettier 76 0 0 5.82s
✅ XML xmllint 2 0 0.03s
✅ YAML prettier 2 0 0 1.45s
✅ YAML v8r 2 0 2.71s
✅ YAML yamllint 2 0 0.26s

See errors details in artifact MegaLinter reports on CI Job page
Set VALIDATE_ALL_CODEBASE: true in mega-linter.yml to validate all sources, not only the diff

@lars-reimann lars-reimann enabled auto-merge (squash) February 10, 2022 12:13
@lars-reimann lars-reimann merged commit d31031c into main Feb 10, 2022
@lars-reimann lars-reimann deleted the feature/merge_with_api-editor branch February 10, 2022 12:18
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@lars-reimann @aserge16 @duklin @mmdoja @AdrianBajraktari