Dev (#376)

* Python GA support

* updated mac address rule (#225)

* updated mac address rule

* removed workforce dir

---------

Co-authored-by: “Hitesh <hitesh.bedre@privado.com>

* Added rules for third party sdks in the db (#266)

* add - cookie as sink

* add - merged cookie rules

* Remove redundant ruby sdk rules

* Ruby/third party sdk secondary (#269)

* Added rules for third party sdks in the db

* added some more third parties

* refactor rules

* Handling of rule to select element split over multiple lines (#270)

* Handling of rule to select element split over multiple lines

If the tag is split into multiple lines then the existing regex was
failing to detect the element. Like the below code sample.
Handled respective use case along with the addition of corresponding
unit test

```
<ComplexInputField
                    name={FIELDS.SURNAME}
                    label={messages.surName.placeholder}
                    value={values?.surname}
                    error={errors.surname}
                  />
```

* rule fix

* Update README.md (#272)

* Js sdk fixes (#274)

* Python GA support

* updated mac address rule (#225)

* updated mac address rule

* removed workforce dir

---------

Co-authored-by: “Hitesh <hitesh.bedre@privado.com>

* Fix the sdk rules for javascript

* Update the okta rule

* Remove unwanted rules

---------

Co-authored-by: hiteshbedre <32206192+hiteshbedre@users.noreply.github.com>
Co-authored-by: KhemrajSingh Rathore <khemraj.rathore@privado.ai>
Co-authored-by: “Hitesh <hitesh.bedre@privado.com>
Co-authored-by: Pandurang Patil <pandurang.patil@gmail.com>

* Ruby http libraries (#278)

* Support for Ruby HTTP client rules

* Update the apiHttpLibraries rule for ruby

* Update personal_characteristics.yaml

* Update biometric_data.yaml

* Ruby SDK + JS Fixes (#282)

* Support for Ruby HTTP client rules

* Update the apiHttpLibraries rule for ruby

* Additional ruby sdk + minor JS fixes

* Minor corrections in rules

* Fix the typo in fetchapi

* JS Fixes (#283)

* Ruby http libraries (#278)

* Support for Ruby HTTP client rules

* Update the apiHttpLibraries rule for ruby

* Update personal_characteristics.yaml

* Ruby SDK + JS Fixes (#282)

* Support for Ruby HTTP client rules

* Update the apiHttpLibraries rule for ruby

* Additional ruby sdk + minor JS fixes

* Minor corrections in rules

* Fix the typo in fetchapi

---------

Co-authored-by: Hitesh Mahajan <hiteshvm1998@gmail.com>

* Sources/click stream ddhq (#284)

* add datadog initialization params which default track click behaviour

* Add IndexedDB storage rule

---------

Co-authored-by: Datta Mundada <dbmundada333@gmail.com>

* vehicle registration rule

* Laundarkly fix (#287)

* Remove useFlags method from launchdarkly sdk

* Minor correction in laundarkly fix

* JS sdk fixes -1

* Fix code-review comment

* Dev (#285)

* Sources/click stream ddhq (#284)

* add datadog initialization params which default track click behaviour

* Add IndexedDB storage rule

---------

Co-authored-by: Datta Mundada <dbmundada333@gmail.com>

* vehicle registration rule

* Laundarkly fix (#287)

* Remove useFlags method from launchdarkly sdk

* Minor correction in laundarkly fix

* JS sdk fixes -1

* Fix code-review comment

---------

Co-authored-by: Hitesh Mahajan <hiteshvm1998@gmail.com>
Co-authored-by: Hitesh Mahajan <hitesh@Hiteshs-MacBook-Pro.local>

* updated leakage rule (#292)

* Ruby august release (#294)

* change - wechat rule

* fix - tencent cloud

* add - thirdParty api

* add - new rules in ruby

* add more rules

* Python: Stripe Rule (#297)

* Python: Stripe Rule (#298)

* Add Graphql rules as API rules (#299)

* Add Graphql rules as API rules

* Add client creation baseurl detection pattern

* Add mongodb rule for models

* Add OpenAI sdk rules (#302)

* Add OpenAI sdk rules

* Fix code review comments

* Removed duplicate rule (#303)

* Removed duplicate rule (#303) (#304)

* Update the LLM rules + Add Weviate DB (#306)

* Added rules for ruby third party (#295)

* Added rules for ruby third party

* removed already existing rule

* added hellosign rule

* code refactoring

* Update online_identifiers.yaml (#305)

* Update online_identifiers.yaml

* fix - redis rule in ruby

* new rules

---------

Co-authored-by: root <root@ip-172-31-4-154.eu-west-1.compute.internal>

* [goSrc2cpg] - Added Golang storage rule (#309)

* Added golang storage rule

* golang rule change

* Go: leakage sink (#310)

* leakage sink

* added leakage rules based on package name

---------

Co-authored-by: “Hitesh <hitesh.bedre@privado.com>

* default config for zeromq sockets (#311)

* Dev (#285)

* Sources/click stream ddhq (#284)

* add datadog initialization params which default track click behaviour

* Add IndexedDB storage rule

---------

Co-authored-by: Datta Mundada <dbmundada333@gmail.com>

* vehicle registration rule

* Laundarkly fix (#287)

* Remove useFlags method from launchdarkly sdk

* Minor correction in laundarkly fix

* JS sdk fixes -1

* Fix code-review comment

---------

Co-authored-by: Hitesh Mahajan <hiteshvm1998@gmail.com>
Co-authored-by: Hitesh Mahajan <hitesh@Hiteshs-MacBook-Pro.local>

* Python: Stripe Rule (#298)

* Removed duplicate rule (#303) (#304)

* default config for zeromq sockets

---------

Co-authored-by: Datta Mundada <dbmundada333@gmail.com>
Co-authored-by: Hitesh Mahajan <hiteshvm1998@gmail.com>
Co-authored-by: Hitesh Mahajan <hitesh@Hiteshs-MacBook-Pro.local>
Co-authored-by: KhemrajSingh Rathore <khemraj.rathore@privado.ai>
Co-authored-by: hiteshbedre <32206192+hiteshbedre@users.noreply.github.com>

* Golang storage rule (#312)

* Added golang storage rule

* golang rule change

* fix go rule

* fix amazons3 domains rule

* Added rule for go sqlx

* Updated s3 rules

---------

Co-authored-by: Ankit Kumar <ankit.kumar@privado.ai>

* added zerolog rules for logs

* changed redis rules

* Remove SQL Alchmey rule  for python (#315)

* Go: Storage Rules (#317)

* go: database rule

* go: storage support

* go: divided into read and readandwrite rules

* divided rule into two

* go: corrected weaviate rule

---------

Co-authored-by: “Hitesh <hitesh.bedre@privado.com>

* Golang storage rules (#318)

* storage rules

* arangodb go rule fix

* added storage rules

* parallel processing for comparisom report (#321)

* parallel processing for comparison report (#324)

* Parallel comparison report (#326)

* parallel processing for comparison report

* change standalone ref to parallel-test

* change zip command to get only one folder

* change ref to main

* Delete zamaon sdk (#328)

* added go exclusion rule (#329)

* vertx.io is not a third party, its a toolkit

* Update Account Password.

* Go GORM rules (#338)

* go: API Tagger (#331)

* go: API Tagger

* added internal api rule

* added do method

---------

Co-authored-by: “Hitesh <hitesh.bedre@privado.com>

* [gosrc2cpg] - Added third party rules (#335)

* Added go third party rules

* Added more GO third parties

* Added more rules

* added more rules

* go rule fix

* go third party rule fix

* go third party backslash fix

* go third parties rule fix

* [gosrc2cpg] - storage framework rules (#340)

* added gorm rule

* fix gorm rule

* added more storage rules

* Added more framework rules

* dummy-commit

* Update national_identification_numbers.yaml (#334)

* Python/3p/cherrypy (#308)

* Dev (#285)

* Sources/click stream ddhq (#284)

* add datadog initialization params which default track click behaviour

* Add IndexedDB storage rule

---------

Co-authored-by: Datta Mundada <dbmundada333@gmail.com>

* vehicle registration rule

* Laundarkly fix (#287)

* Remove useFlags method from launchdarkly sdk

* Minor correction in laundarkly fix

* JS sdk fixes -1

* Fix code-review comment

---------

Co-authored-by: Hitesh Mahajan <hiteshvm1998@gmail.com>
Co-authored-by: Hitesh Mahajan <hitesh@Hiteshs-MacBook-Pro.local>

* Python: Stripe Rule (#298)

* Removed duplicate rule (#303) (#304)

* Delete rules/sinks/third_parties/sdk/cherrypy/python.yaml

cherrypy is a framework not a thirdparty

---------

Co-authored-by: Datta Mundada <dbmundada333@gmail.com>
Co-authored-by: Hitesh Mahajan <hitesh@Hiteshs-MacBook-Pro.local>
Co-authored-by: KhemrajSingh Rathore <khemraj.rathore@privado.ai>
Co-authored-by: hiteshbedre <32206192+hiteshbedre@users.noreply.github.com>

* change slack action to upload file (#342)

* Kotlin poc (#344)

* kotlin logging rules

* api rules

* add - kotlin rule

* modified mongo rule

* add - more rules

* Added go comparison report support (#345)

* fix path in comparison results (#346)

* add - regex for exclusion templates in kotlin (#348)

* add - regex for exclusion templates in kotlin

* update regex

* Revert "update regex"

This reverts commit c7b9507.

* Google map and google play rule (#349)

* kotlin logging rules

* api rules

* add - kotlin rule

* modified mongo rule

* add - more rules

* add - more google rules

* lng  matches with short for language (#350)

* add rule for authy (#351)

* spring java patch collection (#353)

* go exclusion rule fix (#354)

* Revert "go exclusion rule fix (#354)"

This reverts commit 69b9f9b.

* add datadog java sdks

source : https://docs.datadoghq.com/developers/community/libraries/

* add android collection rules (#356)

* added go third party rules (#358)

* Go additional rule (#361)

* added go third party rules

* go temporal rule

* Go third party rule (#363)

* added go third party rule

* go slack rule added

* go github rule added (#365)

* add tags to link source to collections (#367)

* add tags to link source to collections

Required for: Privado-Inc/privado-core#862

* remove rule

---------

Co-authored-by: Pandurang Patil <pandurang.patil@gmail.com>
Co-authored-by: Hitesh Mahajan <hiteshvm1998@gmail.com>
Co-authored-by: KhemrajSingh Rathore <khemraj.rathore@privado.ai>

* refine cookie storage rule (#369)

* refine cookie storage rule

* refine cookie storage rule

---------

Co-authored-by: Hitesh Mahajan <hitesh.mahajan@privado.ai>

* Go APIHttp Libraries rule (#371)

* added go apihttp libraries

* go apihttp improvement

* Add few more android rules (#372)

* add more android rules

* education rules

* Update any.yaml

* fix

---------

Co-authored-by: Pandurang Patil <pandurang.patil@gmail.com>
Co-authored-by: Hitesh Mahajan <hiteshvm1998@gmail.com>
Co-authored-by: KhemrajSingh Rathore <khemraj.rathore@privado.ai>

* fix java google pubsub

---------

Co-authored-by: hiteshbedre <32206192+hiteshbedre@users.noreply.github.com>
Co-authored-by: “Hitesh <hitesh.bedre@privado.com>
Co-authored-by: Pandurang Patil <pandurang.patil@gmail.com>
Co-authored-by: Karan Batavia <118820668+karan-batavia@users.noreply.github.com>
Co-authored-by: Dattaprasad Mundada <dbmundada333@gmail.com>
Co-authored-by: Hitesh Mahajan <hiteshvm1998@gmail.com>
Co-authored-by: Hitesh Mahajan <hitesh@Hiteshs-MacBook-Pro.local>
Co-authored-by: Ankit Kumar <118803988+ankit-privado@users.noreply.github.com>
Co-authored-by: root <root@ip-172-31-4-154.eu-west-1.compute.internal>
Co-authored-by: Suchakra Sharma <suchakra@privado.ai>
Co-authored-by: Ankit Kumar <ankit.kumar@privado.ai>
Co-authored-by: Hitesh Mahajan <hitesh.mahajan@privado.ai>

.github/workflows/comparison-result.yml

@@ -1,15 +1,40 @@
-name: Monitoring Stability and Comparing Results for privado 
+name: Monitoring Stability and Comparing Results for privado
 
-# Triggers when a pull_request or a push action is configured on master branch
+# Triggers when a pull_request is created
 on:
   pull_request_target:
+    branches:
+      - "**"
 
 jobs:
+  start_workflow:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Send message to slack
+        id: initial-message
+        uses: archive/github-actions-slack@master
+        with:
+          slack-optional-parse: full
+          slack-bot-user-oauth-access-token: ${{ secrets.SLACK_TOKEN }}
+          slack-channel: ${{ secrets.SLACK_CHANNEL_ID }}
+          slack-text: "Comparison workflow started for ${{github.event.pull_request.html_url}}"
+
+      - name: Save output to env
+        id: save-output
+        run: echo "INIT_MSG_TS=${{ fromJson(steps.initial-message.outputs.slack-result).response.message.ts }}" >> $GITHUB_OUTPUT
+    outputs:
+      init_message_ts: ${{steps.save-output.outputs.INIT_MSG_TS}}
+
   setup_and_scan:
+    needs: start_workflow
+    strategy:
+      matrix:
+        language: ['java', 'python', 'js', 'ruby-1', 'ruby-2', 'go']
+    continue-on-error: true
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v3
-          
+
       - name: Install JDK-18
         uses: actions/setup-java@v3
         with:
@@ -31,48 +56,91 @@ jobs:
         with:
           repository: Privado-Inc/standalone-monitoring-stability
           path: ./temp/standalone-monitoring-stability
-          ref: main 
-      
+          ref: main
+
       - name: Run the script for ${{github.head_ref}} and ${{github.base_ref}}
-        run: cd ./temp/standalone-monitoring-stability && pip install -r requirements.txt && python3 ./run.py -rbb ${{github.base_ref}} -rbh ${{github.head_ref}} -brr ${{ github.event.pull_request.base.repo.html_url }} -hrr ${{ github.event.pull_request.head.repo.html_url }} -guf -urc
+        run: cd ./temp/standalone-monitoring-stability && pip install -r requirements.txt && python3 ./run.py -r ./repos/${{matrix.language}}.txt -rbb ${{github.base_ref}} -rbh ${{github.head_ref}} -brr ${{ github.event.pull_request.base.repo.html_url }} -hrr ${{ github.event.pull_request.head.repo.html_url }} -guf -urc
 
       - name: Run aws-export
-        run: cd ./temp/standalone-monitoring-stability/ && python3 aws-export.py ${{github.event.number}}
+        run: cd ./temp/standalone-monitoring-stability/ && python3 aws-export.py ${{matrix.language}}-${{github.event.number}}
 
       - name: Move results to a folder
-        run: cd ./temp/standalone-monitoring-stability/ && mkdir results && mv output-${{github.event.number}}.xlsx ./results/output-${{github.event.number}}.xlsx && mv ./temp/result-${{github.event.number}}.zip ./results/result-${{github.event.number}}.zip && mv slack_summary.txt ./results/slack_summary.txt
+        run: cd ./temp/standalone-monitoring-stability/ && mkdir results && mv output-${{matrix.language}}-${{github.event.number}}.xlsx ./results/output-${{matrix.language}}-${{github.event.number}}.xlsx && mv ./temp/result-${{matrix.language}}-${{github.event.number}}.zip ./results/result-${{matrix.language}}-${{github.event.number}}.zip && mv slack_summary.txt ./results/slack_summary.txt
+
+      - name: Zip the results
+        run: cd /home/runner/work/privado/privado/temp/standalone-monitoring-stability && zip result-${{matrix.language}}-${{github.event.number}}.zip -r ./results
+
+      - name: Set summary variable
+        run: |
+          echo "MESSAGE<<EOF" >> $GITHUB_ENV
+          echo "$(cat /home/runner/work/privado/privado/temp/standalone-monitoring-stability/results/slack_summary.txt)" >> $GITHUB_ENV
+          echo "EOF" >> $GITHUB_ENV
+      
+      - name: Post results to slack
+        uses: adrey/slack-file-upload-action@master
+        with:
+          thread_ts: ${{needs.start_workflow.outputs.init_message_ts}}
+          channel: ${{ secrets.SLACK_CHANNEL_ID }} # check
+          path: "/home/runner/work/privado/privado/temp/standalone-monitoring-stability/result-${{matrix.language}}-${{github.event.number}}.zip"
+          initial_comment: "Comparison Results generated on ${{github.event.repository.name}} by PR ${{github.event.number}} from branch ${{github.head_ref}} to ${{github.base_ref}} \nPR link https://github.com/Privado-Inc/privado/pull/${{github.event.number}}\n Language: ${{matrix.language}} \nSummary Report:\n ${{ env.MESSAGE }}"
+          filetype: "zip"
+          token: ${{ secrets.SLACK_TOKEN }}
+
+      - name: Export workflow output
+        run: cd ./temp/standalone-monitoring-stability && python3 ./workflow_check.py /home/runner/work/privado/privado/temp/standalone-monitoring-stability/results/slack_summary.txt
 
-      - name: Upload output and result for next job
+      - name: Set summary variable
+        run: |
+          echo "MESSAGE<<EOF" >> $GITHUB_ENV
+          echo "$(cat ./temp/standalone-monitoring-stability/action_result.txt)" >> $GITHUB_ENV
+          echo "EOF" >> $GITHUB_ENV 
+
+      - name: Upload summary file
         uses: actions/upload-artifact@master
         with:
-          name: results
-          path: /home/runner/work/privado/privado/temp/standalone-monitoring-stability/results
+          name: ${{matrix.language}}
+          path: /home/runner/work/privado/privado/temp/standalone-monitoring-stability/results/slack_summary.txt
 
-  send-result:
-    needs: setup_and_scan
-    runs-on: ubuntu-latest
-    steps: 
-      - uses: actions/checkout@v3
+      - name: Workflow report analysis
+        if: ${{ env.MESSAGE != 'true' }}
+        run: exit 1
 
-      - name: Download result folder
+  collate_summary:
+    needs: [start_workflow, setup_and_scan]
+    runs-on: ubuntu-latest
+    steps:
+      - name: Download summary file
         uses: actions/download-artifact@master
         with:
-          name: results
-          path: ./results
-
-      - name: Zip the results
-        run: zip result-${{github.event.number}}.zip -r ./results
+          path: ./language_summary
+
+      - name: Install Python 3.10
+        uses: actions/setup-python@v4
+        with:
+          python-version: '3.10'
+
+      - name: Clone standalone-monitoring-stability/flow-test
+        uses: actions/checkout@v3
+        with:
+          repository: Privado-Inc/standalone-monitoring-stability
+          path: ./temp/standalone-monitoring-stability
+          ref: main
+
+
+      - name: Collate summary
+        run: cd ./temp/standalone-monitoring-stability && pip install -r requirements.txt && python3 ./collate_summary.py -s /home/runner/work/privado/privado/language_summary
 
       - name: Set summary variable
         run: |
           echo "MESSAGE<<EOF" >> $GITHUB_ENV
-          echo "$(cat ./results/slack_summary.txt)" >> $GITHUB_ENV
+          echo "$(cat /home/runner/work/privado/privado/temp/standalone-monitoring-stability/global_summary.txt)" >> $GITHUB_ENV
           echo "EOF" >> $GITHUB_ENV
-      - name: Post results to slack
-        uses: MeilCli/slack-upload-file@v3
+
+      - name: Send summary to slack
+        uses: slackapi/slack-github-action@v1.24.0
         with:
-          slack_token: ${{ secrets.SLACK_TOKEN }}
-          channel_id: ${{ secrets.SLACK_CHANNEL_ID }}
-          file_path: "/home/runner/work/privado/privado/result-${{github.event.number}}.zip"
-          initial_comment: "Comparison Results generated on ${{github.event.repository.name}} by PR ${{github.event.number}} from branch ${{github.head_ref}} to ${{github.base_ref}} \nPR link https://github.com/Privado-Inc/privado/pull/${{github.event.number}} \nSummary Report:\n ${{ env.MESSAGE }}"
-          file_type: "zip"
+          update-ts: ${{needs.start_workflow.outputs.init_message_ts}}
+          channel-id: ${{ secrets.SLACK_CHANNEL_ID }}
+          slack-message: "\nComparison Results generated on ${{github.event.repository.name}} by PR ${{github.event.number}} from branch ${{github.head_ref}} to ${{github.base_ref}} \nPR link https://github.com/Privado-Inc/privado/pull/${{github.event.number}}\nLanguage: All \nSummary Report:\n ${{ env.MESSAGE }}"
+        env:
+          SLACK_BOT_TOKEN: ${{ secrets.SLACK_TOKEN }}

.gitignore

@@ -246,3 +246,6 @@ dist
 # files
 privado
 notes.md
+
+#Directory created by IDE
+workspace

config/exclusions/go.yaml

@@ -0,0 +1,5 @@
+exclusions:
+  - id: Exclusions.Test
+    name: Exclude test source code
+    patterns:
+      - '.*_test\(s\)?.*'

config/exclusions/java.yaml

@@ -12,4 +12,4 @@ exclusions:
   - id: Exclusions.Empty
     name: Exclude file which cannot be read
     patterns:
-      - "<empty>"
+      - "<empty>"

config/exclusions/kotlin.yaml

@@ -0,0 +1,5 @@
+exclusions:
+  - id: Exclusions.Template
+    name: Exclude template file
+    patterns:
+      - "(?i)(.*template.kt|.*template(s)?/.*)"

config/systemConfig/default.yaml

@@ -0,0 +1,3 @@
+systemConfig:
+  - key: maxSocketCount
+    value: "4096"

config/systemConfig/go.yaml

@@ -0,0 +1,9 @@
+systemConfig:
+  - key: apiHttpLibraries
+    value: ^(?i)(net/http|github.com/parnurzeal/gorequest|(gopkg.in|github.com/go-resty)/resty|valyala/fasthttp|github.com/gojektech/heimdall/v\\d/httpclient|github.com/levigross/grequests|github.com/PuerkitoBio/rehttp|github.com/machinebox/graphql).*
+
+  - key: apiSinks
+    value: (?i)(?:url|client|open|request|execute|newCall|load|host|access|list|set|put|post|proceed|trace|patch|Path|send|remove|delete|write|read|postForEntity|call|createCall|createEndpoint|dispatch|invoke|getInput|getOutput|getResponse|do)
+
+  - key: apiIdentifier
+    value: (?i).*((hook|base|auth|prov|endp|install|request|service|gateway|route|resource)(.){0,12}url|(slack|web)(.){0,4}hook|(rest|api|request|service)(.){0,4}(endpoint|gateway|route)).*

config/systemConfig/kotlin.yaml

@@ -0,0 +1,12 @@
+systemConfig:
+  - key: apiHttpLibraries
+    value: ^(?i)(org.apache.http|okhttp|org.glassfish.jersey|com.mashape.unirest|java.net.http|java.net.URL|org.springframework.(web|core.io)|groovyx.net.http|org.asynchttpclient|kong.unirest.java|org.concordion.cubano.driver.http|javax.net.ssl|javax.xml.soap|org.apache.axis2|com.sun.xml.messaging.saaj|org.springframework.ws.client|com.eviware.soapui|org.apache.cxf|org.jboss.ws|com.ibm.websphere.sca.extensions.soap|com.sun.xml.ws|org.apache.camel.component.cxf|org.codehaus.xfire|org.apache.synapse|org.apache.wink.client|com.oracle.webservices.internal.api.databinding.Databinding|com.sap.engine.interfaces.webservices.runtime.client).*
+
+  - key: ignoredSinks
+    value: (?i).*(?<=map|list|jsonobject|json|array|arrays|jsonnode|objectmapper|objectnode).*(put:|get:).*
+
+  - key: apiSinks
+    value: (?i)(?:url|client|openConnection|request|execute|newCall|load|host|access|fetch|get|getInputStream|getApod|getForObject|getForEntity|list|set|put|post|proceed|trace|patch|Path|send|sendAsync|remove|delete|write|read|assignment|provider|exchange|postForEntity|call|createCall|createEndpoint|dispatch|invoke|newMessage|getInput|getOutput|getResponse|marshall|unmarshall|send|asyncSend)
+
+  - key: apiIdentifier
+    value: (?i).*((hook|base|auth|prov|endp|install|request|service|gateway|route|resource)(.){0,12}url|(slack|web)(.){0,4}hook|(rest|api|request|service)(.){0,4}(endpoint|gateway|route)).*

config/systemConfig/ruby.yaml

@@ -0,0 +1,12 @@
+systemConfig:
+  - key: apiHttpLibraries
+    value: (?i)(multipart|faraday|rest-client|httparty|http.client|net.http|curb|sawyer|unirest|excon|typhoeus|.*(Http(.){0,2}Client|RestClient|HTTParty|Faraday|Unirest)).*
+
+  - key: ignoredSinks
+    value: (?i).*(?<=map|list|jsonobject|json|array|arrays|jsonnode|objectmapper|objectnode).*(put:|get:).*
+
+  - key: apiSinks
+    value: (?i)(?:new|url|client|openConnection|request|execute|newCall|load|host|access|usequery|fetch|get|getInputStream|getApod|getForObject|getForEntity|list|set|put|post|proceed|trace|patch|Path|send|sendAsync|remove|delete|write|read|assignment|provider|exchange|postForEntity|call|createCall|createEndpoint|dispatch|invoke|newMessage|getInput|getOutput|getResponse|marshall|unmarshall|send|asyncSend|emit)
+
+  - key: apiIdentifier
+    value: (?i).*((hook|base|auth|prov|endp|install|cloud|host|request|service|gateway|route|resource|upload|api|worker)(.){0,12}url|(slack|web)(.){0,4}hook|(sentry|segment)(.){0,1}(dsn)|(rest|api|host|cloud|request|service)(.){0,4}(endpoint|gateway|route)).*

rules/collections/android/any.yaml

@@ -0,0 +1,63 @@
+collections:
+  - id: Collections.Android.Form.Email
+    name: Android Form Email
+    patterns:
+      - ".*(?i)email.*"
+    tags:
+      sourceId: Data.Sensitive.ContactData.EmailAddress
+
+  - id: Collections.Android.Form.User
+    name: Android Form User Account
+    patterns:
+      - ".*(?i)(user|login).*"
+    tags:
+      sourceId: Data.Sensitive.AccountData.AccountID
+
+  - id: Collections.Android.Form.OrderDetails
+    name: Android Form Order Details
+    patterns:
+      - "(?i).*((order|shipping|billing|invoice)(subscription|charge)?[^\\s/(;)#|,=!>]{0,5}(number|code|num|no|id))"
+    tags:
+      sourceId: Data.Sensitive.PurchaseData.OrderDetails
+
+  - id: Collections.Android.Form.FirstName
+    name: Android Form Personal Characterstics
+    patterns:
+      - "(?i).*((?:first|given)[^\\s/(;)#|,=!>]{0,5}|full)[_]?name"
+    tags:
+      sourceId: Data.Sensitive.PersonalIdentification.FirstName
+
+  - id: Collections.Android.Form.LastName
+    name: Android Form Personal Characterstics
+    patterns:
+      - "(?i).*((?:last|sur(?!geon))[^\\s/(;)#|,=!>]{0,5}name)"
+    tags:
+      sourceId: Data.Sensitive.PersonalIdentification.LastName
+
+  - id: Collections.Android.Form.Address
+    name: Android Form Address
+    patterns:
+      - ".*(?i)address.*"
+    tags:
+      sourceId: Data.Sensitive.ContactData.Address
+
+  - id: Collections.Android.Form.PhoneNumber
+    name: Android Form Phone Number
+    patterns:
+      - ".*(?i)phone.*"
+    tags:
+      sourceId: Data.Sensitive.ContactData.PhoneNumber
+
+  - id: Collections.Android.Form.ZipCode
+    name: Android Form Zip Code
+    patterns:
+      - ".*(?i)zip.*"
+    tags:
+      sourceId: Data.Sensitive.ContactData.Address
+
+  - id: Collections.Android.Form.Password
+    name: Android Form Password
+    patterns:
+      - ".*(?i)password.*"
+    tags:
+      sourceId: Data.Sensitive.AccountData.AccountPassword

rules/collections/annotations/java.yaml

@@ -2,7 +2,7 @@ collections:
   - id: Collections.Annotation.Spring
     name: Spring Web Interface Annotation
     patterns: 
-      - "RequestMapping|PostMapping|PutMapping|GetMapping|DeleteMapping"
+      - "RequestMapping|PostMapping|PutMapping|PatchMapping|GetMapping|DeleteMapping"
     tags:
 
   - id: Collections.Annotation.Struts

rules/collections/default/javascript.yaml

@@ -2,5 +2,5 @@ collections:
   - id: Collections.Express
     name: Express framework restendpoint
     patterns: 
-      - "express.(post|get|all|delete|put|patch|head|subscribe|unsubscribe)"
+      - "(?:express|fetch|@feathersjs/feathers|fastify|restify|@nestjs/cli|itty-router|koa-router|@ioc[:]Adonis|@adonisjs|@sails|sails|.*loopback|.*(?:socket[.](io|on|to).*)|(?:io[.]on.*(connection|leave-room|join-room))).*"
     tags:

rules/collections/webforms/any.yaml

@@ -2,5 +2,5 @@ collections:
   - id: Collections.Webforms
     name: Webform data collection
     patterns: 
-      - "^<(?i)(?:\\w{0,}(input|upload)\\w{0,}|\\w{0,}(textarea|Text|TextBox|Select|Field|Autocomplete|Checkbox))"
+      - "^<(?i)(?:\\w{0,}(input|upload)\\w{0,}|\\w{0,}(textarea|Text|TextBox|Select|Field|Autocomplete|Checkbox))[^>]*.*"
     tags:

rules/sinks/internal_apis/api/go.yaml

@@ -0,0 +1,6 @@
+sinks:
+  - id: Sinks.API.InternalAPI
+    name: Internal APIs
+    patterns: 
+      - "((http|https|ftp|ssh):\\/\\/){0,1}(((25[0-5]|(2[0-4]|1\\d|[1-9]|)\\d)\\.?\\b){4}|(localhost))(:[0-9]{2,4}){0,1}(\\/([a-z]){0,1}){0,1}.*"
+    tags:

rules/sinks/internal_apis/api/ruby.yaml

@@ -0,0 +1,6 @@
+sinks:
+  - id: Sinks.API.InternalAPI
+    name: Internal APIs
+    patterns: 
+      - "((http|https|ftp|ssh):\\/\\/){0,1}(((25[0-5]|(2[0-4]|1\\d|[1-9]|)\\d)\\.?\\b){4}|(localhost))(:[0-9]{2,4}){0,1}(\\/([a-z]){0,1}){0,1}.*"
+    tags: