Fix][api] Fix multiple threads creating folders will result in multiple identical folders

.github/ISSUE_TEMPLATE/bug_report.md

@@ -1,14 +1,16 @@
 ---
 name: Bug report
 about: Create a report to help us improve
-title: "[BUG] bug title "
+title: "[Bug][Module Name] Bug title "
 labels: bug
 assignees: ''
 
 ---
 
 *For better global communication, please give priority to using English description, thx! *
 
+*Please review https://dolphinscheduler.apache.org/en-us/docs/development/issue.html when describe an issue.*
+
 **Describe the bug**
 A clear and concise description of what the bug is.
 
@@ -32,5 +34,5 @@ If applicable, add screenshots to help explain your problem.
 **Additional context**
 Add any other context about the problem here.
 
-**Requirement or improvement
+**Requirement or improvement**
 - Please describe about your requirements or improvement suggestions.

.github/ISSUE_TEMPLATE/feature_request.md

@@ -1,12 +1,19 @@
 ---
 name: Feature request
 about: Suggest an idea for this project
-title: "[Feature]"
+title: "[Feature][Module Name] Feature title"
 labels: new feature
 assignees: ''
 
 ---
 
+*For better global communication, please give priority to using English description, thx! *
+
+*Please review https://dolphinscheduler.apache.org/en-us/docs/development/issue.html when describe an issue.*
+
+**Describe the feature**
+A clear and concise description of what the feature is.
+
 **Is your feature request related to a problem? Please describe.**
 A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
 

.github/ISSUE_TEMPLATE/improvement_suggestion.md

@@ -0,0 +1,24 @@
+---
+name: Improvement suggestion
+about: Improvement suggestion for this project
+title: "[Improvement][Module Name] Improvement title"
+labels: improvement
+assignees: ''
+
+---
+
+*For better global communication, please give priority to using English description, thx! *
+
+*Please review https://dolphinscheduler.apache.org/en-us/docs/development/issue.html when describe an issue.*
+
+**Describe the question**
+A clear and concise description of what the improvement is.
+
+**What are the current deficiencies and the benefits of improvement**
+- A clear and concise description of the current deficiencies and the benefits of this improvement.
+
+**Which version of DolphinScheduler:**
+ -[1.1.0-preview]
+
+**Describe alternatives you've considered**
+A clear and concise description of any alternative improvement solutions you've considered.

.github/ISSUE_TEMPLATE/question.md

@@ -1,14 +1,16 @@
 ---
-name: question
-about: have a question wanted to be help
-title: "[QUESTION] question title"
+name: Question
+about: Have a question wanted to be help
+title: "[Question] Question title"
 labels: question
 assignees: ''
 
 ---
 
 *For better global communication, please give priority to using English description, thx! *
 
+*Please review https://dolphinscheduler.apache.org/en-us/docs/development/issue.html when describe an issue.*
+
 **Describe the question**
 A clear and concise description of what the question is.
 
@@ -19,5 +21,5 @@ A clear and concise description of what the question is.
 **Additional context**
 Add any other context about the problem here.
 
-**Requirement or improvement
+**Requirement or improvement**
 - Please describe about your requirements or improvement suggestions.

.github/ISSUE_TEMPLATE/test.md

@@ -0,0 +1,24 @@
+---
+name: Test
+about: Test to enhance the robustness of this project
+title: "[Test][Module Name] Test title"
+labels: test
+assignees: ''
+
+---
+
+*For better global communication, please give priority to using English description, thx! *
+
+*Please review https://dolphinscheduler.apache.org/en-us/docs/development/issue.html when describe an issue.*
+
+**Describe the question**
+A clear and concise description of what the test part is.
+
+**What are the current deficiencies and the benefits of changing or adding this test**
+- A clear and concise description of the current deficiencies, the benefits of changing or adding this test, and the scope involved.
+
+**Which version of DolphinScheduler:**
+ -[1.1.0-preview]
+
+**Describe alternatives you've considered**
+A clear and concise description of any alternative solutions you've considered.

.github/workflows/ci_backend.yml

@@ -46,19 +46,19 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v2
+      # In the checkout@v2, it doesn't support git submodule. Execute the commands manually.
+      - name: checkout submodules
+        shell: bash
+        run: |
+          git submodule sync --recursive
+          git -c protocol.version=2 submodule update --init --force --recursive --depth=1
       - name: Set up JDK 1.8
         uses: actions/setup-java@v1
         with:
           java-version: 1.8
+      - name: Check license
+        run: ./mvnw -B apache-rat:check
       - name: Compile
-        run: mvn -B clean compile package -Prelease -Dmaven.test.skip=true
-  License-check:
-    runs-on: ubuntu-latest
-    steps:
-      - uses: actions/checkout@v2
-      - name: Set up JDK 1.8
-        uses: actions/setup-java@v1
-        with:
-          java-version: 1.8
-      - name: Check
-        run: mvn -B apache-rat:check
+        run: mvn -B clean compile install -Prelease -Dmaven.test.skip=true
+      - name: Check dependency license
+        run: tools/dependencies/check-LICENSE.sh

.github/workflows/ci_e2e.yml

@@ -30,6 +30,12 @@ jobs:
     steps:
 
       - uses: actions/checkout@v2
+      # In the checkout@v2, it doesn't support git submodule. Execute the commands manually.
+      - name: checkout submodules
+        shell: bash
+        run: |
+          git submodule sync --recursive
+          git -c protocol.version=2 submodule update --init --force --recursive --depth=1
       - uses: actions/cache@v1
         with:
           path: ~/.m2/repository
@@ -38,28 +44,31 @@ jobs:
             ${{ runner.os }}-maven-
       - name: Build Image
         run: |
-          export VERSION=`cat $(pwd)/pom.xml| grep "SNAPSHOT</version>" | awk -F "-SNAPSHOT" '{print $1}' | awk -F ">" '{print $2}'`
-          sh ./dockerfile/hooks/build
+          sh ./docker/build/hooks/build
       - name: Docker Run
         run: |
-          VERSION=`cat $(pwd)/pom.xml| grep "SNAPSHOT</version>" | awk -F "-SNAPSHOT" '{print $1}' | awk -F ">" '{print $2}'`
-          docker run -dit -e POSTGRESQL_USERNAME=test -e POSTGRESQL_PASSWORD=test -p 8888:8888 dolphinscheduler:$VERSION all
+          export VERSION=$(cat $(pwd)/pom.xml | grep '<version>' -m 1 | awk '{print $1}' | sed 's/<version>//' | sed 's/<\/version>//')
+          sed -i "s/apache\/dolphinscheduler:latest/apache\/dolphinscheduler:${VERSION}/g" $(pwd)/docker/docker-swarm/docker-compose.yml
+          docker-compose -f $(pwd)/docker/docker-swarm/docker-compose.yml up -d
       - name: Check Server Status
-        run: sh ./dockerfile/hooks/check
+        run: sh $(pwd)/docker/docker-swarm/check
       - name: Prepare e2e env
         run: |
-          sudo apt-get install -y libxss1 libappindicator1 libindicator7 xvfb unzip
+          sudo apt-get install -y libxss1 libappindicator1 libindicator7 xvfb unzip libgbm1
           wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
           sudo dpkg -i google-chrome*.deb
           sudo apt-get install -f -y
-          wget -N https://chromedriver.storage.googleapis.com/80.0.3987.106/chromedriver_linux64.zip
+          wget -N https://chromedriver.storage.googleapis.com/83.0.4103.39/chromedriver_linux64.zip
           unzip chromedriver_linux64.zip
           sudo mv -f chromedriver /usr/local/share/chromedriver
           sudo ln -s /usr/local/share/chromedriver /usr/local/bin/chromedriver
       - name: Run e2e Test
         run: cd ./e2e && mvn -B clean test
       - name: Collect logs
-        run: |
-          mkdir -p ${LOG_DIR}
-          docker logs dolphinscheduler > ${LOG_DIR}/dolphinscheduler.txt
-        continue-on-error: true
+        if: failure()
+        uses: actions/upload-artifact@v2
+        with:
+          name: dslogs
+          path: ${{ github.workspace }}/docker/docker-swarm/dolphinscheduler-logs
+
+

.github/workflows/ci_frontend.yml

@@ -35,6 +35,12 @@ jobs:
         os: [ubuntu-latest, macos-latest]
     steps:
       - uses: actions/checkout@v2
+      # In the checkout@v2, it doesn't support git submodule. Execute the commands manually.
+      - name: checkout submodules
+        shell: bash
+        run: |
+          git submodule sync --recursive
+          git -c protocol.version=2 submodule update --init --force --recursive --depth=1
       - name: Set up Node.js
         uses: actions/setup-node@v1
         with:
@@ -44,12 +50,19 @@ jobs:
           cd dolphinscheduler-ui
           npm install node-sass --unsafe-perm
           npm install
+          npm run lint
           npm run build
 
   License-check:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v2
+      # In the checkout@v2, it doesn't support git submodule. Execute the commands manually.
+      - name: checkout submodules
+        shell: bash
+        run: |
+          git submodule sync --recursive
+          git -c protocol.version=2 submodule update --init --force --recursive --depth=1
       - name: Set up JDK 1.8
         uses: actions/setup-java@v1
         with:

.github/workflows/ci_ut.yml

@@ -15,9 +15,12 @@
 # limitations under the License.
 #
 
-on: ["pull_request"]
+on:
+  pull_request:
+  push:
+    branches:
+      - dev
 env:
-  DOCKER_DIR: ./docker
   LOG_DIR: /tmp/dolphinscheduler
 
 name: Unit Test
@@ -30,40 +33,88 @@ jobs:
     steps:
 
       - uses: actions/checkout@v2
+      # In the checkout@v2, it doesn't support git submodule. Execute the commands manually.
+      - name: checkout submodules
+        shell: bash
+        run: |
+          git submodule sync --recursive
+          git -c protocol.version=2 submodule update --init --force --recursive --depth=1
       - uses: actions/cache@v1
         with:
           path: ~/.m2/repository
           key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }}
           restore-keys: |
             ${{ runner.os }}-maven-
       - name: Bootstrap database
-        run: cd ${DOCKER_DIR} && docker-compose up -d
+        run: |
+          sed -i "s/: root/: test/g" $(pwd)/docker/docker-swarm/docker-compose.yml
+          docker-compose -f $(pwd)/docker/docker-swarm/docker-compose.yml create --force-recreate dolphinscheduler-zookeeper dolphinscheduler-postgresql
+          sudo cp $(pwd)/sql/dolphinscheduler-postgre.sql $(docker volume inspect docker-swarm_dolphinscheduler-postgresql-initdb | grep "Mountpoint" | awk -F "\"" '{print $4}')
+          docker-compose -f $(pwd)/docker/docker-swarm/docker-compose.yml up -d dolphinscheduler-zookeeper dolphinscheduler-postgresql
       - name: Set up JDK 1.8
         uses: actions/setup-java@v1
         with:
           java-version: 1.8
+      - name: Git fetch unshallow
+        run: |
+          git fetch --unshallow
+          git config remote.origin.fetch "+refs/heads/*:refs/remotes/origin/*"
+          git fetch origin
       - name: Compile
         run: |
-          export MAVEN_OPTS='-Dmaven.repo.local=.m2/repository -XX:+TieredCompilation -XX:TieredStopAtLevel=1 -XX:+CMSClassUnloadingEnabled -XX:+UseConcMarkSweepGC -XX:-UseGCOverheadLimit -Xmx3g'
+          export MAVEN_OPTS='-Dmaven.repo.local=.m2/repository -XX:+TieredCompilation -XX:TieredStopAtLevel=1 -XX:+CMSClassUnloadingEnabled -XX:+UseConcMarkSweepGC -XX:-UseGCOverheadLimit -Xmx5g'
           mvn test -B -Dmaven.test.skip=false
+      - name: Upload coverage report to codecov
+        run: |
           CODECOV_TOKEN="09c2663f-b091-4258-8a47-c981827eb29a" bash <(curl -s https://codecov.io/bash)
+      # Set up JDK 11 for SonarCloud.
+      - name: Set up JDK 1.11
+        uses: actions/setup-java@v1
+        with:
+          java-version: 1.11
       - name: Run SonarCloud Analysis
         run: >
-          mvn verify --batch-mode
-          org.sonarsource.scanner.maven:sonar-maven-plugin:3.6.1.1688:sonar
+          mvn --batch-mode verify sonar:sonar
           -Dsonar.coverage.jacoco.xmlReportPaths=target/site/jacoco/jacoco.xml
           -Dmaven.test.skip=true
           -Dsonar.host.url=https://sonarcloud.io
           -Dsonar.organization=apache
           -Dsonar.core.codeCoveragePlugin=jacoco
           -Dsonar.projectKey=apache-dolphinscheduler
           -Dsonar.login=e4058004bc6be89decf558ac819aa1ecbee57682
+          -Dsonar.exclusions=dolphinscheduler-ui/src/**/i18n/locale/*.js,dolphinscheduler-microbench/src/**/*
         env:
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
           SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}
       - name: Collect logs
         run: |
           mkdir -p ${LOG_DIR}
-          cd ${DOCKER_DIR}
-          docker-compose logs db > ${LOG_DIR}/db.txt
+          docker-compose -f $(pwd)/docker/docker-swarm/docker-compose.yml logs dolphinscheduler-postgresql > ${LOG_DIR}/db.txt
         continue-on-error: true
+
+  Checkstyle:
+    name: Check code style
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v2
+      # In the checkout@v2, it doesn't support git submodule. Execute the commands manually.
+      - name: checkout submodules
+        shell: bash
+        run: |
+          git submodule sync --recursive
+          git -c protocol.version=2 submodule update --init --force --recursive --depth=1
+      - name: check code style
+        env:
+          WORKDIR: ./
+          REVIEWDOG_GITHUB_API_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          CHECKSTYLE_CONFIG: style/checkstyle.xml
+          REVIEWDOG_VERSION: v0.10.2
+        run: |
+          wget -O - -q https://github.com/checkstyle/checkstyle/releases/download/checkstyle-8.22/checkstyle-8.22-all.jar > /opt/checkstyle.jar
+          wget -O - -q https://raw.githubusercontent.com/reviewdog/reviewdog/master/install.sh | sh -s -- -b /opt ${REVIEWDOG_VERSION}
+          java -jar /opt/checkstyle.jar "${WORKDIR}" -c "${CHECKSTYLE_CONFIG}"  -f xml \
+               | /opt/reviewdog -f=checkstyle \
+                    -reporter="${INPUT_REPORTER:-github-pr-check}" \
+                    -filter-mode="${INPUT_FILTER_MODE:-added}" \
+                    -fail-on-error="${INPUT_FAIL_ON_ERROR:-false}"