Merge pull request #400 from snyk-tech-services/feat/de-activate-old-…

…projects

feat: de-activate projects no longer in the repo

docs/sync.md

@@ -1,47 +1,82 @@
 # Sync
+
 ## Table of Contents
+
 - [Sync](#sync)
   - [Table of Contents](#table-of-contents)
   - [Prerequisites](#prerequisites)
 - [What will change?](#what-will-change)
   - [Branches](#branches)
+  - [De-activating Snyk projects that represent files that have been renamed/moved/deleted](#de-activating-snyk-projects-that-represent-files-that-have-been-renamedmoveddeleted)
+    - [Scenarios](#scenarios)
+      - [File renamed/moves/deleted](#file-renamedmovesdeleted)
+      - [node\_modules, tests \& fixtures](#node_modules-tests--fixtures)
 - [Kick off sync](#kick-off-sync)
   - [1. Set the env vars](#1-set-the-env-vars)
-  - [2. Download & run](#2-download--run)
+  - [2. Download \& run](#2-download--run)
   - [Examples](#examples)
     - [Github.com](#githubcom)
     - [GitHub Enterprise Server](#github-enterprise-server)
     - [GitHub Enterprise Cloud](#github-enterprise-cloud)
+    - [Only syncing Container projects (Dockerfiles)](#only-syncing-container-projects-dockerfiles)
+    - [Only syncing Open Source + Iac projects (Dockerfiles)](#only-syncing-open-source--iac-projects-dockerfiles)
 - [Known limitations](#known-limitations)
 
 ## Prerequisites
+
 You will need to have setup in advance:
+
 - your [Snyk organizations](docs/orgs.md) should exist and have projects
 - your Snyk organizations configured with some connection to SCM (Github/Gitlab/Bitbucket etc) as you will need the provide which integration sync should use to update projects.
 - you will need your Snyk API token, with correct scope & [admin access for all Organizations](https://snyk.docs.apiary.io/#reference/import-projects/import/import-targets). This command will perform project changes on users behalf (import, update project branch, deactivate projects). **Github Integration Note**: As Github is both an auth & integration, how the integration is done has an effect on usage:
   - For users importing via [Github Snyk integration](https://docs.snyk.io/integrations/git-repository-scm-integrations/github-integration#setting-up-a-github-integration) use your **personal Snyk API token** (Service Accounts are not supported for Github integration imports via API as this is a personal auth token only accessible to the user)
   - For Github Enterprise Snyk integration with a url & token (for Github.com, Github Enterprise Cloud & Github Enterprise hosted) use a **Snyk API service account token**
 
-
 Any logs will be generated at `SNYK_LOG_PATH` directory.
 
 # What will change?
 
 ## Branches
+
 Updating the project branch in Snyk to match the default branch of the repo in the SCM. The drift can happen for several reasons:
-- branch was renamed in Github/Gitlab etc on a repo from e.g. from  `master` > `main`
+
+- branch was renamed in Github/Gitlab etc on a repo from e.g. from `master` > `main`
 - a new default branch was chosen from existing branches e.g. both `main` and `develop` exist as branches and default branch switched from `main` to `develop`
 
+## De-activating Snyk projects that represent files that have been renamed/moved/deleted
+
+During sync a shallow clone of a repo will be done to find all files in the repo and compare them to files monitored by Snyk. If any file is no longer found in the repo, the corresponding Snyk project will be deactivated.
+
+### Scenarios
+
+#### File renamed/moves/deleted
+
+If a file in a repo moved, was re-named, repo was re-named. These will be broken projects in Snyk and therefore deactivated by `sync` command e.g. `src/package.json` > `src/lib/package.json`
+
+#### node_modules, tests & fixtures
+
+Any projects that were imported but match the default exclusions list (deemed to be fixtures or tests) will also be deactivated. The list matches the same pattern used in Snyk during import via UI. The full list is:
+
+- `fixtures`
+- `tests`
+- `__tests__`
+- `test`
+- `__test__`
+- `ci`
+- `node_modules`
+- `bower_components`
+- `.git`
 
 # Kick off sync
+
 `sync` command will analyze existing projects & targets (repos) in Snyk organization and determine if any changes are needed.
 
 `--dryRun=true` - run the command first in dry-run mode to see what changes will be made in Snyk before running this again without if everything looks good. In this mode the last call to Snyk APIs to make the changes will be skipped but the logs will pretend as if it succeeded, the log entry will indicate this was generate in `dryRun` mode.
 
 The command will produce detailed logs for projects that were `updated` and those that needed an update but `failed`. If no changes are needed these will not be logged.
 
-
 ## 1. Set the env vars
+
 - `SNYK_TOKEN` - your [Snyk api token](https://app.snyk.io/account)
 - `SNYK_LOG_PATH` - the path to folder where all logs should be saved,it is recommended creating a dedicated logs folder per import you have running. (Note: all logs will append)
 - `SNYK_API` (optional) defaults to `https://snyk.io/api/v1`
@@ -51,7 +86,6 @@ The command will produce detailed logs for projects that were `updated` and thos
 
 Grab a binary from the [releases page](https://github.com/snyk-tech-services/snyk-api-import/releases) and run with `DEBUG=snyk* snyk-api-import-macos import --file=path/to/imported-targets.json`
 
-
 ## Examples
 
 ### Github.com
@@ -62,7 +96,6 @@ In dry-run mode:
 Live mode:
 `DEBUG=*snyk* SNYK_TOKEN=xxxx snyk-api-import sync --orgPublicId=<snyk_org_public_id> --source=github`
 
-
 ### GitHub Enterprise Server
 
 In dry-run mode:
@@ -71,8 +104,6 @@ In dry-run mode:
 Live mode:
 `DEBUG=*snyk* SNYK_TOKEN=xxxx snyk-api-import sync --orgPublicId=<snyk_org_public_id> --source=github-enterprise --sourceUrl=https://custom.ghe.com`
 
-
-
 ### GitHub Enterprise Cloud
 
 In dry-run mode:
@@ -81,7 +112,17 @@ In dry-run mode:
 Live mode:
 `DEBUG=*snyk* SNYK_TOKEN=xxxx snyk-api-import sync --orgPublicId=<snyk_org_public_id> --source=github-enterprise`
 
+### Only syncing Container projects (Dockerfiles)
+
+`--snykProduct` can be used to specify to sync projects belonging to Open Source, Container (Dockerfiles) or IaC products which represent files in Git repos.
+`DEBUG=*snyk* SNYK_TOKEN=xxxx snyk-api-import sync --orgPublicId=<snyk_org_public_id> --source=github-enterprise --snykProduct=container`
+
+### Only syncing Open Source + Iac projects (Dockerfiles)
+
+`DEBUG=*snyk* SNYK_TOKEN=xxxx snyk-api-import sync --orgPublicId=<snyk_org_public_id> --source=github-enterprise --snykProduct=open-source --snykProduct=iac`
 
 # Known limitations
+
 - Any organizations using a custom branch feature are currently not supported, `sync` will not continue.
 - Any organizations that previously used the custom feature flag should ideally delete all existing projects & re-import to restore the project names to standard format (do not include a branch in the project name). `sync` will work regardless but may cause confusion as the project name will reference a branch that is not likely to be the actual branch being tested.
+- It is not possible to know if a file was moved or renamed in the current implementation as it requires looking through commits history or using webhooks. It is also not currently possible to re-name projects in Snyk. In all cases projects will be deactivated and their replacement re-imported, creating a new projects with new history.

src/cmds/sync.ts

@@ -3,7 +3,10 @@ import * as yargs from 'yargs';
 const debug = debugLib('snyk:orgs-data-script');
 
 import { getLoggingPath } from '../lib/get-logging-path';
-import type { CommandResult } from '../lib/types';
+import type { SnykProductEntitlement } from '../lib/supported-project-types/supported-manifests';
+import { CommandResult, productEntitlements } from '../lib/types';
+import { SupportedProductsUpdateProject } from '../lib/types';
+
 import { SupportedIntegrationTypesUpdateProject } from '../lib/types';
 
 import { updateOrgTargets } from '../scripts/sync/sync-org-projects';
@@ -33,26 +36,47 @@ export const builder = {
     default: false,
     desc: 'Dry run option. Will create a log file listing the potential updates',
   },
+  snykProduct: {
+    required: false,
+    default: SupportedProductsUpdateProject.OPEN_SOURCE,
+    choices: [...Object.values(SupportedProductsUpdateProject)],
+    desc: 'List of Snyk Products to consider when syncing an SCM repo. Monitored Snyk Code repos are automatically synced already, if Snyk Code is enabled any new repo imports will include Snyk Code projects',
+  },
 };
 
 export async function syncOrg(
   source: SupportedIntegrationTypesUpdateProject[],
   orgPublicId: string,
   sourceUrl?: string,
   dryRun?: boolean,
+  entitlements: SnykProductEntitlement[] = [],
+  manifestTypes?: string[],
 ): Promise<CommandResult> {
   try {
     getLoggingPath();
 
-    const res = await updateOrgTargets(orgPublicId, source, dryRun, sourceUrl);
+    const res = await updateOrgTargets(
+      orgPublicId,
+      source,
+      dryRun,
+      sourceUrl,
+      entitlements,
+      manifestTypes,
+    );
 
     const nothingToUpdate =
       res.processedTargets == 0 &&
       res.meta.projects.updated.length == 0 &&
       res.meta.projects.failed.length == 0;
     const orgMessage = nothingToUpdate
       ? `Did not detect any changes to apply`
-      : `Processed ${res.processedTargets} targets (${res.failedTargets} failed)\nUpdated ${res.meta.projects.updated.length} projects\n${res.meta.projects.failed.length} projects failed to update\nFind more information in ${res.fileName} and ${res.failedFileName}`;
+      : `Processed ${res.processedTargets} targets (${
+          res.failedTargets
+        } failed)\nUpdated ${
+          res.meta.projects.updated.length
+        } projects\nFind more information in ${res.fileName}${
+          res.failedFileName ? ` and ${res.failedFileName}` : ''
+        }`;
 
     return {
       fileName: res.fileName,
@@ -76,16 +100,43 @@ export async function handler(argv: {
   orgPublicId: string;
   sourceUrl?: string;
   dryRun?: boolean;
+  snykProduct?: SupportedProductsUpdateProject[];
 }): Promise<void> {
-  const { source, orgPublicId, sourceUrl, dryRun } = argv;
+  const {
+    source,
+    orgPublicId,
+    sourceUrl,
+    dryRun,
+    snykProduct = [SupportedProductsUpdateProject.OPEN_SOURCE],
+  } = argv;
   debug('ℹ️  Options: ' + JSON.stringify(argv));
 
   const sourceList: SupportedIntegrationTypesUpdateProject[] = [];
   sourceList.push(source);
 
+  const manifestTypes: string[] = [];
+  const entitlements: SnykProductEntitlement[] = [];
+
+  const products = Array.isArray(snykProduct) ? snykProduct : [snykProduct];
+  for (const p of products) {
+    entitlements.push(productEntitlements[p]);
+  }
+  console.log(
+    `ℹ️  Running sync for ${source} projects in org: ${orgPublicId} (products to be synced: ${products.join(
+      ',',
+    )})`,
+  );
+
   // when the input will be a file we will need to
   // add a function to read and parse the file
-  const res = await syncOrg(sourceList, orgPublicId, sourceUrl, dryRun);
+  const res = await syncOrg(
+    sourceList,
+    orgPublicId,
+    sourceUrl,
+    dryRun,
+    entitlements,
+    manifestTypes,
+  );
 
   if (res.exitCode === 1) {
     debug('Failed to sync organizations.\n' + res.message);

src/lib/api/project/index.ts

@@ -5,6 +5,7 @@ import { getApiToken } from '../../get-api-token';
 import { getSnykHost } from '../../get-snyk-host';
 import type { requestsManager } from 'snyk-request-manager';
 import type { SnykProject } from '../../types';
+import { straightThroughBufferTask } from 'simple-git/dist/src/lib/tasks/task';
 const debug = debugLib('snyk:api-project');
 
 interface BulkProjectUpdateResponse {
@@ -14,7 +15,10 @@ interface BulkProjectUpdateResponse {
 export async function deleteProjects(
   orgId: string,
   projects: string[],
-): Promise<BulkProjectUpdateResponse> {
+): Promise<{
+  success: BulkProjectUpdateResponse[];
+  failure: BulkProjectUpdateResponse[];
+}> {
   const apiToken = getApiToken();
   if (!(orgId && projects)) {
     throw new Error(
@@ -63,6 +67,41 @@ export async function deleteProjects(
   }
 }
 
+export async function deactivateProject(
+  requestManager: requestsManager,
+  orgId: string,
+  projectPublicId: string,
+): Promise<boolean> {
+  getApiToken();
+  getSnykHost();
+  if (!(orgId && projectPublicId)) {
+    throw new Error(
+      `Missing required parameters. Please ensure you have provided: orgId & projectId.`,
+    );
+  }
+  debug(`De-activating project: ${projectPublicId}`);
+  const url = `/org/${orgId.trim()}/project/${projectPublicId}/deactivate`;
+
+  const res = await requestManager.request({
+    verb: 'post',
+    url: url,
+    body: JSON.stringify({}),
+    useRESTApi: false,
+  });
+
+  const statusCode = res.statusCode || res.status;
+  if (!statusCode || statusCode !== 200) {
+    debug(`Failed de-activating project projectId`);
+    throw new Error(
+      'Expected a 200 response, instead received: ' +
+        JSON.stringify({ data: res.data, status: statusCode }),
+    );
+  }
+
+  debug('Updated projects batch');
+  return true;
+}
+
 export async function updateProject(
   requestManager: requestsManager,
   orgId: string,

src/lib/find-files.ts

@@ -110,7 +110,7 @@ function findFile(
 ): string | null {
   if (filter.length > 0) {
     const filename = pathLib.basename(path);
-    if (matches(filename, filter)) {
+    if (matches(filename, filter) || matches(path, filter)) {
       return path;
     }
   } else {
@@ -154,5 +154,5 @@ async function findInDirectory(
 }
 
 function matches(filePath: string, globs: string[]): boolean {
-  return globs.some((glob) => micromatch.isMatch(filePath, glob));
+  return globs.some((glob) => micromatch.isMatch(filePath, '**/' + glob));
 }

src/lib/git-clone.ts

@@ -21,6 +21,7 @@ interface GitCloneResponse {
   repoPath?: string;
   gitResponse: string;
 }
+
 export async function gitClone(
   integrationType: SupportedIntegrationTypesUpdateProject,
   meta: RepoMetaData,

src/lib/types.ts

@@ -1,3 +1,5 @@
+import { SnykProductEntitlement } from './supported-project-types/supported-manifests';
+
 export interface ImportTarget {
   orgId: string;
   integrationId: string;
@@ -96,6 +98,19 @@ export enum SupportedIntegrationTypesUpdateProject {
   GHE = 'github-enterprise',
 }
 
+export enum SupportedProductsUpdateProject {
+  CONTAINER = 'container',
+  OPEN_SOURCE = 'open-source',
+  IAC = 'iac',
+}
+
+export const productEntitlements: {
+  [key in SupportedProductsUpdateProject]: SnykProductEntitlement;
+} = {
+  [SupportedProductsUpdateProject.IAC]: 'infrastructureAsCode',
+  [SupportedProductsUpdateProject.OPEN_SOURCE]: 'openSource',
+  [SupportedProductsUpdateProject.CONTAINER]: 'dockerfileFromScm',
+};
 // used to generate imported targets that exist in Snyk
 // when we need to grab the integrationId from Snyk
 export enum SupportedIntegrationTypesToListSnykTargets {

src/scripts/import-projects.ts

@@ -9,7 +9,7 @@ import {
   pollImportUrls,
   getConcurrentImportsNumber,
 } from '../lib';
-import { Project, ImportTarget } from '../lib/types';
+import type { Project, ImportTarget } from '../lib/types';
 import { getLoggingPath } from '../lib';
 import { logImportedBatch } from '../loggers/log-imported-batch';
 import { IMPORT_LOG_NAME } from '../common';

src/scripts/sync/clone-and-analyze.ts

@@ -4,6 +4,7 @@ import * as path from 'path';
 
 import { find, getSCMSupportedManifests, gitClone } from '../../lib';
 import type { SnykProductEntitlement } from '../../lib/supported-project-types/supported-manifests';
+import { getSCMSupportedProjectTypes } from '../../lib/supported-project-types/supported-manifests';
 import type {
   RepoMetaData,
   SnykProject,
@@ -35,6 +36,10 @@ export async function cloneAndAnalyze(
   import: string[];
   deactivate: SnykProject[];
 }> {
+  const manifestFileTypes =
+    manifestTypes && manifestTypes.length > 0
+      ? manifestTypes
+      : getSCMSupportedProjectTypes(entitlements);
   const { success, repoPath, gitResponse } = await gitClone(
     integrationType,
     repoMetadata,
@@ -54,7 +59,7 @@ export async function cloneAndAnalyze(
     // TODO: when possible switch to check entitlements via API automatically for an org
     // right now the product entitlements are not exposed via API so user has to provide which products
     // they are using
-    getSCMSupportedManifests(manifestTypes, entitlements),
+    getSCMSupportedManifests(manifestFileTypes, entitlements),
     6,
   );
   const relativeFileNames = files.map((f) => path.relative(repoPath, f));
@@ -69,6 +74,6 @@ export async function cloneAndAnalyze(
   return generateProjectDiffActions(
     relativeFileNames,
     snykMonitoredProjects,
-    manifestTypes,
+    manifestFileTypes,
   );
 }