Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Next rebased #5836

Closed
wants to merge 340 commits into from
Closed

Next rebased #5836

wants to merge 340 commits into from

Conversation

brandonrising
Copy link
Collaborator

@brandonrising brandonrising commented Feb 29, 2024

What type of PR is this? (check all applicable)

  • Refactor
  • Feature
  • Bug Fix
  • Optimization
  • Documentation Update
  • Community Node Submission

Have you discussed this change with the InvokeAI team?

  • Yes
  • No, because:

Have you updated all relevant documentation?

  • Yes
  • No

Description

This is a PR to merge the next branch into main. I created a separate branch for the rebase due to branch protection rules. The rebase conflicts on main can be found below.

This PR includes:

  • Session Processor remake
  • Removal of invocation processor and invocation queue
  • Create invocation context for improving the relationship between nodes and services
  • Create invocation context data to provide a reliable path for delivering information to nodes
  • Model Manager rewrite
  • Queues for downloading multiple model installs in parallel
  • Metadata provided on remote model installs
  • New Model Manager UI
  • New request structure for models in graphs

Related Tickets & Documents

diff --cc invokeai/frontend/web/src/services/api/index.ts
index 6a342bb72,1f567d790..000000000
--- a/invokeai/frontend/web/src/services/api/index.ts
+++ b/invokeai/frontend/web/src/services/api/index.ts
@@@ -52,22 -52,10 +52,27 @@@ const dynamicBaseQuery: BaseQueryFn<str
    const baseUrl = $baseUrl.get();
    const authToken = $authToken.get();
    const projectId = $projectId.get();
 +  const isOpenAPIRequest =
 +    (args instanceof Object && args.url.includes('openapi.json')) ||
 +    (typeof args === 'string' && args.includes('openapi.json'));
  
    const fetchBaseQueryArgs: FetchBaseQueryArgs = {
++<<<<<<< HEAD
 +    baseUrl: baseUrl ? `${baseUrl}/api/v1` : `${window.location.href.replace(/\/$/, '')}/api/v1`,
 +  };
 +
 +  // When fetching the openapi.json, we need to remove circular references from the JSON.
 +  if (isOpenAPIRequest) {
 +    fetchBaseQueryArgs.jsonReplacer = getCircularReplacer();
 +  }
 +
 +  // openapi.json isn't protected by authorization, but all other requests need to include the auth token and project id.
 +  if (!isOpenAPIRequest) {
 +    fetchBaseQueryArgs.prepareHeaders = (headers) => {
++=======
+     baseUrl: baseUrl || window.location.href.replace(/\/$/, ''),
+     prepareHeaders: (headers) => {
++>>>>>>> 86a372b02 (refactor(ui): url builders for each router)
        if (authToken) {
          headers.set('Authorization', `Bearer ${authToken}`);
        }
 
 
 Revision 
 
 
diff --cc invokeai/frontend/web/src/services/api/index.ts
index 6a342bb72,1f567d790..000000000
--- a/invokeai/frontend/web/src/services/api/index.ts
+++ b/invokeai/frontend/web/src/services/api/index.ts
@@@ -52,22 -52,10 +52,22 @@@ const dynamicBaseQuery: BaseQueryFn<str
    const baseUrl = $baseUrl.get();
    const authToken = $authToken.get();
    const projectId = $projectId.get();
 +  const isOpenAPIRequest =
 +    (args instanceof Object && args.url.includes('openapi.json')) ||
 +    (typeof args === 'string' && args.includes('openapi.json'));
  
    const fetchBaseQueryArgs: FetchBaseQueryArgs = {
-     baseUrl: baseUrl ? `${baseUrl}/api/v1` : `${window.location.href.replace(/\/$/, '')}/api/v1`,
 -    baseUrl: baseUrl || window.location.href.replace(/\/$/, ''),
 -    prepareHeaders: (headers) => {
++baseUrl: baseUrl || window.location.href.replace(/\/$/, ''),
 +  };
 +
 +  // When fetching the openapi.json, we need to remove circular references from the JSON.
 +  if (isOpenAPIRequest) {
 +    fetchBaseQueryArgs.jsonReplacer = getCircularReplacer();
 +  }
 +
 +  // openapi.json isn't protected by authorization, but all other requests need to include the auth token and project id.
 +  if (!isOpenAPIRequest) {
 +    fetchBaseQueryArgs.prepareHeaders = (headers) => {
        if (authToken) {
          headers.set('Authorization', `Bearer ${authToken}`);
        }
 
--------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
diff --cc invokeai/frontend/web/src/features/parameters/components/MainModel/ParamMainModelSelect.tsx
index 18f780bde,84b112eb5..000000000
--- a/invokeai/frontend/web/src/features/parameters/components/MainModel/ParamMainModelSelect.tsx
+++ b/invokeai/frontend/web/src/features/parameters/components/MainModel/ParamMainModelSelect.tsx
@@@ -1,16 -1,14 +1,22 @@@
++<<<<<<< HEAD
 +import { Box, Combobox, FormControl, FormLabel, Tooltip } from '@invoke-ai/ui-library';
 +import { createMemoizedSelector } from 'app/store/createMemoizedSelector';
 +import { useAppDispatch, useAppSelector } from 'app/store/storeHooks';
 +import { InformationalPopover } from 'common/components/InformationalPopover/InformationalPopover';
 +import { useGroupedModelCombobox } from 'common/hooks/useGroupedModelCombobox';
++=======
+ import { CustomSelect, FormControl, FormLabel } from '@invoke-ai/ui-library';
+ import { createMemoizedSelector } from 'app/store/createMemoizedSelector';
+ import { useAppDispatch, useAppSelector } from 'app/store/storeHooks';
+ import { useModelCustomSelect } from 'common/hooks/useModelCustomSelect';
++>>>>>>> 17f5484f5 (feat(ui): fix main model & control adapter model selects)
  import { modelSelected } from 'features/parameters/store/actions';
  import { selectGenerationSlice } from 'features/parameters/store/generationSlice';
- import { pick } from 'lodash-es';
- import { memo, useCallback, useMemo } from 'react';
+ import { memo, useCallback } from 'react';
  import { useTranslation } from 'react-i18next';
  import { NON_REFINER_BASE_MODELS } from 'services/api/constants';
- import type { MainModelConfig } from 'services/api/endpoints/models';
- import { getModelId, mainModelsAdapterSelectors, useGetMainModelsQuery } from 'services/api/endpoints/models';
+ import { useGetMainModelsQuery } from 'services/api/endpoints/models';
+ import type { MainModelConfig } from 'services/api/types';
  
  const selectModel = createMemoizedSelector(selectGenerationSlice, (generation) => generation.model);
  
@@@ -42,21 -36,9 +44,27 @@@ const ParamMainModelSelect = () => 
    });
  
    return (
++<<<<<<< HEAD
 +    <FormControl isDisabled={!options.length} isInvalid={!options.length}>
 +      <InformationalPopover feature="paramModel">
 +        <FormLabel>{t('modelManager.model')}</FormLabel>
 +      </InformationalPopover>
 +      <Tooltip label={tooltipLabel}>
 +        <Box w="full">
 +          <Combobox
 +            value={value}
 +            placeholder={placeholder}
 +            options={options}
 +            onChange={onChange}
 +            noOptionsMessage={noOptionsMessage}
 +          />
 +        </Box>
 +      </Tooltip>
++=======
+     <FormControl isDisabled={!items.length} isInvalid={!selectedItem || !items.length}>
+       <FormLabel>{t('modelManager.model')}</FormLabel>
+       <CustomSelect selectedItem={selectedItem} placeholder={placeholder} items={items} onChange={onChange} />
++>>>>>>> 17f5484f5 (feat(ui): fix main model & control adapter model selects)
      </FormControl>
    );
  };
 
 
 Revision 
 
 
diff --cc invokeai/frontend/web/src/features/parameters/components/MainModel/ParamMainModelSelect.tsx
index 18f780bde,84b112eb5..000000000
--- a/invokeai/frontend/web/src/features/parameters/components/MainModel/ParamMainModelSelect.tsx
+++ b/invokeai/frontend/web/src/features/parameters/components/MainModel/ParamMainModelSelect.tsx
@@@ -1,16 -1,14 +1,15 @@@
- import { Box, Combobox, FormControl, FormLabel, Tooltip } from '@invoke-ai/ui-library';
+ import { CustomSelect, FormControl, FormLabel } from '@invoke-ai/ui-library';
  import { createMemoizedSelector } from 'app/store/createMemoizedSelector';
  import { useAppDispatch, useAppSelector } from 'app/store/storeHooks';
 +import { InformationalPopover } from 'common/components/InformationalPopover/InformationalPopover';
- import { useGroupedModelCombobox } from 'common/hooks/useGroupedModelCombobox';
+ import { useModelCustomSelect } from 'common/hooks/useModelCustomSelect';
  import { modelSelected } from 'features/parameters/store/actions';
  import { selectGenerationSlice } from 'features/parameters/store/generationSlice';
- import { pick } from 'lodash-es';
- import { memo, useCallback, useMemo } from 'react';
+ import { memo, useCallback } from 'react';
  import { useTranslation } from 'react-i18next';
  import { NON_REFINER_BASE_MODELS } from 'services/api/constants';
- import type { MainModelConfig } from 'services/api/endpoints/models';
- import { getModelId, mainModelsAdapterSelectors, useGetMainModelsQuery } from 'services/api/endpoints/models';
+ import { useGetMainModelsQuery } from 'services/api/endpoints/models';
+ import type { MainModelConfig } from 'services/api/types';
  
  const selectModel = createMemoizedSelector(selectGenerationSlice, (generation) => generation.model);
  
@@@ -42,21 -36,9 +37,11 @@@ const ParamMainModelSelect = () => 
    });
  
    return (
-     <FormControl isDisabled={!options.length} isInvalid={!options.length}>
+     <FormControl isDisabled={!items.length} isInvalid={!selectedItem || !items.length}>
 -      <FormLabel>{t('modelManager.model')}</FormLabel>
 +      <InformationalPopover feature="paramModel">
 +        <FormLabel>{t('modelManager.model')}</FormLabel>
 +      </InformationalPopover>
-       <Tooltip label={tooltipLabel}>
-         <Box w="full">
-           <Combobox
-             value={value}
-             placeholder={placeholder}
-             options={options}
-             onChange={onChange}
-             noOptionsMessage={noOptionsMessage}
-           />
-         </Box>
-       </Tooltip>
+       <CustomSelect selectedItem={selectedItem} placeholder={placeholder} items={items} onChange={onChange} />
      </FormControl>
    );
  };
 
--------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
diff --git a/invokeai/frontend/web/public/locales/en.json b/invokeai/frontend/web/public/locales/en.json
index 26d95e4c0..2f9064a77 100644
--- a/invokeai/frontend/web/public/locales/en.json
+++ b/invokeai/frontend/web/public/locales/en.json
@@ -1130,8 +1130,8 @@
         "codeformerFidelity": "Fidelity",
         "coherenceMode": "Mode",
         "coherencePassHeader": "Coherence Pass",
-        "coherenceSteps": "Steps",
-        "coherenceStrength": "Strength",
+        "coherenceEdgeSize": "Edge Size",
+        "coherenceMinDenoise": "Min Denoise",
         "compositingSettingsHeader": "Compositing Settings",
         "controlNetControlMode": "Control Mode",
         "copyImage": "Copy Image",
@@ -1478,6 +1478,7 @@
             "heading": "Mode",
             "paragraphs": ["Method used to create a coherent image with the newly generated masked area."]
         },
+<<<<<<< HEAD
         "compositingCoherenceSteps": {
             "heading": "Steps",
             "paragraphs": ["Number of steps in the Coherence Pass.", "Similar to Generation Steps."]
@@ -1485,6 +1486,18 @@
         "compositingStrength": {
             "heading": "Strength",
             "paragraphs": ["Amount of noise added for the Coherence Pass.", "Similar to Denoising Strength."]
+=======
+        "compositingCoherenceEdgeSize": {
+            "heading": "Edge Size",
+            "paragraphs": ["The edge size of the coherence pass."]
+        },
+        "compositingCoherenceMinDenoise": {
+            "heading": "Minimum Denoise",
+            "paragraphs": [
+                "Minimum denoise strength for the Coherence mode",
+                "The minimum denoise strength for the coherence region when inpainting or outpainting"
+            ]
+>>>>>>> 30b6a0ee2 (wip(ui): Replace 2 Layer Coherence pass with Gradient Mask)
         },
         "compositingMaskAdjustments": {
             "heading": "Mask Adjustments",
@@ -1762,6 +1775,9 @@
         "clearCanvasHistoryMessage": "Clearing the canvas history leaves your current canvas intact, but irreversibly clears the undo and redo history.",
         "clearHistory": "Clear History",
         "clearMask": "Clear Mask (Shift+C)",
+        "coherenceModeGaussianBlur": "Gaussian Blur",
+        "coherenceModeBoxBlur": "Box Blur",
+        "coherenceModeStaged": "Staged",
         "colorPicker": "Color Picker",
         "copyToClipboard": "Copy to Clipboard",
         "cursorPosition": "Cursor Position",
 
 
 Revision 
 
 
diff --git a/invokeai/frontend/web/public/locales/en.json b/invokeai/frontend/web/public/locales/en.json
index 26d95e4c0..3beb57add 100644
--- a/invokeai/frontend/web/public/locales/en.json
+++ b/invokeai/frontend/web/public/locales/en.json
@@ -1130,8 +1130,8 @@
         "codeformerFidelity": "Fidelity",
         "coherenceMode": "Mode",
         "coherencePassHeader": "Coherence Pass",
-        "coherenceSteps": "Steps",
-        "coherenceStrength": "Strength",
+        "coherenceEdgeSize": "Edge Size",
+        "coherenceMinDenoise": "Min Denoise",
         "compositingSettingsHeader": "Compositing Settings",
         "controlNetControlMode": "Control Mode",
         "copyImage": "Copy Image",
@@ -1486,6 +1486,17 @@
             "heading": "Strength",
             "paragraphs": ["Amount of noise added for the Coherence Pass.", "Similar to Denoising Strength."]
         },
+        "compositingCoherenceEdgeSize": {
+            "heading": "Edge Size",
+            "paragraphs": ["The edge size of the coherence pass."]
+        },
+        "compositingCoherenceMinDenoise": {
+            "heading": "Minimum Denoise",
+            "paragraphs": [
+                "Minimum denoise strength for the Coherence mode",
+                "The minimum denoise strength for the coherence region when inpainting or outpainting"
+            ]
+        },
         "compositingMaskAdjustments": {
             "heading": "Mask Adjustments",
             "paragraphs": ["Adjust the mask."]
@@ -1762,6 +1773,9 @@
         "clearCanvasHistoryMessage": "Clearing the canvas history leaves your current canvas intact, but irreversibly clears the undo and redo history.",
         "clearHistory": "Clear History",
         "clearMask": "Clear Mask (Shift+C)",
+        "coherenceModeGaussianBlur": "Gaussian Blur",
+        "coherenceModeBoxBlur": "Box Blur",
+        "coherenceModeStaged": "Staged",
         "colorPicker": "Color Picker",
         "copyToClipboard": "Copy to Clipboard",
         "cursorPosition": "Cursor Position",

QA Instructions, Screenshots, Recordings

Merge Plan

This PR can be merged when approved
#dev-chat on discord needs to be advised of this change when it is merged

Added/updated tests?

  • Yes
  • No : please replace this line with details on why tests
    have not been included

brandonrising and others added 30 commits February 29, 2024 14:29
This is useful for the zod schemas and types we have created to match the backend.
- Update most model identifiers to be `{key: string}` instead of name/base/type. Doesn't change the model select components yet.
- Update model _parameters_, stored in redux, to be `{key: string, base: BaseModel}` - we need to store the base model to be able to check model compatibility. May want to store the whole config? Not sure...
The MM2 router is at `api/v2/models`. URL builder utils make this a bit easier to manage.
- ModelMetadataStoreService is now injected into ModelRecordStoreService
  (these two services are really joined at the hip, and should someday be merged)
- ModelRecordStoreService is now injected into ModelManagerService
- Reduced timeout value for the various installer and download wait*() methods
- Introduced a Mock modelmanager for testing
- Removed bare print() statement with _logger in the install helper backend.
- Removed unused code from model loader init file
- Made `locker` a private variable in the `LoadedModel` object.
- Fixed up model merge frontend (will be deprecated anyway!)
- Rename old "model_management" directory to "model_management_OLD" in order to catch
  dangling references to original model manager.
- Caught and fixed most dangling references (still checking)
- Rename lora, textual_inversion and model_patcher modules
- Introduce a RawModel base class to simplfy the Union returned by the
  model loaders.
- Tidy up the model manager 2-related tests. Add useful fixtures, and
  a finalizer to the queue and installer fixtures that will stop the
  services and release threads.
- Replace AnyModelLoader with ModelLoaderRegistry
- Fix type check errors in multiple files
- Remove apparently unneeded `get_model_config_enum()` method from model manager
- Remove last vestiges of old model manager
- Updated tests and documentation

resolve conflict with seamless.py
Also default CNet preprocessors to "RGB"
We use pydantic to validate a union of valid invocations when instantiating a graph.

Previously, we constructed the union while creating the `Graph` class. This introduces a dependency on the order of imports.

For example, consider a setup where we have 3 invocations in the app:

- Python executes the module where `FirstInvocation` is defined, registering `FirstInvocation`.
- Python executes the module where `SecondInvocation` is defined, registering `SecondInvocation`.
- Python executes the module where `Graph` is defined. A union of invocations is created and used to define the `Graph.nodes` field. The union contains `FirstInvocation` and `SecondInvocation`.
- Python executes the module where `ThirdInvocation` is defined, registering `ThirdInvocation`.
- A graph is created that includes `ThirdInvocation`. Pydantic validates the graph using the union, which does not know about `ThirdInvocation`, raising a `ValidationError` about an unknown invocation type.

This scenario has been particularly problematic in tests, where we may create invocations dynamically. The test files have to be structured in such a way that the imports happen in the right order. It's a major pain.

This PR refactors the validation of graph nodes to resolve this issue:

- `BaseInvocation` gets a new method `get_typeadapter`. This builds a pydantic `TypeAdapter` for the union of all registered invocations, caching it after the first call.
- `Graph.nodes`'s type is widened to `dict[str, BaseInvocation]`. This actually is a nice bonus, because we get better type hints whenever we reference `some_graph.nodes`.
- A "plain" field validator takes over the validation logic for `Graph.nodes`. "Plain" validators totally override pydantic's own validation logic. The validator grabs the `TypeAdapter` from `BaseInvocation`, then validates each node with it. The validation is identical to the previous implementation - we get the same errors.

`BaseInvocationOutput` gets the same treatment.
The change to `Graph.nodes` and `GraphExecutionState.results` validation requires some fanagling to get the OpenAPI schema generation to work. See new comments for a details.
`GraphInvocation` is a node that can contain a whole graph. It is removed for a number of reasons:

1. This feature was unused (the UI doesn't support it) and there is no plan for it to be used.

The use-case it served is known in other node execution engines as "node groups" or "blocks" - a self-contained group of nodes, which has group inputs and outputs. This is a planned feature that will be handled client-side.

2. It adds substantial complexity to the graph processing logic. It's probably not enough to have a measurable performance impact but it does make it harder to work in the graph logic.

3. It allows for graphs to be recursive, and the improved invocations union handling does not play well with it. Actually, it works fine within `graph.py` but not in the tests for some reason. I do not understand why. There's probably a workaround, but I took this as encouragement to remove `GraphInvocation` from the app since we don't use it.
@github-actions github-actions bot added documentation Improvements or additions to documentation api python PRs that change python files Root PythonDeps invocations PRs that change invocations backend PRs that change backend files services PRs that change app services frontend-deps PRs that change frontend dependencies frontend PRs that change frontend files CICD labels Feb 29, 2024
Copy link
Collaborator

@psychedelicious psychedelicious left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ty! I'm going to do the rebase also and compare the diff just as a sanity check

@brandonrising brandonrising deleted the next-rebased branch February 29, 2024 23:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api backend PRs that change backend files CICD documentation Improvements or additions to documentation frontend PRs that change frontend files frontend-deps PRs that change frontend dependencies invocations PRs that change invocations python PRs that change python files Root services PRs that change app services
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants