NIFI-13976 Added REST Endpoints for Copying and Pasting Flows (#9535)

- Including the revision for the current Process Group in the flow response.
- Introduce a method propose built for adding new versioned components to an existing Process Group.
- Updating the paste response payload to only include the entities for the components that were just added.
- Always generating new IDs when copying components.
- Creating an ID for the copy action.
- Code clean up.
- Mapping IDs on paste to ensure connections are created correctly.
- Authorizing any components whose instance IDs are specified.
- Copying any sensitive properties from copied instances when pasting.
- Fixing mapping of copy response to ensure that nested Controller Services can be referenced.
- Including external Controller Services in copy response to they can be resolved on paste.
- Excluding unreferenced services from copy response.
- Including property descriptors to allow external services to resolve when pasting.
- Including Parameters and Parameter Providers in copy response.
- Fixing issue returning unreferenced services in the current process group.
- Reverting changes building up external service references.
- Only including external services that are references by the copy selection.
- Adding some additional test coverage.
- Adding consideration for the current ids in the paste id mapping.
- Adding an endpoint response merger for the paste endpoint.
- Including version control information in copy response.
- Being more lenient when pasting unknown registry clients.
- Ensuring groups IDs are mapped correctly when and when not versioned.
- Including registry client id in vci of versioned process groups.
- Updating copy/paste logic in existing system tests to leverage new endpoints.
- Restoring originally proposed port name if possible.
- Allow ports to be pasted into the root group.
- Fixing authorization issue when pasting a flow containing Parameter Providers.
- Fixing issue when authorizing exporting a Flow that contains a Process Group bound to a Parameter Context.
- Introducing some Copy/Paste system tests.
- Adding system tests for verify various copy paste functionality.
- Ensuring appropriate access around Parameter Providers following resolution based on locally available Parameter Providers.
- Ensuring user has access to controller services directly referenced.
- Fixing NPE in log message of Parameter Context.
- Adding more copy/paste system tests.
- Moving copy/paste version tests into the CopyPasteIT.
- Fixing unit test mocking.
- Include service authorization based on versioned component id since it may resolve during synchronization.
- Skipping properties with no values when resolving service identifiers.

Signed-off-by: David Handermann <exceptionfactory@apache.org>

nifi-framework-api/src/main/java/org/apache/nifi/authorization/resource/VersionedComponentAuthorizable.java

@@ -0,0 +1,32 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.authorization.resource;
+
+import java.util.Optional;
+
+/**
+ * ComponentAuthorizable that also exposes the versioned identifier of the underlying component.
+ */
+public interface VersionedComponentAuthorizable extends ComponentAuthorizable {
+
+    /**
+     * The versioned component identifier of the underlying component.
+     *
+     * @return the versioned component identifier
+     */
+    Optional<String> getVersionedComponentId();
+}

nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/entity/CopyRequestEntity.java

@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.web.api.entity;
+
+import io.swagger.v3.oas.annotations.media.Schema;
+import jakarta.xml.bind.annotation.XmlType;
+
+import java.util.HashSet;
+import java.util.Set;
+
+/**
+ * A request to copy a portion of the flow.
+ */
+@XmlType(name = "copyRequestEntity")
+public class CopyRequestEntity extends Entity {
+
+    private Set<String> processGroups = new HashSet<>();
+    private Set<String> remoteProcessGroups = new HashSet<>();
+    private Set<String> processors = new HashSet<>();
+    private Set<String> inputPorts = new HashSet<>();
+    private Set<String> outputPorts = new HashSet<>();
+    private Set<String> connections = new HashSet<>();
+    private Set<String> labels = new HashSet<>();
+    private Set<String> funnels = new HashSet<>();
+
+    /**
+     * @return the ids of the connections to be copied.
+     */
+    @Schema(description = "The ids of the connections to be copied."
+    )
+    public Set<String> getConnections() {
+        return connections;
+    }
+
+    public void setConnections(Set<String> connections) {
+        this.connections = connections;
+    }
+
+    /**
+     * @return the ids of the funnels to be copied.
+     */
+    @Schema(description = "The ids of the funnels to be copied."
+    )
+    public Set<String> getFunnels() {
+        return funnels;
+    }
+
+    public void setFunnels(Set<String> funnels) {
+        this.funnels = funnels;
+    }
+
+    /**
+     * @return the ids of the input port to be copied.
+     */
+    @Schema(description = "The ids of the input ports to be copied."
+    )
+    public Set<String> getInputPorts() {
+        return inputPorts;
+    }
+
+    public void setInputPorts(Set<String> inputPorts) {
+        this.inputPorts = inputPorts;
+    }
+
+    /**
+     * @return the ids of the labels to be copied.
+     */
+    @Schema(description = "The ids of the labels to be copied."
+    )
+    public Set<String> getLabels() {
+        return labels;
+    }
+
+    public void setLabels(Set<String> labels) {
+        this.labels = labels;
+    }
+
+    /**
+     * @return the ids of the output ports to be copied.
+     */
+    @Schema(description = "The ids of the output ports to be copied."
+    )
+    public Set<String> getOutputPorts() {
+        return outputPorts;
+    }
+
+    public void setOutputPorts(Set<String> outputPorts) {
+        this.outputPorts = outputPorts;
+    }
+
+    /**
+     * @return The ids of the process groups to be copied.
+     */
+    @Schema(description = "The ids of the process groups to be copied."
+    )
+    public Set<String> getProcessGroups() {
+        return processGroups;
+    }
+
+    public void setProcessGroups(Set<String> processGroups) {
+        this.processGroups = processGroups;
+    }
+
+    /**
+     * @return The ids of the processors to be copied.
+     */
+    @Schema(description = "The ids of the processors to be copied."
+    )
+    public Set<String> getProcessors() {
+        return processors;
+    }
+
+    public void setProcessors(Set<String> processors) {
+        this.processors = processors;
+    }
+
+    /**
+     * @return the ids of the remote process groups to be copied.
+     */
+    @Schema(description = "The ids of the remote process groups to be copied."
+    )
+    public Set<String> getRemoteProcessGroups() {
+        return remoteProcessGroups;
+    }
+
+    public void setRemoteProcessGroups(Set<String> remoteProcessGroups) {
+        this.remoteProcessGroups = remoteProcessGroups;
+    }
+
+}

nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/entity/CopyResponseEntity.java

@@ -0,0 +1,221 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.web.api.entity;
+
+import io.swagger.v3.oas.annotations.media.Schema;
+import jakarta.xml.bind.annotation.XmlType;
+import org.apache.nifi.flow.ExternalControllerServiceReference;
+import org.apache.nifi.flow.ParameterProviderReference;
+import org.apache.nifi.flow.VersionedConnection;
+import org.apache.nifi.flow.VersionedFunnel;
+import org.apache.nifi.flow.VersionedLabel;
+import org.apache.nifi.flow.VersionedParameterContext;
+import org.apache.nifi.flow.VersionedPort;
+import org.apache.nifi.flow.VersionedProcessGroup;
+import org.apache.nifi.flow.VersionedProcessor;
+import org.apache.nifi.flow.VersionedRemoteProcessGroup;
+
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * A response to copy a portion of the flow.
+ */
+@XmlType(name = "copyResponseEntity")
+public class CopyResponseEntity extends Entity {
+
+    private String id;
+
+    private Map<String, ExternalControllerServiceReference> externalControllerServiceReferences = new HashMap<>();
+    private Map<String, VersionedParameterContext> parameterContexts = new HashMap<>();
+    private Map<String, ParameterProviderReference> parameterProviders = new HashMap<>();
+
+    private Set<VersionedProcessGroup> processGroups = new HashSet<>();
+    private Set<VersionedRemoteProcessGroup> remoteProcessGroups = new HashSet<>();
+    private Set<VersionedProcessor> processors = new HashSet<>();
+    private Set<VersionedPort> inputPorts = new HashSet<>();
+    private Set<VersionedPort> outputPorts = new HashSet<>();
+    private Set<VersionedConnection> connections = new HashSet<>();
+    private Set<VersionedLabel> labels = new HashSet<>();
+    private Set<VersionedFunnel> funnels = new HashSet<>();
+
+    /**
+     * The id for this copy action.
+     *
+     * @return The id
+     */
+    @Schema(description = "The id for this copy action."
+    )
+    public String getId() {
+        return id;
+    }
+
+    public void setId(String id) {
+        this.id = id;
+    }
+
+    /**
+     * The external controller service references.
+     *
+     * @return The external controller service reference
+     */
+    @Schema(description = "The external controller service references."
+    )
+    public Map<String, ExternalControllerServiceReference> getExternalControllerServiceReferences() {
+        return externalControllerServiceReferences;
+    }
+
+    public void setExternalControllerServiceReferences(Map<String, ExternalControllerServiceReference> externalControllerServiceReferences) {
+        this.externalControllerServiceReferences = externalControllerServiceReferences;
+    }
+
+    /**
+     * The referenced parameter contexts.
+     *
+     * @return The referenced parameter contexts
+     */
+    @Schema(description = "The referenced parameter contexts."
+    )
+    public Map<String, VersionedParameterContext> getParameterContexts() {
+        return parameterContexts;
+    }
+
+    public void setParameterContexts(Map<String, VersionedParameterContext> parameterContexts) {
+        this.parameterContexts = parameterContexts;
+    }
+
+    /**
+     * The referenced parameter providers.
+     *
+     * @return The referenced parameter providers
+     */
+    @Schema(description = "The referenced parameter providers."
+    )
+    public Map<String, ParameterProviderReference> getParameterProviders() {
+        return parameterProviders;
+    }
+
+    public void setParameterProviders(Map<String, ParameterProviderReference> parameterProviders) {
+        this.parameterProviders = parameterProviders;
+    }
+
+    /**
+     * @return the connections being copied.
+     */
+    @Schema(description = "The connections being copied."
+    )
+    public Set<VersionedConnection> getConnections() {
+        return connections;
+    }
+
+    public void setConnections(Set<VersionedConnection> connections) {
+        this.connections = connections;
+    }
+
+    /**
+     * @return the funnels being copied.
+     */
+    @Schema(description = "The funnels being copied."
+    )
+    public Set<VersionedFunnel> getFunnels() {
+        return funnels;
+    }
+
+    public void setFunnels(Set<VersionedFunnel> funnels) {
+        this.funnels = funnels;
+    }
+
+    /**
+     * @return the input port being copied.
+     */
+    @Schema(description = "The input ports being copied."
+    )
+    public Set<VersionedPort> getInputPorts() {
+        return inputPorts;
+    }
+
+    public void setInputPorts(Set<VersionedPort> inputPorts) {
+        this.inputPorts = inputPorts;
+    }
+
+    /**
+     * @return the labels being copied.
+     */
+    @Schema(description = "The labels being copied."
+    )
+    public Set<VersionedLabel> getLabels() {
+        return labels;
+    }
+
+    public void setLabels(Set<VersionedLabel> labels) {
+        this.labels = labels;
+    }
+
+    /**
+     * @return the output ports being copied.
+     */
+    @Schema(description = "The output ports being copied."
+    )
+    public Set<VersionedPort> getOutputPorts() {
+        return outputPorts;
+    }
+
+    public void setOutputPorts(Set<VersionedPort> outputPorts) {
+        this.outputPorts = outputPorts;
+    }
+
+    /**
+     * @return The process groups being copied.
+     */
+    @Schema(description = "The process groups being copied."
+    )
+    public Set<VersionedProcessGroup> getProcessGroups() {
+        return processGroups;
+    }
+
+    public void setProcessGroups(Set<VersionedProcessGroup> processGroups) {
+        this.processGroups = processGroups;
+    }
+
+    /**
+     * @return The processors being copied.
+     */
+    @Schema(description = "The processors being copied."
+    )
+    public Set<VersionedProcessor> getProcessors() {
+        return processors;
+    }
+
+    public void setProcessors(Set<VersionedProcessor> processors) {
+        this.processors = processors;
+    }
+
+    /**
+     * @return the remote process groups being copied.
+     */
+    @Schema(description = "The remote process groups being copied."
+    )
+    public Set<VersionedRemoteProcessGroup> getRemoteProcessGroups() {
+        return remoteProcessGroups;
+    }
+
+    public void setRemoteProcessGroups(Set<VersionedRemoteProcessGroup> remoteProcessGroups) {
+        this.remoteProcessGroups = remoteProcessGroups;
+    }
+}