Add migration tools for function

[tuteng]

Fixed #77

• Add migration tools for function

Mapped field

puslar     function-mesh
tenant <=> tenant
namespace <=> namespace
name <=> name
className <=> className
autoAck <=> autoAck
processingGuarantees <=> ProcessingGuarantee
retainOrdering <=> retainOrdering
retainKeyOrdering <=> RetainKeyOrdering
deadLetterTopic <=> DeadLetterTopic
forwardSourceMessageProperty <=> ForwardSourceMessageProperty
maxMessageRetries <=> MaxMessageRetry
parallelism <=> replicas, MaxReplicas
inputs <=> topics list
topicsPattern <=> TopicPattern
customSerdeInputs <=> CustomSerdeSources
customSchemaInputs <=> CustomSchemaSources
inputSpecs <=> sourceSpecs
   schemaType <=> schemaType
   SerdeClassName <=> SerdeClassName
   isRegexPattern <=> isRegexPattern
   schemaProperties <=> SchemaProperties
   consumerProperties <=> ConsumerProperties
   receiverQueueSize <=> ReceiverQueueSize
   cryptoConfig <=> CryptoConfig
resources <=> resources(limits, requests)
   cpu <=> cpu
   ram <=> memory
cleanupSubscription <=> cleanupSubscription
Topic <=> topic
outputSerdeClassName <=> SinkSerdeClassName
outputSchemaType <=> SinkSchemaType
customSchemaOutputs <=> CustomSchemaSinks
producerConfig <=> ProducerConf
   maxPendingMessages <=> MaxPendingMessages
   maxPendingMessagesAcrossPartitions <=> MaxPendingMessagesAcrossPartitions
   useThreadLocalProducers <=> UseThreadLocalProducers
   cryptoConfig <=> CryptoConfig
subscriptionPosition <=> SubscriptionPosition
subName <=> SubscriptionName
userConfig  <=> funcConfig?
runtimeFlags <=> RuntimeFlags
runtime <=> Runtime
logTopic <=> LogTopic
? <=> SourceType
? <=> SinkType

[Huanli-Meng]

leave my comments

[nlu90]

parallelism <=> replicas, MaxReplicas

parallelism should be mapping to replicas, not MaxReplicas

? <=> SourceType
? <=> SinkType

The SourceType and SinkType are shortcut for built-in connectors based on the doc here. Currently we don't have built-in connectors, so the migration tool might want to reference how pulsar handles such built-in connectors translating

[freeznet]

parallelism <=> replicas, MaxReplicas

parallelism should be mapping to replicas, not MaxReplicas

? <=> SourceType
? <=> SinkType

The SourceType and SinkType are shortcut for built-in connectors based on the doc here. Currently we don't have built-in connectors, so the migration tool might want to reference how pulsar handles such built-in connectors translating

I think there are some misusage of SourceType and SinkType in operator controller.

According to pulsar's doc, when running builtin IO connectors, user can just use --source-type or --sink-type to refer a built-in connector's name, such as pulsar-admin sinks create --sink-type data-generator, which means it create a sink from built-in connector pulsar-io-data-generator.

But for function-mesh, it acts differently. SourceType and SinkType are the ClassName of message's schema.

Like in SinkSpec, we have SourceType and SinkType

function-mesh/api/v1alpha1/sink_types.go

Lines 36 to 37 in d7deb90

	SourceType string `json:"sourceType,omitempty"`
	SinkType string `json:"sinkType,omitempty"`

SourceType is used to generate proto.SourceSpec

function-mesh/controllers/spec/utils.go

Line 242 in d7deb90

TypeClassName: sink.Spec.SourceType,

SinkType is used to generate proto.SinkSpec

function-mesh/controllers/spec/utils.go

Line 266 in d7deb90

TypeClassName: sink.Spec.SinkType,

I have created typeClassName in SourceSpec and SinkSpec #133, to avoid naming inconsistencies with the official Pulsar doc.

[tuteng]

./bin/pulsar-admin functions get --name test-func --tenant public --namespace default
{
  "tenant": "public",
  "namespace": "default",
  "name": "test-func",
  "className": "org.apache.pulsar.functions.api.examples.ExclamationFunction",
  "inputSpecs": {
    "test_src": {
      "isRegexPattern": false,
      "schemaProperties": {},
      "consumerProperties": {}
    }
  },
  "output": "test_result",
  "producerConfig": {
    "useThreadLocalProducers": false,
    "batchBuilder": ""
  },
  "processingGuarantees": "ATLEAST_ONCE",
  "retainOrdering": false,
  "retainKeyOrdering": false,
  "forwardSourceMessageProperty": true,
  "userConfig": {
    "PublishTopic": "test_result"
  },
  "runtime": "JAVA",
  "autoAck": true,
  "parallelism": 1,
  "resources": {
    "cpu": 1.0,
    "ram": 1073741824,
    "disk": 10737418240
  },
  "cleanupSubscription": true
}

I don't think we can get the values of SourceType and SinkType with the above command.

So there are two possible ways to solve this problem

1. after the yaml file is generated, the user manually configures the two values
2. by other method, such as reading the values directly from some configuration file
   Anyway, at the now I think there are still some problems with the migration, such as correctness and reliability that need to be verified
   I suggest we create a separate epic to follow up on this issue
   https://github.com/streamnative/cloud-epics/issues/144