feat: Redesign the architecture of the plugin system

[nfrankel]

Issue description

Current situation

At the moment, you add a plugin in the request flow by referencing with its name and setting the configuration. The plugin is effectively a singleton.

This approach has some downsides:

• You can use a plugin only once in the request flow. Users cannot add the same plugin multiple times with different configurations.
• For maintainers, it means that they need to duplicate code and reference it with different names and priorities. For example, the ext-plugin-pre-req and the ext-plugin-post-req do the same - run arbitrary Lua code.
• Users cannot change the priority of a plugin, it's hard-coded (see feat: As a User, I want to be able to configure the priority of each plugin so I don't need to change the source code #6504)
• Plugin developers need to handle every possible configuration case (see request help: How to current limit both in minutes and day by using plugin limit-count #6197)

Proposal

In Object-Oriented speak, make Plugin a class and introduce PluginInstance "objects" (or something similarly named):

• Plugin defines the code
• PluginInstance references the Plugin and associates it with a priority and a configuration

Decoupling the code from the configuration/priority would bring much more freedom for users.

[liangliang4ward]

yes! Some time I want use a plugin to config diffent properties.
just like #6527

[tokers]

@nfrankel Could you give a configuration snippet about how to configuring plugins when using the new arch?

Also, we need to evaluate the cost for the re-design, such as how to refactor plugins so that plugin codes are reentrant.

[nfrankel]

@tokers Would something like that be possible?

curl -i https://my.host/apisix/admin/plugin_configs/1  -H 'X-API-KEY: abc' -X PUT -d '{
  "plugins": [
    "plugin": {
      "id": "ext-plugin-req",
      "priority": 500,
      "conf": [{
        "name": "ext-plugin-A",
        "value": "{\"enable\":\"feature\"}"}
      ]},
    },
    "plugin": {
      "id": "ext-plugin-req",
      "priority": -500,
      "conf": [{
        "name": "ext-plugin-B",
        "value": "{\"disable\":\"feature\"}"}
      ]},
    }
  ]
}'

[tokers]

Got it.

But since we use the array to orchestrate plugins, I think we can remove the priority field.

[nfrankel]

But since we use the array to orchestrate plugins, I think we can remove the priority field.

I don't think so. AFAIK, plugins can be set on different levels (Route or Service). Hence, you need a way to merge their orders - be explicit about it. Does it make sense or am I wrong?

[tokers]

But since we use the array to orchestrate plugins, I think we can remove the priority field.

I don't think so. AFAIK, plugins can be set on different levels (Route or Service). Hence, you need a way to merge their orders - be explicit about it. Does it make sense or am I wrong?

IMHO that's confusing, the intuitive order is Route > Service, if we let users decide the order, it's error-prone.

[nfrankel]

Imagine the user has defined a Service with plugin A and priority 5. Now, they define a route with plugin B and C.

How would you compute the order based on the array?

[tokers]

Imagine the user has defined a Service with plugin A and priority 5. Now, they define a route with plugin B and C.

How would you compute the order based on the array?

That's tricky. We may think about another way to allow a plugin can be run multiple times but still use the object to store their config. I'm wondering if there are some circumstances that people want to run the same plugin in a single route but not continuously (e.g. run plugin A, plugin B, plugin A).

[tokers]

Imagine the user has defined a Service with plugin A and priority 5. Now, they define a route with plugin B and C.
How would you compute the order based on the array?

That's tricky. We may think about another way to allow a plugin can be run multiple times but still use the object to store their config. I'm wondering if there are some circumstances that people want to run the same plugin in a single route but not continuously (e.g. run plugin A, plugin B, plugin A).

If not, what about the following way:

{
    "uri": "/api/*",
    "host": "foo.com",
   "plugins": {
       "limit-count": [
          {
             "time_window": 60,
             "count": 10,
             "key": "remote_addr"
          },
          {
             "time_window": 1,
             "count": 5,
             "key": "route_id"
          }
       ] 
    }
}

[liangliang4ward]

Imagine the user has defined a Service with plugin A and priority 5. Now, they define a route with plugin B and C.
How would you compute the order based on the array?

That's tricky. We may think about another way to allow a plugin can be run multiple times but still use the object to store their config. I'm wondering if there are some circumstances that people want to run the same plugin in a single route but not continuously (e.g. run plugin A, plugin B, plugin A).

If not, what about the following way:

{
    "uri": "/api/*",
    "host": "foo.com",
   "plugins": {
       "limit-count": [
          {
             "time_window": 60,
             "count": 10,
             "key": "remote_addr"
          },
          {
             "time_window": 1,
             "count": 5,
             "key": "route_id"
          }
       ] 
    }
}

Is it possible to support more flexible configuration methods? For example, hamc-auth supports multiple configurations and uses the or mode

[tzssangglass]

hamc-auth supports multiple configurations and uses the or mode

I think it's not that hmac-auth itself supports or, but that the plugin orchestration system supports or for the same plugin.

[liangliang4ward]

I think it's not that hmac-auth itself supports or, but that the plugin orchestration system supports or for the same plugin.

yes. but may be not the same plugin. perhaps the same type.

[membphis]

hi @nfrankel , I create a new example follow your way. I think it looks good now.

$ curl -i https://my.host/apisix/admin/plugin_configs/1  -H 'X-API-KEY: abc' -X PUT -d '{
    "desc": "blah",
    "plugins": [
        {
            "id": "limit-count",
            "priority": 500,
            "conf": [
                {
                    "time_window": 60,
                    "count": 10,
                    "key": "remote_addr"
                },
                {
                    "time_window": 1,
                    "count": 5,
                    "key": "route_id"
                }
            ]
        },
        {
            "id": "ip-restriction",
            "priority": 200,
            "conf": [
                {
                    "whitelist": [
                        "127.0.0.1",
                        "113.74.26.106/24"
                    ]
                }
            ]
        }
    ]
}'

[nfrankel]

@membphis LGTM 🙂

[tokers]

hi @nfrankel , I create a new example follow your way. I think it looks good now.

$ curl -i https://my.host/apisix/admin/plugin_configs/1  -H 'X-API-KEY: abc' -X PUT -d '{
    "desc": "blah",
    "plugins": [
        {
            "id": "limit-count",
            "priority": 500,
            "conf": [
                {
                    "time_window": 60,
                    "count": 10,
                    "key": "remote_addr"
                },
                {
                    "time_window": 1,
                    "count": 5,
                    "key": "route_id"
                }
            ]
        },
        {
            "id": "ip-restriction",
            "priority": 200,
            "conf": [
                {
                    "whitelist": [
                        "127.0.0.1",
                        "113.74.26.106/24"
                    ]
                }
            ]
        }
    ]
}'

I still think it'll confuse users, an array has an order attribute, but indeed we don't run the plugin as per their order in the array.

[nfrankel]

@tokers That's a legitimate concern. What about:

$ curl -i https://my.host/apisix/admin/plugin_configs/1  -H 'X-API-KEY: abc' -X PUT -d '{
    "desc": "blah",
    "plugins": {
        "limit-count": {
            "priority": 500,
            "conf": [
                {
                    "time_window": 60,
                    "count": 10,
                    "key": "remote_addr"
                },
                {
                    "time_window": 1,
                    "count": 5,
                    "key": "route_id"
                }
            ]
        },
        "ip-restriction": {
            "priority": 200,
            "conf": [
                {
                    "whitelist": [
                        "127.0.0.1",
                        "113.74.26.106/24"
                    ]
                }
            ]
        }
    }
}'

[tokers]

@tokers That's a legitimate concern. What about:

$ curl -i https://my.host/apisix/admin/plugin_configs/1  -H 'X-API-KEY: abc' -X PUT -d '{
    "desc": "blah",
    "plugins": {
        "limit-count": {
            "priority": 500,
            "conf": [
                {
                    "time_window": 60,
                    "count": 10,
                    "key": "remote_addr"
                },
                {
                    "time_window": 1,
                    "count": 5,
                    "key": "route_id"
                }
            ]
        },
        "ip-restriction": {
            "priority": 200,
            "conf": [
                {
                    "whitelist": [
                        "127.0.0.1",
                        "113.74.26.106/24"
                    ]
                }
            ]
        }
    }
}'

Looks better than the original one, but still has the priority field, which means users can define the plugin running order by themselves?

[membphis]

@tokers That's a legitimate concern. What about:

$ curl -i https://my.host/apisix/admin/plugin_configs/1  -H 'X-API-KEY: abc' -X PUT -d '{
    "desc": "blah",
    "plugins": {
        "limit-count": {
            "priority": 500,
            "conf": [
                {
                    "time_window": 60,
                    "count": 10,
                    "key": "remote_addr"
                },
                {
                    "time_window": 1,
                    "count": 5,
                    "key": "route_id"
                }
            ]
        },
        "ip-restriction": {
            "priority": 200,
            "conf": [
                {
                    "whitelist": [
                        "127.0.0.1",
                        "113.74.26.106/24"
                    ]
                }
            ]
        }
    }
}'

It is too similar to the old plugins configuration structure. How to be compatible with the old version?

{
    "desc": "blah",
    "plugins": {
        "limit-count": {
            ...
        },
        "ip-restriction": {
            ...
        }
    }
}

[spacewander]

@tokers That's a legitimate concern. What about:

$ curl -i https://my.host/apisix/admin/plugin_configs/1  -H 'X-API-KEY: abc' -X PUT -d '{
    "desc": "blah",
    "plugins": {
        "limit-count": {
            "priority": 500,
            "conf": [
                {
                    "time_window": 60,
                    "count": 10,
                    "key": "remote_addr"
                },
                {
                    "time_window": 1,
                    "count": 5,
                    "key": "route_id"
                }
            ]
        },
        "ip-restriction": {
            "priority": 200,
            "conf": [
                {
                    "whitelist": [
                        "127.0.0.1",
                        "113.74.26.106/24"
                    ]
                }
            ]
        }
    }
}'

From a developer's aspect, it is hard to distinguish this from the current arch, as "priority" / "conf" can be a valid field.

What about using a new field instead of "plugins" for the route conf?

[tokers]

What about using a new field instead of "plugins" for the route conf?

Do you have any suggestion?

[nfrankel]

Let's take a step back. At the moment, we are not discussing what is the best developer experience but how not to make it too similar to the existing approach. That's another problem, and we can tackle it in another thread using API versioning, for which there are several solutions (e.g., path, header, query parameter).

[tokers]

I still cannot capture the meaning for using a "priority" field.

[spacewander]

Currently, APISIX is route-based proxy (every configuration is bound to the route). This proposal will require changing APISIX to a plugin-based proxy (every configuration is organized as a plugin configuration), which needs to rewrite APISIX totally, so IMHO I can't accept such a proposal.

What about a smaller change, like adding a priority field in the route's plugin? Like:

curl http://127.0.0.1:9080/apisix/admin/routes/1 -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
    "methods": ["GET"],
    "uri": "/index.html",
    "plugins": {
        "limit-conn": {
            "_meta": {"priority":1200}
            "conn": 1,
            "burst": 0,
            "default_conn_delay": 0.1,
            "rejected_code": 503,
            "key_type": "var",
            "key": "http_a"
        },
        "ip-restriction": {
              "_meta": {"priority":200}
              "whitelist": [
                    "127.0.0.1",
                     "113.74.26.106/24"
               ]
        }
    },
    "upstream": {
        "type": "roundrobin",
        "nodes": {
            "127.0.0.1:1980": 1
        }
    }
}'

[membphis]

I think this is acceptable.

[nfrankel]

Why the extra verbosity?

"_meta": {"priority":1200} vs. "priority": 1200

[tokers]

@nfrankel _meta means metadata for a plugin, encapsulating the priority into it is useful for distinguishing it from the plugin configuration, and if we want to introduce more fields, we can also put them into _meta.