feat: file-logger add vars

[kamly]

Description

file-loggerr plugins add vars

fixes #9596

Checklist

• I have explained the need for this PR and the problem it solves
• I have explained the changes or the new features added to this PR
• I have added tests corresponding to this change
• I have updated the documentation to reflect this change
• I have verified that this change is backward compatible (If not, please discuss on the APISIX mailing list first)

[monkeyDluffy6017]

@kamly please fix the code lint error

[monkeyDluffy6017]

test cases and docs are needed

[kamly]

hi, @monkeyDluffy6017 all done

• add test
• add doc
• fix code lint

[kamly]

hi, @monkeyDluffy6017 fix~

[kamly]

@monkeyDluffy6017 , fix again ~

[Gallardot]

Refer to include_req_body and include_req_body_expr to see if it is better to implement an include_req_header and include_req_header_expr feature so that other logging plugins can support this feature without each plugin implementing it separately.

@kamly @monkeyDluffy6017 WDYT?

[kamly]

Refer to include_req_body and include_req_body_expr to see if it is better to implement an include_req_header and include_req_header_expr feature so that other logging plugins can support this feature without each plugin implementing it separately.

@kamly @monkeyDluffy6017 WDYT?

Good idea, but I read in the docs that file-logger plugin doesn't support include_req_body_expr.

In the http-logger.lua

function _M.check_schema(conf, schema_type)
    if schema_type == core.schema.TYPE_METADATA then
        return core.schema.check(metadata_schema, conf)
    end

    local ok, err = core.schema.check(schema, conf)
    if not ok then
        return nil, err
    end
    return log_util.check_log_schema(conf)
end

Compared in the file-logger.lua

function _M.check_schema(conf, schema_type)
    if schema_type == core.schema.TYPE_METADATA then
        return core.schema.check(metadata_schema, conf)
    end
    return core.schema.check(schema, conf)
end

I think of these few steps:

1. The file-logger plugin can support this feature first.
2. This feature add to other loggers plugin.
3. Another pr refactors file-logger plugin to support log_util.check_log_schema.

The main reason is that I need this feature to be used in a production environment, and it is a big challenge for me to refactor file-logger plugin. I want features to be implemented in small steps.

@Gallardot @monkeyDluffy6017 WDYT ?

[Gallardot]

Good idea, but I read in the docs that file-logger plugin doesn't support include_req_body_expr.

In fact, file-logger plugin support include_req_body and include_req_body_expr. You can refer to the kafka plugin for configuration and have a try.

apisix/apisix/plugins/file-logger.lua

Lines 151 to 153 in bb0a0e6

	function _M.log(conf, ctx)
	local entry = log_util.get_log_entry(plugin_name, conf, ctx)
	write_file_data(conf, entry)

apisix/apisix/utils/log-util.lua

Line 228 in bb0a0e6

entry = get_full_log(ngx, conf)

apisix/apisix/utils/log-util.lua

Line 166 in bb0a0e6

if conf.include_req_body then

If it doesn't meet your scenario, you can implement the include_req_header and include_req_header_expr feature in get_full_log.

apisix/apisix/utils/log-util.lua

Line 108 in bb0a0e6

local function get_full_log(ngx, conf)

[kamly]

In fact, file-logger plugin support include_req_body and include_req_body_expr. You can refer to the kafka plugin for configuration and have a try.

I got it ~

But file-logger plugin does not support include_req_body_expr because the schema does not define include_req_body_expr .

apisix/apisix/plugins/file-logger.lua

Lines 27 to 44 in bb0a0e6

	local schema = {
	type = "object",
	properties = {
	path = {
	type = "string"
	},
	log_format = {type = "object"},
	include_resp_body = {type = "boolean", default = false},
	include_resp_body_expr = {
	type = "array",
	minItems = 1,
	items = {
	type = "array"
	}
	}
	},
	required = {"path"}
	}

Maybe the author who wrote it before missed it, I can try to align kafka-logger plugin

@monkeyDluffy6017 WDYT ?

[kamly]

@Gallardot Please take a look ?

[Gallardot]

But file-logger plugin does not support include_req_body_expr because the schema does not define include_req_body_expr .

@kamly
Although it is not defined in the schema, but it can be used directly in file-logger plugin. Therefore, I think it is only necessary to clearly define this feature in file-logger so that users can know more clearly that this feature is supported.

[kamly]

@kamly Although it is not defined in the schema, but it can be used directly in file-logger plugin. Therefore, I think it is only necessary to clearly define this feature in file-logger so that users can know more clearly that this feature is supported.

@Gallardot got it, I have resubmitted following your solution. Please take a look ?

in 900e438

[kamly]

@monkeyDluffy6017 @Gallardot , please take a look

[monkeyDluffy6017]

so that other logging plugins can support this feature without each plugin implementing it separately.

Hi @Gallardot, I don't get your point, how can we do this? each plugin is independent

[Gallardot]

I don't get your point, how can we do this? each plugin is independent

@monkeyDluffy6017
Yes, each plugin is independent on runtime. However, in terms of implementation, some logging plug-ins use the same method log_util.get_log_entry.

[kamly]

@monkeyDluffy6017 What else do I need to update?

[kamly]

@monkeyDluffy6017 all fix again ~