feat: added dataset recall testing API

[gubinjie]

Checklist:

Important

Please review the checklist below before submitting your pull request.

• Please open an issue before creating a PR or link to an existing issue
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I ran dev/reformat(backend) and cd web && npx lint-staged(frontend) to appease the lint gods

Description

A knowledge base recall test interface for external calls is added

Fixes #8959
Close #8477

Type of Change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update, included: Dify Document
• Improvement, including but not limited to code refactoring, performance optimization, and UI/UX improvement
• Dependency upgrade

Testing Instructions

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

• Test A
• Test B

[hwzhuhao]

1.Many contents in controllers/service_api/dataset/hit_testing.py and controllers/console/datasets/hit_testing.py are duplicated. Can a common method be abstracted?
2.It would be even better if there were a test screenshot included

[gubinjie]

@hwzhuhao
sorry, I reviewed these two pieces of code and I think it's already an atomic operation that doesn't need to extract generic methods.
The following is a screenshot of the test call interface.

[hwzhuhao]

@gubinjie Maybe you can add a base class, for example:

class HitTestingBase:
    def get_and_validate_dataset(self, dataset_id_str):
        dataset = DatasetService.get_dataset(dataset_id_str)
        if dataset is None:
            raise NotFound("Dataset not found.")
        try:
            DatasetService.check_dataset_permission(dataset, current_user)
        except services.errors.account.NoPermissionError as e:
            raise Forbidden(str(e))
        return dataset

    def parse_args(self):
        parser = reqparse.RequestParser()
        parser.add_argument("query", type=str, location="json")
        parser.add_argument("retrieval_model", type=dict, required=False, location="json")
        parser.add_argument("external_retrieval_model", type=dict, required=False, location="json")
        return parser.parse_args()
    
    def hit_testing_args_check(self, args):
        HitTestingService.hit_testing_args_check(args)

    def perform_hit_testing(self, dataset, args):
        try:
            response = HitTestingService.retrieve(
                dataset=dataset,
                query=args["query"],
                account=current_user,
                retrieval_model=args["retrieval_model"],
                external_retrieval_model=args["external_retrieval_model"],
                limit=10,
            )
            return {"query": response["query"], "records": marshal(response["records"], hit_testing_record_fields)}
        except services.errors.index.IndexNotInitializedError:
            raise DatasetNotInitializedError()
        except ProviderTokenNotInitError as ex:
            raise ProviderNotInitializeError(ex.description)
        except QuotaExceededError:
            raise ProviderQuotaExceededError()
        except ModelCurrentlyNotSupportError:
            raise ProviderModelCurrentlyNotSupportError()
        except LLMBadRequestError:
            raise ProviderNotInitializeError(
                "No Embedding Model or Reranking Model available. Please configure a valid provider "
                "in the Settings -> Model Provider."
            )
        except InvokeError as e:
            raise CompletionRequestError(e.description)
        except ValueError as e:
            raise ValueError(str(e))
        except Exception as e:
            logging.exception("Hit testing failed.")
            raise InternalServerError(str(e))

class HitTestingApi(Resource, HitTestingBase):
    @setup_required
    @login_required
    @account_initialization_required
    def post(self, dataset_id):
        self.dataset_id_str = str(dataset_id)
        dataset = self.get_and_validate_dataset(self.dataset_id_str)
        args = self.parse_args()
        self.hit_testing_args_check(args)
        return self.perform_hit_testing(dataset, args)

[gubinjie]

Okay, big guy, wait a minute, modify it immediately~~0.0

[gubinjie]

@hwzhuhao I'm okay, please take a look

[hwzhuhao]

@gubinjie The file name might be better as hit_testing_base.py. Additionally, please review the comments and address them. Thank you.

[gubinjie]

@hwzhuhao Hmm, I found out that the path in the code was wrong before, and now it's been modified

[gubinjie]

@hwzhuhao These have been corrected

[hwzhuhao]

lgtm, thanks. @crazywoola