[RFC] Directory Refactor

[gaocegege]

Now the directory layout is a little confusing:

./pkg
├── api
│   ├── health
│   ├── operators
│   │   └── apis
│   │       ├── common
│   │       │   └── v1alpha2
│   │       ├── experiment
│   │           └── v1alpha2
│   │       ├── studyjob
│   │       │   └── v1alpha1
│   │       └── trial
│   │           └── v1alpha2
│   ├── v1alpha1
│   └── v1alpha2
├── controller
│   ├── v1alpha1
│   │   └── studyjob
│   └── v1alpha2
│       ├── experiment
│       └── trial
├── db
│   ├── v1alpha1
│   └── v1alpha2
├── earlystopping
│   └── v1alpha1
├── manager
│   ├── modelstore
│   └── v1alpha1
│       ├── file-metricscollector
│       ├── metricscollector
│       ├── studyjobclient
│       └── visualise
│           └── tensorboard
├── mock
│   ├── modelstore
│   ├── v1alpha1
│   └── v1alpha2
├── suggestion
│   ├── v1alpha1
├── ui
└── util
    └── v1alpha2

I propose some changes:

• Move pkg/mock/modelstore to pkg/mock/v1alpha1/db. Maybe we need to change the generate script to do it automatically.
• Move pkg/manager/modelstore to pkg/manager/v1alpha1, since it is version specific implementation. And it will be changed in v1alpha2.
• Change pkg/controller/v1alpha2/experiment to pkg/controller/experiment/v1alpha2/. Becuase the API layout is pkg/api/operators/apis/experiment/v1alpha2
• Change pkg/controller/v1alpha2/trial to pkg/controller/trial/v1alpha2/
• Change pkg/controller/v1alpha1/studyjob to pkg/controller/studyjob/v1alpha2/
• Rename pkg/api/operators to pkg/apis/controller/. The new name follows the same convention with pkg/controller
• Put pkg/api/v1alpha1 and pkg/api/v1alpha2 in pkg/apis/manager. At the same time, make API proto about suggestions in pkg/apis/suggestion/v1alpha1 and pkg/apis/suggestion/v1alpha2. Becuase it will be called directly in v1alpha2.

The new layout will look like:

./pkg
+── apis
-── api
│   ├── health
-   ├── operators
+   ├── controller
│   │   ├── common
│   │   │   └── v1alpha2
-   │   ├── experiment
+   │   ├── experiments
│   │   │   └── v1alpha2
│   │   ├── studyjob
│   │   │   └── v1alpha1
-   │   └── trial
+   │   └── trials
│   │       └── v1alpha2
│   ├── v1alpha1
│   └── v1alpha2
├── controller
-   ├── v1alpha1
-   │   └── studyjob
-   └── v1alpha2
-       ├── experiment
-       └── trial
+   ├── studyjob
+   │   └── v1alpha1
+   └── experiment
+       ├── v1alpha2
+   └── trial
+       └── v1alpha2
...
├── manager
-   ├── modelstore
│   └── v1alpha1
│       ├── file-metricscollector
│       ├── metricscollector
│       ├── studyjobclient
│       └── visualise
│           └── tensorboard
├── mock
-   ├── modelstore
│   ├── v1alpha1
│   └── v1alpha2
...

[gaocegege]

/cc @richardsliu @YujiOshima @johnugeorge @hougangliu

WDYT

[YujiOshima]

Though currently, health is only used in the manager, it can be used in all services.
It is based on the offical grpc health check service https://github.com/grpc/grpc/blob/master/doc/health-checking.md

[gaocegege]

@YujiOshima Make sense. Then I think we should keep it. I updated the proposal, Thanks for your reminder

[hougangliu]

it can make katib directory style consistent now.
modelstore is not in use now, should we drop it? or make a backup directory in root path to store it momentarily since I am not sure if modeldb storage plugin can reuse part of the code

[gaocegege]

Is modelstore used in v1alpha1?

[johnugeorge]

Model store is currently not used. Most of the code is obsolete and it is written for v1alpha1 api. I don't think, we need to keep the code. @YujiOshima

[gaocegege]

Then maybe we could just delete it. model store interface confuses me for a long time.

[richardsliu]

Do we need to do this for v1alpha2? This can cause a lot of conflicts with ongoing PRs.

[gaocegege]

It is just a suggestion. We can do it after v1alpha2 is released.

[YujiOshima]

@johnugeorge Yes, the Model store was used before v0.4. We can drop it.
@gaocegege Agree, we should do this after making v1alpha2 stable.

[gaocegege]

#502 (comment)

I feel, Kubebuilder recommends plural form of CR. Ideally, it should have been apis/experiments, apis/trials as per Kubebuilder notation. I was planning to take this up post 0.6 release

We should do it, too.

[gaocegege]

/close by #737