Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[doc][taier-website] update doc #1156

Merged
merged 1 commit into from
Mar 5, 2024
Merged

[doc][taier-website] update doc #1156

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions k8s/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,11 +30,11 @@ k8s-n1 Ready,SchedulingDisabled master 165d v1.19.4

快速部署的组件

| 服务 | 版本 |
| ----------------- |--------|
| mysql（单机版） | 5.7 |
| 服务 | 版本 |
|---------------|--------|
| mysql（单机版） | 5.7 |
| zookeeper（集群） | 3.5.7 |
| taier 前后端 | master |
| Taier 前后端 | master |

## 2. 快速部署

Expand Down
4 changes: 2 additions & 2 deletions taier-ui/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@
<yarnVersion>${yarn.version}</yarnVersion>

<!-- If it is inconvenient to visit the official website, you can use the domestic Taobao mirror-->
<nodeDownloadRoot>https://npm.taobao.org/dist/</nodeDownloadRoot>
<nodeDownloadRoot>https://mirrors.aliyun.com/nodejs-release/</nodeDownloadRoot>
<yarnDownloadRoot>https://mirrors.huaweicloud.com/yarn/</yarnDownloadRoot>
</configuration>
</execution>
Expand All @@ -51,7 +51,7 @@
<goal>yarn</goal>
</goals>
<configuration>
<arguments>config set registry https://registry.npm.taobao.org</arguments>
<arguments>config set registry https://registry.npmmirror.com</arguments>
</configuration>
</execution>

Expand Down
2 changes: 1 addition & 1 deletion website/docs/expand/task.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ OCEANBASE_SQL(8, "OceanBaseSQL", EJobType.SQL.getType(), 4, EComponentType.OCEAN
:::

### 保存任务
在taier页面上配置好对应的属性参数和任务内容之后，点击保存即可
在Taier页面上配置好对应的属性参数和任务内容之后，点击保存即可
默认会对sqlText内容去除注释
:::tip
如果需要对任务保存做自定义的参数拼接或处理
Expand Down
4 changes: 0 additions & 4 deletions website/docs/functions/component/datax.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,10 +19,6 @@ DataX 计算组件用于运行 DataX 脚本任务,DataX不依赖任务前置组
:::tip
DataX任务运行在Taier本地服务器,通过Taier来执行，并获取执行结果信息

```shell
python datax.py task.temp/task.json
```

:::

:::caution
Expand Down
16 changes: 13 additions & 3 deletions website/docs/functions/component/flink-on-standalone.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,14 +9,16 @@ sidebar_label: Flink On Standalone

* flink官网下载 [release 1.12.7](https://flink.apache.org/downloads.html)

:::tip
如果不了解Flink Standalone 可以参考官网文档 [Flink Standalone](https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/deployment/resource-providers/standalone/overview/#standalone-cluster-reference)
:::
**2、chunjun的插件**

* 编译1.12版本的chunjun [master](https://github.com/DTStack/chunjun)插件，将生成的chunjun-dist 拷贝至 $FLINK_HOME/lib

:::tip
chunjun 有些依赖包依赖是provided，需要将缺省的jar放到$FLINK_HOME/lib
例如：在chunjun-core最新版本中将logback的依赖改为了provided, 需要将以下logback包放到$FLINK_HOME/lib logback-core-1.2.11.jar
logback-classic-1.2.11.jar
chunjun 有些依赖包依赖是provided，需要将缺省的jar放到$FLINK_HOME/lib
例如：在chunjun-core最新版本中将logback的依赖改为了provided,需要将以下logback包放到$FLINK_HOME/lib logback-core-1.2.11.jar logback-classic-1.2.11.jar
:::

**3、flink1.12 standalone服务参数**
Expand Down Expand Up @@ -78,6 +80,14 @@ lib/

```

部署完Flink Standalone 打开Flink Web页面
确认部署正常
1. 页面正常打开
2. chunjun的插件包 在`jobmanger`的log中显示正常加载

![Flink Standalone](../../../static/img/readme/flink-standalone.png)



## Standalone 控制台参数

Expand Down
2 changes: 1 addition & 1 deletion website/docs/functions/multi-cluster.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ sidebar_label: 集群配置

## 集群配置
1. 集群配置按照`公共组件`-->`资源调度组件`-->`存储组件`-->`计算组件`的顺序进行配置
2. 不同厂商Hadoop集群环境各不一样，遇到适配问题，可以尝试修改taier源码支持
2. 不同厂商Hadoop集群环境各不一样，遇到适配问题，可以尝试修改Taier源码支持

## 环境依赖
- [x] SFTP服务器
Expand Down
6 changes: 4 additions & 2 deletions website/docs/quickstart/build.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,9 @@ oracle 插件因为商业版本原因，官方没有提供 ojdbc 的驱动，开
```shell
$ yarn install
$ yarn run build
# or
```
或者使用
- ```shell
$ pnpm install
$ pnpm run build
```
Expand All @@ -60,6 +62,6 @@ oracle 插件因为商业版本原因，官方没有提供 ojdbc 的驱动，开
- taier-ui/dist

:::tip
编译前端代码，建议node版本为`v16.14.0`，包管理工具为`pnpm` / `yarn`
编译前端代码，建议node版本为`v16.16.0`，包管理工具为`pnpm` / `yarn`
源码中包含部分scala的代码，通过idea源码启动需要添加scala的sdk 建议scala版本为`2.11.1`
:::
66 changes: 64 additions & 2 deletions website/docs/quickstart/faq.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,55 @@ sidebar_label: 常见问题


## 常见问题
### 任务运行 直接报错
```java
com.dtstack.taier.common.exception.ClientAccessException: Client access exception.
at com.dtstack.taier.common.client.ClientCache.getClient(ClientCache.java:114)
at com.dtstack.taier.common.client.ClientOperator.judgeSlots(ClientOperator.java:155)
at com.dtstack.taier.scheduler.WorkerOperator.judgeSlots(WorkerOperator.java:93)
at com.dtstack.taier.scheduler.WorkerOperator$$FastClassBySpringCGLIB$$b285a6bd.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:769)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:747)
at org.springframework.aop.framework.adapter.AfterReturningAdviceInterceptor.invoke(AfterReturningAdviceInterceptor.java:55)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:747)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:747)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:689)
at com.dtstack.taier.scheduler.WorkerOperator$$EnhancerBySpringCGLIB$$9dd74da7.judgeSlots(<generated>)
at com.dtstack.taier.scheduler.jobdealer.JobSubmitDealer.submitEngineJob(JobSubmitDealer.java:340)
at com.dtstack.taier.scheduler.jobdealer.JobSubmitDealer.submitJob(JobSubmitDealer.java:329)
at com.dtstack.taier.scheduler.jobdealer.JobSubmitDealer.lambda$run$0(JobSubmitDealer.java:220)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
```

:::tip
taier部署目录下没有worker-plugins
:::

### 新建任务下拉框为空 或者 没有数据同步任务类型
:::tip
任务下拉框为空 查看初始化SQL是否正常执行 `dict`表中type = 30 是否正常 不包含重复数据
没有数据同步任务类型 或者没有其他任务类型 确认前置组件是否正常配置 [组件任务依赖关心](./functions/multi-cluster.md)
:::


### 不依赖hadoop 可以实现异构数据源之间的数据同步吗
:::tip
可以本地部署`Flink Standalone` 或 `DataX` 通过数据同步任务或者DataX来实现同步
:::

### 数据同步版本可以用其他版本吗
:::tip
不行 Chunjun对接的版本为1.12_release Taier项目依赖的也为1.12 建议使用Flink版本1.12.7
:::

### 任务提交报错 提示找不到core-jar
```java
Expand Down Expand Up @@ -94,7 +143,7 @@ Spark组件`sparkSqlProxyPath` 配置的路径下上传对应依赖jar包
### 本地部署登录之后提示未登录
![login_error](../../static/img/example/login_error.png)
:::tip
本地通过访问http://localhost:8090/
本地通过访问http://ip:8090/
:::


Expand All @@ -113,6 +162,13 @@ com.dtstack.taier.datasource.api.exception.InitializeException: get classloader
确保taier部署环境存在同级目录datasource-plugins
:::

### 数据同步任务一直等待运行
:::tip
session 模式 确认 flink session是否拉起 可以参考github issue
standalone 默认 确认 standalone是否正常启动 组件参数配置是否正常
等待运行的信息可以在`taier.log`日志中查看
:::

### 数据同步任务运行class not found

![login_error](../../static/img/example/class_not_find.png)
Expand Down Expand Up @@ -153,7 +209,7 @@ chunjun的connector更换过之后，需要重启Taier和Flink Session
### Taier-UI编译失败

:::tip
手动通过npm等方式编译或者删除Taier-ui下node_modules目录再次编译
手动通过npm等方式编译或者删除Taier-ui下node_modules目录再次编译 npm版本建议v16.16
:::

### Taier等插件包编译失败
Expand Down Expand Up @@ -194,4 +250,10 @@ application.properties 配置hadoop.user.name=hdfs 重启Taier 更多参数参
### 任务提交显示资源不足
:::tip
确认集群的cpu、内存或flink slot数量是否足够
:::

### 如果访问页面提示空白
:::tip
![img.png](/img/readme/no_dist.png)
如果访问页面提示空白 原因为前端dist文件不存在 需要编译后在启动 参考本地启动文档
:::
50 changes: 50 additions & 0 deletions website/docs/quickstart/idea.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,11 +26,61 @@ $ build/mvn-build-plugins.sh
|-- datasource-plugins
```

- 构建前端项目文件
前端文件默认指向到data-develop模块下的/resource/static/ 下 需要构建出来才能正常访问页面
```java
public void addResourceHandlers(ResourceHandlerRegistry registry) {
registry.addResourceHandler("/static/**").addResourceLocations("classpath:/static/static/");
registry.addResourceHandler("/images/**").addResourceLocations("classpath:/static/images/");
registry.addResourceHandler("/assets/**").addResourceLocations("classpath:/static/assets/");
registry.addResourceHandler("swagger-ui.html").addResourceLocations(
"classpath:/META-INF/resources/");
registry.addResourceHandler("/webjars/**").addResourceLocations(
"classpath:/META-INF/resources/webjars/");
registry.addResourceHandler("/taier/**").addResourceLocations("classpath:/static/");
registry.addResourceHandler("/**").addResourceLocations("classpath:/static/");
super.addResourceHandlers(registry);
}
```

- 前端文件构建分为2种方式
1. 通过datadevelop.sh脚本构建
datadevelop.sh会构建前端脚本文件并复制到data-develop模块下的/resource/static/的目录下
```shell
$ build/mvn-build-datadevelop.sh
```

static目录结构
```shell
static/
├── assets
├── images
├── static
├── **.js
├── **.js
```
:::tip
如果datadevelop.sh脚本构建失败 请检查是否是node代理仓库无法访问导致
:::
2. 手动通过npm构建
参考 [源码编译](./quickstart/build.md)
编译完成之后 将taier-ui下`dist`目录拷贝至data-develop模块下的`/resource/`下
将`dist`文件目录重命名为`static`即可




- 在conf/application.properties配置好正确的Zookeeper、MySQL信息
- IDEA选择TaierApplication启动项目
![idea-run](/img/readme/idea-run.png)


:::tip
源码中包含部分scala的代码，通过idea源码启动需要添加scala的sdk 建议scala版本为`2.11.1`
Module Setting > Global Libraries > scala sdk
:::

:::tip
![img.png](/img/readme/no_dist.png)
如果访问页面提示空白 原因为前端文件不存在 需要编译后在启动
:::
Binary file added website/static/img/readme/flink-standalone.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added website/static/img/readme/no_dist.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading