Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

StreamPark FAQ #507

Open
xinzhuxiansheng opened this issue Dec 9, 2021 · 19 comments
Open

StreamPark FAQ #507

xinzhuxiansheng opened this issue Dec 9, 2021 · 19 comments
Labels
question Further information is requested

Comments

@xinzhuxiansheng
Copy link
Contributor

xinzhuxiansheng commented Dec 9, 2021

StreamPark ── A magical framework make flink&spark easier!

FAQ

Here is a compilation of frequently mentioned popular issues based on user feedback. If you have a new question, please submit issue . Please do not ask your question here. This is not a question area.


这里记录总结了用户反馈较多的热门问题, 如果你有新的问题,请提issue ,不要在这里提问. 不要在这里提问. 不要在这里提问. 这里不是提问区.

@xinzhuxiansheng
Copy link
Contributor Author

xinzhuxiansheng commented Jan 12, 2022

1. maven install error,Failed to run task: 'npm install' failed?

69691639122373_ pic_hd

because the front end uses nodejs, make sure that nodejs is installed on the compiling machine when compiling, and make sure that the nodejs version is not too old. You can enter streamx-console-webapp and manually execute the cmd to try to compile: npm install, if still If it fails, you can check the information related to nodejs compilation by yourself and try to solve this problem by yourself

@xinzhuxiansheng

This comment was marked as resolved.

@xinzhuxiansheng

This comment was marked as resolved.

@xinzhuxiansheng

This comment was marked as resolved.

@xinzhuxiansheng

This comment was marked as resolved.

@xinzhuxiansheng
Copy link
Contributor Author

xinzhuxiansheng commented Jan 12, 2022

  1. Caused: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient

图片

配置文件缺失metastore.uri,添加上即可, 请参考#219

@xinzhuxiansheng
Copy link
Contributor Author

xinzhuxiansheng commented Jan 12, 2022

  1. java.lang.RuntimeException: java.io.IOException: com.sun.jna.LastErrorException: [2] No such file or directory

java.lang.RuntimeException: java.io.IOException: com.sun.jna.LastErrorException: [2] No such file or directory
	at com.github.dockerjava.httpclient5.ApacheDockerHttpClientImpl.execute(ApacheDockerHttpClientImpl.java:187)
	at com.github.dockerjava.httpclient5.ApacheDockerHttpClient.execute(ApacheDockerHttpClient.java:9)
	at com.github.dockerjava.core.DefaultInvocationBuilder.execute(DefaultInvocationBuilder.java:228)
	at com.github.dockerjava.core.DefaultInvocationBuilder.lambda$executeAndStream$1(DefaultInvocationBuilder.java:269)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: com.sun.jna.LastErrorException: [2] No such file or directory
	at com.github.dockerjava.transport.DomainSocket.<init>(DomainSocket.java:63)
	at com.github.dockerjava.transport.BsdDomainSocket.<init>(BsdDomainSocket.java:43)
	at com.github.dockerjava.transport.DomainSocket.get(DomainSocket.java:138)
	at com.github.dockerjava.transport.UnixSocket.get(UnixSocket.java:27)
	at com.github.dockerjava.httpclient5.ApacheDockerHttpClientImpl$2.createSocket(ApacheDockerHttpClientImpl.java:145)
	at org.apache.hc.client5.http.impl.io.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:125)
	at org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:407)
	at org.apache.hc.client5.http.impl.classic.InternalExecRuntime.connectEndpoint(InternalExecRuntime.java:168)
	at org.apache.hc.client5.http.impl.classic.InternalExecRuntime.connectEndpoint(InternalExecRuntime.java:178)
	at org.apache.hc.client5.http.impl.classic.ConnectExec.execute(ConnectExec.java:136)

Check if Docker is started


@wolfboys
Copy link
Member

  1. log4j conflict #571

image

methods of resolution

Dependency hierarchy, see where the log conflicts are

fix udf log4j conflict

Requirement or improvement(诉求 & 改进建议)

add java parameter
Start the job of Flink job add: -Dlog4j.ignoreTC=true

@wolfboys
Copy link
Member

  1. Could not find a suitable table factory for 'org.apache.flink.table.factories.TableSourceFactory' in the classpath

2021-12-24T04:43:28.901627819Z Caused by: org.apache.flink.table.api.NoMatchingTableFactoryException: Could not find a suitable table factory for 'org.apache.flink.table.factories.TableSourceFactory' in
2021-12-24T04:43:28.901630469Z the classpath.
2021-12-24T04:43:28.901632969Z
2021-12-24T04:43:28.901635342Z Reason: Required context properties mismatch.
2021-12-24T04:43:28.901637804Z
2021-12-24T04:43:28.901640154Z The matching candidates:
2021-12-24T04:43:28.901642578Z org.apache.flink.table.sources.CsvAppendTableSourceFactory
2021-12-24T04:43:28.901645059Z Mismatched properties:
2021-12-24T04:43:28.901647812Z 'connector.type' expects 'filesystem', but is 'kafka'
2021-12-24T04:43:28.901650202Z 'format.type' expects 'csv', but is 'json'

Flink version: 1.14.0

解决方案:flink-kafka-connector 的使用参数不对,请参考flink官网:

CREATE TABLE user_log (
user_id VARCHAR,
item_id VARCHAR,
category_id VARCHAR,
behavior VARCHAR,
ts TIMESTAMP(3)
) WITH (
'connector' = 'kafka',
'topic' = 'user_behavior',
'properties.bootstrap.servers' = 'kafka-1:9092,kafka-2:9092,kafka-3:9092',
'properties.group.id' = 'testGroup',
'scan.startup.mode' = 'earliest-offset',
'format' = 'json'
);

CREATE TABLE pvuv_sink (
dt VARCHAR primary key,
pv BIGINT,
uv BIGINT
) WITH (
'connector' = 'jdbc', -- 使用 jdbc connector
'url' = 'jdbc:mysql://test-mysql:3306/test', -- jdbc url
'table-name' = 'pvuv_sink', -- 表名
'username' = 'root', -- 用户名
'password' = '123456' -- 密码
);

INSERT INTO pvuv_sink
SELECT
DATE_FORMAT(ts, 'yyyy-MM-dd HH:00') dt,
COUNT(*) AS pv,
COUNT(DISTINCT user_id) AS uv
FROM user_log
GROUP BY DATE_FORMAT(ts, 'yyyy-MM-dd HH:00');

另外kafka的消息格式

{"user_id": "543462", "item_id":"1715", "category_id": "1464116", "behavior": "pv", "ts":"2021-02-01T01:00:00Z"}
{"user_id": "662867", "item_id":"2244074","category_id":"1575622","behavior": "pv", "ts":"2021-02-01T01:00:00Z"}
{"user_id": "662867", "item_id":"2244074","category_id":"1575622","behavior": "pv", "ts":"2021-02-01T01:00:00Z"}
{"user_id": "662867", "item_id":"2244074","category_id":"1575622","behavior": "learning flink", "ts":"2021-02-01T01:00:00Z"}

要修改为

{"user_id": "543462", "item_id":"1715", "category_id": "1464116", "behavior": "pv", "ts":"2021-02-01 01:00:00"}
{"user_id": "662867", "item_id":"2244074","category_id":"1575622","behavior": "pv", "ts":"2021-02-01 01:00:00"}
{"user_id": "662867", "item_id":"2244074","category_id":"1575622","behavior": "pv", "ts":"2021-02-01 01:00:00"}
{"user_id": "662867", "item_id":"2244074","category_id":"1575622","behavior": "learning flink", "ts":"2021-02-01 01:00:00"}

否则日志解析失败。


@wolfboys
Copy link
Member

wolfboys commented Jan 12, 2022

  1. window idea environment Could not submit Flink job to remote yarn cluster

动态修改提交job的入口参数, 入口点 YarnClientImpl类 , 方法 submitApplication, 提交点 this.rmClient.submitApplication(request); 对入参 request的 CLASSPATH and _FLINK_CLASSPATH 参数值的分隔符 windows is ";" 替换为 linux is ":"

这个问题应该和官网上报的是一个问题
https://issues.apache.org/jira/browse/FLINK-17858

@bulolo
Copy link
Contributor

bulolo commented Dec 14, 2022

请问 2.0.0版本 docker 部署 .env是引用已有mysql,那么数据库 streampark数据库是要手工导入sql吗?还是在docker首次启动的时候会自己创建?
image
image

@0akarma
Copy link

0akarma commented Dec 28, 2022

请问 2.0.0版本 docker 部署 .env是引用已有mysql,那么数据库 streampark数据库是要手工导入sql吗?还是在docker首次启动的时候会自己创建? image image

得自己进入mysql容器,执行sql,默认不会初始化

@apache apache deleted a comment from midoll3 Dec 29, 2022
@apache apache deleted a comment from douxiaofeng99 Jan 9, 2023
@apache apache deleted a comment from wanghuan2054 Jan 9, 2023
@apache apache deleted a comment from Yougetadad Jan 9, 2023
@apache apache deleted a comment from Su1024 Jan 9, 2023
@apache apache deleted a comment from moranyuwen Jan 10, 2023
@wolfboys
Copy link
Member

  1. 请问streampark 集成 flinkcdc不,根据日志实时同步功能支持不?

支持, 不论是datastream写的flinkcdc同步的作业还是flinksql 作业都支持, 只要是一个标准的flink作业都支持, 如果是flink sql作业的话, connector 必须是按照flink的规范实现的标准的 flink sql connector, 引入对应的依赖jar或者pom即可.

@2000liux
Copy link

compile using this command : mvn clean install -DskipTests -Dcheckstyle.skip -Dmaven.javadoc.skip=true

@changeme2012
Copy link

changeme2012 commented Jun 5, 2023

提交flink sql的任务运行失败,找不到失败原因
image

Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.streampark.flink.client.FlinkClient$.$anonfun$proxy$1(FlinkClient.scala:80)
at org.apache.streampark.flink.proxy.FlinkShimsProxy$.$anonfun$proxy$1(FlinkShimsProxy.scala:60)
at org.apache.streampark.common.util.ClassLoaderUtils$.runAsClassLoader(ClassLoaderUtils.scala:38)
at org.apache.streampark.flink.proxy.FlinkShimsProxy$.proxy(FlinkShimsProxy.scala:60)
at org.apache.streampark.flink.client.FlinkClient$.proxy(FlinkClient.scala:75)
at org.apache.streampark.flink.client.FlinkClient$.submit(FlinkClient.scala:49)
at org.apache.streampark.flink.client.FlinkClient.submit(FlinkClient.scala)
at org.apache.streampark.console.core.service.impl.ApplicationServiceImpl.lambda$start$10(ApplicationServiceImpl.java:1544)
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)
... 3 more
Caused by: java.lang.NoSuchFieldError: CANCEL_ENABLE
at org.apache.streampark.flink.client.trait.FlinkClientTrait.submit(FlinkClientTrait.scala:102)
at org.apache.streampark.flink.client.trait.FlinkClientTrait.submit$(FlinkClientTrait.scala:63)
at org.apache.streampark.flink.client.impl.YarnApplicationClient$.submit(YarnApplicationClient.scala:44)
at org.apache.streampark.flink.client.FlinkClientHandler$.submit(FlinkClientHandler.scala:40)
at org.apache.streampark.flink.client.FlinkClientHandler.submit(FlinkClientHandler.scala)

Upgrading the version to 2.1.1

@3yekn1
Copy link

3yekn1 commented Sep 4, 2023

image

  1. 可以通过修改依赖 为 provided解决。
  2. 是否还有其他手段解决,例如fat jar?
  3. 为什么修改child-first 与 parent-first选项未生效?

@caicancai
Copy link
Member

caicancai commented Sep 4, 2023

An error is reported when the source code is compiled

image

At present, you can comment these two files to compile, then uncomment and compile again.You can try it.

@liyichencc
Copy link

liyichencc commented Sep 6, 2023

  1. after streamx-console started, app.home is not set, and throw NullPointerException

image

streamx-console initialization check failed. If started local for development and debugging, please ensure the -Dapp.home parameter is clearly specified in vm options, more detail: http://www.streamxhub.com/docs/user-guide/development/#vm-options

http://www.streamxhub.com/docs/user-guide/development/#vm-options Link has expired,Now you can refer to this address:https://streampark.apache.org/zh-CN/docs/user-guide/deployment/

@apache apache deleted a comment from bluesky2535 Sep 13, 2023
@apache apache deleted a comment from cyrusmaster Sep 13, 2023
@apache apache deleted a comment from westPointz Sep 13, 2023
@apache apache deleted a comment from moranyuwen Mar 21, 2024
@wangyg007
Copy link

局部截取_20240914_151613
dev分支编译本地部署,前端spark applications点击add new报404

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
question Further information is requested
Projects
None yet
Development

No branches or pull requests

10 participants