[Spark load][Fe 5/6] Fe submit spark etl job #3716
Conversation
imay
added
area/load
| public class Pair<F, S> { | ||
| public static PairComparator<Pair<?, Comparable>> PAIR_VALUE_COMPARATOR = new PairComparator<>(); | ||
|
|
||
| @SerializedName(value = "first") |
There was a problem hiding this comment.
I'am not sure this is ok, cause there is no guarantee that the F and S object can also be serialized by GSON
There was a problem hiding this comment.
Users guarantee this when use Pair class？like Map and List.
There was a problem hiding this comment.
I checked, this is not work
There was a problem hiding this comment.
I add a comment.
When using Pair for persistence, users need to guarantee that F and S can be serialized through Gson
| private static final String CONFIG_FILE_NAME = "jobconfig.json"; | ||
| private static final String APP_RESOURCE_LOCAL_PATH = PaloFe.DORIS_HOME_DIR + "/lib/" + APP_RESOURCE_NAME; | ||
| private static final String JOB_CONFIG_DIR = "configs"; | ||
| private static final String MAIN_CLASS = "org.apache.doris.load.loadv2.etl.SparkEtlJob"; |
There was a problem hiding this comment.
How about get it from SparkEtlJob.class.getXXX()?
There was a problem hiding this comment.
ok, I comment and will replace with it when SparkEtlJob class is merged.
| throw new LoadException(errMsg + "spark app state: " + state.toString()); | ||
| } | ||
| if (retry >= GET_APPID_MAX_RETRY_TIMES) { | ||
| throw new LoadException(errMsg + "wait too much time for getting appid. spark app state: " |
There was a problem hiding this comment.
| throw new LoadException(errMsg + "wait too much time for getting appid. spark app state: " | |
| throw new LoadException(errMsg + " wait too much time for getting appi d. spark app state: " |
There was a problem hiding this comment.
errMsg already have a space at the end.
wyb
changed the title
wyb
changed the title
wyb
changed the title
wyb
force-pushed
the
spark_load_fe_submit_etl_job
branch
2 times, most recently
from
6d0ef4e to
20df6f2
Compare
| public class Pair<F, S> { | ||
| public static PairComparator<Pair<?, Comparable>> PAIR_VALUE_COMPARATOR = new PairComparator<>(); | ||
|
|
||
| @SerializedName(value = "first") |
There was a problem hiding this comment.
I checked, this is not work
| + ", msg=" + tReadResponse.getOpStatus().getMessage()); | ||
| } | ||
| failed = false; | ||
| return tReadResponse.getData(); |
There was a problem hiding this comment.
broker's pread() method does not guarantee to read the specified length of data currently.
But #3881 is trying to solve this problem. Just for remind.
wyb
force-pushed
the
spark_load_fe_submit_etl_job
branch
from
ee27b34 to
efcc796
Compare
3 participants
After user creates a spark load job which status is PENDING, Fe will schedule and submit the spark etl job.
2.1 Create etl job configuration according to Spark load interface #3010 (comment)
2.2 Upload the configuration file and job jar to HDFS with broker
2.3 Submit etl job to spark cluster
2.4 Wait for etl job submission result
#3433