Skip to content

Latest commit

 

History

History
35 lines (25 loc) · 3.07 KB

README.md

File metadata and controls

35 lines (25 loc) · 3.07 KB
Raw

Schedule a Kafka producer appplication (taxi-source)

Step 1: Review the taxi-source Kafka producer task marathon configuration found at spatiotemporal-event-source/taxi-source.json. Breaking the marathon app configuration file down:

  • deploys one instance of a 'taxi-source' deployed as a amollenkopf/spatiotemporal-event-source Docker container

  • each container is allocated 1 cpu shares & 5GB of memory (needed to load the large simulation file into memory)
  • each container starts up with a java command with lots of application specific parameters (including the Kafka Mesos DNS entry)
  • the --class gets resolved as part of the amollenkopf/spatiotemporal-event-source Docker image


Step 2: To schedule 'task-source' go to the DC/OS dashboard and navigate to 'Services - Services'. To run a new Service click the '+' button at the top right of the Services screen and click the 'Single Container' option.


Step 3: Toggle the 'JSON EDITOR' button to on and cut & paste the contents of spatiotemporal-event-source/taxi-source.json into the JSON area.


Step 4: Click the 'REVIEW & RUN' button, review the service configuration & click the 'RUN SERVICE' button to schedule 'taxi-source'.


Step 5: On the 'Services' page note that 'taxi-source' is in 'Deploying' status. note: The first time you deploy the service it will download the .csv simulation file from S3 and will likely take a couple of minutes so be patient.


Step 6: Once the 'taxi-source' shows a status of 'Running' click on 'taxi-source' to see more information.


Step 7: 'taxi-source' is a custom Scala source application that reads a CSV file from S3, loads it's contents into memory and then produces taxi vehicle movement events to a Kafka topic. Here we can see the host where the app was scheduled to as well as the status of the app. To see the progress of the app we can dive into the Mesos Dashboard.


Step 8: Open the Mesos dashboard to view the task of 'taxi-source'.


Step 9: Here we can see the application task for 'taxi-source' and can monitor the progress by clicking into the 'Sandbox' and opening it's 'stdout' file.


Step 10: In the Mesos dashboard navigate to open the 'stdout' file of the 'taxi-stream' Spark Streaming driver. Here we can see that the Spark Streaming workers tasks are now recieving the events from Kafka.




Congratulations: You have successfully ...