faq 272793601

Reducing computation time for large scenarios by running in a cluster

by Jishnu Narayan on 2018-04-26 13:01:52

---

Hello,

I am working on a scenario of the city of Amsterdam with 168403 agents, 17375 nodes, 31502 links, 2517 pt stops, and 2000 taxis. The modes are car, walk, bike, pt, taxi of which only walk and bike are teleported. Due to the size of the scenario I am running it in a cluster ( cartesius@surfsara.nl) in the Netherlands. What necessary changes should be made in the config file or in the code to achieve the benefits of parallel computing? Currently, I have converted the project in eclipse to an executable jar file and is then running it on the cluster. But I am not gaining on computation time, which is not expected. So clearly there is something that I am not doing right. Any insights will be highly appreciated.

The configuration file is given below:

Thanks,

Jishnu

<?xml version="1.0" ?>
 <!DOCTYPE config SYSTEM "http://www.matsim.org/files/dtd/config_v2.dtd">
 <config>
 	<module name="dvrp">
 		<param name="mode" value="taxi"/>
 	</module>

 	<module name="taxi">
 		<param name="destinationKnown" value="false"/>
 		<param name="vehicleDiversion" value="false"/>
 		<param name="pickupDuration" value="120"/>
 		<param name="dropoffDuration" value="60"/>
 		<param name="onlineVehicleTracker" value="false"/>
        
         <param name="breakIfNotAllRequestsServed" value="false"/>
 		<param name="changeStartLinkToLastLinkInSchedule" value="true"/>
		
 		<param name="taxisFile" value="taxi_vehicles(2000).xml"/>

 		<param name="timeProfiles" value="true"/>
 		<param name="detailedStats" value="true"/>

 		<parameterset type="optimizer">
 			<param name="type" value="RULE_BASED"/>
 			<param name="id" value="example"/>
 			<param name="goal" value="DEMAND_SUPPLY_EQUIL"/>
 			<param name="nearestRequestsLimit" value="99999"/>
 			<param name="nearestVehiclesLimit" value="99999"/>
 			<param name="cellSize" value="1000"/>
 		</parameterset>
 	</module>

 	<module name="global">
 		<param name="randomSeed" value="4711"/>
 		<param name="coordinateSystem" value="Atlantis"/>
 	</module>

 	<module name="network">
 		<param name="inputNetworkFile" value="new.xml"/>
 	</module>

 	<module name="plans">
 		<param name="inputPlansFile" value="AlbatrossAgents.xml"/>
 	</module>

 	<module name="controler">
 		<param name="outputDirectory" value="Output"/>
 		<param name="firstIteration" value="0"/>
 		<param name="lastIteration" value="10"/>
 		<param name="eventsFileFormat" value="xml"/>
 		<param name="mobsim" value="qsim"/>
 	</module>

 	<module name="qsim">
 		<!-- "start/endTime" of MobSim (00:00:00 == take earliest activity time/ run as long as active vehicles exist) -->
 		<param name="startTime" value="00:00:00"/>
 		<param name="endTime" value="30:00:00"/>
 		<param name="simStarttimeInterpretation" value="onlyUseStarttime"/>
 	    <param name="insertingWaitingVehiclesBeforeDrivingVehicles" value="true"/>
 		<param name = "snapshotperiod"	value = "00:00:00"/> <!-- 00:00:00 means NO snapshot writing -->
		
 	</module>

 	<module name="planCalcScore">
	
 	<param name="lateArrival" value="-18"/>
 			<param name="earlyDeparture" value="-0"/>
 			<param name="performing" value="+6"/>
 			<param name="traveling" value="-6"/>
 			<param name="waiting" value="-0"/>
			
 			<parameterset type="activityParams">
 			<param name="activityType" value="work"/>
 			<param name="typicalDuration" value="08:00:00"/>
 			<param name="openingTime" value="04:00:00"/>
 			<param name="latestStartTime" value="18:00:00"/>
 			<param name="closingTime" value="23:00:00"/>
 			<param name="priority" value="1.0"/>
 		</parameterset>

 		<parameterset type="activityParams">
 			<param name="activityType" value="home"/>
 			<param name="priority" value="1.0"/>
 			<param name="typicalDuration" value="12:00:00"/>
 		</parameterset>
		
 		<parameterset type="activityParams">
 			<param name="activityType" value="bringget"/>
 			<param name="priority" value="1.0"/>
 			<param name="typicalDuration" value="08:00:00"/>
 		</parameterset>
		
 		<parameterset type="activityParams">
 			<param name="activityType" value="business"/>
 			<param name="priority" value="1.0"/>
 			<param name="typicalDuration" value="08:00:00"/>
 		</parameterset>
		
 		<parameterset type="activityParams">
 			<param name="activityType" value="shoppingD"/>
 			<param name="priority" value="1.0"/>
 			<param name="typicalDuration" value="04:00:00"/>
 		</parameterset>
		
 		<parameterset type="activityParams">
 			<param name="activityType" value="shoppingND"/>
 			<param name="priority" value="1.0"/>
 			<param name="typicalDuration" value="04:00:00"/>
 		</parameterset>
		
 		<parameterset type="activityParams">
 			<param name="activityType" value="service"/>
 			<param name="priority" value="1.0"/>
 			<param name="typicalDuration" value="04:00:00"/>
			
 		</parameterset>
		
 		<parameterset type="activityParams">
 			<param name="activityType" value="sozializing"/>
 			<param name="priority" value="1.0"/>
 			<param name="typicalDuration" value="08:00:00"/>
 		</parameterset>
		
 		<parameterset type="activityParams">
 			<param name="activityType" value="leisure"/>
 			<param name="priority" value="1.0"/>
 			<param name="typicalDuration" value="08:00:00"/>
 		</parameterset>
		
 		<parameterset type="activityParams">
 			<param name="activityType" value="touring"/>
 			<param name="priority" value="1.0"/>
 			<param name="typicalDuration" value="12:00:00"/>
 		</parameterset>
 	   <parameterset type="modeParams">
 			<param name="constant" value="0.0"/>
 			<param name="marginalUtilityOfDistance_util_m" value="0.0"/>
 			<param name="marginalUtilityOfTraveling_util_hr" value="0.0"/>
 			<param name="mode" value="taxi"/>
 			<param name="monetaryDistanceRate" value="0.0"/>
 		</parameterset>
	
 	</module>

 	<module name="strategy">
	    
 		<param name="maxAgentPlanMemorySize" value="5"/> <!-- 0 means unlimited -->

 		<param name="ModuleProbability_1" value="0.7"/> <!--0.7 -->
 		<param name="Module_1" value="BestScore"/>

 		<param name="ModuleProbability_2" value="0.1"/>   <!--0.1 -->
 		<param name="Module_2" value="ReRoute"/>

 		<param name="ModuleProbability_3" value="0.1"/>  <!--0.1 -->
 		<param name="Module_3" value="TimeAllocationMutator"/>

 		<param name="ModuleProbability_4" value="0.1"/>  <!--0.1 -->
 		<param name="Module_4" value="ChangeTripMode"/>
 	</module>

 	<module name="transit">
 		<param name="useTransit" value="true"/>
 		<param name="transitScheduleFile" value="transitSchedule.xml"/>
 		<param name="vehiclesFile" value="transitVehicle.xml"/>
 		<param name="transitModes" value="pt"/>
 	</module>
	
 	<module name="changeMode">
 		<param name="modes" value="pt,car, taxi"/>
 	</module>

 </config>

---

Comments: 6

---

Re: Reducing computation time for large scenarios by running in a cluster

by Davi Bicudo on 2018-04-26 13:35:16

You can set the number of Threads to be used by the QSIM and for replanning.

Just add <param name="numberOfThreads" value="XX" /> to the global and qsim modules.

I use the following rule to define the (arguably) optimal number of threads independently of the computer:

int numThreads = Runtime.getRuntime().availableProcessors() + 1;

config.global().setNumberOfThreads(numThreads);

config.qsim().setNumberOfThreads(numThreads);

---

Re: Reducing computation time for large scenarios by running in a cluster

by Michal Maciejewski on 2018-04-26 13:48:14

With this setup, the biggest speedup you will achieve by specifying lower limits (e.g. 20 or 40):

<param name="nearestRequestsLimit" value="40"/>
 <param name="nearestVehiclesLimit" value="40"/>

---

Re: Reducing computation time for large scenarios by running in a cluster

by Michal Maciejewski on 2018-04-26 13:54:47

As for parallelisation, the taxi contrib needs to be executed in a single-threaded QSim. If you look into the output config file, there are a couple of params named {{numberOfThreads}}. You can specify them in your input config file (all except the one for QSim).

Nevertheless, the major gain is from using lower limits (previous post).

Cheers,

Michal

---

Re: Reducing computation time for large scenarios by running in a cluster

by Jishnu Narayan on 2018-04-26 14:01:41

Dear Davi,

Thank you for the reply. I tried increasing the numebr of threads of qsim and global, but got the error:

Exception in thread "main" java.lang.RuntimeException: Only a single-threaded QSim allowed

---

Re: Reducing computation time for large scenarios by running in a cluster

by Thibaut Dubernet on 2018-04-26 14:16:39

Dear Jishnu,

the answer of Michal is probably the best one given your particular setup, but to answer your particular question about parallelism, in particular on a cluster, for future users who would find this answer by searching those topics:

MATSim supports parallelism on a single machine, in the following way:

• the config group "global" has a parameter names "numberOfThreads", that controls the number of threads used by the Controler

• additionally, the config group "qsim" has a parameter "numberOfThreads", that controls the number of threads used by the QSim (mobility simulation)

The first parameter is used to dispatch agents between threads at replanning. Agents are replanned independently, so this is an "embarrassingly parallel" problem: n a private machine, it can basically make it as high as the number of cores on your machine. On a cluster, the answer is slightly different (see below).

The second parameter is used by the QSim. Why a second parameter, will you ask? The problem is that the mobility simulation is much more difficult to parallelize: what happens somewhere can have an influence on any part of the network. For this reason, increasing the number of threads too much does not really help, and can even make computation time worse. Some experiments suggest that 5 is the maximum number you should use there. It might be even less with taxi/DVRP.

Now, why not put the first number as high as you can on the cluster? I do not know what system the cluster you mention uses, but LSF, the system used on the Swiss HPC, computed CPU usage as (number of cores requested)*(duration of the job). So to run 32 replanning threads, I will "pay" as if I did use those 32 threads the whole time, even if in practice replanning only takes a small fraction of the time. "Paying" can be actual money, or losing priority in the scheduling of next jobs, depending on your cluster.

MATSim does not support distributed computation, where chunks of work are sent to different machines (I assume this was part of your expectation with a "cluster"), and will probably never do. There were some experiments, but they required a very drastic change in the MATSim process (that, in particular, would be incompatible with taxis). As far as I know, the idea never made it to a nicely packaged extension and might have been abandoned.

TL;DR: if you want to get some speedup with parallelization, on a cluster, setting the two "numberOfThreads" parameters to 5 might be a good idea. Not sure how this interacts with taxi/dvrp.

---

Re: Reducing computation time for large scenarios by running in a cluster

by Jishnu Narayan on 2018-05-03 09:46:19

Dear Michal and Thibaut,

Thank you for the reply. That was indeed helpful.