What are the computational details behind DEP?
As of July 2016, DEP makes over 200,000 WEPP model runs each day. The WEPP model itself is a single threaded job, but thankfully only takes a second to run for the 10+ years we are simulating. Even at a one-second execution time, having 200k runs means that we have to implement some sort of parallel execution to have all the runs complete within a day (only ~100k seconds in a day!).
DEP's goal is to have daily output from the previous day by approximately 6 AM of the current day. This represents a maximum time window of about six hours to accomplish the following tasks:
Update input climate files with yesterday's data
Run the WEPP model 200k times
Process the WEPP output files and create summaries within the database
As of this writing, the DEP processing is done on a Dell PowerEdge R820 with 4 physical processors (48 threads), 256 GB of memory, and two mirrored 1 TB SSD drives. These assets permit the DEP execution to complete by about 5 AM each morning. Ironically, the most computationally "expensive" component of this processing is the input-output associated with the numerous ASCII files used by the WEPP model. The SSD drives allow these small operations to complete with limited latency.
The parallelization happens with a programming script that launches multiple instances of WEPP at the same time. This Python code uses the standard library multiprocessing to proctor the runs. Again, the SDD drives really make this possible as the multiple jobs are not stuck in I/O wait.