You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We are working on projects where each realization has more than 200 million geometries. Assuming that values are stored as Float64, we have ~0.5Gb per realization. If we request 10 parallel processes, we can consume up to 5Gb of RAM.
Given these RAM constraints, it would be useful to introduce an OnlineEnsemble or some other mechanism to consume realizations in an online fashion. The pseudo-code for this could look something like the following:
# request N realizations with P processes
reals =rand(proc, domain, N, pool =1:P, async =true)
while!isempty(reals)
real =pop!(reals)
# do work# save to diskend
In this case the rand call would return immediately, and the N realizations would be consumed on batches by P parallel processes. At any single point in time, the maximum RAM required will be a function of P as opposed to N.
The text was updated successfully, but these errors were encountered:
We are working on projects where each realization has more than 200 million geometries. Assuming that values are stored as
Float64
, we have ~0.5Gb per realization. If we request 10 parallel processes, we can consume up to 5Gb of RAM.Given these RAM constraints, it would be useful to introduce an
OnlineEnsemble
or some other mechanism to consume realizations in an online fashion. The pseudo-code for this could look something like the following:In this case the
rand
call would return immediately, and theN
realizations would be consumed on batches byP
parallel processes. At any single point in time, the maximum RAM required will be a function ofP
as opposed toN
.The text was updated successfully, but these errors were encountered: