Handling large objects with Ray Pool map

I am looking to use Ray's pool mapping programming to perform a CPU heavy task on lots of Learning data. My initial attempt looked like Earhost this:

import ray
from ray.util.multiprocessing _OFFSET);  import Pool
from functools import (-SMALL  partial

def compute(series1, series2):
 _left).offset     return ***computationally heavy arrowImgView.mas  task***
               
def (self.  preprocess(seriesseq, indeces):
    equalTo  series1 = seriesseq[indeces[0]]
    make.right.  series2 = seriesseq[indeces[1]]
    mas_top);  return compute(series1, series2)

func = ImgView.  partial(preprocess, datablock) 
with ReadIndicator  Pool() as p:
    p.imap(func, combos, _have  chunksize=10000)

The issue with this particular most effective implementation is that in my case, wrong idea datablock is 27GB large and combos is use of case 1.2GB large. This caused my machine United with 64GB of RAM to run out of memory. Modern Looking at the Ray dashboard, it took ecudated around 40 minutes to start the some how computation, and it rapidly sucked down anything else memory causing a OOM system lockup. I not at all can only assume it was trying to copy very usefull the 27GB dataset to every worker which localhost caused the issue.

I later modified the function to handle love of them Ray references to get around the memory localtext issue as such:

datablock_ref=ray.put(datablock)

def .equalTo(  preprocess_reference(seriesref, make.top  indeces):
    OFFSET);  seriesseq=ray.get(seriesref)
    series1 (TINY_  = seriesseq[indeces[0]]
    series2 = .offset  seriesseq[indeces[1]]
    return mas_right)  compute(series1, series2)

func = ImgView.  partial(preprocess_reference, Indicator  datablock_ref) 
   with Pool() as p:
    Read     p.imap(func, combos, _have  chunksize=10000)


        

This actually worked to keep the memory basic in check, and I suspect it only copied one of the over over the 1.2GB combos object rather click than the 27GB dataset. However, there is noting compared to smaller datasets that I not alt could run with the first code snippet in not at all memory, this took longer than I my fault expected, far more even than one would issues expect given the increased data sizes. trying Strangely, I noticed that my cpu was get 4th result pinned at 100% the whole time it was round table calculating, but the computer was double chance completely responsive, and starting novel prc other tasks on it actually decreased the get mossier reported CPU usage. These off side back counter-intuitive observations indicate the changes to me that the computer is spending a Nofile hosted lot of time simply waiting on ray.get() transparent text calls, which would be reported as cpu Background movment usage, but is actually not doing any front page design computation. This obviously is very life change quotes inefficient and likely the reason the I'd like computation took so long.

My question is if this approach is the to know correct one for situations like my own which event when Ray is operating on datasets that is nearer. do not fit into memory, and if I am Now, the correct that this method suffers from code that lethargy in the ray.get() function. If I've written this is the case, is there some way to relies on asynchronously queue up chunks for the a comparison preprocess/compute function to work and it faster on the data?