Memory efficient way to yield signal chunks

[danielhrisca]

@NicolaSabino wrote

Hi there

I'm using this library to process a huge file (5GB) containing video frames containing only one group and 12 indexes. I'm interested only in the 12th index.

The problem is that if I try to get these data with file.get(group=0, index=12, raw=True) because it is trying to load a whole Signal in RAM.

Is there a way to yield that specific Signal (group and index) sample by sample?

[danielhrisca]

Hello @NicolaSabino

you should use iter_get

[NicolaSabino]

Thanks for the quick reply

I've tried this

for samples,_ in mdf.iter_get(group=0, index=12, raw=True, samples_only=True):
    pdb.set_trace()
    pass

But the code crashed in any case because I think it is trying to yield a single tuple with all the samples.

Is there any combination of parameter of this method that let me to yield each single sample, or a method-like iter_signal() ?

[danielhrisca]

The problem stems from the way the SDBLOCKs (signal data blocks) referenced by that channel are loaded. Since they are not part of the channel group raw data blocks it is complicated to have the same chunked access as with the regular channels (the ones that have the samples in the raw channel group data blocks)

[NicolaSabino]

Ok, one last question, debugging I've noticed that the program crash while calling an extract() method from .cutils or similar. In the same file (mdf_v4.py) I've also noticed that you defined also by your own that function. Should I use your extract() method or the one from .cutils? What's the difference?

[danielhrisca]

the one from cutils should work faster but it segfaulted because of a bug

[danielhrisca]

@NicolaSabino can you check if the lastest development code crashes on your files? There was a recent PR which fixed some problems in the cutils code