Description
BufferedData::add_batch uses compute_suggested_batch_size_for_output to determine when staged record batches should be sorted and flushed.
However, compute_suggested_batch_size_for_output(mem_size, num_rows) returns a suggested number of rows, while the current condition compares that value with staging_mem_used, which is measured in bytes:
let suggested_batch_size =
compute_suggested_batch_size_for_output(
self.staging_mem_used,
self.staging_num_rows,
);
if self.staging_mem_used > suggested_batch_size {
self.flush_staging()?;
}
Description
BufferedData::add_batchusescompute_suggested_batch_size_for_outputto determine when staged record batches should be sorted and flushed.However,
compute_suggested_batch_size_for_output(mem_size, num_rows)returns a suggested number of rows, while the current condition compares that value withstaging_mem_used, which is measured in bytes: