Reduce memory usage in distance normalization

[iannesbitt]

During distance normalization, radar array is expanded all at once, which is untenable for medium (0.1 GB) to large files because the expanded array can exceed the RAM size (speaking from experience). Lines 96 and 99 of arrayops.py (below) are the culprits and should be modified.

readgssi/readgssi/arrayops.py

Lines 96 to 99 in bb3bb2c

	ar = np.repeat(ar, norm_vel['normalized'].astype(int, casting='unsafe').values, axis=1)
	nvm = int(round(norm_vel['normalized'].mean()))
	del norm_vel
	ar = reducex(ar, by=nvm, verbose=verbose)

Solution: chunk-ify distance normalization by breaking the original array into int(norm_vel['normalized'].mean()) + 1 parts and tacking each normalized chunk on to a processed array to return. That way, max memory usage should only be ~3x filesize, instead of norm_vel['normalized'].mean() *filesize (which is far too often something extreme like 75x).

This will require a slicing for loop around the np.repeat() and readgssi.arrayops.reducex() functions to build the new array block by block.

[iannesbitt]

Admittedly this is not what I want to be doing the week I have to turn in the first draft of my manuscript, but here's the fix:

readgssi/readgssi/arrayops.py

Lines 97 to 103 in 50cea3e

	for c in np.array_split(ar, nvm, axis=1):
	# takes (array, [transform values to broadcast], axis)
	c = np.repeat(c, norm_vel['normalized'].astype(int, casting='unsafe').values[on:on+c.shape[1]], axis=1)
	c = reducex(c, by=nvm, chnum=i, number=nvm, verbose=verbose)
	proc = np.concatenate((proc, c), axis=1)
	on += c.shape[1]
	i += 1