Problem in selecting Argo referenced data

[kamwal]

Durring running the OWC analysis using Argo referenced data only, I encountered the below error:

` UPDATE_SALINITY_MAPPING: Working on profile 3
Traceback (most recent call last):
File "L:\users\argo\kamwal\OWC_Python_SSH\owc-software-python\ow_calibration\get_region\get_region_data.py", line 172, in get_region_data
grid_long = wrap_longitude(grid_long)
File "L:\users\argo\kamwal\OWC_Python_SSH\owc-software-python\ow_calibration\get_region\data_functions\wrap_longitude.py", line 27, in wrap_longitude
neg_long = np.argwhere(grid_long < 0)
TypeError: '<' not supported between instances of 'list' and 'int'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "L:/users/argo/kamwal/OWC_Python_SSH/owc-software-python/owc_calibration.py", line 73, in
update_salinity_mapping("/", FLOAT_NAME, USER_CONFIG)
File "L:\users\argo\kamwal\OWC_Python_SSH\owc-software-python\ow_calibration\update_salinity_mapping\update_salinity_mapping.py", line 330, in update_salinity_mapping
best_hist_lat, best_hist_long, best_hist_dates] = get_region_data(wmo_numbers,
File "L:\users\argo\kamwal\OWC_Python_SSH\owc-software-python\ow_calibration\get_region\get_region_data.py", line 198, in get_region_data
raise Exception("NO DATA FOUND")
Exception: NO DATA FOUND

Process finished with exit code 1

`
In running the same analysis for (1) ctd+ argo and (2) CTD only, there was not such error.

FLOAT SOURCE NUMBER: 3901960
REFERENCE DATA: ARGO_for_DMQC_2020V01.mat
WMO boxes: wmo_boxes_argo.mat
OBJECTIVE MAPPING PARAMETERS:

` # ===============================
#
# Objective Mapping Parameters
#

    # max number of historical casts used in objective mapping
    'CONFIG_MAX_CASTS': 300,

    # 1=use PV constraint, 0=don't use PV constraint, in objective mapping
    'MAP_USE_PV': 0,

    # 1=use SAF separation criteria, 0=don't use SAF separation criteria, in objective mapping
    'MAP_USE_SAF': 0,

    # spatial decorrelation scales, in degrees
    'MAPSCALE_LONGITUDE_LARGE': 8,
    'MAPSCALE_LONGITUDE_SMALL': 2,
    'MAPSCALE_LATITUDE_LARGE': 4,
    'MAPSCALE_LATITUDE_SMALL': 1,

    # cross-isobath scales, dimensionless, see BS(2005)
    'MAPSCALE_PHI_LARGE': 0.1,
    'MAPSCALE_PHI_SMALL': 0.02,

    # temporal decorrelation scale, in years
    'MAPSCALE_AGE_LARGE': 20,
    'MAPSCALE_AGE_SMALL': 10,

    # exclude the top xxx dbar of the water column
    'MAP_P_EXCLUDE': 100,

    # only use historical data that are within +/- yyy dbar from float data
    'MAP_P_DELTA': 200`

[edsmall-bodc]

Is there a way for us to check how much argo data is in each WMO box?

[edsmall-bodc]

After a quick discussion with @kamwal we have found that the program isn't actually doing anything wrong. The matlab version also finds no data for the same profile.

However, rather than throwing an error here, we should stop the analysis at this point and move on to the next profile.

So my proposed fix is it remove the try/except clause from get_region_data and instead just check to see if we get any data that fits the analysis criteria. If we don't, we should just move on to the next profile.

[edsmall-bodc]

Duplicate of #32

[edsmall-bodc]

This is not a duplicate - one crash occurs whilst fetching data (this one). The other crashed occurs whilst interpolating.

[edsmall-bodc]

@kamwal Can you attach your float_source to this issue please?

[kamwal]

Yes, sure.
source_3901960.zip

[edsmall-bodc]

So you ran this for just Argo data?

[kamwal]

yes, I was using only Argo reference data here.

ARGO_for_DMQC_2020V01
WMO_BOXES: wmo_boxes_argo.mat

[gmaze]

It's probably still too early, but argopy is able to fetch and return data from the Argo reference dataset !

from argopy import DataFetcher as ArgoDataFetcher
loader = ArgoDataFetcher(src='erddap', ds='ref')
ds1 = loader.region([-85, -45, 10, 20, 0, 1000, ‘2012-01’, ‘2012-02’]).to_xarray()

we could develop an access point that would work with owc, ie using distances instead of a rectangular space/time domain

[matdon17]

Shall we make this another ticket for an enhancement?

[gmaze]

yes, that would be more specific and help us get what is really required by pyowc

[kamwal]

The fix to issue of reading Argo reference data is in branch https://github.com/euroargodev/argodmqc_owc/tree/kamwal/argo_ref

The issue of not reading these data was associated with use of the os.path.sep.join which was not correctly joining the parts of the string of directory where the Argo data were stored.

The branch is also fixing issue of not read WMO boxed by scipy.io loadmat python function that was not able to read .mat data with version of 7.4.

[kamwal]

This issue has been fixed using the fix