Work on BGC from 2024 LOV visit

[gmaze]

Place holder for work kicked off during 2024 LOV visit.
Closes #361
Adressed #363

What's in this PR:

• Implement machinery to be able to post-process data according to user modes for BGC from the erddap
  • This includes a new transform_data_mode, that is doing what was doing the filter_data_mode method before, but this method was very badly named: merge PARAM with PARAM_ADJUSTED according to DATA_MODE values
  • The already existing filter_data_mode is now really doing what is was supposed to do: filter points according to data mode values
  • Refactor the postprocessing method in the data fetcher facade, so that it clearly implement user mode requirements for 'phy' and 'bgc' dataset
• Implement support for another index file, the auxiliary index in the ``etc` GDAC folder
• Implement nutrients and carbonate system predictions for the Mediterranean Sea using the CANYON-MED model. This brings in a new submodule in argopy to easily support argo accessor extensions.

Poke @Sauzede and @catsch

Breaking changes in the API:

• Argo xarray dataset attribute history is replaced by Processing_history
• change in signature for the xarray.argo accessor method filter_data_mode
• refactorised datafetcher prototype method filter_data_mode as transform_data_mode

[gmaze]

Hi folks, @catsch @Sauzede @grgdll @HCBScienceProducts,

Could I have some advice with regard to how can I easily identify all variables related to Radiometry in the Erddap list of variables below (the BGC dataset filling the Erddap is from synthetic profiles).

I would need this in order to filter out radiometry variables not in R data mode for the "Standard user mode", following recommendation in this issue #280 (comment)

List of variables to filter

List of erddap BGC dataset from synthetic profile files:

['wmo_inst_type',
 'data_type',
 'date_update',
 'pi_name',
 'bbp470',
 'bbp470_qc',
 'bbp470_adjusted',
 'bbp470_adjusted_qc',
 'bbp470_adjusted_error',
 'bbp532',
 'bbp532_qc',
 'bbp532_adjusted',
 'bbp532_adjusted_qc',
 'bbp532_adjusted_error',
 'bbp700',
 'bbp700_qc',
 'bbp700_adjusted',
 'bbp700_adjusted_qc',
 'bbp700_adjusted_error',
 'bbp700_2',
 'bbp700_2_qc',
 'bbp700_2_adjusted',
 'bbp700_2_adjusted_qc',
 'bbp700_2_adjusted_error',
 'bisulfide',
 'bisulfide_qc',
 'bisulfide_adjusted',
 'bisulfide_adjusted_qc',
 'bisulfide_adjusted_error',
 'cdom',
 'cdom_qc',
 'cdom_adjusted',
 'cdom_adjusted_qc',
 'cdom_adjusted_error',
 'chla',
 'chla_qc',
 'chla_adjusted',
 'chla_adjusted_qc',
 'chla_adjusted_error',
 'cndc',
 'cndc_qc',
 'cndc_adjusted',
 'cndc_adjusted_qc',
 'cndc_adjusted_error',
 'cp660',
 'cp660_qc',
 'cp660_adjusted',
 'cp660_adjusted_qc',
 'cp660_adjusted_error',
 'downwelling_par',
 'downwelling_par_qc',
 'downwelling_par_adjusted',
 'downwelling_par_adjusted_qc',
 'downwelling_par_adjusted_error',
 'down_irradiance380',
 'down_irradiance380_qc',
 'down_irradiance380_adjusted',
 'down_irradiance380_adjusted_qc',
 'down_irradiance380_adjusted_error',
 'down_irradiance412',
 'down_irradiance412_qc',
 'down_irradiance412_adjusted',
 'down_irradiance412_adjusted_qc',
 'down_irradiance412_adjusted_error',
 'down_irradiance443',
 'down_irradiance443_qc',
 'down_irradiance443_adjusted',
 'down_irradiance443_adjusted_qc',
 'down_irradiance443_adjusted_error',
 'down_irradiance490',
 'down_irradiance490_qc',
 'down_irradiance490_adjusted',
 'down_irradiance490_adjusted_qc',
 'down_irradiance490_adjusted_error',
 'down_irradiance555',
 'down_irradiance555_qc',
 'down_irradiance555_adjusted',
 'down_irradiance555_adjusted_qc',
 'down_irradiance555_adjusted_error',
 'down_irradiance665',
 'down_irradiance665_qc',
 'down_irradiance665_adjusted',
 'down_irradiance665_adjusted_qc',
 'down_irradiance665_adjusted_error',
 'down_irradiance670',
 'down_irradiance670_qc',
 'down_irradiance670_adjusted',
 'down_irradiance670_adjusted_qc',
 'down_irradiance670_adjusted_error',
 'doxy',
 'doxy_qc',
 'doxy_adjusted',
 'doxy_adjusted_qc',
 'doxy_adjusted_error',
 'doxy2',
 'doxy2_qc',
 'doxy2_adjusted',
 'doxy2_adjusted_qc',
 'doxy2_adjusted_error',
 'doxy3',
 'doxy3_qc',
 'doxy3_adjusted',
 'doxy3_adjusted_qc',
 'doxy3_adjusted_error',
 'nitrate',
 'nitrate_qc',
 'nitrate_adjusted',
 'nitrate_adjusted_qc',
 'nitrate_adjusted_error',
 'ph_in_situ_total',
 'ph_in_situ_total_qc',
 'ph_in_situ_total_adjusted',
 'ph_in_situ_total_adjusted_qc',
 'ph_in_situ_total_adjusted_error',
 'pres',
 'pres_qc',
 'pres_adjusted',
 'pres_adjusted_qc',
 'pres_adjusted_error',
 'psal',
 'psal_qc',
 'psal_adjusted',
 'psal_adjusted_qc',
 'psal_adjusted_error',
 'temp',
 'temp_qc',
 'temp_adjusted',
 'temp_adjusted_qc',
 'temp_adjusted_error',
 'turbidity',
 'turbidity_qc',
 'turbidity_adjusted',
 'turbidity_adjusted_qc',
 'turbidity_adjusted_error',
 'up_radiance412',
 'up_radiance412_qc',
 'up_radiance412_adjusted',
 'up_radiance412_adjusted_qc',
 'up_radiance412_adjusted_error',
 'up_radiance443',
 'up_radiance443_qc',
 'up_radiance443_adjusted',
 'up_radiance443_adjusted_qc',
 'up_radiance443_adjusted_error',
 'up_radiance490',
 'up_radiance490_qc',
 'up_radiance490_adjusted',
 'up_radiance490_adjusted_qc',
 'up_radiance490_adjusted_error',
 'up_radiance555',
 'up_radiance555_qc',
 'up_radiance555_adjusted',
 'up_radiance555_adjusted_qc',
 'up_radiance555_adjusted_error',
 'latitude',
 'longitude',
 'position_qc',
 'time',
 'time_qc',
 'direction',
 'platform_number',
 'platform_type',
 'cycle_number',
 'config_mission_number',
 'data_centre',
 'profile_bbp470_qc',
 'profile_bbp532_qc',
 'profile_bbp700_2_qc',
 'profile_bbp700_qc',
 'profile_bisulfide_qc',
 'profile_cdom_qc',
 'profile_chla_qc',
 'profile_cndc_qc',
 'profile_cp660_qc',
 'profile_down_irradiance380_qc',
 'profile_down_irradiance412_qc',
 'profile_down_irradiance443_qc',
 'profile_down_irradiance490_qc',
 'profile_down_irradiance555_qc',
 'profile_down_irradiance665_qc',
 'profile_down_irradiance670_qc',
 'profile_downwelling_par_qc',
 'profile_doxy_qc',
 'profile_doxy2_qc',
 'profile_doxy3_qc',
 'profile_nitrate_qc',
 'profile_ph_in_situ_total_qc',
 'profile_pres_qc',
 'profile_psal_qc',
 'profile_temp_qc',
 'profile_turbidity_qc',
 'profile_up_radiance412_qc',
 'profile_up_radiance443_qc',
 'profile_up_radiance490_qc',
 'profile_up_radiance555_qc']

[HCBScienceProducts]

All variables that contain "up_radiance", "downwelling_par" or "down_irradiance" are radiometry-related. "up_radiance" and "down_irradiance" are followed by 3 numerical digits that give the associated wavelength in nm, if you want to refine your filter.

[gmaze]

awesome ! thanks @HCBScienceProducts !

[gmaze]

one more question @catsch @Sauzede @grgdll @HCBScienceProducts :

what about CDOM ?
it's all in real time data mode, so how can we consider it in "standard" user mode ?

[catsch]

We are presently waiting for new calibration factors for CDOM or a list of QC=4 from Seabird
so presently, it should not be used at all,
more to come at the next ADMT...

[gmaze]

so presently, it should not be used at all,

thanks @catsch !

[gmaze]

do you think we can keep Radiometry in the 'research' user mode ?

[catsch]

Most of the time, Radiometry is good data (in R mode)
and Radiometry exists in D and (1,2,5,8) is Ok
Do I answer the question ?

[HCBScienceProducts]

do you think we can keep Radiometry in the 'research' user mode ?

Yes to radiometry both in 'research' mode with selection like all other variables (only D-mode, QC flags '1', '5', or '8'). On radiometry in 'standard' mode ***@***.***): Should this rather be R-mode or A-mode or D-mode with QC flags '1', '2', '5', or '8' (instead of R-mode only). I know there's been some DMQC on radiometry done, i.e., D-mode exist (and are better data than the R-mode data). On CDOM: No separate treatment on the argopy side should be needed for the 'standard' mode but can be treated like all other variables (no R-mode but only A-mode or D-mode, QC flags '1', '2', '5', or '8'). The CDOM issues are to be resolved on the Argo data management side. Once solved, it may eventually transition from today's "only R-mode and better not be used" towards A-mode or D-mode and can then appear just like any other variable. Same for the 'research mode': No separate treatment of CDOM (or radiometry) needed.