Fix example validation issues before merging `nmdc-schema/main` into `berkeley-schema-fy24/main` #73

turbomam · 2024-01-19T17:34:28Z

merged a local copy of the nmdc-schema into a local duplicate of berkeley-schema-fy24

now comparing against berkeley-schema-fy24's main

will need to review the example data files that have been moved into the problem sub-directory

@eecavanna who else should we invite as reviewers?

Updating slots to required for class FunctionalAnnotationAggMember, adding new class and slots for the metap aggregation collection.

Add is_best_protein slot

Update slots for MetaPFunctionAggMember

Updated to use new is_best_protein slot

Update range for is_best_protein to Boolean from BooleanValue.

Updating to Boolean instead of BooleanValue.

Make slots for KEGG aggregation tables required.

Updating to failed data can have no has_output.

Also, fix underscore-ness of other pseudo-"private" variables.

That way, we don't have to instantiate the class in order to invoke them.

…nation schema version

Removing rule to have has_output be optional if quality_control_report.status=fail

Testing removing invalid data of quality_control_report.status=pass mixed with failure_categorization.

snakecasing permissible values for FailureWhatEnum

Updating failure_what values to snakecase permissible values

Updating failure_where value to snakecase permissible value.

Resolving merge conflicts, updating format of qc information to what is in the nmdc-schema.

aclum · 2024-01-22T21:28:02Z

src/schema/basic_slots.yaml

+      Slot to store additional comments about laboratory or workflow output. For workflow output
+      it may describe the particular workflow stage that failed. (ie Failed at call-stage due to a malformed fastq file).
+
+  instrument_name:


This needs to get removed, re-adding this is why Database-data-generation-instrument_name-invalid.yaml is now passing

@turbomam not sure if you want to use the new deprecation way we talked about or if this should just be deleted.

cc @mslarae13

My vote is just deleted since it's the fork & we just deleted everything else.

if you mean remove it from nmdc-schema production (main), then deprecation protocol we decided on. I think it would be a good spot for Sierra to help (if so)

oops I already deleted it! but we can reconstruct a deprecation if you want @aclum

Deleting is fine with me.

aclum · 2024-01-22T23:01:20Z

@turbomam Database-img_mg_annotation_objects.yaml is failing b/c for most documents everything but type slot is commented out, edits in nmdc-schema for this file are from about a month ago. Recall what you were testing or can things be uncommented or deleted to resolve this?

This reverts commit a69ad73.

aclum · 2024-01-23T00:15:19Z

@turbomam i need help with Extraction-NEON.yaml, if I strip this down to just id, type, has_input, has_output and conver to a extraction-set record I can get data to validate properly but not as an Extraction record itself.
That is, this validates:
cat Database-extraction.test.yaml
extraction_set:

id: nmdc:extrp-11-00r2pk65
type: nmdc:Extraction
has_input:
- nmdc:procsm-11-9gjxns61
  has_output:
- nmdc:procsm-11-0wxpzf07
but this does not
cat Extraction-NEON.yaml
id: nmdc:extrp-11-00r2pk65
type: nmdc:Extraction
has_input:
nmdc:procsm-11-9gjxns61
has_output:
nmdc:procsm-11-0wxpzf07

aclum

Things that need to be addressed:

Database-Extraction-invalid-sample_mass.yaml is failing but for unexpected reasons
slot instrument_name needs to be removed to make Database-data-generation-instrument_name-invalid.yaml
I have fixed a number of the valid example data files in the problem folder, those fixes are in branch https://github.com/microbiomedata/berkeley-schema-fy24/tree/berkeley-schema-replica-feedback/src/data/valid as discussed over slack. I'll leave it to @turbomam to decide how to get these back in
I would like to change the permissible value names for FailureWhereEnum to match the berkeley-schema-fy24 class names, dropping 'Activity'. I'm willing to do this and update the example data but we might want to wait until after we've worked through the other issues.

turbomam · 2024-01-24T23:45:24Z

I fixed all of the tests except src/data/problem/valid/Database-img_mg_annotation_objects.yaml but I haven't read all of the comments here yet.

Database-Extraction-invalid-sample_mass.yaml: there's no sample_mass slot any more. Thre is an input_mass slot that serves the same purpose
some of the unexpected behavior you were seeing in your validations may come from the fact that you we'rent using the --target-class option.
I did remove instrument_name

I still have to work on the permissible value names for FailureWhereEnum

turbomam · 2024-01-25T00:29:36Z

I'm really not motivated to hand-fix Database-img_mg_annotation_objects.yaml. @brynnz22 I think all of the commented sections need t be uncommented, and then it needs the type-injection migration. If it needs a little manually tweaking after that, I'm happy to do it. Can we work on this together?

brynnz22 · 2024-01-25T17:12:10Z

I'm really not motivated to hand-fix Database-img_mg_annotation_objects.yaml. @brynnz22 I think all of the commented sections need t be uncommented, and then it needs the type-injection migration. If it needs a little manually tweaking after that, I'm happy to do it. Can we work on this together?

@turbomam can you point to this file? I'm not sure what you are referring to

turbomam · 2024-01-25T17:27:09Z

@turbomam can you point to this file? I'm not sure what you are referring to

https://github.com/microbiomedata/berkeley-schema-fy24/blob/berkeley-main-replica/src/data/problem/valid/Database-img_mg_annotation_objects.yaml

brynnz22 · 2024-01-25T18:08:13Z

Thanks @turbomam that sounds good to me.

mslarae13 · 2024-01-31T20:16:35Z

@turbomam @brynnz22 @aclum what is left to do here?

turbomam · 2024-02-01T14:59:24Z

Somebody said that Database-img_mg_annotation_objects.yaml was all commented out in nmdc-schema, so we shouldn't worry about making it fully valid in berkeley-schema-fy24. But it isn't commented out at all in nmdc-schema. If it has unique patterns in it that aren't validated anywhere else, then I would like to fix this file eventually, even if we do leave it in src/data/problem.

@aclum Database-biosamples-infiltrations.yaml has become invalid since the last time I tried to validate this branch.

I still think we should merge now. The build and test work.

turbomam · 2024-02-01T15:07:15Z

@aclum @brynnz22 Database-biosamples-infiltrations.yaml is failing the new pattern for relating a Biosample to a Study

poetry run linkml-validate 
  --schema src/schema/nmdc.yaml \
  --target-class Database src/data/problem/valid/Database-biosamples-infiltrations.yaml

[ERROR] [src/data/problem/valid/Database-biosamples-infiltrations.yaml/0] Additional properties are not allowed ('part_of' was unexpected) in /biosample_set/0
[ERROR] [src/data/problem/valid/Database-biosamples-infiltrations.yaml/0] 'associated_studies' is a required property in /biosample_set/0
[ERROR] [src/data/problem/valid/Database-biosamples-infiltrations.yaml/0] Additional properties are not allowed ('part_of' was unexpected) in /biosample_set/1
[ERROR] [src/data/problem/valid/Database-biosamples-infiltrations.yaml/0] 'associated_studies' is a required property in /biosample_set/1
[ERROR] [src/data/problem/valid/Database-biosamples-infiltrations.yaml/0] Additional properties are not allowed ('part_of' was unexpected) in /biosample_set/2
[ERROR] [src/data/problem/valid/Database-biosamples-infiltrations.yaml/0] 'associated_studies' is a required property in /biosample_set/2

brynnz22 · 2024-02-01T17:48:58Z

@turbomam I think we just need to change part_of in the Database-biosamples-infiltrations.yaml to associated_studies and this should validate properly.

turbomam · 2024-02-02T13:17:47Z

@turbomam I think we just need to change part_of in the Database-biosamples-infiltrations.yaml to associated_studies and this should validate properly.

Thanks @brynnz22 . I was hoping that if I acted clueless then somebody else would do it for me :-)

aclum and others added 30 commits November 7, 2023 16:00

Update nmdc.yaml

b962823

Updating slots to required for class FunctionalAnnotationAggMember, adding new class and slots for the metap aggregation collection.

Create MetaPFunctionAggMember-minimal.yaml

df4f336

Update basic_slots.yaml

d214c65

Add is_best_protein slot

Update nmdc.yaml

10b4ced

Update slots for MetaPFunctionAggMember

Update MetaPFunctionAggMember-minimal.yaml

2adf2f7

Updated to use new is_best_protein slot

Update basic_slots.yaml

76aedf3

Update range for is_best_protein to Boolean from BooleanValue.

Update MetaPFunctionAggMember-minimal.yaml

232a19e

Updating to Boolean instead of BooleanValue.

Update nmdc.yaml

29e0b40

Make slots for KEGG aggregation tables required.

Update MetaPFunctionAggMember-minimal.yaml

58d2070

works because everthing is commented out

cce8856

everything workigne except rules and enums

5088245

old enums working

7515b20

new class-equivalent permissible values

708b836

Update Database-ReadQcAnalysisActivity-quality_fail.yaml

fb58cbd

Updating to failed data can have no has_output.

Support transformers that don't modify the dictionary in place

e64eddc

Clarify variable names

0c9bc76

Rename all migrator classes to Migrator

2823578

Add explanatory comments within base class

f9fbde0

Convert self.agenda into a pseudo-"private" variable

7272230

Also, fix underscore-ness of other pseudo-"private" variables.

Convert "to/from" version getters into class methods

26f8f1b

That way, we don't have to instantiate the class in order to invoke them.

Implement function that retrieves migrator class having a given desti…

4d22499

…nation schema version

Clarify comment

9fccb15

Update workflow_execution_activity.yaml

2264ea1

Removing rule to have has_output be optional if quality_control_report.status=fail

Update Database-ReadQcAnalysisActivity-invalid.yaml

7ff8d20

Testing removing invalid data of quality_control_report.status=pass mixed with failure_categorization.

Merge branch 'main' into 1427-try-to-fix-qc-report-pr-1334-in-new-branch

e7130e6

Update nmdc.yaml

236b14b

snakecasing permissible values for FailureWhatEnum

Update Database-ReadQcAnalysisActivity-quality_fail.yaml

06f17b6

Updating failure_what values to snakecase permissible values

Update Database-ReadQcAnalysisActivity-invalid.yaml

13d0baf

Updating failure_where value to snakecase permissible value.

Create infiltrations slot

71189e3

Add infiltrations to Biosample

8be1fb1

eecavanna changed the title ~~Review merge conflict resolution (nmdc-schema/main -> berkeley-schema-fy24/main)~~ Fix example validation issues before merging nmdc-schema/main into berkeley-schema-fy24/main Jan 20, 2024

Update Database-Extraction-invalid-sample_mass.yaml

a69ad73

Resolving merge conflicts, updating format of qc information to what is in the nmdc-schema.

aclum reviewed Jan 22, 2024

View reviewed changes

Revert "Update Database-Extraction-invalid-sample_mass.yaml"

10a558b

This reverts commit a69ad73.

aclum added 3 commits January 22, 2024 16:44

Fixing valid example data files after conflict resolution

c7153d9

Fixing valid example data files after conflict resolution

7d4923c

Fixing problems with valid and invalid data

098bf3d

aclum requested changes Jan 23, 2024

View reviewed changes

all but Database-img_mg_annotation_objects.yaml

3581992

turbomam added 2 commits January 24, 2024 18:56

merge conflict

6a56872

input_mass not smapl_mass

fe10d65

backmerge main

caa9cb8

turbomam requested review from aclum and mslarae13 February 1, 2024 14:59

one more problem example data file

68731c6

Database-img_mg_annotation_objects.yaml sole problem

a36bdec

turbomam merged commit c673555 into main Feb 2, 2024
2 checks passed

turbomam deleted the berkeley-main-replica branch February 2, 2024 13:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix example validation issues before merging `nmdc-schema/main` into `berkeley-schema-fy24/main` #73

Fix example validation issues before merging `nmdc-schema/main` into `berkeley-schema-fy24/main` #73

turbomam commented Jan 19, 2024

aclum Jan 22, 2024

aclum Jan 22, 2024

aclum Jan 22, 2024

mslarae13 Jan 22, 2024

mslarae13 Jan 22, 2024

turbomam Jan 25, 2024

aclum Jan 25, 2024

aclum commented Jan 22, 2024

aclum commented Jan 23, 2024

aclum left a comment •

edited

Loading

turbomam commented Jan 24, 2024 •

edited

Loading

turbomam commented Jan 25, 2024 •

edited

Loading

brynnz22 commented Jan 25, 2024

turbomam commented Jan 25, 2024 •

edited

Loading

brynnz22 commented Jan 25, 2024

mslarae13 commented Jan 31, 2024

turbomam commented Feb 1, 2024 •

edited

Loading

turbomam commented Feb 1, 2024

brynnz22 commented Feb 1, 2024 •

edited

Loading

turbomam commented Feb 2, 2024

Fix example validation issues before merging nmdc-schema/main into berkeley-schema-fy24/main #73

Fix example validation issues before merging nmdc-schema/main into berkeley-schema-fy24/main #73

Conversation

turbomam commented Jan 19, 2024

aclum Jan 22, 2024

Choose a reason for hiding this comment

aclum Jan 22, 2024

Choose a reason for hiding this comment

aclum Jan 22, 2024

Choose a reason for hiding this comment

mslarae13 Jan 22, 2024

Choose a reason for hiding this comment

mslarae13 Jan 22, 2024

Choose a reason for hiding this comment

turbomam Jan 25, 2024

Choose a reason for hiding this comment

aclum Jan 25, 2024

Choose a reason for hiding this comment

aclum commented Jan 22, 2024

aclum commented Jan 23, 2024

aclum left a comment • edited Loading

Choose a reason for hiding this comment

turbomam commented Jan 24, 2024 • edited Loading

turbomam commented Jan 25, 2024 • edited Loading

brynnz22 commented Jan 25, 2024

turbomam commented Jan 25, 2024 • edited Loading

brynnz22 commented Jan 25, 2024

mslarae13 commented Jan 31, 2024

turbomam commented Feb 1, 2024 • edited Loading

turbomam commented Feb 1, 2024

brynnz22 commented Feb 1, 2024 • edited Loading

turbomam commented Feb 2, 2024

Fix example validation issues before merging `nmdc-schema/main` into `berkeley-schema-fy24/main` #73

Fix example validation issues before merging `nmdc-schema/main` into `berkeley-schema-fy24/main` #73

aclum left a comment •

edited

Loading

turbomam commented Jan 24, 2024 •

edited

Loading

turbomam commented Jan 25, 2024 •

edited

Loading

turbomam commented Jan 25, 2024 •

edited

Loading

turbomam commented Feb 1, 2024 •

edited

Loading

brynnz22 commented Feb 1, 2024 •

edited

Loading