[RFC FS-1060] Nullness checking

[dsyme]

Continuation of #5790 and #6804

This is a prototype implementation of RFC FS-1060 nullable reference types

See tests\adhoc\nullness for testing and samples including baselines of outputs from

• existing compiler
• updated compiler
• updated compiler with /langversion:preview
• updated compiler with /langversion:preview /checknulls

TODO:

• revise RFC and resolve all unresolved issues (@dsyme)
• implement syntax string | null and : not null (@T-Gro )
• implement import of .NET metadata (@T-Gro )
• implement emit of .NET metadata (@T-Gro )
• Import+Export TODOs (to test and adjust if needed)
• Nullness used in inheritance (e.g. inherit System.Collections.Generic.List<string | null>)
• Soundness of import of CLI EventHandlers (possibly overly defensive)
• Nullness and import of TypeProvider-generated types + In general the overall TP scenario whet it comes to nullness support - do in separate PR
• Object overrides can change nullness (e.g. boolean.ToString() does or StringBuilder.ToString()), make it work to avoid false alarms
• Type.GetType(..) |> Option.ofObj produces a false alarm, fix that
• printfn "%s" someNullableString produces a false alarm, check the printing pattern annotations
• (+) operator for strings gives false alarms on nullable strings, should allow nullables since String.Concat also does
• Pattern matching flow analysis on match x with | null -> Support this as a scenario
• Allow incoming arrays to be null (this causes false alarms as of now, it's a bug)
• resolve all // TODO NULLNESS (@dsyme)
• use it in the codebase and get it green
• there's a case in tests\adhoc\nullness we're getting The types 'System.String (...)' and 'System.String (...)' do not have compatible nullability. which is wrong - either the nullability aren't shown for some reason or the types should be considered compatible

Testing:

• add proper testing, moving tests across from tests\adhoc\nullness
• test and check signature compatibility. While integrating master I noticed a case in the compiler where a signature file had a non-nullable string type for ther return of a function and an implementation had a nullable string type (and a later soundness problem arose)

To be moved to RFC and resolved, then tested here:

• should nullness be supported on ref tuples and anon tuples
• work out what to do about compat of Option.ofObj and others
• the "type equiv" relation is currently directional only giving a nullness warning if "actual" has a null and "expected" is non-null. This includes a dubious change of direction in SolveFunType thing - for the contravariance of inputs. This has caused a few outputs to change in tests. This is both unsound in the general case (nested cases of equivalence should demand true equivalence) and is dubious for functions because the "MatchingOnly" case of type equivalence is also directional.
• assess nullness checking in conditionals, e..g if x then null else "" and if x then "" else null

[psfinaki]

A non-binding question: do you see any opportunities here for further feature optimization, performance-wise?

If you do, it might be interesting to submit some benchmark along the way. This was often done when doing big features (but obviously it doesn't have to apply everywhere).

[vzarytovskii]

A non-binding question: do you see any opportunities here for further feature optimization, performance-wise?

If you do, it might be interesting to submit some benchmark along the way. This was often done when doing big features (but obviously it doesn't have to apply everywhere).

Not at this point for sure. We should merge it as soon as we can so we can contribute fixes before November

[psfinaki]

So I was mostly following the RFC to find any inconsistencies with expected behavior and things look generally well.

A few points (so far):

1. Not sure we want "nullness warning" to prefix nullness warnings. The prefix only takes space everywhere. Also, it's currently inconsistent, some topic related warnings don't have this anyway.

2. Getting 2 warnings for this for some reason:

let getLength (x: string | null) = x.Length

Also, the text is not the best IMO. Consider inspiring by C#, like "Dereference of a possibly null reference".

3. This code:

let len (str: string | null) =
    match str with
    | null -> -1
    | NonNull s -> s.Length  // binds a non-null result

...produces this:

One - a UX improvement would be to be exact: You can remove this *|NonNull|* pattern usage

Two - since this basically says "hey this is redundant", I would expect to not have the second warning, since if I remove NonNull, I stop getting it.

---

Stay tuned!

[psfinaki]

Does isNull work as expected?

This code produces a warning:

let test (x : int32) = isNull x

Why? I'd imagine this to be still a plain error, like it used to:

This code also produces a warning:

let test (x: string) = isNull x

It used to not produce it. I'd imagine this behavior to be kept.

[T-Gro]

Does isNull work as expected?

isNull or more in general the T:null constraint did have to change the behaviour.
It is important that code not opting in checknullness still receives an error (i.e. new compiler + langversion preview + checknulls disabled).

fsharp/src/Compiler/Checking/ConstraintSolver.fs

Line 2616 in 972fa34

if not g.checkNullness && not (TypeNullIsExtraValue g m ty) then

Code opting in into nullness indeed gets a different behavior, since the semantics for "does type allow nulls" have changed, and diagnostics take a different form.

The last example, isNull nonNullableString is correctly a warning when using --checknulls+. A value without a | null in such project is automatically considered to be non-nullable.

[T-Gro]

A few points (so far):

Good for spotting this, thumbs up. I created new issues to cover for them.
Especially the diagnostics is better to be covered with a separate issue so that all options for various texting can be designed and presented there.

#17410
#17409

Created to cover these, can be again taken independently

[T-Gro]

A non-binding question: do you see any opportunities here for further feature optimization, performance-wise?

If you do, it might be interesting to submit some benchmark along the way. This was often done when doing big features (but obviously it doesn't have to apply everywhere).

It is a compile-time feature without runtime impact.
One could in theory have a look at the added library features, but they are all pretty much basic checks against null and a branching instruction.

For the compilation times, the e2e build tests under different configurations remained the same as far as I can see (comparing this branch, which does not apply checknulls, and its sibling PR which does).

[psfinaki]

It is a compile-time feature without runtime impact.
One could in theory have a look at the added library features, but they are all pretty much basic checks against null and a branching instruction.

No that makes, it was more about optimization potential in the implementation details. I am totally fine with not adding benchmarks here.

[psfinaki]

A few more notes.

[vzarytovskii]

I intend to merge it.
@psfinaki could you please make an issue (or multiple) with your findings.
The faster we get it to upstream, the sooner we can start inserting again and the sooner we will be able to test it ourselves. It might even make it to p7

[psfinaki]

Great job @dsyme and @T-Gro.

I played a lot of with the feature, IMO this definitely passes all the smoke tests, as per other things - see comments.

What would be amazing is to actually update the RFC because it has some TBDs and some inconsistencies, both internal ones (e.g. withNull function) and w.r.t. implementation (e.g. obj | null support). Splitting the helper RFCs will be also benefecial.

Let's

[psfinaki]

@vzarytovskii it's all green now, I fixed the trimming checks. As for the issues, I'd prefer force-merging it with unresolved comments so that @T-Gro can later make sense of them.

[dlidstrom]

It’s fantastic how you are bringing F# forward! 🎉