[API Proposal]: Future of Numerics and AI - Provide a downlevel System.Numerics.Tensors.TensorPrimitives class

[tannergooding]

Future of Numerics and AI

.NET provides a broad range of support for various development domains, ranging from the creation of performance-oriented framework code to the rapid development of cloud native services, and beyond. In recent years, especially with the rise of AI and machine learning, there has been a prevalent push towards improving numerical support and allowing developers to more easily write and consume general purpose and reusable algorithms that work with a range of types and scenarios. While .NET's support for scalar algorithms and fixed-sized vectors/matrices is strong and continues to grow, its built-in library support for other concepts, such as tensors and arbitrary length vectors/matrices, is behind the times. Developers writing .NET applications, services, and libraries currently need to seek external dependencies in order to utilize functionality that is considered core or built-in to other ecosystems. In particular, for developers incorporating AI and copilots into their existing .NET applications and services, we strive to ensure that the core numerics support necessary to be successful is available and efficient, and that .NET developers are not forced to seek out non-.NET solutions in order for their .NET projects to be successful.

.NET Framework Compatible APIs

While there are many reasons for developers to migrate to modern .NET and it is the goto target for many new libraries or applications, there remains many large repos that have been around for years where migration can be non-trivial. And while there are many new language and runtime features which cannot ever work on .NET Framework, there still remains a subset of APIs that would be beneficial to provide and which can help those existing codebases bridge the gap until they are able to successfully complete migration.

It is therefore proposed that an out of band System.Numerics.Tensors package be provided which provides a core set of APIs that are compatible with and appropriate for use on .NET Standard 2.0. There are also two types that would be beneficial to polyfill downlevel as part of this process, System.Half and System.MathF, which will significantly improves the usability of the libraries for common scenarios.

Provide the Tensor class

Note

The following is an extreme simplification meant to give the general premise behind the APIs being exposed. It is intentionally skimming over several concepts and deeper domain specific terminology that is not critical to cover for the design proposal to be reviewed.

At a very high overview, you have scalars, vectors, and matrices. A scalar is largely just a single value T, a vector is a set of scalars, a matrix is a set of vectors. All of these can be loosely thought of as types of tensors and they have various relationships and can be used to build up and represent higher types. That is, you could consider that T is a scalar, or a vector of length 1, or a 1x1 matrix. Likewise, you can then consider that T[] is a vector or unknown length and that T[,] represents a 2-dimensional matrix and so on. All of these can be defined as types of tensors. -- As a note, this terminology is partially where the name for std::vector<T> in C++ (and similar in other languages) comes from. This is also where the general considerations of "vectorization" when writing SIMD or SIMT arise, although they are typically over fixed-length, rather than arbitrary length.

From the mathematical perspective, many of the operations you can do on scalars also apply to the higher types. There can sometimes be limitations or other considerations that can specialize or restrict these operations, but in principle they tend to exist and work mostly the same way. It is therefore generally desirable to support many such operations more broadly. This is particularly prevalent for "element-wise" operations which could be thought of as effectively doing array.Select((x) => M(x)) and where an explicit API can often provide significant performance improvements over the naive approach.

The core set of APIs described below cover the main arithmetic and conversion operators provided for T as well as element-wise operations for the functionality exposed by System.Math/System.MathF. Some design notes include:

• Where the scalar API Method would return a bool, the vector API has two overloads MethodAny and MethodAll.
• APIS will require that developers utilize Span<T> to access these APIs, overloads taking T[] are not provided
• APIs will support the destination buffer and one of the source buffers being the same.
• APIs taking more than 1 operand will support the other operands being a scalar and it indicating the same value is used for every element.
• In order to make usage of the APIs less "clunky", particularly given these are mathematical APIs where building up complex expressions may be more prevalent, it is proposed a slight deviation from the normal API signature is taken. Rather than simply returning the number of elements written to the destination span, it is proposed that the APIs return a Span<T> instead. This would simply be destination sliced to the appropriate length. This still provides the required information on number of elements written, but gives the additional advantage that the result can be immediately passed into the next user of the algorithm without requiring the user to slice or do length checks themselves.

For targeting modern .NET, there will be a separate future proposal detailing a Tensor<T> type. This then matches a similar split we have for other generic types such as Vector<T> and Vector. The non-generic Tensor class holds extension methods, APIs that are only meant to support a particular set of T, and static APIs. While Tensor<T> will hold operators, instance methods, and core properties. The APIs defined for use on .NET Framework are effectively the "workhorse" APIs that Tensor<T> would then delegate to. They more closely resemble the signatures from the BLAS and LAPACK libraries, which are the industry standard baseline for Linear Algebra operations and allow tensor like functionality to be supported for arbitrary memory while allowing modern .NET to provide a type safe and friendlier way to work with such functionality that can simultaneously take advantage of newer language/runtime features, such as static virtuals in interfaces or generic math.

namespace System.Numerics.Tensors;

public static partial class Tensor
{
    // Element-wise Arithmetic

    public static Span<float> Add(ReadOnlySpan<float> left, ReadOnlySpan<float> right, Span<float> destination);
    public static Span<float> Add(ReadOnlySpan<float> left, T right, Span<float> destination);

    public static Span<float> Subtract(ReadOnlySpan<float> left, ReadOnlySpan<float> right, Span<float> destination);
    public static Span<float> Subtract(ReadOnlySpan<float> left, T right, Span<float> destination);

    public static Span<float> Multiply(ReadOnlySpan<float> left, ReadOnlySpan<float> right, Span<float> destination);
    public static Span<float> Multiply(ReadOnlySpan<float> left, T right, Span<float> destination); // BLAS1: scal

    public static Span<float> Divide(ReadOnlySpan<float> left, ReadOnlySpan<float> right, Span<float> destination);
    public static Span<float> Divide(ReadOnlySpan<float> left, T right, Span<float> destination);

    public static Span<float> Negate(ReadOnlySpan<float> values, Span<float> destination);

    public static Span<float> AddMultiply(ReadOnlySpan<float> left, ReadOnlySpan<float> right, ReadOnlySpan<float> multiplier, Span<float> destination);
    public static Span<float> AddMultiply(ReadOnlySpan<float> left, ReadOnlySpan<float> right, T multiplier, Span<float> destination);
    public static Span<float> AddMultiply(ReadOnlySpan<float> left, T right, ReadOnlySpan<float> multiplier, Span<float> destination);

    public static Span<float> MultiplyAdd(ReadOnlySpan<float> left, ReadOnlySpan<float> right, ReadOnlySpan<float> addend, Span<float> destination);
    public static Span<float> MultiplyAdd(ReadOnlySpan<float> left, ReadOnlySpan<float> right, T addend, Span<float> destination); // BLAS1: axpy
    public static Span<float> MultiplyAdd(ReadOnlySpan<float> left, T right, ReadOnlySpan<float> addend, Span<float> destination);

    public static Span<float> Exp(ReadOnlySpan<float> x, Span<float> destination);
    public static Span<float> Log(ReadOnlySpan<float> x, Span<float> destination);

    public static Span<float> Cosh(ReadOnlySpan<float> x, Span<float> destination);
    public static Span<float> Sinh(ReadOnlySpan<float> x, Span<float> destination);
    public static Span<float> Tanh(ReadOnlySpan<float> x, Span<float> destination);

    // Vector Arithmetic

    // A measure of similarity between two non-zero vectors of an inner product space.
    // It is widely used in natural language processing and information retrieval.
    public static float CosineSimilarity(ReadOnlySpan<float> x, ReadOnlySpan<float> y);

    // A measure of distance between two points in a Euclidean space.
    // It is widely used in mathematics, engineering, and machine learning.
    public static float Distance(ReadOnlySpan<float> x, ReadOnlySpan<float> y);

    // A mathematical operation that takes two vectors and returns a scalar.
    // It is widely used in linear algebra and machine learning.
    public static float Dot(ReadOnlySpan<float> x, ReadOnlySpan<float> y); // BLAS1: dot

    // A mathematical operation that takes a vector and returns a unit vector in the same direction.
    // It is widely used in linear algebra and machine learning.
    public static float Normalize(ReadOnlySpan<float> x); // BLAS1: nrm2

    // A function that takes a collection of real numbers and returns a probability distribution.
    // It is widely used in machine learning and deep learning.
    public static float SoftMax(ReadOnlySpan<float> x);

    // A function that takes a real number and returns a value between 0 and 1.
    // It is widely used in machine learning and deep learning.
    public static Span<float> Sigmoid(ReadOnlySpan<float> x, Span<float> destination);

    public static float Max(ReadOnlySpan<float> value);
    public static float Min(ReadOnlySpan<float> value);

    public static int IndexOfMaxMagnitude(ReadOnlySpan<float> value); // BLAS1: iamax
    public static int IndexOfMinMagnitude(ReadOnlySpan<float> value);

    public static float Sum(ReadOnlySpan<float> value);
    public static float SumOfSquares(ReadOnlySpan<float> value);
    public static float SumOfMagnitudes(ReadOnlySpan<float> value); // BLAS1: asum

    public static float Product(ReadOnlySpan<float> value);
    public static float ProductOfSums(ReadOnlySpan<float> left, ReadOnlySpan<float> right);
    public static float ProductOfDifferences(ReadOnlySpan<float> left, ReadOnlySpan<float> right);

    // Vector Conversions

    public static Span<Half> ConvertToHalf(ReadOnlySpan<float> value, Span<Half> destination);
    public static Span<float> ConvertToSingle(ReadOnlySpan<Half> value, Span<float> destination);
}

Polyfill System.Half in Microsoft.Bcl.Half

System.Half is a core interchange type for AI scenarios, often being used to minify the storage impact for the tens of thousands to millions of data points that need to be stored. It is not, however, as frequently used for computation.

We initially exposed this type in .NET 5 purely as an interchange type and it is therefore lacking the arithmetic operators. These operators were later added in .NET 6/7 as a part of the generic math initiative. For .NET Framework, the interchange surface area should be sufficient and will follow the general guidance required for polyfills that they meet the initial shape we shipped, even if the version it shipped on is no longer in support (that is, the .NET Standard 2.0 surface area needs to remain compatible with .NET 5, even though .NET 5 is out of support).

namespace System;

public readonly struct Half : IComparable, IComparable<Half>, IEquatable<Half>, IFormattable
{
    public static Half Epsilon { get; }
    public static Half MaxValue { get; }
    public static Half MinValue { get; }
    public static Half NaN { get; }
    public static Half NegativeInfinity { get; }
    public static Half PositiveInfinity { get; }

    public static bool operator ==(Half left, Half right);
    public static bool operator !=(Half left, Half right);

    public static bool operator >(Half left, Half right);
    public static bool operator >=(Half left, Half right);

    public static bool operator <(Half left, Half right);
    public static bool operator <=(Half left, Half right);

    public static explicit operator double(Half value);
    public static explicit operator Half(float value);
    public static explicit operator Half(double value);
    public static explicit operator float(Half value);

    public int CompareTo(Half other);
    public int CompareTo(object? other);

    public bool Equals(Half other);
    public override bool Equals(object? other);

    public override int GetHashCode();

    public static bool IsFinite(Half value);
    public static bool IsInfinity(Half value);
    public static bool IsNaN(Half value);
    public static bool IsNegative(Half value);
    public static bool IsNegativeInfinity(Half value);
    public static bool IsNormal(Half value);
    public static bool IsPositiveInfinity(Half value);
    public static bool IsSubnormal(Half value);

    public static Half Parse(string s);
    public static Half Parse(string s, NumberStyles style);
    public static Half Parse(string s, IFormatProvider? provider);
    public static Half Parse(string s, NumberStyles style = NumberStyles.AllowThousands | NumberStyles.Float, IFormatProvider? provider = default);
    public static Half Parse(ReadOnlySpan<char> s, NumberStyles style = NumberStyles.AllowThousands | NumberStyles.Float, IFormatProvider? provider = default);

    public override string ToString();
    public string ToString(string? format);
    public string ToString(string? format, IFormatProvider? provider);
    public string ToString(IFormatProvider? provider);

    public bool TryFormat(Span<char> destination, out int charsWritten, ReadOnlySpan<char> format = default, IFormatProvider? provider = default);

    public static bool TryParse(string? s, out Half result);
    public static bool TryParse(string? s, NumberStyles style, IFormatProvider? provider, out Half result);
    public static bool TryParse(ReadOnlySpan<char> s, out Half result);
    public static bool TryParse(ReadOnlySpan<char> s, NumberStyles style, IFormatProvider? provider, out Half result);
}

Polyfill System.MathF in Microsoft.Bcl.MathF

System.MathF was added in .NET Core 2.0 to provide float support that had parity with the double support in System.Math. Given that float is the core computational type used in AI scenarios, many downlevel libraries currently provide their own internal wrappers around System.Math. .NET ships several such shims for its own scenarios and the proposed System.Numerics library would be no exception. As such, it would be beneficial to simply provide this functionality officially and allow such targets to remove their shims. This simplifies their experience and may give additional performance or correctness over the naive approach.

namespace System;

public static class MathF
{
    public const float E = 2.71828183f;
    public const float PI = 3.14159265f;

    public static float Abs(float x);

    public static float Acos(float x);
    public static float Asin(float x);
    public static float Atan(float x);
    public static float Atan2(float y, float x);

    public static float Cos(float x);
    public static float Sin(float x);
    public static float Tan(float x);

    public static float Cosh(float x);
    public static float Sinh(float x);
    public static float Tanh(float x);

    public static float Ceiling(float x);
    public static float Floor(float x);
    public static float Truncate(float x);

    public static float Exp(float x);
    public static float Log(float x);
    public static float Log(float x, float y);
    public static float Log10(float x);

    public static float IEEERemainder(float x, float y);

    public static float Max(float x, float y);
    public static float Min(float x, float y);

    public static float Pow(float x, float y);

    public static float Round(float x);
    public static float Round(float x, int digits);
    public static float Round(float x, int digits, MidpointRounding mode);
    public static float Round(float x, MidpointRounding mode);

    public static int Sign(float x);

    public static float Sqrt(float x);
}

[ghost]

Tagging subscribers to this area: @dotnet/area-system-numerics
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Future of Numerics and AI

.NET provides a broad range of support for various development domains, ranging from the creation of performance-oriented framework code to the rapid development of cloud native services, and beyond. In recent years, especially with the rise of AI and machine learning, there has been a prevalent push towards improving numerical support and allowing developers to more easily write and consume general purpose and reusable algorithms that work with a range of types and scenarios. While .NET's support for scalar algorithms and fixed-sized vectors/matrices is strong and continues to grow, its built-in library support for other concepts, such as tensors and arbitrary length vectors/matrices, is behind the times. Developers writing .NET applications, services, and libraries currently need to seek external dependencies in order to utilize functionality that is considered core or built-in to other ecosystems. In particular, for developers incorporating AI and copilots into their existing .NET applications and services, we strive to ensure that the core numerics support necessary to be successful is available and efficient, and that .NET developers are not forced to seek out non-.NET solutions in order for their .NET projects to be successful.

.NET Framework Compatible APIs

While there are many reasons for developers to migrate to modern .NET and it is the goto target for many new libraries or applications, there remains many large repos that have been around for years where migration can be non-trivial. And while there are many new language and runtime features which cannot ever work on .NET Framework, there still remains a subset of APIs that would be beneficial to provide and which can help those existing codebases bridge the gap until they are able to successfully complete migration.

It is therefore proposed that an out of band System.Numerics.Tensors package be provided which provides a core set of APIs that are compatible with and appropriate for use on .NET Standard 2.0. There are also two types that would be beneficial to polyfill downlevel as part of this process, System.Half and System.MathF, which will significantly improves the usability of the libraries for common scenarios.

Provide the Tensor class

NOTE: The following is an extreme simplification meant to give the general premise behind the APIs being exposed. It is intentionally skimming over several concepts and deeper domain specific terminology that is not critical to cover for the design proposal to be reviewed.

At a very high overview, you have scalars, vectors, and matrices. A scalar is largely just a single value T, a vector is a set of scalars, a matrix is a set of vectors. All of these can be loosely thought of as types of tensors and they have various relationships and can be used to build up and represent higher types. That is, you could consider that T is a scalar, or a vector of length 1, or a 1x1 matrix. Likewise, you can then consider that T[] is a vector or unknown length and that T[,] represents a 2-dimensional matrix and so on. All of these can be defined as types of tensors. -- As a note, this terminology is partially where the name for std::vector<T> in C++ (and similar in other languages) comes from. This is also where the general considerations of "vectorization" when writing SIMD or SIMT arise, although they are typically over fixed-length, rather than arbitrary length.

From the mathematical perspective, many of the operations you can do on scalars also apply to the higher types. There can sometimes be limitations or other considerations that can specialize or restrict these operations, but in principle they tend to exist and work mostly the same way. It is therefore generally desirable to support many such operations more broadly. This is particularly prevalent for "element-wise" operations which could be thought of as effectively doing array.Select((x) => M(x)) and where an explicit API can often provide significant performance improvements over the naive approach.

The core set of APIs described below cover the main arithmetic and conversion operators provided for T as well as element-wise operations for the functionality exposed by System.Math/System.MathF. Some design notes include:

• Where the scalar API Method would return a bool, the vector API has two overloads MethodAny and MethodAll.
• APIS will require that developers utilize Span<T> to access these APIs, overloads taking T[] are not provided
• APIs will support the destination buffer and one of the source buffers being the same.
• APIs taking more than 1 operand will support the other operands being a scalar and it indicating the same value is used for every element.
• In order to make usage of the APIs less "clunky", particularly given these are mathematical APIs where building up complex expressions may be more prevalent, it is proposed a slight deviation from the normal API signature is taken. Rather than simply returning the number of elements written to the destination span, it is proposed that the APIs return a Span<T> instead. This would simply be destination sliced to the appropriate length. This still provides the required information on number of elements written, but gives the additional advantage that the result can be immediately passed into the next user of the algorithm without requiring the user to slice or do length checks themselves.

For targeting modern .NET, there will be a separate future proposal detailing a Tensor<T> type. This then matches a similar split we have for other generic types such as Vector<T> and Vector. The non-generic Tensor class holds extension methods, APIs that are only meant to support a particular set of T, and static APIs. While Tensor<T> will hold operators, instance methods, and core properties. The APIs defined for use on .NET Framework are effectively the "workhorse" APIs that Tensor<T> would then delegate to. They more closely resemble the signatures from the BLAS and LAPACK libraries, which are the industry standard baseline for Linear Algebra operations and allow tensor like functionality to be supported for arbitrary memory while allowing modern .NET to provide a type safe and friendlier way to work with such functionality that can simultaneously take advantage of newer language/runtime features, such as static virtuals in interfaces or generic math.

namespace System.Numerics.Tensors;

public static partial class Tensor
{
    // Element-wise Arithmetic

    public static Span<float> Add(ReadOnlySpan<float> left, ReadOnlySpan<float> right, Span<float> destination);
    public static Span<float> Add(ReadOnlySpan<float> left, T right, Span<float> destination);

    public static Span<float> Subtract(ReadOnlySpan<float> left, ReadOnlySpan<float> right, Span<float> destination);
    public static Span<float> Subtract(ReadOnlySpan<float> left, T right, Span<float> destination);

    public static Span<float> Multiply(ReadOnlySpan<float> left, ReadOnlySpan<float> right, Span<float> destination);
    public static Span<float> Multiply(ReadOnlySpan<float> left, T right, Span<float> destination); // BLAS1: scal

    public static Span<float> Divide(ReadOnlySpan<float> left, ReadOnlySpan<float> right, Span<float> destination);
    public static Span<float> Divide(ReadOnlySpan<float> left, T right, Span<float> destination);

    public static Span<float> Negate(ReadOnlySpan<float> values, Span<float> destination);

    public static Span<float> AddMultiply(ReadOnlySpan<float> left, ReadOnlySpan<float> right, ReadOnlySpan<float> multiplier, Span<float> destination);
    public static Span<float> AddMultiply(ReadOnlySpan<float> left, ReadOnlySpan<float> right, T multiplier, Span<float> destination);
    public static Span<float> AddMultiply(ReadOnlySpan<float> left, T right, ReadOnlySpan<float> multiplier, Span<float> destination);

    public static Span<float> MultiplyAdd(ReadOnlySpan<float> left, ReadOnlySpan<float> right, ReadOnlySpan<float> addend, Span<float> destination);
    public static Span<float> MultiplyAdd(ReadOnlySpan<float> left, ReadOnlySpan<float> right, T addend, Span<float> destination); // BLAS1: axpy
    public static Span<float> MultiplyAdd(ReadOnlySpan<float> left, T right, ReadOnlySpan<float> addend, Span<float> destination);

    public static Span<float> Exp(ReadOnlySpan<float> x, Span<float> destination);
    public static Span<float> Log(ReadOnlySpan<float> x, Span<float> destination);

    public static Span<float> Cosh(ReadOnlySpan<float> x, Span<float> destination);
    public static Span<float> Sinh(ReadOnlySpan<float> x, Span<float> destination);
    public static Span<float> Tanh(ReadOnlySpan<float> x, Span<float> destination);

    // Vector Arithmetic

    // A measure of similarity between two non-zero vectors of an inner product space.
    // It is widely used in natural language processing and information retrieval.
    public static float CosineSimilarity(ReadOnlySpan<float> x, ReadOnlySpan<float> y);

    // A measure of distance between two points in a Euclidean space.
    // It is widely used in mathematics, engineering, and machine learning.
    public static float Distance(ReadOnlySpan<float> x, ReadOnlySpan<float> y);

    // A mathematical operation that takes two vectors and returns a scalar.
    // It is widely used in linear algebra and machine learning.
    public static float Dot(ReadOnlySpan<float> x, ReadOnlySpan<float> y); // BLAS1: dot

    // A mathematical operation that takes a vector and returns a unit vector in the same direction.
    // It is widely used in linear algebra and machine learning.
    public static float Normalize(ReadOnlySpan<float> x); // BLAS1: nrm2

    // A function that takes a collection of real numbers and returns a probability distribution.
    // It is widely used in machine learning and deep learning.
    public static float SoftMax(ReadOnlySpan<float> x);

    // A function that takes a real number and returns a value between 0 and 1.
    // It is widely used in machine learning and deep learning.
    public static Span<float> Sigmoid(ReadOnlySpan<float> x, Span<float> destination);

    public static float Max(ReadOnlySpan<float> value);
    public static float Min(ReadOnlySpan<float> value);

    public static int IndexOfMaxMagnitude(ReadOnlySpan<float> value); // BLAS1: iamax
    public static int IndexOfMinMagnitude(ReadOnlySpan<float> value);

    public static float Sum(ReadOnlySpan<float> value);
    public static float SumOfSquares(ReadOnlySpan<float> value);
    public static float SumOfMagnitudes(ReadOnlySpan<float> value); // BLAS1: asum

    public static float Product(ReadOnlySpan<float> value);
    public static float ProductOfSums(ReadOnlySpan<float> left, ReadOnlySpan<float> right);
    public static float ProductOfDifferences(ReadOnlySpan<float> left, ReadOnlySpan<float> right);

    // Vector Conversions

    public static Span<Half> ConvertToHalf(ReadOnlySpan<float> value, Span<Half> destination);
    public static Span<float> ConvertToSingle(ReadOnlySpan<Half> value, Span<float> destination);
}

Polyfill System.Half in Microsoft.Bcl.Half

System.Half is a core interchange type for AI scenarios, often being used to minify the storage impact for the tens of thousands to millions of data points that need to be stored. It is not, however, as frequently used for computation.

We initially exposed this type in .NET 5 purely as an interchange type and it is therefore lacking the arithmetic operators. These operators were later added in .NET 6/7 as a part of the generic math initiative. For .NET Framework, the interchange surface area should be sufficient and will follow the general guidance required for polyfills that they meet the initial shape we shipped, even if the version it shipped on is no longer in support (that is, the .NET Standard 2.0 surface area needs to remain compatible with .NET 5, even though .NET 5 is out of support).

namespace System;

public readonly struct Half : IComparable, IComparable<Half>, IEquatable<Half>, IFormattable
{
    public static Half Epsilon { get; }
    public static Half MaxValue { get; }
    public static Half MinValue { get; }
    public static Half NaN { get; }
    public static Half NegativeInfinity { get; }
    public static Half PositiveInfinity { get; }

    public static bool operator ==(Half left, Half right);
    public static bool operator !=(Half left, Half right);

    public static bool operator >(Half left, Half right);
    public static bool operator >=(Half left, Half right);

    public static bool operator <(Half left, Half right);
    public static bool operator <=(Half left, Half right);

    public static explicit operator double(Half value);
    public static explicit operator Half(float value);
    public static explicit operator Half(double value);
    public static explicit operator float(Half value);

    public int CompareTo(Half other);
    public int CompareTo(object? other);

    public bool Equals(Half other);
    public override bool Equals(object? other);

    public override int GetHashCode();

    public static bool IsFinite(Half value);
    public static bool IsInfinity(Half value);
    public static bool IsNaN(Half value);
    public static bool IsNegative(Half value);
    public static bool IsNegativeInfinity(Half value);
    public static bool IsNormal(Half value);
    public static bool IsPositiveInfinity(Half value);
    public static bool IsSubnormal(Half value);

    public static Half Parse(string s);
    public static Half Parse(string s, NumberStyles style);
    public static Half Parse(string s, IFormatProvider? provider);
    public static Half Parse(string s, NumberStyles style = NumberStyles.AllowThousands | NumberStyles.Float, IFormatProvider? provider = default);
    public static Half Parse(ReadOnlySpan<char> s, NumberStyles style = NumberStyles.AllowThousands | NumberStyles.Float, IFormatProvider? provider = default);

    public override string ToString();
    public string ToString(string? format);
    public string ToString(string? format, IFormatProvider? provider);
    public string ToString(IFormatProvider? provider);

    public bool TryFormat(Span<char> destination, out int charsWritten, ReadOnlySpan<char> format = default, IFormatProvider? provider = default);

    public static bool TryParse(string? s, out Half result);
    public static bool TryParse(string? s, NumberStyles style, IFormatProvider? provider, out Half result);
    public static bool TryParse(ReadOnlySpan<char> s, out Half result);
    public static bool TryParse(ReadOnlySpan<char> s, NumberStyles style, IFormatProvider? provider, out Half result);
}

Polyfill System.MathF in Microsoft.Bcl.MathF

System.MathF was added in .NET Core 2.0 to provide float support that had parity with the double support in System.Math. Given that float is the core computational type used in AI scenarios, many downlevel libraries currently provide their own internal wrappers around System.Math. .NET ships several such shims for its own scenarios and the proposed System.Numerics library would be no exception. As such, it would be beneficial to simply provide this functionality officially and allow such targets to remove their shims. This simplifies their experience and may give additional performance or correctness over the naive approach.

namespace System;

public static class MathF
{
    public const float E = 2.71828183f;
    public const float PI = 3.14159265f;

    public static float Abs(float x);

    public static float Acos(float x);
    public static float Asin(float x);
    public static float Atan(float x);
    public static float Atan2(float y, float x);

    public static float Cos(float x);
    public static float Sin(float x);
    public static float Tan(float x);

    public static float Cosh(float x);
    public static float Sinh(float x);
    public static float Tanh(float x);

    public static float Ceiling(float x);
    public static float Floor(float x);
    public static float Truncate(float x);

    public static float Exp(float x);
    public static float Log(float x);
    public static float Log(float x, float y);
    public static float Log10(float x);

    public static float IEEERemainder(float x, float y);

    public static float Max(float x, float y);
    public static float Min(float x, float y);

    public static float Pow(float x, float y);

    public static float Round(float x);
    public static float Round(float x, int digits);
    public static float Round(float x, int digits, MidpointRounding mode);
    public static float Round(float x, MidpointRounding mode);

    public static int Sign(float x);

    public static float Sqrt(float x);
}

Author:	tannergooding
Assignees:	-
Labels:	area-System.Numerics, api-ready-for-review
Milestone:	-

[jkotas]

It is therefore proposed that an out of band System.Numerics.Tensors package be provided

What is going to happen with the existing content of System.Numerics.Tensors package that never shipped a stable version?

[tannergooding]

What happens with the existing content of System.Numerics.Tensors package that never shipped a stable version?

They will be removed and exist purely in source control history for the near term.

The general design of the types in the existing preview package isn't incorrect. However, they was done as a prototype and done several releases ago at this point, so it doesn't take into account newer features such as generic math, vectorization support, etc. We've also since determined that we do in fact need to provide basic operation support as we've hadmultiple requests from both first and third party customers.

What we're doing here is providing the foundations on which the package can be stabilized and shipped properly. With the initial stabilization being done for .NET Standard to answer some core needs. The Tensor<T> type (and corresponding support for DenseTensor/SparseTensor) will then be redone (building off what's already been done) taking into account recent .NET features and will be only available for modern .NET (targeting .NET 9/10). So they will come back in a future version of the package.

[jkoritzinsky]

Can we rename the Distance methods to EuclideanDistance to make sure it's clear which definition of distance they're using (as there's many different kinds of "distance" in Euclidean spaces)?

[gfoidl]

Sigmoid should be more specific, i.e. what sigmoid-function it's exactly -- e.g. SigmoidLogistic?

For the tensor operations should there be double operations as well?
(For my usecases double would be preferable)

[gfoidl]

Re: EuclideanDistance perf-wise it could be the squared distance, as the (square) root is monotonic, so for comparisons (i.e. what's the nearest) one doesn't need to calculate the square root.
If so, maybe name it EuclideanDistance2?

[tannergooding]

Can we rename the Distance methods to EuclideanDistance to make sure it's clear which definition of distance they're using (as there's many different kinds of "distance" in Euclidean spaces)?

We could, but we already define other APIs named Distance with them being the standard "pythagorean distance" (or straight line distance) that is most typical. I imagine other distances are then the ones that would need/get additional clarification as to their kind.

This is also common in other libraries/spaces. If we were to prefix it as EuclideanDistance, it would set precedent for other APIs such as Normalize to also be prefixed or named differently.

Re: EuclideanDistance perf-wise it could be the squared distance, as the (square) root is monotonic, so for comparisons (i.e. what's the nearest) one doesn't need to calculate the square root.
If so, maybe name it EuclideanDistance2?

It would have to be named *Squared to match the existing convention we use elsewhere. 2 itself is ambiguous. That being said, the cost of the square root tends to be trivial, particularly for large inputs, and its likely not worth it except for very small inputs.

We could also expose EuclideanDistanceSquared or DistanceSquared, but that may "open the floodgates" to doing this in many places.

Sigmoid should be more specific, i.e. what sigmoid-function it's exactly -- e.g. SigmoidLogistic?

Same here as with "Euclidean Distance". Many libraries simply expose the core SigmoidLogistic function as simply Sigmoid and only differentiate other variants as necessary.

There's a balance between clarity and matching existing industry standards we want to reach.

[tannergooding]

For the tensor operations should there be double operations as well?
(For my usecases double would be preferable)

Double isn't on the table, currently at least, for the .NET Standard support. It is very much on the table for the modern .NET support which will utilize generics and likely generic math.

[gfoidl]

There's a balance between clarity and matching existing industry standards we want to reach.

I agree for the distance part. Keep Distance as the dominant one is the euclidean distance. Other distance measures could be given as separate overload if there's demand -- or maybe better abstract it with IDistance and generics, which defaults to the euclidean one, others can be given easily. But that won't work for .NET Standard.

For the sigmoid part I disagree. There are too many of them available and used, each with somewhat different characteristics.
Although the name "sigmoid" is used by many libraries, one has to look into the docs to see what function exactly is implemented -- if it's documented.
So why not make it obvious by adding the concrete function type to the name?
Also I don't see a problem be being incosistent in the naming of Distance (w/o specifying the conrete euclidean distance) and SigmoidLogistic (w/ specifying the concrete type), as "distance" is in general refered to the euclidean one, whilst for sigmoid it's a group of functions with characteristic S-shape.

This is also common in other libraries/spaces.

Well...if other libraries have it that way it doesn't mean it's correct and that .NET has to follow along...if could be done better 😉 (I hope you get how I mean this)

[tannergooding]

Well...if other libraries have it that way it doesn't mean it's correct and that .NET has to follow along...if could be done better 😉 (I hope you get how I mean this)

The consideration is that while it is technically an entire class, there is effectively a domain standard that simply "sigmoid" by itself is the logistic function.

This is true in Intel MKL, Apple Accelerate, Wolfram Alpha and many other libraries oriented towards Linear Algebra (BLAS/LAPACK), as well as Machine Learning and math reference sources.

It is the de-facto response when you search "sigmoid function programming", the reference image used in almost any explanation of what a sigmoid is, and so on. Being different in this case is likely to hurt things more than it would help, particularly for the domain of users that are most likely to use these types; particularly if they have ever done similar in another language or domain. Simply because sigmoid(x) is almost definitively treated as y=1/(1+e^(-x))

Notably it is also what is done on many scientific or graphic calculators which support it.

[antonfirsov]

// A mathematical operation that takes a vector and returns a unit vector in the same direction.
// It is widely used in linear algebra and machine learning.
public static float Normalize(ReadOnlySpan<float> x); // BLAS1: nrm2

The description is misleading since the method presumably returns the scalar representing the Euclidean norm. Other API-s like Vector4.Normalize() actually return the normalized vector, so should this be named EuclideanNorm() instead?

[eutist]

Is the System.Numerics.Tensors library a sort of an attempt to imitate the classical tensors from tensor calculus? If so, it looks like it's need to implement the main concepts from tensor calculus and differential geometry, like co-/contravariance, contraction, differentiation, Cristoffels and so on. But if it is intended only for ML/AI, there is not obligatory. Who is the expected client of Tensor<T>, and what types of apps (except ML/AI) needs on it?

[terrajobst]

Video

• The T should be float
• We should replace left and right with x and y for consistency
• ConvertToXxx: first parameter should be named source
• All span-returning methods should return void
• For Half and MathF we'll follow up offline; they might not be needed.

namespace System.Numerics.Tensors;

public static partial class TensorPrimitives
{
    public static void Abs(ReadOnlySpan<float> x, Span<float> destination);

    public static void Add(ReadOnlySpan<float> x, ReadOnlySpan<float> y, Span<float> destination);
    public static void Add(ReadOnlySpan<float> x, float y, Span<float> destination);

    public static void AddMultiply(ReadOnlySpan<float> x, ReadOnlySpan<float> y, ReadOnlySpan<float> multiplier, Span<float> destination);
    public static void AddMultiply(ReadOnlySpan<float> x, ReadOnlySpan<float> y, T multiplier, Span<float> destination);
    public static void AddMultiply(ReadOnlySpan<float> x, float y, ReadOnlySpan<float> multiplier, Span<float> destination);

    public static void Cosh(ReadOnlySpan<float> x, Span<float> destination);

    public static float CosineSimilarity(ReadOnlySpan<float> x, ReadOnlySpan<float> y);

    public static float Distance(ReadOnlySpan<float> x, ReadOnlySpan<float> y);

    public static void Divide(ReadOnlySpan<float> x, ReadOnlySpan<float> y, Span<float> destination);
    public static void Divide(ReadOnlySpan<float> x, float y, Span<float> destination);

    public static float Dot(ReadOnlySpan<float> x, ReadOnlySpan<float> y);

    public static void Exp(ReadOnlySpan<float> x, Span<float> destination);

    public static int IndexOfMax(ReadOnlySpan<float> value);
    public static int IndexOfMin(ReadOnlySpan<float> value);

    public static int IndexOfMaxMagnitude(ReadOnlySpan<float> value);
    public static int IndexOfMinMagnitude(ReadOnlySpan<float> value);

    public static float Norm(ReadOnlySpan<float> x);

    public static void Log(ReadOnlySpan<float> x, Span<float> destination);
    public static void Log2(ReadOnlySpan<float> x, Span<float> destination);

    public static float Max(ReadOnlySpan<float> value);
    public static void Max(ReadOnlySpan<float> x, ReadOnlySpan<float> y, Span<float> destination);

    public static float MaxMagnitude(ReadOnlySpan<float> value);
    public static void MaxMagnitude(ReadOnlySpan<float> x, ReadOnlySpan<float> y, Span<float> destination);

    public static float Min(ReadOnlySpan<float> value);
    public static void Min(ReadOnlySpan<float> x, ReadOnlySpan<float> y, Span<float> destination);

    public static float MinMagnitude(ReadOnlySpan<float> value);
    public static void MinMagnitude(ReadOnlySpan<float> x, ReadOnlySpan<float> y, Span<float> destination);

    public static void Multiply(ReadOnlySpan<float> x, ReadOnlySpan<float> y, Span<float> destination);
    public static void Multiply(ReadOnlySpan<float> x, float y, Span<float> destination);

    public static void MultiplyAdd(ReadOnlySpan<float> x, ReadOnlySpan<float> y, ReadOnlySpan<float> addend, Span<float> destination);
    public static void MultiplyAdd(ReadOnlySpan<float> x, float y, ReadOnlySpan<float> addend, Span<float> destination);
    public static void MultiplyAdd(ReadOnlySpan<float> x, ReadOnlySpan<float> y, float addend, Span<float> destination);

    public static void Negate(ReadOnlySpan<float> values, Span<float> destination);

    public static float Product(ReadOnlySpan<float> value);
    public static float ProductOfSums(ReadOnlySpan<float> x, ReadOnlySpan<float> y);
    public static float ProductOfDifferences(ReadOnlySpan<float> x, ReadOnlySpan<float> y);

    public static void Sigmoid(ReadOnlySpan<float> x, Span<float> destination);

    public static void Sinh(ReadOnlySpan<float> x, Span<float> destination);

    public static float SoftMax(ReadOnlySpan<float> x);

    public static void Subtract(ReadOnlySpan<float> x, ReadOnlySpan<float> y, Span<float> destination);
    public static void Subtract(ReadOnlySpan<float> x, float y, Span<float> destination);

    public static float Sum(ReadOnlySpan<float> value);
    public static float SumOfMagnitudes(ReadOnlySpan<float> value);
    public static float SumOfSquares(ReadOnlySpan<float> value);

    public static void Tanh(ReadOnlySpan<float> x, Span<float> destination);

    // .NET 7+
    public static void ConvertToHalf(ReadOnlySpan<float> source, Span<Half> destination);
    public static void ConvertToSingle(ReadOnlySpan<Half> source, Span<float> destination);
}

[eerhardt]

All span-returning methods should return void

Looks like Sigmoid was missed in the above approved API.

public static Span<float> Sigmoid(ReadOnlySpan<float> x, Span<float> destination);

[bartonjs]

Looks like Sigmoid was missed in the above approved API.

Fixed.

[Lanayx]

There's already a tensor library in .NET space, namely https://github.com/dotnet/TorchSharp. I wonder what is the benefit of putting thousands of working hours on reinventing the wheel, rather than just spend a bit of time of improving the quality and stability of the existing one?
The API above covers about 1% of the operations that should be available for the ML developer, who is going to develop the other 99%? And if hypothetically it is implemented, is there any demand for this? Is there a crowd of people who want ML in .NET but TorchSharp doesn't work for them so they want something brand new from ground up?

[huoyaoyuan]

There's already a tensor library in .NET space, namely https://github.com/dotnet/TorchSharp. I wonder what is the benefit of putting thousands of working hours on reinventing the wheel, rather than just spend a bit of time of improving the quality and stability of the existing one?

Strictly speaking, TorchSharp isn't in .NET space. The implementation of tensor storage and operations are in native code. I believe this proposal is more about tensor operations in managed memory.

[Lanayx]

Strictly speaking, TorchSharp isn't in .NET space. The implementation of tensor storage and operations are in native code. I believe this proposal is more about tensor operations in managed memory.

And this doesn't make sense as well - using managed memory and CPU rather than video memory and GPU is at least 10x slower, so nobody will use it for real training

[saint4eva]

There's already a tensor library in .NET space, namely https://github.com/dotnet/TorchSharp. I wonder what is the benefit of putting thousands of working hours on reinventing the wheel, rather than just spend a bit of time of improving the quality and stability of the existing one? The API above covers about 1% of the operations that should be available for the ML developer, who is going to develop the other 99%? And if hypothetically it is implemented, is there any demand for this? Is there a crowd of people who want ML in .NET but TorchSharp doesn't work for them so they want something brand new from ground up?

TorchSharp gears towards python developers who want to use .NET and the APIs and naming convention are not ideomatic. Moreover, it is a wrapped over c++ implementation

[tannergooding]

The purpose of this isn't to compete with or replace something like TorchSharp/Tensorflow/etc. It's to provide a core set of primitive linear algebra APIs that are currently "missing" from .NET.

That core set is effectively ensuring we have BLAS/LAPACK like functionality provided out of the box and will entail these core "workhorse" algorithms and a friendly Tensor<T> type for making it easier to use.

This will be beneficial for prototyping and potentially various types of interchange as well. It also doesn't exclude the ability for some other tooling to come in and allow execution of such code on the GPU (see ComputeSharp which does this for regular C# code), etc.

---

Really this can be viewed as simply the "next step" on top of the existing math and numerics "primitives" we already provide and simply expanding it to include some additional APIs/algorithms that have been the de-facto industry standard for nearly 50 years. With this proposal covering the bare minimum support and a future proposal extending that to the rest of the relevant surface area.

[Lanayx]

Right, so I tried to question the fact that linear algebra APIs are really "missing" :) If you look at the description of torch (which is what gets wrapped by torchsharp) then it

contains data structures for multi-dimensional tensors and defines mathematical operations over these tensors.

which is exactly as you describe as "tensors and workhorse algorithms" + some utilities and this is a huge amount of thorough work

In the API above there is a series of choices that are highly opinionated and are good to be done in a separate nuget library rather than in .NET framework itself, namely

1. Tensor is generic type with one generic parameter. This can become pretty serios restriction in future, for example if this proposal lands we might want to add tensor dimensionality as constant generic parameter. Also there are reasons why TorchSharp didn't make Tensor generic.
2. The operations above are described as Spans and ReadonlySpans which makes them to be not usable directly (I assume you are envisioning Tensor to be directly castable to span) - in many cases where performance is important you'd want to perform scalar operations in-place, e.g. myTensor.Add(1), also there are cases where it has to be tensor (to understand dimensions), not just span, for example dot product.
3. There is no agreement in industry what is a Half type - it can be FP16 or BFLOAT16 and really it should be up to developer what to choose, not to hardcode it into framework

[AaronRobinsonMSFT]

It's to provide a core set of primitive linear algebra APIs that are currently "missing" from .NET.

These APIs are not in the current BCL, but I'm not sure that equates to missing. Based on the above queries I think we should understand who we believe the users of these will be as opposed to people consuming the existing high quality NuGet packages. What makes the existing .NET solutions, TorchSharp/Tensorflow/etc, need in box replacements?

[stephentoub]

In some cases it is the platform itself needing the API, e.g. various places we handroll vectorized implementations, such as with Sum/Average/Min/Max/etc. in LINQ. There are also places where we'd use the optimized functionality if it were available, but it's not worth the implementation maintaining its own optimized implementation, so it doesn't. APIs in the platform can't depend on external sources.

In other cases, whether we like it or not, there are consumers that are very limited in what non-platform dependencies they're able to take. We know of a non-trivial number in this camp; I'd be happy to chat offline, @AaronRobinsonMSFT, if you'd like.

In other cases, consumers can take a dependency, but don't want to force a large external dependency purely for the one or two functions they need. If they're in the platform, they'd use them; otherwise, they end up rolling their own.

We want developers reaching for and using nuget. We will still push developers to solutions like TorchSharp. This is not a replacement or competitor for that. This is about building some widely used, core routines into the core libraries, so that they're available to the breadth of applications that need them.

[AaronRobinsonMSFT]

there are consumers that are very limited in what non-platform dependencies they're able to take

but don't want to force a large external dependency purely for the one or two functions they need

This is about building some widely used, core routines into the core libraries, so that they're available to the breadth of applications that need them.

The above three statements are a good insight into the purpose of these APIs, thanks. I agree with this intent and I think it aligns with what was mentioned in #89639 (comment).

Given the above statement I think some confusion is being introduced with the name of the class (that is, Tensor). This is a precise mathematical term that is being used in a generalized/future looking form here, but it is also now overloaded in the software community and comes with certain expectations for more than what is described above. I think seeing a class called "Tensor" implies a level of breadth that is likely ill-suited for the BCL and should be left to the domain of experts in the field rather than in a base library.

Given the queries we've seen here and in some other issues, the community is expressing a misalignment of the aforementioned intent with potential expectations and that, to me, is a red flag for the name. For example, if the class was named LinearOps, not suggesting just a straw-man, then the intent as described above would be clear without inadvertently expressing broader aspirations.

@stephentoub I'll reach out tomorrow.

[tannergooding]

This is a precise mathematical term that is being used in a generalized/future looking form here

Many names/terms in programming come from mathematics and they are almost all molded to fit into some similar but not quite right abstraction. In programming, Tensor isn't some new name nor is it being overloaded recently, its been used as it is here for the last 20+ years. For better or worse, it is the "standardized" term for what is being added here and it matches the mathematical definition about as well as the typical usages of vector does and in a more appropriate way than many ecosystems use such terms.

This proposal is starting the addition of a set of linear algebra APIs that have a 45+ year history and which have broad baseline support in multiple languages/ecosystems. This ends up being support for scalar, vector, and matrix operations, of various dimensions, all of which are encompassed by the name Tensor. Hence why packages frequently use the term Tensor when providing such functionality. Many such packages go far beyond the "baseline" support and get into the much deeper domain specific specializations and optimizations. This proposal does not represent the entire set of what will be added to the BCL, it represents the core set that fits the immediate needs of several 1st party partner teams until the point they can transition to modern .NET. As described in the original post, the plan is to expand this to be more feature complete and cover the additional core areas. This will eventually complete out the initial goal of the System.Numerics.Tensors package when it was first being prototyped. That is to provide a set of core interchange types/APIs. There will be a set of primitive operation support provided such that the developers can do minimal work with such data. This effectively encompasses the core BLAS/LAPACK specs that more xpansive functionality is then built on. Much as System.Math provided the core functionality required for scalar support and which itself was "lacking" many core APIs until .NET Core came around.

Developers who need more advance or complete functionality should opt to use more advanced libraries, such as TensorFlow.NET, TorchSharp, ML.NET, ONNX, Semantic Kernel, etc. That, doesn't, however, negate the benefits that these will provide, it also doesn't require them to be in corelib, etc.

I think seeing a class called "Tensor" implies a level of breadth that is likely ill-suited for the BCL and should be left to the domain of experts in the field rather than in a base library.

I would strongly disagree. What's being provided here is core functionality that several other ecosystems provide and which higher level libraries build on top of in other ecosystems. There are many benefits to providing such baseline functionality in box and it is not a far step from what we're already providing, it's simply expanding that support from scalars and fixed-length vectors/matrices to arbitrary length vectors/matrices. It allows .NET developers to achieve more with .NET, to get the expected behavior/performance across all platforms we support (Arm64, x64, etc), and ensures that they don't require large (and sometimes limited -- many of the most prominent packages for BLAS/LAPACK like functionality are hardware or vendor specific and may not work or behave well on other platforms/hardware) external dependencies for what is effectively baseline functionality. They only need the additional packages when going beyond the basics, which is the same experience they'd get in other prominent languages/ecosystems.

Given the queries we've seen here and in some other issues, the community is expressing a misalignment of the aforementioned intent with potential expectations and that, to me, is a red flag for the name. For example, if the class was named LinearOps, not suggesting just a straw-man, then the intent as described above would be clear without inadvertently expressing broader aspirations.

There are users, as with any larger proposal, who are engaging in conversation on the topic and direction. As is typical, users who are happy with the direction have less reason to engage directly and are simply leaving thumbs ups or other positive reactions (or no reaction). Users who are unhappy are engaging more directly with questions/concerns.

Listening to and responding to those questions/concerns is important; but it doesn't mean there is now a big red flag or that this is the wrong direction. There will always be a subset of the community that is unhappy with any feature provided (BCL, Language, Runtime, etc). There will always be a subset of the community that believes a given feature isn't worthwhile, that it is the wrong direction, etc. So the feedback has to be taken and weighed against what domain experts are asking for, what the area owners believe the correct direction is, what other language ecosystems are doing, etc.

We also got a lot of feedback on generic math that we should be exposing concepts like rings, groups, and fields. We got feedback that having concrete interfaces for nominal typing was the wrong direction and that instead it should be a more structurally typed based approach. Such considerations were discussed at length and a decision was made that benefited the entirety of .NET.

[jkotas]

What's being provided here is core functionality that several other ecosystems provide and which higher level libraries build on top of in other ecosystems.

Are there examples in other ecosystems where the basic algebra operations like Span<float> Add(ReadOnlySpan<float> left, ReadOnlySpan<float> right, Span<float> destination); are under Tensor type name or namespace? I think the name choice is what is leading to the confusion here.

[AaronRobinsonMSFT]

The namespace here includes Tensors so I think the domain is rather well-defined. I would suggestion in the vein of MathF that the static class called Tensor be renamed to something else. Speaking with @stephentoub offline he was able to express some additional context that reduces my concerns and I would say also speaks to those expressed in both #89639 (comment) and #89639 (comment). I'm not necessarily going to push hard, but I think a rename of the class is appropriate. Below are a few options that seem reasonable to me and I think capture the currently expressed intent by both @stephentoub and @tannergooding.

• MathT - is troublesome due to T being the common generic parameter, but does follow the MathF pattern.
• VectorMath - Expresses a much closer statement to the existing APIs here. It does introduce confusion with our Vector types, but within the Tensors namespace I think that is reduced.
• LinearMath - Personally my preference as it precisely defines the current API surface area and expresses the present well supported goals of these APIs.

I can of course accept Tensor but I think it is introducing some longer term confusion and personally outside of the domain of what the BCL should be providing.

[AaronRobinsonMSFT]

Are there examples in other ecosystems where the basic algebra operations

@jkotas I found the following:

Julia has LinearAlgebra. The description there matches much of what was said about BLAS and LAPACK above.

Python has NumPy and that has a linalg category. This description also matches statements about BLAS and LAPACK.

Rust has a myriad of linear algebra crates, most are clear about "linear algebra". It also has a "standard" tensor crate that isn't generic - food for thought with respect to the one we have been working on. The generic aspect also aligns with question (1) from #89639 (comment).

Based on this, happy to see counter examples, I think a rename is reasonable given the stated intent of this API as found in #89639 (comment).

It's to provide a core set of primitive linear algebra APIs that are currently "missing" from .NET.

[tannergooding]

We have been explicitly moving away from Math/MathF like classes due to problems that show up with regards to overload resolution and usability. Particularly with the presence of things like generic math and many of the operations that are intended to be supported here, having a concrete type under which users can locate and perform their operations and use operators is desirable. I don't believe exposing a LinearAlgebra type will be desirable and believe that it will be a step in the wrong direction and significantly hinder our ability to expose and integrate this functionality using newer features.

It will also hinder the ability to design and expose a tensor interchange type/interfaces that will allow larger libraries to have a common interface and expand the functionality themselves -- i.e. One of the original reasons the System.Numerics.Tensors package was started as an experiment several years back and which saw a lot of positive response. Just which we didn't finish due to time constraints and a few open questions around whether basic operations (some of whats in this proposal) should also be provided.

---

I think people are seeing the name Tensor and making a mountain out of a molehill. Tensor is the industry standard name and encompasses what we are looking to support. I see no chance for confusion here and there isn't any deviation from what it means in other libraries or ecosystems. Some more advanced libraries may provide even more functionality, but that doesn't change what a tensor is/means nor what the foundations (which is all we're looking to provide) are. It's no different from anywhere else we provide a type and the foundations and which another library may define its own version of, or build on top of ours, or provide their own extended/different functionality around. Which they do for float, double, the integer primitives, and many other types throughout.

This static class here is just the first part and its designed and intended to be paired with a Tensor<T> and potentially ITensor<T> type, much as we've done and succeeded around elsewhere (Vector and Vector<T> is one simple example). It only encompasses the bare minimum needed for .NET Framework with the intent for everything else to be provided in .NET 9/10 and for modern .NET only. It is not the complete picture and was not intended to be due to time constraints and the needs of .NET Standard vs modern .NET. The entire picture has already been considered (with a design doc being worked on), was discussed a bit in API review and with interested parties, is taking into account input from major parties and our findings from the initial System.Numerics.Tensors prototype, etc.

[tannergooding]

We could potentially split it out into DenseVector, SparseVector, DenseMatrix, and SparseMatrix. However, there is a risk of potential confusion with the existing Vector types (which are for SIMD vectors) and we'd need to rely on docs or something else to help explain the difference. There would also be potential problems with slicing and mixing the two types which is solved by having a single Tensor type in most other cases.

[AaronRobinsonMSFT]

I don't believe exposing a LinearAlgebra type will be desirable and believe that it will be a step in the wrong direction and significantly hinder our ability to expose and integrate this functionality using newer features.

How would this impact future features? Right now the functions are all specfically using Span<T>/ReadOnlySpan<T> and primitive floating point types. The two examples, Python and Julia, above align with industry standards at least Python seems to be the most popular and accepted entry into this domain. If those two have adopted LinearAlgebra, what are we gaining by diverting?

Note that I am focused on the stated reason for this work in #89639 (comment). Providing future types and scenarios that have yet to be reviewed or that are planned seems premature when the industry seems to have no confusion with the existing languages in the space (that is, Python). I don't see how placing these linear algebra APIs within a LinearAlgebra class is detrimental.

[jkotas]

This static class here is just the first part and its designed and intended to be paired with a Tensor and potentially ITensor type

This assumes that we won't need to end up introducing non-generic Tensor to deliver a usable set of exchange types. I am not convinced that it is a safe assumption to make. If we need to introduce a non-generic Tensor, we won't have a good option to go with. In other words, this proposal is burning the good name that we may want to use in future for something else.

[tannergooding]

In other words, this proposal is burning the good name that we may want to use in future for something else.

The other name will already be burned by having a generic Tensor<T>. We typically pair things as struct/class Type<T> and static class Type, since that makes it simple and intuitive to access the relevant APIs. It also makes it a convenient place to hold extension methods if needed for perf/etc.

A non-generic tensor would effectively end up like Array which itself is very limited and is not the goto for the vast majority of scenarios. The interchange type would likely be better suited as a non-generic ITensor at that point (where helper APIs or extensions are then on the static class Tensor).

How would this impact future features?

We'd end up, once again, with APIs in two distinct locations and having to face the problems around overload resolution and discoverability. Its the same problem we had for years on Math vs double and later float vs MathF where some APIs were in one place and other APIs in another. Having to direct devs on how to find things was increasingly problematic; particularly given that the primitives were different from how almost all other types operated (where APIs were directly on the type in question or a non-generic type of the same name).

If those two have adopted LinearAlgebra, what are we gaining by diverting?

We aren't diverting IMO. We're exposing the same BLAS/LAPACK APIs in a way that allows developers to have type safety, use operators, etc. In a way that fits in with the broader design goal and with the way .NET itself has been moving towards doing these things. In a way that makes them friendlier and easier to use.

If we didn't need to support .NET Standard at all, we might consider not exposing these driver APIs at all and only exposing Tensor<T>. However, since these driver APIs, are simply taking in a linearized view of the underlying storage and are how the actual Tensor APIs would be implemented; they provide a bit more flexibility and provide an allocation free path to doing the operations. Other specialized library may also opt to use them to get optimized CPU based computation for primitive operations of their own tensor types (much as they already use things like Sqrt or Cos for scalars around). They could then focus instead on things like GPU compute without needing to worry about the foundational layers.

Providing future types and scenarios that have yet to be reviewed or that are planned seems premature when the industry seems to have no confusion with the existing languages in the space (that is, Python).

Part of any API proposal needs to take into account past, current, and future directions to try and help ensure we don't end up with something that cannot evolve, to ensure we don't end up with work that will just be thrown away, etc.

We have an overall design of where this supposed to end up and we have a more immediate goal of needing to provide a .NET Standard OOB for a subset of that functionality. This is simply taking that subset and exposing it.

What is effectively being suggested as an alternative is we take what's proposed for System.Numerics.Tensors.Tensor and move it into System.Numerics.LinearAlgebra. We would then later end up with Tensor<T>/Tensor, SparseTensor<T>/SparseTensor that defer their implementations to LinearAlgebra.*. It's worth noting that some of the APIs aren't strictly linear algebra themselves and are instead efficient implementations of core math APIs over vectors/matrices/tensors.

This ultimately splits things apart and makes it more nuanced when we decide to expose a particular API that isn't quite linear algebra but which does fit into the broader tensor support (e.g. supporting most of the core operations that INumber<T> supports) and where "driver" APIs that operate over raw spans is still desirable.

I think that will just hurt things longer term and we're ultimately better having them on Tensor

[tannergooding]

There was an offline discussion and then a confirmation with API review that we will go with TensorPrimitives as the class name.

Using the name TensorPrimitives as the class name is like other classes, such as BinaryPrimitives. It would only be used for low-level implementation primitives, that is these "driver" APIs that operate over spans. In the future, when we look at providing a concrete Tensor<T> type is when we would look at providing the static Tensor class to avoid burning the name or causing potential confusion in exposed API surface.

[MichalPetryka]

Using the name TensorPrimitives as the class name is like other classes, such as BinaryPrimitives.

Wouldn't using TensorOperations like BitOperations make more sense?

[tannergooding]

We didn't believe so. BitOperations is a bit different in terms of what it provides and how it operates as compared to BinaryPrimitives. The tensor APIs are more like the latter than the former.

[Lanayx]

Taking into account that none of those operations have anything to do with tensors or AI, but rather with spans, would it make more sense to name it SpanPrimitives or BlasPrimitives to completely remove any misunderstandings?

[tannergooding]

The APIs here are the primitive driver/workhorse APIs and therefore take a span representing the backing memory for a tensor. They are still "over tensors" even if there isn't a concrete type for them.

As more of BLAS/LAPACK is added, that will also include APIs supporting sparse data and matrix data. Concrete Tensor types will (provided the plan doesn't change) then come later to provide a more type safe way of operating over the same.

[jetersen]

What about vector math for Pow, at least the current System.Numerics does not provide a pow that uses Vector math.

I have vector math project (That I always intended to open source but never got around to it) that did some of these math operations we used for signal processing.

    [MethodImpl(MethodImplOptions.AggressiveInlining)]
    public static void Pow(Span<float> left, float exponent, Span<float> result, int length)
    {
        var leftVector = MemoryMarshal.Cast<float, Vector256<float>>(left);
        var resultVector = MemoryMarshal.Cast<float, Vector256<float>>(result);
        var numberOfVectors = resultVector.Length;
        var vectorCeiling = numberOfVectors * Vector256<float>.Count;
        var exponentVector = Vector256.Create(exponent);

        for (var i = 0; i < numberOfVectors; i++)
        {
            resultVector[i] = Pow(leftVector[i], exponentVector);
        }

        // Finish operation with any numbers leftover
        for (var i = vectorCeiling; i < length; i++)
        {
            result[i] = Pow(left[i], exponent);
        }
    }
    
    [MethodImpl(MethodImplOptions.AggressiveInlining)]
    public static Vector256<float> Pow(Vector256<float> left, Vector256<float> right)
    {
        return Exp(Avx.Multiply(right, Log(left)));
    }

    [MethodImpl(MethodImplOptions.AggressiveInlining)]
    public static float Pow(float left, float right)
    {
        return MathF.Pow(left, right);
    }

For Log and Exp we took inspiration from here: https://github.com/dotnet/runtime/blob/c1ba5c50b8ae47dda48a239be6dd9c7b6acc5c48/src/tests/JIT/Performance/CodeQuality/HWIntrinsic/X86/PacketTracer/VectorMath.cs

For anyone interested we took some python code and turned it into c# and made the code 44400 times faster. From 222 milliseconds to 5 microseconds 😅
This was calculating certain concentration of molecules based on absorbance spectroscopy with proprietary models. Thanks to reducing array copying and reference copying using Spans and Vector math.

The reason for our Pow API to expose length was that the spans might be bigger than the range that we were computing on or the fact that different ranges were using different exponent for the pow.

[tannergooding]

This proposal was only about the "core" surface area needed for the .NET Standard target.

APIs like Pow (and most of what MathF provides) are on the general backlog of what to provide as part of the next steps. Whether they will be only available to modern .NET or available to .NET Standard as well will have to be determined when they go to API review.

[xoofx]

I think that System.Numerics.Tensors.TensorPrimitives makes sense, but I would suggest to not bring any higher level Tensor<T> to BCL, if it is to bring types similar in the way Vector3/Vector4 are done, while the practical usage of tensors is in the end vastly different.

I have prototyped a tensor library earlier this year (not OSS yet), and discovered that in order to have a meaningful system that can plug with heterogeneous computation (CPU, GPU, Tensor cores...etc.) and symbolic calculation (for computing derivatives and backward propagation), the Tensor types have to be built around a computation graph implicitly built with operations (that are operation nodes while tensors are on the edges), and that computation is completely async and deferred to a computation engine that might execute things on the CPU or GPU...etc. You need also to have all these operations on tensors to be pluggable (e.g new kind of execution engine or dispatch between CPU/GPU workloads). Anything that would be designed around a straight implementation (e.g like Vector3/Vector4) would be quite limited in their usage/usefulness.

Otherwise for System.Numerics.Tensors.TensorPrimitives I did implement similar helpers (for the CPU engine), including vectorized versions of Log/Exp/Sigmoid...etc and their derivatives. Though internally I add to implement it with operations so that I could batch them in similar ways (binary operations, that can work SIMD + scalar, unary that can work similarly...etc.)