If I understood correctly, then LanguageExt.Meta will extend the C# syntax.

Not quite, the way to think about this is it's a little bit like the Roslyn API, but it works only for types. But because values are also types it is effectively a whole language API. That's because generic-types that have values applied as their generic arguments are no different to lambda-functions. It's all just a case of when it runs.

The when is important. Because that's where the flexibility comes in. You could:

• Build your own standalone language compiler that uses the Meta API to do type-inference, type-checking, and ultimately walk the type-trees and generate your compiled output.
• Build a source-generator that uses the Meta API instead of the Roslyn API - to do more powerful type-based processing, before spitting out some C#
• Use the Meta API live as the ultimate validator, for things like API surfaces and the like.

For example, you can use the raw API to define an option-type:

var optionA = 
    TLam("a",
        Sum(Case("Some", TVar("a")),
            Case("None", Empty)));

This creates a 'type lambda abstraction' (this is the generic parameter), with a Sum type within that has two cases: Some and None. The Some case uses the lambda argument for its parameter.

Now you have an in-code specification for a generic sum-type.

Then you can apply an argument to it:

var optionInt = TApp(optionA, Ty.Integer);

Now you have an Option Integer.

Or, you could instead apply a value:

var option123 = TApp(optionA, Ty.Value(123));

Now you have an Option 123.

You can then check if one is the sub-type of the other:

option123.IsSubTypeOf(optionInt)

Which means you can create data 'types' that represent the raw data in your application (think: a live web-request, or whatever). You can then check if that raw, live, data-type is a sub-type of the wider expected type, which validates the input in the same way a compiler validates your code.

In reality you'd want to apply the 123 to Some and not to the whole Option type. So, applying the constructor is like this:

var some123 = Cons(optionInt, "Some", Ty.Value(123));

Then you can do the sub-typing check again:

some123.IsSubTypeOf(optionInt)

This will validate that Some 123 is a sub-type of Option Integer.

Because the type-system is super flexible, it has other types that you just don't get in C#, like true union-types:

var option1or2or3 = TApp(optionA, Exists(Ty.Value(1), Ty.Value(2), Ty.Value(3)));

Now option1or2or3 will only allow None | Some 1 | Some 2 | Some 3. So this will fail:

some123.IsSubTypeOf(option1or2or3)

Because Some 123 is not a valid value for the type.

The sub-typing aspect of this (not OOP subtyping, but partial orders) is fundamental. It means we can treat data as types and we can describe types that the data should be 'less-than or equal to'. If a type is less-than or equal to another type then it's the subtype of it and can therefore be assigned to it (i.e. it type checks).

Now obviously if we're declaring types, functions, etc. then we should have a type-system to put them in; to build more complex types (like records, etc.) that refer to each other.

Here's a small example from one of my prototypes:

Universe.Ours.Process(
    TypeSystem("Global",
        Declare("Option",
            TLam("a",
                Sum(Case("Some", TVar("a")),
                    Case("None", Empty)))),
        Declare("Either",
            TLam("l",
                TLam("r",
                    Sum(Case("Left", TVar("l")),
                        Case("Right", TVar("r")))))),
        
        Declare("Seq", TLam("a", List(TVar("a"))))),
    TypeSystem("Reader",
        Declare("Either", DotNet(typeof(EitherReader<,>))),
        Declare("Option", DotNet(typeof(OptionReader<>))),
        Declare("Seq", DotNet(typeof(SeqReader<>)))
        ),
    TypeSystem("Writer",
        Declare("Either", DotNet(typeof(EitherWriter<,>))),
        Declare("Option", DotNet(typeof(OptionWriter<>))),
        Declare("Seq", DotNet(typeof(SeqWriter<>)))));

This uses the 'top level' universe, called Ours, but you can create parallel universes (other instances of Universe)

• The Ours universe has three type-systems added to it: Global, Reader, and Writer.
• Declare adds a type-declaration to the type-system
• You should see that Option, Either, and Seq are added as types to the Global type-system
• But there are also equivalently named types added to the Reader and Writer type-systems.
  • This is how (my prototype) knows how to convert the .NET types; Option, Either, and Seq into their Meta equivalents.

The Reader and Writer type-systems are, if you squint, a kind of type-class/trait like system. Being able to move between .NET types and Meta types opens up a door to a huge amount of future functionality. I have already written generic serialisers/deserialisers that will serialise/deserialise any Meta types to and from JSON. Which means any type that gets added to the Reader or Writer type-system automatically becomes serialisable.

Obviously, having to make the reader and writer implementations per-type is awkward, but that's where the source-generators will come in. The ability to generate the reader and writer implementations automatically and add them to the Ours type-system save a lot of typing.

You would only need to implement the reader/writers manually when you have a type that doesn't 'play by the rules'. An example is Option<A> in language-ext. It's not a true sum-type. So the reader/writers need to be written manually. This is what it looks like for Option<A>:

public class OptionReader<A> : TyReader
{
    public static Context<Option<A>> Read(Ty obj) =>
        obj is TyCons cons
            ? cons.Constructor switch
            {
                "Some" when 
                    cons.Items.Count == 1 && 
                    cons.Items[0] is TyValue item =>
                    Context.Pure(Option<A>.Some(item.CastUnsafe<A>())),
                "None" when 
                    cons.Items.Length == 0 
                    => Context.Pure(Option<A>.None),
                _ => Context.Fail<Option<A>>(
                    ErrorsExt.SerialisationFailed($"Unknown constructor for Option: {cons.Constructor} or invalid number of arguments"))
            }
            : Context.Fail<Option<A>>(ErrorsExt.SerialisationFailed($"Unknown type for Option: {obj}"));
    public Context<Ty> ReadValue(Ty data) =>
        Read(data).Map(Ty.Value);
}
public class OptionWriter<A> : TyWriter
{
    static readonly Ty Type =
        Ty.TApp(Ty.Id("Option"), Ty.DotNet<A>());
    Context<Ty> Write(Option<A> ma, Func<object?, Context<Ty>> valueConstruct) =>
        ma.Match(
            Some: r => valueConstruct(r).Map(x => Ty.Cons(Type, "Some", x)),
            None: () => Context.Pure(Ty.Cons(Type, "None")));
    public Context<Ty> WriteValue(Ty value, Func<object?, Context<Ty>> valueConstruct) =>
        value is TyValue<object> { Annotation: Option<A> val } 
            ? Write(val, valueConstruct)
            : Context.Fail<Ty>(ErrorsExt.SerialisationFailed($"Unknown type for Option: {value}"));
}

Again, this is all from my prototype, so it might look better or worse in the final version!

But for generated code, where we know everything about the type (like with [Union] and [Record] in the CodeGen system) then the creation of the reader/writer types should be automated.

Now, the real beauty of this is that one you're in the Meta type-system rather than the C# type-system, you're not limited to C#'s types and can therefore build more expressive types, do more complex projections, and generally go crazy. But once you're done you can then project back to a C# afterwards.

Imagine:

JSON text comes into API --> Convert to Meta type --> inputTy.IsSubTypeOf(expectedTy) --> Convert to C# type

This is a very simple example that shows a validation pipeline that validates before we ever get a C# concrete type. But there could be myriad intermediate stages, which leveraged: universals, existentials, unions, records, multiple type-systems, even multiple universes, etc.

For new source-generators, being able to convert .NET types into Meta types and then performing transformations on them using proper type-algebra will make it soooo much more powerful.

Finally, there is a scripting system. This is not necessarily expected to turn into some major language or anything - although I guess it could if it got some momentum. Really, it's to make the authoring of types slightly easier than using the API (for where that makes sense).

Here are some examples from my prototype (which actually all works, type-checks, and runs as expected):

// Either discriminated union type
Either l r = union {
    Left l,
    Right r
};
// Option discriminated union type
Option a = union {
    Some a,
    None
};
// Example of a union type (not discriminated)
OtherOpt a = Some a | None | Integer;
// Creates a concrete OtherOpt type 
OtherOptInt = OtherOpt Integer;
// Creates a concrete Option type
OptionInt = Option Integer;
// Is Option 1 a subtype of Option Integer? (True)
OptSub1 = Option 1 <: OptionInt;
// Is OtherOpt a subtype of Option a? (False)
OptSub2 = OtherOpt <: Option;
// Is Option a subtype of OtherOpt? (True)
OptSub3 = Option <: OtherOpt;
// Example of a higher-kinded type
Monad (m : * -> *) = {
    bind a b = m a -> (a -> m b) -> m b,
    pure a   = m a 
};
// Example of a higher-kinded type
Functor (f : * -> *) = {
   map a b = (a -> b) -> f a -> f b
};
// Create a concrete Option monad
MOption = Monad Option;
// Create a concrete Option functor
FOption = Functor Option;
// Create an Either monad
MEither l = Monad (Either l);
// Create an Either functor
FEither l = Functor (Either l);
// Create an Option 100
Foo = Some 100;
// Create an Option Integer
Bar = Some Integer;
// Is Foo a subtype of Bar (True)
Test1 = Foo <: Bar;
// Is Bar a subtype of Foo (False)
Test2 = Bar <: Foo;
// Higher-kinded sub-type checker
Sub (m : * -> *) x y = 
    m x <: m y;
// Is Option 100 a subtype of Option Integer (True)
Test3 = Sub Option 100 Integer;
// Is Option Integer a subtype of Option 100 (True)
Test4 = Sub Option Integer 100;
// Linked-list type
List a = union {
    Cons a (List a),
    Nil
};
// Union-type that can only be one of three values
Val = 1 | 2 | 3;
// Example of constructing an empty list
Items1 = Nil;
// Example of constructing a list
Items = Cons (1 : Val) (Cons (2 : Val) (Cons (3 : Val) Nil));

Much is missing from these examples, but already you can see that higher-kinds, records, unions, discriminated unions, values-as-types, and sub-typing are all there.

You could probably also build some of your types DSL nicely with Expression<T>. In my experience, Expression<T> is an under-appreciated gem in C#'s implementation of LINQ. It's what the DLR used to make DSLs and I've built some fascinating DSLs out of it myself, in past lives where I no longer have access to that code, for better and worse.

Spitballing on your first example, without time at the moment to build a working prototype, but you could get things out of:

Expression<Func<AType, SomeMetaType>> Some = a => a;
SomeMetaType None = new();
Expression<Func<AType, SomeMetaType>> OptionA = a => Some(a) | None;
// Something like this might even be possible with the right function definition
SomeMetaType Meta<T>(Expression<Func<in T, SomeMetaType>> expression) => /* … */;
var Some = Meta(a => a);
var None = Meta();
var OptionA = Meta(a => Some(a) | None);

All great questions, however I'm away from my machine for the weekend and typing a full reply on my phone would take ages! So I'll get back to you with a full reply next week.

But, in terms of not being that visible recently: that's mostly because I've been focusing on the v5-transducers branch (there's a huge amount of activity on there), and the Meta prototype which is private at the moment (will open it up soon).

Prototyping is just how I write code. I brute force every difficult coding problem by building, sometimes many, prototypes. Some might say that's inefficient and would instead advocate more upfront design, but most who know me know I have extraordinarily high output (quality and quantity).

In the case of the current roadmap, this is a junction of many big ideas that have been percolating in my mind for a long time. My comments about reducing complexity is entirely where my brain is going. And so when I prototype I am trying to make real these ideas in a way that has the lowest cognitive load (which is another reason why prototypes are good, you get too feel the complexity for real).

I really want to create a toolkit that everyone can use to build complex functional compositions. A bit like Haskell, but even more. Transducers can be thought of as the LEGO bricks of monads, functors, etc. Meta can be thought of as 'the truth' of your distributed application.

Marrying these ideas and making them easy to use is complex, hence why it's taking me a while to release some of this stuff.

My biggest worry is having to undo something, or fundamentally change an implementation (of say the Transducer type) not long after releasing it (with the breaking changes it entails).

So this one needs to be right.

It also needs to be right for my new startup, which builds on these ideas and takes them into the stratosphere. So really it will be a case of releasing once I feel comfortable everything is in a good place.

A follow up to your specific questions:

1. For now, I'm not sure I understand the future of current set of projects (Core, Parsec, Sys, etc.). Recently, you've introduced an upcoming update #1269. Does it mean that all these projects will be updated to v5 with keeping old features as an obsoleted and will be just referencing new LanguageExt.Transducers to gain new functionality?

LanguageExt.Transducers will just be a namespace within LanguageExt.Core.dll, so not a new library. I will be replacing the inner workings of Eff, Proxy (pipes), Reader, Writer, State, and RWS with transducers. These are all monads that are implemented internally with functions (rather than raw data, like Option, Either, etc.) - these naturally lend themselves to transducers.

Other function based monads (Try, TryAsync, TryOption, Aff) will be deprecated (made [Obsolete]) - as per the async proposal #1269.

Data-based monads (like Option, Either, etc.) will all support a .ToTransducer() natural-transformation. This will allow them to be lifted into compositions with other Transducer supporting monads.

Sequence based monads (Seq, Lst, IEnumerable, IAsyncEnumerable, IObservable) will also support a .ToTransducer() method and a function called many(...) that will allow their values to stream through an transducer composition. Meaning that everything that supports transducers (which will be everything) will also support synchronous and asynchronous streaming.

2. Do you have more info on what exactly will be dual-liscence prices and features? I understand it's too early, just want to know your initial thoughts.

LanguageExt.Meta and anything that depends on it will be dual-license, everything else will be MIT licensed (in the headings of the roadmap items you can see what's been marked as dual license).

3. Currently, I think I understand the approach of building and injecting a "runtime" you've described on wiki and in some other discussions. How difficult will be to integrate already existing libs and frameworks with new LanguageExt.Transducers, LanguageExt.Meta and others?

Transducers themselves have a small, fixed, runtime: TState. Which is just some state that is passed through the transducer pipeline that contains a CancellationToken, a SynchronizationContext, and a collection of resources for cleanup (disposables).

In terms of the other types, they will only have runtimes if that type needs a runtime, it's not a requirement of the Transducer system. And so, from that point, there's not much difference to how language-ext is now.

Meta itself doesn't need a runtime outside of its own internal workings (it's an entirely pure system).

However, one of the future benefits of Meta is that tooling can be built to auto-generate runtimes from selected traits and implementations. For example, you might specify: HasFile, HasEnvironment, HasConsole - with Live.FileIO, Live.EnvironmentIO, and Live.ConsoleIO. A source generator that uses Meta could read the types and then be used to project the runtime. Automating the process of generating runtimes completely. This has been on my mental wish-list for a while, but I don't want to over-promise too much right now.

Not that this couldn't be done with a regular source-generator that leverages Roslyn, but should be much easier with Meta.

4. Do you think some changes from MS team that may be made into language or platform may affect your big idea too hard? For example, recent reports about green-threads testing and possible upcoming changes in async machinery in the next .NET releases? For now, this looks more like "ideal" roadmap rather than real one, correct me if I'm wrong.

Green threads won't happen in .NET. Even if they did, it would just be something I could use internally with the Transducers (which is where my 'green thread' code is). It may reduce some overhead but unlikely to make much difference.

In terms of anything else they might do. That's just a bit of a "yeah, they might do anything" question - it's obviously impossible to know completely. My impression is that, from the language point-of-view, we're unlikely to ever see higher-kinds, we definitely aren't going to see dependent-types, and we almost certainly won't see refinement-types. These are all things I'm bringing into Meta.

Some things I hope we see:

• struct based delegates, so we can have lambdas that don't allocate
• Linear/affine types - which, again, will help reduce allocations

My old-skool games-programmer brain really wants to find ways to reduce allocations. The above two items would really help.

5. You've said that you've been prototyping quite a lot recently. It may be toooo long to explain why some approaches failed, but it still interesting: what specific restrictions does C# or .NET or any other part of this system impose when trying build such complex own one?

I mentioned in my previous answer why I prototype so much. In terms of restrictions in C#, well, a library like this is very much at the edge of C#'s capabilities, so I bump into a number of things that it can't do that I want it to do. It's mostly limitations of the type-system, but it can also be the fact that LINQ and lambdas allocate all the time - and because this is a library I want to use (and I want others to use) I have to take things like memory-allocations very seriously and try to find ways of mitigating them as much as possible.

On the whole though C# is excellent and the CLR is excellent - so this really is just trying to build on some exceptional foundations.

6. What do you think about F*, proof-oriented systems and dependent types in general? Do these have any chances to become more popular in the future, or it way too academic stuff just for researching?

Its maths. And maths works. And so having more rigourous and expressive type-systems will always improve the quality of the code that you write. It does come with more upfront effort by the developer and that can sometimes be a bit frustrating; but if your motivation is to write complex software that is stable and easy to manage then (I believe) it's the best way.

But, that isn't even the biggest issue: trying to find good, well supported libraries, for common things needed in business domains - like RDMBS support, or good cloud access libraries, etc. is hard. And if they don't exist you have to build them yourself, slowing everything down. I even noticed this when working on a Haskell project for a year - it felt like we spent 50% of our time trying to find suitable libraries to support the scope of the project (which granted was a complex project, but they all existed for C#, Java, etc.)

For any of these ecosystems to become popular it needs some good momentum and some dedicated individuals to build what already exists elsewhere, it's tough.

They also need to 'talk' to the 'professional engineer'. If I see one more Fibonacci example I think I'll scream! Haha! Real world examples for real world engineers.

That's why I think having something that can give you a lot of the benefits of a more 'academic' language, whilst still having access to the entire C# ecosystem could pay dividends. This is especially true at the 'edges' of your application, to stop bad data getting in, or leaving. But really, it's true for all code.

7. How will LanguageExt.Meta's type system differ from the concept of dependent types? Your explanation somehow reminds me these, but I'm not sure if it's the same thing.
For example, in F* there's a nat type that represents all natural numbers, so the solver-system uses its defenition to prove some other theorems. Will it be possible to do similar stuff with LanguageExt.Meta? From your other reply, I see the one way of doing so by manually creating discriminated union with all possible cases (e.g. Option a, some123).

Dependent-types are just types that depend on values, so from that point-of-view it's the same. I will also be supporting refinement-types which I think really is the killer feature. We're not going to be using Meta for proofs, but for making our systems more robust through better control of data in and out of our systems. I'm coming at this from the view that data in C# applications usually isn't constrained well enough, so I wanted to build something that understands (at runtime***) if something is correct or not. We do this all the time with if statements (refinements), this is an attempt at formalising that idea.

*** I'm saying "at runtime" because you can use the type-system directly in your applications; but also it can be used in source-generators - which makes them compilers (effectively). But the source-generators may also defer certain type-checking that C#'s type-system can't do by injecting Meta code directly into the generated code (into constructors for example).

I knew I wanted something that could represent any C# value - and so Meta has primitive-types that align with C#'s primitive types.

Therefore, I don't define nat as:

nat = Zero | Succ nat

Which is one difference.

But my system does support values as types. So, for example:

Foo = 1 | 2 | "fred";
IsFred (x : Foo) = {
    match(x)
    {
       1 => False,
       2 => False,
       "fred" => True
    };
}
Bar1 = IsFred 1;
Bar2 = IsFred "fred";

1, 2, and "fred" aren't values, they're singleton types. Foo is a type. IsFred is not a function, it's a generic-type that takes one type-argument and constructs a Bool type. You can think of this a bit like how Some constructs an Option.

If that all still seems confusing, here's what it might look like in hand-written C#:

record N1;
record N2;
record Fred;
abstract record Foo;
record Foo_Case1(N1 Value) : Foo;
record Foo_Case2(N2 Value) : Foo;
record Foo_Case3(Fred Value) : Foo;
record IsFred<X> where X : Foo
{
    public static readonly bool Result =
        typeof(X) == typeof(Foo_Case1) ? false  
      : typeof(X) == typeof(Foo_Case2) ? false  
      : typeof(X) == typeof(Foo_Case3) ? true
      : throw new NotSupportedException();      
}
record Bar1 : IsFred<Foo_Case1>;   // Bar.Result == false
record Bar2 : IsFred<Foo_Case3>;   // Bar.Result == true

One thing I've noticed in dependently-types languages is that they still have a distinction between values used in types and what we would classically think of types. I've not done that. Everything is a type, full stop. I don't know, yet, whether that will come back to bite me.

Another difference is that I have the concept of multiple type-systems, which can support morphisms between them. I think that's pretty unique to any language, but maybe there's something out there.

8. Currently, some countries can't use payment methods to transfer all over the world. I understand that's its not your problem and even maybe the percentage of language-ext users with such trouble is very low, but I just want to know will there be any chance to legally bypass these obstacles (if needed)?

Remember, no-one pays unless their turnover is at least $1,000,000 (proposed). If there's no way an organisation like that can transfer funds then they won't be able to use the dual-licensed features unfortunately. Being a 1 person operation doesn't really give me the bandwidth to look for solutions to that problem.

As always, thanks for the detailed response, I think I get your thoughts and mood about all of this.

If I see one more Fibonacci example I think I'll scream!

Couldn't agree more :D

Just a few simple questions I forgot to ask:

1. What do you think are approximate time frames (apart from the year) on these features? As I understood, these advanced projects (e.g. Meta and SourceGen) are far away from fully prototyping and implementation due its massive complexity, so it may take even more than one year. But what about Transducers, i.e. v5 (correct me if I'm wrong)?
2. What are your plans on your book and courses? Are they gonna replace current repo's wiki and discussions or it will be something more than guidelines on programming with language-ext?

1. What do you think are approximate time frames (apart from the year) on these features? As I understood, these advanced projects (e.g. Meta and SourceGen) are far away from fully prototyping and implementation due its massive complexity, so it may take even more than one year. But what about Transducers, i.e. v5 (correct me if I'm wrong)?

• Transducers and implementation of the no-async proposal will be Q1.
• Meta, at least in beta form, will be Q2
  • Features that depend on Meta (source-generators, serialisation): probably not long after, because I will need to build some of these to prove Meta is useful.

2. What are your plans on your book and courses? Are they gonna replace current repo's wiki and discussions or it will be something more than guidelines on programming with language-ext?

They won't ever replace the wiki or any other documentation. Documentation shouldn't be pay-walled. But more in-depth presentations of language-ext (that take significant portions of my time) will need to be paid for. A. because I need a motivation to do it, and B. because there are other things I could do with that time!

Also ready to pre-order it. Functional Programming in C# from Enrico Buonanno is great and lots of those concepts can then be applied when using Language-Ext but a proper book on Language-Ext would be a blessing. I always found your wiki articles well written and the theory well explained. Really looking forward to a book full of them :)

Thanks for the support 👍

I know that if I sit down to write a book it'll probably never happen. So, what I'm doing is building up articles on my blog - this has immediate value to users of the library.

I am hoping that I'll get to a point where I have enough content, so that I can then adapt it into a book.

This seems like an approach that's got a greater chance of succeeding and if it doesn't there's still plenty of content on the blog.