Archive for the 'Emily' Category

My Lambdaconf talk on the Emily programming language

Tuesday, March 1st, 2016

You might have seen my posts here about Emily, my hybrid functional/object-oriented programming language project. I actually gave a talk at Lambdaconf in May of 2015 about the project, and it is now available on Youtube. While the documents on the Emily website are intended to be introductions to the language for a general audience, the talk was about design, and was aimed at functional programmers and programming language enthusiasts. I’m pretty happy with how it turned out. Here it is if you’d like to watch it:

Announcing the Emily programming language, version 0.1

Sunday, January 18th, 2015

This last year I’ve been working on a programming language. Around April, I sat down and I wrote out a list of all the things I want out of a language. Nothing that exists right now felt satisfying, so I decided to make my own. I’ve named this language Emily, and as of today I have a finished “version 0.1” release that you can download and run. This version is very primitive in certain ways, but you can write a program in it and it points enough toward what the language can eventually be that I’m excited.

An example

Here’s a small program in Emily:

width = 80

foreach ^upto ^perform = {
    counter = 0
    while ^(counter < upto) ^( perform counter; counter = counter + 1; )

inherit ^class = [ parent = class ]

line = [                               # Object for one line of printout
    createFrom ^old = {
        foreach width ^at { # Rule 135
            final = width - 1
            here   = old at
            before = old ( at == 0 ? final : at - 1 )
            after  = old ( at == final ? 0 : at + 1 )
            this.append: ( here && before && after ) \
                     || !( here || before || after )
    print ^ = {
        this.each ^cell { print: cell ? "*" : " " }
        println ""                                          # Next line

repeatWith ^old = {  # Repeatedly print a line, then generate a new one
    do: old.print
    new = inherit line
    new.createFrom old
    repeatWith new

starting = inherit line        # Create a starting line full of garbage
next = 1
foreach width ^at (
    starting.append: at != next
    if (at == next) ^( next = next * 2 )
repeatWith starting                                             # Begin

This executes the 1d rule 135 cellular automata, or in other words, it prints strange pyramids:

*  * *** ******* *************** ******************************* ***************
      *   *****   *************   *****************************   **************
 ****   *  ***  *  ***********  *  ***************************  *  ************
  **  *     *       *********       *************************       **********
*       ***   *****  *******  *****  ***********************  *****  ********  *
  *****  *  *  ***    *****    ***    *********************    ***    ******
*  ***          *  **  ***  **  *  **  *******************  **  *  **  ****  ***
    *  ********         *               *****************               **    **
 **     ******  *******   *************  ***************  *************    **
    ***  ****    *****  *  ***********    *************    ***********  **    **
 **  *    **  **  ***       *********  **  ***********  **  *********      **
       **          *  *****  *******        *********        *******  ****    **
 *****    ********     ***    *****  ******  *******  ******  *****    **  **
  ***  **  ******  ***  *  **  ***    ****    *****    ****    ***  **        **
   *        ****    *           *  **  **  **  ***  **  **  **  *      ******
**   ******  **  **   *********                 *                 ****  ****  **
*  *  ****          *  *******  ***************   ***************  **    **    *
       **  ********     *****    *************  *  *************      **    **
******      ******  ***  ***  **  ***********       ***********  ****    **    *
*****  ****  ****    *    *        *********  *****  *********    **  **    **
 ***    **    **  **   **   ******  *******    ***    *******  **        **
  *  **    **        *    *  ****    *****  **  *  **  *****      ******    ****
        **    ******   **     **  **  ***               ***  ****  ****  **  **
*******    **  ****  *    ***          *  *************  *    **    **
 *****  **      **     **  *  ********     ***********     **    **    ********
  ***      ****    ***         ******  ***  *********  ***    **    **  ******
*  *  ****  **  **  *  *******  ****    *    *******    *  **    **      ****  *
       **               *****    **  **   **  *****  **       **    ****  **
******    *************  ***  **        *      ***      *****    **  **      ***
*****  **  ***********    *      ******   ****  *  ****  ***  **        ****  **
****        *********  **   ****  ****  *  **       **    *      ******  **    *
***  ******  *******      *  **    **         *****    **   ****  ****      **
 *    ****    *****  ****       **    *******  ***  **    *  **    **  ****
   **  **  **  ***    **  *****    **  *****    *      **       **      **  ****

Anyway, even without knowing the syntax, you might notice a few things looking at this program:

  • This language is extremely extensible. Near the beginning, I realized there were two things that I didn’t get a chance to put in the standard library: a foreach function, and a way to instantiate an object of a class. So I just implemented them in the program. Very simple.
  • Support for both [“unpure”] functional and object-oriented styles: Functions are being passed as values all over the place, functions are being created offhandedly (that’s the ^), objects with prototype inheritance are created very casually (that’s []), tail recursion works.
  • Clean, familiar syntax: At least, assuming you’ve been writing a lot of Python, JS, or Lua, this looks a lot like code you’ve written before, other than the ^s.

Why’s Emily special?: For language geeks

The one central New Idea in Emily is that everything is a function, or maybe put another way, objects and functions are interchangeable. You can think of an “object”, or a structure, as being a function that maps key names to values; Emily actually treats objects this way. If an object can be a function, then anything can be; 3.add can be the function that adds 3 to another number, 3 can be the function that maps .add to 3.add.

The important thing about this isn’t that functions specifically are important; what turns out to be important is simply that everything in Emily is the same thing. Everything acts like a unary (one-argument) function, so where other languages might have several “verbs” (field lookup, variable definition, function definition, function application, arithmetic) Emily has exactly one verb (function application). All you’re doing is applying functions in a particular order, and writing code just means plugging those functions together like pipes. Even the “conventional” syntax elements, like + or the . lookups, are just unary function applications in disguise. Even declaring a variable is done with a function call.

So why does this matter?

Why’s Emily special?: The practical side

I like to build abstractions in my code. If I’m writing a C program, and I find I’m writing for(int c = 0; c < upto; c++) {something} over and over, I get annoyed; I wish I could just define a “foreach upto {something}”, like I did in the pyramid program up top.

I like to mix different kinds of tools when I write software. I really like Lua, but Lua isn’t very good for certain kinds of binary manipulation and threading, so I write programs that are part C++ and part Lua. But this is tricky; the languages each have their own complexities, and the complexities clash. Languages don’t work well with each other.

I assert both these things get a lot easier if you have a language whose underlying model is very simple.

In Emily, where everything is function applications, building complex abstractions just means plugging the applications together in a particular order. The language itself doesn’t need to be this simple– again, if you look above, a lot of stuff is happening in the code that doesn’t look like a function application. But that complexity is something built on top of the language, rather than being a fundamental part of the model. This means it can be extended further, and it also means it can be replaced.

Syntax like + or =, for example, is actually performing macro transformations– 3 + 5 * 4 gets rewritten into .times 4). This makes these operators easy to extend– if you want to design an object that “acts like” a number, you just define an object that implements the .plus and .times methods. It also makes them possible to replace– the transformations are just little programs, and in a later version of Emily you’ll be able to define your own operator transformations, if there’s a different syntax you’d like better.

That’s not all that impressive, though– operator overloading is a pretty standard feature in languages, and macros are not the way you’d prefer to implement abstractions. Rather the more basic element in Emily of everything is function application is what opens up the really powerful possibilities. Let’s try something a little more unusual. As mentioned above, Emily has prototype inheritance. But it’s single inheritance– only one parent per object. What if you’d prefer multiple inheritance? Well, you can implement it yourself, by writing a single function:

# A function that returns a function-- it generates the fake "parent"
dualInherit ^parent1 ^parent2 = ^key {
    thisUpdate this: \
        (parent1.has key ? parent1 : parent2) key

hasAddOne = [
    addOne ^ = this.value = this.value + 1
hasAddTwo = [
    addTwo ^ = this.value = this.value + 2

child = [
    parent = dualInherit hasAddOne hasAddTwo
    value  = 0

do: child.addOne
do: child.addTwo
println: child.value

So what’s happening here? The object that child inherits from is chosen by setting the parent key. But objects and functions in Emily are interchangeable. So child inherits from a function. dualInherit takes two desired parents and manufactures a function which takes a key and executes it on whichever of the two parents knows how to respond. child then uses dualInherit to inherit from both the classes hasAddOne (from which it inherits the method addOne) and hasAddTwo (from which it inherits the method `addTwo).

What else?

There’s a couple more interesting things that made it into the language, even as early as this version is:

  • \version 0.1
    This feature is small, but I believes it solves a somewhat fundamental problem with programming languages. Each Emily program is encouraged to start with a line identifying the language version it was developed against. When an Emily interpreter– current or future– encounters the \version line, it makes a decision about whether that code can be run or not. If the hosting interpreter is backward-compatible with the code’s version, it just runs. But if backward-incompatible changes have been made to the language since then, it will either enter a compatibility mode or politely refuse to run. At some point, it will be possible to install these compatibility modes as pluggable modules.

    I write a lot of Python, and a huge running problem is compatibility between versions. In Python, as in most programming languages, the implementation version is the same as the language version. Python 2.4 runs Python 2.4, Python 2.7 runs Python 2.7, Python 3.1 runs Python 3.1, etc. Meanwhile Python 2.7 can run Python 2.4 code, but Python 3.1 can’t run Python 2.7 code, which means Python is competing with itself and nobody uses Python 3 because all the code’s written for 2.7. (And even before the big 3.0 switch, forward compatibility created a huge problem all by itself: If you had a program that used a feature from 2.5, but what you had installed was 2.4, you wouldn’t know it until you tried to run it and something would break strangely, possibly at runtime.)

    This is all silly! Language versions define interfaces, and interpreters are engines. We shouldn’t be holding back on upgrading our engines because the interface changed (and if there’s some reason a new engine can’t handle the old interface, it should at least fail very early). It’s generally possible at least in principle to convert between these interfaces, so it should be possible to install something that does conversion for an incompatible past interface (probably even a future one!). It should be possible to mix code written against different interfaces in the same program– maybe even the same file. There’s surely a point at which this becomes untenable (library cross compatibility probably gets awkward quick), but language implementors not being able to get updates adopted because nobody wants to lose back compatibility with 15-year-old versions doesn’t sound very tenable either.

    Anyway, for now: Just tag each file with its version, and all this becomes a lot easier to sort out later.

  • Proper Unicode support

    This really ought to be something we just expect of a language these days, but Emily is being developed for full Unicode support and the interpreter treats source files as UTF-8. Right now this only extends as far as handling Unicode whitespace– well, and the macro system supports unicode symbols, but since you can’t make macros yet that’s not so useful. As soon as possible though my goal is to implement UAX #31 so you can use Unicode in identifiers, and I’m hoping to work with coders fluent in non-latin-script languages to make sure Emily is usable for those purposes.

    Oh, also: At some point I’m just gonna make smart quotes work as quotes. If we’re gonna keep pasting them in to our code by accident, they might as well work.

  • Return continuations

    I’m… not sure I should be calling too much attention to this, but it is a bit unique. As I’ve said, everything in Emily is a function, at least in form. There are no “keywords” as syntax constructs– any “syntax” is just shorthand for function calls. So if I’m going to implement, say, return, it has to be something that can be treated like a function. return in Emily winds up being what’s called a “continuation”– a function-like object that when called just jumps to a particular place, in this case the end of the method call.

    This has an interesting side effect that I had not entirely intended:

    timeMachine ^ = (           # Function takes no arguments
        goBackward = return     # Store return in a variable. Yes, really.
        return 1                # Return. This line only runs once.
    counter = do timeMachine    # The return value of "do TimeMachine" is 1, right?
    println: counter            # Only the first time-- every time we call goBackward,
    goBackward: counter + 1     # we return "counter + 1" from timeMachine,
                                # even though timeMachine already ended.

    …it turns out “return” for a particular function call can be treated like any other value, and even outlive the function call it was born in. As with any other kind of continuation, this creates the opportunity for some very powerful constructs (I’m trying to work out how I can implement Java-style named breaks with it) and also the opportunity for some really bad ideas. I’m going to see if I can find a way to encourage the former while maybe putting some limits on the latter.

What next?

What you see here is part of a more ambitious set of ideas (link goes to my original design writeup before I started writing any code). Here’s some things I want in Emily eventually; some of this may not be completely feasible, but I think even making it part of the way there would be great.

  • Types

    There’s a lot of things I want to try with typing in Emily, and I’m nervous about elaborating too far on what they are before I know for sure what I can pull off. But in general: Emily should have type annotations. Typing should be “gradual”; you should be able to have a part of your program with type annotations and a part without. There should be a prover for types, and an inference engine. If you assign something to a variable or argument which has a type annotation, that should mean a runtime check for correctness. There should be a “compile time” check also; the compile time check should be able to tell some calls are definitely type-correct and optimize out the correctness check, and it should be able to tell some calls are definitely type-incorrect and refuse to run the script until they’re fixed. (Right now, nothing stops you from typing 3(4) except that this will fail at runtime).

    This is all pretty standard for a language written in the last ten years. But I want to try some odder things, based around Emily’s core ideas that everything in the language is “the same thing”, interchangeable with functions, everything is extensible and there is no syntax “magic”. So: Types should be constructable at runtime. It should be possible to use a function as a type; there should be a syntax for turning x such x > 4 into a type. Some types should just be language-level assertions, like there should be a type for “this variable’s value is known at runtime” or “executing this function has no side-effects”. There should be an ability along the lines of assigning a type as a key for an object– something like [ .name="minusone"; ^x of int = x - 1 ] for a function that returns "minusone" when applied on .name or subtracts 1 when called with an int. In other words, I want average objects to be able to act like match or pattern-matching guards from a functional programming language. (This would mean that you could build up guards that inherit from other guards, which I kind of like.)

  • C++

    I need to be able to interoperate with other languages. I write video games, so there’s a whole bunch of C++ libraries I really need access to. Interoperability with C++ is hard. Interoperability with anything is hard. As alluded above I used to write a lot of Lua, and the interaction layer with C++ was just incredibly complicated and involved generating code with a Python script.

    Complicated or no, I’m going to have to write an interop layer for Emily. But I think that there are some design choices in Emily that are going to make this easier. Because Emily is so dynamic, I can generate a bunch of stuff at runtime that otherwise would have required a code generator, and Emily’s everything-is-functions rule means that wrappers can be much less awkward than they might in another language. Objects and classes and methods from different languages all have different semantics; if you have an object in one language which actually wraps a guest object from another language, you’re going to wind up with weird mismatches where the objects and methods you’re interacting with in the host language don’t quite behave like their analogues in the guest language (the “complexity clash” I mentioned earlier). But an object in Emily is a smooth ball; its semantics are just a matter of what arguments you feed it. If the type system is smart, it could even verify that the arguments you feed the wrappers are always consistent with the semantics of the guest object.

  • Reader macros

    Emily has macros that transform the AST into a different AST before it is executed. A lot of functional languages have the concept of a “reader macro”, which generates AST from whole cloth. There’s a piece of implied code called the “reader” which is consuming the text of your program and parsing it into an AST; \ will eventually be a way of sending messages to this reader, possibly enabling transformations that a normal macro couldn’t perform because it would need access to the program source as a string. Right now the only reader instruction is \version, and the only thing it can do is halt the program if the version is wrong. There will be more reader instructions in future (for example, \op will probably be the way to define the normal, non-reader macros).

    What really interests me though is the possibility of a \reader, which would cause the reader to just plain get out of the way and hand off parsing the rest of the file to a piece of Emily code. My daydream here is that if you could write a \reader that parses some completely other programming language and emits Emily ASTs (hopefully easy since Emily really only barely has an AST, the AST is just a tree of applications)– and if I’m right that it will be unusually easy to write Emily code that accepts other languages’ objects as guests– then Emily could become a useful intermediate layer that knows how to translate between several different languages and mediate their differences.

These are all Big Ideas though, and the ideas the language already has could stand some cleanup, so for 0.2 I’m going to be focused on basic functionality improvements (less confusing scoping, operators on strings, short-circuiting booleans, user-defined operators, unicode, package loading, IO).

Downloading and running Emily

As mentioned, Emily is available from a BitBucket page (or, if you can’t use Mercurial, its GitHub mirror), but not yet in any other form. You will need to compile it yourself; it’s written in Objective Caml, so you’ll need to install that first. For instructions, see By the time you read this, there may also be install packages; see

What’s “BitBucket”?

If you’re not familiar with BitBucket or Github, go to the Downloads page on BitBucket, click the “Branches” tab, and to the right of the word “stable” click “zip”. This will download a zip file of the most recent stable release.

Can’t I just download something?

If you happen to use “homebrew” for Mac OS X, Misty De Meo has helpfully put together a homebrew package. You can install Emily via brew by running:

brew install mistydemeo/emily/emily

Hopefully by the next release we’ll have Debian packages too!

Getting started

If you’re interested in Emily and want to give it a shot, a good thing to read next would be the tutorial. If you just want to know more, there’s a bunch of documentation including some design documents and information about modding the language in the docs folder.


Emily is MIT licensed, meaning it is open source and only attribution for the authors is required. The list of contributors is here.

Emily programming language: Status update one

Wednesday, June 4th, 2014

About a month ago, I put up some plans for a programming language. It actually got some interest!, so I thought I’d go over some things that have happened since then.

First off, I decided the name of the language is definitely Emily, unless I’m somehow legally forced to call it something else. I made a website for it and an announcement-feed Twitter, at and @emilylanguage, respectively.

Second off, I now have a prototype implementation! It’s a very, very minimal implementation– you seriously couldn’t develop anything in it, and it’s missing most of the language’s better planned elements– but it does demonstrate a couple of the language’s striking features, and it’s turing-complete. (And the turing completeness part can be demonstrated in an amusing way: It turns out to be possible to embed Unlambda programs in it.)

Programs written in the language don’t look very pretty right now. Most of the “operators” in Emily, like = or +, are eventually intended to be defined as macros, and those macros aren’t implemented in the prototype yet, so setting a variable looks a little funny:

# Prints "3.0"
set .test 3
print test

And adding looks a little funny:

# Prints "7.0"
print (3 .plus 4)

In the final version of Emily, writing “3 + 4” will do the same thing as writing “3 .plus 4”– the only difference will be that + has operator precedence, and .plus does not.

There’s one macro that *is* in already– ^, or “lambda”, which turns whatever comes right after it into a function:

# Prints "4.0"
^x( print x ) 4

This defines an “anonymous function”– if you aren’t familiar with that idea, it’s a function that doesn’t have a name but just shows up in an expression, like a literal– that takes one argument and prints it. It then feeds the anonymous function the argument “4”.

With these couple things explained, here’s a more complicated Emily program that works in the current interpreter:

set .countup ^arg {
    set .count (arg.from)
    loop ^(
        print count
        print "\n"
        set .count ( count .plus (arg.step) ) (

countup[ set .from 10; set .to 20; set .step 2 ]

Again, not very pretty, but this demonstrates a couple of interesting things that Emily tries to do.

To recap my last blog post a bit, everything in Emily is formally a function– everything. Objects are functions pretending to be objects, functions which take key names (in the form of strings like .plus— assume this is just a funny way of saying “plus”) and map them to values. Numbers are objects, which are functions. Scopes are objects (which are functions); the language sets up and shuffles around the scope functions silently, and it forwards any bare symbols it finds (like “set” or “count” in the examples) to the current scope function as a string argument. A lot of the time these various functions don’t really *act* like functions, of the kind we normally recognize (for this reason I’ve been calling them “blocks”, after Smalltalk) but the fact the language doesn’t distinguish the different kinds of functions semantically opens up a lot of possibilities for useful abstraction. The only “verb” in Emily is “apply”– apply function A to argument B. Everything else is just built on top of that.

Not a lot happens in this example, but a couple things do stand out: assigning a variable value just means invoking the “set” method on the current scope (i.e., passing .set to the current scope block, doing which returns a function that can be used to alter that scope block’s mappings). There are no special flow control constructs– “loop” is just a function that takes a closure as argument and executes it until it returns false. The thing that really stands out to me here though are the appearance of {} and []– the countup language body is wrapped in a {} to give it its own variable scope, and when we invoke countup the argument list is written as an “object literal”, denoted by []. What’s interesting here is how these two things are implemented by the language.

Parenthesis in Emily create “groups”– I have to call them that because I used the word “block” for something else already, maybe a mistake. Any parenthesis in Emily can contain multiple statements, separated by semicolons. A trick here is that group (), scoped group {}, and object literal [] are all the same thing, distinguished only by what the scope is within the group and what is returned at the end. More interestingly, there’s no “magic” to the group operators– effectively, what defines the different group types are a couple of lines of setup/teardown code that run at the beginning and end of the group, and that setup/teardown could in principle be written in Emily. In other words, {} and [] could in the final version just be implemented as macros! In the current implementation, all three group types are handled by the same code in the interpreter, with the difference being:

    (Assume any time a statement executes, its evaluated value is assigned to a variable named “last”)
    Plain groups: Set [scope] to the enclosing scope; after final statement return “last”.
    Scoped groups: Create a block [newscope], which prototype-inherits from the enclosing scope, and set [scope] to it. After final statement return “last”.
    Object literals: Create a block [object]. Create a block [newscope], which prototype-inherits from the enclosing scope. Assign [newscope].set = [object].set and [newscope].this = [object]. After final statement, return [object].

Did I get too language wonky there? The point I’m trying to make here is that “an object literal syntax” is a really basic thing to a language, but this language kind of doesn’t have one, because it doesn’t need one. That’s just normal code running inside of the [], and the thing that makes [] “special”– the setup/teardown– could have been implemented by an end user. This is exciting to me because it implies end users can implement other language extensions, things as complicated as an object literal syntax, on their own.

The implementation

The prototype implementation of Emily is hosted on BitBucket, although it will not do you a lot of good right now. The main problem is that it is written in Objective-C, which means that it will only easily build if you have OS X and XCode. Compiling on other operating systems is possible but will require setting up something like GNUStep. I haven’t investigated this myself, mostly because I intend to port to something-other-than-Objective-C rather than bother with GNUStep. If you do have a mac, or do have GNUStep set up, you’ll probably find it pretty easy to set up and extend. The current implementation is only about five or six source files. (Sorry if the BitBucket page is a little confusing, they just changed their layout and now it’s all weird– the cloud with the arrow in it is the “download source tarball” button.)

Remember, this is literally the very first thing I got to run at all– as of this exact writing there’s no code documentation, no documentation of the exact language as implemented so far (the closest is the sample code files), and there are known memory leaks. I’m not even giving this a version number yet. Still, it’s a start! If you want to try actually using or extending it, feel free to contact me and I’ll help you get set up.

I designed a programming language and it looks like this.

Tuesday, April 22nd, 2014

For reasons I talk about here, I’m going to try to create a programming language. So far I’ve got basically a first-draft design.

There is a specific idea I decided I want to explore, when I do this: Programming languages duplicate too much. Programming languages often have multiple syntaxes that do very similar things, or multiple underlying concepts that do very similar things. It is sometimes possible to collapse these similar things into one thing, and when we do, I usually like the results better.

For example, many languages (Perl, Python, PHP) have both a dictionary type and an object type, but the two are used in effectively the same way; on the other hand Lua collapses dictionaries and objects into one type (tables), and makes an object field lookup identical to a dictionary string lookup. Or most object oriented languages distinguish objects and classes, but prototype based languages show that you can get by with just objects; if objects can inherit from other objects, then a “class” is just a pattern for a particular kind of object. When you collapse ideas together like this, or build language features on top of existing features rather than adding new primitives, you reduce both the amount of mental overhead in thinking about code implementation and also the amount of redundant syntax. There’s usually a lot of redundant syntax. C++ uses . to access a field from a reference, and -> to access a field from a pointer. Most languages use (x) to indicate an argument to a function, and [x] to indicate an index to an array. Why? If pointers and references were just special cases of one underlying concept, or if arrays and functions were, you could use one syntax for each pair and you wouldn’t have to mentally track what each variable is, you wouldn’t have to do all the obnoxious manual refactoring when you suddenly decide to replace a reference with a pointer somewhere or vice versa.

In the language I’ve been thinking about, I started with Lua’s “Table” idea– what if i built objects out of dictionaries?– and decided to take it one step further, and build both objects and dictionaries out of functions. In this language, there’s one underlying data structure that functions, objects, dictionaries, and some other stuff besides are just special cases of– design patterns of.

Taking a cue from Smalltalk, I’m going to call this underlying structure “blocks”.

Blocks are just functions

A block, for purposes of this blog post, is a unary function. It takes exactly one argument, and it returns a value. Anywhere in this blog post I say “blocks”, I could have just written “functions”. I’m going to mostly use the “block” jargon instead of saying “functions” because some blocks will be “used like” functions and some very much will not be.

In my language, you’ll define a function just with “=”:

    addOne ^x = x + 1

The ^x is an argument binding. The ^ signals to the language that the right side of the = needs to be a function body (a closure). If on the next line you just said

    y = addOne 3

That would just assign 4 to the variable “y”, it would not create a function.

Blocks are pattern-matched functions

A big part of this project is going to be that I really like the ideas in functional languages like ML or Haskell, but I don’t actually enjoy *writing* in those languages. I like OO. I want a language that gives me the freedom and expressiveness of FP, but comfortably lets me code in the OO style I use in Python or Lua or C++. So I’m going to steal as many ideas from FP languages as I can. Three really important ideas I’m going to steal are closures, currying, and pattern matching.

In case you don’t know those languages, let me stop and explain pattern matching real quick. You know how C++ lets you function overload?

    // In C++
    void addOneHour(int &k) { k = (k + 1) % 12; }
    void addOneHour(float &k) { k = fod(k + 1.0, 12); }

Well, pattern matching is as if you could switch not just on type, but also on value:

    // In hypothetical-C++
    void addOneAbsolute(int &k where k > 0) { k = k + 1; }
    void addOneAbsolute(int &k where k < 0) { k = k - 1; } void addOneAbsolute(0) { } // Do nothing

That last line– the one demonstrating we could write a function whose pattern matches only *one single value*– is going to be important to this language. Why?

Blocks are dictionaries

In my language, if I want to assign more than one pattern to a single block, I just use = multiple times:

    factorial ^x = x * factorial (x - 1)
    factorial 0 = 1

“Factorial” is a block. The way I’m looking at it, a block is just a data structure which maps patterns to closures. It’s like a dictionary, but some of the keys (the ones with bound variables) match multiple values.

However we could not bother assigning any bound-variable patterns, and then we’d just have a dictionary or an array:

    nameOfMonth 1 = "January"
    nameOfMonth 2 = "February"
    nameOfMonth 3 = "March"

Blocks are objects

Here I want to introduce a data type called an “atom”. This is an idea stolen from Erlang (and possibly Ruby?). Technically an atom is an “interned string”. It’s something that the programmer sees as a string, but the compiler sees as an integer (or a pointer, or something which has a constant-time comparison). You get at the atom by putting a . before a symbol; the symbol is the name of the atom:

    x = .atomname

It’s cheaper to compare atoms than strings (.atomname == .atomname is cheaper than “atomname” == “atomname”) and cheaper to use them as dictionary lookup keys. This means atoms work well as keys for fields of an object. Objective-C for example actually uses atoms as the lookup keys for its method names, although it calls them “selectors”. In my language, this looks like:

    constants.pi = 3.14
    constants.e = 2.71
    constants.phi = 1.61

Notice this looks like normal object syntax from any number of languages. But formally, what we’re doing is adding matching patterns to a function. What’s cool about that is it means we’ll eventually be able to use machinery designed for functions, on objects. Like to skip ahead a bit, eventually we’ll be able to do something like

    map constants [.pi, .e, .phi]

and this will evaluate to an array [3.14, 2.71, 1.61].

What’s up with the square brackets? Oh, right. well, I think all this “constants.” nonsense is gonna get kinda tiresome. So let’s say there’s a syntax like:

    constants = [ pi = 3.14, e = 2.71, phi = 1.61 ]

Notice I say “pi” and not “.pi”– on the left side of an =, the initial “.” is implicit. More on that in a moment.

One other thing. Inside of the [ ], there exists an implicit “this” variable, corresponding to the object the [ ] creates. so if you say

    counter = [
        count = 0
        increment ^x = { this.count = this.count + x }
        decrement ^x = { this.count = this.count - x }
    counter.increment 1
    counter.increment 3

Then at the end of this string of code “counter.count” is equal to four.

Blocks are prototypes

What if we want more than one counter object? Well, you’ll notice an interesting consequence of our pattern matching concept. Let’s say I said:

    counter = [
        init ^x = { this.count = x }
        increment ^x = { this.count = this.count + x }
        decrement ^x = { this.count = this.count - x }

    counter_instance ^x = counter x
    counter_instance.init 3
    counter_instance.increment 5

When we say “counter_instance.whatever”, the language interprets this as calling the block counter_instance with the argument .whatever. So if counter_instance is defined to just re-call “counter”, then on the next line saying “counter_instance.init 3” will fetch the block stored in counter.init, and then that block gets called with the argument 3. The way the “this” binding works is special, such that counter_instance.init gets invoked “on” counter_instance– “this” is equal to counter_instance, not counter.

The syntax we used to make counter_instance “inherit” is pretty ugly, so let’s come up with a better one:

    counter_instance.ditch = counter

I haven’t explained much about how = works, but when we say “counter_instance ^x = “, what we’re really doing is taking a closure with an argument binding and adding it to counter_instance’s implementation-internal key-value store, with the key being a pattern object that matches “anything”. “.ditch” is a shortcut for that one match-anything key slot. In other words, by setting counter_instance.ditch to counter, we are saying that counter is counter_instance’s “prototype”.

Something to make clear here: the lines inside a [ ] aren’t “magic”, like C++ inside a struct declaration or anything. They’re just normal lines of code, like you’d find inside a { }. The difference is the insides of [ ] are using a specially prepared scope with access to a “this” and a “super”, and at the end of the [ ] the [ ] expression returns the scope into which all these values are being assigned (“this”). The upshot is you could easily have the first line of your [ ] be something like an “inherit counter;” call that sets the ditch and does some various other fix-up to make this prototype system act more like some other kind of object system, like a class system (I like classes). This sort of thing is possible because

Blocks are scopes

Like most languages, this one has a chain of scopes. You’ll notice above I offhandedly use both ( ) and { } ; these are the same thing, in that they’re a series of statements which evaluate to the value of the final statement:

    x = ( 1; 2; 3 )

…sets x equal to 3. The 1; and 2; are noops. (Semicolon is equivalent, in the examples I’ve given here, to line-ending. There’s also a comma which is effectively a semicolon but different and the difference is not worth explaining right now.)

The one difference between { } and ( ) is that { } places its values into a new scope. What is a scope? A scope is a block. When you say

    a = 4

The unbound variable a is atom-ized, and fed into the current scope block. In other words “a” by itself translates to “scope.a”. When you create a new inner scope, say by using { }, a new scope block is created, and its ditch is set to the block for the enclosing scope. The scope hierarchy literally uses the same mechanism as the prototype chain.

Block constituents are properties (or: blocks are assignment statements)

Non-language geeks may want to skip this section.

I’ve been pretty vague about what = does, and that’s because it has to do several layers of things (matching items that already exist, binding variables, wrapping closures, and actually performing assignment). However, ultimately = must write [pattern, closure] pairs into one or more blocks. = cannot, however, actually write anything by itself. Ultimately, when = decides it needs to assign something, it is calling a “set” method.

    a = 4

Is ultimately equivalent to

    scope.set .a 4

That = is sugar for .set is a small detail, but it has some neat consequences. For one thing, since everything that happens in this language is curryable, it means you can trivially make a function:

    a_mutator = set.a

…which when called will reassign the “a” variable within this current scope (remember, “set” by itself will just be “scope.set”). For another thing, this means you can create a “property” for a particular variable:

    set.a ^x = ( b = x + 1 )
    a = 3

After this code runs, “b” will be equal to 4 (and “a” will still be equal to a function that mutates “b”).

The existence of .set will also have some interesting effects once we have types and therefore a type checker. I’ve been kinda vague about whether = has “set” or “let” semantics– that is, if you assign to a variable does it auto-instantiate or must you predeclare it, if there is a variable by the assigned name in the ditch does assignment shadow in the assigned block or reassign in the parent block, etc. And the answer is it doesn’t much matter for purposes of this post, because any of the possible things that happen when you set a field (“not declared” error thrown, assigned to top-level block, assigned to a parent block) could just be all things that could and do happen in different blocks, depending on what that block’s .set is set to. For example, it would probably make sense for object blocks and scope blocks to have a different last-ditch “.set” behavior, or be sensible to allow different source files to have different “.set”s for their file-level scopes (“use strict”).

On that note, let’s talk about types. There’s a lot of very exciting stuff happening in the study of types in programming languages right now, both types as used in languages and types as used in extra-lingual static analysis tools. I don’t understand a lot of this research yet (and I want to learn) but I think I understand enough to have an idea of what’s possible with types right now, and that means I know how I want types in this language to work.

Blocks are types

Let’s say we have a syntax variable:type that we can use to constrain the arguments of a function.

    factorial ^x : int = x - 1

When this function is called, there will be a runtime check, if “x” is not an int, it will be a runtime failure. Let’s say we can use the a:b construct inside expressions too:

    square ^x = ( x * x:float ) :: stateless

Let’s say that :: instead of : indicates that the type is being applied not to the value returned by that parenthesis, but to the implicit “function” defined by the parenthesis itself. “stateless” is a type that applies to functions; if we assert a function is “stateless” we assert that it has no side-effects, and its resulting value depends only on its inputs. (In other words, it is what in another language might be called “pure”.)

There’s some kind of a inferred typing system in place. There’s a compile time type checker, and when it looks at that “square” function it can tell that since “x” is a float in one place inside the expression, that the “x” passed into square must itself be a float. It can also tell that since the only code executed in “square ^x” is stateless, that the function “square ^x” is also stateless. Actually the “stateless” is from the checker’s perspective unnecessary, since if the checker has enough information about x to know the * in (x * x) is a stateless operation– which, if it knows x is a float, it does know that– then square ^x would be stateless anyway.

There’s some kind of a gradual typing system in place. There is a compile-time step which, everywhere square ^x is called, tries to do some kind of a type-proving step and determine if the argument to square is a float. If it can prove the argument is a float, it actually omits the runtime check to save performance. If it *can’t* prove the argument is a float, or it can prove the argument *isn’t* a float, it adds the check and maybe prints some kind of compile-time warning. (To stress: some of these properties, like “stateless”, might be in many cases *impossible* to prove, in which case the checker is conservative and treats “can’t prove” as a failure.) Besides omitting safety checks, there are some other important kinds of optimizations that the type checker might be able to enable. Critically, and this will become important in a moment, if a function is stateless then it can be potentially executed at runtime.

So what are types? Well, they’re just functions. “int” and “stateless” are language-builtin functions that return true if their arguments are an int, or a provably stateless function, respectively. (For purposes of a type function, if the type *doesn’t* match, then either a runtime failure or a return false are okay.) Types are values, so you can construct new ones by combining them. Let’s say that this language has the || and && short-circuit boolean operators familiar from other languages, but it also has & and | which are “function booleans”– higher level functions, essentially, such that a | b returns a function f(x) which is true if either a(x) or b(x) is true. So if “stateless” and “nogc” are two of the builtin type functions, then we can say:

    inlineable = stateless | nogc

And if we want to define a totally unique type? Well, you just define a function:

    positive ^x = x > 0
    sqrt ^x : positive = x / x    # Note: There might be a bug here

Obviously you can’t use just any function here– there would have to be some specific type condition (probably something like the “inlineable” I describe above) that any function used as a type in a pattern would be required to conform to. This condition would begin and end with “whatever the type checker can efficiently prove to apply or not at compile-time”.

Let’s finally say there’s some sugar for letting you define these “type condition” functions at the same time you define the function to whose parameters they apply; we could reduce that last block down to

    sqrt (^x >= 0) = x / 2    # Square root implementation, WIP 2

One other bit of sugar that having a type system makes easy:

Blocks are argument lists

So everything so far has been a unary function, right? There’s only so much we can do with those. This language is set up for currying– that’s how method lookup works, after all– and I would like to offer explicit sugar for curry:

    curryadd ^x ^y = x + y

But ehh, I don’t actually like using currying for everything. I like argument lists. And I really, *really* like named arguments, like Python uses. Let’s say we have this syntax:

    divide [^numerator, ^denominator = 1] = numerator / denominator

The “parameters” block there? Is totally just a block. But there’s some kind of block wiring such that:

    divide [4, 2]           # Evaluates to 2
    divide [4]              # Evaluates to 4-- "denominator" has a default argument
    divide [9, denominator=3]                       # Evaluates to 3
    divide [denominator = 4, numerator = 16]        # Evaluates to 4
    divide [ ]       # Compile-time error -- assignment for "numerator" not matched

There’s some sort of block “matching” mechanism such that if the argument block can be wired to the parameter block, it will be. I don’t have an exact description handy of how the wiring works, but as long as blocks remember the order in which their (key, value) pairs are assigned, and as long as they can store (key, value) pairs where exactly one of key and value is (no value), then such a matching mechanism is at least possible.

My expectation is that almost all functions in this language will use the argument blocks for their parameters, and almost all invocations will have an argument block attached.

Blocks are macros

I wanna go back here and look at something closer: We’ve defined that there’s some subset of this language which can be run at compile time, and that the type checker can identify which functions are in that category. I think this is a pretty powerful concept, because it means the language can use *itself* as its macro language.

So far in this post, you’ve seen three main kinds of syntax in the code samples: Unary function application (again, a field lookup like a.b.c is really just a bunch of currying), “=”, and little extra operators like “+”. What I’m going to assert is that the extra operators– and also maybe =, and maybe even [ ]– are actually just rewrite rules. So for the line:

    3 + square 4

Before actually being executed, this line is transformed into

    3 .plus ( scope .square 4 )

“3”, like anything else, is a block. Like in Io or Self, adding three to four is just invoking a particular method on the 3 object. In this language “+”, the symbol, is just a shortcut for .plus, with parser rules to control grouping and precedence. (If we actually just wrote “3 .plus square 4”, then the currying would try to interpret this as “(3 .plus square) 4”, which is not what we want.)

There’s some kind of a syntax for defining line-rewrite rules, something like:

    op [ symbol = "!", precedence = 6, replace = .not, insert = .unary_postfix, group = .right_inclusive ]
    op [ symbol = "*", precedence = 5, replace = .times, insert = .infix, group = .both ]
    op [ symbol = "+", precedence = 4, replace = .plus, insert = .infix, group = .both ]
    op [ symbol = "==", precedence = 3, replace = .eq, insert = .infix, group = .both ]
    op [ symbol = "&&", precedence = 2, replace = .and, insert = .infix, group = .both ]
    op [ symbol = "||", precedence = 1, replace = .or, insert = .infix, group = .both ]

Which means for something like

    result = 12 * 2 + 9 == 3 + 8 * 4
    result = !parser.valid 34 && result

Ultimately what’s actually being executed is:

    scope .set .result ( ( ( 12 .times 2 ) .plus 9 ) .eq ( 3 .plus ( 8 .times 4 ) ) )
    scope .set .result ( ( ( scope .parser .valid 34 ) .not ) .and ( scope .result ) )

So this is fine for symbols like + and – which operate on two clearly-defined values, but what about something more complicated like “=”? Well, there ought to be some kind of way to pass “op” a .custom function, which takes in a list of lexed tokens representing a line and returns a transformed list of tokens. At that point you can do pretty much anything. “=” might be the *one* thing that you can’t implement this way because = does special things involving adding bindings. But short of that, custom “op”s would be sufficient even for things like, I don’t know, flow control:

    if ( a == 4 ) { k.x = 3 } else { k.x = 4 }

I may be getting into the language-geek weeds again here but I’m gonna walk through this: Let’s say I have a higher order function “if ^pred ^exec” which takes functions “pred” and “exec”, executes “pred” (pred is probably nullary… which I haven’t decided what that means in this language yet), if the result is true it executes “exec” and returns the void combinator (v ^x = v), if the result is false it returns a function which expects as argument either .elsif (in which case it returns if) or .else (in which case it returns a function that takes a nullary function as argument and evaluates it). We’ve now defined the familiar if…elsif…else construct entirely in terms of higher order functions, but actually *using* this construct would be pretty irritating, because the “pred” and “exec” blocks couldn’t just be ( ) or { } as people expect from other languages, they’d have to be function-ized (which means annoying extra typing, toss some ^s in or however lambdas are made in this language). But, we can declare “if”, “else” and “elsif” rewrite ops: “if” ^-izes the next two tokens and then replaces itself with just “if” again; “else” and “elsif” ^-ize the next one token each and then replace themselves with .else or .elsif. If we do this, then the familiar if… else syntax above just *works*.

…why am I going into all this, about “if” “else”? Well, because I want to stress that it means *flow control constructs can be implemented in the language itself*, and they will be truly first-class equals with builtins like “if” or “while”. In my eventual vision of this language, the *only* language-level syntactical elements are

    ( )
    [ ]
    { }

And *everything* else, including comment indicators and the end-of-line statement-terminator, is just rewrite rules, ideally rewrite rules written in the language itself. Which implies if you don’t like the language’s syntax much, you could just unload the builtin “stdops” module that contains things like “+” and “if”, and substitute your own. “op” rules are local to scopes, so syntax could vary hugely file to file. Which… well, shouldn’t it? I know people who avoid entire languages because they don’t like one or two things about the syntax. Say, people who go “well, Objective-C has a neat object model, but I can’t get used to all those square brackets”. Or I in my last blog post specifically said that although they both have lots of features I like, I personally won’t use LISP because I can’t make visual sense of S-expressions, I won’t use Javascript because of the casting rules. None of this makes any sense! Languages should be about *features*. They should be models of computation, and we should be evaluating them based on how expressive that model is, based on the features of the underlying flow control or object model or type system or whatever. Syntax shouldn’t have to be part of the language selection process, and if languages let us put the sugar on ourselves instead of pre-sugaring everything then it wouldn’t have to be. I’m probably getting carried away here. What was I talking about? Did I say something just now about casting rules? Let’s talk about casting rules.

Blocks are language machinery

Some syntactical elements, like [ ] and =, might be too complex for the programmer to plausibly implement themselves. The programmer should still have a fair amount of control over these builtins work. One way to do this would be to have things like [ ] and = implicitly call functions that exist in the current scope. For example, instead of calling .set, = might call a function “assign” that exists in current scope; this would allow individual scopes to make policy decisions such as the variable auto-instantiation rules I mentioned earlier. [ ], at the moment it instantiates the new block, might call a function “setup” that exists in the current scope, allowing the programmer to do things like change the default ditch (base class) or the exact meaning of “inherit”. There might be a function that defines the default type constraints for numbers, or strings, or lines of code. Maybe somewhere there’s a Haskell fan who wants to be able to have every ( ) wrapped up to be ^-ized and every line wrapped in ( ) :: stateless, so that any code *they* write winds up being effectively lazy-evaluated and side-effect-free and they can only communicate with the rest of the language using unsafe monads. They should be able to do that.

One thing I definitely want in is for there to be something like a “fallback” function which, if a particular block is called with an argument whose type doesn’t fit any pattern the block has defined, attempts to map the argument to one of the patterns the block *can* handle. In other words questions about whether different but intraconvertable types like ints and floats can be converted between without a cast would be a decision made on a per-project or per-file basis. Or for example if there’s a function

    square ^x:int = x*x

and one of the patterns on the fallback block is

    fallback ^fn : function( ^type, _ ) [^x : type] = fn x    # Follow all that?

(Let’s assume “function” is a higher-order type function such that function(a,b) is the type of a function a -> b, and let’s assume _ has the magic property of “match anything, but don’t capture it” when used in a pattern.)

…then even though the function is only defined for (square x) we could totally get away with calling square[ x ], because the fallback function could match [ x ] to x.

Uh, incidentally, I’m not totally sure this thing with the fallback function is actually in general possible or possible to make performant. But as with most of the stuff in this language, I think it would be fun to try!

Blocks are C++ or Javascript objects in disguise, potentially

There’s one last thing I want to talk about here, although it’s one of the most important features from my perspective. The model we have here– where formally speaking all field accesses execute functions, all field assignments execute functions, and there’s some kind of type checker at work capable of tracking fine detail about what kinds of operations get performed on individual blocks– means that the underlying language-level implementation of “a block” could differ from block to block.

The model I’ve described here for blocks is extremely dynamic and flexible– *too* flexible, such that it would be very difficult to make code using all these dynamic features performant. Except not every block will be using all of the features blocks have. Some blocks will only contain “value” keys (i.e. never a ^var:type pattern), and the type inferrer will be able to prove this the case. The compiler/interpreter could represent this one block internally as a plain hashtable, rather than taking the overhead to enable executing arbitrary code on every access. Some blocks, despite being mutable, will have a fixed known set of keys; the language could maybe represent these in memory as plain structs, and translate atoms to fixed memory offsets at compile time.

And some blocks, at the programmer’s direction, might be doing something else altogether. It’s easy to imagine a “proxy object” where each invocation of an atom and an argument on the block is actually copying the atom and argument and shipping them into another thread or across a network, and the type checker ensures the contract is followed and objects are actually copyable; you could build an Erlang style messaging system this way.

Of particular interest to me, some blocks might actually be guests from some totally other system, say a different language with its own object model. An FFI for some other language could make wrapper blocks for that language’s objects, and put in place type guarantees that the programmer does not interact with those blocks in any way the guest language does not support. The two languages I’d personally really like to be able to interface with this way are C++ and Javascript, because these languages have valuable platform and library support, but also are languages I do not actually want to *write*.

C++ in particular interests me, because I’m not aware of any languages which are “higher level” in the sense that interests me but which can currently adequately interface with C++. C++ is actually pretty tricky to interface with– the big problem here, to my mind, being that method calling conventions (name mangling) vary from compiler to compiler. Actually, on some platforms (by which I mean “Windows”) it’s the case that shared libraries (DLLs) can’t be shared between compilers even if you are writing in C++ yourself. It would probably be necessary, if making a C++ FFI, to target one particular compiler (I’d vote Clang, because it’s extensible and has good platform support). Choosing to target one particular compiler would have a neat side effect: With some knowledge of the compiler’s implementation details, it *ought* to be possible to make blocks that inherit from C++ classes, and have those blocks actually construct fake vtables at runtime that jump into the compiled code for (or interpreter for) my language. Since in my language “classes” and “objects” get constructed by calling functions whose execution could be potentially deferred to runtime, it would be essentially invisible to the programmer when they say [ inherit QObject; objectName = “Block” ] whether a normal block or a pseudo-C++ class is being constructed.


Anyway, here’s what I think I’ve got here. I started with one single idea (pattern-matched unary functions that remember the order in which their patterns were assigned), and asked the question “how much of what a language normally does could I collapse into this one concept?”. The answer turns out to be “very nearly EVERYTHING”, including stuff (like type specifications, macros and FFIs) that most languages would wind up inventing effectively an entire sub-language just to support (templates… ugh). I actually *do* want a new programming language, mostly because of that thing I mentioned with not liking any existing language’s C++ interop, and I actually do intend to at least attempt this project. I’m basically just gonna download the Clang source at some point and see how far I get. One thing in my favor is that since this language is based on a small number of simple things that interact in complex ways, I could probably get a minimal implementation going without too much difficulty (especially if I don’t go for types on the first pass).

Oh, one very final thing: I never said what all of this is called. In my head I’m planning to name this language either Emily, because despite being fundamentally OO it strikes me as a fairly “ML-y” language; or Emmy, after Emmy Noether. I’ll decide which later.

That’s all.

Note to commenters: Harsh criticisms are very much welcomed. Criticisms based on demographics or assumptions about my background are not. Thanks.

Statement of Intent

Tuesday, April 22nd, 2014

I think I have decided that these are the things I most want in a programming language:

    1a. Objects with string keys, which double as dictionaries.
    2a. Closures.
    3a. A way to store a closure in an object and invoke it with the relevant object bound to some variable within the function (“this”).
    4a. Prototype-based inheritance for objects.
    5a. The potential to instantiate a closure, or an arbitrary singleton object, inside of an expression (“lambda”).
    6a. Gradual typing (with calls from untyped code into typed code being protected by something like a check and an exception throw at runtime).
    7a. Two-way communication with C++ (C++ objects may be visible within the language, language objects may be passed into C++ code).
    8a. Two-way communication with Javascript.
    9a. Support for multiple concurrent threads of execution, with interaction between by means of message passing.
    10a. It must be possible for me to develop a program on one operating system and execute it on another operating system (i.e. there is a VM or a cross compiler).

And here are some things that I think would be very nice in a programming language:

    1b. Pattern matching.
    2b. Syntax support to allow me to declare curried functions, as transparently as if they were non-curried functions.
    3b. Function invocation with named arguments (i.e. Python keyword args).
    4b. Suspension and resumption of stacks (coroutines or generators).
    5b. Atoms (interned strings which can be used as object keys without paying for a string lookup).
    6b. Properties (i.e. overloading of assignment).
    7b. The language itself should be ultimately defined as a AST, with multiple possible syntaxes reducing down to that single AST.
    8b. Language features (ints, exceptions) should be possible to enable or disable for individual programs or pieces of code. Not all features fit all projects.
    9b. There should be a potential for (arbitrary and user-defined?) type “adjectives” which are assertions that are remembered as part of the type and checked as part of the type checker. (Is this object mutable? Is this object mutable with regard to its set of keys? Does this function have side effects? Are these two properties recursive for the related object/call graph? Does this function ever use “this”? Is this object ever accessed with a non-atom key? Is this integer greater than three?)
    10b. Potential to directly manipulate the scope of a closure (I.E. mappings of unbound variables in the function body) after function declaration.
    11b. The collapsing of functions and objects. (“Look up key ‘f’ on object a” and “Invoke method a with unary argument ‘f'” should be the same operation, same syntax, same behavior. Function scope and prototype chain should be, insofar as the user can tell, the same mechanism).
    12b. Customizable automatic casting rules (if we have 1b, applied by a user-selected pattern-matched function).
    13b. Syntax support for object “classes” that have the familiar semantics of Python or Java. Ideally, classes would just be a pattern implemented on top of prototypes, but the syntax should make this pattern easy and the class identity should be visible to things like debuggers.
    14b. Some simple user-configurable syntax redefinition (like the ability to define an operator ^ where invoking 3^4 should be actually interpreted as exponent(3,4), and ^ has a user-defined operator precedence, etc).
    15b. The ability to have some sort of object which is ultimately represented at runtime as a packed memory buffer, such that saying “set element 4 of x to 3” is actually writing “3” into a specific well-defined byte or bytes in memory. (This is a useful thing both because of performance optimizations it makes possible, and for interacting with hardware like GPUs which only understand byte streams.)

I would like to explore ways to get to a point where I have a programming environment that satisfies everything from the A list and some emotionally satisfying number of items from the B list. In seeking such a language, there are incidentally some conditions that any language I spend time using will have to satisfy. I don’t want to have an argument with anyone about this final list; just think of them as personal preferences.

    1x. It must not require me to program with S-expressions.
    2x. It must not require the use of the JVM or CLR. (I am potentially willing to compromise on the CLR.)
    3x. It must not force me to program in a way which avoids the use of side-effects or state.
    4x. It must not require me to use any closed-source compiler, core library, or development tool.

No language currently exists which provides everything on my A list. Some of the items on the B list, particularly 10b and 11b, do not exist in any language that I am aware of. Most of the most interesting items on the B list only seem to exist in languages which fail one or more items on the X list. Any currently existing language will be a compromise. How fair do these compromises look?

If I use Lua, I am happiest. Lua gets me 1a, 2a, 3a, 4a, 5a, 10a, 4b, 6b, and 13b, 15b if I’m using Luajit, and since I have 5a I can awkwardly approximate 3b. No language I know of does 1a better than Lua. Unfortunately, 7a and 9a, which Lua does *not* support, are two of my *most* important conditions– I literally can’t develop without those two, whereas missing 1a through 5a will merely make me sad.

    (Lua gets something close to 7a– two-way communication with *C*– and two-way communication with C can be used to construct trampolines that *approximate* communication with C++, but in my experience using this technique is so frustrating I would rather avoid Lua altogether than attempt it again. There are also incidentally projects which purport to offer 8a for Lua, but I have not seen these projects actually working and I believe that they would need additional development before being actually used.)

If I use Javascript, I do get 1a, 2a, 3a, 4a, 5a, 8a obviously, 10a, and 6b. If I am using Javascript ES6, I also get 9a, 3b and 4b. Actually, Javascript does pretty well against my lists. Unfortunately, some of the personally important items on the list Javascript satisfies, it messes up quite badly. For example it offers 4a, but uses a baroque method for defining an object with a prototype, and it does not offer any way when overriding a prototype method to invoke the “super” method. That last one’s pretty huge; “super” is a basic feature of all object systems which, since it is present in Self, I would argue without “super” you don’t have a prototype-based inheritance system at all. On similar lines, Javascript offers 2a but essentially ruins it by not (before ES6) offering proper block scope; if I can’t readily create a closure in (for example) a loop, that is so awkward I probably won’t ever use closures at all.

There is another problem with Javascript: It is an unending cavalcade of horrors. Javascript feels *shoddy*. Many extremely basic language features are riddled with weird exceptions, and exceptions to exceptions, such that you never feel certain what the code you have just written does; many kinds of simple typos result in a silent failure or unexpected behavior rather than any kind of error. (The type coercion rules alone would feel like a great argument against using Javascript even if everything else about the language were perfect.) I very much like what is in this toolbox, but the individual tools all feel as if they will fall apart in my hands if for example I hit anything too hard with that hammer. This shifting-sand problem is exacerbated by the dramatic variance between Javascript versions and implementations; the primary benefit of Javascript is its huge installed base, but only a small subset of the syntax is “safe” in the sense that an acceptable portion of your installed base is compatible with it. This is an unfortunate property when we are talking about mere syntactical convenience features: 3b and 4b are nice, but I would not literally give up users to get them.

    (I have not investigated if Javascript satisfies 7a).

If I use C++, I get 4a (or rather 13b, class-inheritance alone, which is close enough for me), 6a (not gradual typing but strong typing, which I don’t mind), 7a (sort of– assuming you never dynamically link against anything from a competing C++ compiler), 9a (with memory sharing instead of events, but you can add the events via a library), 10a (with *great* difficulty), 15b (although since 15b is *all* C++ offers for data storage, it isn’t very pleasant), and an extremely limited 14b that can be used to get 6b (although only for some kinds of variables). There is also a horrible, confusing mockery of 12b which really the language would be better off if they had not tried to offer this feature. If I use C++11, I get 2a, but it’s gross (the closures capture variables manually, rather than automatically; they’re only created if you use the “lambda” feature, which is totally separate from any other kind of function or method; and the syntax is incredibly ugly). C++11 also has “auto”, which is… one step closer to my ideal form of 6a than strong typing alone would be, I guess. Overall C++ and C++11 don’t offer *anything* exactly the way I’d most prefer it, and the only items on my list I’d even feel C++ is really “good enough” at are 4a/13b and 9a.

    (At this point one notices something interesting about my criteria 7a: Not even C++ does very well against it. This is a hard criteria to meet.)

I’ll leave it as an exercise to the reader to test my criteria against Python, ML, Haskell, and Erlang.

Looking at my options, I feel that if I could somehow hack Luajit to offer 6a and 7a and also support Lua Lanes (which provides 9a), I’d have a language I were totally satisfied with. I spent a while seriously considering doing just that. However, Luajit is big, and 7a in *particular* is such a hairy thing to implement that I wouldn’t want to attempt it in someone else’s codebase.

So I think if I want a language that satisfies all of these things, it would have to be a new language. Along those lines, I have written a follow-up post here.