Archive for the 'Uncategorized' Category

Announcing the Emily programming language, version 0.1

Sunday, January 18th, 2015

This last year I’ve been working on a programming language. Around April, I sat down and I wrote out a list of all the things I want out of a language. Nothing that exists right now felt satisfying, so I decided to make my own. I’ve named this language Emily, and as of today I have a finished “version 0.1″ release that you can download and run. This version is very primitive in certain ways, but you can write a program in it and it points enough toward what the language can eventually be that I’m excited.

An example

Here’s a small program in Emily:

width = 80

foreach ^upto ^perform = {
    counter = 0
    while ^(counter < upto) ^( perform counter; counter = counter + 1; )
}

inherit ^class = [ parent = class ]

line = [                               # Object for one line of printout
    createFrom ^old = {
        foreach width ^at { # Rule 135
            final = width - 1
            here   = old at
            before = old ( at == 0 ? final : at - 1 )
            after  = old ( at == final ? 0 : at + 1 )
            this.append: ( here && before && after ) \
                     || !( here || before || after )
        }
    }
    print ^ = {
        this.each ^cell { print: cell ? "*" : " " }
        println ""                                          # Next line
    }
]

repeatWith ^old = {  # Repeatedly print a line, then generate a new one
    do: old.print
    new = inherit line
    new.createFrom old
    repeatWith new
}

starting = inherit line        # Create a starting line full of garbage
next = 1
foreach width ^at (
    starting.append: at != next
    if (at == next) ^( next = next * 2 )
)
repeatWith starting                                             # Begin

This executes the 1d rule 135 cellular automata, or in other words, it prints strange pyramids:

*  * *** ******* *************** ******************************* ***************
      *   *****   *************   *****************************   **************
 ****   *  ***  *  ***********  *  ***************************  *  ************
  **  *     *       *********       *************************       **********
*       ***   *****  *******  *****  ***********************  *****  ********  *
  *****  *  *  ***    *****    ***    *********************    ***    ******
*  ***          *  **  ***  **  *  **  *******************  **  *  **  ****  ***
    *  ********         *               *****************               **    **
 **     ******  *******   *************  ***************  *************    **
    ***  ****    *****  *  ***********    *************    ***********  **    **
 **  *    **  **  ***       *********  **  ***********  **  *********      **
       **          *  *****  *******        *********        *******  ****    **
 *****    ********     ***    *****  ******  *******  ******  *****    **  **
  ***  **  ******  ***  *  **  ***    ****    *****    ****    ***  **        **
   *        ****    *           *  **  **  **  ***  **  **  **  *      ******
**   ******  **  **   *********                 *                 ****  ****  **
*  *  ****          *  *******  ***************   ***************  **    **    *
       **  ********     *****    *************  *  *************      **    **
******      ******  ***  ***  **  ***********       ***********  ****    **    *
*****  ****  ****    *    *        *********  *****  *********    **  **    **
 ***    **    **  **   **   ******  *******    ***    *******  **        **
  *  **    **        *    *  ****    *****  **  *  **  *****      ******    ****
        **    ******   **     **  **  ***               ***  ****  ****  **  **
*******    **  ****  *    ***          *  *************  *    **    **
 *****  **      **     **  *  ********     ***********     **    **    ********
  ***      ****    ***         ******  ***  *********  ***    **    **  ******
*  *  ****  **  **  *  *******  ****    *    *******    *  **    **      ****  *
       **               *****    **  **   **  *****  **       **    ****  **
******    *************  ***  **        *      ***      *****    **  **      ***
*****  **  ***********    *      ******   ****  *  ****  ***  **        ****  **
****        *********  **   ****  ****  *  **       **    *      ******  **    *
***  ******  *******      *  **    **         *****    **   ****  ****      **
 *    ****    *****  ****       **    *******  ***  **    *  **    **  ****
   **  **  **  ***    **  *****    **  *****    *      **       **      **  ****

Anyway, even without knowing the syntax, you might notice a few things looking at this program:

  • This language is extremely extensible. Near the beginning, I realized there were two things that I didn’t get a chance to put in the standard library: a foreach function, and a way to instantiate an object of a class. So I just implemented them in the program. Very simple.
  • Support for both ["unpure"] functional and object-oriented styles: Functions are being passed as values all over the place, functions are being created offhandedly (that’s the ^), objects with prototype inheritance are created very casually (that’s []), tail recursion works.
  • Clean, familiar syntax: At least, assuming you’ve been writing a lot of Python, JS, or Lua, this looks a lot like code you’ve written before, other than the ^s.

Why’s Emily special?: For language geeks

The one central New Idea in Emily is that everything is a function, or maybe put another way, objects and functions are interchangeable. You can think of an “object”, or a structure, as being a function that maps key names to values; Emily actually treats objects this way. If an object can be a function, then anything can be; 3.add can be the function that adds 3 to another number, 3 can be the function that maps .add to 3.add.

The important thing about this isn’t that functions specifically are important; what turns out to be important is simply that everything in Emily is the same thing. Everything acts like a unary (one-argument) function, so where other languages might have several “verbs” (field lookup, variable definition, function definition, function application, arithmetic) Emily has exactly one verb (function application). All you’re doing is applying functions in a particular order, and writing code just means plugging those functions together like pipes. Even the “conventional” syntax elements, like + or the . lookups, are just unary function applications in disguise. Even declaring a variable is done with a function call.

So why does this matter?

Why’s Emily special?: The practical side

I like to build abstractions in my code. If I’m writing a C program, and I find I’m writing for(int c = 0; c < upto; c++) {something} over and over, I get annoyed; I wish I could just define a “foreach upto {something}”, like I did in the pyramid program up top.

I like to mix different kinds of tools when I write software. I really like Lua, but Lua isn’t very good for certain kinds of binary manipulation and threading, so I write programs that are part C++ and part Lua. But this is tricky; the languages each have their own complexities, and the complexities clash. Languages don’t work well with each other.

I assert both these things get a lot easier if you have a language whose underlying model is very simple.

In Emily, where everything is function applications, building complex abstractions just means plugging the applications together in a particular order. The language itself doesn’t need to be this simple– again, if you look above, a lot of stuff is happening in the code that doesn’t look like a function application. But that complexity is something built on top of the language, rather than being a fundamental part of the model. This means it can be extended further, and it also means it can be replaced.

Syntax like + or =, for example, is actually performing macro transformations– 3 + 5 * 4 gets rewritten into 3.plus(5 .times 4). This makes these operators easy to extend– if you want to design an object that “acts like” a number, you just define an object that implements the .plus and .times methods. It also makes them possible to replace– the transformations are just little programs, and in a later version of Emily you’ll be able to define your own operator transformations, if there’s a different syntax you’d like better.

That’s not all that impressive, though– operator overloading is a pretty standard feature in languages, and macros are not the way you’d prefer to implement abstractions. Rather the more basic element in Emily of everything is function application is what opens up the really powerful possibilities. Let’s try something a little more unusual. As mentioned above, Emily has prototype inheritance. But it’s single inheritance– only one parent per object. What if you’d prefer multiple inheritance? Well, you can implement it yourself, by writing a single function:

# A function that returns a function-- it generates the fake "parent"
dualInherit ^parent1 ^parent2 = ^key {
    thisUpdate this: \
        (parent1.has key ? parent1 : parent2) key
}

hasAddOne = [
    addOne ^ = this.value = this.value + 1
]
hasAddTwo = [
    addTwo ^ = this.value = this.value + 2
]

child = [
    parent = dualInherit hasAddOne hasAddTwo
    value  = 0
]

do: child.addOne
do: child.addTwo
println: child.value

So what’s happening here? The object that child inherits from is chosen by setting the parent key. But objects and functions in Emily are interchangeable. So child inherits from a function. dualInherit takes two desired parents and manufactures a function which takes a key and executes it on whichever of the two parents knows how to respond. child then uses dualInherit to inherit from both the classes hasAddOne (from which it inherits the method addOne) and hasAddTwo (from which it inherits the method `addTwo).

What else?

There’s a couple more interesting things that made it into the language, even as early as this version is:

  • \version 0.1
    This feature is small, but I believes it solves a somewhat fundamental problem with programming languages. Each Emily program is encouraged to start with a line identifying the language version it was developed against. When an Emily interpreter– current or future– encounters the \version line, it makes a decision about whether that code can be run or not. If the hosting interpreter is backward-compatible with the code’s version, it just runs. But if backward-incompatible changes have been made to the language since then, it will either enter a compatibility mode or politely refuse to run. At some point, it will be possible to install these compatibility modes as pluggable modules.

    I write a lot of Python, and a huge running problem is compatibility between versions. In Python, as in most programming languages, the implementation version is the same as the language version. Python 2.4 runs Python 2.4, Python 2.7 runs Python 2.7, Python 3.1 runs Python 3.1, etc. Meanwhile Python 2.7 can run Python 2.4 code, but Python 3.1 can’t run Python 2.7 code, which means Python is competing with itself and nobody uses Python 3 because all the code’s written for 2.7. (And even before the big 3.0 switch, forward compatibility created a huge problem all by itself: If you had a program that used a feature from 2.5, but what you had installed was 2.4, you wouldn’t know it until you tried to run it and something would break strangely, possibly at runtime.)

    This is all silly! Language versions define interfaces, and interpreters are engines. We shouldn’t be holding back on upgrading our engines because the interface changed (and if there’s some reason a new engine can’t handle the old interface, it should at least fail very early). It’s generally possible at least in principle to convert between these interfaces, so it should be possible to install something that does conversion for an incompatible past interface (probably even a future one!). It should be possible to mix code written against different interfaces in the same program– maybe even the same file. There’s surely a point at which this becomes untenable (library cross compatibility probably gets awkward quick), but language implementors not being able to get updates adopted because nobody wants to lose back compatibility with 15-year-old versions doesn’t sound very tenable either.

    Anyway, for now: Just tag each file with its version, and all this becomes a lot easier to sort out later.

  • Proper Unicode support

    This really ought to be something we just expect of a language these days, but Emily is being developed for full Unicode support and the interpreter treats source files as UTF-8. Right now this only extends as far as handling Unicode whitespace– well, and the macro system supports unicode symbols, but since you can’t make macros yet that’s not so useful. As soon as possible though my goal is to implement UAX #31 so you can use Unicode in identifiers, and I’m hoping to work with coders fluent in non-latin-script languages to make sure Emily is usable for those purposes.

    Oh, also: At some point I’m just gonna make smart quotes work as quotes. If we’re gonna keep pasting them in to our code by accident, they might as well work.

  • Return continuations

    I’m… not sure I should be calling too much attention to this, but it is a bit unique. As I’ve said, everything in Emily is a function, at least in form. There are no “keywords” as syntax constructs– any “syntax” is just shorthand for function calls. So if I’m going to implement, say, return, it has to be something that can be treated like a function. return in Emily winds up being what’s called a “continuation”– a function-like object that when called just jumps to a particular place, in this case the end of the method call.

    This has an interesting side effect that I had not entirely intended:

    timeMachine ^ = (           # Function takes no arguments
        goBackward = return     # Store return in a variable. Yes, really.
        return 1                # Return. This line only runs once.
    )
    
    counter = do timeMachine    # The return value of "do TimeMachine" is 1, right?
    println: counter            # Only the first time-- every time we call goBackward,
    goBackward: counter + 1     # we return "counter + 1" from timeMachine,
                                # even though timeMachine already ended.
    

    …it turns out “return” for a particular function call can be treated like any other value, and even outlive the function call it was born in. As with any other kind of continuation, this creates the opportunity for some very powerful constructs (I’m trying to work out how I can implement Java-style named breaks with it) and also the opportunity for some really bad ideas. I’m going to see if I can find a way to encourage the former while maybe putting some limits on the latter.

What next?

What you see here is part of a more ambitious set of ideas (link goes to my original design writeup before I started writing any code). Here’s some things I want in Emily eventually; some of this may not be completely feasible, but I think even making it part of the way there would be great.

  • Types

    There’s a lot of things I want to try with typing in Emily, and I’m nervous about elaborating too far on what they are before I know for sure what I can pull off. But in general: Emily should have type annotations. Typing should be “gradual”; you should be able to have a part of your program with type annotations and a part without. There should be a prover for types, and an inference engine. If you assign something to a variable or argument which has a type annotation, that should mean a runtime check for correctness. There should be a “compile time” check also; the compile time check should be able to tell some calls are definitely type-correct and optimize out the correctness check, and it should be able to tell some calls are definitely type-incorrect and refuse to run the script until they’re fixed. (Right now, nothing stops you from typing 3(4) except that this will fail at runtime).

    This is all pretty standard for a language written in the last ten years. But I want to try some odder things, based around Emily’s core ideas that everything in the language is “the same thing”, interchangeable with functions, everything is extensible and there is no syntax “magic”. So: Types should be constructable at runtime. It should be possible to use a function as a type; there should be a syntax for turning x such x > 4 into a type. Some types should just be language-level assertions, like there should be a type for “this variable’s value is known at runtime” or “executing this function has no side-effects”. There should be an ability along the lines of assigning a type as a key for an object– something like [ .name="minusone"; ^x of int = x - 1 ] for a function that returns "minusone" when applied on .name or subtracts 1 when called with an int. In other words, I want average objects to be able to act like match or pattern-matching guards from a functional programming language. (This would mean that you could build up guards that inherit from other guards, which I kind of like.)

  • C++

    I need to be able to interoperate with other languages. I write video games, so there’s a whole bunch of C++ libraries I really need access to. Interoperability with C++ is hard. Interoperability with anything is hard. As alluded above I used to write a lot of Lua, and the interaction layer with C++ was just incredibly complicated and involved generating code with a Python script.

    Complicated or no, I’m going to have to write an interop layer for Emily. But I think that there are some design choices in Emily that are going to make this easier. Because Emily is so dynamic, I can generate a bunch of stuff at runtime that otherwise would have required a code generator, and Emily’s everything-is-functions rule means that wrappers can be much less awkward than they might in another language. Objects and classes and methods from different languages all have different semantics; if you have an object in one language which actually wraps a guest object from another language, you’re going to wind up with weird mismatches where the objects and methods you’re interacting with in the host language don’t quite behave like their analogues in the guest language (the “complexity clash” I mentioned earlier). But an object in Emily is a smooth ball; its semantics are just a matter of what arguments you feed it. If the type system is smart, it could even verify that the arguments you feed the wrappers are always consistent with the semantics of the guest object.

  • Reader macros

    Emily has macros that transform the AST into a different AST before it is executed. A lot of functional languages have the concept of a “reader macro”, which generates AST from whole cloth. There’s a piece of implied code called the “reader” which is consuming the text of your program and parsing it into an AST; \ will eventually be a way of sending messages to this reader, possibly enabling transformations that a normal macro couldn’t perform because it would need access to the program source as a string. Right now the only reader instruction is \version, and the only thing it can do is halt the program if the version is wrong. There will be more reader instructions in future (for example, \op will probably be the way to define the normal, non-reader macros).

    What really interests me though is the possibility of a \reader, which would cause the reader to just plain get out of the way and hand off parsing the rest of the file to a piece of Emily code. My daydream here is that if you could write a \reader that parses some completely other programming language and emits Emily ASTs (hopefully easy since Emily really only barely has an AST, the AST is just a tree of applications)– and if I’m right that it will be unusually easy to write Emily code that accepts other languages’ objects as guests– then Emily could become a useful intermediate layer that knows how to translate between several different languages and mediate their differences.

These are all Big Ideas though, and the ideas the language already has could stand some cleanup, so for 0.2 I’m going to be focused on basic functionality improvements (less confusing scoping, operators on strings, short-circuiting booleans, user-defined operators, unicode, package loading, IO).

Downloading and running Emily

As mentioned, Emily is available from a BitBucket page (or, if you can’t use Mercurial, its GitHub mirror), but not yet in any other form. You will need to compile it yourself; it’s written in Objective Caml, so you’ll need to install that first. For instructions, see https://bitbucket.org/runhello/emily/src/stable/doc/build.md. By the time you read this, there may also be install packages; see emilylang.org.

What’s “BitBucket”?

If you’re not familiar with BitBucket or Github, go to the Downloads page on BitBucket, click the “Branches” tab, and to the right of the word “stable” click “zip”. This will download a zip file of the most recent stable release.

Can’t I just download something?

If you happen to use “homebrew” for Mac OS X, Misty De Meo has helpfully put together a homebrew package. You can install Emily via brew by running:

brew install mistydemeo/emily/emily

Hopefully by the next release we’ll have Debian packages too!

Getting started

If you’re interested in Emily and want to give it a shot, a good thing to read next would be the tutorial. If you just want to know more, there’s a bunch of documentation including some design documents and information about modding the language in the docs folder.

License

Emily is MIT licensed, meaning it is open source and only attribution for the authors is required. The list of contributors is here.

Open source index, Indiecade

Monday, October 6th, 2014

Hi hi. Two small blog-relevant updates.

  1. It got frustrating to me how hard it is to get around the list of projects on my BitBucket page, so I made a sorted index of all my open source code. The list is hosted on BitBucket and includes all my game projects, including a number I’ve never released.

  2. The game BECOME A GREAT ARTIST IN JUST 10 SECONDS I made with Michael Brough is an Official Selection at Indiecade 2014. The game will be demoed at Indiecade’s Night Games event this Saturday, with this customized keyboard made by Rachel Yonamine:

    GREAT ARTIST keyboard

    Here’s our Indiecade trailer:

Emily programming language: Status update one

Wednesday, June 4th, 2014

About a month ago, I put up some plans for a programming language. It actually got some interest!, so I thought I’d go over some things that have happened since then.

First off, I decided the name of the language is definitely Emily, unless I’m somehow legally forced to call it something else. I made a website for it and an announcement-feed Twitter, at emilylang.org and @emilylanguage, respectively.

Second off, I now have a prototype implementation! It’s a very, very minimal implementation– you seriously couldn’t develop anything in it, and it’s missing most of the language’s better planned elements– but it does demonstrate a couple of the language’s striking features, and it’s turing-complete. (And the turing completeness part can be demonstrated in an amusing way: It turns out to be possible to embed Unlambda programs in it.)

Programs written in the language don’t look very pretty right now. Most of the “operators” in Emily, like = or +, are eventually intended to be defined as macros, and those macros aren’t implemented in the prototype yet, so setting a variable looks a little funny:

# Prints "3.0"
set .test 3
print test

And adding looks a little funny:

# Prints "7.0"
print (3 .plus 4)

In the final version of Emily, writing “3 + 4″ will do the same thing as writing “3 .plus 4″– the only difference will be that + has operator precedence, and .plus does not.

There’s one macro that *is* in already– ^, or “lambda”, which turns whatever comes right after it into a function:

# Prints "4.0"
^x( print x ) 4

This defines an “anonymous function”– if you aren’t familiar with that idea, it’s a function that doesn’t have a name but just shows up in an expression, like a literal– that takes one argument and prints it. It then feeds the anonymous function the argument “4″.

With these couple things explained, here’s a more complicated Emily program that works in the current interpreter:

set .countup ^arg {
    set .count (arg.from)
    loop ^(
        print count
        print "\n"
        set .count ( count .plus (arg.step) )
        count.lt (arg.to)
    )
}

countup[ set .from 10; set .to 20; set .step 2 ]

Again, not very pretty, but this demonstrates a couple of interesting things that Emily tries to do.

To recap my last blog post a bit, everything in Emily is formally a function– everything. Objects are functions pretending to be objects, functions which take key names (in the form of strings like .plus– assume this is just a funny way of saying “plus”) and map them to values. Numbers are objects, which are functions. Scopes are objects (which are functions); the language sets up and shuffles around the scope functions silently, and it forwards any bare symbols it finds (like “set” or “count” in the examples) to the current scope function as a string argument. A lot of the time these various functions don’t really *act* like functions, of the kind we normally recognize (for this reason I’ve been calling them “blocks”, after Smalltalk) but the fact the language doesn’t distinguish the different kinds of functions semantically opens up a lot of possibilities for useful abstraction. The only “verb” in Emily is “apply”– apply function A to argument B. Everything else is just built on top of that.

Not a lot happens in this example, but a couple things do stand out: assigning a variable value just means invoking the “set” method on the current scope (i.e., passing .set to the current scope block, doing which returns a function that can be used to alter that scope block’s mappings). There are no special flow control constructs– “loop” is just a function that takes a closure as argument and executes it until it returns false. The thing that really stands out to me here though are the appearance of {} and []– the countup language body is wrapped in a {} to give it its own variable scope, and when we invoke countup the argument list is written as an “object literal”, denoted by []. What’s interesting here is how these two things are implemented by the language.

Parenthesis in Emily create “groups”– I have to call them that because I used the word “block” for something else already, maybe a mistake. Any parenthesis in Emily can contain multiple statements, separated by semicolons. A trick here is that group (), scoped group {}, and object literal [] are all the same thing, distinguished only by what the scope is within the group and what is returned at the end. More interestingly, there’s no “magic” to the group operators– effectively, what defines the different group types are a couple of lines of setup/teardown code that run at the beginning and end of the group, and that setup/teardown could in principle be written in Emily. In other words, {} and [] could in the final version just be implemented as macros! In the current implementation, all three group types are handled by the same code in the interpreter, with the difference being:

    (Assume any time a statement executes, its evaluated value is assigned to a variable named “last”)
    Plain groups: Set [scope] to the enclosing scope; after final statement return “last”.
    Scoped groups: Create a block [newscope], which prototype-inherits from the enclosing scope, and set [scope] to it. After final statement return “last”.
    Object literals: Create a block [object]. Create a block [newscope], which prototype-inherits from the enclosing scope. Assign [newscope].set = [object].set and [newscope].this = [object]. After final statement, return [object].

Did I get too language wonky there? The point I’m trying to make here is that “an object literal syntax” is a really basic thing to a language, but this language kind of doesn’t have one, because it doesn’t need one. That’s just normal code running inside of the [], and the thing that makes [] “special”– the setup/teardown– could have been implemented by an end user. This is exciting to me because it implies end users can implement other language extensions, things as complicated as an object literal syntax, on their own.

The implementation

The prototype implementation of Emily is hosted on BitBucket, although it will not do you a lot of good right now. The main problem is that it is written in Objective-C, which means that it will only easily build if you have OS X and XCode. Compiling on other operating systems is possible but will require setting up something like GNUStep. I haven’t investigated this myself, mostly because I intend to port to something-other-than-Objective-C rather than bother with GNUStep. If you do have a mac, or do have GNUStep set up, you’ll probably find it pretty easy to set up and extend. The current implementation is only about five or six source files. (Sorry if the BitBucket page is a little confusing, they just changed their layout and now it’s all weird– the cloud with the arrow in it is the “download source tarball” button.)

Remember, this is literally the very first thing I got to run at all– as of this exact writing there’s no code documentation, no documentation of the exact language as implemented so far (the closest is the sample code files), and there are known memory leaks. I’m not even giving this a version number yet. Still, it’s a start! If you want to try actually using or extending it, feel free to contact me and I’ll help you get set up.

Statement of Intent

Tuesday, April 22nd, 2014

I think I have decided that these are the things I most want in a programming language:

    1a. Objects with string keys, which double as dictionaries.
    2a. Closures.
    3a. A way to store a closure in an object and invoke it with the relevant object bound to some variable within the function (“this”).
    4a. Prototype-based inheritance for objects.
    5a. The potential to instantiate a closure, or an arbitrary singleton object, inside of an expression (“lambda”).
    6a. Gradual typing (with calls from untyped code into typed code being protected by something like a check and an exception throw at runtime).
    7a. Two-way communication with C++ (C++ objects may be visible within the language, language objects may be passed into C++ code).
    8a. Two-way communication with Javascript.
    9a. Support for multiple concurrent threads of execution, with interaction between by means of message passing.
    10a. It must be possible for me to develop a program on one operating system and execute it on another operating system (i.e. there is a VM or a cross compiler).

And here are some things that I think would be very nice in a programming language:

    1b. Pattern matching.
    2b. Syntax support to allow me to declare curried functions, as transparently as if they were non-curried functions.
    3b. Function invocation with named arguments (i.e. Python keyword args).
    4b. Suspension and resumption of stacks (coroutines or generators).
    5b. Atoms (interned strings which can be used as object keys without paying for a string lookup).
    6b. Properties (i.e. overloading of assignment).
    7b. The language itself should be ultimately defined as a AST, with multiple possible syntaxes reducing down to that single AST.
    8b. Language features (ints, exceptions) should be possible to enable or disable for individual programs or pieces of code. Not all features fit all projects.
    9b. There should be a potential for (arbitrary and user-defined?) type “adjectives” which are assertions that are remembered as part of the type and checked as part of the type checker. (Is this object mutable? Is this object mutable with regard to its set of keys? Does this function have side effects? Are these two properties recursive for the related object/call graph? Does this function ever use “this”? Is this object ever accessed with a non-atom key? Is this integer greater than three?)
    10b. Potential to directly manipulate the scope of a closure (I.E. mappings of unbound variables in the function body) after function declaration.
    11b. The collapsing of functions and objects. (“Look up key ‘f’ on object a” and “Invoke method a with unary argument ‘f’” should be the same operation, same syntax, same behavior. Function scope and prototype chain should be, insofar as the user can tell, the same mechanism).
    12b. Customizable automatic casting rules (if we have 1b, applied by a user-selected pattern-matched function).
    13b. Syntax support for object “classes” that have the familiar semantics of Python or Java. Ideally, classes would just be a pattern implemented on top of prototypes, but the syntax should make this pattern easy and the class identity should be visible to things like debuggers.
    14b. Some simple user-configurable syntax redefinition (like the ability to define an operator ^ where invoking 3^4 should be actually interpreted as exponent(3,4), and ^ has a user-defined operator precedence, etc).
    15b. The ability to have some sort of object which is ultimately represented at runtime as a packed memory buffer, such that saying “set element 4 of x to 3″ is actually writing “3″ into a specific well-defined byte or bytes in memory. (This is a useful thing both because of performance optimizations it makes possible, and for interacting with hardware like GPUs which only understand byte streams.)

I would like to explore ways to get to a point where I have a programming environment that satisfies everything from the A list and some emotionally satisfying number of items from the B list. In seeking such a language, there are incidentally some conditions that any language I spend time using will have to satisfy. I don’t want to have an argument with anyone about this final list; just think of them as personal preferences.

    1x. It must not require me to program with S-expressions.
    2x. It must not require the use of the JVM or CLR. (I am potentially willing to compromise on the CLR.)
    3x. It must not force me to program in a way which avoids the use of side-effects or state.
    4x. It must not require me to use any closed-source compiler, core library, or development tool.

No language currently exists which provides everything on my A list. Some of the items on the B list, particularly 10b and 11b, do not exist in any language that I am aware of. Most of the most interesting items on the B list only seem to exist in languages which fail one or more items on the X list. Any currently existing language will be a compromise. How fair do these compromises look?

If I use Lua, I am happiest. Lua gets me 1a, 2a, 3a, 4a, 5a, 10a, 4b, 6b, and 13b, 15b if I’m using Luajit, and since I have 5a I can awkwardly approximate 3b. No language I know of does 1a better than Lua. Unfortunately, 7a and 9a, which Lua does *not* support, are two of my *most* important conditions– I literally can’t develop without those two, whereas missing 1a through 5a will merely make me sad.

    (Lua gets something close to 7a– two-way communication with *C*– and two-way communication with C can be used to construct trampolines that *approximate* communication with C++, but in my experience using this technique is so frustrating I would rather avoid Lua altogether than attempt it again. There are also incidentally projects which purport to offer 8a for Lua, but I have not seen these projects actually working and I believe that they would need additional development before being actually used.)

If I use Javascript, I do get 1a, 2a, 3a, 4a, 5a, 8a obviously, 10a, and 6b. If I am using Javascript ES6, I also get 9a, 3b and 4b. Actually, Javascript does pretty well against my lists. Unfortunately, some of the personally important items on the list Javascript satisfies, it messes up quite badly. For example it offers 4a, but uses a baroque method for defining an object with a prototype, and it does not offer any way when overriding a prototype method to invoke the “super” method. That last one’s pretty huge; “super” is a basic feature of all object systems which, since it is present in Self, I would argue without “super” you don’t have a prototype-based inheritance system at all. On similar lines, Javascript offers 2a but essentially ruins it by not (before ES6) offering proper block scope; if I can’t readily create a closure in (for example) a loop, that is so awkward I probably won’t ever use closures at all.

There is another problem with Javascript: It is an unending cavalcade of horrors. Javascript feels *shoddy*. Many extremely basic language features are riddled with weird exceptions, and exceptions to exceptions, such that you never feel certain what the code you have just written does; many kinds of simple typos result in a silent failure or unexpected behavior rather than any kind of error. (The type coercion rules alone would feel like a great argument against using Javascript even if everything else about the language were perfect.) I very much like what is in this toolbox, but the individual tools all feel as if they will fall apart in my hands if for example I hit anything too hard with that hammer. This shifting-sand problem is exacerbated by the dramatic variance between Javascript versions and implementations; the primary benefit of Javascript is its huge installed base, but only a small subset of the syntax is “safe” in the sense that an acceptable portion of your installed base is compatible with it. This is an unfortunate property when we are talking about mere syntactical convenience features: 3b and 4b are nice, but I would not literally give up users to get them.

    (I have not investigated if Javascript satisfies 7a).

If I use C++, I get 4a (or rather 13b, class-inheritance alone, which is close enough for me), 6a (not gradual typing but strong typing, which I don’t mind), 7a (sort of– assuming you never dynamically link against anything from a competing C++ compiler), 9a (with memory sharing instead of events, but you can add the events via a library), 10a (with *great* difficulty), 15b (although since 15b is *all* C++ offers for data storage, it isn’t very pleasant), and an extremely limited 14b that can be used to get 6b (although only for some kinds of variables). There is also a horrible, confusing mockery of 12b which really the language would be better off if they had not tried to offer this feature. If I use C++11, I get 2a, but it’s gross (the closures capture variables manually, rather than automatically; they’re only created if you use the “lambda” feature, which is totally separate from any other kind of function or method; and the syntax is incredibly ugly). C++11 also has “auto”, which is… one step closer to my ideal form of 6a than strong typing alone would be, I guess. Overall C++ and C++11 don’t offer *anything* exactly the way I’d most prefer it, and the only items on my list I’d even feel C++ is really “good enough” at are 4a/13b and 9a.

    (At this point one notices something interesting about my criteria 7a: Not even C++ does very well against it. This is a hard criteria to meet.)

I’ll leave it as an exercise to the reader to test my criteria against Python, ML, Haskell, and Erlang.

Looking at my options, I feel that if I could somehow hack Luajit to offer 6a and 7a and also support Lua Lanes (which provides 9a), I’d have a language I were totally satisfied with. I spent a while seriously considering doing just that. However, Luajit is big, and 7a in *particular* is such a hairy thing to implement that I wouldn’t want to attempt it in someone else’s codebase.

So I think if I want a language that satisfies all of these things, it would have to be a new language. Along those lines, I have written a follow-up post here.

Double Union Game Jam recap

Sunday, February 23rd, 2014

Yesterday I cohosted with a woman named Snail a game jam at Double Union, a feminist hacker space in San Francisco. The game jam was targeted at women and gender-nonbinary people and the goal was to reach out to people who might not be participating in SF’s existing indie/gamejam culture. We opened with a class on Twine I taught (my class notes are here)and then ran for about four hours before showing what we’d made. It seemed to go really well! We got about 12 people plus the normal Double Union Saturday crowd, and every single person who stayed to participate wound up making a game.

Here’s some of the games we made– all of them can be played in a browser:

Gina made A Grim Task, a really intense short story about volunteering at a nonprofit.

Haley made “LET ME TELL YOU ABOUT LIL B THE BASED GOD”. The name describes it pretty well.

Wonja made two games:
Lizard Quest, an INCREDIBLY AMAZING adventure game.
Hallway, a, uh… experimental experience.

Kanane made The Huntress, an impressively polished fantasy story.

I actually made a couple small things!:
San Francisco Quest, which I actually made during the Twine class at the beginning as a demonstration of how to use Twine. It’s very very short, probably the smallest a game could legally be without getting into trouble with the Game Police.
I then *tried* to make a Pacman clone, but it took me a really long time to fix some bugs where I couldn’t get anything at all to draw (it turns out C++/OpenGL is a really bad choice for a four-hour jam). By the time I got that fixed, I was out of time, so I couldn’t add anything like interactivity. So I decided the result was Social Anxiety Pac Man, a game where all you can do is stand in a maze immobilized by your choices. (The link goes to an image, not the game. Trust me, it’s the same experience.)

Snail, my cohost, made Another City, a really funny hallucinatory story that to me feels reminiscent of San Francisco Quest only better. She says this link is a “demo” she’ll finish later, some of the paths don’t complete.

Seanna made How To Be Successful. This one is actually really cool, it’s a Twine game consisting of nothing except pictures.

Alicia made the one non-Twine game of the jam, Make the kitties happy!. This one is totally adorable, you have some cats and you have to make them all happy at the same time by giving them what they want. Hint: You can change the tool selected (at the top) by clicking on it. (You may have to run this in Firefox.)

Jenn made Lady Bird and Tiger Bear, a short fairy tale.

And our two last participants, Tyler and Hannah, made complex Twine stories about, respectively, memories you can hold in your hand and a epic struggle to make yourself a sandwich. These haven’t been posted on the Internet yet but I played the sandwich one and it was really good.

Overall I’m really happy with how the jam turned out. We talked at the end and we are planning to do this again at minimum and hopefully expand this into a larger recurring women+nonbinary game making group. There seemed to have been more people who were interested in participating but couldn’t because of the limited space at DU, so we want to give them a chance to do that next time. Our current plan now that the jam is out of the way is to start doing a monthly Twine writer’s group at Double Union, and in a couple months do another game-making-class+gamejam like the one yesterday. Next time we’ll be teaching Stencyl. We should have more details soon, so keep an eye out!

A Game of the Year 2013 Poll: Results

Tuesday, January 14th, 2014

CLICK HERE TO JUMP TO THE PRETTY COLOR-CODED FULL RESULTS

This explanation will look a lot like that of previous years, but:

Every year since 2004 I’ve been hosting this Game of the Year poll for the users of some forums I read. There are a lot of GOTY polls out there, but this one I think is kind of special. Most polls, you’re given a list of four or five options and you’re asked to pick the one you liked best. This poll, people are given a list of a couple of hundred options, consisting of every new game released in the previous year– and asked to rate their top ten or twenty.

This does a few interesting things. First off, we get to see all the information about what people’s second, third etc choices are. Second off, because the second, third etc choices count, people are more likely to vote for the game they want to win, rather than the game they think is likely to win– they’re less likely to engage in “strategic voting”. Finally, because we have all this information, we’re actually able to provide somewhat reasonable rankings for something like the top hundred or so games of last year.

The full results– showing the exact number of voters who ranked each game first, second, third place etc– can be found here. In the meantime, the final results were:

  1. Gone Home (3426) *** GAME OF THE YEAR ***
  2. Bioshock Infinite (3373)
  3. Papers, Please (2506)
  4. Saints Row IV (2497)
  5. Tomb Raider 2013 (2490)
  6. The Legend of Zelda: A Link Between Worlds (2440)
  7. Pokémon X and Y (2423)
  8. The Last Of Us (2263)
  9. The Stanley Parable (2187)
  10. Fire Emblem: Awakening (2161)
  11. Animal Crossing: New Leaf (1924)
  12. Grand Theft Auto V (1721)
  13. Assassin’s Creed IV: Black Flag (1631)
  14. Brothers: A Tale of Two Sons (1554)
  15. Rogue Legacy (1547)
  16. Super Mario 3D World (1393)
  17. Antichamber (1334)
  18. Gunpoint (1302)
  19. Metal Gear Rising: Revengeance (1214)
  20. Kentucky Route Zero (1130)

The numbers in parentheses are the final scores each game got under the poll’s ranking system. Thanks if you voted, and some more elaborate analysis of the results (plus an explanation of the scores) can be found below.

NOTEWORTHY WINNERS

GOTY 2013:

#1, Gone Home

Top-ranked PC Exclusive:

#1, Gone Home

Top-ranked 3DS Exclusive:

#6, The Legend of Zelda: A Link Between Worlds

Top-ranked PS3 Exclusive:

#8, The Last Of Us

Top-ranked WiiU Exclusive:

#16, Super Mario 3D World

Top-ranked Browser Game:

#24, Depression Quest

Top-ranked Mobile Exclusive:

#46, 868-HACK
(OR)
#56, Ridiculous Fishing, if you disqualify 868-HACK because of its PC prototype, “86856527″

Top-ranked Vita exclusive:

#67, Tearaway

Top-ranked PS4 Exclusive:

#85, Resogun

Top-ranked 360 Exclusive:

#104, BattleBlock Theater

Top-ranked XB1 Exclusive:

#124, Killer Instinct

Top-ranked Ouya Exclusive:

#166, Towerfall

Top-ranked Wii Exclusive:

#189, Pandora’s Tower

Top-ranked FPS:

#1, Gone Home

Top-ranked “Indie” Game:

#1, Gone Home

Top-ranked RPG:

#7, Pokémon X and Y

Top-ranked Sports Game:

#45, Divekick

“Cult” Award (see below):

#46, 868-HACK

NOTEWORTHY LOSERS

Best game of 2013 which somehow nobody considered to be their #1 pick: #30, Guacamelee!
Worst game of 2013 that at least one person considered their #1 pick: Three-way tie between three games tied for the #326 slot: “I Hate the Dark”; Wizardry Online; and “Heroine’s Quest: The Herald of Ragnarok”. Each of these games got only one vote, but each of these voters considered it their game of the year.
Worst game of 2013: Two-way tie between the games tied for #402: “Composition 62″ and “Ultionus: A Tale of Petty Revenge”. Both of these games scored only one vote each, each from someone who considered it their 20th best game of the year.

There were a whole 57 games on the nominations list that no one voted for at all.

ALTERNATE SCORING METHODS

The rankings listed above are based on a version of the Borda count voting method. Each vote cast for a game gives that game a certain number of points. If someone ranks a game #1, that game gets 20 points. If they rank it #2, the game gets 19 points. If they rank it #3 the game gets 18 points… and so on. I have a script that checks a couple of alternate ways of ranking the same data, though.

For example, if we rank games only by the number of first place votes they got, the winner remains the same but almost the entire rest of the list changes dramatically– a lot more movement than usual this year, it seems like. I bolded entries that are different in the first-place-votes count:

First Past the Post

  1. Gone Home (52)
  2. The Last Of Us (51)
  3. The Legend of Zelda: A Link Between Worlds (38)
  4. Saints Row IV (35)
  5. Bioshock Infinite (33)
  6. Fire Emblem: Awakening (27)
  7. Papers, Please (23)
  8. Pokémon X and Y (23)
  9. Grand Theft Auto V (19)
  10. Dota 2 (19)
  11. The Stanley Parable (17)
  12. Animal Crossing: New Leaf (17)
  13. Kentucky Route Zero (16)
  14. Metal Gear Rising: Revengeance (15)
  15. Tomb Raider 2013 (14)
  16. StarCraft II: Heart of the Swarm (12)
  17. Assassin’s Creed IV: Black Flag (11)
  18. 868-HACK (10)
  19. Super Mario 3D World (9)
  20. Hate Plus (9)

Most years when I look at the first-past-the-post list a “cult” game emerges that received very few overall votes, but where an overwhelming percentage of those votes were #1 votes (I think of this as the “Persona award”); this year the standout was 868-HACK, which managed to grab #18 in the first past the post rankings despite being all the way down at #46 in the overall rankings. Also of note here are Hate Plus, which jumped from #33 to a tie for #19; and DOTA 2, which jumped from #22 to #10; and The Last Of Us, which jumped from #8 to #2 (actually, if it had received one more #1 vote, it would have tied Gone Home for first place).

I also did two more ways of sorting the rankings: an “approval” vote, where nothing is counted except the number of votes a game received (i.e. a first-place and a twentieth-place ranking count the same– all the matters is if the game was on someone’s list); and an instant runoff vote. Usually these two track the main count very closely, but this year, something rare happens in IRV: The first and second place games switch place! If you are qualified to comment on the differences between instant runoff and Borda-based ranked voting, feel free to tell us what that means.

APPROVAL

  1. Bioshock Infinite (223)
  2. Gone Home (205)
  3. Papers, Please (176)
  4. Tomb Raider 2013 (162)
  5. Saints Row IV (157)
  6. Pokémon X and Y (155)
  7. The Stanley Parable (149)
  8. The Legend of Zelda: A Link Between Worlds (144)
  9. Fire Emblem: Awakening (140)
  10. The Last Of Us (136)
  11. Animal Crossing: New Leaf (125)
  12. Rogue Legacy (117)
  13. Grand Theft Auto V (113)
  14. Assassin’s Creed IV: Black Flag (108)
  15. Brothers: A Tale of Two Sons (107)
  16. Gunpoint (98)
  17. Antichamber (91)
  18. Super Mario 3D World (84)
  19. Don’t Starve (83)
  20. Far Cry 3: Blood Dragon (82)

IRV

  1. Bioshock Infinite (223)
  2. Gone Home (205)
  3. Papers, Please (176)
  4. The Legend of Zelda: A Link Between Worlds (144)
  5. Saints Row IV (157)
  6. Pokémon X and Y (155)
  7. The Last Of Us (136)
  8. Tomb Raider 2013 (162)
  9. The Stanley Parable (149)
  10. Fire Emblem: Awakening (140)
  11. Animal Crossing: New Leaf (125)
  12. Grand Theft Auto V (113)
  13. Assassin’s Creed IV: Black Flag (108)
  14. Rogue Legacy (117)
  15. Brothers: A Tale of Two Sons (107)
  16. Gunpoint (98)
  17. Antichamber (91)
  18. Super Mario 3D World (84)
  19. Metal Gear Rising: Revengeance (77)
  20. Far Cry 3: Blood Dragon (82)

FINALLY: PER-FORUM BREAKDOWNS

Okay, so this is where things get… interesting.

When this poll first started, it was run out of the forums for Penny Arcade, and historically, that one forum has totally dominated the results. I traditionally link on a couple more small forums, but these usually only provide a handful of votes– and anyway, most of the forums I tended to target were themselves spinoff from the PA forum community. The final results invariably look almost exactly like the PA-specific results.

Except not this year. This year, for whatever reason, my efforts to promote the poll on Twitter took off like crazy; one of the tweets about it got 51 retweets. The result was a HUGE voter influx, to the point where PA was almost outnumbered– in the end PA contributed about 300 votes, whereas Twitter contributed about 250.

And PA and Twitter voted *really differently*. My vote script tracks “where votes came from”, and lets me run results isolated to votes from a particular source. The tracking isn’t perfect, but ought to be able give us some idea how different internet communities voted. Here’s the breakdowns from the different major vote contributors with their respective color-coded listings linked; here’s what we find:

Penny Arcade Forums (296 voters)

  1. Bioshock Infinite
  2. Tomb Raider 2013
  3. The Last Of Us
  4. Saints Row IV
  5. The Legend of Zelda: A Link Between Worlds
  6. Grand Theft Auto V
  7. Assassin’s Creed IV: Black Flag
  8. Fire Emblem: Awakening
  9. Pokémon X and Y
  10. Papers, Please
  11. Brothers: A Tale of Two Sons
  12. Rogue Legacy
  13. Gone Home
  14. The Stanley Parable
  15. Starcraft II: Heart of the Swarm
  16. Super Mario 3D World
  17. Gunpoint
  18. Metal Gear Rising: Revengeance
  19. Ni No Kuni
  20. Guacamelee!
Twitter (244 voters)

  1. Gone Home
  2. Papers, Please
  3. The Stanley Parable
  4. Animal Crossing: New Leaf
  5. Pokémon X and Y
  6. Depression Quest
  7. Saints Row IV
  8. Kentucky Route Zero
  9. Antichamber
  10. Hate Plus
  11. The Legend of Zelda: A Link Between Worlds
  12. Bioshock Infinite
  13. Fire Emblem: Awakening
  14. Tomb Raider 2013
  15. The Last Of Us
  16. Gunpoint
  17. Rogue Legacy
  18. Super Mario 3D World
  19. Candy Box
  20. 868-HACK
Platformers.net (32 voters)

  1. The Legend of Zelda: A Link Between Worlds
  2. Fire Emblem: Awakening
  3. Bioshock Infinite
  4. Rogue Legacy
  5. Pokemon X and Y
  6. The Stanley Parable
  7. Gone Home
  8. Far Cry 3: Blood Dragon
  9. Papers, Please
  10. Cookie Clicker
  11. Shin Megami Tensei IV
  12. Metal Gear Rising: Revengeance
  13. Animal Crossing: New Leaf
  14. Super Mario 3D World
  15. Salty Bet
  16. Phoenix Wright: Ace Attorney
  17. Brothers: A Tale of Two Sons
  18. Grand Theft Auto V
  19. Assassin’s Creed IV: Black Flag
  20. Saint’s Row IV
Tigsource.com (22 voters)

  1. Papers, Please
  2. The Stanley Parable
  3. The Last Of Us
  4. Path of Exile
  5. Brothers: A Tale of Two Sons
  6. Shelter
  7. Save the Date
  8. Monaco: What’s Yours is Mine
  9. Jelly No Puzzle
  10. Risk of Rain
  11. Don’t Starve
  12. SUPERHOT
  13. The Wonderful 101
  14. Starcraft II: Heart of the Swarm
  15. Dota 2
  16. Metal Gear Rising: Revengeance
  17. Towerfall
  18. Samurai Gunn
  19. Far Cry 3: Blood Dragon
  20. Dragon’s Crown

Looking at these breakdowns, the most pronounced difference between the big two voting blocks– Twitter and Penny Arcade– is how they treated the top two entries from the combined results: Twitter put Gone Home at #1 whereas PA put it at #13, and Bioshock Infinite was voted #1 by PA but ranked down at #12 for twitter. One thing that might have hurt Bioshock here is that although Twitter voted overwhelmingly for Gone Home, the PA bloc did not vote for Bioshock Infinite nearly as solidly; in fact, going by “first place” votes alone, PA actually preferred The Last Of Us (39 first place votes to Bioshock’s 23).

The voting differences get even more interesting when we compare the Twitter results to the Tigsource results– the Twitter results seem to show a very heavy influence, and Tigsource is an indie community, but these are apparently slightly different parts of the indie community because a few of the games that made strong showings on the Twitter list (in particular, Gone Home and Depression Quest) didn’t rank at all among Tigsource’s 20 or so voters.

Anyway, that’s it, thanks so much for voting and I’ll be doing this again next year!

Game of the Year 2013: Vote Here

Sunday, January 5th, 2014

Hello anyone out there: I’ve got this Game of the Year poll that I run on some web forums I frequent. The way it works is that you rank your favorite games of the year– up to 20, though vote for as many or as few as you want– and the script will sort out the top 200 or so out of everyone’s votes. Here’s last year’s results if you want to see what this looks like (or previous years here). I will run this poll script for one week and then post the results. If you’d like to give it a try:

Vote here

When one week is up (Jan 12) I will delete this post and post the results here on this blog. Thanks!

Player Piano

Tuesday, November 19th, 2013

For my submission to the SHARECART 1000 project, a simple FM synthesizer/sequencer. When installed with other SHARECART 1000 games, Player Piano will allow you to listen to your SHARECART save files as music.

Note that auto-save on quit will only work if you quit by pressing “ESC”.

Download

How do I figure out the original git revision of a GitHub zip I downloaded?

Sunday, October 20th, 2013

This is a problem I ran into today: Awhile back I downloaded a copy of a project hosted on GitHub, using the “Download Zip” button on the project page.

A few months later I came back, and I needed to know exactly which git revision it was I downloaded. Given this zip file, how does one “go backward” and figure out what revision it was? (The GitHub zips contain just a folder containing a single revision; there is no .git directory in it, so command line git can’t do anything.) I got some help on Twitter (thanks Robert!) and eventually figured out how— but I couldn’t find an explanation anywhere Google had picked up, so here’s a quick summary for future generations.

Method 1: Zip comment

This one’s pretty simple: There’s a thing in the headers of zip files called the “zip comment”, and apparently GitHub stores the original revision hash in there. You can get this out using the command line zip tool and the “unzip -z” flag:

If you’re using Windows, the makers of the WinZip tool claim they display the comment automatically when you open the zip in WinZip.

Method 2: Dates

Maybe you’ve lost the original zip, and you just have the folder? We still have one piece of useful information: the “modified” dates. If you look, all the files in the unzipped repo will have their created/modified dates set to the same date and time:

In fact, this is the date and time corresponding to the exact timestamp on the original Git commit. So we can just go look through the git logs until we find a revision with that exact timestamp:

By the way: If you download a zip from BitBucket, they helpfully actually put the exact revision hash into the name of the downloaded zip, so you’re unlikely to run into this problem. Just so you know though BitBucket git repositories do work with both the “zip comment” and the timestamp rules above. BitBucket mercurial repositories don’t have the “comment” in the zip file, but they do contain an invisible file named .hg_archival at the root level of the unzipped directory which contains the same information and then some.

Exacto: How to copy scenes between Stencyl files

Sunday, June 2nd, 2013

Warning: This has only been successfully tested in one specific case and it could potentially leave Stencyl files unusable. Only use it after backing up the Stencyl projects involved.

So this is a tool with a pretty narrow application, but, I made a thing. If you’re familiar with the (generally pretty nice) game-making tool Stencyl, you may be aware of a curious limitation it has: It is not possible to copy a scene between two Stencyl files. You can import and export assets, or scripts; but not entire scenes. My friend Liz wound up getting stuck on this hard when she was making her game Problem Attic (it’s really good, you should play it); she had split her project file into two, and was experimentally making levels in each. Then she went back to merge them and found out Stencyl won’t let you do this.

To unstick Liz, I wound up making a Python script that just goes behind Stencyl’s back and copies scene data between games. You can find the script, named Exacto, here; click “Download”, or look under “source” (it’s the only file). From our testing, the script appears to work— it was used to make the final version of Problem Attic.

Caveats

The script copies all the data related to a Stencyl scene that I am aware of. There are two problems here though. First off, it’s possible there’s some data I’m not aware of, and which the script therefore misses. Second off, the script does not even try to copy things which are global to the game— such as actors, or assets like tilemaps. So if you want this script to work, you will need to copy those things over first. You will need to separately load all the assets used in game A, into game B.

There’s one more, even more awkward thing though. Remember I said above our test programs were two versions of the same game? Well, all the assets have “ID numbers”. You can’t see them while you’re making the game, but they’re in the game files. I’m not sure, but it’s possible that if you copied between two *very* different games, and you used a lot of assets in the scene you copied, and both games contained the same assets *but* the “ID numbers” were different, then my script might get the ID numbers confused and cause Stencyl to use the wrong actors or the wrong assets or tilemaps after you copy the scene. I’m not sure; I haven’t had a chance to try this. Because Liz again was just working off two versions of one game file, I expect the ID numbers would be the same in both for her anyway.

I’m going into all this technical detail for a simple reason: Stencyl does not deal well when it sees something it doesn’t expect in the game file. Stencyl expects it will only see game files it made and that people like me won’t be mucking around behind its back. So if you copy something into a game file that Stencyl doesn’t understand, it might just crash, or freeze, or maybe you’ll think the copy worked but afterward you’ll find certain windows in the interface just refuse to open. (This is why I am making such a big deal about backing up your files.) So I’m wanting to warn about places where my script might confuse Stencyl, and the assets/asset ID thing is the biggest danger point I know of.

By the way though— if you *do* find a case where running this script left Stencyl unable to work on your game file, I’d be curious to talk to you. If I had more examples to work with, I’d be able to make the script more resistant to bugs.

Usage

Download the exacto.py script from the above URL. If you are using Windows, you’re also going to need to install something called “Python”. Go here and download the “Python 2.7 Windows Installer” or “Python 2.7.5 Windows installer”— it doesn’t much matter which you pick as long as it has “2.7″ in the name and not “3.0″. (If you are on a Mac, you already have Python.) Run that installer.

Close Stencyl, and open up a command line window— on Windows this is “Command prompt” in the start menu, on the Mac this is Applications->Utilities->Terminal, but if you don’t already know how to use the command line this next part might not make much sense to you anyway. Cd to the directory you downloaded exacto.py to. Here’s a few examples of using Exacto– things you can type into the command line:

python exacto.py --help

This has Exacto print out exactly how to use it. There’s a couple small features it lists in the –help but which I don’t document in this blog post.

python exacto.py -i ~/stencylworks/games/gameFROM list

This prints out a list of all the scenes, and their “Scene ID” numbers. By the way, where I say “gameFROM”, put the name of your game.

python exacto.py -i ~/stencylworks/games/gameFROM -o ~/stencylworks/games/gameTO copy "scene one" "scene two" "scene three"

This is what you actually want to do to copy scenes. You’ll want to put the name of the game you want to pull from instead of gameFROM after the -i, the name of the game you want to write into instead of gameTO after the -o, and at the end instead of “scene one” “scene two” put the names of the scenes you want to copy (in quotation marks). Once you’ve run this command, you can open up Stencyl again and see if it worked.

By the way– maybe you run “list” and realize the names are kinda complicated or messy to type? You could also just use numbers. Say you ran the “list” command and you saw the scenes you want to copy have scene IDs 6 and 8. You could say:

python exacto.py -i ~/stencylworks/games/gameFROM -o ~/stencylworks/games/gameTO copy --raw-id 6 8

Notice all these examples are for the Mac command line. If you run on Windows, you want to write it a little different. Instead of saying “python” you want to say “C:\Python27\python.exe”, and everywhere I wrote “~/stencylworks/games/” above you’ll want to say “%APPDATA%\Stencyl\stencylworks\games\”. (The point with this “stencylworks/games” folder is that you need to find the place where stencylworks stores your game’s *directory*– the thing containing the game.xml. The script doesn’t work on .stencylworks files.)

TODO

I’m not sure if I intend to do any more work on this— it depends on how much interest there is. But some things I think would be worth exploring in a future version of this script:

  • It’s probably pretty easy to make a Windows GUI so you don’t have to do all this downloading-Python/command line junk. cx_Freeze or something.
  • Fix for the “ID numbers” thing I’m worried about above?
  • It’s probably possible to copy assets and actors and stuff! I just haven’t looked into it. At least, it’s probably pretty easy to *check* if the assets are present in the destination file, and if they aren’t, complain and refuse to copy, instead of just going ahead and breaking stuff.
  • It would be pretty easy to be able to operate on .stencylworks files, or even make new .stencylworks files. They’re just zips.

Anyway

Anyway, good luck, and if you try this script out, I’d be curious to hear how it went (whether it works or not!).