Monday, March 7, 2011

On writing your own programming language

In about three days, you can push out your own programming if you are a master at recursion, understand parsers, understand tree transformations, and have a good grasp on the goals of programming.

Problematically, a programming language is a small thing. The next step is the tools and the platform. This will take about one to three years to build.

Now, once the platform is built, it will take about two to ten years to market it provided the right people pick it up. This how Ruby went from obscure to mainstream, and this is why I give DHH mad props since he put a huge surge into Ruby.

I had spent about 1.5 years working on my platform for my programming language, and I had shipped web properties on it. Then I killed it.

I killed it because I realized the world didn't need yet another shitty programming language. It was categorically shitty because I didn't build it to be marketable and the problem it was solving was kind of silly to the market place and my ability to support it.

Basically, the idea was to build a programming language around a statically typed schema and then provide a language integrated query language similar to SQL in such a way that it could automate scalability.

It's extremely intellectually satisfying problem to work on, but it was a poor use of my time. However, I believed (and still believe) that the problem is worthwhile. It was intoxicating to work with statically typed SQL with a statically typed schema and then solve the induced persistence problem with replication and automated infrastructure management tools.

Over all, the experience was wonderful. I recommend anyone capable or willing to start to write a programming language. Once it is built, then you will see a lifetime of work ahead of you since it will never be finished.

Lessons Learned.

Working on the problem lead ultimately to dealing with replication at a serious level. Replication is non-trivial, and the CAP theorem is a harsh mistress. It's very easy to get going and then the shit will hit the fan, and then you are screwed.

Building a taxonomy of how different patterns in SQL lead to different types of systems gives a greater appreciation and understanding of the NoSQL movement. I am fully confident that in the next five years, we will have some very interesting solutions that enable a agile database movement. The entire process also gave me insight into why SQL really does suck and is very limited.

I also built my own map reduce framework which gave me a greater understanding of how to solve problems with MapReduce. I basically did a differential on a "join space". I would use relations to build a "join space" by using joins and then map a function on the changes. This looked very similar to how you can do MapReduce in CouchDB, but mine sucked. Actually, mine was terrible in comparison.

I invented my own heroku like interface for launching back-ends. It's fairly easy with AWS to automate your entire infrastructure management. I'm kicking myself for now launching a product off of it. It's wicked cool to push a button and have three MySQL servers pop up and connect to each other. It's even more wicked cool to see shards re-balance and push data into the new servers.

I added closures to PHP in a way that enabled you to serialize the closure to Amazon's SimpleQueue. While it was only a true closure in memory, it was very neat to add asynchronous processing to PHP. A simple C + curl driver would launch and basically run the event loop in apache2.

Documenting a programming language's more esoteric features can be hard since you have to teach new ways of thinking. This is why I generally avoid a DSL in most production environments. Generally, the process of building the DSL will give you enough insight to make a rock solid library.

People really hate static typing, and if you want to build a statically typed programming language, then it is going to cost a lot of money. Most of the commercial used statically typed programming languages have companies behind them with deep pockets; I don't think this is a accident.


  1. Not me. I enjoy a statically typed language. It just feels right.

  2. @XML Home, it does feel right, but it isn't a popular choice. :/

  3. Nice.. I hope you would be inspired for next version of your platform). Important to believe that it won't be shitty :)