Betting the Farm on JavaScript

So, I can't sleep and what am I thinking about? Type Systems...

Two years ago, I would have cursed the dynamic typing landscape as fools and miscreants because I'm a performance junkie. I like my code to run insanely fast. This is my typical salvo in any type system argument.

Now, I realize that I was just being closed minded and rather stupid (and even more so ignorant). Consider this JavaScript code.

function (a,b) { return a + b; }

What is its type? Well, if I may bastardize OCaml type notation for a second, then I would say something like:

(a',b') -> c'

What the fuck is c' ? The reason I don't know what c' is because JavaScript's + operator is overloaded in a poorly defined way that is plain stupid from a type inference perspective. This question combined with my lack of creativity ultimately lead me to the conclusion that JavaScript is doomed to win the performance argument.

Well, JavaScript is doomed until you run the program for significant lengths of time and realize that we have multiple cores that are going to be wasted on consumer machines. Let me explain this black magic that is bouncing around in my head.

Type Inference in JavaScript

Given a function like the above, I can't type it in a meaningful way unless I know the type of the inputs. However, when the function is evaluated I have the types and the return type can be established. If I utilize just in time compiling, then I can use type inference to produce types and compile a native version of the code given the correct type signature.

Type inference is fairly expensive (at least, the ways that I've solved it) but is fortunately 100% cpu bound. It seems perfect to fork off into a queue for some poor core to solve.

My suspicion is that this is what JaegerMonkey is doing, but I'm rather ignorant of its code. At least if they are not doing it, then I can add it in as a patch If I suddenly find myself with a wealth of time on my hands.

Future of JavaScript

I wager that JavaScript will enable C level optimizations by learning how to game the next generation of JavaScript environments. I look forward to optimizing my JavaScript libraries to have tight inner loops.

I believe in this future because there is commercial interest in enabling it.

This is why I'm very optimistic about node.js and the future of server side JavaScript. Sure my node.ocaml is three times faster than node.js now, but it is just a matter of time until node.js catches up.

It would also be a good time to take v8 or a build of JaegerMonkey and start building a game engine with it and provide a canvas/WebGL compliant C back-end.

Also, if I was back in academics, then this would be the perfect topic for a PhD dissertation.

API as your Queen (or how I feel in love with NoSQL)

I like to think and play around with software architectures. About 1.5 years ago, I was inspired by twitter. Or, more precisely, I was inspired by the twitter developer community and the developing mash up scene. I have also watched the growing number of NoSQL solutions. The two are connected.

Developers are performing joins in the application space (or they are not doing joins at all).

Joins are the most important aspect of the relational database model, and if we don’t need them at the data tier any more, then do we need a relational database?

The answer is "it depends", but you don't even want to start there. You want to start by building an API that enables the product to work. This is where I start as I can get a good flow of the ebb and flow of data.

With node.js, node.ocaml, or KayakHTTP, it is very easy to create a REST-ish server for enabling developers to build products against a back-end. You can also see how much of a back-end you really are going to need and which products you will want to utilize in the future.

Always start by building your product as an API.

When you do this, your API is now a NoSQL back-end. You would be surprised what people can do when they don't have to deal with SQL.

Green Computing; do more with less

It seems these days that anyone can do "real-time" internet these days due to awesome technologies like node.js or node.ocaml (shameless self promotion). Realistically thou, no one cares if your software is optimized in C using pointer arithmetic to squeeze an extra hundred requests per second out of a box. Very few care that one has taken the time to read the Black Book. Why? computers are really fast and many people have figured out how to make software scale out horizontally. Computing resources are way cheaper than developer mind-share, so where is the motivation for business?

Well, that's a good question.

This can be addressed with "green computing" marketing campaign. Unfortunately, It would be ultimately doomed out of the gate.

If I were to write another IDE (or, being too productive and unwise)

So, two years ago, I wrote an IDE for a collection of tools that me and some friends used to build an awesome Ajax site. It was madness!

This is what it had:

• Text editing (standard stuff like syntax coloring, color based find and replace)
• Gui editor (box manipulation and property manipulation, pixel perfect editing)
• Locking Source Code via Live Server to edit against
• Cross Machine Synchronization (that is, drag a file from your local computer and it gets uploaded to the remote computer and changes are synchronized as the local file is changed = very cool for designers)
• A big compiler that did a lot of grunt work (and a lot of automated pre-optimizations, so it automated evil)
  
• 100K lines of code.
  

And, all the IDE was used for was to manage the input to "the compiler". The compiler took all the css, validated it, turned all the parametric css (think Less except Turing complete and with an image library to splice and make buttons really easy to stamp out) into images and css, optimize all the image file names, optimize all the images (using optipng and some custom algorithms that removed superfluous colors that don't add much value), took the state machines and combined it with the JavaScript code, compiled the GUI into JavaScript, brought in the JavaScript kernel, and it would spit out one JavaScript file, build an RPC library for JavaScript to communicate to the PHP backend, one (or two) css files, and all the images (some in sprite sheets) per "application".

It was a revolutionary Ajax platform, but it also sucked for reasons I'm not going to discuss.

If I were to do it all again, here is what I would do differently (and kinda doing now).

1. Write the GUI editor as a stand alone editor that could be launched from command line.
2. Not use XML at all for the Gui file format; XML was a terrible choice. Now, I would use JavaScript to construct the object and transfer it to the host program as JSON which it could then trivial serialize out as JavaScript code that merges nicely. Plus, it would be hackable by a text editor.
3. Open source the custom template engine / (Or use Mustache)
4. Write the state machine compiler (the thing that made Ajax real easy for us) as a library rather than a DSL; open source it. (The closest thing available now is zef's mobl which I would use now rather than invent my own).
   
5. Not worry about image file names, rather just map out all images via CSS and then use a CDN.
6. Skip GUI editing entirely, it isn't needed and pixel perfect is over-rated.
7. In locking SCM versus distributed SCM, the key to distributed is designing your code and architecture in a way where merging is obviously the correct choice. If you design your code and how your code works together with other developers in a way where merging will work, then merging will work just fine.
8. Not build a custom file system/custom source control. I kid you not. I have a tool called fire that lets me very quickly build a document object model that serializes to giant XML files. I then used some old networking code to turn the DOM into a server where developers would checkout bits and pieces of the DOM to edit it. Every change coming from a developer would then backup the entire XML file. Once the XML file grew to 1MB, I was going through gigabytes of data per hour.
   
9. Open source the image tools (they are coming)
10. Open source the JavaScript tools (or use Google's Closure)
11. Open source the kernel (or use jQuery more effectively)
12. Open source parametric CSS (Or, add image manipulation to Less)
    

What I had built was a web operating system, and I plan to dump the code out there some day. If you want to dig through the code some time, then send me a shout out and I'll expedite the dump (even thou it sucks).

I would love to revisit the problems we were facing, but the fundamental problem with building a web operating system is the user/developer education debt is currently infeasible.

More over, I don't have enough energy to maintain that kind of code base without at least five long beards. As I age, I'm looking more to polishing things up rather than hacking things out. Building an IDE was a great way to learn and make mistakes, but it was also misguided as it prevented be from adding value to the project and the business. Although, I enabled my team to do marvelous things that are still not being done in the marketplace of products.

Learn about the project so awesome that I created an IDE for it: @killer startups. on mashable. Our youtube video (holy crap, we built this!).

Watching that, I realized how awesome the tech and idea was. Maybe we should revive it? I don't know.

On IDEs (A refinement of my 30 lessons)

In my last post on 30 lessons, the most contentious lesson was on IDEs. I would like to clarify a few things.

IDEs in general, are a good thing. For instance, Visual Studio 2010 is by far the best IDE on the planet IF you use C#. Eclipse is arguably the best (and I'm not going to hold myself to this, so feel free to disagree since I don't use Java anymore) Java IDE out there. For languages like C# and Java, you really do need an IDE; namely because they go beyond just editing and introduce language level extensions like re-factoring and code completion.

Now, that I've given praise to IDE land; I'll tear it down.

root

I deploy and primarily work with Linux stuff. For me, if you want sudo/root access at my company, then you need to be able to use either vim or nano in a highly proficient manner (you also need to be able to use scp). I believe that if you can't do this, then you are not a computing professional worthy of the power of root. Why? there is no IDE for /etc where all configuration lives. Sometimes, something is configured wrong for a new use-case and we have to do something non-trivial in real-time to a live production server. After-all, real men use root.

New Stuff

What IDE existed for Ruby when it came out? or JavaScript? or Haskell? If you find yourself limited by the IDE, then you are going to miss out on new technologies and the ideas they offer. At the end of the day, comfort with "primitive tools" enables you to work with new innovative technologies.

Productivity?

I'll concede every productivity claim made my pro-IDE developers. Here is the thing: what are you being productive at?

Nomadic Lifestyle

A long time ago, my computer crashed and I lost about 2 months of work. I had to reinstall windows and my IDE (Visual Studio 6). I made a lot of modifications to it to suite me. They were lost, and I was lost. I didn't like the feeling that I was dependent on customizing the world to suite me, so I decided then that I would never customize software again. If I needed to tweak anything, then I was doing something wrong. Having lived this way for a while, I can say that Life is pretty good this way. I spend more time on my work than messing around.

Although, recently, I must admit that I've changed the background in my Ubuntu install; that's about it thou.

Cluster Editing

I have a cluster of 32 computers running, and I use pico to code on it. I simply use clusterssh to edit them all at once. What IDE can do this?

fin

Ok, I'm done bashing IDEs.

30 lessons learned in computing over the last ten years

In looking at the last ten years of my life, I realized that I've learned many things. Mostly about how wrong I've been, and how stupid I've been. So, having looked at the 80+ projects I've worked on in the past ten years (excluding coursework, current start-ups, and graduate studies), I have reduced what I learned to a blog post. (In bullet format no-less).

• If you plan to write a programming language, then commit to every aspect. It is one thing to translate between languages; it is an entirely different effort to provide good error/warning messages, good developer tools, and to document an entirely different way of thinking. In writing Kira, I invented a whole new way to think about how to code, and while much of it was neat to me; some of it was very wrong and kinda stupid.
• Geometric computing is annoying, always use doubles. Never be clever with floats; floats will always let you down. Actually, never use floats.
• Lisp is the ultimate way to think, but don’t expect everyone to agree with you. Actually, most people will look at you as if you are crazy. The few that listen will revere you as a god that has opened their eyes to computing.
  
• If you plan on writing yet another Object Relational Mapper, then only handle row writing/transactions. Anything else will be wrong in the long term.
• If you want to provide students with a computer algebra system, then make sure they can input math equations into a computer first.
• Don’t build an IDE. Learn to use terminal and some text editor. If you need an IDE, then you are doing something wrong. When you master the terminal, the window environment will be cluttered with terminals and very few “applications”
• Learn UNIX, they had 99% of computing right. Your better way is most likely wrong at some level.
• Avoid XML, use JSON. Usage of text formats is a boon to expressiveness and the fact that computing has gotten cheap. Only use binary based serialization for games.
• If you plan to build an ORM to manage and upgrade your database, then never ever delete columns; please rename them.
• Never delete anything, mv it to /tmp
• Never wait for money to do anything; there is always a place to start.
• Optimize complexity after people use a feature and complain. Once they complain, you have a real complexity problem. I’ve had O(n^3) algorithms in products for years, and it didn’t matter because what they powered were not used.
• Text games can be fun too; if you want to write an MMO, then make a MUD. You can get users, and then you can use that to get traction to build something bigger. Develop rules and a culture.
• Don’t worry about concurrency in your database until you have real liabilities issues.
  
• Backup every day at the minimum, and test restores every week. If your restore takes more than 5 minutes of your time (as in time using the keyboard), then you did something wrong. If you can’t backup, then you have real issues and enough money to solve them with massive amounts of replicationg.
• Never write an IDE; it will always be a mistake. However, if you do make it, then realize most people don’t know that silver bullets don’t exist. You can easily sell it if you find the right sucker; this will of course become a part of your shame that you must own up when you die.
• JavaScript is now the required programming language for the web; get used to it. JavaScript is also going to get crazy fast once people figure out how to do need based type inference. Once JavaScript is uber, learn to appreciate the way it works rather than map your way of thinking to it.
• Master state machines, and you will master custom controls. Learn enough about finite state machines to be able to draw pictures and reason about how events coming into the machine affect the state.
• There is more value in learning to work in and around piles of crappy code than learning to make beautiful code; all code turns into shit given enough time and hands.
  
• If you want to build a spreadsheet program, then figure out how to extend Excel because Excel is god of the spreadsheet market.
• Write five games before writing a game engine.
• Debugging statistical applications is surprisingly difficult, but you can debug it by using R and checking the results with statistics.
• Don’t design the uber algorithm to power a product; instead figure out how to make a simple algorithm and then hire ten people to make the product uber.
• Learn to love Source Control. Backups are not enough. As you age, you will appreciate it more.
• Communicate to people more often, don’t stay in the cave expecting people will know your genius. At some point in your life, you will need to start selling your genius.
• Realizing that every product that exists solves some kind of problem. Rather than dismissing the product, find out more about the problem the product is trying to solve. Life is easier when you can look at new technology and find out that it does solve.
• Learn to be sold. Keep the business card of a good salesman. Sometimes, they actually have good products, but they are always useful.
• You can make developers do anything you want. Normal users on the other hand are not so masochistic.
• If you are debating between Build or Buy, then you should Build. You are debating which means you don’t know enough about it to make a sound decision. When you build, at least you will get something working before you find what to Buy and how to design with it.
• You will pay dearly for being prickly; learn to be goo and flexible to the changing world. Be water, my friend.
  

If you got to this point, then good job. The biggest thing I have learned (and probably the most painful) in the past ten years is how to deal with my ego. Ego is supposedly your best friend, but it also your worst enemy. Ego is a powerful force, but it isn't the right force to use. While I admit that I've used ego to push myself in very positive direction, I think I would have been better off if I didn't as the side effects trump the pros.

last ten years (if I uploaded it to github)

I've gone through about 10 years of backups looking, and I thought about putting it all up on github. However, most of it would just suck. So, instead, I'll put up gravestones for each of them and not look back anymore. Then, I'll delete the files and be done with. Keep in mind, most of these are side projects. Only two are failed start-ups (Hurox and FileSharingAccelerator). This list excludes current strategic technologies I've developed at my current business.

kitchen	a game engine
grill	statically typed clone of JavaScript
juknow	a JavaScript to PHP converter
kapowie	an OpenOffice Spreadsheets to PHP converter
blaze	An animation language for doing periodic animations (very similar to these devices that do music looping)
iknife	a vector editor in C#
diffistory	an algorithm that would reconstruct a new version of a file using aspects from n-versions of a file
cosmic pipeline	a re-invented of unix pipes on Windows (decided just to switch to unix for that problem)
simpledepend	a php dependency analysis toolkit; using a standard procedural style, you could produce libraries and end-points that would have no includes and consume minimal memory.
scripture	a documentation language
istate	a network synchronization language; describe data-structure and then use them it would sync over the wire
butcherblock	3d object modeler and game editor
cauldron	3d rendering engine (pre-kitchen)
fork	2d gui editor/texture atlas packer
fryingpan	3d physics wrapper library (physx, bullet, ode)
zmlc/fire	heap modeler for structure serialization
inferno	better version of fire
bnj	Bayesian network tools in java 3.x
gem	graph editor and modeler
rpgengine	a player system with buffs
arcane works	a functional programming language version of excel
melchior	a new IDE for web editing with symbolic editing
alienautopsy	world of warcraft bot
convexbodyplayground	given any closed model, this broke it down into convex bodies
facebookwalk	crawl facebook and convert image emails into text/stalk students
researchfu	cms with mathml support
phpmathml	a mathml to png renderer
tutorcas	computer algebra system
mathgraphics	a point set visualization language
sliceem	a 3d model by using contour levels
gg+	gui graph + (open scene graph gui system)
mathmyway	a php-like language that enabled tutorcas to come alive on the web
monte carlo localization	construct a map and locate yourself (robotics)
jovian katana 1/2	ajax ide, gui editor (used to build hurox)
hurox	social marketplace
frame compiler	given a single image, split it up into 9 images for building frames (generates php code)
image2php	convert an image to php (useful for combining with given colors to change hue on the fly)
javascriptmaster	a javascript parser that enabled me to search code to find inefficiencies and problems.
evolution4k	a 3d space game
progressive mesh demo	an implementation of Hoppe's progressive mesh stuff
spherex	a physics engine where gravity was opposite (everything repelled)
zengine	marching cubes optimization where space is a grid represented by link lists of open and closed space
zterrain	yet another terrain engine
jove3	next generation ajax platform
spherebands	yet another collision detection engine
primgen	marching cube implementation for building models from math equations
lotr-risk	scanned all the game assets and made it work in a network environment
rt-canvas	a real-time (comet based) canvas toolkit for charting streaming data
tilemake	generates 16 images from 3 images using reflections and rotations
nova	javascript based router/topology for web page linking
gravity	a shared file system for keeping replicas in sync (with distributed locking)
mercurial	a simple deployment system (was completely ignorant of the scm)
kira	programming language targeting php with a built in orm
bench	a simple c# benchmarking library/language for testing how webservers react with different types of requests (large posts, small get, small posts) in a haphazard way.
j.encoder	a simple javascript template language (rolled up into jove)
simpledepend	a php comment dependency factor tool; wrap functions in comments and then wrap destinations in comments; the destinations get split into multiple files and the dependents are traced out included when needed.
particalgen	an image generator for particle system images (using point rendering)
scripture	a immature documentation system
diffhistory	a interface to diff that enabled a fs written in dokan to track a head and a revision log of how to go back in time
peoplemachine	a small scale crm
hi, my name is jeff	a book (did my own version of nanowrmo in march)
massshard	a simple orm built around sharding
bspcompiler	compile a mesh into a bsp tree
chef	an ide for my game engine (turned into jove)
factorg	small library for factoring gaussian integers
font2texture	convert a font to a texture map
quickneighborhood	quicksort using planes (designed for broad-phase of a large collection of particles moving quickly)
pointcloudalgorithms	3d algorithms on point clouds
spoon	agent modeling
poonix	turned microsoft virtual pc into a managed cloud
devsync	a suckier version of rsync in windows
glasseson	add glasses using opencv to a picture (client work that didn't pan out)
gold	transaction manager for s3
horde	simplequeueservice based video transcoding (for hurox)
magistrate	interface to kira's backend
filesharingaccelerator	a centralized file sharing service
wealth	my text based version of acquire (board game)
sudoku	yet another sudoku solver
csdc	first orm
mapgen4galciv	made a map generator for galactic civ
mathmod	a web language for math (second version of mathmyway)
auth	oauth before oauth
rage	rapid game engine (precursor to cauldron)
businesslogichelper	regular expression search for enforcing rules on how to produce and consume sql
defensemaped	a map editor for a tower game
notjavascript	a working javascript interpreter with very strict rules (and no newline detection)
sword	a text editor control (from scratch)
boxplane	a gui control
lamegeo	a c# geometry library (primarily for a future reverse geo-coder)
libquest	asynchronous quest manager (how wow quests work)