Not totally insane

• July 30th, 2014
• neural-networks python javascript

Neurosimulators

I wrote my first neural network simulator, called Neural, in 1988 for a VAX computer. In C of course, because (1) C++ did not exist, and (2) computers were slow and there was no faster language than C (save Assembly but we don’t want to go there). Neural got the job done, but was very limited, though it did include a small scripting language for students to play with. We also had a more ambitious project called MetaNet, which had more neural network paradigms and a visual shell. We even built, and I mean physically built, as in built in the basement of our building, a 400 processor neurosimulator that did dedicated neural network processing: the so call Brain Style Processor 400, or BSP400. All of this was done because we had a need for speed. Computers were slow and brain is large, and we want to develop brain style models.

In the late 1990s, my research group in Amsterdam and I developed the Walnut neurosimulator, which was written in C++. I rewrote part of this code for the then newish language Python 1.5, without a visual shell and with critical routines delegated to code that relied on the Python Numerics library, which consists of optimized C-code to run very fast. The Python code served our research well: it was flexible and easy to script and extend, and alltogether much less hassle than C++.

Fun with JavaScript

Just for fun, in 2010 I wrote a completely different library in JavaScript, mainly for educational purposes: to run neural networks in a browser and have students play a little bit with them.

Recently, I had to decide which of the systems I was going to keep maintaining: the C++, Python, or Javascript system? All of them? Who has the time? The C++ based library had always been hard to maintain. Despite the promise of C (C++) being a long-term stable language, quite a few small changes had crept into the language. More importantly, our graphics shell (called Nutshell. Get it???) was Windows based, and the Windows libraries and compilers kept changing dramatically from over the years. At the same time, the Python libraries worked well enough, and they were fast enough for research purposes. I mainly kept the Walnut Nutshell installer up-to-date for new versions of Windows as, surprisingly, the old code base, even in compiled form, kept working through all generations of Windows, from 95, to 98, to XP, Vista, 7, and 8. Even the scripting in Visual Basic for Applications kept on (and keeps on) working. Praise for Microsoft to keep running the same binary code possible for over 20 years!

Nonetheless, I decided to stop development of the C++-based Walnut Nutshell and in early 2014, I did a speed comparison of Python and JavaScript, just to find out how bad JavaScript really was and whether I could live with that.

I had known JavaScript since it was the inept scripting language I used in 1999 to write a few trainers for my Online Memory Improvement Course (see memory.uva.nl). This was before AJAX and the DOM. Speed did not matter, the trainers worked well enough, and everybody was happy. The course and trainers have been online for over fifteen years now.

Speed of simulations in Python and JavaScript

When I delved into the speed of Python versus JavaScript, I was flabbergasted to find out that JavaScript has by now matured into a very fast language. Yes, it is scripted. Yes, it is high level. Yes, it does not have (strong) types. It does not even generate byte code or anything, like say Python. But large crews of very talented programmers have done a most excellent job of optimizing JavaScript interpreters. There are now projects where you can run Python code in JavaScript interpreters of Python and make it run much faster immediately (except for the parts relying on the Numerics libraries of course, which are still C-based). You can even take a normal Windows program and run it on a simulated processor, written in JavaScript(!), and it will run at a decent speed. And these teams of highly talented and very persistent programmers will continue to tweak JavaScript, so we have not seen the end of this yet.

Yes, C++ (or C) will be faster. It always will, especially since you can embed hand-optimized Assembly in it. But C++ is a challenge to write, a pain to maintain, and a nightmare to debug (ever done loose pointer tracking or try to find a memory leak? This was even more ‘fun’in the days when accidentally writing something in deallocated memory would cause your computer to reboot without warning). It is not impossible, but writing good C++ is high art: It requires experience, persistence, and talent. Interfacing with Windows libraries implies writing a lot of boiler plate code. And then you still cannot run your system on a Mac or on Linux. (I know, you can write for QT or WX windows and such, but I consider that high art as well, especially if you do it in C++, rather than say in Python.)

And how much speed-up do you really need? A factor 2 sounds like a lot, but Moore’s Law says: Just wait one and half years and your new hardware will be that much faster. So, if I continue to work for a while in a slightly slower system that runs anywhere on anything, if I am slow enough and take 1.5 years, it will run twice as fast by the time I am finished.

Development goals

So JavaScript is no longer the runt of the litter of languages. Yes, it is till bad and ugly in places, but these parts are easily avoided. And JavaScript is everywhere. Every browser has it and it seems that almost every device we have has a browser: phones, eReaders, even my TV has a browser. And all of these run JavaScript. Now I am not going to argue that one should run neural network simulations on eReaders or smart TVs; my point is merely that because of the ubiquitous nature of JavaScript, a lot of people (and industries) are working on improving the language, so that applications run smoother, animations are slicker, Buy This buttons are pressed, and cash keeps flowing. It is the economy of all this that ensures that for the time being we’ll see a continuous improvement in the JavaScript language.

Which brings me to the topic of this series: running neural network simulations in JavaScript. The goal of this project is to redesign WalnutJS so that it makes full use of recent HTML5 techniques, notably typed arrays and parallel processing with web workers and Single Instruction Multiple Data extensions (SIMD). Additionally, we will develop a few routines for computationally efficient, but biologically plausible, spiking neurons with synaptic delay. First, we will consider a number of partial implementations of the techniques mentioned to experiment with the current state-of-the-art. In these excursions, we will use only Chrome and Firefox to keep everything manageable. We will first look at web workers and multi-procesing, then at SIMD extensions, and finally at a type of spiking neuron that has recently gained popularity.