https://www.timfelgentreff.de

Git talk @ JaX 2010

2010-06-07T00:00:00+00:00

I had the opportunity to speak at the JaX 2010 in Mainz, Germany, together with Ralph Guderlei from exxcellent solutions. The JaX is one of the largest Java conferences and primarily aimed at enterprise architectures and SOA. Our talk concerned distributed version control systems, how they compare to CVS-style systems and how to integrate them into existing development environments with a special focus on Git. The slides are available in German.

First week of RubySOC

2010-06-18T00:00:00+00:00

First week of Ruby Summer of Code is over an I have started coding on extending JRuby with C extension support.

After I had already looked at wmeissner’s jruby/cext the week before and got it to run with his help, I used the first week of rsoc to import the project into the main JRuby repository (forked on timfel/jruby). I looked at mkmf support initially and quickly got annoyed with the way I had to run the jruby-cext examples:

old_cext_run.sh

LD_LIBRARY_CONFIG=jruby-cext/examples/MyTest:cext/build \
  bin/jruby -J-Djava.library.path=cext/build \
  jruby-cext/examples/MyTest/MyTest.rb

So I started working on runtime resolution of library paths to loose the JVM argument and be able to ship the cext jnilib in the jruby-complete.jar. Right now, the cext jnilib is searched in the jruby.home and, if that is within a Jar, extracted to a temporary directory. I applied the same logic to c extension resolution, so running with cexts looks now like any other jruby run:

new_cext_run.sh

bin/jruby jruby-cext/examples/MyTest/MyTest.rb

The second thing I worked on was Kernel.require support. In JRuby, the LoadService class is responsible for resolution of require paths at runtime and various other classes like ExternalScript and JarredScript are used to do the actual loading. I added a CExtension class which enabled me to require C extensions directly. Before, ‘require’ in MyTest.rb looked like this:

old_require.rb

include Java
m = org.jruby.cext.ModuleLoader.new
ext = File.join(File.dirname(__FILE__), "mytest")
m.load(self, ext)
MyTest.test1 # call the C defined method

Now, this has shrunk to:

new_require.rb

require 'mytest'
MyTest.test1

JRuby commit flag

2010-07-09T00:00:00+00:00

Mid-term is almost upon and there are quite a few notable changes, but not much to show.

If have spent most of my time getting various gems to compile. Upon a request from konstantinhaase/rkh I looked into getting the Thin webserver to run. After some tinkering it compiled and ran. Benchmarks run, too, but print no numbers (they don’t with REE, either, at least for me). I tried running this very blog with Thin on JRuby, it won’t process requests right now, though. This is probably a problem with my rb_str* method implementations. I am still looking into this.

Apart from that I have gotten the official C-Api specs to run. The changes needed are at timfel/rubyspec. Also, I have been given commit access to the main repository at jruby.org. The cext branch is now the playground for Wayne Meissner and me. While Wayne is focussing on speed and synchronization, I am more focused on getting gems to run, no matter how. I don’t expect any speed wonders from this C-Api anytime soon.

Thin on JRuby

2010-07-13T00:00:00+00:00

After much work from Wayne Meissner and myself I have managed to get the Thin webserver running on JRuby. It actually loaded for a couple of weeks or so already, but only recently stopped crashing when receiving a request. Today I have managed to get it to serve pages as well, so I guess it’s time for some speed checking :)

The tests are just quickly done using ab and were run on REE 2010.02, JRuby Head on the cext branch and Rubinius 1.0.0-20100514. I did 1000 requests at different concurrency levels and checked the average req/s.

The numbers

concurrency	REE	JRuby	Rubinius
1	3102	298	214
11	3563	521	544
21	4836	584	754
31	3402	668	728
41	5056	686	486
51	4025	672	797
61	5427	1046	704
71	3874	1080	736
81	5249	947	685
91	3919	678	579

As we can see, JRuby starts something like 15x slower than REE and only recovers to about 4 times slower with a concurrency of 71. It is generally faster than Rubinius, though. This is not a fair comparison, though, as JRuby has a Java implementation of EventMachine.

Apart from that, overall coverage of the C API specs is getting better, too. Currently out of the 348 examples, 26 fail and 110 throw errors. Failures are mostly incomplete implementations of Ruby functions, errors are almost exclusively missing functions.

Benchmarking RDiscount

2010-08-14T00:00:00+00:00

The last benchmarks I posted, running Thin, were rather unfair against Rubinius (as Evan has pointed out). Time has passed in which I have been busy getting more specs to pass (only 8 failures and 2 errors are left) and I have found another gem with a benchmarking script for me to run ;) The RDiscount gem which implements Markdown in C.

I just ran rake build benchmark on Ree (2010.02), Rubinius (1.0.1 20100603) and the JRuby cext branch and got these numbers:

JRuby

BlueCloth:	07.433000s total time	00.074330s average
RDiscount:	00.124000s total time	00.001240s average
Maruku:	11.619000s total time	00.116190s average

Ree

BlueCloth:	08.029169s total time	00.080292s average
RDiscount:	00.046740s total time	00.000467s average
Maruku:	06.500247s total time	00.065002s average

Rubinius

BlueCloth:	00.062746s total time	00.000627s average
RDiscount:	00.056345s total time	00.000563s average
Maruku:	21.178625s total time	00.211786s average

It’s clear how Ree is fastest with the C extension, but Rbx is not much behind. And even though JRuby’s cext implementation seems to be around 3 times slower for RDiscount than the other implementations, it’s still almost 60times faster than BlueCloth and around 90times faster than Maruku (compared to 1/171/139 on Ree and 1/1/375 on Rbx for RDiscount/BlueCloth/Maruku).

Also note how Ree is the only one where Maruku is actually fast and also how bloody fast BlueCloth runs on Rubinius!

So while the cext support for JRuby is comparatively slow, there are clearly cases where it might yield a performance gain.

A Redcar debugger interface

2010-09-26T00:00:00+00:00

Bringing some debugging to Redcar

The Redcar

I have been using the excellent Redcar editor for about a year now, and I must say I am very happy with this project and the pace it’s moving forward at. Hardly a day goes by where there is not some new, hot feature that makes coding easier in this project.

But the best part: this editor is written almost entirely in JRuby, which makes it a breeze to get into the code and start hacking on it.

Redcar-Debug

When working on C extension support for JRuby, I often find myself having to attach a Java debugger and a GDB to the same process. For Java debugging, I used Eclipse, which comes with very handy features like inspection and hot code reloading, but the thing I like most about the Eclipse debug perspective is how it organizes the information about the process.

With GDB’s commandline interface, I often found myself typing bt, info locals, info args at each breakpoint, and returning back to Redcar for context.

Luckily, there are very good frontends for GDB, too, the most sophisticated I could find was the debugger interface for Emacs, which offers me a layout similar to the Eclipse debug perspective. The Emacs debugger interface even came with JDB support. Hurray!

Why Redcar-debug?

I turns out, while I do like Emacs for TeX editing, I prefer programming in Redcar. Switching between Redcar, Eclipse was bad enough, throwing Emacs into that was too much.

This is why I started redcar-debug, the Grand Debugging Interface for Redcar which I have been using for a couple of months now for my JRuby debugging needs.

It currently includes modules for GDB, JDB, RDebug and Hijack support, and it shouldn’t be too hard to add other commandline-driven debuggers to it. And because you can describe a GUI for something best with pictures I have some moving pictures for you, too:

Sorry. Your browser does not support the video tag. Get a decent browser

Update The video wasn’t working for some people, so here’s a plain link

Syntax Checking For Redcar

2010-10-23T00:00:00+00:00

I spent the last couple of days thinking about how to implement syntax checking for Redcar. Konstantin Haase and I discussed using the Melbourne gem, which is basically Rubinius’ parser extracted into a gem. Melbourne is a C extension which has been passing all specs for some time now on JRuby master with cext support, so that seemed like a viable way to go.

Before I got started on my plan, however, I remembered that JRuby, like MRI, supports the -c switch to just do Syntax checking. So I dug into that a little and came up with a short script to run the JRuby parser on a piece of code from within a JRuby process:

ruby_syntax_checker.rb

require 'java'

module Java::IO
  import 'java.io.File'
  import 'java.io.FileInputStream'
end

module JRuby
  import 'org.jruby.Ruby'
end

class RubySyntaxChecker
  # Why this *args here? Well, this forces a ManyVarsDynamicScope for the Parser
  def check(path, *args)
    begin
      runtime = JRuby::Ruby.global_runtime
      io = Java::IO::FileInputStream.new(Java::IO::File.new(path))
      runtime.parse_from_main(io, File.basename(path))
    rescue SyntaxError => e
      e.exception.message
    end
  end
end

From this I have now created two Redcar plugins: redcar_syntax_check which offers some general features pertaining to syntax checking: an abstract checker class, a syntax error class which knows how to annotate an edit view in Redcar and a hook to run available syntax checkers after saving a file.
To do this I had to add a few methods to Redcar’s edit_view plugin, so currently this runs only on master.

Using this plugin it is now fairly easy to offer syntax checking for any language: All you have to do is inherit from Redcar::SyntaxCheck::Checker and implement your custom syntax checking logic in the check method. If you discover syntax errors in the given file, build instances of Redcar::SyntaxCheck::Error from them - Error’s need only know which document they belong to, the line number and a message. Calling annotate on them will then draw the annotation and a wiggly line at the error position.

Finally, tell Redcar which grammar’s you support using the supported_grammars class method and a list of grammar names in your Checker subclass’ class body and you’re done!

See redcar_syntax_check_ruby for an example of how to do it.

When all went well, you can just open a document, write some code and, upon saving, it will automatically check the syntax and show you your errors:

Ruby Summer of Code Wrap-Up

2010-11-10T00:00:00+00:00

In the first quarter of this year I applied for a Ruby Summer of Code project, proposing to work on support for Ruby C extensions on JRuby. JRuby has for some time been faster than 1.8 an in in most cases as fast as 1.9 for running Ruby code. But C extensions have been unusable on JRuby, although Wayne Meissner has written a proof-of-concept for loading C code into the JRuby runtime.

Why C Extensions?

MRI 1.8 has been widely critizised (often outside the Ruby community), for being slow. However, the interpreter is written in a way that allows to tap into its structures very easily and Ruby has for some time offered a way to link extensions in C using mkmf.rb, which is part of the standard library.
Processing intensive tasks or interfaces to tried’n’tested C libraries could thus be written without the overhead imposed by an unnecessary abstraction.

When alternative Implementations like Rubinius and JRuby came along, people tried to position a new Ruby FFI as the successor to C extensions, because an FFI API could be supported much easier on different VMs.
I personally feel that FFI will never fully replace C extensions, because it tries to abstract from a low-level concept without being able to hide many of the low-level details. Quite often it is just much easier to simply write and compile a C extension, than to write C code and try to adapt its structures in an FFI wrapper library. Ruby is simply not very comfortable, if you have to think about things like pointer sizes in your wrapper code.

Widespread adoption of FFI is still not happening, and maybe further away now that Rubinius’ support for native C extensions is pretty much complete for most use cases.
Thus, the situation for people wanting to:

Deploy Ruby on Windows easily
Use Java libraries
Deploy in a war-file
…

is, that they have to stay away from C extensions no matter what.

For some extensions, like Nokogiri, EM or OpenSSL, this has lead to pure-Java ports, while other libraries have been fast enough as pure Ruby code on JRuby. However, multiple versions of extensions are a pain to maintain, especially when they rely on other libraries, like RMagick does on ImageMagick or Nokogiri on libxml. In such cases, people have tried re-writing those libraries in Java, which not only makes maintenance even harder, but compatibility becomes an issue, too. Nokogiri-jruby had for some time a lot of problems which came from the Java XML implementation simply not being up to par with libxml.

So, How Does This Work On JRuby?

My Summer of Code pretty much consisted of integrating Wayne’s groundwork with JRuby and its LoadService, which I will briefly explain here, and then trying to get the C-API Rubyspecs passing and trying to work every Gem that somebody seemed to care about.

Requiring Files On JRuby

Rubys require logic is fairly complex, as proven by a ~450 LOC size of basic require specs. JRuby’s is even more complex, as require must also work for .class and .jar files, and not only consider the filesystem, but also the JVM classpath and thus jar file contents.

I learned how to hack the JRuby loading by debugging and stepping through the LoadService class in JRuby using Eclipse. A good entry point to break in there is the smartLoad method, which receives a the require String as it is passed down from Ruby. Then, JRuby goes uses a number of Searcher classes to try to find a load candidate. There is a script searcher, a classloader searcher, and now an extension searcher, too. Each one of those tries different paths and different file extensions to find a match, and there are quite a few things done “under the hood”.
For example, regardless of the platform, you can always require ‘something.so’ to load an extension. The ExtensionSearcher on JRuby will then try to find a jar file or a file with the platform specific shared object file-extension.

When a match is found, it is loaded by a method appropriate for its type: Ruby scripts are loaded by evaluating into the current runtime, C extensions are loaded into the process memory and its Init_* methods are called. To still be able to deploy JRuby applications in a single Jar, extensions are extracted to the default java.io.tmpdir (which you can pass as JVM parameter) and loaded from there. This is because Java’s System.loadLibrary relies on the underlying operating system to actually load the shared object into the process, and that will only work if the library is in the actual filesystem.

There are a few restrictions when using extensions on JRuby. For example, multiple Runtimes will not work, as we can only dynamically link a library to the JVM once. Since we cannot know wether a given extension is thread-safe or if it has global state, we can’t simply re-use the already loaded library and expose it to another Runtime. Currently, we will throw an error if one attempts to load a C extension twice - which means Rakefiles won’t work if they load a C extension and then try spawning a sub-process. JRuby’s system(), magic will detect the launch of another Ruby command and just create a new Runtime in the same JVM. If you want to support JRuby in this case, you can do the following: if RUBY_PLATFORM =~ /java/ require ‘jruby’ JRuby.runtime.instance_config.run_ruby_in_process = false
end ### Running C Code In JRuby - Pieces Of Advice In order for extensions to work, we had to implement as much of the Ruby C API as is possible to support on JRuby. In most cases, where the C methods are simply implementations of STL functionality in C (things like rb_ary_push, rb_str_cat, rb_str_new, …) we can just upcall to Java and have JRuby run the Ruby code for this. You can imagine that this is quite slow, but it is safe, and easily supported. Eventually, we can speed those calls up by caching call targets and methods, too.

More difficult are most macros. Things like RSTRING_PTR or RARRAY expose internal structs of MRI, which we cannot support directly. On JRuby, once you use such a macro, the objects you are accessing are added to a synchronization list and upon each switch Java -> C -> Java the contents of the object are copied downwards and upwards; until the object is eventually garbage collected. We are copying all active objects, because we cannot know if a reference in form of a variable to the content pointers remains in use.

Something we currently do not support at all, or only in a very limited way, are macros and methods which rely on or deal with MRI’s AST (e.g. the NODE(x) macro), its threading model (thread_critical = true won’t work), file descriptors (changing an fd’s access might not work reliably) and stack layout (functions like rb_each and rb_iterate don’t work for arbitrary data structures).

So people, if you’re building C extensions, and want them to work on JRuby, avoid methods that could make assumptions about any of the areas above. If you want your extensions to work faster on JRuby, avoid structs and struct-macros: use accessor methods instead. If you need read-only access the contents of a string as a char array, make sure those Ruby objects are GC’d soon enough, for example by using rb_str_cstr(x).
Finally, if you want some code to be specific for JRuby, #ifdef JRUBY will help you decide what you are compiling for (this works with RUBINIUS, too).

Generally, C extensions on JRuby will never be fast. Never as fast as on MRI, and never as fast as plain Ruby code on JRuby. As extensions go:

C Extension < FFI < Java callout < Java Extension

I have seen actual speedups only in extensions that do a lot of work in a single call. Ryan Tomayko’s RDiscount gem is a good example. There’s only a single call down, the hard work is done in C, and the String will probably be garbage collected pretty soon in Ruby land (see also my micro-benchmark; just don’t interpret too much into it).

Linux PM on Macbook Pro

2010-11-28T00:00:00+00:00

I have recently switched back to Linux as my main OS on my Macbook Pro (7,1 series). After half a year of using Mac OS X, I found I couldn’t force it to yield to my workflow and preferences in too many ways. I still think it is a great system, and a great user experience, it’s just that I have grown accustomed to so many little kinks (like sloppy-focus) that I find I cannot use Mac OS X as effectively as my customized Linux desktop.

After customizing my environment (which can be done easily using GUI tools on modern desktop linux distributions), I realized I had to tweak a little to get battery life up to par with what it was under Mac OS X.

Much of these steps are drawn from the Ubuntu Wiki.

First, there’s a PPA with Macbook specific daemons, drivers, tools, so I added that and installed a few things from there:

$ sudo add-apt-repository ppa:mactel-support && sudo apt-get update

$ sudo apt-get install pommed macfanctld nvidia-bl-dkms xcalib bcm5974-dkms xserver-xorg-input-synaptics btusb-dkms

This will install:

keyboard backlight support
fan control
screen backlight support
screen color calibration
better touchpad support
bluetooth drivers

After that, a few module tweaks are in order:

Activate coretemp on boot
$ sudo su -c 'echo coretemp >> /etc/modules'
Activate proper sound driver variant, with powersaving
$ sudo -s
# echo "options snd-hda-intel model=mbp55 power_save=1" >> /etc/modprobe.d/alsa-base.conf

Then, to get reboot working and have usb devices automatically suspend, use this to add a few kernel paramters to /etc/default/grub. Where it reads GRUB_CMDLINE_LINUX_DEFAULT, add:

usbcore.autosuspend=1 reboot=pci hpet=force
To write those changes to the active grub configuration, run:

$ sudo update-grub

I found that this did not cause all usb devices be set to auto-suspend, even though they could. So I added a one-liner to /etc/rc.local which I took from powertop:

for i in /sys/bus/usb/devices/*/power/control; do echo "auto" > "$i"; done
Don’t forget to make /etc/rc.local executable with chmod +x /etc/rc.local.

Finally, after installing the proprietary NVidia drivers, you can add the following options to the Device section in /etc/X11/xorg.conf:

Option "Coolbits" "1"

Option "OnDemandVBlankInterrupts" "True"

Option "RegistryDwords" "PowerMizerEnable=0x1; PerfLevelSrc=0x2233; PowerMizerDefault=0x3"

This will activate the frequency override capabilities, enable VBlank interrupts to only fire on demand and set the performance on battery to the fixed-minimum, so that Compiz doesn’t trigger the NVidia performance scaling.

After a reboot with these tweaks, I’m getting between 7.5 and 9.5 Watts usage, which is a little higher than on Mac OS X, but still good enough for around 6h of battery under conservative use (on Wifi, twittering, emailing and surfing).

Debugging on steroids - what Ruby should learn from Smalltalk

2011-09-08T00:00:00+00:00

A prototype of Seaside-like debugging in Ruby

The premise

I have used Smalltalk and the Seaside web-framework for some time before Konstantin introduced me to Ruby. I’ve come to use Ruby for most of my scripting needs, but what I’ve always preferred about Smalltalk is the tool integration: because you’re working with a live system, you have access to all these objects you write code for, to play with and see how they behave whenever you want to.

So in Seaside, when you break your application, you get a stacktrace page not unlike the pages in Rails, Sinatra, or Rack. However, in addition to the links for framework and full traces, you have a link to debug the thing directly. Simply at the point where it has thrown the error. And then resume it. As if nothing ever happened.

In Rails, when you break your application, you try to figure out what’s wrong from the trace and by looking at the code. Then you change the code and hope the Rails reloader works (and that you cannot always rely on, see Konstantin’s post on the matter). In Sinatra or Rack apps, you’ll usually have to restart your app. This means waiting for the server to come up and the request to be handled again. Even if it’s just a second or two … it does add up.

Also, it’s bloody annoying.

Here comes Everybody…

’s favorite Ruby implementation MagLev! MagLev is a Ruby implementation build on the excellent, mature, and battle-tested Gemstone/S Smalltalk product. Ruby classes and methods on MagLev are almost indistinguishable from Smalltalk methods and classes, and most of the core classes are simply shared between the implementations.

Since I’ve recently started an internship with Gemstone (now owned by VMware) in Beaverton, OR, I have the opportunity to try and provide a bridge for some of that Smalltalk goodness to reach the Ruby world!

First thing I added enough reflection to the Ruby core classes (mostly Thread and Method) to allow writing a debugger in pure Ruby.

This means being able to inspect, change, step through, and restart frames.
This means stopping, copying, saving, and restarting Threads, all in pure Ruby.
This also means having an API to change methods and classes, in-process, and have them write back to the file-system to keep it consistent with the process’ contents.

Later, that same day

Being well equipped with Awesome Reflective Powers™ now, I wrote an error handling Rack middleware for inclusion in any Rails, Rack and Sinatra app, and whipped up a very simple, web-based debugger that can connect to a running process and, well, debug it.

I presented the demo in the video below at the pdx.rb meeting in September. So what’s going on here? First, I’m using an old Sinatra app which I’ve added a bug to. This bug triggers the Rack middleware to save a continuation of the thread and present a (very, very simple) error page explaining what you can do from there. That page has a link to resume the thread that handled the connection. If you click it, without actually debugging, the thread just continues and shows the Sinatra error page as it would without the middleware.

We, however, having installed MagLev’s Webtools, can connect to the other process, find the right thread and look at the stack. We can quickly find the last frame that executes some of our code: the class method Bithug::User.find.

Because we’re dealing with an actual, living and breathing thread object saved to the stone repository, we can send messages to it to look at the arguments and locals of a method. We can see that self is the Bithug::User class, and that the Db constant that was involved in the error is a KeyValueDictionary. We can also find out that the value the method selects from the Db is non-existent. After noticing our mistake (self.class.name should really be self.name, because we’re already on the class side) we can edit the code and save it, thereby

changing the code in process,
writing it back to the file system,
and instructing the thread to reload and re-execute that frame

The thread is still suspended and in a persistent state. But if we go back to our application now and tell it to resume the request, it will prepare the thread for execution and run it - and we can see our app load. The error is fixed!

Watch the demo on YouTube

Why this is cool

Just using plain old Ruby, we can debug an application, we can inspect and change it while it is running. We don’t have to restart the process, we don’t even have to restart the request, it’s all in the thread, persisted to the repository.

The actual application doesn’t need to expose all these things, either. The thread was saved to the stone repository anyway, we can fire up a VM on a completely different machine, pull the thread from the stone and debug there! So even production, no error will ever need to go undebugged, because you can keep every error around for later inspection! No more trying to reproduce what’s on your production system - triggering the error just once is enough to step through it again and again and again and again.

This is one of the many advantages of having a malleable Smalltalk stack and real object persistence built into your VM!

Other use-cases come to mind. Remember, all this is accessible using pure Ruby.

You might want to handle other extraordinary cases in your application by simply persisting a continuation of the interesting state, so you can look at it later.

Or whenever you look at a method wondering what’s going on, you can just pull up a thread from your life app and see what kinds of arguments you might get, and what the state of your application might look like in this method.

Soon, at a Ruby implementation near you

I am currently working on providing and documenting enough API for people to do all these things with their Ruby apps. I hope some of this will show Rubyists that having an extraordinarily dynamic language doesn’t mean you can’t have great tools as well…

CloudFoundry for Smalltalk applications

2012-02-22T00:00:00+00:00

Finally!

After James Foster posted about getting Aida running on CloudFoundry, I just had to give it a go.

The first step was installing CloudFoundry. I created a vcap user on the server to run the cloud services. Using the dev_setup script from https://github.com/cloudfoundry/vcap/ got me up and running in just under an hour. The installer will download a few packages (it needs sudo for this) and then install all the runtimes (i.e. language VMs) and frameworks as the local user. Afterwards, the websites hosted here were broken, because there’s another thing that the installer changes, and that is your nginx.conf. By default, it’ll redirect everything to the CloudFoundry router process, which is not what I wanted. I just changed it to only redirect anything under *.vcap.bithug.org to the router, so all other websites could work as previously.

The last thing the installer told you was, how to run the cloud: ~/cloudfoundry/vcap/dev_setup/bin/vcap_dev start I found that, since we have a global RVM setup on this server, this interfered with the local rvm install the cloud foundry script set up, so I changed the .bashrc of this user to not load RVM and only use this minimal PATH

bashrc

export PATH=/usr/bin:/bin

After starting, I tested the setup with this simple Sinatra app:

env.rb

require 'rubygems'
require 'sinatra'

get '/' do
  host = ENV['VMC_APP_HOST']
  port = ENV['VMC_APP_PORT']
  "<h1>XXXXX Hello from the Cloud! via: #{host}:#{port}</h1>"
end

get '/env' do
  res = ''
  ENV.each do |k, v|
    res << "#{k}: #{v}<br/>"
  end
  res
end

To do this, I needed to create a user on the server, using the vmc script. vmc adduser Afterwards, it might be a good idea to change ~/cloudfoundry/.deployments/dev_box/config/cloud_controller.yml, insert your own email address into the list of admin users and disallow registering users through vmc. This way, only you’ll be able to add new users.

With this out of the way, we can just put the Sinatra app in an empty directory and run vmc push It will ask a series of questions, I accepted the defaults after choosing env as the name for my app. I verified that it worked by going to http://env.vcap.bithug.org.

Smalltalk, the Server Side

Now, the good stuff :) For getting Smalltalk to run, I pretty much followed James’ instructions. But I wasn’t interested in Aida. For my master’s thesis, I will need a few small apps running under different URLs. For one of those apps, I’ll be using Smalltalk apps written in a tiny web library called RatPack (yes, it’s a Sinatra look-alike in Smalltalk - even the error page is copied). So I needed a way to run arbitrary Smalltalk apps.

To do this, I changed James’ instructions to define a Squeak, rather than a Aida framework. While I was at it, I also changed a few other things:

I use the latest CogVM builds from Eliot Miranda’s website.
I install a patch during staging that will mirror the Transcript to a logfile, so you can have ‘some’ debugging
I installed the smalltalk services stuff in the vcap user’s home directory

You can grab the complete patch here.

Because CloudFoundry assigns a random port to its apps, I used James’ solution of passing the $VCAP_APP_PORT as a commandline argument for the Squeak application to use. However, because I don’t know what the app is going to look like, the developer will have to make sure that his/her app honors that argument.

Smalltalk, the Smalltalk Side

Now I created a (very simple) RatPack application:

app.st

RPApplication subclass: #TestApp
    instanceVariableNames: ''
    classVariableNames: ''
    poolDictionaries: ''
    category: 'TestApp'.

echoSomething
    <get: '/'>
    self error: 'Shows error info about me'.

Usually, with RatPack, you will use the Morphic control panel to start applications, assign them a port and so on. On CloudFoundry, the cloud controller might start an image on any port it chooses, or run multiple instances in parallel. To make sure our application is always running on the right port, I added these methods to the class side of the TestApp class:

classside.st

initialize
        "self initialize"
    Smalltalk addToStartUpList: self.
    Smalltalk addToShutDownList: self.

startUp
    "Start this app if running on cloud foundry"
    (SmalltalkImage current getSystemAttribute: 3) ifNil: [^ self].
    self
        runIn: #deployment
        on: (SmalltalkImage current getSystemAttribute: 3) asNumber
        named: 'TestApp'.

shutDown
    self services do: [:each | each stop; waitForStop].

This will make sure the application is properly shutdown and restarted whenever an image is stopped and re-run.

Now, to push this to the cloud, we need a squeak.st script that creates our initial image. To make this step fast and easy, I created a small project to create pushable directories and scripts from your applications. Install it and run:

script.st

CloudFoundry new
    base: 'squeak';
    generateDropletWithVNC.

It asks whether you want to install any MetacelloConfigurations and/or MCZs. I chose ConfigurationOfRatPack in version 1.0 with the default group, as well as my TestApp MCZ. The script creates a folder named vmc in your default Smalltalk directory with two MCZs copied from your package-cache, as well as a squeak.st script to run during the staging process:

squeak.st

Installer mc directory: '.'; package: 'ConfigurationOfRatPack'; install.
((Smalltalk at: #'ConfigurationOfRatPack') project version: '1.0') load: #('default').
Installer mc directory: '.'; package: 'TestApp'; install.
(Installer ss project: 'RFB') install: 'RFB'.RFBServer start: 5.

As you can see, this script will install the configuration and the mcz, as well as the RFB server, so you can connect to your image via VNC for debugging.

To actually push this application, again follow James’ instructions to patch the vmc gem on your development machine (second to last paragraph, adjust it so it reads squeak instead of aida in all places).

With that done, it’s as easy as going into the vmc directory and running vmc push. It should recognize that you’re working with a Squeak application and ask for deployment details.

In my case, because Squeak actually has to load quite a few things during staging, the client then timed out and reported a failure. You can check the transcript logfile or with VNC to make sure everything works as expected. Once you got it working, use

deploy.st

CloudFoundry generateDroplet.

to generate a deployment without VNC access to lock down your image.

Go say thanks to James and Peter for doing the hard work and finding all that out!

Master Thesis Progress

2012-05-31T00:00:00+00:00

Tracing and Debugging for Distributed Programs

Qoppa - a language to learn about Fexpr's

2012-08-27T00:00:00+00:00

Qoppa - a language to learn about Fexpr’s

Recently, I gave a talk at Fun Club Berlin about Fexpr’s. I had only recently heard about those and worked myself through a blogpost describing a minimal language, Qoppa, that uses only Fexpr’s and has no other sepcial forms. The author then built a library on top of Qoppa that implemented enough of Scheme to implement Qoppa on top of that again.

To do is to understand, so I implemented a Qoppa AST interpreter with Scheme syntax on top of PyPy, using the parser and much of the object model (as much as there is) from their Scheme implementation. You can find the code on GitHub, if you’re interested. I’m going to talk about my experiences with PyPy in a later post, so suffice it to say that the generated binary is reasonably fast, even though I haven’t really thought about optimizing anything.

So, if you’re interested in the What and Why of Fexprs, go read Keegan McAllister’s blogpost and look also at his Scheme implementation on GitHub. I described some of his explanations in my talk, and the screenshots of the “slides” I had are also available.

RSqueak/VM - A research VM for Squeak/Smalltalk

2015-04-10T00:00:00+00:00

RSqueak/VM - A research VM for Squeak/Smalltalk

For the past few years, we have been quietly working on RSqueak/VM, an RPython-based VM for Squeak/Smalltalk. The VM is still under development, mostly in short bursts and pushes, but is beginning to be useable and shows promising results.

RSqueak/VM grew out of the SpyVM work done by Carl Friedrich Bolz, Adrian Kuhn, Adrian Lienhard, Nicholas D. Matsakis, Oscar Nierstrasz, Lukas Renggli, Armin Rigo, and Toon Verwaest, but has since been extended by Lars Wassermann, Anton Gulenko, Tobias Pape, and myself.

We have done many interesting experiments with RSqueak/VM, including using STM, strategies, and most recently we have attempted to get performance up to run as many primitives as possible in pure Smalltalk, including such primitives used for large integer arithmetic, string operations, TrueType font rendering, bézier curves, display compositing, and more. We are not quite there yet for an alpha release, but a set of benchmarks we ran today show that we are getting close :)

PhD Thesis Progress

2015-09-25T00:00:00+00:00

Babelsberg

I am writing my dissertation on Babelsberg, a family of language experiments and that implement a design for adding constraint solving and automatically maintained assertions to ordinary object-oriented languages. You know how you sometimes write assert some.code.that.calls(stuff). And if your assertion fails, you crash? In Babelsberg, you can write always some.code.that.calls(stuff), and rather than crashing directly, the system will attempt to “heal” the program state to make the assertion pass (if it doesn’t already). This works great more often than you would think, and sometimes it even completely frees you from writing the code to establish a correct state in the first place! Just let the system deal with it - a Sudoku solver? No problem, I know what a valid solution should look like, I can just always that. Load balancing for video streaming, automatic repair of partially downloaded image data, layouting. We’ve found a number of interesting applications.