There’s a behaviour in rake I keep stumbling on. It’s the inability to dynamically add prerequisite tasks on a task from one of it’s prerequisites.
I've used this gist a couple of times now:
I'm pretty sure the first time I actually stumbled on this was sometime in 2005.
Why?
The output from the rakefile above is
when, if the dependency tree was built dynamically, ‘c’ should pop up before ‘a’:
Why would you need this you will ask?
Well, it helps to do things like task :build => :determine_sources instead of task :build => determine_sources() and avoid the performance hit of scanning for dependencies before you actually call the task.
This improves rake responsiveness which has a positive effect on team moral (no kidding, this is important!)
The workaround is to calculate all dependencies for all tasks when the rakefile loads and it does not scale.
Case in point: Currently our rakefile builds a C system with a changing number of small applications. Manually adding tasks for every new application costs too much so it is done automatically. Because no dynamic prerequisites are allowed, the method that creates the tasks also needs to scan the source and determine the complete dependency tree for every app task.
This adds unnecessary delays in rake’s startup. As we usually only call one build task we only need to calculate the dependencies for that one task (this is actually done but with a whole bunch of code that essentially controls task creation based on the task called and it is fragile and prevents us from calling multiple tasks in the same command line – basically it’s a hack)
Put your money…
The patch is actually trivial, a two line change in lib/rake/task.rb in the invoke_prerequisites method1.
Where’s the catch?
Well, this actually only works if a prereq is changing the prerequisite list it is a member of. Anything else is dependent on the sequence the tasks appear in the prerequisites list and has unexpected results.
There’s also the slight delay introduced by iterating over the prerequisites again, but this barely registers compared with the actual task invocation times.
1 And yet it took me almost 7 years to fix. Talk about procrastination…
Off to the comments…
Posted by Vassilis Rizopoulos on Jan 13, 2012
Problematic luxury
Recently in our current project we ran up against a luxury problem: The number of our tests has grown so much that it became unfeasible to let them run on a per-commit basis.
There are also so called ‘long running’ tests that test processes with a standard duration of several minutes1. We actually want to run all tests on all builds but we were faced with a few hurdles.
The build used for the tests is big – meaning it contains the whole system and thus has the slowest build times. Running the tests in the same build job only exasperates the problem blowing the self-imposed 10 minute limit sky high.
Currently the “monster” build (all platforms, all components) clocks in at ~30 minutes under high load or ~12 minutes if it gets the build server to itself2. The build required for tests comes second at ~8 minutes.
Throw hardware at the problem
The conventional approach fits our problem well: Split the tests in separate test suites and throw hardware at the problem. Since we are using VMs for our development environment it is actually very easy to instantiate a VM for every test suite and let the testing commence in parallel (for other reasons we cannot run test suites in parallel in the same VM).
Where it gets a bit complicated is coordinating the builds and the tests. We use Jenkins for build management and
it is feasible to create a master-slave configuration feeding the build package to the test jobs but it still strikes me as unnecessarily complicated3.
It also means that the data on the dependencies between the different build jobs is distributed in the Jenkins instances. I have a particular aversion to this type of configuration: ideally any job management system should be able to function without built-in knowledge of it’s place in the system. It just needs to know where to go for the information and we get to manage it centrally4.
What do you actually want?
In order to kick off the tests all we need is a URL pointing to the build package and a bit of metadata, in this case the revision the build corresponds to.
Once the tests are executed we need to know the status of the tests for the revision and a URL pointing to the detailed logs.
Given that we want to test all change sets and that test suites have greatly varying execution times we want to know as soon as possible which revision breaks a test suite…per test suite.
Please join the queue in an orderly fashion
So we want to distribute one build to N testers without knowing when they are going to be available to process it. Sounds like a background job thingy doesn’t it?
Setup a queue for each tester, publish the build information (URL & revision) and let the tester grab the build and do it’s thing. Fire and forget!
This type of job processing is bread and butter for web applications and there’s a lot of libraries/systems that do this: resque, BackgroundJob, DelayedJob etc.
Still too complicated. You need Redis, or Rails, or some SQL database or you’re tied to a POSIX OS5. You get all kinds of goodies like priority scheduling, persistence and monitoring (Resque is my particular favorite with the nifty web GUI) but honestly this is the best case of YAGNI I have yet encountered.
Not created here
Why can’t I find something simple, with minimal dependencies that will run everywhere? Probably because it takes less time to built it than to look for it!
Take Sinatra and Ruby’s built-in Queue class, sprinkle some JSON and you get rplex.
How it works
The rplex service waits. In Jenkins at the end of the build we simple post the data to rplex.
On the tester side an endless loop (implemented in Rplex::Processor) runs the tests whenever a build appears.
How do we get the test results? Same concept: At the end of the test run we post the data back to rplex, this time targeting a single rplex client. That process just updates our information radiator.
Simple complexity
We have a whole bunch of rplex clients now but nowhere near the maintenance overhead of Jenkins jobs. The process looks like this:
Start a VM
Start the rake task with the Rplex::Processor
* Do something else
Fiddling with the data format allows us to do everything we need without touching rplex. If we get a huge backlog for slow test suites we simply restart rplex and empty the queues6.
Come to think of it, I could replace Jenkins completely using rplex as the core…hmmmm!
1 We’re talking integration and system tests here. Where unit tests are concerned we exert draconian control to ensure fast build times.
2 Network based license schemes for compilers suck big time!
3 Some initial problems with setting up master-slave in Jenkins with our firewall and network zone configuration did not help.
4 Which means we get to version control a single source of configuration and handle redundancy by cloning instances.
5 Oh Windows! Thou art the bane of my developer life!
6 While in theory you would not want to leave any build untested when development is in full swing and you get 5 or 10 patches in the space of 5 minutes – just because somebody was careless – you only care about the last one.
Off to the comments…
Posted by Vassilis Rizopoulos on Jan 03, 2012
A little bit of background
I've worked with a lot of teams over the years both on-site with customers and in-house with independent teams and one of the constants was the need for a persistent knowledge archive.
“persistent knowledge archive” is just a pompous term for a repository where the team can easily maintain draft documentation, document decisions that affect development and also maintain documentation that is not actually part of the product, like how to use the testing rig, how to setup the development environment etc.
Single point of reference
One of the first things I ever introduced 1 in our team workflow was the use of a wiki for exactly that purpose.
The previous alternatives of mass emails and gasp Word documents were just too transient or cumbersome.
There are several reasons for that, but the most important in my mind is that a wiki (and a blog) provides a single point of reference that is easy to edit and keep current and has a very easy learning curve.
When you get new team members you just point them to the URL and give a little speech about the project conventions and you’re done.
Scope
I'm going to narrow a bit the scope for this article in order to get my point across.
We’re talking about wikis vs. blogs as tools for team communication, information dissemination that demands the least amount of maintenance and ensures that the information persists and is accessible at any point in the future.
While I have for a long time advocated the use of wikis for my teams, recently I have substituted the wiki for a blog and this is an attempt to explain and discuss the pros and cons of this decision.
But first a couple of definitions
Wiki
Google “wiki definition” and you get
wi·ki /ˈwikē/
Noun: A Web site developed collaboratively by a community of users, allowing any user to add and edit content.
I our scope the community is the team developing the software
Blog
Google “blog definition” and you’ll get
blog /blôg/
Noun: A Web site on which an individual or group of users record opinions, information, etc. on a regular basis.
Again in our scope the group of users is the team and everyone on the team can add and edit posts.
Wiki vs. Blog
Technically there is no substantial difference between a wiki and a blog in the scope we have just defined. Both are essentially content management systems that allow their users to create and edit content.
For both types there is software that provides versioning facilities, search, commenting, file uploads and notifications for new content. So, no difference there.
The experiment of substituting the wiki with a blog showed that there are small but significant differences on the usage uptake between the two tools.
My recent experience showed that a blog was more easily accepted and started seeing use by the team a lot faster than a wiki. There seemed to be more incentive in adding to the blog than in adding to the wiki.
Now the evidence is empirical and also there are significant differences between projects for there to be a definite verdict but the indications are nonetheless strong.
One reason for it might be that the blog format is by now very familiar to the overwhelming majority of software developers. Many of my colleagues maintain their own blogs and have no difficulties with the concept.
I find this hard to believe though. Most blog engines use text markup that originated with wikis (Markdown, Textile etc.) and as I already said there is hardly any difference in how content is created or updated.
I attribute the difference to the concept of content ownership: In a wiki a single page has no clear owner. It’s a collaborative effort by definition and that ever present edit button seems to scare some people off2. Editing something someone else wrote feels I guess invasive.
In the blog the pattern of usage has a clear ownership attribute: the post is owned by whoever created it, any discussion about the content happens in the comments section and there is the implicit understanding that any changes needed to keep the content current will be undertaken by the content owner.
Now this is a purely psychological difference, since on both platforms everyone on the team can view and edit everything, but the empirical evidence suggests it is significant.
There is one usage for which wikis are undisputedly the better solution and that is for collaboratively developing the official project documentation but there the “ownership” of the content is explicitly shared which to my mind strengthens the argument.
In the case of official documentation we can also view it as a single unit of structured work where the interlinking facilities of a wiki are a huge advantage and impose a kind of unification layer while in a blog each post is a single unit and does not necessarily have to fit with everything else, which makes it ideal for informal information dissemination.
Recap
I find that blogs make a better communication medium for medium to long term persistence of project relevant knowledge (design decisions, howtos, status reports) while wikis are much better for developing official documentation. Transient information (“yoohoo, new version for our favorite library”) is still best handled through IM and emails.
Feel free to flame me as an ignorant idiot.
1 “introduce” is another pompous term that means I ran around extolling the virtues of a tool until my significantly more intelligent colleagues get tired of hearing the noise and start using it just to shut me up.
2 I initially was myself baffled by this since collective code ownership is something we strive for in our projects until I noticed that no matter how many developers can understand and change the code the developer who originally was tasked with creating a feature was always referred to as “the guy who wrote it”.
Off to the comments…
Posted by Vassilis Rizopoulos on Sep 12, 2011
Code performance has two aspects: the actual performance measured hard seconds and the perceived performance AKA the responsiveness of the application.
When scripting, which for the purposes of this post I define as writing code that is executed once in the command line and then exits, a crucial metric is the startup time: the time it takes to load your script and start processing.
In the continuing battle to improve the performance and responsiveness of our rake build system (see here, here and here in reverse chronological order) the startup delay has now become the number one issue.
We have a name for it: “The Ruby tribute minute”. It’s not actually a minute, but in the last month it has climbed back up to 11 seconds which is a very noticeable delay, especially when you’re doing ‘rake -T’.
After a bit of experimentation trying (unsuccesfully) to get ruby-prof and profiler.rb running on Windows 7, I forked the code and started decorating it with Benchmark.measure calls.
The results proved very interesting. The overwhelming majority of the startup time was spent requiring couchrest, couchrest_model and active_record. We use couchrest to collect metrics (tasks called, time spent building, lines of code per build, firmware image sizes etc.) in a CouchDB instance which is then used to feed our information radiator.
Requiring couchrest costs ~2 seconds, couchrestmodel costs ~4.5 seconds, activerecord ~3.5 seconds (couchrestmodel requires activerecord but so do other gems we use).
Isolating the problem does not actually help since these numbers fall significantly. For example the following test
gives significantly reduced times:
btw. the stock Snow Leopard Ruby was a lot faster with rubygems 1.5.0:
but suffers with rubygems 1.8.7
I will take a wild guess and assume side effects from the additional gems loaded increase loading times but I can only theorize.
The current workaround consists of removing the require statements from the top of the files and adding them in the tasks where the gems are actually used. This shaves about 6.5 seconds from our startup times and improves rake responsiveness significantly but has the drawback of peppering require statements all over the place. I'm extremely interested in figuring out a better way to do this and also why the current rubygems version performs so bad.
Now to go back and refactor rutema so that active_record is not loaded unless needed…
Off to the comments…
Posted by Vassilis Rizopoulos on Aug 09, 2011
It’s an idea that has been bouncing in my head for months now.
I had reached unprecedented playground fatigue levels (where the sufferer is overcome by feelings of despair at the mention of a playground) and due to the nature of my work had little or no contact with any local hackers/rubyists/geeks.
I love local user groups, I actually started down the path to coding in one such group, in this here city. We didn’t call it a user group, we didn’t have internet (it was Fidonet, BBSs and 2400 baud modems back then) and we drank mostly retsina but the spirit was the same.
It is astonishing how long one can procrastinate and even more astonishing how easy and fast one can organize a user group with fellow geeks.
I started by tweeting a question. The answer led to rubyst.es a greek ruby community site, which unfortunately seems to lack any activity (still waiting on the approval of my registration guys). After a bit of ominous silence and a feeling of dread – currently very familiar in Greece – I reframed my tweeted question and this time I got pointed to ruby-hellug.
It’s a mailing list so I fired an email and forgot about it. Or at least tried to. It took exactly 25 minutes for the first email to reach me. Apparently there iswas no Ruby group in Thessaloniki, GR.
James Gray (James Edward Gray III as I know him ;) ) offered the final push the next day in response to an email for help. Essentially “just do it and get the newbies in, you won’t regret it”.
It was June 22nd. The thessaloniki.rb founders meeting (meaning we went out for beers) took place on July 7th.
thessaloniki.rb #1 (meaning we talked about Ruby and then went out for beers) took place on July 12th.
Reception was very positive. The list has 15 members already, which – taking into account that it is the middle of the summer and traditionally things are super slow here – I consider a huge success.
We even have swag: Our group counts as it’s member the first and (currently) only GitHub employee in Greece ( Petros Amiridis in a league of his own) and GitHub rocks when it comes to community support.
Also Avdi Grim of Exceptional Ruby fame was extra helpful when I asked him to essentially ripoff his book and make a presentation – not only did he give permission, he offered a generous discount to the group as well. A thousand thanks to you Avdi!
Thanks are due to City College and George Eleftherakis for kindly allowing us to use their space.
We have a group, we have a GitHub repository, a twitter hashtag (#thessrb – works on so many levels: thess for thessaloniki, thess==‘you want’ in greek, so thessrb==‘you want ruby’ etc.), ideas and apparently voracious appetite for learning.
At a time of great hardship and uncertainty meeting a group of intelligent people with energy and drive to learn, help each other and move forward is priceless.
See you all in September!
Off to the comments…
Posted by Vassilis Rizopoulos on Jul 15, 2011