Out of my Mind...

Stu Halloway tweeted about the following Pastie earlier today:

He questioned the counterintuitive result of blank? returning false on an empty Java array. He then asked which whether Ruby, JRuby or Rails were the culprit.

I played with the Rails console a little bit and started to dig into the code. It turns out that Rails adds a blank? method to the Ruby Object class. Here is it's implementation:

It also aliases blank? to empty? on arrays further down. As a result, the expected behavior of calling blank? on an empty array is to return true:

The Java array does not respond to the empty? method:

We can see that the Java native array does not share the same ancestry as the Ruby Array:

None of the classes and modules returned by x.class.ancestors implement the empty? method. Rails has no choice to fall back to the default implementation and to return false to blank? since any instantiated object isn't nil by definition (except for nil, which is always nil).

The real culprit here isn't Ruby as Stu alluded in a later tweet. Rather, it is the lack of an empty? implementation on the Java native array class ancestry that is the root cause of the surprising behavior. The best fix would be to implement the empty? method on the ArrayJavaProxy class to Rails could call it.

My first mandate when I joined Project Kenai was to improve it's performance. Kenai wasn't known for being speedy back then. We had some serious performance issues with both page rendering in the browser as well as generating the content on the server side.

I had work on projects before were performance testing (and the resulting avalanche of bug reports from it) were held off until the QA phase of the project. A small team of QA engineers would write scripts that exercised the application to get performance data and generate information on where things were slow. I had a few issues with this approach:

1. The testing almost always occurred near the end of a release. Many of the performance issues discovered in this process could not be fully resolved in the time frame allocated for the release itself. Most were preempted by other feature bugs as well. Performance always seemed to be job 1.0.1
2. The data never provided information on where the application was spending it's time. This lead to a fair amount of time spent looking at the application code and database queries to see where the hotspot were.
3. The performance tests occurred based on a static set of features and actions. Each URL would be tested in isolation. There were none of the interactions that occur from multiple users doing different things in the application at the same time.

Doing performance testing was still valuable. It helped us understand where we needed to do more work. It was just too little too late for many projects that I worked on.

I have tried different approaches in the past. I would instrument the application that I worked on to get data, but that was usually done against a development or QA environment. Reading the logs to find the instrumentation data made it difficult to extract valuable information. The instrumentation had a measurable impact on the response time so it was generally turned off in the production deployments.

Performance testing was different with Project Kenai. Kenai uses the NewRelic Rails plugin to get performance data from the application and generates reports that helps understand where the application is spending it's time and what areas could benefit for further investigation. Here is how we normally deal with performance issues:

1. We use NewRelic to determine what functions are the slowest and have the most impact to our users.
2. I go through the web site the morning after a deployment to see how the deployment changed the performance of the application.
3. I schedule one or two performance issues if nothing stands out of the ordinary. Any serious performance degradation takes precedence of course.
4. I use the development mode to research the performance problem and work toward a solution. My work environment has different performance than our production and staging sites, so I look for relative improvements rather than absolute measurements.
5. I review the impact of the change in our staging environments. This helps me understand the performance of the application with a full production-like dataset as compared to the developer dataset. Many performance issues only showed themselves when running against a larger dataset than what was available to me in my development environment.

This cycle allowed the team to continually improve the performance of the application. We still have work to do with regards to performance. Most of the work that is left to do is in areas that are difficult to resolve, but we know where they are and we keep looking for ways to keep improving the performance.

I firmly believe that any developer who doesn't use the different tools available to communicate outside of the bubble that is the people they work with or the company they work for is short changing him (or her) self.

These tools are varied. Twitter is one of them. It is a great way to exchange barbs and keep track of what like-minded (and not so like-minded) developers are doing. It's 140 character format is conductive to bursty exchanges. I certainly enjoy the ongoing conversation that I have with individuals that I respect who happen to hang out on Twitter. The banter might be light at time, but it is still great banter.

It's not the only medium that developers should use though. Developers need to establish an Internet footprint. It is especially critical for those who are working on web applications or infrastructure. Blogging is another tool available to help developers to establish and expand this Internet footprint. Contributing to projects on Project Kenai, GitHub or other project hosting infrastructure is another good way of doing so as well.

Finally, developers should reach out and network locally. It is not a big investment to take an evening each month to head out and attend a local user group meeting. Doing so shows that you do care about your craft and are willing to invest some time in expanding your knowledge. It is even better when you actively participate into the meetings by presenting or actively networking with other professionals.

To participate, whether through Twitter, blogs, projects or users group shows that you care. That alone is often enough for an individual to rise above the vast sea of indifferent developers that are prevalent in the so called real world.

The Colorado Springs Open Source Users Group record the presentation that are given to their group. Here is the recording that was made of my abbreviated introduction to JRuby:

Basic Concepts - Frederic Jean (Intro to JRuby) from Kevin Werner on Vimeo.

I have yet to watch the whole presentation as I cringe each time I hear me speak.

The JRuby MemCache Client uses Greg Whalin's Java based MemCache client to connect to the servers that implement the MemCache protocol. This client uses a connection pool to connect to and communicate with the memcached servers.

There is one pool by default. It is called "default". You do not need to specify a pool name if you only need one connection pool. This would happen if you only need to talk to a group of servers.

This becomes a little more complicated if you need to communicate with multiple memcached server groups for different purposes. A great example would be using both memcached and Kestrel on the same cluster. This is the situation that we find ourselves under on Project Kenai.

The solution is simple. You can pass a :pool_name parameter when you are building the configuration for the MemCache client:

<script src="http://gist.github.com/157876.js"></script>

This results in a separate connection pool being created and allows you to use the JRuby MemCache client to talk to both Kestrel and memcached servers.

You can just feel the panic rising. You were logged into the production database fixing a few records when you made a small mistake with disastrous consequence.

So what should you do next?

Stop.

Step away from the keyboard.

Breath.

Get help.

Odds are that someone who isn't in a near panic state will be able to think clearly about the problem and guide you through a solution that will reduce or even remove the impact of the incident and help you get things back to their normal state.

Of course, you should really ask yourself why you were in the production database in the first place... But that's another discussion.

I wrote a post about using the jruby_memcache_client Rails plugin to manage sessions. Since then, Abhi Yerra made a fork of Ikai Lan's repository and turned it into a gem. I promptly merged his changes into my repository and I am now publishing my own version of the gem (with a few minor changes).

The new JRuby MemCache Client Gem is API compatible with the Ruby MemCache Client. You should now be able to install the gem into your application and use it directly.

Installing The Gem

You need to add github as a source for gems before you can install the new gem. Thsi is a simple one time command to run:

memcache_options = {
:namespace => 'fortaleza:production_live:'
}
memcached_servers = [ ENV['MEMCACHED_LOCATION'] || '127.0.0.1:11211']

# Constant used by libs
CACHE = MemCache.new memcached_servers, memcache_options

You will then be able to configure Rails to use the :mem_cache_store for both session and cache store.

This is pretty much what any presenter could hope for :)

Here are the slides:

We've all seen people who have 2 or 3 work in progress items within a single workspace. Even the workspace owner doesn't always know what file belongs where, nor can they really prove that their change will integrate cleanly with the rest of the code. I see this more often with people that have used centralized source code repositories to manage their code. Branching is expensive and checking out a fresh repository sometimes takes more time than actually fixing the bug. It is tempting to cut a corner here and there to be able to do this quick fix and move back to the task at hand. It is a dangerous thing to do though.

I find that I do my best work when I am working on one item at a time, whether we are talking about a feature or a bug. This is one of the reasons why I like git. It's local branching capabilities makes it easy to do just that while still being able to quickly go back to a clean slate when reality intrudes and I have to switch to a different task.

My Workflow

Project Kenai still hosts its repository in a Subversion repository. I quickly started using git-svn to managed my workspace and I base my daily workflow on Lennon's daily git-svn workflow. Here is how I approach a task:

1. I make sure that I am at a clean slate and up-to-date. I checkout my master branch (git checkout master) and bring it up to date with the upstream subversion repository (git svn rebase).
2. I create a branch against the master branch and switch to it (git checkout -b branch-name). I normally name the branch based on the Jira issue that I'm working against. This makes it easy for me to link the work that I'm doing to the related Jira issue. I do not using any of the git svn commands in this branch. I add files to the index and commit frequently as I work as a safety net. It is easier to revert to a previous state that way.
3. I then switch to my svn merge branch once work has completed. I make sure that it is up to date (git svn rebase) and run our automated tests. This lets me know that I am working against a clean branch. I then merge my changes from the work branch (git merge --squash branch-name). This condenses all of the commits I did in the branch into one commit that will be merged into the svn merge branch, resolve any merge issues and run the tests again.
4. I commit the merge into the svn merge branch (git commit) and then push my changes to the Subversion repository (git svn dcommit).
5. I finally go back to the master branch, update it (git svn rebase) and run the tests to make sure that all is working as expected.

It may seem a little involved, but it gives me many points where I have a chance to run tests and verify that the work will integrate cleanly with the rest of the team's work.

When Reality Intrudes

I don't always have the luxury of a linear workflow. there are times where I have to jump to a different work item (critical bug) and work on a fix. The workflow that I use allows me to do this very easily since I can go back to a clean slate at any point in time by:

1. Committing the work in progress
2. Going back to the master branch
3. Updating the master branch
4. Creating a new branch for the new work
5. Complete the new work and commit it to the Subversion repository.
6. Go back to the work in progress branch and resume work

As a result, my workspace always contains the changes associated with one work item. There isn't confusion on what file needs to be committed nor do I run the risk of having multiple patches applied to a single file and having to sort through them manually when it is time to commit.

The good news is that this very same workflow will work just as well if we switch the Project Kenai's repository to git or Mercurial.

The JRuby MemCache Client Plugin has been replaced by the JRuby MemCache Client Gem. Details can be found at JRuby MemCache Client Gem

We realized a few weeks ago that we spent a fair amount of time writing Project Kenai's sessions to our database. We already had decided to start using fragment caching and we were exploring different options on the client side.

Project Kenai's front end is a Ruby on Rails web application that is deployed to Glassfish and running under JRuby. We do have multiple JRuby runtimes per domain. There are issues with running the Ruby MemCache client with a JRuby on Rails application, so we paid some attention to Ikai Lan's jruby-memcache-client library. It is a wrapper around a Java based MemCache client that support connection pooling back to the memcached servers.

I quickly discovered that Ikai had focused on the Rails cache store but hadn't put work on the session store. I forked his project and started adding support for a session store.

Using the jruby_memcache_client plugin

Here are the steps to install the client:

1. Run "script/plugin install git://github.com/fredjean/jruby_memcache_client.git" in the root of your Rails application. This will install my forked copy of the plugin.
2. Configure the MemCache client by following the instructions located at http://wiki.rubyonrails.org/rails/pages/HowtoChangeSessionStore .
3. Finally, change the config.action_controller.session_store to use :j_mem_cache_store instead of :mem_cache_store

Make sure that you have a memcached server running and start your rails application.

Making sure that it is running