Things to keep from RubyForge

Executive summary: File uploads, mailing lists, and virtual hosts

Everyone knows that the RubyForge gem index is getting merged into the GemCutter gem index to form a single massive community gem index at rubygems.org. That leaves open the question of what happens to the other RubyForge services. We've been saying that most of the RubyForge services - the ones others can do better - are going away as we move towards the community hub concept that Rich outlined.

So, which RubyForge stuff do we keep? Based on feedback and further discussion, here are the services which are making the cut:

  • mailing lists : these seem to be popular and alternatives seem to be not so good
  • file uploads : these can be done elsewhere, but it seems like keeping them with the projects is worthwhile
  • virtual hosts : these provide Ruby projects with a convenient place to have a home page
  • projects: since we need something off of which to hang the other features

These features will be part of the new community hub, albeit in a much-improved format. Stay tuned... and comments welcome!

October 30, 2009 in Ruby | | Comments (12) | TrackBack (0)

Transferring gem namespaces on RubyForge

A while back I blogged about how you could now see what gem namespaces your RubyForge project owns. That's been improved a bit; as of today you have the ability to release a namespace back into the wild. Here's a screenshot when I was logged into RubyForge and looking at the bottom of the codeforpeople files page:



So, if you go to the "Files" tab on your project and scroll to the bottom of the page you'll see a list of the namespaces your project owns. Each has a little "x" next to it. Just click that "x", go through a confirmation page, and your project will no longer own that namespace.

Note that you also need to delete any gems that claimed that namespace in that project. For example (from the FAQ entry on this topic), suppose you had a mygames project and released a file asteroids-0.0.1.gem. This means that mygames owns that asteroids namespace. Suppose you start an asteroids project and want to move the namespace over. To do this, you would release the gem on the new asteroids project and then delete the namespace from the mygames project. The next time RubyForge deploys gems it'll notice that the asteroids project has a new gem that uses a now-unclaimed namespace, and there you are, namespace transferred.

You can see that there is a gap here - if someone else has released an asteroids gem on their project there's a chance they might get the namespace instead. If that happens, please file a support request and we'll figure it out. Eventually I'll probably modify the user interface so that you can explicitly transfer the namespace from one project to another.

For now, though, this is better than having to file a support request every single time. Enjoy!

October 21, 2009 in Ruby | | Comments (0) | TrackBack (0)

The rubyforge gem and the RubyForge REST API

Yesterday Ryan Davis and I released v2.0.0 of the "rubyforge" gem. The big change for this version is that it no longer interacts with RubyForge by scraping HTML; instead, it uses the new RubyForge REST API. So instead of POST'ing a form to login and fetch your project list, it uses HTTP Basic authentication and hits /users/zenspider/groups.js.

If you're using the new gem, you may see this when you run rubyforge config:

$ rubyforge config
/Library/Ruby/Gems/1.8/gems/json-1.1.7/lib/json/common.rb:122:in `parse': 
618: unexpected token at '<?xml version="1.0" encoding="UTF-8"?> (JSON::ParserError)
<!DOCTYPE html
[ ... lots more ... ]

That's happening because your ~/.rubyforge/user-config.yml file is telling the rubyforge gem to hit https://rubyforge.org, and it needs to hit http://api.rubyforge.org instead. To fix that, either run rubyforge setup, or just edit your ~/.rubyforge/user-config.yml and change the uri setting.

I'm happy to have a REST API for RubyForge available; this has been in the works for a few months and I hope folks come up with some interesting ways to use it. It's certainly much easier (and much more efficient) than the old way of using Mechanize or whatever to parse the HTML. Not all the RubyForge resources are available yet, so if you want to access something and don't see it in the API please let me know and I'll add it. The code is here and feedback is of course welcome. Enjoy!

September 22, 2009 in Ruby | | Comments (0) | TrackBack (0)

Applying the Rails XSS vulnerability patch

I'm probably making this harder than it should be... but if so, the Internet will correct me :-)

Anyhow, I wanted to apply the Rails XSS vulnerability patch on a machine that was running Rails 2.3.2. The gems weren't frozen to the app, though, they were just out there in /usr/local/lib/ruby/gems/1.8/gems/.

I moved over to the gems directory and tried to apply the patch, but I got the "which file do you want to apply the patch to" message. It makes sense; the patch wants to make the change to (for example) activesupport/lib/active_support/multibyte.rb and all the gem directories have the version numbers attached - e.g., activesupport-2.3.2. I was getting ready to kind of pick the patch apart but then thought "hold on, symlinks!" So I did this:

$ cd /usr/local/lib/ruby/gems/1.8/gems/
$ sudo ln -s activesupport-2.3.2 activesupport
$ sudo ln -s actionpack-2.3.2 actionpack
$ sudo patch -p1 < /home/tom/2-3-CVE-2009-3009.patch
patching file activesupport/lib/active_support/multibyte.rb
patching file activesupport/lib/active_support/multibyte/chars.rb
Hunk #2 succeeded at 283 (offset -15 lines).
Hunk #4 succeeded at 622 (offset -15 lines).
patching file activesupport/lib/active_support/multibyte/utils.rb
patching file activesupport/test/multibyte_utils_test.rb
patching file actionpack/lib/action_view/helpers/tag_helper.rb
$ sudo rm -f actionpack
$ sudo rm -f activesupport

Restart the app, and huzzah! All's well.

September 04, 2009 in Ruby | | Comments (7) | TrackBack (0)

What gem namespaces does that RubyForge project own?

Back in January I posted a list of which RubyForge projects own what RubyGem. That's all well and good, but it's a one-time dump of data and it's more or less hidden off in a corner.

So now to fix that, you can see the gem namespaces each project owns at the bottom of the "files" page. For example, here's the seattlerb project "files" page with their massive list of namespaces.

This list gets updated daily, so it should stay fresh.

Next task: the ability to transfer gem namespaces to another project. That's been a long time coming, but this gets us a step closer.

August 25, 2009 in Ruby | | Comments (1) | TrackBack (0)

How much disk space is my RubyForge project using?

To find out, just log in to RubyForge, go to your project, and click the Admin tab. The totals are split out for "SCM" (source code mgmt, e.g., cvs/git/svn), space used in your project's virtual host, and space used by the files you've released:

Here are the top 10 projects in terms of SCM usage:

gforge=> select g.unix_group_name, d.scm_space_used/1000 \
as MB from disk_usages d, groups g where g.group_id = d.group_id \
 order by d.scm_space_used desc limit 10;
 unix_group_name | mb  
-----------------+-----
 blacklight      | 598
 cougar          | 576
 instantrails    | 547
 easygameengine  | 249
 ogrerb          | 228
 rubyes          | 215
 fxruby          | 187
 dojo-pkg        | 177
 restore         | 174
 tubix           | 150

And for released files:

gforge=> select g.unix_group_name, d.released_files_space_used/1000 \
as MB from disk_usages d, groups g where g.group_id = d.group_id \
order by d.released_files_space_used desc limit 10;
 unix_group_name |  mb  
-----------------+------
 backlog         | 1130
 instantrails    |  960
 rubyinstaller   |  726
 rhodes          |  666
 rmagick         |  603
 wxruby          |  535
 rails           |  369
 linnet          |  326
 rubricks        |  323
 fxruby          |  316

And for virtual host space used:

gforge=> select g.unix_group_name, d.virtual_host_space_used/1000 \
as MB from disk_usages d, groups g where g.group_id = d.group_id \
order by d.virtual_host_space_used desc limit 10;
 unix_group_name | mb  
-----------------+-----
 instantrails    | 328
 funfx           | 171
 rubyworks       | 116
 fxruby          | 109
 roby            |  97
 scrubyt         |  95
 freeride        |  91
 twitter4r       |  84
 rpa-base        |  76
 rubyhackerblog  |  74

Not surprising that InstantRails would be the leader or close to it in all three categories, I guess; such is the way of projects with large binaries. "backlog" has a bunch of war files - targetting JRuby, I reckon.

This information is populated via a cronjob, so if you clean up some stuff it'll be a while before the numbers get updated. Right now I've got it scheduled to run once a week. Mechanics-wise, it's very low-ceremony - it just iterates over the active projects and runs du -sk on various directories.

August 21, 2009 in Ruby | | Comments (0) | TrackBack (0)

Refactoring RubyForge svn+http directives

I was inspired by the guys over at beanstalkapp to look at the way the RubyForge virtual hosts handle svn+http. Here's the old way:

<Location /svn>
  DAV svn
  SVNPathAuthz off
  SVNPath /var/svn/support
  <LimitExcept GET PROPFIND OPTIONS REPORT>
    deny from all
  </LimitExcept>
</Location>

Pretty straightforward, but since all the virtual hosts have pretty much the same directives, here's the new way:

<Location /svn>
  SVNPath /var/svn/support
</Location>

Much more concise! All of the duplicated stuff got moved up into a global Location directive and only the host-specific stuff remains. This makes each of the virtual hosts that include svn+http support about a third smaller, which means faster Apache restarts and all around goodness.

The downside is that now all virtual hosts include a /svn URL - it's there even if that project only uses git or cvs. Kind of clonky. It'd also be nice to get rid of SVNPath, but unfortunately I think the way we have /svn hanging off the root of the virtual host prevents that.

One day I should look at mod_vhost_alias. 99% of the RubyForge virtual hosts are exactly the same less the project name. Seems like it should be possible to refactor away almost all of them.

August 18, 2009 in Ruby | | Comments (0) | TrackBack (0)

Monit, Mongrel, and /usr/local/bin

If you're using Monit to restart a Mongrel cluster, you'll probably run into the problem detailed here - that is, Monit reports "execution failed" when trying to restart a mongrel instance. The problem is that mongrel_rails is usually in /usr/local/bin/, and Monit doesn't put /usr/local/bin in the PATH. That blog post's comments have a variety of suggestions and you'll find similar ideas around the web, such as:

  • Edit /usr/local/bin/mongrel_rails and munge ENV['PATH']
  • Uglify the Monit script's start_program to call /usr/bin/env PATH=/usr/local/bin
  • Symlink /usr/bin/mongrel_rails to /usr/local/bin/mongrel_rails

That last suggestion seems fairly painless... but still clumsy.

So, here's another way. Are you compiling monit from source? Good! Edit env.c and change line 158:

/* BEFORE */
  char   *path = "PATH=/bin:/usr/bin:/sbin:/usr/sbin";
/*  AFTER */
  char   *path = "PATH=/bin:/usr/bin:/sbin:/usr/sbin:/usr/local/bin";

A quick make && sudo make install and there you go! /usr/local/bin is now on the PATH. Enjoy!

July 31, 2009 in Ruby | | Comments (2) | TrackBack (0)

The Subversion Ruby bindings and the 'relocation R_X86_64_32' error

I was setting up some servers with the Beanstalk guys and was building the Subversion Ruby bindings - you know, the ones that use SWIG. All was well until I got to make swig-rb. That resulted in an error which I will present in its entirety for easier Googling:

$ make swig-rb
cd subversion/bindings/swig/ruby/libsvn_swig_ruby && /bin/sh /home/beanstalk/src/subversion-1.5.5/libtool \
--tag=CC --silent --mode=link gcc  -g -O2  -g -O2 -pthread    -rpath /usr/local/lib -lruby-static -o \
libsvn_swig_ruby-1.la  swigutil_rb.lo ../../../../../subversion/libsvn_client/libsvn_client-1.la \
../../../../../subversion/libsvn_wc/libsvn_wc-1.la ../../../../../subversion/libsvn_delta/libsvn_delta-1.la \
../../../../../subversion/libsvn_subr/libsvn_subr-1.la /usr/local/apache2/lib/libaprutil-1.la -lexpat \
/usr/local/apache2/lib/libapr-1.la -luuid -lrt -lcrypt  -lpthread -ldl 
/usr/bin/ld: /usr/local/lib/libruby-static.a(array.o): relocation R_X86_64_32 against `a local symbol' can 
not be used when making a shared object; recompile with -fPIC
/usr/local/lib/libruby-static.a: could not read symbols: Bad value
collect2: ld returned 1 exit status
make: *** [subversion/bindings/swig/ruby/libsvn_swig_ruby/libsvn_swig_ruby-1.la] Error 1

The problem is that the linker is looking for a shared object library and finding only a static library. The solution is to recompile Ruby with the --enable-shared and reinstall it. This will create a shared object library (lib/libruby.so.1.8.6) in addition to the static library (lib/libruby-static.a), and if you retry make swig-rb it'll work.

I'm used to seeing Ruby libraries that depend on underlying native code - e.g., libxml-ruby depends on the libxml2 headers and such being in place, the mysql gem depends on the MySQL client, etc. But the Subversion bindings are different - they come with Subversion and depend on Ruby being already installed. It kind of turns things around.

Why doesn't the Subversion autoconf script (or the Makefile?) detect that there's no shared object library and fail with a more helpful error? I'm not sure... but that would be nice.

July 22, 2009 in Ruby | | Comments (0) | TrackBack (0)

libmysqlclient.so.16: cannot open shared object file

This is another post under the "simple solutions to annoying problems" category. I was deploying a Rails app to a server on which I'd compiled MySQL from source. The deployment went fine until the deploy:migrate task:

  * executing "cd /path/to/app/releases/20090714141105; rake RAILS_ENV=production  db:migrate"
    servers: ["foo.net"]
    [foo.net] executing command
*** [err :: foo.net] !!! The bundled mysql.rb driver has been removed from 
Rails 2.2. Please install the mysql gem and try again: gem install mysql.
*** [err :: foo.net] rake aborted!
*** [err :: foo.net] libmysqlclient.so.16: cannot open shared object file: 
No such file or directory - /usr/local/lib/ruby/gems/1.8/gems/mysql-2.7/lib/mysql.so
*** [err :: foo.net] 
*** [err :: foo.net] (See full trace by running task with --trace)
 ** [out :: foo.net] (in /path/to/app/releases/20090714141105)
    command finished
failed: "sh -c 'cd /path/to/app/releases/20090714141105; rake RAILS_ENV=production  db:migrate'" on foo.net

The problem was that I'd compiled MySQL with --prefix=/usr/local/mysql and that directory wasn't on the runtime linker path. So I edited /etc/ld.so.conf and added /usr/local/mysql/lib/mysql, ran ldconfig, and voila, all was well.

'

This problem was a little more confusing since the Rails app itself was running fine, and running script/console worked fine too. But that was because I had added the MySQL lib path to LD_LIBRARY_PATH in /etc/profile. Anyhow, hope this helps someone!

July 14, 2009 in Ruby | | Comments (2) | TrackBack (0)

Next »