Cobalt Edge

I've recently spent a considerable amount of time rectifying problems caused by SVN externals. In one of the codebases I work on, it had been developed with a heavy number of Rails plugins as SVN externals. In general, it was a good approach as these were external code, or shared code, etc. This I think is at least better than directly checking the code in, as you have a more precise record of where it's from, etc. I should also note that our externals were all set to specific tags or branches specific to our code (i.e. not to trunk, where you'd be getting updates without your control). Sounds good, what about this "evil"?

The problem comes in when you need to make changes to the code of an external. You might think, well, go change the root code and then adjust your tag, etc. In some cases you can't do that - maybe it's not code you have commit rights to, or you're making a change that's specific to your app and can't be done another way, or, as was often in the case I had, we were on a much older version, and the trunk and other tags had major differences that I didn't want to integrate.

Thus, what I needed to do was remove this as an external, and check the code in directly. Another approach would be to branch it from where you were and modify that, etc. I wasn't able to do that due to various Subversion permissions (probably not a common case, but I had no choice). This action itself (remove external, add code) is not a real problem in SVN. But, it IS a problem when you go to update. A simple "svn up" on other machines failed. That is pathetic. Instead, what I had to do was go delete the existing (svn externaled) directories, then do "svn up". This of course broke our continuous integration server, and I also had to go manually fix this up on machines I was deploying to. Crappy, but if that was the end of it, I'd probably not be as unhappy...

When it comes to merging these kinds of changes into branches, watch out! This is where SVN just flails. First if you happen to use svnmerge.py to manage your branch merging, forget it. It just can't deal with it, and will leave you with a partially complete merge. Doing it manually, even with things like --ignore-ancestors, does not work either. I had to do something similar as to the "svn up" fix: I had to go in and delete all the directories that were previously svn externals, and then do my merge. And note, do NOT delete the parent directories. For example, if all of your Rails app's plugins were externals, do go and nuke "vendor/plugins". It will then be totally confused and just not do anything, and fail. Nope, you need to specifically delete each offending svn external directory. I make extensive use of branches (I do most work on a branch for daily work), so you can multiply these problems across the number of branches you might need to be merging to, etc.

Having said all that, this problem isn't really all that illogical. I don't know how SVN works internally, but the whole svn:externals thing seems a bit like a hack, or at least not a first class citizen in SVN land. SVN merge or update, should be able to see: hey, you were up to date (for your current revision) on directory X, but this update is going to replace that with new code with the same dir name. But, it doesn't, maybe because it doesn't look at the externals properly in relation. I don't know, and I don't care, since it's broken, and my fix is that I'm moving to Git soon enough :) Also, as another point of view, I know Perforce handles this kind of thing just fine (we used remote mounted Perforce depots all the time at Adobe, and made seriously extensive use of branches (in fact, we required working on a branch)).

Now that I've spent entirely too much time on the build-up, what's the solution? Simple: use Piston (or Braid if using Git). What Piston does, is to not use svn:externals, and instead check the code in directly, yet maintain linkage to the external it came from. My take is this is really probably how svn:externals should've worked (I presume that constantly updating an external is actually a rarely desired trait). You import an svn external using Piston, and it will pull the latest code from whatever SVN URL you supply. In this case, you could use trunk, or you could as usual use a tag or branch. But then it's fixed - it will not update that anytime you do "svn update". Instead, it is up to you to explicitly tell it to update. This avoids svn externals as far as your daily operations go, and also causes zero problems for merges. It does more though.

The second benefit of Piston is that you can then modify the external code, but still bring down updates from the external, allowing a synergy between using external code and your app's specific needs. This is exactly what I needed on a couple of plugins we use, where those plugins' code had deviated significantly from our codebase so I couldn't use a newer version, but I needed to make some changes.

To summarize, the evil is SVN itself not handling changing of externals (i.e. to/from an external) in basic operations like updates and merges, which may cause a lot of manual work on your end, and break automated builds or similar. The solution: use Piston or Braid and get the best of everything.

Loading mentions Retweet

There's a post today on the Relevance blog about Frozen Gems Generator. I tried posting a comment there, but it seems to have not gone through, so I'll blog my solution here instead.

Chad Woolley at Pivotal Labs created GemInstaller to solve the problem of specifying exactly what gems you want your Rails app (or other Ruby code) to use. I've dealt with this issue a lot over the course of building Rails apps, and while at first blush I didn't think this was a good solution, I'm now really like it, and use it on most of my projects (basically all the projects I control or can :)

So, why is it better than other solutions, or at least the other solutions I've seen? First, let me give a quick synopsis: it is a simple gem that allows you to create a geminstaller.yml file that specifies the version(s) of gems your app wants. This can be an explicit version, or can use things like >= version, etc. It can then automatically install the gems for you on app launch, on deploy, or just at the command line. The benefits of this solution for a Rails app include:

• Solves the arhictecture/platform-specific gems problem. I haven't seen any of the other solutions do this, or do it well. Most just punt on it, others require a convoluted process or hacking up your other code. Because geminstaller simply relies on the gems being installed on your system, it will use the proper version for whatever system it is running on. This also ties into the next point...


• No polluting your source control with gems. This speeds up your source control, as well as your deploys. Further, for architecture specific gems, you now don't have to have every version of each gem in your source control for each platform you need (which is quite likely at least two: your dev boxes (e.g. Macs) and your deployment boxes (Linux), but could be even more).


• Easier, single location, statement of what gems your app requires. By using the geminstaller.yml file, you have a single place to go see what gems and which versions of those gems your app uses. This is much better than trying to look through your vendor directory, and determine what version of a gem you might have.


• Great for bootstrapping your development environment. Sure, frozen gems usually solve this too (except for the architecture specific ones!). You can just run geminstaller after pulling down the code and it'll go install all the specific version gems you need.


• Allows for multiple config files, so that you can build common ones you use across projects, etc. Or even cooler, your plugins or whatever can provide a file to specify what they need and you can integrate that into your config!


• Easy to install and use. In Rails 2 environments, you can simply drop the few lines needed to use it into its own file in config/initializers. In Rails 1.x, you add these lines to your environment.rb.


• You determine what level of function you want geminstaller doing in your app: e.g. do you want it automatically installing missing gems or just warning you? Should it put them on the load path so you are guaranteed the proper version loads, or do you want to just use it to bootstrap and live dangerously otherwise ;-)


• Makes it easier to experiment with new versions of gems. Since you'll have to install the gem anyway (or most solutions need that to freeze them in, but not all), you can experiment by simply changing the version number in your geminstaller.yml file. To undo it, just change the number back. No need to copy the gem into vendor or a private gem repo, etc. Easy.


• GemInstaller can tell you what gems you have on your system, but are not in your config file, as a way to see what you might need.

Check out the GemInstaller page for more details. I highly recommend this, and thanks Chad for creating it.

Loading mentions Retweet