It’s pretty great.

But something that happens as you get more experienced with Rails is you start adding folders to your app/ folder. So, in addition to the standard models, views, controllers, and mailers directories, you might add a presenters, workers, or serializers directory. This is actually a big step toward leveling up as a Rails developer; it’s a sign that you’re comfortable decomposing your app into more than just models, views, and controllers.

One of the downsides of doing this, though, is that it feels painful to work with these new types of classes, because rails.vim doesn’t have handy navigation commands for them. But that’s okay, because it’s actually really easy to add those navigation commands. In addition to rails.vim, Tim Pope also makes projectionist.vim, which allows you to create custom Vim commands via a simple configuration file. And it’s not limited to Ruby or Rails; it can be used for any kind of project.

To get started, you put a .projections.json file in your project root directory. Here’s what that file looks like on one of my projects, where it’s adding navigation commands for workers and presenters:

{
  "app/workers/*_worker.rb": {"type": "worker"},
  "app/presenters/*_presenter.rb": {"type": "presenter"}
}

You can see that it’s pretty simple. You specify a pattern that matches certain types of files in your project, then you specify some options for files which match that pattern. In this case, the only option I’m specifying is "type", which is what creates the navigation commands. So, that first line creates the :Eworker and :Vworker commands (and some others, but those are the only ones I actually use) and then specifies that they will open files in the app/workers directory which have filenames ending in _worker.rb.

And that’s it!

Well, I’ll mention one more thing: I specified the "type" option in the configuration file above, but there are other options that you can specify as well, such as a "template" for creating new files or an "alternate" for navigating to related files. I’ll leave you to peruse the documentation if you want to learn more about those, though.

$ rpn_party
> 3 2 +
5.0
> 6 -
-1.0
> q
$

Naturally, I wanted to write tests that verified that it behaved as expected. But I’m a big believer in integration tests, so I wanted my tests to actually interact with the calculator as if they were a user. That is, I wanted them to run the executable, then send commands to the REPL over stdin, and, finally, verify the results by reading them from stdout. Essentially, use the app exactly as a human would.

The Solution

The solution can be found in a class from Ruby’s standard library called PTY. PTY allows you to spawn an external process and then interact with that process by using puts to write to it’s stdin and gets to read from it’s stdout. You can read the documentation for PTY, but it probably won’t make much sense unless you have a pretty good understanding of how pseudoterminals work (PTY is an unixism for pseudoterminal). It’s okay if you don’t, though, because using PTY to interact with command line apps is pretty simple:

PTY.spawn('path/to/executable') do |stdout, stdin, pid|
  stdin.puts 'some command'
  stdout.gets

  response = stdout.gets
  assert_equal 'expected response', response
end

Here’s what’s going on in this chunk of code:

1. We’re calling the spawn class method on the PTY class.
2. We’re passing the path to our executable as the first argument
3. Our second argument is a block which is where we specify how we want to interact with the process.
4. The block takes three arguments: two IO objects representing the stdout and stdin of the spawned process and then the pid of the spawned process, which will be useful later on.
5. In the first line of the block, we are interacting with the spawned process by writing text to it’s stdin.
6. Then we are consuming one line of text from the process’s stdout. We have to do this because our input from the previous line is echoed to the process’s stdout, so we need to consume that before we can get to the actual response. I’ll go into more detail on this down below.
7. Then, in the next two lines, we are getting the process’s response and asserting that it is the value we are expecting.

That’s basically it, other than a few tips and gotchas, which I will dive into below.

Echoed Input on stdout

The first gotcha to be aware of is that, for many command line apps, everything that is typed on stdin is echoed back to stdout. If you think about it, this actually makes a lot of sense. If it wasn’t, the user wouldn’t be able to see what they were typing. There are some cases where input doesn’t get echoed to stdout, though, like when a user is typing a password.

This isn’t a big deal, more just something to keep in mind. I dealt with this in my tests by creating a method that would send a command to my process and then immediately consume a line of output from stdout. This made my tests easier to comprehend. Here’s what that looked like:

def send_command(pty, command)
  pty[1].puts command
  pty[0].gets
end

This is a little tricky because for this method to do what it needs to do, it needs access to both stdin (to send the command) and stdout (to consume the line containing the command). So at the beginning of each of my blocks, I put all three of the block arguments into an array which would be less cumbersome to pass around:

PTY.spawn('path/to/executable') do |stdout, stdin, pid|
  pty = stdout, stdin, pid
end

And that pty variable is what gets passed as the first argument to the send_command method.

Process Termination

If you’ve come this far in pursuit of integration testing your command line app, you want to go all the way. And that means you want to verify that your program exits correctly. You could have your app print an exit message and then verify in your test that this message gets printed to stdout when you input the command to exit. The problem is that you don’t actually know that your CLI app exited. It could have printed the exit message and kept running.

Better to actually verify that your process is no longer running. This is where the pid argument to the block from up above comes into the picture:

PTY.spawn('bin/rpn_party') do |output, input, pid|
  stdin.puts 'exit'
  assert PTY.check(pid)
end

Here we’re sending the exit command to our process, then using the check class method on PTY to assert that the process is no longer running. The semantics of check are the opposite of what I, personally, would expect, but I’m not a systems programmer, so I will assume that I’m wrong on this one. Anyway, check returns nil if the process is running and a truthy value if the process is not running, so you want to assert that PTY.check(pid) returns true to verify that your process has exited.

Results Race Condition

Another gotcha you need to be aware of is that, when you spawn another process with PTY like this, you now have two separate processes, which means that you can run into timing issues when making assertions about the output from your spawned process. So this assertion can fail sometimes:

PTY.spawn('path/to/executable') do |stdout, stdin, pid|
  # send a command and clear it from stdout
  response = stdout.gets
  assert_equal 'expected response', response
end

What’s going on is that your test process has sent a command to your spawned process, then instantaneously tries to read the response from stdout. But if your spawned process has a small delay in writing to it’s stdout (for whatever reason; the delay only needs to be miniscule), then the response your test gets will be blank and the assertion will fail. You can get around this by having your test process sleep:

PTY.spawn('path/to/executable') do |stdout, stdin, pid|
  # send a command and clear it from stdout
  sleep 0.1
  response = stdout.gets
  assert_equal 'expected response', response
end

Having your test process sleep before getting every response is not really optimal, though. With a large enough test suite, tenths of a second start to add up. What you really want is a method that tries getting the response for a certain amount of time before giving up. Something like this:

def get_response(stdout)
  start = Time.now
  try_for = 2
  response = nil

  loop do
    response = stdout.gets.chomp
    break if response || Time.now > start + try_for

    sleep 0.1
  end

  response
end

What’s going on there is that, in each iteration of the loop, we try to get a response. If we get a response, or if we have exceeded the total amount of time we are going to try for, then we break out of the loop. Otherwise, we sleep for 0.1 seconds, after which point the loop runs again.

Prompt Race Condition

Another race condition issue can pop up if your CLI app shows the user a prompt, like this (the > is the prompt):

$ rpn_party
> commmand
result
>

What can happen is that, because the test is running at computer speed, rather than human speed, the test can send the next command in the sequence before the CLI app’s stdout has printed the > after the last command. Say you have a test like this:

PTY.spawn('path/to/executable') do |stdout, stdin, pid|
  stdin.puts 'command1'
  response1 = stdout.gets

  stdin.puts 'command2'
  response2 = stdout.gets
  assert_equal 'response2', response     # fails sometimes
end

This assertion will sometimes fails in a not so obvious way (because reasoning about async is hard). When it does fail, you’ll get a message like:

Expected "response2"
Actual   "> response2"

The reason it fails like this is because what you think is happening is this sequence of events:

1. Command 1 is sent
2. Response 1 is received
3. Prompt 1 is printed to screen
4. Command 2 is sent
5. Response 2 is received
6. Prompt 2 is printed to screen

But what actually happens sometimes is this:

1. Command 1 is sent
2. Response 1 is received
3. Command 2 is sent before the prompt has been printed
4. Prompt + Response 2 are then received together

That’s how a response like > response happens instead of response.

To resolve this, we need to wait until the prompt is printed before we send a request. I think it’s best to wrap this up in a method, like so:

def wait_for_prompt(stdout)
  start = Time.now
  try_for = 1

  loop do
    prompt = stdout.getc
    break if prompt == '>' || Time.now > start + try_for

    sleep 0.1
  end
end

And there you go. Now you can integration test your CLI apps.

Before I get started, I’ll just mention that all of the code examples in this post are from the demo app, which you can find on Github. Also, the demo app is live, so you can see everything in action.

Webcal

Webcal is what allows a user to click on a link in their browser to subscribe to a calendar in their calendar app.

So, you click on a link in your browser, then a dialog window pops up in your calendar app to subscribe to the calendar:

Then you click Subscribe and, voilà, now that power lunch you have scheduled is in your calendar:

In the future, your calendar app will periodically poll the calendar’s URL to check for updates.

Controller

Here’s what your controller should look like:

class CalendarsController < ApplicationController
  def show
    respond_to do |format|
      format.ics do
        cal = Icalendar::Calendar.new
        cal.x_wr_calname = 'Awesome Rails Calendar'
        cal.event do |e|
          e.dtstart     = DateTime.now + 2.hours
          e.dtend       = DateTime.now + 3.hours
          e.summary     = 'Power Lunch'
          e.description = 'Get together and do big things'
        end
        cal.publish
        render plain: cal.to_ical
      end
    end
  end
end

View code on Github

I’ll highlight the important parts:

1. Your controller needs to be unauthenticated, or else the user’s calendar app won’t be able to access the calendar. So, if your ApplicationController is authenticated, your CalendarsController will probably need to inherit directly from ActionController::Base to avoid authentication.

2. If your calendar contains sensitive data that should only be available to a specific user, you will need to use a strategy similar to password reset links so that you aren’t leaking sensitive data. So, your calendar should be available at a path like /calendar/u/:some_long_random_string, instead of /calendar/u/:user_id.

3. You respond_to the ics format.

4. You set cal.x_wr_calname to specify the default calendar name that will be displayed in the user’s calendar app.

5. You use render plain: cal.to_ical to actually render the calendar. In older versions of Rails you would specify text instead of plain.

You can learn about everything else going on in this controller by reading the icalendar gem documentation. It’s all standard stuff.

Link

Here’s what your subscription link should look like:

<%= link_to 'Subscribe', calendar_url(protocol: :webcal, format: :ics) %>

View code on Github

I’ll highlight the important parts:

1. You have to use a full URL, not just a path. The reason for this is the same as for Rails mailers: unlike the browser, the user’s calendar app won’t know the domain. So, in this example, we’re using calendar_url instead of calendar_path.

2. You specify the webcal protocol. This will create a link that starts with webcal://, instead of http://, which is what tells the browser to hand the link off to the user’s calendar app.

3. You specify the ics format. This will create a link that ends in /calendar.ics, instead of just /calendar, which is what tells Rails to handle the request with the format.ics block in the controller.

That’s it, you’re done!

Just a reminder that you can find the demo app on Github, or see it in action on Heroku.

A View Backed Model

Let’s start by unpacking the term View Backed Model. In Rails, each model is backed by a table in the underlying database. So, our Author model is backed by the authors table and our Post model is backed by the posts table. A view backed model is a model that is backed by a database view, instead of by a normal table.

Great, so what exactly is a database view? Well, you could go read the Postgres documentation on views, but the short version is that views allow you to encapsulate a query behind a table interface. Basically, a database view lets you assign a name to a query and then interact with the result that query generates as if it were any other table in your database (filter it, join to it, etc.). Don’t worry if that seems a little fuzzy to you, because we’re about to make it more concrete by applying it to our Greatest Per Group problem.

Let’s start by looking at the query we came up with back in Part 1 of this series:

SELECT authors.*, newest_posts.title AS newest_post_title
FROM authors
LEFT JOIN (SELECT DISTINCT ON (author_id) *
           FROM posts
           ORDER BY author_id, created_at DESC)
           AS newest_posts
ON newest_posts.author_id = authors.id

Working from the inside out, what we’re doing in this query is:

1. Using a subquery to creating a temporary table.
2. Naming the temporary table newest_posts.
3. And then joining it to the authors table.

A database view allows us to encapsulate the subquery that makes up our temporary newest_posts table and make it permanently available throughout our database. Here’s how we create that database view:

CREATE VIEW newest_posts AS
  SELECT DISTINCT ON (author_id) *
  FROM posts
  ORDER BY author_id, created_at DESC

As you can see, it looks remarkably similar to our subquery from above, just rearranged a little bit. Now we have a view called newest_posts that we can treat like a table. And every time we refer to it, PostgreSQL will run this query and treat the results as a table.

So if we want the newest post for each author, we can do:

SELECT * FROM newest_posts

And to get the newest post for a specific author, we can do:

SELECT * FROM newest_posts WHERE author_id = 42

Or, if we want to accomplish our original query from above, we can do:

SELECT authors.*, newest_posts.title AS newest_post_title
FROM authors
LEFT JOIN newest_posts
ON newest_posts.author_id = authors.id

As you can see, our view is a drop in replacement for the original subquery. Which makes sense, because PostgreSQL is literally running the exact same query in the spot where it says newest_posts.

Database views are a powerful way to encapsulate commonly used or complicated queries. And they can be an incredibly useful tool in a Rails app, because it’s fairly easy to hit the edge of how well ActiveRecord maps to what you’re trying to accomplish with a query. But ActiveRecord understands tables pretty well and, since a database view acts like a table, they are a powerful way to force ActiveRecord to just do what you need it to do.

Great, so how do we use database views in a Rails app?

Database Views in Rails

Since a database view is something that exists in the database, we’ll create ours in a migration:

class CreateNewestPostsView < ActiveRecord::Migration
  def up
    execute <<-sql
      CREATE VIEW newest_posts AS
        SELECT DISTINCT ON (author_id) *
        FROM posts
        ORDER BY author_id, created_at DESC
    sql
  end

  def down
    execute <<-sql
      DROP VIEW newest_posts
    sql
  end
end

And since a database view behaves like a table, all we need to do is create a model that follows the Rails naming conventions and things will just work (well, mostly, but we’ll get to that in a bit):

class NewestPost < ActiveRecord::Base
end

Of course, Rails has no idea that the newest_posts table that it expects to back the NewestPost model is not actually a table at all, but is instead a dynamic query result comprised of rows from the posts table. PostgreSQL takes care of all of that behind the scenes. All Rails knows is that there is a table-like thing called newest_posts which returns one row per Author and has an author_id column that it can use to join to the authors table. So, if we do:

NewestPost.all

Rails will generate SQL like:

SELECT * FROM newest_posts;

But PostgreSQL will actually do:

SELECT *
FROM (SELECT DISTINCT ON (author_id) *
      FROM posts
      ORDER BY author_id, created_at DESC)

This allows us to use use our View Backed Model just like any other model in our app. For instance, we can set up associations:

class NewestPost < ActiveRecord::Base
  belongs_to :author
end

Then we can create the corresponding has_one association in our Author model:

class Author < ActiveRecord::Base
  has_many :posts
  has_one :newest_post
end

And now we have a normal ActiveRecord association that we can use for eager loading:

@authors = Author.includes(:newest_post)
@authors.each do |author|
  puts "#{author.name}'s latest post: #{author.newest_post.title}"
end

And Rails only generates two queries: one to load the authors and another to load the newest posts.

A few things to be aware of

If you’ll recall, we created our database view in a Rails migration, which is all fine and dandy. Except that we had to execute raw SQL instead of using one of ActiveRecord::Migration’s handy methods (like create_table or add_index). That’s because Rails doesn’t really know what views are. And a side effect of this is that, if you take a look at your schema.rb after creating a view, you won’t see your view in there.

So if somebody tries to get their development environment up and running with rake db:create, they won’t have any database views. Which is probably not what you want if you’re trying to use views in your Rails app.

That’s okay, though, because there are two ways around this.

The first option is to configure your app to use SQL instead of Ruby for schema dumps. By default, Rails is configured to do Ruby schema dumps. That’s why schema.rb ends in .rb and consists of a series of ActiveRecord::Migration methods designed to recreate your schema from scratch.

But you can configure Rails to do SQL schema dumps instead. In which case you’ll get a structure.sql file filled with a bunch of SQL statements that look like the output of pg_dump. Critically, this type of schema dump will include statements to recreate any database views (and extensions) you’ve created.

You can read the documentation on how to do this, but it’s pretty simple. In config/application.rb, you change this line:

config.active_record.schema_format = :ruby

to:

config.active_record.schema_format = :sql

You’ll probably want to delete your schema.rb file, too.

And that’s all there is to it.

There is another, more comprehensive approach to solving the schema dump problem, which is to use the Scenic gem. Scenic actually adds ActiveRecord::Migration methods which allow you to avoid using SQL formatted schema dumps. Plus it has a bunch of other convenient tools for working with database views in your Rails app. I haven’t used it, personally, but I plan to.

The other major consideration with View Backed Models is a little bit more subtle. And it is that instances of a View Backed Model are not instances of the model that is backed by the table they’re derived from. That’s a bit much to digest, so let me explain it with some code.

Let’s grab a NewestPost:

@newest_post = NewestPost.first

Because the newest_posts database view selects all of the columns from the posts table that it is derived from (that little * at the end of the second line of our view):

CREATE VIEW newest_posts AS
  SELECT DISTINCT ON (author_id) *
  FROM posts
  ORDER BY author_id, created_at DESC

The data from those columns is available on instances of NewestPost. So we can do:

@newest_post.title
@newest_post.body
@newest_post.created_at

And they all work as we would expect them to. But any methods or associations defined on Post will not work. So if Post has_many :comments and you try to do something like:

@newest_post.comments

It will not work.

It’s easy to start thinking of instances of NewestPost as instances of Post, since that’s what they are in a logical sense, but to the dumb machine, NewestPost and Post are just two completely unrelated model classes.

So, around now you might be thinking to yourself, “Aha, but this is easy to solve. I’ll just make NewestPost inherit from Post and then it’ll have access to all of the methods and associations defined on Post.”

In a sane world, that would be a perfectly reasonable way to proceed. But, unfortunately, we exist in a world where Rails has Single Table Inheritance (STI). I’ll spare you a deep dive on STI, but the gist of it is that, for descendants of ActiveRecord::Base, Rails stomps all over your ability to have one class inherit from another in a way that would be useful to us here.

But, as STI taketh away, so STI giveth.

Because STI is such a convoluted mess, it necessitates the existence of the very tool that will allow us to escape the conundrum that it has created for us. And that tool is the becomes method. You can read the API docs, but becomes uses the attributes of the object you call it on to create an instance of the class you pass to it. Yet another thing that is easier understood with code. Here it is in action:

@newest_post.becomes(Post)

Which is more or less the equivalent of doing:

Post.new(@newest_post.attributes)

And since we now have an instance of Post, we can use all of the methods and associations defined on Post without having to redefine them. Which allows us to stuff like this:

@newest_post.becomes(Post).comments

So there you go. It’s a bit of a kludge and I wish it were more elegant, but it works. And, in my opinion, the power and utility of View Backed Models are worth a modest sacrifice in elegance.

And that’s it. Now you know way more than you ever wanted to about this strange little corner of the Rails world.

A scoped has_one association

We’ll need our Greatest Per Group query from Part 1 fresh in our heads for this next part, so here it is again:

SELECT authors.*, newest_posts.title AS newest_post_title
FROM authors
LEFT JOIN (SELECT DISTINCT ON (author_id) *
           FROM posts
           ORDER BY author_id, created_at DESC)
           AS newest_posts
ON newest_posts.author_id = authors.id

There are three things happening in this query:

1. We’re creating a temporary newest_posts table that returns a single post – the newest one – for each author.
2. Then, we’re joining this temporary table to our authors table on authors.id = newest_posts.author_id.
3. Finally, we’re selecting some data from that newest_posts table to return along with our author data.

Let’s forget about the third item – selecting the data – for now and instead focus on the first two.

We’re creating a table that has a single row per author and we’re joining it to the authors table on the author_id foreign key. If that feels eerily familiar to you, it’s because this is essentially exactly what happens when we have a has_one association defined on a model. What we need to do is somehow get Rails to use our subquery for a has_one we define. Here’s our subquery again:

SELECT DISTINCT ON (author_id) *
FROM posts
ORDER BY author_id, created_at DESC

Luckily, Rails allows us to customize – to scope – the query used for a has_one association. Here’s how we do that with our subquery:

class Author
  has_many :posts
  has_one :newest_post, -> {
    select("DISTINCT ON (author_id) *")
    .order(:author_id, created_at: :desc)
  }, class_name: "Post"

This basically tells Rails that 1) we have a has_one called newest_post, 2) it is essentially a Post, and 3) to use .select("SELECT DISTINCT ON (author_id) *").order(:author_id, created_at: :desc) when generating the query. And because it is just like any other has_one association, you can eager load it:

@authors = Author.includes(:newest_post).all
@authors.each do |author|
  puts "Newest post for #{author.full_name}: #{author.newest_post.title}"
end

And it only fires off two queries.

Why this approach is awesome

Let’s compare and contrast. Here’s how we achieved this with custom query methods:

@authors = Author
  .select("authors.*,
           newest_posts.title AS newest_post_title")
  .joins("LEFT JOIN (SELECT DISTINCT ON (author_id) *
                     FROM posts
                     ORDER BY author_id, created_at DESC)
          AS newest_posts ON newest_posts.author_id = authors.id")

And here’s what we just did:

class Author
  has_many :posts
  has_one :newest_post, -> {
    select("DISTINCT ON (author_id) *")
    .order(:author_id, created_at: :desc)
  }, class_name: "Post"

Already this feels like a big improvement. It eliminates a lot of the inline SQL of the customer query methods, it succinctly describes – has_one :newest_post – what is going on, and it’s just more intuitive. The idea that an author has a newest post makes sense. It is easy idea to hold in your head. All that SQL stuffed inline into your Ruby code is not easy to hold in your head.

But there are other benefits to this approach as well.

One of the problems we identified with using custom query methods is that they aren’t very composable. If you abstract them away behind named scopes, you can’t combine the named scopes because the SELECT clauses interfere with each other. But scoped has_one associations are as composable as you can get.

Let’s say we also wanted an oldest_post association:

class Author
  has_many :posts
  has_one :newest_post, -> {
    select("DISTINCT ON (author_id) *")
    .order(:author_id, created_at: :desc)
  }, class_name: "Post"
  has_one :oldest_post, -> {
    select("DISTINCT ON (author_id) *")
    .order(:author_id, created_at: :asc)
  }, class_name: "Post"

This allows us to do something like this:

@authors = Author.includes(:newest_post, :oldest_post)
@authors.each do |author|
  puts "Newest post for #{author.name}: #{author.newest_post.title}"
  puts "#{author.name} first posted on #{author.oldest_post.date}"
end

Compared to the mess of SQL and named scopes that we would need if we were using custom query methods, this is a breath of fresh air. Not only is it composable, it’s way less code. And it works the way Rails wants to work. What do I mean by that? Well, because we’ve set these up as has_one assocations – and that’s really what they were all along; an author’s newest or oldest post is a single post that is related to the author – we get a bunch of stuff for free.

For instance, let’s say we want to show a list of authors with the title of their newest post, but not include any authors who have not posted yet. Now that’s simple. We can use our new has_one relations with joins:

@authors = Author.joins(:newest_post)
@authors.each do |author|
  puts "Newest post for #{author.name}: #{author.newest_post.title}"
end

And, since these are associations, they’ll return full-fledged Post instances. So if we decide that we want to display other attributes of the newest post, there’s no need to modify any SQL. We simply call the attribute we need:

<%= pluralize(@author.newest_post.word_count, "word") %>

And if we want to have a link to the newest post, that’s simple too:

<%= link_to @author.newest_post.title, post_path(@author.newest_post) %>

Or we can use it in a form builder:

<%= form_for @author.newest_post do |f| %>
  <%= f.text_field :title %>
  <%= f.submit %>
<% end %>

Or we can access it’s assocations:

<%= pluralize(@author.newest_post.comments.count, "comment") %>

You get the idea. You can do whatever you want with it, because it’s an instance of one of your models. But each one of these would have required different custom SQL if we were just using custom query methods to add attributes to our Author instances.

How this approach works

So that covers why this approach is better, but I want to dive into how this approach works because, even though it might not look like it, we’re getting ActiveRecord to use pretty much the same SQL query in both approaches. The main difference is in where we put the parts of that query.

Let’s revisit our original query to return a list of the newest posts for each author:

SELECT DISTINCT ON (author_id) *
FROM posts
ORDER BY author_id, created_at DESC

In the first approach, we put this into a subquery, joined the result of that subquery to our authors table, then selected the column we needed:

SELECT authors.*, newest_posts.title AS newest_post_title
FROM authors
LEFT JOIN (SELECT DISTINCT ON (author_id) *
           FROM posts
           ORDER BY author_id, created_at DESC)
           AS newest_posts
ON newest_posts.author_id = authors.id

In this approach we don’t have to do all of that because Rails already understands how to join has_one relations. Instead, all we have to do is create a has_one relation that executes our query and then Rails will know what to do with it. And if you take a closer look at the has_one we defined above, you’ll see that it looks remarkably like that original query of ours:

   has_one :newest_post, -> {
    select("DISTINCT ON (author_id) *")
    .order(:author_id, created_at: :desc)
  }, class_name: "Post"

In the last argument, we’re telling Rails that this has_one relation should return an instance of a Post. Normally, Rails will figure this out automatically based on the name we give our has_one relation, but that won’t work in this case, so we have to specify the model class. And Rails knows that the table for the Post class is posts. Then, in the lambda passed as the second argument, we’re specifying the SELECT DISTINCT ON and ORDER clauses that we want Rails to use when querying the posts table.

And the rest, as they say, is magic.

Well, almost

In my opinion, the approach we covered here – using a scoped has_one association – really hits the sweet spot for implementing Greatest Per Group queries in Rails. It works for most use cases and, because it does things in a Rails-y way, it’s intuitive and composable. But it breaks down with more complicated stuff that’s at the boundary of how much ActiveRecord “understands” SQL. Like, for instance, if you try to use joins in the custom scope on the has_one relation, or in more complicated uses of the has_one relation when doing eager loading.

That’s okay, though, because View Backed Models, which we’ll cover in Part 4, are the knife that cuts through any knot.

Taking advantage of ActiveRecord’s dynamic model attribute behavior

One of the things about Ruby on Rails that I found most confusing when I first got started is the way that you don’t have to define the attributes of your models in your models. Instead, you just add columns to the underlying database table – usually with migrations, but it doesn’t actually matter how they get there – and ActiveRecord will automatically add those attributes to your model instances and create accessors for them.

Though this confused me when I was newer to Rails, I have come to appreciate it as I’ve gained experience. It strikes a nice balance between having an easy default and allowing you to roll up your sleeves and get your hands dirty when necessary. And getting our hands dirty is exactly what we’ll do here.

Because Rails dynamically creates attributes on our model for any columns returned by the database, all we have to do is get Rails to execute our query from part 1 when loading our collection of authors. Then each author will have an attribute called newest_post_title with the title of the newest post.

Here’s that query again:

SELECT authors.*, newest_posts.title AS newest_post_title
FROM authors
LEFT JOIN (SELECT DISTINCT ON (author_id) *
           FROM posts
           ORDER BY author_id, created_at DESC)
           AS newest_posts
ON newest_posts.author_id = authors.id

Part of the magic of Rails is that it’s intelligent naming defaults allow us to avoid writing SQL most of the time. But the tools to get our hands dirty with SQL are right there, waiting for us when we need them. Here’s how we get Rails to execute that query:

@authors = Author
  .select("authors.*,
           newest_posts.title AS newest_post_title")
  .joins("LEFT JOIN (SELECT DISTINCT ON (author_id) *
                     FROM posts
                     ORDER BY author_id, created_at DESC)
          AS newest_posts ON newest_posts.author_id = authors.id")

It’s actually incredibly straightforward. We’re using the select method to specify the SELECT statement and then chaining on the joins method to specify the LEFT JOIN. If you do this in a Rails console, you’ll see that ActiveRecord executes the exact SQL query we want.

Now, we can access the newest_post_title when we iterate over our list of authors like this:

@authors.each do |author|
  puts "Newest post for #{author.name}: #{author.newest_post_title}"
end

And it only generates a single query, rather than the N + 1 queries we would generate otherwise.

Taking advantage of the way ActiveRecord dynamically creates an attribute for every column present in the query result is an incredibly powerful technique. It allows us to essentially add any attribute we need to any model, all on the fly, without having to set anything up in our model class ahead of time.

All we have to do is craft the right SQL statement.

Some shortcomings with this approach

The most obvious shortcoming with this approach is that it is rather verbose and ugly. You’ve got a lot of SQL inside those chained select and joins methods and, unless you wrote the code yourself, it takes a bit of puzzling to figure out what’s going on. This shortcoming is easy to overcome, though. We can encapsulate all that logic in a named scope:

scope :with_newest_post_title, -> {
  .select("authors.*,
           newest_posts.title AS newest_post_title")
  .joins("LEFT JOIN (SELECT DISTINCT ON (author_id) *
                     FROM posts
                     ORDER BY author_id, created_at DESC)
          AS newest_posts ON newest_posts.author_id = authors.id")
}

This named scope allows you to just do Author.with_newest_post_title instead of typing that huge long thing out each time, which is definitely a win.

A less obvious shortcoming with this approach is that it is not composable. Let’s say you also want to display how long each author has been writing for the site. Now you need the date of each author’s oldest post as well as the title of each author’s newest post. You could create another named scope for the date of the author’s first post:

scope :with_oldest_post_date, -> {
  select("authors.*,
          oldest_posts.date AS oldest_post_title")
  .joins("LEFT JOIN (SELECT DISTINCT ON (author_id) *
                     FROM posts
                     ORDER BY author_id, created_at ASC)
          AS oldest_posts ON oldest_posts.author_id = authors.id")
}

But when you try to chain the two named scopes together – that is, when you try to compose them – with something like:

Author.with_newest_post_title.with_oldest_post_date

It won’t work. That’s because the SELECT statements from each named scope conflict with each other. Instead, you would need a third named scope that combines the two:

scope :with_newest_post_title_and_oldest_post_date, -> {
  select("authors.*,
          newest_posts.title AS newest_post_title,
          oldest_posts.date AS oldest_post_title")
  .joins("LEFT JOIN (SELECT DISTINCT ON (author_id) *
                     FROM posts
                     ORDER BY author_id, created_at DESC)
          AS newest_posts ON newest_posts.author_id = authors.id")
  .joins("LEFT JOIN (SELECT DISTINCT ON (author_id) *
                     FROM posts
                     ORDER BY author_id, created_at ASC)
          AS oldest_posts ON oldest_posts.author_id = authors.id")
}

Does that look like a hot mess to you? Because it looks like a hot mess to me. We’ve got three named scopes to account for all the possible combinations of the two attributes we might want to include in our query. That is literally the definition of combinatorial explosion. And anyway, who wants big blocks of SQL mixed in with their Ruby?

But this approach is a pretty handy tool to have in your back pocket when you know exactly how to achieve what you want in SQL and you just need to get it done fast. The next two parts will cover approaches that improve on the shortcomings of this approach, plus bring other benefits to the table:

1. Using a scoped has_one association (clean and elegant)
2. Using a view backed model (when all else fails)

Something that I find myself needing to do frequently in my Ruby on Rails apps is to get the largest/smallest/newest/oldest/whateverest element in a model’s has_many association. So, if my app has a Author model which has_many :posts, I might want to show the title of an author’s most recent post when I’m displaying other information about the author.

If I’m only showing a single author in my view, this is easy to do without encountering a performance problem. I would just do something like this:

@post = @author.posts.order(created_at: :desc).first
@post.title

That generates two queries: one to load the author and another to load the most recent post. Pretty simple.

But if I want to show a list of authors along with the title of each author’s most recent post, then we have to do something different. If we try the same approach, we can generate a lot of queries: one to load the list authors and then another N queries as we loop over each author and load their most recent post. This is the dreaded N + 1 query problem.

We really only want to fire off two queries: one to load the authors and then one more to load just their most recent posts. I am calling this a Greatest Per Group query because it is a special case of a Greatest N Per Group query, where N = 1. And it doesn’t have to be the greatest per group, it could be the largest, smallest, newest, oldest, or whatever-est per group. The point is that you are trying to rank the records on the many side of a one-to-many relationship and only return a single record.

Okay, now that we know what a Greatest Per Group query is and where it’s useful, let’s figure out how they work.

How Greatest Per Group queries work

The good news is that SQL’s reason for existence is to enable exactly this kind of thing. What we want to do is JOIN our authors table to a table of posts which only has one post for each author: the most recent post. So we know the overall structure will look like this:

SELECT authors.*, newest_posts.title AS newest_post_title
FROM authors
LEFT JOIN newest_posts ON newest_posts.author_id = authors.id

That’s a fairly straightforward join. In this case we’re using a LEFT JOIN since we want to list all authors, even if they have never written a post. What we glossed over, though, was that newest_posts table. The problem is that we don’t have a newest_posts table. So how are we going to join a table that doesn’t exist?

One of the great things about SQL is that it allows you to use the results of a query as a temporary table that you can join to. So, we’re going to create a temporary newest_posts table and then join it to our authors table. First, though, we need to figure out how to generate a table that only has the most recent post for each author.

Your first thought might be that this is a perfect place to use GROUP BY. That was my first thought, too. But GROUP BY won’t work for our purposes here because – in Postgres, at least – it limits what columns you can select to the column(s) you’re grouping by and columns created by aggregate functions like MIN, MAX, AVG, SUM, and COUNT. So a query like this will return an error:

SELECT posts.*
FROM posts
GROUP BY author_id
ORDER BY created_at DESC

The best we can do with GROUP BY is something like this:

SELECT author_id, MAX(created_at) AS newest_post
FROM posts
GROUP BY author_id

That will return a result that looks like this:

author_id | newest_post
----------+------------
        1 | 2015-08-07
        2 | 2015-11-15
        3 | 2014-04-23
        4 | 2015-01-18

This could be useful if all we want to do is show when the last post was created. But we want to show the title of the newest post’s, so this isn’t very useful to us. That means GROUP BY is out as an option, at least in Postgres. So, what are our options?

Well, Postgres has this marvelous thing called DISTINCT ON. You can go read the documentation for the full run down, but the short description is this: DISTINCT ON will only return the first row for each group of rows that share a unique combination of your distinct column(s). So you can use ORDER BY to sort your results and that will determine which row DISTINCT ON returns. This allows you to get only the newest, oldest, largest, smallest, etc. records for an association. Plus, unlike GROUP BY, it allows you to SELECT any and all columns that you desire.

Here’s how we get a list of each of the most recent posts per author using DISTINCT ON:

SELECT DISTINCT ON (author_id) *
FROM posts
ORDER BY author_id, created_at DESC

That’s it. Fairly simple when it’s all said and done. There are a couple of things which tripped me up the first time I used DISTINCT ON, though, so I’ll list them here:

• There is no comma directly after the DISTINCT ON (author_id) part of the SELECT statement. But there are commas after that if you’re selecting more than one column.
• The first (leftmost) column in the ORDER BY clause has to be the distinct column – author_id in our case. This is kinda goofy, since I think it’s an implementation detail leaking out, but it’s a thing that Postgres will make you do.
• The remaining items in the ORDER BY clause work as you normally expect and are what determine whether you get the newest, oldest, largest, smallest, or whateverest record, so pay attention to them.

Okay, let’s put it all together now. So, we have our overall query:

SELECT authors.*, newest_posts.title AS newest_post_title
FROM authors
LEFT JOIN newest_posts ON newest_posts.author_id = authors.id

And we know that the newest_posts table that we’re joining to the authors table is actually going to be a temporary table that we generate with this query:

SELECT DISTINCT ON (author_id) *
FROM posts
ORDER BY author_id, created_at DESC

But how do we actually do that? Well, we wrap the query in parentheses, which makes it a subquery, then we give it a name with AS newest_posts, and then we join it. Here’s what that looks like:

SELECT authors.*, newest_posts.title AS newest_post_title
FROM authors
LEFT JOIN (SELECT DISTINCT ON (author_id) *
           FROM posts
           ORDER BY author_id, created_at DESC)
           AS newest_posts
ON newest_posts.author_id = authors.id

This will return a result that looks like this:

author_id | first_name | last_name | newest_post_title
----------+------------+-----------+------------------
        1 | Jane       | Austen    | Persuasion
        2 | Charles    | Dickens   | Oliver Twist
        3 | J.D.       | Salinger  | Nine Stories

Okay, now we know what our SQL should look like. But how do we put that into our Rails app in a way that’s useful? Well, there are three approaches that I will cover:

1. Using custom query methods (quick and dirty)
2. Using a scoped has_one association (clean and elegant)
3. Using a view backed model (when all else fails)

So far I’ve been happy with how this has turned out. For very little work on my part (and with some help from others), our YouTube channel now has 70 videos, 69 subscribers, and ~4,800 views. Considering that we typically get 70-80 attendees at each meeting, this has greatly expanded the reach of our merry little band of Ruby programmers.

In this post I’m going to run through the gear and software I use to make these recordings.

Camera

I use an iPhone 6+. Any recent model will do, though. I recommend having at least 10gb of free space on whatever iPhone you use. If you plan on using your personal phone, this will probably mean having at least a 32gb model.

Tripod

Any cheap tripod will do. I’m using an Amazon Basics 60" Lightweight Tripod w/ Bag ($24).

iPhone Tripod Mount

There are a bazillion iPhone tripod mounts out there. I’m using the Glif Adjustable Tripod Mount for Smartphones ($30). It works well enough, but there are probably better and/or less expensive options.

Wireless Microphone

You can spend a lot of money on microphones. And it can be tempting to do so, since having a wireless lavalier microphone is critical to capturing high-quality audio of the speaker. But, as far as I can tell, the “pro” options all cost at least $600, which is way outside of my price range.

Thanks to a suggestion from my friend, Chuck Lauer Vose, I learned that there is a segment of the microphone market that is oriented toward teachers, instead of audio/video professionals. And since teachers and schools are not typically flush with cash, these options tend to be much more affordable. The wireless microphone that I settled on is the Movo WMIC50 2.4GHz Wireless Lavalier Microphone System ($115).

Technically speaking, there is a tiny bit of lag with this microphone. In practice, though, you can’t tell unless you’re looking for it. They operate on AAA batteries and there is essentially zero setup; they just work.

Microphone Mount

You’ll need a way to mount the wireless mic receiver to your tripod, since it needs to plug into your phone and, thus, be near it. I’m using a generic clip on mount ($10). I actually don’t know what the trade name is for these things, but they’re essentially a clip with a 1/4" threaded mount where you can screw on the mic receiver (or other gear, from what I gather).

Microphone Adapter

I have no idea why, but the output from battery powered microphones, including the one I linked to above, is not compatible with the iPhone’s microphone jack. So, you’ll need an adapter, like this one from kVconnection ($25) that I found after Googling about this for a while. You can probably find this elsewhere, but I just cargo-culted this product from somebody else’s blog post and it works, which is far enough down the rabbit hole for my tastes.

Batteries & Power

You’ll need AAA batteries for the wireless microphone system. I like Sanyo’s Eneloop rechargeables. Here’s an 8-pack of AAA Eneloops ($16) and the Eneloop charger ($21).

You’ll also want to bring an extension cord, power strip, and the power adapter for your phone. Just trust me on this. Your mic batteries will run out. Your phone battery will run out. It’s better to just be prepared for this eventuality.

Telephoto Lens (if you need it)

I’m lucky enough to be able to set my phone & tripod up nice and close to the speakers at the Portland Ruby Brigade monthly meetings, but distance to the speaker is largely dictated by the venue. You may run into a situation where you need some sort of optical zoom (you don’t want to use digital zoom). There are a bunch of iPhone compatible telephoto lenses. I bought the Olloclip Telephoto Lens ($100), but can’t recommend it, per se, since I haven’t actually needed to use it.

Software

I originally used the built-in camera app on the iPhone, but videos made with that app get automatically synced to Apple’s photo cloud service if you have syncing turned on (I do), and that syncing process takes forever and destroys my home Internet connection while it’s happening. So I recently switched to using Apple’s iMovie app ($5). This keeps the videos from being synced to the cloud and provides some useful editing options (and, for videos under a certain length, you can upload directly to YouTube from within the app). My videos are usually too large to upload directly to YouTube, though, so I Airdrop the finished videos to my Mac, then upload to YouTube from there.

The idea is to put all the static pages in a PagesController which has a single action and corresponding view for each static page. So your homepage would have an index action and a corresponding index.html.erb view file. And your About Us page might be an about action with an about.html.erb view. And so on. Here’s what it all looks like.

In your config/routes.rb:

App::Application.routes.draw do
  root to: "pages#index"
  get :contact, to: "pages#contact"
  get :about, to: "pages#contact"
end

In your app/controllers/pages_controller.rb:

class PagesController < ApplicationController
  def index
  end

  def contact
  end

  def about
  end
end

And then you just have a single view to match each action in app/views/pages/.

This is all fine and straightforward, but those route declarations really bothered me. They strike me as very repetitious, especially as you add more and more static pages, so I went digging through the Rails documentation and came across the controller route macro. It essentially scopes a bunch of routes to a specific controller. In other words, exactly what I was looking for.

Now your routes look like this:

App::Application.routes.draw do
  controller :pages do
    root to: :index
    get :contact
    get :about
  end
end

This solution is two lines longer, yet so much more concise. It nicely groups together a set of routes that are related to each other. I think the Rails route scoping documentation is worth a read if you find yourself repetitively adding the same controller value to a bunch of different routes.

So, if you have routes defined like this:

resources :companies do
  resources :addresses
end

The route /companies/1/address/2 will point to the AddressesController, instead of, say, the Companies::AddressesController. To me, this is confusing and unintuitive, but it is not a huge problem for smaller apps. It gets to be difficult once a routes file starts to look more like this:

resources :companies do
  resources :addresses
  resources :people
  resources :orders
end

resources :products
resources :industries
resources :salespeople

Now, when you look in your controllers directory, it will look like this:

# ls app/controllers
addresses_controller.rb
companies_controller.rb
industries_controller.rb
orders_controller.rb
people_controller.rb
products_controller.rb
salespeople_controller.rb

To figure out that the AddressesController, PeopleController, and OrdersController all handle nested routes requires either 1) opening each one up and reading it, or 2) perusing the routes file. Essentially, the app’s routes, which are hierarchical in nature, have been flattened into a single level of hierarchy. Okay, maybe you don’t think it’s that big of a deal, but imagine trying to make sense of this when you have dozens of controllers and many of them handle nested routes. Luckily, Rails has some tools to help organize your controllers.

What we really want to do is namespace all the controllers that handle nested routes for Company associations in a module called Companies. So, the AddressesController from above would become the Companies::AddressesController and so on. When we do this, because of the way Rails looks for classes we have defined in our app, we will end up with a companies directory in app/controllers that neatly contains all of our namespaced controllers.

Some code will explain this better. Here’s how you tell the router that a set of routes should be directed to controllers namespaced under a certain module:

resources :companies do
  scope module: "companies" do
    resources :addresses
    resources :people
    resources :orders
  end
end

resources :products
resources :industries
resources :salespeople

And now, when you look in your controllers directory, it should look like this (with the companies directory containing the namespaced controllers):

# ls app/controllers
companies_controller.rb
companies/
industries_controller.rb
products_controller.rb
salespeople_controller.rb

This is much more comprehensible. You know that all the controllers you see in the root controllers directory stand on their own, so to speak. And all the controllers in the companies directory are for handling nested Company routes (just remember to prefix the all of the controller class names with Companies:: when you define them). Plus, now there is a shared hierarchy between your routes file, the names of your controller classes, and the names and layout of the files on disk.

In all the Rails tutorials, you will see models organized something like this:

class Company < ActiveRecord::Base
  attr_accessible :name, :billing_address_id, :shipping_address_id,
    :industry_id, :salesperson_id, :price_level_id

  validates :name, :billing_address, :shipping_address,
    :industry, :salesperson, :price_level, presence: true

  validates :name, uniqueness: true

  belongs_to :billing_address, class: "Address"

  # And so on
end

I think this is just wrong.

All the logic related to a single attribute is spread out across the entire model. If you want to know what the name attribute is all about, you have to look in three different places, which means it’s easy to miss something. You can get away with this when a model is small, but it’s simply unworkable as a model grows larger than the height of your screen. Here’s how I would rewrite this model:

class Company < ActiveRecord::Base
  attr_accessible :name
  validates :name, presence: true, uniqueness: true

  attr_accessible :status
  validates :status, presence: true

  attr_accessible :billing_address_id
  belongs_to :billing_address, class: "Address"
  validates :billing_address

  # And so on
end

This is functionally equivalent to the first example, but it’s easy to immediately grasp everything there is to know about the name attribute – because it’s all there on those first two lines and nowhere else. The only drawback is that this approach results in even longer model files (since each attr_accessible and validates_presence_of macro gets its own line), but so what?

Do you really want to be reading the scribblings of a madman? Because that’s what the first approach is.

Now, there is no reason, ever, for a non-admin user to do anything in the admin section, so I created an Admin::ApplicationController and put a before_filter in it to redirect requests from non-admin users to the admin login page. Then, I had all of my Admin:: controllers inherit from it.

So I’m all set, right?

Of course, the weak link in this chain is that I have to remember to make my Admin:: controllers inherit from Admin::ApplicationController. But I’m just a human – and I frequently use generators to create my controllers – so I’m always forgetting to do this.

But this is a huge sercurity problem. And, now that I know how to test for orphaned routes, testing that all of my /admin routes are secure is a piece of cake.

In fact, this test caught me forgetting to inherit from Admin::ApplicationController not more than 3 hours ago. Enjoy!

require 'test_helper'

class Admin::InsecureRoutesTest < ActionDispatch::IntegrationTest
  ROUTES = Rails.application.routes.routes.map do |route|
    # Turn the route path spec into a string:
    # - Remove the "(.:format)" bit at the end
    # - Use "1" for all params
    path = route.path.spec.to_s.gsub(/\(\.:format\)/, "").gsub(/:[a-zA-Z_]+/, "1")
    # Route verbs are stored as regular expressions; convert them to symbols
    verb = %W{ GET POST PUT PATCH DELETE }.grep(route.verb).first.downcase.to_sym
    # Return a hash with two keys: the route path and it's verb
    { path: path, verb: verb }
  end

  test "admin routes should redirect to admin login page" do
    admin_routes = ROUTES.select { |route| route[:path].starts_with? "/admin" }
    insecure_routes = []

    admin_routes.each do |route|
      begin
        reset!
        # Use the route's verb to access the route's path
        request_via_redirect(route[:verb], route[:path])
        
        # If we aren't redirected to the admin login, the route is insecure
        unless path == admin_login_path
          insecure_routes << "#{route[:verb]} #{route[:path]}"
        end

        rescue ActiveRecord::RecordNotFound
        # Since we are blindly submitting "1" for all route params,
        # this error can pop up. If it does, it means that the request
        # got past the `before_filter` and to the code in the controller
        # where it attempts to locate the record in the database. That
        # means this is an insecure route.
        unless path == admin_login_path
          insecure_routes << "#{route[:verb]} #{route[:path]}"
        end

        rescue AbstractController::ActionNotFound
        # This error means the route doesn't connect to a controller
        # action. This is a problem if it happens, but this is a separate
        # concern and should be tested elsewhere. See my previous post
        # about testing for orphaned routes.
      end
    end

    # Fail if we have insecure routes.
    assert insecure_routes.empty?, "The following routes are not secure: \n\t#{insecure_routes.uniq.join("\n\t")}"
  end

Well, I recently went through routes.rb and did some major housekeeping. But this app has a bunch of models, with controllers for most of them in both the default namespace and an admin namespace.

So, even after I had spent the better part of a day cleaning house, my spidey-sense was still tingling. I just didn’t feel like I could be sure that everything was copacetic. And I’ve learned the hard way that this means it’s time for some tests…

Since routes are the first interface to a web app, I figured a good place to start would be to make sure there were no orphaned routes – routes that do not lead to a controller action. Also, I wanted a single test that would test all of my routes, so there would be no forgetting to add tests down the road.

Here’s that test.

require 'test_helper'

class OrphanedRoutesTest < ActionDispatch::IntegrationTest
  ROUTES = Rails.application.routes.routes.map do |route|
    # Turn the route path spec into a string:
    # - Remove the "(.:format)" bit at the end
    # - Use "1" for all params
    path = route.path.spec.to_s.gsub(/\(\.:format\)/, "").gsub(/:[a-zA-Z_]+/, "1")
    # Route verbs are stored as regular expressions; convert them to symbols
    verb = %W{ GET POST PUT PATCH DELETE }.grep(route.verb).first.downcase.to_sym
    # Return a hash with two keys: the route path and it's verb
    { path: path, verb: verb }
  end

  test "no orphaned routes exist" do
    orphaned_routes = []

    ROUTES.each do |route|
      begin
        reset!
        # Use the route's verb to access the route's path
        request_via_redirect(route[:verb], route[:path])
      rescue ActionController::RoutingError, AbstractController::ActionNotFound
        # ActionController::RoutingError means the controller doesn't exist
        # AbstractController::ActionNotFound means the action doesn't exist
        orphaned_routes << "#{route[:verb]} #{route[:path]}"
      rescue ActiveRecord::RecordNotFound
        # This error means the controller couldn't locate the specified record.
        # This happens because we are using 1 for all the route params
        # But, it means that the route points to a valid controller action,
        # so we ignore it
      end
    end

    # Fail if we have any orphaned routes
    assert orphaned_routes.empty?, "The following routes lead to nowhere: \n\t#{orphaned_routes.uniq.join("\n\t")}"
  end
end

That was a mistake on my part.

Most of his achievements are pretty high level – the kind of stuff you achieve over the span of an entire career. While it’s vitally important to poke your head above ground every once in a while and make sure you’re headed in the right direction, it turns out that’s not really what I need at the moment.

Instead, I’m looking to make a list that I will revisit every few weeks or so. For something like that, I find it’s more useful to look at things I want to accomplish within a horizon of six months to a year.

So, with that all said, here’s what my list looks like at the time of writing. I removed most of the high level stuff and added in a bunch of skills related to Ruby and Ruby on Rails.

Become more fluent in Ruby

• Commit the Ruby syntax to memory
• Commit significant standard libraries to memory
• Utilize a metaprogramming technique in an app

Become more fluent in Ruby on Rails

• Use Haml in an app
• Use JavaScript in an app the Rails way
• Use Coffeescript in an app
• Deploy an app with Capistrano
• Implement background jobs in an app
• Commit form helpers API to memory
• Commit ActiveRecord::Migration API to memory
• Commit ActiveRecord validation API to memory

Learn by teaching others:

• Present at a local user group
• Teach at a RailsBridge workshop
• Mentor a new Rails programmer

1. Identify the experiences that advance a person as a developer.
2. Select a particular experience to pursue.
3. Pursue that experience to completion. (Achievement unlocked!)
4. Reflect on that experience. Really soak it in.
5. Return to Step 2, this time selecting a new experience.

This was along a common vein with some things I had been doing, and generally seemed like a good fit for me, since I’m fairly new to programming, so I forked his list and modified it to suit my circumstances:

Program in a variety of different programming paradigms

• Write an application in a functional language
• Write an application in an object-oriented language
• Write an application in a prototype-based language
• Write an application in a logic programming language
• Write an application using the Actor model
• Write an application in Forth

Program for different platforms:

• Write a nontrivial web app
• Write a nontrivial desktop app
• Write a nontrivial mobile app

Enhance my understanding of the building blocks:

• Use machine learning to solve a problem
• Write a networking client (e.g., HTTP, FTP)
• Write a B-tree database
• Wrap an existing library to provide a better user experience
• Write an application or framework that provides a plugin model
• Write a testing framework
• Write a lexer & parser

Enlighten myself:

• Complete five code katas
• Complete the programming koans for a language that you want to learn
• Attend a code retreat
• Read SICP and complete all the exercises

Program in the open:

• Contribute to an open source project (Added three words of documentation. Ha!)
• Have a patch accepted at an open source project
• Earn commit rights on a significant open source project
• Publish an open source project
• Perform a Refactotum of an open source project

Learn by teaching others:

• Present a lightning talk
• Present at a local user group
• Present at a conference
• Publish a tutorial
• Publish a constructive code review of an open source project

EDIT: I have since updated this list.

test "weight is not required" do
  assert @product.valid?, "product should be valid to begin with"
  @product.weight = nil
  assert @product.valid?, "product should still be valid"
end

This might seem too trivial to test, or like we’re testing core Rails functionality, but an optional attribute is a feature of your application. In other words, it’s easy for parts of your app to depend on an attribute being optional, which means it’s easy to break something by making that attribute required down the road.

class CreateProducts < ActiveRecord::Migration
  def change
    create_table :products do |t|
      t.text :name
      t.decimal :weight
    end
  end
end

class ProductTest < ActiveRecord::TestCase
  test "name is required" do
    @product = products(:one)
    @product.name = nil
    assert !@product.valid?
  end
end

class Product < ActiveRecord::Base
  validates :name, presence: true
end

Here we have a model with two attributes: name, which is required, and weight, which is optional. We also have a test to make sure that name is required. Now we decide that the weight attribute shouldn’t accept negative numbers, so we add a test and modify the model.

class ProductTest < ActiveRecord::Base
  # more tests
  test "weight should not be negative" do
    @product.weight = -5.4
    assert !@product.valid?
  end
end

class Product < ActiveRecord::Base
  validates :name, presence: true
  validates :weight, numericality: { greater_than_or_equal_to: 0 }
end

Easy enough, right? Nope! We just make a small but insidious mistake.

First, because we didn’t add allow_nil: true to the numericality validation, weight is now required, but that’s not what we wanted to do. We just need to make sure weight isn’t a negative number; it should still be optional.

Okay, that’s not too bad; we might have caught it if it broke some of our other tests. Something else is much more diconcerting; if we’re using a fixture without the weight attribute set, the "name is required" test isn’t actually testing anything.

one:
  name: Cat 5e

class Product < ActiveRecord::Base
  # validates :name, presence: true
  validates :weight, numericality: { greater_than_or_equal_to: 0 }
end

Boom! Both of our tests still pass, but we’re clearly not validating that name is required. This should strike fear into your heart; we have a reasonable looking situation that completely fails to do what we expect. In other words, we have built our app on top of quicksand. Maybe it’s time to tweak our testing methodology.

There are two ways we could have caught this bug. First, we could have validated our fixtures, which is good hygiene but kind of a separate issue. Second, we could have tested that our optional attribute was actually optional. We’ll focus on this second one.

test "weight is not required" do
  assert @product.valid?
  @product.weight = nil
  assert @product.valid?
end

Look familiar? This test is how we got this whole blog post started. More importantly, it will start a chain of failing tests that, as we cause them to pass, lead us back to code which does what we expect it to do.

It’s kinda wordy for my taste though, and will result in egregious violation of the Don’t Repeat Yourself (DRY) principle, especially if we have a bunch of optional attributes. Why not create a helper method so we can test all of our optional attributes in one fell swoop?

class ActiveSupport::TestCase
  # some other stuff
  def self.test_optional_attribute(fixture_name, *attrs)
    attrs.each do |attr|
      test "#{attr} is optional" do
        fixture = instance_variable_get("@#{fixture_name}")
        assert fixture.valid?, "#{fixture} isn't a valid fixture"
        fixture.send "#{attr}=", nil
        assert fixture.valid?, "#{attr} should be optional"
      end
    end
  end
end

Now we have a nice, succinct way to test all of our optional attributes in a single line of code.

class ProductTest < ActiveSupport::TestCase
  # other tests
  test_optional_attribute :product, :weight
end

Maybe this all sounds like paranoia to you, but mistakes like this one are shockingly easy to make. Indeed, this is the reason we do any testing at all; we humans, with our inconsistency and tendency toward errors, and computers, with their unforgiving precision, are like oil and water; we don’t mix.

And yet, here we are anyway, trying to master computation. Rather than pretend that humans and computers aren’t fundamentally incompatible, embrace it. Go all the way. Test every single assumption. Just be smart about it; make it easy and succint. Make it joyful.

test "weight is required" do
  @prod_one = products(:one)
  @prod_one.weight = nil
  assert !@prod_one.valid?
end

Simple enough; I don’t think we’re treading new ground here. The problem is that this test assumes the fixture is valid. If the fixture isn’t valid, the test will still pass, but it won’t actually be testing anything. Yikes! Better to assert that the fixture is valid.

test "weight is required" do
  @prod_one = products(:one)
  assert @prod_one.valid?
  @prod_one.weight = nil
  assert !@prod_one.valid?
end

That’s definitely better, but we’ve got a bunch of tests to write; it’s going to be tiresome (and error-prone) to have to assert the validity of our fixtures in each test. Plus, we want to keep our tests laser focused on what they’re actually testing. Why not just get it all out of the way at once?

setup do
  @prod_one = products(:one)
end

test "fixtures are valid" do
  assert @prod_one.valid?
end

test "weight is required" do
  @prod_one.weight = nil
  assert !@prod_one.valid?
end

Look at how tight that test is. There’s nothing in there to distract you from immediately comprehending what’s being tested. Truly a thing of beauty. There is still one thing that’s bothering me though…

We’re going to be testing our fixtures this way a lot – at least once for each unit and functional test class. It’s tiresome to actually have to write an entire test each time we want to do this. Why not just write a helper method so we can turn this into a one liner?

class ActiveSupport::TestCase
  # some other stuff

  def self.test_valid_fixture(*fixts)
    fixts.each do |fixture|
      test "@#{fixture} fixture is valid" do
        fixture = instance_variable_get("@#{fixture}")
        assert fixture.valid?,
          "Fixture is invalid: #{fixture.errors.full_messages.join(', ')}"
      end
    end
  end

end

Now we can be concise and expressive in our test class, which allows us to conserve our brainpower for where it’s really needed.

class ProductTest < ActiveRecord::TestCase
  setup do
    @prod_one = products(:one)
    @prod_two = products(:two)
  end

  test_valid_fixture :prod_one, :prod_two

  test "weight is required" do
    @prod_one.weight = nil
    assert !@prod_one.valid?
  end
end

It was one of those things, a Fundamental Truth About The Universe, like that genes, not organisms, are the basic units of reproduction; or that evolution, not the id, is what explains human behaviour. One of those “Aha!” moments when the haze of confusion is swept away and suddenly everything makes sense.

And so my eyes were opened a little wider.

Wandering off into the wilderness

Anybody else, coming across a subject which they found so compelling, and which also offered lucrative career opportunities, would have made an immediate course change. But I was missing a couple of crucial skills:

1. Setting goals for myself
2. Making plans to achieve those goals
3. Executing said plans

So this beautiful, golden opportunity floated right past me while I was busy taking the path of least resistance.

Soon, though, it became apparent to me that I needed to learn how to accomplish things, and so began the multiyear process of pulling myself up by the bootstraps: learning how to set goals, make plans and execute on those plans by setting a goal, making a plan and executing on that plan.

Finding my way back

And once I was started in a different direction, the momentum of life kept me going in that direction for a good long while. All told, it took 8 years for programming to find it’s way back into my life.

It started off so innocuously. We needed a better way to manage ordering from our overseas factories. We had an Excel spreadsheet listing all of our current products.

“Yo Bossman, check this out,” says Rico. “I added our current inventory and monthly sales numbers to the product spreadsheet, then wrote a simple formula to calculate how much of each product we need to buy.”

“Sweet, that should save us a bunch of time,” replies The Bossman, “but it’s kinda confusing. Is there a way to only show a quantity for the products we do need to order and leave the others blank?”

“Yeah, I’m sure Excel can do that,” says Rico as he Googles around a bit. “Hmmm, this should work: IF(QuantityToBuy=0, "", QuantityToBuy).”

Woah! A conditional. It’s kind of a weird one – if looks like a call to a function, instead of how I remember it – but it’s a conditional nonetheless.

Building a home for myself

Making software with Excel turned out to be alternately deeply strange and downright blissful, but it marked the beginning of my return to Computation. The next big step was converting my company’s <table> & <img> based website to an HTML & CSS based design.

This is the point where things really start to go fractal. It starts with selector specificity. Then it’s the CSS box model. Then semantic markup. Soon I realize that I need something to handle templating for the website. Now it’s Wordpress and writing little snippets of PHP. Oh, and JQuery happened somewhere in there too.

The thing is, it just doesn’t stop. Soon I’m making command line utilites and Coda plugins in Perl. Somewhere along the way I start using MacVim. And Git. Then command line Vim + tmux. And regular expressions, object oriented programming, recursion and Big O notation.

Then I get started with Ruby on Rails, so it’s models, views, controllers, migrations, ActiveRecord, partials, REST, routes, HAML, SASS, YAML, unit testing, integration testing, Rake, Capistrano.

It’s insane how much stuff you need to cram into your brain to be a decent web developer, and yet web development is just one tiny facet of the whole of Computation; a facet which isn’t even that interesting.

Where things get really interesting is when you begin to realize that computation is every where, that there are other, very different forms of computation, and that some of those are way better at certain tasks than what we have now.

Thinking about all these things to learn, I can’t help but remark to myself, “There’s a whole lifetime of work in there for me.”

What a thought!