kellyfelkins.org: web technologies

replication in rails

I’m working on a project that has 3 tables that are very similar and each should have a very similar edit UI. Of course I don’t want to duplicate code or coding effort if I don’t have to. I’m always trying to explore new areas of ruby and rails and this seemed like a good opportunity.

My first thought was to copy the scaffolding generator. I think most of the generators are pretty trivial but scaffolding is significant since it does reflection on the database in order to build the new and edit forms. Copying and modifying the scaffold generator was a bigger job than I expected but I got it working. Since the scaffold generator invokes the model generator I ended up modifying model too.

A modified generater is neat because you can make a change to the generator and completely regenerate the model, controller and views instantly.

I wasn’t satisfied with this though. Basically I now had a lot of automatically generated code. In addition I worked on it longer than if I had simply used the default scaffold generator and modified the forms and controllers individually.

It seemed to me that if you can run a generator 3 times you should be able to refactor things so that you only have one copy of that code instead of 3.

I wanted to experiment with run time code generation like rails associations (and many other features of rails). So I modified the controller adding a method that would generate other methods—the methods that are normally generated when you use scaffolding.

In the end I found that my 3 models were not as close as I had originally thought. The exceptions required to get these techniques to generate all of the code created a lot of complexity that started spilling out to other parts of the system.

Bottom line: these are useful tools to have in your back pocket especially if you are building plugins or other reusable code.
However, they probably should not be used in building a single application due to the additional complexity and non-standard application structure.

Share this article on

Comment [325]

* * *

permalink

listening reimplemented in ruby on rails

Ruby on Rails is a web application framework that uses Ruby as the scripting (and implementation) language. The rails system includes apis for various pieces of the framework as well as applications and scripts that provide both target application functions and application building functions. The integrated apis provide useful web page generation services and database access with object relational mapping. There are other features that I’ve not yet explored.

To evaluate and learn Rails I reimplemented an existing small perl web application which I’ve called “Listening”. You can check out the original description of the listening project, view the perl listening page or view the rails listening page.

The Listening Application

The Listening Application is a simple web page that displays music play information. The play data comes from a Audiotron networked audio player. Early on the play data was extracted from samba logs. Now play data is captured by a daemon that polls the audiotron. In both cases the data is written to a mysql database. The application displays various summaries of music play activity, such as the most recent tracks played, the most played tracks during various time ranges and the most played compilations during various time ranges.

Perl Listening Overview

The perl implementation of the Listening application was started while I was attending a perl programming class with the explicit purpose of exploring and practicing perl. I continued to evolve the application over time in my spare time. The perl implementation is rough and immature, reflecting the application’s history and my perl skills during its development.

The structure of Perl Listening is fairly simple. There is a perl data access api organized in a perl library file. The primary UI is a perl cgi script that pulls data via the data access api and formats it for display.

Reimplementation Goals The goals of the reimplementation project are:

explore features of rails

compare the rails implementation to the perl implementation

Number 2 is a bit of a challenge. There are many aspects of the perl implementation that I am not happy with and that I do not want to reimplement in the rails version. However, if I don’t reimplement them, I will not be able to do a fair comparison. Furthermore, I want to compare the displays generated by both systems as a means of verification, so I want them both to use the same data.

Also, during this project I had a hard disk failure. While I was able to recover most of the information I did lose the latest version of the audiotron monitoring daemon. This daemon had gone through significant development recently so recreating it would be a relatively big job. Furthermore, I’m planning to create a rails implementation of this component. Therefore, I’ve decided to not recreate the perl auditron monitoring daemon.

My approach is to migrate both applications to a common set of features that work off a common database. This means that the rails project will implement new the target features of the final application. It also means that the perl implementation will be modified to the target feature set. These changes to the perl implementation fall in to 2 categories:

changes to support a modified schema, and
dropping of features that are not expected to be in the rails implementation.

Item one because rails imposes some database structure, and to allow both applications to run off the same database, I’ve evolved the database to support rails, then evolved the perl implementation to use the new database schema. This was facilitated by the perl listening data access api, allowing me to make focused changes in a few functions to accomodate the schema changes.

Item 2 because there are features that are deemed no longer relevant, or that were never fully implemented.

Rails Lessons

do it my way and I’ll make it easy for you

One of the advantages of rails is its ‘‘full stack integration’‘. To achieve this, Rails maps names between scripts, classes and database objects and expects a certain organization to the schema (basically, a normalized schema) and table and column name standards. There are 2 sides to this: first, to take full advantage of rails you need to adopt its naming scheme and to organize your data as rails expects. Initially this is a bit of a hassle and feels like a waste of time. In return, rails makes certain operations very easy. For example, in the Listening application, when you have a track object, it is trivial to get the corresponding cd object — you simply use object notation to reference it: my_track.cd This covers the loading of the specific object from the database, instantiation of the object and population of its fields. Very cool!

automation means lack of control

It seems like there are advantages and disadvantages to object/relational mapping systems. It is certainly nice to interact with database information using the object model as I mentioned above. The potential downside is the lack of control of the the database access and the potential for inefficient access. Specifically in this project I see many database queries when direct access to the data could have combined these into a single query. Rails has a system called ‘eager associations’ for dealing with this. Unfortunately, there are some limitations to eager associations that restrict its use. Another problem is data access that does not match or is not implemented by the OR system. Specifically in this project I needed to do data aggregation functions. I needed to sum the time for all plays. Fortunately, while rails did not provide (or I did not find) a feature to do this, it neither got in the way of doing it. I was able to do queries that returned values not represented by the object model. However, the combo was a little confusing when I wanted to aggregate data and use the object model. For example, I needed to know the most played tracks and then display information about those tracks. The second part would best be served by a collection of track objects. However, the track class has no notion of a play time, much less an aggregated play time. I was able to do this but it required constructing a query. Fortunately the framework did not complain when the query returned columns that were not part of the original table.

automation means lack of control, again

The rails processing model starts with a controller receiving a request, talking to the model to gather appropriate data, then invoking a view to format the data. The Listening application actually displays a fair amount of data. That data takes a while to assemble — at least I think that’s the issue. Bottom line, the rails version of the application takes longer to display the listening page.

I suspect there are things I can do, such as enabling fastcgi as is recommended on the rails site. However, the perl cgi version is not using fastcgi.

One advantage the perl cgi version has is that it does not do all of its data gathering before it starts displaying stuff. It gathers the data for the first section and then displays it, then it grabs the data for the next section and displays it. This process continues. You can watch the scroll bar shrink over time. This is a good thing. Its almost a slight of hand. Your perception as a user is that the page is done. In reality, while you are reading the first page of information, additional information is loading “below the fold”.

I created a web templating system in java before the first jsp standard was released. This system had 2 modes. In the first mode, you instantiated a template object, did all of the substitutions, then sent the string version of the template to the browser — basically the same way that rails operates.

One of the first applications implemented using this system was a query interface to a large (approaching 1 terabyte) oracle database. I was very concerned about perceived performance. For that reason, the system implemented a second mode. In this mode, the model was turned inside out. Instead of instantiating a template and telling the template to fill in the various values, this time we would instantiate the template and tell it to “go”. It would start streaming the page. When it encountered a variable that required substitution it would do a call back. In addition, the system was designed with events — the objects that did the substitutions could be notified ahead of time. They were touched at the beginning, saying “get ready…go get your data…I’ll be back when I need it”. I had a separate api I called “simple server”. If these data providing objects extended simple server, the initial touch kicked off a separate thread. Later, the request to load the data would check to see if the data load was complete. If so, it returned with the data. If not, it waited.

It worked. The pages seemed very responsive even though there were no real performance enhancements implemented. From the users perspective the page started displaying immediately. Rails may need something like this.

some things in life are free

Rails has a thing they call “scaffolding”. Scaffolding generates a CRUD (Create, Read, Update, Delete) web interface for your tables. Each of these was a separate page, a separate development effort in perl. Now scaffolds are not fancy. They will not be a part of the exposed interface of a professional application. But non-professional applications can get access to the data quickly and easily. And professional applications may find a use for these behind a firewall.

some things in life are very inexpensive

I implemented the add-on authentication system called Login Generator by Tobias Lütke. After some initial simple configuration, requiring a page to be authenticated is this simple:

<pre>class ArtistController < ApplicationController scaffold :artist before_filter :login_required </pre>

You see that “before_filter: login_required” line? That’s it. If you are not logged in and you try to touch this page you get redirected to a login page.

Metrics

measurement	rails listening	perl listening
files	46	2
code lines	479	651

There are a lot of rails files and few perl files. These rails files are created somewhat automatically by rails — many of them are initially empty and some are still empty. This is a big complaint I have for such systems. I consider this a barrier to entry. I spent a lot of time trying to figure out not just how to do something, but where.

There is a bit more perl code. I thought the code difference would be greater because my perception was that the perl program had become really bloated. It had. To be fair with these metrics I removed perl code that was not represented in the ruby code.

While there are fewer ruby code lines, the ruby system has a bit more functionality. It has this authentication system, and the maintenance screens for each of the table types. The later is close to free though, especially concerning these metrics since it is implemented in the rails framework. The former is pretty cheap too. This is one of the key benefits of a system like rails — I can add key features with little coding of my own.

Overall, while I wrote both systems at similar skill levels with the specific tools, I feel the rails version is much more organized. It’s cleaner and stronger. I think there will be fewer bugs.

Lines counted with
<pre> grep -rv "\$ˆ *#\$\\|\$ˆ *$\$" * | wc -l </pre>

Share this article on

Comment [326]

* * *

permalink

You know you're a geek

You know you’re a geek

This issue is way too familiar. Since I’ve been playing with logging the music I play, I’ve had the thought “does it make sense to turn on my computer, so that I can bring up my web site, to list the most popular songs in my collection, so that I can click the link which launches the song on the audiotron…or should I just turn on a radio?”

Share this article on

Comment [1]

* * *

permalink

shop talk: frames humor

I have done some terrible things in my life but I have never made a site with frames.

from http://photomatt.net/about/

Share this article on

Comment

* * *

permalink

the listening project

What’s Kelly listening to?

The Listening Project

The listening project is an evolving collection of tools that support my own enjoyment of music. The project has also been an avenue for me to experiment with new-to-me software techniques. I started the project during a perl programming class, as a means to learn and practice perl programming.

History

May 16 2004

View saved page.

Currently Playing On Audiotron: Added a “currently playing on audiotron” section to the top of the page. The first try used the audiotron api to poll the audiotron. This turned out to be a bit slow, so I made it a separate page and include it on this page as an iframe. This way, the main page loads, and the iframe can load at its leisure. Bonus: The iframe even refreshes! I’ve now added this to the right border of all my pages.

Updating Stats Every Minute: I’m scanning the logs every minute for updates. This was done in anticipation of adding a list of most recently played. Don’t know at this time if this will create unacceptable work for the system, but so far it seems ok. The log scanning routines really try hard to scan the minimal amount, ie. what’s new in the logs. There is more discussion of this here.

Circa May 2004

View saved page.
This was a major upgrade to the system. It was almost completely re designed and reimplemented. Some of the changes:

Store Play Information in a Database: previously, play information was written to a text file. A database was thought to provide much needed flexibility in generating the stats for the system, as well as efficiently storin
g stats. Both of these have shown to be true.
Store Open Times, Not Just Open Counts: The thought was that file open counts were a poor indicator of actual play time for a couple of reasons. One, some of the tunes I enjoy the most are long—sometimes the whole side of an album. File opens would not indicate how much I’m listening to those tunes, compared to others. There are a lot of opens that are not listening events, such as when the audiotron builds its index of music. Now that the system looks at open time, the brief opens that are associated with non-listening events can be discarded.
Amazon Associates Links: I’m curious—would anyone view this page and decide to order one of these CDs? So far (mid May, 2004), the answer is no :-). It’s possible that no one has visited the page, other than myself.
New Page Designs: I continue to experiment with the display of the play information. I want the page to convey information and be esthetically pleasing.

To Do

pri	what	discussion/rationale	status
x	web stats	I need to know if anyone is viewing these pages.
2	rate tunes/cds	Play time is a passive way of rating music, but I would like to be more active, especially for individual tunes. I usually play whole albums. If a tune plays that I know I don’t like, and I’m sitting at a pc somewhere or near the audiotron, I’ll skip it. The fact is that on most albums I only like a few of the tunes. I’d like to be able to randomly play any tune that I like, for example.
x	cd publish date	I’m curious. I think the stuff I listen to is a decade old or more… :-}
x	edit mp3 tags	Often, music is mis-tagged. Sometimes the artist is listed in different ways for different CDs, like “The Dave Mathews Band” and “Dave Mathews Band”. Many of my compilation CDs are coded inconsistently.
x	audiotron start cmd	I need to start the audiotron if it is not playing. The queue links do nothing if the audiotron is not already playing.
x	log atron playing in db	I need a list of the last few items played, and I don’t want to wait for them to be added to the db via the samba log scan (though I realize now I haven’t thought this through). Also, I want a log of internet radio activity. This is a big part of what I listen to, and I want to generate stats on it.	partial
x	annotate tunes	The list of recently played is now displayed. Now we need the ability to annotate these, as in rating, or add to playlist, etc.
x	audiotron controls	I’m sorely in need of some basic audiotron controls on this page. I’m playing some fairly iffy music, and it would really be nice to be able to easily say “next track please”, or, in other words, “please stop torturing me”.
x	schema enhancements	need to make a separate artist/cd/tune table, and take that stuff out of the other tables.
x	project history	I want to capture the history of the project for my own grounding and to share with others. I’m interested in the evolution of software systems and so I want to track the evolution of my own systems. Also I want to share that with others.	done
1	recent play list	To implement the above, the UI needs a list of what is currently and recently played. Also other thoughts on this.	done 5/30/04

Share this article on

Comment [1]

* * *

Older

kellyfelkins.org

replication in rails

listening reimplemented in ruby on rails

The Listening Application

Perl Listening Overview

Reimplementation Goals The goals of the reimplementation project are: explore features of rails compare the rails implementation to the perl implementation

Rails Lessons

do it my way and I’ll make it easy for you

automation means lack of control

automation means lack of control, again

some things in life are free

some things in life are very inexpensive

Metrics

You know you're a geek

shop talk: frames humor

the listening project

The Listening Project

History

May 16 2004

Circa May 2004

To Do

Reimplementation Goals The goals of the reimplementation project are:

explore features of rails

compare the rails implementation to the perl implementation