|« October 2005||December 2005 »|
I'm Ryan Lowe, a Software Engineering graduate living in Ottawa, Canada. I like agile software development and Ruby on Rails.
I write this blog in Canadian English and don't use a spell checker. Typos happen.
» Full-time Ruby on Rails freelancer
» Full-time with Rails since May 2005
» Former committer for RadRails (now Aptana)
» I also have a few Rails side-projects in development:
1. wheretogoinTO.com Toronto nightlife
2. Hey Heads Up! TODO list and sharing
3. Layered Genealogy family history research
4. foos for foosball scoring
5. fanconcert for music fans (on hold)
Hiring Rails developers? I can telecommute by the hour from Ottawa, Canada
»» Email: rails AT ryanlowe DOT ca
Now hosted on Hey! Heads Up -- check it out!
Derek Lowe's (Ryan's older brother) words at Ryan's funeral
[email protected] no more
Forging Email Headers: Good, Bad or Ugly?
Sarcastic Dictionary (Part 1 of Many)
Twisting Rails is Risky Business
Risky Business? My Take on Early Alphas
Whoa, it's August 2007
A Postscript to "Growth at the grassroots"
»» All Blog Posts
David Heinemeier Hansson
James Duncan Davidson
Signal vs. Noise
Amy Hoy: (24)slash7
Luis de la Rosa
# FanConcert's Scoring Improved
In past blog posts I've outlined problems with wikis as "collaborative" sources of information. To combat those problems FanConcert is going to have ways to identify the quality of each piece of information that was submitted by users.
A lot of Internet sites that grade the quality of information call this process moderation. It's not a very user-friendly name for it, especially when you consider the many definitions of the word. So instead I've decided to refer to moderation on FanConcert as scoring.
Before I get into scoring though, I need to explain the concept of context. When individual bits of information about an object can change, people need a point of reference that never (or very infrequently) moves. For example, on FanConcert a venue has a name that doesn't change because the name is the context. From that name, you can provide values for attributes like the street address, official website, etc. The name can't change because that would change the context that all of the other attributes are based upon.
Imagine if FanConcert didn't enforce this. You could add a venue and fill in all of its attributes, like it's address and website. Then someone else could change the venue name to something that makes the address and website invalid. It doesn't make much sense to allow that.
Wikipedia enforces context with its article titles (and URLs). If you're editing the Wikipedia page for Corel Centre -- which has the distinct Wikipedia URL
Wikipedia can also handle many things that have the same name, with its disambiguation pages. Each of those things then gets a separate unique name on Wikipedia to differentiate it from other things with the same English name.
On FanConcert you can change all attributes except the context attribute by editing, a voting process where majority rules. For example, if the majority of people say that the official website of Madison Square Garden is
Unlike a wiki that only allows corrections, people can submit the same value for an attribute show that they agree. When a lot of people agree it's harder to outvote a large majority and vandalize FanConcert with bad information. The ability to agree is a very important aspect of FanConcert that wikis lack. If enough people submit edits then FanConcert's data can have higher reliability and correctness at any given time, unlike a wiki like Wikipedia that can be instantly vandalized.
There is one twist though: the original submitter of the venue (the creator) can change the venue's name. I allow this so that users can fix typos, not change the venue's context. This means that the user that submitted the venue is responsible for the venue's context as well as the venue name's spelling and capitalization. Other users can't change a venue name so they can't change the venue's context.
BUT other users can agree or disagree with a venue's name by scoring. When a venue has a high score a lot of people are saying: this venue exists in real life and has this official name. Users can disagree with the venue name and provide reasons for their disagreement, such as: the venue name is spelled incorrectly or doesn't even exist in real life.
All of the other attributes of the venue -- like the address -- are not related to the venue's score since they can be changed individually. I recognize that this is a little counter-intuitive at first. I'm not sure how much of a problem this will be -- it's one of those wait and see things.
Scoring can be done on each object individually but some scoring can also be done on lists of many objects. In the last week I've changed the search results for artists, venues and concerts so that you can agree with many objects at the same time. This makes it much easier for correct objects to accumulate high scores, which is really important.
I also allow users to quickly remove their scores from lists of objects just in case they make scoring errors.
Why not allow users to disagree with many objects at once? Each object has a separate reason for being incorrect and I don't want to trivialize negative scoring. This is not as much of a problem as it may seem: objects that are correct will accumulate high scores easily while objects that people aren't sure about will have low or negative scores. If people are too lazy to dig into the object and disagree with it, the object will still have a low score.
 People could confuse "scoring" with a rating system: how good an artist is, rather than its correctness. I have to keep this in mind.
# Fall From Grace, The Blog Post
What does a songwriter do when they're all tapped out of lyrics? Go with a Sure Thing: something that always works.
One of those Sure Things probably started out as an innocent metaphor, snatched from a poet and put into a rock song. Now it's completely out of control. Yes, I'm talking about the phrase "fall from grace".
I hope to find out where this phrase originally came from and what it really means. I'd also like to find out how we got to where we are today: hearing this phrase in every second rock/pop song on the radio.
What is this 'grace' that people 'fall' from? It seems to refer to a sacred position with or love from God (definition 8) that people could lose by doing something that's explained in the rest of the song. But 'grace' seems to have a lot of meanings and a wide range. Are followers of God protected by grace, or is everyone and how much? That determines who can fall in the first place and 'how far', right? How does one lose the grace of God?
Even though I'm not a very religious person, that's probably why I loathe hearing it used in songs so much: it's overdramatic bordering on hyperbole. It doesn't add value to a song it just makes it more ridiculous, diluting the song's intended message. Pop songs still have messages and themes, don't they?
It's not ridiculous because of the religious connotations, far from it. In fact it's quite handy for songwriters that they are only inferring God, whereas mentioning Him specifically may seem too preachy for popular music. I don't mind the inference. It's ridiculous because songwriters often use a 'fall from grace' as a religious consequence to overdramatize a seemingly trivial situation described in the song.
I'd be interested in hearing what religious people think of this. Maybe they see popular music as equally trivial, though it's hard to argue that it's not impressionable. Maybe they are just words in a pop song that not many people pay attention to. Maybe it's just artistic license.
There's something else about the phrase "fall from grace" that makes a lot of sense for songwriters. Not just the meaning of the phrase but how well it rhymes with other words. The word 'grace' rhymes with a lot of words that end in either -ase or -ace. Couple the rhyming capabilities of the phrase with the fact that a person could apparently "fall from grace" for doing just about anything and you've got a real winner! No wonder the phrase is in so many songs.
Here's a list of some of those songs ... some of them have been pretty popular.
The "Fall From Grace" List
Here's the format, sorted by reverse date:
Our Lady Peace - Angels/Losing/Sleep
The Pogues - If I Should Fall from Grace with God
Thornley - Beautiful
The Get Up Kids - Fall From Grace
Starsailor - Tie Up My Hands
Coldplay - God Put a Smile Upon Your Face
Sara Evans - Saints and Angels
Phish - Heavy Things
Live - Run to the Water
2Pac - If My Homie Calls
Korn - Freak on a Leash
Puff Daddy - Come With Me
The Tea Party - Gyroscope
Fall From Grace (the band)
Amanda Marshall - Fall From Grace
Red Hot Chili Peppers - Falling into Grace
Stevie Nicks - Fall From Grace
Def Leppard - Blood Runs Cold
Michael Jackson - Stranger in Moscow
Bon Jovi - Keep the Faith
Dire Straights - Lady Writer
Madonna - The Look of Love
Asia - Heat of the Moment
The Moody Blues - The One
If you know of any other examples, feel free to use my comments and I'll add them to the list. It'll keep you busy on those boring days at work. ;)
# U2 and The Arcade Fire at the Corel Centre in Ottawa
I was lucky enough to be invited by Roy to see U2 and The Arcade Fire at the Corel Centre last night. Thanks again, Roy! It was a really great concert and I was impressed with the audio where we were: in the very top row (S) of section 304, left of center stage.
It's great that U2 invited The Arcade Fire to play with them in Ottawa and their two Montreal dates. The Arcade Fire is one of those great Canadian indie rock bands that's not getting enough exposure from our local radio stations. Though I have seen their videos on MuchMusic.
A lot of the crowd at the U2 concert looked over 30 and I'd hazard a guess many of them don't follow the indie rock scene so it's great exposure for the band. The Arcade Fire is not on tour right now but they mentioned they are recording a new album.
I was particularly impressed with band member Régine Chassagne, who played accordion, drums, keyboards and xylophone in as many songs and also sang. The band has an amazing stage presence -- I can't wait to see them at a smaller venue, like Capital Music Hall or Le Metropolis in Montréal.
The U2 show was quite long -- over two hours -- and the crowd was really into it. On a few of the songs Bono let the crowd sing the choruses, which was pretty cool.
The stage was an oblong circle with the VIP crowd in the middle of it. Unlike other shows I've seen at the Corel Centre, there was no "backstage" area and people were sitting behind the stage, looking at U2's backs. There was a "video screen" made up of dangling columns of lights that displayed pictures throughout the concert. It actually froze early on in the concert and had to be retracted. Like I mentioned to Roy: at least it didn't blue screen! Ha. The circular stage was lined with lights that moved around it in different colours on upbeat songs. We had an excellent vantage point for the lighting effects where we were. :)
U2 played a lot of new songs at the start and end of the concert. In the middle they played old favourites like Sunday Bloody Sunday and talked politics for bit. Overall I liked the positive message and they didn't overdo it. U2 was just amazing on stage. There's a sense of confidence about bands that have been together that long -- I felt the same thing about the Red Hot Chili Peppers when I saw them a few years ago -- and it adds an extra dimension to the whole performance.
Overall a great show! I'm not sure I would have paid the $100 CAN+ that some people paid, even for 300 level seats. But for $60 it was well worth it. U2 was one of those bands I just had to see live before they called it quits.
# The Agile White Box Tester
Jim just wrote an interesting post about black box testers. In it he says: "Ultimately it's a developer's responsibility to ensure that things are working correctly..."
I like this line of thinking but unfortunately in my experience (with both large and small software companies) it's quite new and uncommon. The best thing we can do while we're extolling the benefits of that approach is remember that most teams don't believe or understand it.
Why? Many managers think that having developers test is a waste of development resources which will slow down the project. Many developers think testing is boring. Jim and I may see those justifications as short-sighted -- not in the overall long-term best interests of the project. Until managers and developers change their minds, the old way of thinking -- "let QA do it" -- will reign supreme.
Just like there's no reason why developers can't test, there's no reason why testers can't have the source code and get the white box perspective. I read an excellent book called Testing Extreme Programming that had a major theme: some people are really good at testing, and agile processes like XP do have a place for them. Not everyone on an agile team has to be a developer that tests.
Testers in an agile process can ensure that developers are testing their code properly (broken tests won't help find broken code) and help developers make tests faster and with better quality. Testers can really teach developers a thing or two about testing (testing mentor) for the benefit of the entire project. Testers can also ensure that the tests are complete: that no functional or unit tests are missing -- because missing tests will not find missing or broken behaviour in the software.
Developers may be tempted to take shortcuts when testing in the name of short-term efficiency and the testing team can ensure that developers are being honest. It's a little bit of policing, sure ... but everyone on the team needs to understand it's for the overall health of the project and to help people learn to code and test better, not to embarass people. A good team attitude about this type of feedback can really help.
If developers have a defensive attitude about the quality of their code -- as they sometimes do on teams with seperate development and QA camps -- the project will have a detrimental internal battle going on, and no one wins.
It's the responsibility of the project manager(s) to set the tone of the project so these teams are learning from each other instead of at each other's throats as adversaries, trying to make each other look stupid.
# Non-Fattening Syntactic Sugar: Ruby's unless
I have to tell you, while working on FanConcert with Ruby on Rails I've found Ruby to be a complete pleasure to work with. Only after using it for a few months do I understand all of the advantages of languages like Ruby that James Robertson has been blogging about. He uses Smalltalk but it seems very similar to Ruby (maybe I can find a comparison to link to). Honestly the best way is to get knee-deep into it, then you'll get it too.
First of all, when you're using a dynamically typed language I recommend you unit test. There's no compile-time type checking but your tests can do something even better: make sure all of the objects are behaving the way you expect them to. Then type-checking becomes unnecessary.
Ruby calls this "duck typing": if the object quacks like a duck, it must be one. If the object responds to a method call, it's quacking. Unit testing ensures your objects quack how you intend. The approach makes it easier to refactor Ruby code than Java, even when Java gets help from an IDE like Eclipse (refactoring Java code can be a search-and-replace nightmare without help from an IDE). You don't need to change declared types or method signatures with Ruby code, you just make the object quack differently and it will.
But let's get to something I really like about Ruby: the keyword
The thing I like the most about
You can't call a method on a
Another example is compound boolean statements, which are easy to mess up when you're trying to negate them. Remember:
That's just pseudocode and it's giving me a headache. All software and hardware people know that rule -- originally by De Morgan -- but it's easy to mess up in an if statement.
I like the
# Check out José González
When I listen to music, I mostly listen to whole albums front-to-back (except songs I really don't like). I have the luxury of being able to do that because I'm at a desk all day.
Sometimes I have to listen to a CD a few times before I like it. But having the time to listen to lots of whole CDs gives me a chance to give bands a few listens before saying I don't like them. Some bands that didn't make an immediate impact on me but I ended up really liking were The Beta Band, Ben Harper, Dashboard Confessional, Death from Above 1979, Four Tet, Modest Mouse and Wilco. If it's a sound I'm unfamiliar with sometimes it takes me a bit longer to like it.
Sometimes music hits me like a ton of bricks and I know right away that I like the sound. Artists like Muse, The Futureheads, Kasabian, Nick Drake, Kings of Leon, Metric, The Organ, Death Cab for Cutie, Bloc Party, Billy Talent, The Shins, The Stills, Interpol, Teitur, Yellowcard, Underworld, The Used and finally José González.
José González was born in Sweden to Argentinean parents. His music is what I'd call folk rock but unlike his contemporaries it's often more up-tempo. The South American influences are obvious on some songs, like "Remain" on his album Veneer but I don't think you'll have to like that sound to like José. You'll probably like him if you like other folk/rock artists like Iron & Wine, Teitur or Nick Drake. The finger-plucking, dramatic volume changes and the (acoustic) lack of backing drums particularly remind me of Nick Drake.
# FanConcert is Tagging with AJAX
While I was doing that I was getting feedback about how artists were tagged. People were checking on boxes but not clicking on the link to tag the selected artists, so no artists were being tagged! Definitely a usability issue.
To fix the problem I've used some AJAX magic to tag artists in the background and made tagging more like a form submission, which makes it more obvious that you need to click a button to tag the artists.
I've used the return value from the background asynchronous AJAX form submission to update the tags on the page the user is looking at. It looks amazing in action. You can try it out by logging into FanConcert and doing an artist search.
Why can't artists just be tagged when you check the box? Because the checkboxes will be used for other things in the future, like FanConcert's so-called 'bulk editing' -- editing the same attribute for more than one artist at once -- and also 'bulk moderating/scoring'.
# Lists as Searches
Agile development has a lot of advantages, the biggest being the ability to respond to a change in 'requirements'. I put requirements in quotes because agile development doesn't really have some 100 page specification sitting around -- the requirements can change as the product is developed and people give feedback.
This used to be a custom list that looked something like:
Now it's a concert search instead. Users can see the search form with the parameters filled in and then change them if they want to. Searching gives every list a common look and usability.
The URL has all of the search parameters in it, so it can be easily manipulated. Not only on the URL bar in the browser but also by third party tools that hook into web searching, like Roy's FireFox Search Box extension -- I'll get him to update it. ;) Sure, the URL isn't as readable as it could be but I think it's more useful. The URLs wouldn't have been very readable for non-English speakers anyway.
When I get all of the lists converted to searches, I'd also like to allow users to save their custom searches so they can easily go back to them. A list of saved searches will be on the user's summary page, which appears after the user logs in.
I don't know how it started but lately I've been really into history, especially the history of Ottawa and Canada, though the Simon Schama's History of Britain on the BBC was very interesting.
After seeing all of that history on TV and reading about history on sites like Wikipedia, I find the most amazing thing is how far we've come as a (worldwide) civilization over the last few hundred years. There's a long way to go before we are any kind of Star Trek Utopia, but seeing the kind of progress we've had gives me a lot of hope for the future of mankind. Even the relatively small glancing view of history that I've had over the past year can have that impact.
Children of the generations X and Y live in a new reality: where so many things are taken for granted -- like freedom -- it's hard to imagine them not being there. We've never truly had our freedom threatened in our lifetimes. The upside is that the new reality allows us to concentrate on solving new problems to get to that Star Trek Utopia, like helping other countries instead of our own. The downside is that it's easy to forget to appreciate our past.
On this Remembrance Day, I try to think of a world where we lost World War I or II. I think of a world without democracies or the freedom to say what we think. I think of a world where ethnic or religious cleansing was seen as beneficial and heathly to a society. I think of a world without the Internet and all of the freedoms it affords. I think of a world where North America was defeated and we are under the control of some external superpower. I think of a world where our grandfathers didn't make it back from the front lines because they fought to the bitter end for our freedom -- and lost. I think of a world destroyed by nuclear war. Where would we be? Imagine how much different your life would be. It would probably be many times worse than you can imagine. You may not even have been born.
I'd like to thank everyone that's had a role in giving me the freedom I have today. I will do my best to become less ignorant of the significance of their sacrifices by paying attention to history. I can never truly appeciate what it must have felt like to be in their positions -- but I can try to, and that's the least I can do.
Today I'm proud to be a world citizen, not just Canadian. There are problems in this world, yes, but I think things have turned out alright so far under the circumstances. The world corrected itself and restored balance. It's in a relatively good place. I try to imagine where we could be and it's frightening to think how close we were to losing everything.
# Stop, Think, Turn
The luxury of working by one's self is that plans can change very rapidly with little communication overhead. ;) However, here are a few things I'd like do with FanConcert this week and next. Notice they all have a high impact and priority in one way or another, which is how agile processes are meant to attack. I've talked about specifics of these issues in blog posts over the past two weeks.
I'd like to reorganize FanConcert away from being object-centric, especially with respect to its Rails controllers. The object-centric approach works fine for many Rails apps, but for FanConcert objects have such similar operations that it's violating DRY, making it difficult to share code and tests. Another disadvantage is that the URLs aren't as readable as they could be. FanConcert is well unit tested but during this re-org there could be some breakage -- please keep your eyes peeled for weirdness.
I'd like to get the payment system in, not because I need money (I do) but because it's an area I think will need lots of work and flexibility. Initially I'd like it to support 6 and 12 month subscriptions as well as donations (if someone feels like being more generous than the subscription cost while FanConcert grows). Gift certificates will follow shortly after that, which you'll be able to buy for other people without having to sign up yourself.
I also want to get an initial run at translation in, not only to see how it impacts URLs but also the user interface and usability. A user should be able to see a translation mistake and fix it (get their vote in) easily. Collaborative translation will be a big competitive advantage and a distinguishing feature of FanConcert. The sooner I get it in, the sooner people can start translating.
The output templates I talked about earlier may be a lot of work, so they won't be going in immediately. However, I should think of their probable impact on URLs and other aspects when reorganizing FanConcert.
Editing and moderation will go through some minor changes during the re-org but stay mostly the same. Arbitrary attributes would be nice, but I'm thinking they'll be pushed back. Weighted moderation is coming but it's just not a top priority right now, when the number of users is so low.
Update 11:57 AM - I forgot something big: I should start using Rails caching now. Caching is dependent on so many other factors that it's a serious integration issue. Sure, at the root it's about performance but it forces a type of separation and grouping of pages/URLs I should be following anyway. I realised (in time) that it's not the type of thing that can be easily bolted on later if things aren't organized properly. For those reasons I don't consider it premature optimization, I consider it prudent planning of an integration weak spot.
# FanConcert is (Mostly) a Secondary Source
I'd like to explain my thoughts on FanConcert's information quality, and how that affects other things. FanConcert falls under a group of websites sometimes known as collaborative, meaning that all of the content comes from the users themselves. This is in constrast to news sites that have writers and some sort of editorial process.
I like to compare FanConcert to Wikipedia, since in effect I'm trying to build a better wiki. If you're familiar with wikis and their weaknesses, then you can expect that sometimes the information on them is just wrong for a variety of possible reasons. That's why they make poor primary resources, like for academic research.
I would put FanConcert on much the same level. People are contributing new information and moderating other people's information. Just because a piece of information is modded highly that doesn't mean it's true, it just means that a lot of other people have agreed with it.
The advantage of using a moderation scoring system over a wiki, is that on a wiki there's no way to tell how many people have agreed with a piece of information. If you disagree with a wiki, you correct it. If you agree, you'll probably just leave it alone. There's no permanent record of the fact that you've agreed. On FanConcert you'll be able to say "yes, I agree that's a true piece of information" or say "no, I think that information is false" or just do nothing at all.
All of that moderation doesn't make FanConcert infallible. The hardest part will be conveying to users that while the information on FanConcert isn't 100% reliable and never can be, it is pretty good most of the time. I'd rather make this clear than have them find out for themselves because they believed bad information. Make a prominent disclaimer on every page that users should verify things like concert dates on primary sources like artist, venue or ticket seller official web sites.
So we'll emphasize the fact that FanConcert is a secondary source of information, not a primary source. Wikipedia has a place for links to primary sources but doesn't explain/disclaim its status as a secondary source very well, in my opinion. Maybe it's because they don't want to admit it? That's a real shame because it seems that people that think Wikipedia is a primary source (like the mainstream media, who criticize it on that basis) hurt Wikipedia's credibility as a fabulous secondary source. Critics are looking at Wikipedia, seeing it's infallibility and dismissing it outright because they don't understand it.
Once FanConcert users realize that the information on FanConcert is incorrect sometimes, they will be more likely to contribute. That may seem a little weird but it's just human nature. Would you contribute information to a site if you thought it would just get rejected because it's not perfect? Probably not, you wouldn't want to waste your time double or triple checking it.
I'd like to allow casual submissions to the extent that it's easy and quick to submit new things. That includes having a minimum number of required fields of forms -- or better: really small forms for new submissions. That doesn't mean that the submitted information will be moderated up but it means that it's easy for anyone to submit information to FanConcert in the first place, just like it's easy to edit a wiki. The easier it is to enter information, the more contributors there will be, which as a bonus makes FanConcert more timely with new information.
So there's a whole psychological domino effect going on there and we have wikis to thank for it. While I'm busy trying to improve and supplant the wiki as a collaborative tool (for some types of sites) I'm picking out its weaknesses but I also shouldn't forget why it succeeds.
 FanConcert could be a primary source of information sometimes if Artists, Venues or Record Labels submit information themselves as an "official source". That would be interesting! Obviously information from one of these sources would have a higher initial moderation score, but people are still infallible, so it could be corrected by other regular users. I talked a bit about "expert" contributors in my criticism of wikis called Unequal Voices.
# FanConcert Login and Authentication
At the time I wasn't comfortable enough with Ruby or Rails to attempt that sort of thing, yet it's one of the first pieces of code a personalized website like FanConcert needs to function. I'm grateful for this code but I think it's time to move on from it.
Another side benefit of using that pre-made login code is that usability weakness are obvious from the feedback I've been getting. I'm going to attack these points in this blog post, as well as go over the general login and authentication situation on FanConcert. From now on I'll refer to login and authentication as just "login".
When looking at authentication and calculating risk, it's good to examine worst case scenarios. In the event of a full security breach on FanConcert, how severe will the break-in be? What kind of information will be revealed?
FanConcert will not be storing credit card numbers for recurring subscription payments, for example. A third-party service like PayPal, which has a pretty good security reputation, could handle that. Most of the information on FanConcert will be personal, like the users first and last name (which are optional) and the user's global location.
I don't see any reason why FanConcert would need to store full addresses (right now it's only as specific as city) but users could enter extremely accurate GPS latitude and longitude values to describe their locations. In some situations this could easily reveal the user's exact address (a house) but others not quite so easily (an apartment).
Even taking that into consideration, I see the consequences of full disclosure of a user's information to be relatively moderate. There could also be other sensitive personal information in users' profiles or preferences in the future that could change this.
The choice of security measures is based on not only the consequences of disclosure but the threat of an attack. Together they give an idea of the possible risk that the site may be attacked.
The threat of an attack can be based on the attractiveness of the target. People could target a website for financial gain or just to disrupt a popular site. FanConcert is neither popular nor can be exploited for financial gain, other than maybe free access to the site.
When and if FanConcert operates as a web service, higher security may be needed to ensure that users cannot use the web service more frequently than they have paid for, or that the site cannot be attacked and the exploited for free data.
Passwords are not stored on FanConcert, only a hash of the password. When you enter your password to log in, that password is hashed as well and compared against the stored hashed password to determine if they match. Not even I can see people's passwords when I browse the database, only the hashed passwords.
Yes, these hashes can be broken by methods such as brute force calculation. The amount of calculation required to break it depends on the number of bytes in the hashed values. When the benefit of a FanConcert break-in is so low, why would anyone go to the trouble? Still, it's an aspect of the overall risk I need to keep in mind.
FanConcert does not use a secure connection for passwords, like SSL, so passwords are sent "in the clear" on the back of an HTTP POST. For that reason it would obviously be vulnerable to people snooping on the connection. Some investigation into SSL and https:// would be prudent.
Given that the overall risk seems relatviely low, a hashed-password/session/cookie-based approach like the one I'm presently using seems to be enough security for now. The main problems concern the usability of the login portion of FanConcert.
Geez, where to start with the problems of FanConcert's current login system? I found the Salted Hash Login Generator produced rather ugly pages, rife with spelling mistakes and usability problems. The good news is that the code appears to be technically sound and well unit-tested. It's a good starting point for some iterative improvements.
I need to remove the dependency on the translation library completely. Given that FanConcert will use a custom approach to translation, the dependency on an unnecessary library (and others it in turn depends on) only serves to complicate the login code, especially in the views where every translatable title or phrase is a method call to a helper function off in another file.
I need to fix the confirmation email problems. First of all, it's not obvious that users must confirm their account before they can log in. I've added an FAQ entry to FanConcert's sign in page but I can do a better job conveying that.
The confirmation email is important so that malicious users can't 1) open many accounts at once and 2) can't open more than one account with the same email address. It's rudamentary but it prevents some basic problems. More can be done here to prevent mass signups, including the use of captcha tests, which are becoming more common on the web.
A big problem with the confirmation email is that the URL is long and sometimes breaks over two lines depending on the email client. This is an absolutely unnecessary and inexcusable usability problem and should be fixed.
Another problem is that some web email providers think the confirmation email is junk mail. I can't do much about this but I can tell the user to check their junk email folder for the confirmation email.
However that doesn't help if the email is filtered out completely and deleted and not included in their junk mail folder. Some FanConcert users have reported not receiving confirmation emails at all, and I suspect this is what has happened. A user should be able to ask FanConcert to resend the confirmation email, though that won't help if it's just filtered out again.
Travis pointed out that I can make it easier to change forgotten passwords by allowing security questions such as "what is your mother's maiden name?". Currently the only way to recover a forgotten password is to have an email sent to you, which provides a link with a hash in its querystring to change your password. Because the old password is stored as a hashed value, it's not possible to send the user their existing password so they must change it.
Security questions, if made poorly, can be a security risk for users but there are two factors that make me feel better about them: firstly, as I mentioned above there is little to lose in the event of a break-in and secondly, the user makes his own security questions and should take some responsibility for their quality. If a user makes a security question that is easy for other people to answer, that is largely their own fault. Should I allow that or make the user select from a list of questions I think are more secure? It's a fair question, I've seen both approaches used on the web.
The user's login should be remembered between sessions for a fixed period, like two weeks. This is a common usability feature and again, is reasonable based on the fact that the consequences of disclosure are low. This was actually working on FanConcert until I started using the database to store session values, at which point it broke. I'm still investigating why.
To accomplish that feature, a key is stored as a browser cookie and when the user returns to FanConcert the key is used to authenticate the user instead of having them log in again. Any user with this cookie key could impersonate another user but that requires access to the cookie and therefor the machine it is stored on. Again, this is the type of security users should be expected to take some responsibility for -- but I could warn people about the dangers and provide an option on login NOT to remember the password between sessions, a common feature of web-based email.
Other improvements include consistency of terminology (pick either "log in" or "sign in" and stick to it), readable URLs, good overall usability, navigation, attractiveness and integration with the rest of the site.
Login will also be indirectly connected to the subscription payment system but only in terms of where the user can navigate to and what functions are available to them. I'm pretty sure they will remain relatively uncoupled. Payment is a whole other ball of yarn!
# Perception Problem or Technology?
I got in the mood for a rant, so I posted the following comment over on David's blog concerning Robert Scoble's list of perceived problems with Microsoft web development products. I'm copying it here because I forgot to trackback and I wanted to make it more visible. Some of it refers to other comments on David's post.
Though I respect people's opinions -- and some are quite interesting -- I find it amazing that people waste their cycles on arguments like this, especially when the conversation actually starts on the topic of why "Web 2.0 entrepreneurs like Ross tell me that they aren’t using Microsoft’s stuff" and actually *not* "why the Microsoft suite of tools just doesn't cut it for web-application development".
I'm not dismissing Web 2.0 as a fad but let's not ignore the hype. Scoble (and a lot of other bloggers) have a habit of jumping on hype in the guise of expediency, so I'll let it slide as blogging status quo. It's a knee-jerk and somewhat panicked reaction and companies like Microsoft don't respond to that sort of thing with any kind of speed. That's actually a good thing for the thousands of OTHER companies that DEPEND on Microsoft's web technology "stack", not that they were worried about Microsoft moving too quickly.
There's absolutely no reason why Microsoft should worry either. Large-scale enterprise development, the core of Microsoft's "web business", is not under attack by Rails (or it's cousins) yet and I doubt it would ever be. Enterprise web development moves at a calculated and glaciated pace and allows for changes at an even slower pace. Let's not kid ourselves and think that large organizations will suddenly remove their self-imposed internal war-zones and all agree to be agile and play nice together for the sake of the company, cause it just doesn't happen like that. People have their own internal agendas and careers that trump all of that. People also have a justified fear of being obsoleted by new technology, furthering agendas.
A great example is the cordoned off database camp that springs up in some big companies, to "protect data" at the expense of developer flexibility. Incidentally, that kind of situation is exactly why an MS SQL Server adaptor for Rails would be needed.
There are no upper managers with enough clout to do a systematic far-reaching change like that in a large company salivating over agile sofware development practises and especially new open source software like Rails. Nevermind that they are probably just ignorant of new and relatively unproven technology, it would just never work in a large company, where managers rule and so do checks and balances.
For you sofware process geeks out there like me, think Capability Maturity Model (CMM) and the higher levels of CMM some large companies aspire to. Rails does not shoehorn into the CMM and neither do agile practises, let alone iterative processes. The CMM is a dinosaur but it's proven to make reliable software (and money, just ask Motorola, who are CMM 5 in some areas).
Rails is great work and has innovative uses, but let's not pat each other on the back just yet for destroying the old paradigms that were the "Web 1.0". Rails is still a niche product and it's fabulous for small companies because it integrates a lot of agile best practises, which also work well for small companies.
Should Microsoft be worried? I'm not sure they should be. Like other people have said, small companies can't afford Microsoft stuff anyway so they prefer "free" software. Why would Microsoft want to bother trying to compete in this space against a community that can respond as quickly as Rails can? It's not a race worth getting into. Can David teach Microsoft some things about Rails? Definitely, I'm not discounting that. But it's more about getting to business value faster than a technology war of Ruby and Rails vs C# and .NET vs Java and J2ee. They are all different animals, with specific target "markets".
Like Scoble emphasizes after his list, they are PERCEPTIONS and everyone's got a different one. Scoble is describing marketing problems, not technical ones, that Microsoft needs to overcome. I think he hits it mostly on the head, except that everyone's been intepreting the list as technical problems.
If Microsoft responds with a software solution to the "Web 2.0" any time soon (ie. less than 2 years), I'd be very surprised. By that time, Rails will be onward and upward, the "Web 2.0" will be long forgotten and Microsoft won't care about catching up until their enterprise customers wake up and care about it first, which is perfect timing for their glaciated pace.
Everyone wins, and I can go back to not caring one way or the other ... except from a software process geek standpoint, of course.
# FanConcert URLs
Developing iteratively in little bits at a time it can make it easy to miss the big picture. One of the important -- and often underrated -- aspects of the big picture for websites is the URLs they use.
For example, FanConcert URLs are grouped by object class. In Ruby on Rails, this is first grouping is known as the controller. All of the Artist operations are in the same controller (artist). Some of the URLs look like:
Doesn't exactly roll of the tongue. I'd like to change FanConcert to use a more readable format, like:
That looks a lot better. The significant bonus is having all of the operations for each object grouped together by Rails controller. All of the add pages for every object will be in the add controller, like:
It makes it a lot easier to share common controller and view code for common operations like
Why am I so concerned about URLs now, you might ask? Thinking into the future isn't very agile, right? Maybe, but agile processes aren't hard and fast rules for me: sometimes I feel I need to think ahead. Sometimes I need to use common sense, not rules or processes.
People don't like it when you change URLs, especially if you don't gracefully redirect. I won't put in redirection code in this case because there aren't enough people impacted by this change and it adds complexity. If you're just using FanConcert, you probably won't even notice the different URLs.
As well, search engines are very unforgiving about changing URLs. It will take a while before FanConcert is properly re-indexed by the search engines after I do this switch. During that transition time people will get a lot of 404 errors from search engine links to FanConcert. I'll just make sure FanConcert's 404 page is more helpful than it is now, so people can find the new page location.
For the above reasons, the earlier the URLs are decided on the better. I need to do a little bit of forward thinking here and try to come up with future possible features that will have an impact on URLs. Maybe I can leave a hole for them to gracefully slip into? That's the idea.
A feature that will have a large impact in the future is translation for locales. When FanConcert is translated into different languages, you should be able to see the same page (and information) in any language that FanConcert has been translated into. What could the URLs for the different languages look like? Let's take the Add an Artist page as an example:
I don't have to worry about, because it's an HTTP redirect to:
which is the page in the default language (probably British English).
is the page in the default English (British).
The page in a specific localization of Canadian English. Others would be:
The convention is to have the localization part of the language in upper case, but URLs are typically not case sensitive. Some servers support that (ie. Apache) but it doesn't make sense for URLs to be case-sensitive on FanConcert. Case-sensitivity is unnecessary here and becomes a usability issue.
Why have different English translations for each locale? I think we take for granted how different the English nations actually are. We all have local customs, jargon, units of measure, date formatting, etc. I'd like to facilitate all of this and allow FanConcert to be a unique experience for each locale. Then people will feel more at home, instead of having to compromise or concede to larger nations that speak the same language. As a Canadian I think I speak from some experience in this area. :)
An obvious downside to this approach is that URLs for non-English users will not be as readable for those languages, for example:
This doesn't read well to a French person but is somewhat of a necessity from a few standpoints. First, it would be overly complicated to translate/alias all of the Rails controllers and actions used on FanConcert (the first and second words after the /).
Secondly, even though URLs are supporting more internationational characters, like those that are accented, if I'm not mistaken all languages cannot be represented in URLs, so there is a technical limitation there.
Thirdly, it would make it harder to switch between languages on the URL. If I want to switch between English and Japanese for the same page, all I'd need to do is change:
Translation and localization is definitely an area where I'd like/need to take a lot of feedback. It's hard to serve the minority well when you are living (mostly) in a larger group (English) that has, up to this point, dominated the Internet (hint: that is changing, my friend).
It's hard to relate and I'm going to need to do my best to sympathize with people that speak other languages because I want them to feel at home on FanConcert. I'd like every language and every locale in those languages to be able to be first-class citizens as much as possible. It would be up to the people in that locale to do the translation work, again I'd just facilitate it.
While I'm on the topic of languages: you'll be able to change localization preferences for your user. If you select you're from Canada, it would use the en-CA translation/localization and metric units by default but you could override it. That's the idea: FanConcert makes an assumption and in most cases it Just Works. The assumption can then be overriden by users that are exceptions.
A sensitive URL area is searching. I won't write too much about this area right now, but I will say this: search URLs should not only be readable but also generally hackable in the location text box -- people should be able to replace words in search URLs to make different searches. This is pretty standard on search engines and popular websites.
This not only makes the URL prettier but it also makes it easier to create third-party tools that point to FanConcert. A simple example is FireFox's flexible search box, which can be programmed to search most sites. If it's easy to connect tools like that to FanConcert, more people will search and use FanConcert. It's that simple.
Another area is output variations on the same data as a webpage in a different form like syndication feeds such as RSS, Atom and OPML. I'd like the URL to show that the information comes from the same source as much as possible. Right now that looks like:
which is a bit ugly, especially when you condsider that every new output format is a new Rails action, like
Remember how earlier I talked about wanting to be flexible on output formats and templates? That also factors into this. Maybe something like:
Where the parameter
For personalized output, a user's custom template could start with the user's unique username:
When you blend syndication and languages with URLs, you get something that looks like:
The data, language and template could all be independent of each other. On the other hand, a custom template could just use one language for simplicity.
It's all up in the air, these are just ideas. What do you think? Can you think of other areas of FanConcert that need consideration with respect to URLs?
# FanConcert Speculation 1 (More)
The first is internationalization (aka I18N), localization (aka L10N) and translation of FanConcert. I'd like to try something different than most sites: a collaborative approach to translation.
How will that work? Every string used in the FanConcert UI will be translatable, and people will vote on the translation for a given language. Just like voting on object attributes, the translated value with the highest score -- based on weighted votes -- will be used.
This means that the translation is not only flexible but probably more pedestrian. It also means that phrases may not necessarily be correct but errors will probably be glaring to a native speaker. FanConcert would need to make it easy to submit and moderate translations, so that incorrect phrases are replaced quickly and easily.
As well, since FanConcert will be added to regularly throughout it's lifetime -- a competitive advantage of an agile process and unit testing -- quick translation of new features will also be important. I could add a new feature and it would use English everywhere by default until it is translated. Obviously some English phrases in the middle of another language would be glaring, and users of FanConcert in that language would have some motivation to translate it as quickly as possible.
This also solves another problem with translation: it can be expensive. Not only am I not paying people to translate FanConcert but I also won't have to speculate about which languages to translate to next in order to get into emerging markets for FanConcert. When enough Japanese people start using FanConcert in English, they may translate it themselves over time into their native language for native Japanese speakers.
Translations would happen as they are needed and would vary in quality based on the number of people using FanConcert in that language because more peer review can produce a better product. There are a lot of technical obstacles to this approach to translation but I believe that it will give FanConcert a very serious competitive advantage.
Artists, concerts, venues, albums and record labels are from everywhere on the globe, so it makes sense to facilitate all of the possibilities not just from a completeness standpoint but also because of the new market possibilities it opens up -- and here's the very important point -- with very little additional work on my part.
The other main idea is making FanConcert a social website. That's as simple as allowing other users to communicate with each other and share their musical preferences and ideas on a limited basis. There are already music websites that do this but it's a natural fit to have these features in something like FanConcert as well.
Besides that, people just want to have the ability to reach out. We are social animals and we like to interact. People are fascinated by other people's thoughts as well as sharing their own opinions, so having a way for people to share will help grow a significant userbase that likes that functionality, which in turn leads to more content and higher quality content because there are more people moderating.
I'm hestitant to jump on the "social software" bandwagon but I do see value in a lot of the common features. FanConcert's priority, I believe, should still be information but some social aspects certainly can't hurt things and in fact probably help quite a bit. Thanks to everyone that's been pushing me on this aspect, it's been helpful.