I was recently reviewing over 500 code check-ins for The Archivist and in the course of my review, a narrative emerged. I could see the evolution of the project: sprint milestones, new features, etc.

Perhaps more interesting (and entertaining) than the narrative was the humor in some of the comments. Now, the vast majority of the comments were informative—and appropriate. But in the heat of the moment, well, some of the comments got more colorful.

Here’s a list of the ten more amusing comments:

1. “more futzing”

2. “don’t f*ck with my connection strings”

3. “fixed wonky export code”

4. “build for legal”

5. “fixed goofy routing logic in mvc controller”

6. “deploy for sprint” then “redeploy for sprint” then “another try” then “ok really the sprint build”

7. “fixed serialization, deserialization crap”

8. “who said dll hell ended with .net?”

9. “remmed out back door”

10. “fixed dirty logic”

Am I totally out of line? If you were my dev lead, would you come flog me? Care to share some of your more amusing check-in comments?

In an earlier post I talked about working on The Archivist Desktop version, and how the software began its evolution to the web and the cloud. In this post, part II of a series, I'll take a step back and explain how we concepted and executed The Archivist Web (alpha)and talk about what came before the design and code.

Audience Research

Once we decided to take on the project, we turned to our established group of experts for guidance. We already had a prototype of a similar application in The Archivist Desktop, so we had a cadre of fellow-Microsoft employees, friends and colleagues in other companies, and current users we could tap for solid user-testing. We focused on two primary groups of users: our internal Microsoft team and users of The Archivist Desktop.

Getting Stakeholders and Internal Microsoft Team Members On Board

We started by discussing the project's intricacies and goals with our internal team members and stakeholders. We set up short individual meetings with people, gathered feedback on how they used The Archivist Desktop and asked what new features they wanted to see.

“Conversation” with current users

Again, because we had already released The Archivist Desktop, we had an already-established user-base. Over the years there had been a few “vocal” users – who either wrote in to thank us, or report problems and failures – who we felt could provide valuable feedback from real-world use-cases. At the beginning stages of building The Archivist Web we simply emailed a few of the users with whom we had been in contact and asked them a simple set of questions surrounding specific topics such as types of visualizations they would use, how often the tool needed to update, the way they were using the data and the like. By asking specific questions rather than asking for general feedback, users were able to focus on key issues and needs that we could act upon. Often, by asking a generalized question such as “What would you like to change,” you get very generalized answers that are hard to turn into actionable tasks.

We also communicated our plans openly, which helped us make contacts in other parts of the company and take advantage of Microsoft's existing technologies and knowledge-base. Discussing our plans with the Windows Azure team, for example, helped us overcome some serious hurdles with our data-storage methodology. Without their help, we might have been derailed at the beginning.

What Current Users Want

At the beginning stages of the project, we simply emailed a few of our most vocal users and asked them questions about what types of visualizations they might use, how often the tool needed to update, how they were using data, etc. Asking specific questions instead of relying on general feedback helped us identify key issues and needs we could easily act on.

User personas

From our sit-downs and email conversations, we were able to segment our audience into 'personas', defined as representations of real users, including their habits, goals and motivation.

There are many ways to approach user personas. Some designers spend a long time fleshing out the behaviors and market segmentations of a target audience. We, on the other hand, took a more rudimentary approach, following the example of Smashing Magazine great article on the same subject.

Drive-by vs. Return Users

We know that most users will come by the site once, do a quick search, check out some visualizations, and then leave. That’s just the nature of building websites, and especially one geared toward a niche audience.

We wanted to provide an engaging experience with few barriers to entry for the drive-by user, but it was more important to concentrate on bringing people back. So, we decided to focus on our return users, Marketing Managers and academics.

The Marketing Manager

Marketing Managers primarily care about tracking sentiment—who is tweeting about what, how often, and how people feel about a brand or site over time. But up until recently, there were only a few ways to gain access to targeted Twitter data, most of which was presented in a very raw format that was difficult to extrapolate from.

Our plan was to change all this with The Archivist, and make it easy to find, share and interpret large swaths of data suited to the Marketing Manager's needs.

The Academic

Because of our experience with The Archivist Desktop, we knew that there was a surprisingly large demand for the Archivist in academia. During the last year or so, we’ve talked with with numerous professors and students who have found The Archivist Desktop useful for their research.

Whereas the Marketing Manager wants to track 'sentiment', the academic is interested in raw data that's accurate, easy to access and appropriate for use in research.

Wireframes

I thought it might be useful to provide the full PowerPoint walk-through of our wireframe set, which gave our team an understanding of how different users might come to and interact the site.

What we couldn't do (Feature Scoping)

Coming to agreement on the wireframes was easy, but brought up questions about what data we could actually display to users. As I noted earlier, I spent a lot of time thinking about the frequent requests for filtering by date, secondary keywords and other items. Because of how Twitter works, some of this data wasn't consistently available. Other items were so gnarly we couldn't scope them into this release.

The Visualizations Visualization explorations

As Karsten and I were wrapping up the early Archivist Desktop version, I explored additional visualizations I thought we might be able to incorporate into the downloadable WPF application. Since we were working in WPF at that point, I felt free to explore visualizations that might be useful to our audience. Some that I came up with:

For a few specific reasons, we decided to ship The Archivist using the ASP.NET charting contols, and made the decision to feature the following charts.

• Tweets over time
• Top Users
• Top Words
• Top URLs
• Tweet vs. Retweet
• Top Sources (software)
• Use ASP.Net Charting Controls

In my next post, I'll talk about taking the site from wireframes and explorations to a functional HTML + CSS3 + jQuery site, and how I worked with Karsten, who made the beauty of the backend happen in the cloud.

Do you use the same process as us, moving from problem to persona to wireframes to design? Leave a comment and be sure to follow us on Twitter @Mixonline.

I recently learned something extremely fascinating from a book called Gut Feelings: The Intelligence of the Unconscious. In Greg Gigerenzer's own words:

Intuitions based on only one good reason tend to be accurate when one has to predict the future, when the future is difficult to foresee, and when one has only limited information. They are also more effective in using time and information.

What Gigerenzer means is that people generally rely on one good reason to make decisions when the outcome is truly unknowable. And, that's OK! It's worked wonders for us for thousands of years.

But the more interesting insight hidden in there for us designers is that our best bet at predicting the future for something truly unknowable—say, how an audience will react to a new product, or what'll happen to the price of a certain stock—is often to follow your gut.

Mo Options Mo Problems

Now, this may seem preposterous to our spreadsheet generation, because it runs counter to the currently dominant microeconomic paradigm called Rational Choice Theory. According to it, human beings are rational—Homo Econonomicus—and given a set of alternatives, will weigh the costs and benefits of each to pick the most suitable one. Increase the alternatives, and you increase the chances of finding a more suitable pick, right?

The credo "more options are better" has become a staple in our generation. From supermarkets that carry hundreds of different types of cereals to our propensity to follow thousands of people on Twitter, we've given in to the notion that more choices lead to better outcomes. Sadly—and you know this in your gut—the exact opposite is true in most cases. More options lead to confusion. Not conversion, as many marketers would have you believe.

Gigerenzer's own take on that:

Complex analysis, by contrast, pays when one has to explain the past, when the future is highly predictable, or when there are large amounts of information.

Gigerenzer cites numerous studies in his book supporting this critical point, but there's a classic one we're all too familiar with—Apple.

An Apple a Day

When Jobs took over Apple in 1997, Apple sold over 40 different types of products, from printers to handhelds. Particularly overwhelming was Apple's lineup of computers, which hovered around 50 different models. Which Performa is right for you? The 5200CD or the 5215CD? Jobs later confessed that even he couldn't figure out the answer to that question. So how could customers?

Among the other unpopular restructuring decisions, Apple cut most of its product lines and reduced its computer line to four machines: novice and professional versions of a desktop and a laptop. History will remember it as a key step in facilitating Apple's comeback.

But, why is it that too many options paralyze us while fewer options spring us into action? Entire books have been written on the topic, but if you want a succinct explanation, you need look no further than Miller's Law.

Miller's Law: The Magic Number Seven

Princeton's celebrity cognitive psychologist, George Miller, published the seminal paper on the topic of human cognition in the Psychological Review in 1956. In it, he argued that the average human could hold 7±2 objects in working memory; this is now fondly known as Miller's Law. Miller's insight has had an immeasurable impact on the design of everyday things in the last 50 years.

The classic application of Miller's Law in the software industry is when a designer tries to convince stakeholders that the navigation menu of a web site needs to be limited to seven (give or take two) choices. But more generally, we can credit our subconscious pull towards "keeping things simple" to Miller; at least within the context of user experiences. That is no small feat.

Now if only someone can do the same for the benefits of white space.

KISS my Point

Your users are already overwhelmed. Jonathan Anderson of UXMag illustrated this in a very eye-opening post—These are your users... read and be horrified. But as Gigerenzer writes, even when they're not overwhelmed, they make their choices based on one or two criteria. Providing your users with more options is generally counter-productive. Of course, there are exceptions to this rule of thumb.

The bottom-line? Prune. Cut. Shorten. Simplify.

And once you're done?

Simplify some more.

You may be wondering how frequently The Archivist updates archives. Well, the answer to the question is more complicated that it may first appear. Let’s dig in.

The Archivist interacts with Twitter using the Twitter Search API, which it polls at variable intervals based on the frequency with which a particular archive is updated. We call this the elastic degrading polling function. This algorithm helps The Archivist be a good Twitter citizen, allowing us to poll Twitter conservatively while at the same time maintaining archives with the latest tweets.

Here’s how the algorithm works: When a user makes an archive ‘active’, the polling process begins. Every archive is inspected once an hour to determine how ‘hot’ it is. We determine how hot an archive is by recording how many results we get back each time we poll Twitter (the maximum we can pull at any one time is 1500).  We use this number to determine how frequently to poll Twitter for that archive. Depending on the number, we either hold off on polling for a given interval or query again, based on the following buckets:

So, let’s look at an example. Say we have an archive going for the term ‘Wittgenstein’. When the Archivist checks on this archive at 10 AM, it discovers that the last query for Wittgenstein only returned 10 tweets. It also discovers that this archive was last updated at 9 AM. The Archivist won’t poll Twitter for this archive, because the tweet count isn’t high enough and the archive had been queried within 24 hours. Since the archive is in the 24 hour bucket, the same thing will happen when The Archivist checks on this archive each hour.

Once 9 AM rolls around on the next day, since 24 hours have passed, Twitter will be polled for the Wittgenstein archive .

Now, let’s say for some reason there’s a flurry of tweets about Wittgenstein—when that archive was updated at 9am, it pulled 600 tweets. In this case, the archive adjusts because it has become hot. It is now in the 1 hour bucket instead of the 24 hour bucket.  So, when 10 AM rolls around, the Wittgenstein archive gets updated again. 

But let’s say at 10 AM it pulls only 250 tweets. Well, now the archive moves to the 8 hour bucket. So,the Wittengenstein archive will not be polled again again until 6 PM. Let’s say it pulls 1000 tweets. Well, it goes back to the 1 hour bucket, since it appears to be hot. At 7 PM the term is checked again.  This time, the response is only 10 tweets. It seems to have cooled off quickly, so we’ll move it back to the 24 hour bucket.  

Some of you may notice that there’s a chance that The Archivist could possibly miss tweets when a term becomes hot. This is a reality of our architecture and is justified by the following: First, once a term gets hot, the amount of data can grow quickly. Ultimately, in that scenario,The Archivist becomes a statistical sample as opposed to a true historical record.  Second, Twitter itself doesn’t guarantee that all tweets will be returned for a given search. See http://help.twitter.com/entries/66018-my-tweets-or-hashtags-are-missing-from-search and http://dev.twitter.com/doc/get/search for more on this. Consequently there is no way that The Archivist can ever claim to be a true historical record. Third, The Archivist is optimized for following non-trending topics over a long period of time, as opposed to trending topics over a short time.  For a tool optimized for the latter scenario, see Archivist Desktop. Another option would be to run your own instance of The Archivist Web and tweak the polling algorithm, which would be trivial to do. Contact me if you are interested in doing so.

A couple years ago, my friend irhetoric and I were chatting about Twitter, as we were at the end of building Flotzam, a social networking aggregator tool that collects and visualizes the ephemeral information floating around in the social sphere. In some ways, this ephemerality is beautiful: all this information swirls around us like leaves that appear and decay over time. On the other hand, if this information is valuable, capturing and keeping it could be useful.

The most striking—and easily accessible—of these potentially useful data streams is Twitter. Tweets, while often inane or funny or what-have-you, can impart important information, whether by themselves or in aggregate. Take, for instance, Twitter’s role in the Iranian Elections of 2009, in which Iranian protesters used Twitter to broadcast information that would have otherwise been suppressed, using the hashtag #iranelection. The election also provoked people from all nations to “show solidarity with the protesters” by changing their Twitter avatar to green. What good this did is unclear, but it’s an interesting way of disseminating information and sentiment.

The problem, though, is that you can only view tweets as far as two weeks in the past (or less for very active terms such as #iranelection). Fortunately, by 2009 Karsten and I had designed and developed The Archivist Desktop for MIX Online, and were able to save and capture this data. We ended up with 22,000+ individual tweets over a 2-week span.

The Archivist Desktop

The Archivist Desktop was originally released as a “mini-lab” and has been installed by users from such varied fields as marketing, software development, social sciences and research. It was a ‘small project’, but Karsten and I thought it was a worthwhile for a few reasons. With The Archivist, users are now able to:

• Save tweets Users can save the tweets they’re interested in.
• Export the data to analyze The Archivist provides an easy way to export to Excel for visualizations and data-mining.
• Visualizations to interpret There are two useful visualizations shipped with The Archivist, which can provide quick and easy understanding of the data the user has collected.
• Distributed network The Archivist is an installable program and works on a distributed network, which reduces the likelihood of one user being affected by another. This is especially important when dealing with Twitter rate-limiting.
• Private, secure data Sometimes users are sensitive about the data they’re collecting. By having an installable program, the data is saved to a user’s hard-disc and accessible only to those users who have direct access to that computer and file.

Challenges with The Archivist Desktop

While The Archivist Desktop is great in many ways, we needed to address a few problems we saw as the project matured. Over time, we were able to connect with users around the globe and encountered a few recurring requests:

• “Always-on and connected?” Because The Archivist Desktop is an application, it needs two things to work: it must be running and have an internet connection. If your computer goes to sleep, for instance, the application stops running. Or, if you’re running it on a laptop and you travel, you get no connectivity, no archives.
• “Not Just Windows, please.” Many of our users, including some of the MIX Online team at times, are Mac users. By making this project Windows-only and relying on various other technology dependencies, we were shutting out a large and potentially enthusiastic part of the community.
• “I’d rather not install a program.” We found that a large population of users were hesitant to install a program they were unsure of. The Archivist is a small file and, for Internet Explorer users, a one-click process. To many users, however, installing a program is a hurdle that’s very difficult to cross.
• “I wish I could go further back in time.” Suppose User A started an archive on the term “Microsoft.” If four months later User B started an archive on the same “Microsoft” keyword, all of the data User A collected would not be available to User B. That seems a waste. Data should be ultimately shareable. Free the data!
• “More eye-candy, please.” The users of The Archivist expressed they found great value in the data-visualizations The Archivist provided, but wanted more. The data visualizations people asked for were often the same, and included top keywords, top users, top links, and volume over time. There were also many other great suggestions on how we could parse the data from Twitter. Users had very specific requests that coincided with their goals, whether they were focused around brands, politics or pop-culture.
• “I’d rather share a link or image than a bunch of data.” While it’s easy to attach an Excel spreadsheet or .xml file to an email, that isn’t the best experience for sharing data or understanding around that data. How could we help people take full advantage of this new data and information?

The Archivist Web Alpha

We decided to move The Archivist to “the cloud” and create a web-based, Windows Azure-powered web tool. There are numerous benefits to creating the new version:

• Always-on By creating a web service, we were able to take advantage of the always-on nature of the internet and could poll Twitter for data whether the end user was connected or not. This enables users to set up archives without having to monitor their status more than once every 30 days to keep the archive active. We no longer needed to require users to actually run the program—because essentially, we’re running it for them.
• Cross-platform The Archivist Web runs on multiple browsers and platforms. This opens up The Archivist to essentially all internet-connected users. Some sort of authentication is required, however, to save archives. To make this as seamless as possible, we tapped into the Twitter authentication system and now have people simply use their Twitter credentials to authenticate themselves.
• Data-sharing By making The Archivist a cloud-based application and storing collected information in shared database, we can make archives that have already been created available to new users. For instance, if User A creates a “Microsoft” archive on January 2 and User B starts a “Microsoft” archive on March 2, User B’s archive should reach back to January 2. This data is unavailable through a normal Twitter search. This feature, in our opinion, is super valuable.
  
  An additional type of sharing is the ability to tweet both The Archivist Web itself and your archive set—which will only display those archives you have set to “public” – individual archives, as well as individual visualizations within that archive. It’s as easy as sharing the url that appears in the address bar at the top of any browser.
• No installation required By no longer requiring installation, users can use the service without the commitment of trusting the source, waiting for the download, ensuring they have the right system requirements and the other barriers to entry.
• More visualizations Our users had asked for a lot of new visualizations. Of course, we had ideas on what *we* would like to see and found useful, but talking to our current and potential future users helped us determine which visualizations were the most useful to our audience.

In my next Opinion, I’ll be writing about the process of scoping the new project to our resources, time-constraints and technical limitations. From there I’ll try to explain the process of releasing a project and making it be successful.

In school, we’re taught that we should write at a sixth to eighth grade level when writing for the general public. Newspapers have historically required their writers to write at a low grade level, and in his book "The Vanishing Newspaper", author Philip Meyer argues that newspapers are struggling, in part, because "Many newspaper stories are too hard to read". Meyer arrived at this conclusion by measuring thousands of newspaper articles with the Flesch-Kincaid readability test. He found that only a quarter of the articles were written at or below the eighth grade level. As a point of comparison, the Flesch-Kincaid grade level for this post so far is 10.4.

Of course, readability is just one element of simplicity. We praise students who are good at communicating complex topics in simple language, and Albert Einstein even said, “You do not really understand something unless you can explain it to your grandmother.” But I’m worried that all of this talk about simplicity may be missing the point.

Another Perspective

Tyler Cowen is an economist at GMU, and a founder of the hugely popular "Marginal Revolution" (MR), an economics blog I’ve enjoyed for several years. Cowen and blog co-author Alex Tabarrok consistently post on complex topics and never "talk down" to readers. Despite this, the readership has steadily grown, and MR boasts some of the most engaged and thoughtful commenters on the Web. In fact, Cowen sometimes writes posts he knows will be above most readers’ heads. Far from repelling or discouraging readers, most find this challenging and inspiring.

One could argue that MR has a unique audience, and Cowen certainly wouldn’t disagree, but I’ve noticed a similar pattern in other places. Our team runs a couple of large conferences each year: the Professional Developer’s Conference (PDC), for developers, and the MIX conference for Web developers and designers. Attendees can turn in session evaluations for each session, including verbatim comments, so we can evaluate speakers and topics. If we’ve learned one thing, it’s this: if you talk down to attendees or make your content too simple, you’ll be slaughtered in the evaluations. But if you challenge the attendees, you’ll make them happy, even if many find your content hard to follow.

Making Grandma Work for It

Does this mean that blog readers and conference attendees are vain posers who need to be treated like "big kids" to feel good about themselves? I don’t think so. Over the past few decades, research into a phenomenon called "contrafreeloading" has provided some clues that help explain why people don’t like being talked down to. In short, "contrafreeloading" means that animals, including humans, prefer to work for their food if given the choice between completely free food and identical food that requires work to obtain. There are only two exceptions to this rule: housecats and economists are both perfectly rational, and will happily freeload when given the chance.

The implications are clear. If it’s a topic she’s interested in, Grandma will prefer to work things out for herself rather than be spoon-fed. Don’t water things down, and don’t talk down to her!

What do you think? Do you have situations where spoon-feeding is definitely the right thing to do? Or have you had spoon-feeding backfire on you? Leave us a comment below.

I’m not quite sure how or why it happened, but at some point, the user experience community—the group of people who are truly experts in interaction design—lost their grip on the social media space.

In the void, a number of self-appointed social media experts (some knowledgeable, some completely faking it), have catapulted themselves into prominence. The success of these so-called experts has much to do with the fact that many of them are marketing inclined, and thus talk the talk. Proof positive can be found on Facebook, where the majority of corporate pages are simply channels for one-way messaging.

The thing is, social media does not play by traditional marketing rules. It plays by digital rules, where conversation is key and authenticity reigns. “Doing it right” means knowing how to develop and design interactive programs and environments.

Why Did UX Drop Out?

We, as user experience professionals, understand the rules of interaction, yet are reluctant to play. Why? If you’re a user experience pro, my guess is that you bailed out of the social scene for one or more of the following reasons:

You’re Bored.

You were probably one of the first users on Facebook. Now everyone, including our mother, is there and you’re ready to move on. How could you possibly help a client develop a Facebook strategy when you, yourself, can’t stand updating your status?

Remember that social media is more than Facebook and Twitter. There’s always a popular channel—that’s easy. But developing a strategy is much more complex, and way more interesting than you think.

Strategy starts with users and their needs and propensities. Are they creators or collectors? Do they blog or rate? Once we understand users’ needs, then we can develop a meaningful strategy, whether that be on Facebook or LinkedIn or a stand-alone experience. You should try it sometime.

You’re Extremely Hip.

Social media is very much en vogue, and you’re just too cool to be following trends. You’re a leader, after all (see point number one). If you have to read about someone else’s lunch again, you just might lose yours.

But consider this: throughout history there have always been certain rights attached to certain people. Some folks get to talk, some folks don’t. And that was true until very recently.

Somewhere in the mid 20th century, somewhere between William James and ARPANET, somewhere in the span between Hunter S. Thompson and fanzines, we (in the broadest possible sense) found our voices. In our post-aural, post-structural, post-modern society, communication is a right, not a privilege.

And that’s why you should rejoice to the high holy heavens every time that guy posts what he ate for lunch—because it means you live in a world where anyone can say anything at any time, when anything can be published to anyone in the world instantly.

This is bigger than you. Embrace it.

You Don’t Want to Be Associated With the Crap

You’ve witnessed a number of brands and marketers dabble in the social media space and you’re embarrassed by the display of low quality experiences. You should be. It’s pathetic. But that’s because marketers are trying to apply old marketing tactics to entirely new paradigms.

Marshall McLuhan said that all forms of new media take on the likeness of existing forms before they find themselves. That’s what happened to the Web (brochure-ware anyone?) and that’s what’s been happening to social media. The space is in dire need of experts who really understand it.

If you loathe the crap, stop whining and do something about it. Commit to bringing your “A” game to the table and treat social seriously.

You Cut Your Teeth At the Wrong Time

You entered the industry just after the Internet bubble bust, when budgets were very tight and there was no room for experimentation. During this time, community functionality (what we used to call social media) was often stripped out of projects, as was anything that wasn’t absolutely necessary.

It’s not your fault, but it’s time to learn. See the next point for a good reading list to get started.

You Just Don’t Get It.

No, it’s not that you don’t care, or that you’re too cool. You just seriously don’t get that the Web is social. But here’s an analogy that might help: separating social from digital projects is like separating plumbing from a kitchen redesign. It makes no sense.

For every project you strategize or wireframe, you should be thinking social. It’s not separate.

If you’re not sure where to start, there are numerous academics who have been writing about social media since The Well (Google it). For example, Howard Rheingold, or Barry Wellman at the University of Toronto. Henry Jenkins at USC and Jenny Preece at the University of Baltimore. There’s also former Microsoft researcher Marc Smith, and currently with Microsoft, Danah Boyd. Clay Shirky at NYU and Charlene Li with the Altimeter Group. Entire communities of people exist who study social media and their research and thoughts are easily assessable.

It’s Time To Step It Up

It’s time for the user experience community to step it up and realize that we have a real contribution to make in the social media space. Social media is about identity and reputation management, user needs and behaviors, motivations and interaction—all user experience specialties.

Don’t let the space go to so-called experts, some of whom weren’t even in digital three years ago. And please don’t make me hire a social media specialist for my team when I really just want to hire a user experience pro with a solid understanding of social media.

The time is now. More is asked of you. Let’s take it back.

When the Nobel Committee announced that President Obama would receive the peace prize, how many tweets did that produce? 90,000. When Oprah announced that she would end her TV program when the contract was completed, were there more or fewer than the Obama tweets? Fewer. 73,000 fewer. How many tweets have there been about health care reform (#hcr)? Over 600,000.

Using The Archivist In Class

In the fall of 2009, I wanted to teach two courses about politically oriented messages on Twitter. My students were going to be the first in the country to have an entire course devoted to this subject. But there was a problem: how could we learn how many tweets there were about Obama receiving the Peace Prize? And anything else we might want to know?

It was difficult to find a solution. There are many systems for searching Twitter, including Twitter itself, which allows you to enter a search term and watch the tweets flow by. But we needed a system that would not only search for tweets, but also count, collect and export them. It turned out that The Archivist was the only system that could do the job.

Politics On Twitter

Many believe that Twitter messages are largely about the concerns of an individual. They interest a few friends, but have little wider interest. That may be true of most messages, but my students found that politically oriented tweets aren’t like that at all; People interested in politics circulate a lot of ideas, and those ideas are as likely to come from outside Twitter as in. URLs are much more frequently found in politically-oriented twitter messages than in other kinds of messages.

Even more convincing: Twitter was the tool of choice for celebrating President Obama's election in the fall of 2008. The number of tweets about his inaugural address in 2009 broke every record up to that point.

By studying Twitter, my students were able to explore a lot of politics—comparing members of congress tweeting with members of the British House of Commons, for example. What they did was only possible because of The Archivist. And the new web version is going to be even better.

note website on Bob’s computer :)

The Archivist is a new lab/website from Mix Online that lets people archive, analyze and export tweets. Here’s a little more about why we built The Archivist and who we built it for.

The Transitoriness of Twitter

Twitter isn’t going away anytime soon. From Twitter:

The tweets keep coming. And coming. Exponentially.

Then again, tweets are disappearing all the time.  Tweets come and go.  They’re ephemeral.

Tweets are transitory because that’s the way Twitter is designed. Twitter doesn’t allow access into its database for more than two weeks—and for a trending topic, you may not be able to access tweets older than 24 hours.

Recently, Twitter announced that it’s going to donate its entire archive of public tweets to the Library of Congress. How cool is that?

Bummer, though, that the database of tweets isn’t accessible yet.

Why a bummer? Because there are all kinds of things you can find when analyzing a dataset of tweets.  Patterns emerge from the noise. Discoveries are made.

So how do you capture all those tweets before they disappear into the ether? 

Enter The Archivist.

The Archivist:  Archive, Analyze and Export Tweets

The Archivist sets out to accomplish three things:

• Archive tweets
• Analyze tweets
• Export tweets

Now, to be clear, the Archivist is very different than what the Library of Congress will make available. The Archivist is not an archive of all tweets. The Archivist can’t go back in time.

But what The Archivist can do is allow individuals to begin archives right now on searches they care about.

Let’s take a look at features of The Archivist.

Creating Archives

The Archivist can create archives of tweets equivalent to any search done at http://search.twitter.com.  But there’s a difference between searching directly on Twitter and searching from The Archivist: When you start a search from The Archivist, your search will continue to run—day and night, every day, every night. And The Archivist will create an archive based on that search.

Any tweet that matches your search gets saved and is available for you to peruse. You can come back later and see what’s been harvested in a day, a week or a month.

Again, it’s important to note that The Archivist can’t go back in time to get old tweets. But once you start an archive, you can be sure that, moving forward, The Archivist will monitor that term. So, using The Archivist takes some forethought: you need to know in advance that you’ll care about a given set of tweets later.

Doing Analysis

So now you’ve got a few giant archives of tweets based on your searches. What to do with an archive of 1,000 tweets? 10,000? 100,000?  1,000,000?

Well, this is where data mining comes into play. The Archivist data mines on your behalf, providing six visualizations that offer insight into a given archive of tweets.

The six visualizations are:

• Volume Over Time
• Top Users
• Tweet vs. Retweet
• Top Words
• Top Urls
• Source Of Tweet

These six visualizations are shown in a dashboard.

You can click each chart for a larger view with more data and interactivity. You can also immediately see a top user’s Twitter profile or go to the web page of a top url, for example.

You can read more about the specifics of these visualizations.

People often want to do deeper analysis than the six visualizations provided allow, or combine visualizations. (For example: “Who were the top users on a given date for a set of tweets that weren’t retweets?”)

Unfortunately, we can’t provide that level of analysis. But we know how you can do it, which gets to our next topic: Export.

Exporting Tweets

If you want a DIY analysis, you can download an archive and start slicing and dicing it yourself.

We allow all archives to be saved as tab delimited text files. These are easily loaded into Excel, imported into a database or parsed with a program.  Here’s a screenshot of an archive that’s been pulled into Excel 2007 and then had pivot tables created:

  

And, with Excel 2010 and the PowerPivot plug-in, huge archives can be turned into pivot tables with lightening-fast analysis.

Who Is The Archivist For?

We designed The Archivist around three core scenarios. The first is what we called the “drive-by” scenario; the second is the “data miner”; the third is the “developer”.

The Drive-By User

The drive-by user is someone who comes to the site simply to get a real time snapshot on a given search.  He/she doesn’t sign in, but remains anonymous on the site. The drive-by user can view anyone else’s public archives and do searches, see visualizations and even export searches. However, any searches started by the drive-by user are not persisted. That’s because you have to sign in if you want The Archivist to start archiving.

Searches by the Drive-By user create a non-persistent archive, including visualizations, of 500 tweets and will display the recent 100 tweets for that search:

So, for example, you could do a drive-by search on a term and quickly get a sense of what is happening with a given search. But, because you didn’t sign in, The Archivist won’t continue to monitor that search.

The Data Miner

The Data Miner is another scenario.  The key difference between the drive-by user and data miner is that the data miner signs into the site.  Authentication happens through Twitter. Once the data miner has signed into The Archivist, his searches are saved and persisted.

So, going back to the scenario, a data miner signs in and starts a search. The Archivist then continues to monitor that search term ad infinitum. If the archive gets to a certain size, The Archivist will close it and start a new one based on the same search.

Data miners have a homepage on The Archivist that coordinates with their Twitter username. So, for example, if I sign into The Archivist with the username of mixonline, I will have a homepage on The Archivist that looks like: http://archivist.visitmix.com/mixonline. I can see all the archives I created, how many tweets are in each archive, and whether the archive is active/closed and public/private on the homepage. Here’s an example of an Archivist homepage (signed in):

Note that you can see each archive’s size, activity and whether it’s public or private.

Data Miners can make an archive available to the public. By default, though, all archives are private. Here’s a screenshot of what my profile page looks like when someone else hits it:

Only archives that I’ve made public are available for others to see.

Archives created by Data Miners can grow to 500,000 tweets.  Once an archive hits that number, it gets closed an a new archive gets started for that term.

 

The Developer

We’d be remiss if we didn’t provide a way for developers to access all this data and analysis.  That’s why all public archives and data analysis we do to create our visualizations are available for download programmatically. So you could download the dataset of top users for a given archive and then create your own data visualization with that data, for example.

We’re also providing all the source code used to build The Archivist (You can download it here). With this, you can run your own instance of the Archivist. Read more about the developer documentation here.

In Summary

We are excited to release The Archivist and look forward to your feedback. Give it a whirl and let us know what you think!

Flotzam at a conference is fun. It gives the audience the ability to immediately participate in the event—their photos, tweets, etc, show up on the big screen, giving them immediate satisfaction and making them feel like they are part of the event.

If you’d like to see Flotzam in action, you can try it out by installing it yourself. Or watch videos that shows off the application on YouTube.

Here’s a guide for using Flotzam at a conference.

Flotzam and Social Networks

Flotzam works with a variety of social networks. Here’s a brief intro:

• Twitter integrations lets you see what people are saying about the conference in almost real-time.
• Flickr and Facebook integration allows attendees to upload their photos of the event; Flotzam will pull and display the photos.
• If there will be a lot of on-the-fly press about your conference, Flotzam’s Digg integration will be helpful.
• Flotzam will display any RSS feed
• Flotzam lets you see and display Youtube videos about your conference

The end result is that Flotzam aggregates all these networks and creates a compelling visualization.

A few things to note:

• If you want to add another social network to Flotzam, you’ll have to write some code. Here’s how to do it: http://visitmix.com/labnotes/How-To-Add-a-New-Data-Source-to-Flotzam. If the social network you want to add supports RSS, it will be easier to integrate.
• If you don’t like the skins we have, you can find a designer fluent in Expression Blend to reskin it, as discussed here: http://visitmix.com/labnotes/How-To-skin-Flotzam
• Because Flotzam pulls data in real time, there’s a chance that inappropriate content could appear on the screen.

Getting Flotzam Up and Running

If you want to use Flotzam for your conference, you need to do the following:

1. Install Flotzam here: http://visitmix.com/labs/flotzam/install/flotzam.application  or download the source here:  https://code.msdn.microsoft.com/Release/ProjectReleases.aspx
2. You should have a machine with w/2GB RAM and a video card of 256MB. Be sure to test on your hardware to catch any issues in advance.
3. The machine must have network access. Flotzam does not do well with intermittent connectivity; if you can’t guarantee a stable connection to the internet, don’t use Flotzam.
4. Determine what feeds you want Flotzam to display.

Note that you’ll need to show attendees how to access the services you set up. If you don’t want to do this, one option is to simply have your conference folks update twitter and upload photos.
Here’s how to configure Flotzam:

1. All settings can be changed by clicking SETTINGS AND OPTIONS in the upper left corner.
2. If you want everyone at the event’s tweets to appear, you can create a Twitter account such as myconference, let people know about it, and then tell Flotzam to “Watch My Followers.” To do this, visit the Twitter settings panel and provide the username and password for the myconference Twitter account.
3. If you set up a Facebook event for your conference, you can enter the Event ID in the Facebook settings panel. If the URL for the event is http://www.facebook.com/event.php?eid=2367953648, the event id would be 2367953648. You'll have to provide your credentials to Facebook.
4. You can add a different Flickr tag in the Flickr settings Panel.
5. You can specify a DIGG setting and a YouTube tag.
6. You can manage which RSS feeds show up, but you’ll have to do so in Internet Explorer.
7. Lastly, select from the dropdown in the general settings tab if you want a different skin to show up.

Here’s how to skin Flotzam:

1. Download the latest Flotzam source code and open Flotzam.sln in Expression Blend.
2. Hit F5 to see the application working.  You should see different "flotzams" appear on the screen. By default, you will see public Twitters and Flickr images as well as videos posted to YouTube. You can change what Flotzams you get (personalize them, etc.) by tweaking the settings.
3. If you want to see prettier Flotzams, you can change the skin in the "General" tab of the settings. These are skins that Tim Aidlin made.
4. Now it’s your turn to make a skin.  In Blend, go to the Resources tab and find the file new.xaml.  Expand it and you should see seven templates.  If the Resources tab is blank, go to the Project tab, open the file Window1.xaml and then go back to the Resources tab. You should see new.xaml.  It should look like this:

5. Open DIGG_TEMPLATE.  There you see the fields you have to work with in the designer.

6. Expand the stack panel. Inside the StackPanel you can see the visual tree with the different fields you have to work with.

7. Double-click [TextBlock] "Username".  Change to the Properties tab.  Notice that the Text field (under Common Properties) is highlighted in yellow. This is because the Text field has been databound to the Username field.  DON'T MESS WITH THE DATABINDING.  But feel free to mess with anything and everything else.  For example, you can copy the whole TextBlock into a different container; you can change any of its other properties, but don't mess with any field that is highlighted yellow.

8. A default animation has been applied to each of the templates.  You can see it by clicking the >> in the Objects and Timeline pane. You will now see OnLoaded1.  This animation is very simple: it just fades out the entire Canvas over 5 seconds.  To remove this animation, just go to the Triggers pane and click the minus arrow.

9.   By default, Flotzams are placed on the scene at random. This is done from code.  If you want to turn this off (so that they always enter stage right in the middle of the screen), open Window1.xaml.cs. On line 27, change the randomizerFlag to false.
10. You’re also welcome to change the background of Window1.xaml.

Here’s how to add a new data source to Flotzam.

1. Create a new data model class that derives from DataModel in data.cs – look at the other derivations of DataModel to see how the model works
2. Add this class as a new Object Data Provider to datatemplates.xaml  storyboards.xaml
3. Add new member variable for data at the top of window1.xaml.cs
4. Load member variable and activate it at line 194 of window1.cs.xaml
5. Add to Settings
5a. Create new tab for settings
5b. Add settings to Settings Property in Visual Studio Properties
5c. Update save_click for when user changes settings (window1.cs.xaml line 82)
5d. Update settings form in window loaded
6. Make a new datatemplate (or else muck with the data model to trick the data binding into working by renaming properties) in datatemplates.xaml
6a. Do databinding
7. Update PickPanel() method in window1.xaml
7a. Increase random picker by one
7b. Add to giant if statement
7c. Add member variable to count the items
7d. Update line 299 and add that setting to the condition to prevent infinite loop

Several pages of an upcoming Mix Online lab are fundamentally static, except for a single hyperlink that toggles depending on whether a user is logged in or not.  To optimize the site, I wanted to use caching on these static pages. But by adding the OutputCache attribute to these pages, a user could end up getting a cached page with the wrong logged-in hyperlink: users who weren’t logged in would see a page that stated they were logged in—or vice versa.

The solution? Using the VaryByCustom parameter of the OutputCache attribute, so that both a logged-in and a logged-out version of the page are cached and served up appropriately. Here’s how to do it:

First, add a method to the global.asax.cs file, which overrides the VaryByCustom call:

public override string GetVaryByCustomString(HttpContext context, string arg) { if (arg == "IsLoggedIn") { if (context.Request.Cookies["anon"] != null) { if (context.Request.Cookies["anon"].Value == "false") { return "auth"; } else { return "anon"; } } else { return "anon"; } } else { return base.GetVaryByCustomString(context, arg); } }

Then, decorate the controller actions with the OutputCache attribute:

[OutputCache(CacheProfile = "StaticPage")] public ActionResult Index() { return View(); }

When using caching, I like to keep all the profile information in web.config. This lets me update things without recompiling. Here’s what the cache profile looks like:

                    

               
               
               
           
       
   

Notice how the varybyparam attribute is set with an asterisk, and the varybycustom attribute has the string IsLoggedIn, the term that gets passed to the VaryByCustom override.

I’m using my own kind of authentication (more on that in a future post), so checking a cookie is sufficient. If you’re using ASP.NET Membership Authentication, you can use slightly different logic such that pages are cached based on the username itself. See this post for a sample—and tip o’ the hat to Chris Mullins for setting me on this path!

This is what a prosecutor had to say about the incidents:

"You hate to tell people to be wary of your fellow citizens, but the reality is you've got to be wary when you are doing something when interacting with strangers on Craigslist," prosecutor Lindquist said. (CNN, 2010)

This is Craigslist’s "answer to the problem". Seriously? Is simply "trusting your instincts" and "meeting in a public place like a café" enough? I don’t think so.

The Accountability Problem

Because most people who use sites like Craigslist do so anonymously, accountability is a challenge. You can never be sure that who you’re communicating with is who they say they are. And even if someone truthfully identifies him or herself, there can still be malicious intent.

Since we can't measure intent, there is one obvious way around the problem: Make people correctly identify themselves.

Anonymity and Privacy

To log into Craigslist, all that's required is a verifiable email account, which can be created in 5 minutes by anyone. This protects account owners from having others modify their information, but it doesn't remove anonymity. No one knows who the person on the other end of the email transaction is.

The Craigslist model itself is also a problem. Like "ads wanted" in newspapers, anyone can apply. And unlike newspaper ads, using Craigslist is completely free, so there’s virtually no barrier to entry.

Facebook faces similar challenges. Last year the company implemented OpenID on their site, making it possible for users to log in with a single identity. This is a big step in the right direction, but it doesn’t help verify that people are who they say they are.

At the heart of the anonymity problem is a concern for privacy. How do we force people to identify themselves without invading privacy? How do we protect privacy without neglecting accountability? Case in point: Many Facebook users recently boycotted the site because of its "overly complex" privacy controls and "like" button, which shares too much information with others.

The crux of the issue is that privacy is at odds with a site that's completely free to users. Facebook's business model, for example, requires information about users to survive financially. This is a huge problem that shows just how difficult it is to solve the identity and privacy issue.

A Few Possible Fixes

Amazon.com and eBay have a workaround for privacy: rating systems that allow users to determine how reliable a vendor or seller is. While there is some risk in using Amazon and eBay, it's far less risky than using Craigslist.

And Twitter has introduced "verified accounts", but it took a law suit to get them to do it. Of course, Twitter still deals with impersonation issues.

What Needs To Happen

Craigslist and sites like it need more accountability than just a verified email address. A person's identity needs to be verified. They don’t need to share that info with the world, but there needs to be some assurance.

What Will It Take?

There is very little accountability on the web today. I haven't yet heard of lawsuits brought against Craigslist after the home invasions, but I wonder: what will it take to fix the issue?

What measures would you take to counteract this problem? Can we solve the problem without invading innocent peoples' privacy? Should we just keep rolling the dice?

Leave a comment or if you tweet, follow us on Twitter to learn about new content, opinions and articles.

Frequently, design students or even folks looking to make a career shift to the Web approach me with the following question—"What's the one technology or software package I must learn to make a career out of designing on the Web?" I usually answer with three: HTML, CSS & JavaScript.

Oh no, not another post about the free love and the Open Web! Actually, let me tell you right off the bat that I'm not about to pull a Jean-Jacques Rousseau and get all anti-corporate on you.

Rousseau and The Web

Rousseau was a Genevan philosopher who earned his early ticket to fame by publishing the essay, Discours sur les sciences et les arts ("A Discourse on the Moral Effects of the Arts & Sciences"), in which he argued that arts and sciences were the root cause for all moral corruption.

According to Andrew Potter, the author of The Authenticity Hoax, Rousseau was a key figure in popularizing Romanticism—in other words, he was one of the first guys to put "the cool" into dissing all that is modern and consequently, laid the groundwork for the entire doomsday movie genre, which features films like "The Day After Tomorrow".

While it's certainly true that the Web tends to favor Rousseauian ways, the reality is that corporate blood, sweat and tears play a huge role in fueling its progress. Even the widespread adoption of open standards relies on the collective corporate nod.

And therein lies the simple reasoning for my answer.

Getting In The HTML5 Game

Google, Apple, Opera, Mozilla and Microsoft are all in the HTML5 game now. And you better believe—from Apple's heavy-handed stance on Flash to Google's recent Chrome App Store announcement—that the commitment goes deeper than just the browser. Many of them are betting their business models on positioning HTML, CSS and JavaScript as the front of the future Web.

Make no mistake; I'm not saying that HTML5 has won against Flash and Silverlight. Joshua's argument remains true today. The scenario you're building for will and should determine whether you pick standards, Silverlight, Flash, Java, choose-your-own-flavor or a combination of these.

But, from the perspective of a student investing in your future, I'd choose HTML, CSS and JavaScript in most cases. Between those three, you're going to learn the fundamentals of markup—allowing for migration to Flash's MXML or Silverlight's XAML. You'll also learn programming fundamentals using ECMAScript, which provides a great bridge to both Flash's ActionScript (which is actually based on ECMAScript) as well as, I daresay, C#.

I'm no expert, though. What do you think?

I love these posts. But the thing is, they always come from the agency or designer's side. Here, I intend to give you a little insight from the bad client's side. Because even though I think most of my agency partners find me pleasurable to work with, I sometimes see myself in these "bad clients" posts. I thought maybe I could help bring some perspective to the relationship.

Bad Client Behavior #1: They Want a Ton of Changes At the Last Minute

You, the designer, probably did exactly what you should do. You went through a business requirements gathering exercise, then talked to some actual users (yes, you were that lucky) and delivered an amazing set of wireframes (on time!). And what did you hear from your client? Crickets.

So you pressed ahead. You sent over the visual comp of the homepage expecting everything to go smoothly, when BAM! Suddenly Mr. Bad Client's feedback is flowing and he's telling you, "We're way off track here!" You get more and more frustrated as the delivery date looms heavily and you have to cancel your plans to see the Human Centipede movie this weekend.

Here's the problem: Most of the time, your bad clients are super busy and just can't wrap their heads around a wireframe. In fact, they probably scanned it for about two minutes and declared it good. The visual comp is the closest thing to a real site in their heads, so that's when they begin providing the feedback you needed earlier in the process.

The fix? I recommend getting to the visual comp phase as fast as possible, and allowing room in the schedule to deal with feedback on the design.

Bad Client Behavior #2: Ugh! They Want You to Build a Site Without Content!

Ok, I'll admit it. I'm guilty of this. I've actually read Kristina Halvorson's book, Content Strategy for the Web. But I've been in situations where I was the primary contact with the agency, and my stakeholders just didn't have the content ready before putting together the site. Sorry Kristina!

After I apologized profusely, the great agency I worked with did the next best thing: They asked what kind of content would go in the various content blocks, and tried to use existing content instead of lorem ipsum. It actually worked out pretty well.

I was able to give them some guidance and examples of content we used in the past. For example, Microsoft is notorious for using long product names, titles of posts and titles of session for our events. So the typical "Lorem Ipsum" filler text wouldn't work for us. We were able to plan for that.

I believe that you should have your content prior to building a site. But sometimes reality sets in, and you have to adjust to the situation. This agency has a special place in my heart for working with me vs. lecturing me.

Bad Client Behavior #3: They Want To See 10 Different Comps

As a client, I have to admit I like the idea of having choices. And yes, I will always want a Frankenstein of a web site that combines the best of the three you have shown me. Sorry!

But you have to remember: Most clients are not keeping up-to-date on the latest design trends. Helping give them exposure is a good thing.

Also, remember there are a ton of stakeholders (especially in a large company) that have a say-so in the design (or at least who sign my paycheck). So although you may have created one concept to rule them all, it may be more fruitful in the end to let me choose from different concepts.

Bad Client Behavior #4: They're Obsessed With "Above The Fold"

First of all, let me just say: I'm with you on this one. Also in this bucket is:

• Can you make the logo bigger?
• It just doesn't "pop" for me.
• Can you make it more clean?
• Hey! You left some empty space! Can we put a blog roll or a puppy picture there to fill it up?

These are critical moments for educating the client. Explain the "why". Another great agency I worked with was really good about explaining what was not on the page, and why. This made it very easy to go back to the stakeholders and explain why something was done (or NOT done).

Bad Client Behavior #5: They're Just Idiots

First, your clients are not idiots. They probably know their knowledge space just as much as you know yours.

That said, I do believe in firing really bad clients. If they don't pay up, if they don't respect you, if they burn you—by all means, you should never do anything for them again. I recommend Zeldman's 20 Signs You Don't Want That Web Design Project.

But take time when you are frustrated to look at it from the client's side.

Now Say You…

Do you agree? Do you disagree? Please let me know what you think in the comments below. I would be particularly interested in bad clients you have dealt with. Did you "fire" them? Did you have a Jedi-trick you used to get over a rough patch? Please tell me about it below!

Also, follow me or MIX Online on Twitter if you like.

I use the ASP.NET Chart Control in Azure in an ASP.NET page with both markup and code-behind. I do this because the chart control overlays hrefs and tooltips on the chart. (Check out this post for a little more about how I use the charts: http://visitmix.com/LabNotes/ASPNET-Charts-and-ASPNET-MVC--Controller-vs-View.)

Things work fine if I configure the chart control to generate images in memory. But, with more than one web role, things break. This is expected, as the documentation states:  “Do not use this [the memory storage] option in a server cluster or a multiple-process site.”

Somehow, I need to store the image to disk instead of memory, and then let the chart control know where to do it. Azure supports writing to disk through what they call a ‘local resource’, so I started using a LocalResource. But I kept hitting the following problem: I don't know the path to the LocalResource until runtime, but the Chart Control expects this information to be in the Web.Config file. And, if I changed the Web.Config at runtime, Azure got very unhappy.

I looked into using XDrive, which is similar to using a LocalResource, but the same problem ensued.  Each web role ended up mounting its own drive, and the drive letter became dynamic. I need to know it to pass that drive letter to Web.Config.

But I found a solution: I just implement my own instance of IChartStorageHandler and have it point to blob storage. Here’s what the code looks like:

public class ChartImageHandler : IChartStorageHandler { CloudStorageAccount account; CloudBlobClient client; CloudBlobContainer tweetContainer; public ChartImageHandler() { //wire up azure account = CloudStorageAccount.FromConfigurationSetting("DataConnectionString"); client = account.CreateCloudBlobClient(); tweetContainer = client.GetContainerReference(Urls.archiveContainer); } #region IChartStorageHandler Members public void Delete(string key) { CloudBlob image = tweetContainer.GetBlobReference(Urls.imageURL + key); image.Delete(); } public bool Exists(string key) { bool exists = true; WebClient webClient = new WebClient(); try { using (Stream stream = webClient.OpenRead(Urls.imageURL + key)) { } } catch (WebException) { exists = false; } return exists; } public byte[] Load(string key) { //sometimes load gets called before save is done CloudBlob image = tweetContainer.GetBlobReference(Urls.imageURL + key); byte[] imageArray; try { imageArray = image.DownloadByteArray(); } catch (Exception) { System.Threading.Thread.Sleep(1000); imageArray = image.DownloadByteArray(); } return imageArray; } public void Save(string key, byte[] data) { CloudBlob image = tweetContainer.GetBlobReference(Urls.imageURL + key); image.UploadByteArray(data); } #endregion }

Implementing this interface turns out to be pretty easy, especially because the Azure Storage Client CloudBlob class supports DownloadByteArray and UploadByteArray. Because the signature of the interface uses byte[] all over the place, this comes in real handy. Thanks Azure Storage Client!

You’ll note that I’m using this static Urls.imageURL for pointing to blob storage. This string is simply a path to a container in Azure.

The only tricky bit is that sometimes the Load method gets called before the Save function is finished. This is probably because of the latency with blob storage, which is going to be a little slower than writing to disk. I deal with this by adding try/catch. This simply puts a 1 second sleep delay in the call before trying to download the image. This solves the problem, and I’ve never seen the Save function take any longer than that.

I then update the web.config to specify that the ChartImageHandler key passed the fully qualified namespace and location of my handler, instead of the default handler.

And now I can use the ASP.NET Chart Control in Windows Azure with multiple web roles. Rockin.

Recently, I've been doing a lot of usability testing. The great part about it is, each new person I sit down with reveals a wealth of knowledge that would otherwise have gone untapped. The bad thing is, user testing takes time out of a busy production schedule, adding to an already stretched timeline and minimal budget.

But it’s worth it.

Why Do it?

Clients always want their deliverables last week. We do our best to deliver quickly— sometimes within ridiculous timelines. Because of this, we often don't have time to put the design, product or idea in front of potential customers. Unfortunately, this can lead to myopia , and sometimes to a displeased audience and client.

If the product you deliver doesn’t resonate with your client's audience, the displeasure frequently reflects back on *you*: You didn’t understand the “greatness” of the product. You didn’t convey that greatness to the customer. You just didn’t get the message, and the final product doesn't perform.

It’s rare for clients to admit that their idea needs improving. When presented with actual data, however, they have an empirical foundation on which they can judge their ideas. This is where usability testing comes in.

Usability testing doesn't take long, and it helps give your clients confidence in the choices you make. It's also an excellent opportunity to discover major issues and address them before releasing a product to the public, and before the client begins receiving feedback.

Doing It Inexpensively

User-testing is fairly easy to do, and inexpensive. The best way I’ve found is to just sit down with a user,watch him or her, and take notes. It's best to find someone who's unfamiliar with your product or service, because the goal is to make your product as easy to use for first-time users as it is for veterans.

Generally, all this testing requires is some time and patience. However, if you're testing a user-interface, screen-capturing your session can help you evaluate data after-the-fact. If possible, your video should be accompanied by audio, which you can usually capture via the software you’re using, or even a mobile device with a microphone. Watching and listening to the user can lead to great insights.

Doing It Wisely

I mention listening to the user because I’ve found this is where they generally express frustration. It’s hard to tell when a user is really having problems just by watching where they click or move their mouse.

Ask your test subjects to narrate their actions as much as possible. Encourage them to voice their intentions, expectations and frustrations . Have them tell you what they’re doing with the mouse: What they’re looking at and what they’re looking for. Stream-of-consciousness is what you want here.

And you, you shut up. Of course, give your subjects guidance and answer questions when they ask—but do your best to stay out of the way. If you over-direct, your test subjects won't be as likely to explore, and you might get a skewed result.

Also, keep testing informal. Meet in a comfortable spot. Relax. Talk a bit. Keeping it loose will enable people to relax, talk to you more freely and use the product like they would if you weren't looking over their shoulders.

Telling ‘em What To Do

Users seem to fall into one of two categories: those who like specific direction, and those who like to explore.

Both of these types of users are valuable, so it’s helpful to come to your session with two scenarios in mind: one that's very directed, in which you ask the user to perform a certain set of tasks, or a specific workflow. The other is to simply let the users “go crazy” and use the product how they see fit. This is an interesting way to see what a user would do coming to your product “blind,” and can provide valuable insights on making your product as accessible as possible.

Other Methods

Although the most valuable data is often collected after a design has been developed and is usable, doing usability testing prior to production is a good idea. It’s important to create solid wireframes, if not fully-realized composites, and go through the same process as discussed above.

As well, if you have a serious budget and giant project, it may be worthwhile to invest in large-scale, “professional” usability testing. I’ve been fortunate to have been part of many sessions where there was a group of us sitting in a little room, watching 8 or so users interact with various user-interfaces. The sessions were moderated by a professional usability tester, and were highly focused, video-recorded and analyzed later. Of course, the data from these sessions was very useful.

I think, however, the small shop or individual designer can do quite well and get similar results from doing usability testing on the cheap.

What About You?

What’s your experience with usability-testing? Do you find it useful? Are your clients reluctant to pay for it? What sort of tips could you give other small shops and freelancers to do it on the cheap? Leave a comment below, and be sure to follow us on Twitter at @Mixonline.

The film follows protagonist Ric O'Barry, a reformed dolphin activist from Flipper fame, who's joined forces with Louie Psyhoyo, leader of the Ocean Preservation Society, as they lead a team of vigilantes to covertly film the entire massacre in The Cove—the alleged scene of an annual dolphin massacre that's been deemed off-limits to all but a select group of officials and fisherman by the Japanese authorities.

Despite the one-sidedness, it's a great piece of filmmaking. The documentary positions the cause masterfully by gripping its audience with one of the seven basic plots.

Having said that, I'm completely astonished (but maybe I shouldn't be) at how this film has catapulted dolphins to the new status of the sacred cow. Don't get me wrong, I'm happy for Ric and dolphins worldwide.

But, let's be honest about the double standard here.

Flipper-flopping

I watched the film sitting next to two vegetarians (full disclosure—I like my filet mignon medium-well), one of whom couldn't see the difference between the dolphin massacre and industrial livestock farming in most developed nations, including the US. In essence, my vegetarian friend noted, both are systems in which humans kill animals for monetization and sustenance.

Ric argues that dolphins have Theory of Mind: They're self-aware, extremely intelligent and close to humans in the evolutionary chain, if you will. Hence, their captivity and murder are inexcusable.

As Sam Harris argues in his TED talk, Science Can Answer Moral Questions, our tendency to protect certain forms of life such as elephants, primates and now, dolphins, is rooted in our belief that they can experience a broad range of happiness and suffering. Conversely, we don't show such empathy for rocks, ants or, say, Cattle.

But this reasoning just doesn't sit well with many smart vegetarians I know. Some are infuriated that we bestow upon ourselves the right to make judgments about what species we may kill or hunt. Or that we presume to know which species are off-limits, since our understanding about their consciousness is incomprehensive.

Make no mistake—this isn't just an isolated opinion of a few vegetarians. It's even a core tenet of many religions—Ahimsa (non-violence), a core concept in Jainism, asserts that all living beings have a jiva (a soul) that is equal and thus, must be treated with equal respect. Practicing Jains go to extreme lengths to preserve life of all forms, from ants to root vegetables; many Jains have never tasted a potato because that's considered a sin!

Beliefs about what level of respect different forms of life deserve traverse a broad range of complex and subjective arguments.

The keyword here is subjective.

The Curse of Subjectivity

The word "subjective" is the arch nemesis of user experience. That's because "subjective" is usually an illusion when it comes to design.

Everyday, brilliant creations are adulterated with toxic accoutrements borne from the bowels of a belief that design is subjective. This distortion of reality seems to have become a core tenet in our organizational value system; it often leads to metaphorical Cove massacres of our products.

The challenge with design is that it's not as primitive as logic, but we tend to silently measure it logically. Design is a craft that approaches the complexity of natural language with nuances, contradictions, and emotions. Much like good writing, good design is something that doesn't just evolve, but is intelligently created by a craftsman. And it always grips us in an unexplainable way.

Because it is, often, unexplainable.

Explaining the Unexplainable

The unexplainable part of the practice design is founded on evolved problem-solving capacities that are often hidden from our consciousness. These are the same evolved capacities that allow ball players to catch seemingly impossible fly balls with awe-inspiring accuracy—a task that when solved mathematically relies on solving differential equations in four dimensions—which even the most sophisticated robots can't replicate.

The problem is that our reductionist methods always mislead us into mistaking the unexplainable for incorrect or illogical. The fact is that reductionist approaches generally do a piss-poor job of reverse-engineering the unexplainable when it comes to design—because they neglect the unquantifiable pieces of real equation.

We need to wrap our minds around a new definition of design that builds upon better evidence. Taking talent for granted, the definition lies somewhere between three key points: Edwards' assertion that artists have a trained ability to represent the world as audiences desire to see them, Gigerenzer's assertion that intuition and gut feelings are better suited for certain types of problem-solving and Gladwell's assertion that expertise in a trade is achieved at the magical 10,000 hour mark.

In simpler words, the practice of design occurs at the confluence of talent, training, intuition and experience. The rest is just background noise.

Thank God We're Not Dolphins

My point is that the practice of design isn't really subjective like the ethics of the dolphin trade or personal musical tastes, and while much of it is unexplainable, it's hardly illogical or incorrect.

But, I suppose the real moral is that designers should be thankful that they're not dolphins. I mean, can you imagine being a dolphin swimming up to humans, introducing yourself with a friendly high-pitched language that rivals the sophistication of their own, and impressing them with tasks they can't achieve, including mastering their own language, only to find yourself being held captive to perform circus tricks or simply being marginalized, massacred and consumed without consequence?

Wait a minute...

I once heard an automotive designer remark on the design quality of a competitor, saying, “They have a factory in Dusseldorf that chews up steel and craps out Porsches.” He was conveying his opinion that the cars were created in a blindly mechanical manner, with no soul or intelligence behind the design. Indirectly, he was communicating his belief that design should be intelligent and purposeful—not mechanical.

I think about this anecdote when I hear myself saying things like “This project has evolved over time.” And it’s not just me: I’ve noticed people conflating the word “evolution” with “progress” in at least two print magazine articles just this month. People do it all the time—if you pay attention over the next week, I bet you’ll notice at least a few occurrences.

So here’s the thing: evolution is exactly the opposite of intelligent design! Evolutionary theory was developed to describe how a “Blind Watchmaker”, acting purely through chance variation with no intelligence whatsoever, could produce the various species of plants and animals. So when I say that my project has “evolved over time”, it literally means, “We randomly threw crap at the wall, and this is what stuck.”

This might be true for some projects, but we normally want people to think we applied some personal intelligence to our work. Try to think of other examples where we commonly use the word “evolve” and none of them really hold up. Did AJAX “evolve”, for example? Not really. AJAX grew through widespread experimentation that was intelligent and purposeful, not through chance variation.

OK, so maybe I’m being extreme. Everyone knows you mean “progress” when you say “evolution”, right? But I can’t help thinking of a gigantic machine, blindly chewing up random piles of steel and crapping out Porsches, every time I catch myself using “evolve” this way.

Horses and Porsches

Let’s invert things, just for the sake of argument. If evolution through blind chance variation has no place in our design projects, why does it have a place in the speciation of plants and animals? Whether you believe the original animal species were created by a blind watchmaker or by an intelligent Designer, it should have little bearing on animals going forward.

Our ancestors were able to achieve amazing results with very little technology, turning wolves and foxes into the crazy variety of domesticated dog breeds we see today. Stand up close to a horse, and try to imagine the centuries of intelligent human design that led to the domestication of this powerful beast whose greatest happiness in life is to go exactly where his human masters want him to go.

All of this animal engineering stopped hundreds of years ago, before the industrial revolution. Why? Today, we have immensely better tools. Not only could we do a much better job at the selection process needed to create new breeds, we could even create entirely new species synthetically from DNA! We talk a lot of smack about how much smarter we are than our ancestors, but I haven’t seen anything as spectacular as a Clydesdale or Weimaraner come out of the post-Darwin age. Because of our ancestors' courage, we had horses to usher in the industrial age and Chihuahuas to provide comic relief in so many movies. What new innovations will we bequeath upon our descendants?

Doing What You Do

You probably aren't a position to be creating new species or breeds of animals. But you are in charge of your own creations. Don’t simply go through the motions and then passively expect “evolution” to carry the ball forward. Be purposeful and intelligent, and let your imagination go wild. Whether it’s more like a Clydesdale, Chimera, or the fabled Labradoodle, make your project a reflection of your best ideas.