Meet Silver Surfer aka 500 SuperMentor Dan Martell (@danmartell).  Dan is the Co-Founder of Flowtown, a social marketing application for businesses.  Self diagnosed as having intense ADHD, you can either find him hosting dinners, speaking about marketing or jumping off cliffs around the world.

Note: all data displayed in the mock-ups below are based on real data as of Feb 6th, 2011 except for personal user information.

Over the past 8 weeks, we’ve quietly launched a new app called Timely (http://timely.is).  It’s a simple way to add tweets to a queue and publish them at optimal times throughout the day. It’s been a fun project to build. We currently have over 2500 sign-ups, over 21% avg weekly active users, and 56 upgraded accounts.

The result? We now have 2 simple admin dashboards that improve our ability to learn faster and build a tighter feedback loop – especially in the very early days of building our app.

User Activity Stream

The User Activity Streams are a color-coded log output of actions taken by or done to our users in real time to help us look for meaningful patterns. The mock-up above is equivalent to what you’d see in our dashboard today.

The Activity Stream components are:

• Date/Time Stamp: When the action occurred
• Account: Link to the account owners email
• Action: Color coded action done to / or created by the app
• Details: Meta information to give context

Here’s how I use this view to guide product:

Patterns Highlight User Behavior
Colors help us recognize patterns. First, we just had a simple text log output. Adding color brought the stream to life. When I watch how our users interact with our application, I can easily see the both the positive and negative outcomes.

Get Personal with Your People
Because I can view the actions of account owners (and typically their email), the drill down is super easy. I’ve often discovered bugs and UX issues in our app just by watching users repeatedly take an action or watch 3 duplicate actions logged in a row. Regardless of the reason, I’ll just grab their email and shoot ‘em a quick note to learn more about what they’re looking to do and to see if I can help make it happen for them.   It’s the epitome of being proactive.

The downside? This deeply personalized approach won’t scale to thousands of daily active users – but it’s definitely a huge benefit while you’re just beginning and trying to learn more about your users, their needs and behaviors.

Updating Actions to Drive Better Features
Depending on our backlog of features, I’ll usually go in and change the activity stream to output the actions that will help me determine which, if any, changes are needed. If we deploy a new feature, I’ll usually add that activity to the log and remove another to reduce clutter. I try never to have more than 5 unique activities logged to keep my thinking clear.

Weekly Cohort Metrics

(These are real numbers)

One of the biggest challenges of building a web app is knowing if your changes actually make things better or worse for users. Most people use vanity metrics to “feel” good about their traction (traffic, unique visits, conversions, etc.) There’s nothing wrong with that per se, but at the end of the day, there are really only 2 measurements worth knowing:

• Did your prospects activate?
• Did your new users come back?

Using a simple reporting tool, you can quickly see if people “got it.” For us, activation means: created an account, added a Twitter profile (and tweeted) in their first week, and if that trend (the blue numbers) is improving or not.

Know Your Baseline
Having a baseline for your current metrics is just as important as measuring the impact of your changes.  Before we build a feature, we always add the metric to our cohorts table so we can see how we’ve done previously. This allows us to learn (baseline) and improve from there. Since we write all of our stories in a“Feature X will move Y metric” format, we need to get our baseline for Y. Having a simple framework in place makes it easy to review a feature’s relevance and ultimate impact.

Moving Forward with Confidence
If you don’t use cohort measurements, it’s really tough to know if you’re improving, especially if you’re deploying new production updates multiple times through the day. It makes it really tough to know what change had which effect. That’s why we use metrics in weekly increments, If we screw something up, we can quickly reverse the changes and get things fixed without causing any major issues.

Beyond Metrics: Vision, Creativity and Strong Opinions Wanted
Owning a hammer doesn’t make you a carpenter. You can’t simply depend on “Tools” as a guarantee you’ll get things right. But having the right tools – and knowing how, why and when to use them – can help you  determine what needs to be built AND let you know if you’re moving in the right direction.

Great product requires a vision before you build. You need to let your creativity flow and try different implementations. Lastly, you also need to when to “fold ‘em” and not build at all.  In fact, saying no may actually be the most important skill you can have, so don’t be afraid to use it when the need arises.