Rob Conery Ramblings

I've  been tagged by Scott Hanselman, who's asking:

Did you grow up a geek? Let's see what you looked like as a young burgeoning nerd or nerdette.

I was online the other night, running my new dual locks through Ashenvale (I'm a new dual-boxer and absolutely love it. I keep telling Brad Wilson that Warhammer's fail sauce and that running twin toons around is a sheer blast! I've tried two mages, and Priest/Mage combo which I took to 60 in 3 weeks very casually... seriously I play maybe 8 hours a week, if that... anyway it rocks! Faceroll FTW!) when Communicator goes "PING" and it's Scott:

found pictures of me from 1985 ... when I was like 11 ... HUGE dork ... clearly, they need to be on the internet to embarass me... your turn ... go get the shoebox

I just happen to have this shoebox in my closet as my mom just moved out of the house that I grew up in. Literally - I lived there from day 1 through the age of 18, and she decided just last spring to move to Oregon and into a small place. I got to help move her out and in the (very cliche) dusty attic I found a TON of pictures of me.

My mom's a pack-rat. She kept everything, including trophy's of mine when I was a kid (the token T-ball kind of trophies).

Anyway - the picture here is of my family in 1974. My dad is taking the picture, and from the left is my mom, my sister Teri, my sister Candy, my bearded-genius brother John, and my punk rebel older brother Larry.

I'm in the blue pin-striped shirt, and that is the house I grew up in that my mom just sold (which is kinda sad).

I wrote before about how I got started in programming and really, I sort of fell backwards into it. My brother (the bearded guy) tried to corrupt me digitally around the time that this picture was taken, but I was more into playing ice hockey and baseball.

Crap. That makes my mom a "Hockey Mom". Lord help me.

Now this is going to sound very awkward and probably a bit vain, but I was kind of a cute kid. I tried to find a real dorky picture of me but I think my parents only kept the cute ones (I know this sounds horrifically vain- but it's the truth... what can I say... I was cute):

I did manage to find the revolting "kid in his first tux wondering how to be cool" picture though - you know where you think you'll knock 'em dead by trying to look like James Bond. This picture was taken in 1983 when I was 14, right before the Junior Homecoming dance, and makes my skin crawl:

Look at those watery eyes, that rail-like physique with a stare that could melt cheese. A "cool drink of water" as my gramma would say - if your definition of "cool" is having pizza face and so much mousse (or was it Dep?) in your hair that it almost snaps...

Believe it or not I did have a date that night...

Ahh the fun - thanks for the memory jog Mr. Hanselman! And now, these people need to be publicly humiliated...

Eric Kemp, Jon Galloway, Kevin Dente (you gotta blog sucka!), Steven Harman, and Jeremy Miller.

Yes, I know this is becoming a habit - but the last bits are tough! This one's worth waiting for though, I think :). I'm tackling a number of things with this episode:

• Personalization - how to effectively query for "You may also want"
• PayPal - I'm plugging in everyone's favorite payment engine!
• Reorganizing a bit - I'm following along with my suggestions to the core template the other day - I find it's a good deal easier to get around the app!

A word about PayPal
Integrating PayPal is pretty simple, but the API I've chosen is pretty hard to test so I'm stepping outside the TDD bits unfortunately, but I'm doing my best to make sure I create some reusable elements that you can take a run with.

I'm trying to accomodate a "least-touch" scenario - meaning I want to enable a transaction where you know nothing about the user, and they don't want you to. There are some issue with this - such as how much information can you ask them up front (such as where to ship to so we can calculate shipping costs?) and how much do you retain. PayPal can actually calculate all of this for you - even tax amounts!

Lots of cool stuff with PayPal Standard - and I'm capturing every step (and misstep) along the way. I should have this one ready by Tuesday.

I've been toying around with reorganizing the default MVC template and thought I would share it with people. I like the way it's come together.

Reorganized
When I was working heavily with Rails I came to really enjoy the way the applications are structured. It seemed... clean in the way all parts of the application had an especially relevant bucket to live in. I especially liked the way the application bits (logic, data access, controllers and views...) were kept nicely in the "app" folder.

I'll be honest and say I'd like to see our default MVC template change a little bit. Phil's team and I (I'm not on Phil's team - I just annoy them) discuss this stuff a lot - and it's been brought up a few times that perhaps I should make a template of my own to show what I'm thinking. And I think that's a great idea...

So I reorganized it some. Here's what I'd love to have my site look like:

You'll notice that I'm keeping the "App" folder approach, and sticking the Controllers in there as well as the Views. In addition I have a place for DataAccess and Services (business logic) as well as Logs and a folder for the Dependencies.

I've also partitioned a folder called "Public" - this is where (I think, at least) the support files should go, as well as any static files (like HTML, etc). I've also added CSS and Scripts folders too.

Yes it looks a lot like a Rails site, but if it works - why change it?

Basic Hookups
But why stop at folder organization? I did a few other things as well:

• Every site needs logging - so I've plugged NLog
  
• I like the implementation of the AccountController - but I made it a bit more generic and have it working now with an IAuthenticationService which uses both ASP.NET Membership as well as OpenID. And yes I've included those implementations in the Services folder.
  
• Every site needs to email things - so I've created an IMailerService which implements SmtpMailerService as well as my favorite TestMailerService (which stores sent mail in a List<> rather than try to send it off to an SMTP server).
  
• I expanded the Site.master a little bit to include a ContentPlaceholder in the Head of the document (for script registry).
  
• I added jQuery (even though I know this is coming!) as well as jQuery calendar.
  
• I also setup StructureMap to be the default ControllerFactory. DI comes wired "out of the box" as you can see by the "Bootstrapper.cs" file that's in the root of the  App folder. I've also taken the liberty of configuring a few things already for you - such as the logger and mailer services.

A ViewEngine of My Very Own
The default ViewEngine that comes with ASP.NET MVC (the WebFormViewEngine) looks to the root of the application for the Views folder, and I needed to change that if I wanted to shove everything into the App folder -  so I had to create my own :). Thanks to Phil and team for making this rockin easy - all I had to do was to overwrite the location formats and I was good to go:

public class SubSonicViewEngine : WebFormViewEngine
{

    public SubSonicViewEngine()
    {
        MasterLocationFormats = new[] { 
                "~/App/Views/{1}/{0}.master", 
                "~/App/Views/Shared/{0}.master" 
            };
        ViewLocationFormats = new[] { 
                "~/App/Views/{1}/{0}.aspx", 
                "~/App/Views/{1}/{0}.ascx", 
                "~/App/Views/Shared/{0}.aspx", 
                "~/App/Views/Shared/{0}.ascx" 
            };
        PartialViewLocationFormats = ViewLocationFormats;
    }

}

To implement this ViewEngine I had to add one line of code to Application_Start in the Global.asax:

 ViewEngines.Engines.Add(new SubSonicViewEngine());

... And that's all there was to it!

Open ID Support
I've also added Open ID support to the Login page, and I've left a place there for you to put your own ID Selector. To implement, you simply need to head over to IDSelector.com and fill in 3 bits of information. They give you some code, you come back and put it in place et voila! You have OpenID:

I've included the OpenIdAuthenticationService class that I've been using with the Storefront as well. This class handles the redirect to OpenID (and also the callback) - you simply need to implement the code for integrating the user and their URL.

To get this to work, I have a new Action in the AccountController which I call directly:

        public ActionResult OpenIDLogin()
        {
            bool isValid = false;
            string claimedUrl = Request["openid.claimed_id"];

            if (!String.IsNullOrEmpty(claimedUrl))
            {
                Uri openIDUri = new Uri(claimedUrl);
                var svc = new OpenIDAuthenticationService();
                isValid = svc.IsValidLogin(openIDUri);
                if (isValid)
                {
                    string returnUrl = Request.Form["ReturnUrl"];
                    FormsAuthentication.SetAuthCookie(claimedUrl, false);

                    if (!String.IsNullOrEmpty(returnUrl))
                    {
                        return Redirect(returnUrl);
                    }
                    else
                    {
                        return RedirectToAction("Index", "Home");
                    }            
                }
            }
            return View("Login");
        }

The line "Request["openid.claimed_id"] might look weird - and you might be wondering why I'm 1) using "Request" instead of Request.Form and 2) why I've named it something like that.

The answer is that this action gets called twice - once by the login form (via POST when the user clicks "login"), and once by the OpenID provider (via GET on the redirect).  The Open ID provider will return a querystring which has the "openid.claimed_id" key in it, and I want to be able to read that in - so I need to access the entire Request name/value collection.

The Service class uses DotNetOpenID to redirect off to the provider, as well as receive the callback:

        public bool IsValidLogin(Uri serviceUri)
        {
            bool result = false;
            var openid = new OpenIdRelyingParty();
            if (openid.Response == null)
            {
                // Stage 2: user submitting Identifier
                openid.CreateRequest(serviceUri.AbsoluteUri).RedirectToProvider();
            }
            else
            {
                result = openid.Response.Status == AuthenticationStatus.Authenticated;
                if(result)
                {
                    //synch the users
                }
            }
            return result;
        }

... and there you have it.

And Finally - What This Isn't
I have bigger and badder plans for SubSonic than just this (I'm working on that stuff now) so even though this has the SubSonic name to it - please don't think it's what I've been working  on for the last 9 months :). This is the result of me geeking out over the weekend and wondering if others would find this useful.

What I'd really like is your feedback with respect to the things I've done with the template - specifically:

• Organization
• Setup
• Included Technology
• What I've Left Out

The main reason is that if you like what you see here I can raise the ideas to the team and say "look what comments I've received on my blog". Or, conversely, if this is too complicated I can stop bugging them :).

Download the Template Here (uses Preview 5). Stick the zip file into your C:\Users\[USERNAME]\Documents\Visual Studio 2008\Templates\ProjectTemplates folder and restart Visual Studio. The template will show up under "My Templates":

Looking forward to your thoughts!

As I'm sure you know by now, jQuery is going to ship with Visual Studio in the future and will also be distributed as  part of the ASP.NET MVC installation.

This, friends, is progress.

You can read all of the details on ScottGu's blog (as well as a nice tutorial on Hanselman's) about what jQuery is (if you don't know) and also why you should care. Not only does this represent a significant leap in productivity for you (intellisense for jQuery! Hellloooo!), it also affirms, yet again, a shift in thinking at Microsoft.

Many people have doubted Microsoft's intentions in the past - some have claimed that they're gobbling up Open Source projects and burying them (like, ahem, SubSonic). Hopefully these opinions will start to change. And if you're wondering why/how all this is happening, especially right now, you have one person to thank for it.

I think this is such a cool thing that I'm going to show Scott a little love for it later today. When I'm done with that I might by Resig (the jQuery guy) something as well. He's one of my heroes :).

Wow this one took me a while. As anyone knows the last few miles of any type of journey are the longest and usually most difficult. They can also be the best :). I won't say this is the best screencast of the bunch - there's a lot in here and I had to pare it down some - but I like what's in here. I'm getting very, very close to wrapping this and it's making me happy :).

The first part is devoted to expanding the Order management pipeline bits - Charging, Shipping, and Cancellation. I expand on Windows Workflow and also create a separate code-based pipeline for those who don't want to use WWF.

The second part is all about plugging in User Tracking and what it means to plant data that you can mine later. Looking forward to your feedback.

I need some serious help with the UI and I'd like to formally ask some of you CSS wonks/visual types for help. I need some CSS love with respect to sprucing up the template as well as improving the structure. I'd love anyone's ideas and help.

You can watch it here.

Download the code from here.

It's annual review time at Microsoft and I've been a tad distracted with that and some other things that I hope to show off in the next 2 weeks. I did find time to record about 40 minutes of stuff (usable stuff) over the last 2 weeks and I've been cutting/splicing it together over the last few days.

I have one last section I want to add to "complete the story" and I'll be recording that in the next 45 minutes. After that it's a matter of stitching/editing etc so I should have something ready by tomorrow at the latest.

I just looked at the date of the last recording and it's been a while - I didn't mean to let time slip like this but... well it happens!

I've refactored to Preview 5 and posted the source up to CodePlex and now I'm doing some work on the Order Fulfillment piece. I'm recording all of it but there are some ... *things* ... that are inspiring me at the moment that are related to what I'm doing here (building an application). I've decided that chasing and refining those could help me to finish this faster.

So hang in there. It'll be worth it! I should have another episode ready by this Friday.

Had a funny conversation with a colleague the other day where we talked about coding, project process, platform, etc. and we started talking about "Our Most Successful Project" and what it was and why. Interesting where it took us...

What Was Your Most Successful Project?
The first question that comes to mind, of course, is "what do you call success?" and it's a reasonable one. It's also not easily answered - what are your metrics? Are they:

• Money - did you make some? A lot? Does this mean you wrote good software?
• Time - did it come together like butta and sing its way to the finish line?
• Completeness - did you nail every requirement? Was it bug free with 100% code-coverage?
• Client-happiness - did your client get a promotion? Gush about you and your company?

Perhaps it's a mix of all four. And then again perhaps it's none of these - maybe it's just a "gut feeling" you have that you "nailed it".

I think about these things a lot - especially since I'm putting my bits on the block with the MVC Storefront and it better be good. But WTF is *good* anyway? Will it be *good* if the Alt.NET guys talk to me still? Is it *good* if Ayende says it is?

What if I can go to bed at night and feel like I didn't sell my soul to the Great Soul-Sucker, and people use it?

Meh. You can freak yourself out on this to no end. So rather than look forward and worry - I think it's better to look backward and reflect and ask yourself - What Was the Most Successful Project You Worked On?

Note that i'm not asking about code here - I'm asking about the SUCCESS of the project. Qualify it as you will - based on code, technology, client happiness, money - preferably a sweet mix of all 4!

I had a moment to think about this tonight as I pulled my kids around the block in their Red Wagon - what was MY most "successful" project? I'd say it has to be SubSonic.

My Most Successful Project
People use it, they like it, and it seems to be very useful - which is what I intended. This isn't supposed to be an ego-stroke post, but it's something I want to look back on so I can see what's important as I push ahead with other projects.

SubSonic isn't technically marvelous and exciting, by any stretch. The docs completely suck, the support is not where it should be (though I really, really try), and our test coverage is ... well I spose I shouldn't even use that word with respect to our unit tests :).

I didn't use BDD or TDD, I didn't follow BDUF or make Eric join me in weekly scrums. I didn't do any kind of focus group or marketing either - I just had an idea and it lit me up.

So I spose what I'm saying is that it was accidental :). It was inspired - to be sure, but really it was (as they always say) a matter of right place, right time, and right idea and I made something that people found useful.

Which is market-speak and has nothing to do with technical approaches or project processes. To be honest I'm OK with that - I'm working on 3.0 right now and so far have implemented what I think are pretty decent approaches (like I did with 1.0 2 years ago, and I'm sure I'll do again with 4.0 two years from now). The point I can't get away from is that, technically speaking, it's more about the itch that SubSonic scratches than the fluffy stuff that makes it up.

There's a messaging service out there that shows cats in servers and whales supported by birds (James and the Giant Fail Whale) but I'll be damned if I try to make a comparison there... not good...

I'm interested to hear what you have to say about this - how do you gauge the success of something you've worked on or created?

If you were hired to create some software to count votes in an election, how would you do it? What considerations would you have and how would you implement them. What would be your security considerations?

Think for a moment before reading on, and we'll compare your thoughts with what's actually out there. The results may surprise you.

Disclaimer: This is my personal blog and my opinions are my own and not necessarily that of my employer.

No, This Is Not All About Touch Screens
There has been some major upset in the last few years about touch-screen voting systems and how easy they are to hack - but I don't want you to focus on that, for now. The interesting thing is when you bring up "voting software" most people think these systems are what you're talking about.

In this case I'm more interested in how you would implement vote-counting software - the thing used to count the votes when cards are passed through a reader and tallied. Take 10 minutes or so and specifically think about:

• Voting audit trails
• Counting accuracy
• Security
• Tampering prevention

You keep on musing while we take a break to describe the election process and how votes are counted today...

How Your Vote Is Counted
Every county has the jurisdiction to implement an election as it sees fit, under the guidelines of the Federal Elections Commission (FEC). This means that every county in the US can buy and use "certified" systems from companies such as Diebold, ES&S, and Sequoia.

One of the biggest sellers out there (the state of Maryland paid $50 Million US for this one) is the Diebold's Accu-vote. It consists of a battery of optical readers (one for each polling place) and memory cards for storing election results. It also comes with a license for GEMS, their vote-tallying and reporting software; one license per county for use by the Supervisor of Elections.

On the day that you vote, your ballot gets put into an officious looking black box with the other ballots. At the end of the day your ballot is pushed through one of these readers, and your vote is stored on one of these memory cards:

Once all of the ballots from a polling place are collected and read, a tape is printed by the optical reader that has the results on it, including a full count of the ballots read in. The election workers from that polling place sign the ticket and off it goes to the Department of Elections.

The next step in the voting chain is that the Election Supervisor (or one their appointees) takes the memory cards from each polling place and methodically plugs them into a computer running GEMS, the vote-counting software. GEMS reads the information from the cards and once all the cards are read, a final report is printed out and the Supervisor certifies the results and the election.

Seems simple enough right? Have an idea how you might implement this system?

The Diebold System
In 2003, Bev Harris (the then-housewife and now-founder of BlackBoxVoting.org) wanted to know more about the election software that was being used in her home town near Seattle, WA. She got on the internet and ran Google search after Google search until suddenly...

... when I found that Diebold Election Systems had been storing 40,000 of its files on an open web site, an obscure site, never revealed to public interest groups, but generally known among election industry insiders, and available to any hacker with a laptop, I looked at the files. Having a so-called security-conscious voting machine manufacturer store sensitive files on an unprotected public web site, allowing anonymous access, was bad enough, but when I saw what was in the files my hair turned gray. Really. It did.

The contents of these files amounted to a virtual handbook for vote-tampering: They contained diagrams of remote communications setups, passwords, encryption keys, source code, user manuals, testing protocols, and simulators, as well as files loaded with votes and voting machine software

Turns out that Diebold kept their CVS system up on a public FTP site, with no security. Oops.

She downloaded every file she could find, which included requirements, diagrams, code, and binary files. Of particular interest to her was GEMS - the software that tallies the votes for the county.

It's a windows application written in C++, and the data is kept in an unprotected Acces database. Well that's not entirely true - the Access database is sort of protected - the GEMS application uses a password-protected DSN, with a user name of ADMIN:

Of course if you are even a casual computer user you can just open up the Access database directly, bypassing the DSN security, and have at the results:

If you're wondering how easy it would be to change the results of an election (seriously) - just double-click on the SumCandidateCounter table you see there, and you can change the numbers around:

Howard Dean is shown how this works in this video:

Is this the system that you would have written? Something that gives just about anyone with motive to change an entire election result?

OK, This Was 2004. That Can't Still Be Happening!
You might think that after all this negative exposure, Diebold (and others) would make sure that the system is functioning. Well you would at least hope so. Turns out that things seem to be getting worse:

• The State of California has been through the ringer with Diebold, and just this last year decertified Diebold (and many other software voting machines) completely (for the second time), and wasn't short on reasons why.
• In 2006, and Diebold lost the source code again. Turns out is was stolen in Maryland - the state that paid $50 Million US for the Access database.
• The State of Ohio just invested $20 Million into Diebold touch-screen systems (same ones CA banned). Turns out that a software error causes votes to be dropped. Oops. Problem can't be fixed before November 4 2008. Uh Oh. Turns out that the software doesn't like Anti-virus systems... which you'd think would be OK since a virus could comprimise the entire system. Sigh.

Our presidential election is coming up - how is your county voting? Is it using a touch-screen system? One of the ones that California has categorically thrown out, from every vendor? Or one of the buggy ones in Ohio that has a problem with Anti-virus software?

Or will your vote be stored in an Access database on an election-worker's laptop?

A More Elegant Hack
I know what you're thinking. Buggy systems with security flaws sound like alarmist-speak and opening up and changing an Access database might be a bit hard; after all you'd need to be sitting there on the computer right?

Unless the election workers aren't very protective of their hardware (from the New York Times)...

Ohio is an election battleground state with perennial problems at the polls. So what have election officials in some precincts of the state been doing to keep their voting machines safe from tampering?

Taking the machines home with them and stashing them in their garages in the days before a big election.

If it sounds like something pulled straight out of an episode of Saturday Night Live, or Borat for that matter, it’s not. The practice has become so widespread that it even has a nickname, “sleepovers.”

Meh. That's too easy and it requires some social engineering and sounds alarmist (again!). I wonder if there's a better way to hack into these machines...

Harri's Hack
In 2005 the Leon County (Florida) Supervisor of Elections Ion Sancho contacted BlackBoxVoting to scrutinize their voting systems to be sure they were secure. BBV brought along Harri Hursti, a Finnish computer security expert. Specifically in question was the memory card which carries the vote tallies from the local ballot-reading (at the polling places) machines to the GEMS counter system.

The software vendors have maintained (and still do) that there is no way to tamper with this - the card would know it has been tampered with and scramble the results. Harri, however, found differently.

Harri was given one of these cards by Ion Sancho and asked if there was any way that he could hack it. Harri took the card back to his hotel room (with a memory card reader he bought from a local computer store) and found out that the information was stored in clear text and was easily manipulated. He also found that it was possible to use negative numbers in the vote tallies, and that gave him an idea... no code needed to be written at all - the simplest of all hacks.

This is known as "Harri's Hack":

The Hursti memory card hack performed in Leon County on Dec. 13, 2005 is a variation on stuffing the ballot box prior to any votes being cast. Hursti had pre-loaded the memory card giving one candidate 5 positive votes and one candidate 5 negative votes to create a "zero report." This keeps the machine accurate in votes cast compared to number of voters.

When the scanning machines start up they make sure that the memory cards have a "zero-tally" by essentially adding the numbers together. Harri was able to get around pretty easily.

A mock election was held with the Supervisor of Elections watching, and a total of 8 votes were cast. The results were pretty clear - Harri's "stuffing" attack skewed the election results entirely, and no one had any way of knowing it. It took Harri only 30 minutes to hack the card - and the vote was completely compromised.

If you're thinking that getting access to a card or card-reader might be hard, well you can go right out and buy some from Diebold.

The Keys To The Kingdom
In 2006, Princeton University decided to test the Diebold Accu-vote touch-screen machine. The results are less-than flattering:

One of the main rebuttals to this video is that, under normal situations, you can't access the hardware of a voting machine (even though they show how easy it is to pick the lock). This is true - the machines are indeed locked.

But then again you can also buy some keys from Diebold if you needed to...

There Is So, So Much More
As a geek I am both agitated and intrigued by what I'm finding as I keep looking. As a voter I'm mortified. It's important, I think, to have some alternatives when you write a post like this so we can "keep it positive". And here they are:

• Get informed about your county and their election systems. The elections division is usually quite open and will answer most questions as it's a matter of public record. Find out what system they're using and make sure it's certified. If it's not, write a letter. You can also make a public records request of all the election results (including those signed poll tapes) if you're interested in that.
• Read up on tips from BlackBoxVoting.org. They offer a great free kit for the things you can do if you're interested.
• Learn more about our election process. I've only touched on the very, very tip of this issue and there are more things to be said. The documentary "Hacking Democracy" touches on a lot of these things, and you can watch it online for free. Ballots in the trash, meaningless testing results, donations to voting software companies from lobbyist groups, the State of Ohio allegedly tampering it's own election recount... It's an hour and 20 minutes - but you need to know what's going on as a voter.
• Sound off! Most people are completely unaware of the election process and also the things that are being done "incorrectly". It's time that we know more about what's happening.

If you do anything at all, spend an hour and watch this HBO documentary - it's at the core of all that I've written and what motivated me to spend the last 2 weeks researching this post. I'm trying to do my part by reaching out to you as a voter - not to sling mud but to suggest that there are things you need to know more about. It's far too secret right now.

I'm currently updating the site to work with Preview 5 and there have been some significant changes - as well as opportunities to upgrade the way I've been doing some things. I'm going to take a few days here to make sure all things are working correctly.

Hopefully I'll be able to pop out a new episode on Friday...

I've been working a lot with LINQ over the last few months, trying to approach it sanely with respect to SubSonic. It's not that it's terribly difficult to understand - it can simply be overwhelming at times with how much is in there. And whenever you think you know it, you realize that not only were you wrong, you weren't even in the same universe.

I'm a visual person, and sometimes wrapping my head around a technology involves my "seeing it" in a certain way. Either by a graphic or just a simple outline. LINQ doesn't offer out of the box and I've been writing my expressions out to the console to "see what's hiding in there". No longer - I found a really cool tool to use - and it's already on my machine as part of Visual Studio.

Let's Get Visual
I was trolling around Google today, thinking that someone MUST have written a routine to barf out the Expression Tree generated by a LINQ statement. If you don't know what I'm talking about, consider this LINQ statement:

    var x = from p in query
            where p.ProductID == 1 && p.CategoryID == 6
            select p;

This statement in particular deals with a database - but let's forget that for now and just imagine that it's a query for "something". What this does is generate a thing called an "Expression Tree" - a recursive tree of objects called Expressions that describe the statement. You can see this if you access the Expression of the statement:

    System.Linq.Expressions.Expression expression = x.Expression;

In this statement, "x.Expression" is the entire tree, boxed into the base class of "Expression". Yes, Expressions can (and are expected to) contain other Expressions and branch out all over the place, creating a tree (more on this in a moment).

Hug (and Visit) a Tree
Working with an Expression tree isn't easy, and sometimes I feel like a Geek With Very Little Brain. I can think recursively, if I have the right combination of caffeine and jelly beans, but then I get a major geek hangover and have to go mow the lawn or something. This is tough stuff - but there are a lot of examples out there on how to do it appropriately.

I've written about this before but when it comes to using LINQ and its constructs you really can't write enough. The pattern you want to use when you're dealing with a tree structure is called the Visitor Pattern (from Wikipedia):

In object-oriented programming and software engineering, the visitor design pattern is a way of separating an algorithm from an object structure upon which it operates. A practical result of this separation is the ability to add new operations to existing object structures without modifying those structures. Thus, using the visitor pattern helps conformance with the open/closed principle.

In essence, the visitor allows one to add new virtual functions to a family of classes without modifying the classes themselves; instead, one creates a visitor class that implements all of the appropriate specializations of the virtual function. The visitor takes the instance reference as input, and implements the goal through double dispatch.

If you've never dealt with this kind of pattern before - well neither had I until a year or so ago. The pattern seems simple enough, however applying it to a recursive system isn't the easiest thing. Thankfully, there is an MSDN article on how to do this with Expression Trees and has been something I've reread about 100 times. If you want some more, Matt Warren (another blog I read constantly) shows you how to do this in the scope of a SQL call. It's an amazing read - but give yourself time on this one; Matt's one of "those guys" who's brain is in a different dimension and thankfully he's a very good writer. But it's not easy stuff.

I Think That I Shall Never See...
I'm tits-deep in implementing "this stuff" for SubSonic right now, and routinely I run into issues where an Expression Tree gets generated for a given query, and I need to decide what type the Expression is, how/where it was generated, and what I need to do with it. For this I've used the debugger and immediate window constantly and I've wished I could have this all laid out for me visually.

And thankfully Granville Barnett pointed the way. The "Expression Tree Visualizer" is so very cool that I thought it would be a good idea to pass it along to you:

Again, if you have VS 2008, you already have this on your machine - it's just not hooked up! To hook it up, you need to do a couple of things:

1. Pull the code for this thing out of a zip file in your VS install directory
2. Compile the project
3. Put the DLL into your VS Visualizers directory

It takes just 5 minutes, so let's get started!

First - find your Samples directory for Visual Studio. Mine was located at

There's a lot in here - but the file I'm interested in is the CSharpSamples.zip. If you open that you'll see two directories, one called "Linq Samples". If you open that you'll find what we're looking for! (They didn't make this easy did they):

Almost there - now just drag this to your Desktop and open it. It's a VS 2008 solution and you need to open the project and build it. We're after a single DLL here, and it's the one in the ExpressionTreeVisualizer project. Once it's built, grab the ExpressionTreeVisualizer.dll file from the /bin/Debug, and we're almost done.

Next, find your Visualizers folder for Visual Studio. This should be (by default) in your Docs and Settings directory:

You might not have a Visualizers folder here - if you don't just create it. Then drop in the ExpressionTreeVisualizer.dll! And that's it!

Using the Visualizer works like other visualizers (the data visualizer, XML, and text) - just create a stop when debugging, then hover over the Expression variable you want to look at:

Clicking on the Expression Tree Visualizer will bring it up, and you can view your Expression Tree:

Read Scott Watermasyk's post today on Disqus (spose to read like "discuss") and it sounds very intruiging. I've always hated the way comments are managed in ANY blog platform (although Graffiti does it pretty well). I really dig the voting feature and you can see it in action at the bottom of this post.

Comments are critical to what I'm doing right now with the Storefront, and a good episode can easily bring in 30 or more. I really like the features that allow others to "vote up" comments, and I also like the bi-directional email aspect so you know what's going on. There's a LOT to like here.

However it's a web service, and web services go down from time to time. But, no harm in giving it a try so let's see what happens! Gimme a comment!

In this episode I plug in logging - something that probably should have been implemented sooner.

Not much else to add - it's a pretty simple screencast but I felt it was important enough to have it be it's own deal. If you see anything you don't like here - as always, chime right in. All I ask is that you give me some detail!

Download it here.

(Double click the screen below for full-screen)

I've spent the last few days adding in some functionality that I think would better display what it is I want to do with WWF and the Storefront.

Not much more to say - this is a 10 minute followup wherein I add some more services and buff out the Order submission process, and wrap it all with unit tests (that Ayende will hopefully like).

With revision 482 I added in the ability to work with string values and "LIKE" queries a little more intuitively.

On a recent post, a commenter (liviu) left this comment:

"...Because Subsonic generated code i found it unexplainable why i cannot write something like:

Select<Product>().Where( Schema.Product.ProductName.StartsWith("a"))

or

Select<Product>().Where(Schema.Product.ProductType == "Cool Engine")."

The good news here is that in the second case it's a matter of me writing more complete samples. You can indeed do this query this way:

IList<Product> products=new Select().From<Product>()
   .Where(Product.ProductTypeColum).IsEqualTo("Cool Engine")
   .ExecuteTypedCollection<Product>();

However liviu is correct - you can't use "StartsWith" or "EndsWith" - until now. I just checked in changeset 481 which allows you to use StartsWith() and EndsWith() as well as ContainsString() - here's my Unit Tests:

        [Test]
        public void Select_Using_StartsWith_C_ShouldReturn_9_Records() {


            int records = new Select().From<Product>()
                .Where(Northwind.Product.ProductNameColumn).StartsWith("c")
                .GetRecordCount();
            Assert.AreEqual(9, records);
        }

        [Test]
        public void Select_Using_EndsWith_S_ShouldReturn_9_Records() {


            int records = new Select().From<Product>()
                .Where(Northwind.Product.ProductNameColumn)
                .EndsWith("s").GetRecordCount();
            Assert.AreEqual(9, records);
        }


        [Test]
        public void Select_Using_Contains_Ch_ShouldReturn_14_Records() {


            int records = new Select().From<Product>()
                .Where(Northwind.Product.ProductNameColumn)
                .ContainsString("ch").GetRecordCount();

            Assert.AreEqual(14, records);
        }

Many thanks to liviu for the suggestions!