I’ve had an issue with the training at security conferences for a while now, but I’ve been unable to articulate it. I think I have it now.

The issue is competing incentives.

1. Trainers are incentivized by the conference to get the highest ratings possible on the feedback forms after the class
2. Students of the classes are there to learn something that they can apply in their real job the following Monday
3. These two things are not usually compatible

In short, the type of content that gets trainers the highest ratings is not the content that provides lasting value for students.

It’s the Cheerleader Effect, where the geek finds the cheerleader far more attractive because she’s out of his league. Applied to training, it means that instructors are incentivized to use ninja flips and pyrotechnics for a few days in a way that conveys one thing:

I’m doing something elite! These guys are way smarter than me! I could never do this on my own!

And when it comes time to fill in their class evaluation sheets, they give the class and the instructor full marks.

Why? Because they felt like the content was way above them. Too good for them. Out of their league. Beyond them. And that’s what gets the top scores.

But could they go and execute what they learned? Can they take that knowledge to work the following week? Nope. Not a chance. The stuff was so far above them, or at least it was pitched as such, that they actually cannot use it in the real world.

All that they’re left with is a feeling that the class was amazing. Nothing more. And definitely nothing practical.

The class they really needed was one that is more approachable. Less hand-wavey, and more transparent. Understandable. Practical. Implementable. Something they can take to work and actually use, starting immediately.

But guess what? Those kinds of classes don’t score well on evaluations. They get comments like:

Was hoping for something more advanced. This is the type of stuff I could have thought up myself. Not something I needed to pay $2,500 to see.

Etc.

But in fact, that’s exactly what they should have paid $2,500 to see. Because it’s real. It’s transparent. And it can be put into practice.

The path forward

So here’s what we need to do as an industry with regard to training:

1. Conferences need to select courses that deliver actionable knowledge, not impressive knowledge. They need to rate the course based on how much value it provided instead of how cool it felt. Asking the students two months later if they’ve used anything out of the course would be a great start.
2. Students need to fill in their evaluations based not on how impressed they were with what the instructors could do, but rather what they anticipate they’ll be able to do because of the instructors. The distinction is critical.

Less Kung-fu, more Jujitsu.