How do you know your blind-playtesters got it right?

For me, I ask some questions to probe their experience such as

"What did you focus on"
"Did you gather any resources"
"Did you try any quests"

and as long as their answers "make sense" and they had fun and enjoyed the game.

Then I don't mind if they made mistakes here or there. Games need to work when players play them wrong, as more players will make mistakes with rules than get them right (unless it's a very simple game).

Of course, you still want to get their negative feedback and try and work out any problems. But I wouldn't start blind testing until I was confident the game was pretty solid.

It might be worth going back to normal playtests and observing?

Or yes, ask them to record it!

My brother in law kickstarted his game, Word Domination, last year and one of the ways he did blind play testing prior to his campaign was to sit in the back of the room with a notebook and pen and watch. He let them know that he wouldn't respond in any way to questions or comments. I know this breaks the "blind" part of the play testing to some extent, however, the information he gathered from doing this was invaluable and helped him to know if he was on track when he later did more traditional blind play testing.

Without asking the right questions on a questionnaire, you really won't know.

You need to find the right questions for your game and make sure every blind play-tester has a copy of those questions to answer and return. Then you have to sit back and analyze the responses to those questions.

Realize that if they are getting something seriously wrong then changes absolutely need to be made in the rules to clear up those mistakes. You won't be there to guide anyone who buys your game at retail!

Blind Playtesting is like sending your kids out for the first time. If anything, you're looking for consistency. If there's radical differences in gameplay time, it's either you're not communicating your rules effectively or your game is only getting through to a few people.

But to try and figure out if they're doing it right...you should really ask yourself if your rules are truly consistent. Can some part of your rules be confusing? Ask them what they understood about it and have a discussion.