Rating trivia questions

Short answer:
1) Get a group of 10 or more players who are representative of your target audience for the game and are roughly equal in education and skill.
2) Have them answer each question. Time how long it takes them to answer each question.
3) Take the average of the wrong answer with the longest time and the correct answer with the shortest time. This number is the "difficulty rating" of the question.

Alternately, have test participants answer each question, taking only one minute per question. Count how many times a given question was answered correctly. Higher numbers indicate easier questions. Lower numbers indicate harder questions.

Long answer:
I'm not aware of a specific rating system for trivia (or test) question difficulty, although I suspect that one exists somewhere.

However... Difficulty is subjective. Before you can rate the "difficulty" of your questions, you need to define what difficulty means to you and what you're trying to accomplish with your game.

Your categories are: Algebra, geometry, analysys, combinatorics, and number theory. They're all math-related. I'm assuming that you intend for your game to fill two roles: providing a fun structure for players to rehearse math skills and for them to learn math skills. I'm also assuming that you want to evaluate the difficulty of your questions so you can associate more powerful power-ups with the harder questions. This implies that you want players with a more in-depth understanding of a given topic to be rewarded with cooler power-ups.

The best two ratings of a question's difficulty (that I can think of in this context) are how long it takes to get a correct answer and how many players (out of, say, 10) get the correct answer. Pretend I have a question. The answer can be correct (1), partially-correct (0), or incorrect (-1). I give that question to 10 players (all of whom are educational grade-level peers). I get the following results:

Name: Score (1, 0, or -1), Time in minutes – Commentary
Player 0: 1, 5.8 – Took a long time, got it right.
Player 1: -1, .8 – Did it really fast, got it wrong.
Player 2: 0, 1.5 – Didn't take too long, partially correct.
Player 3: 1, 2.9 – Took a while, got it right.
Player 4: 1, 3.1 – Took a while, got it right.
Player 5: 0, 1.8 – Didn't take too long, partially correct.
Player 6: -1, 1.1 – Did it pretty fast, got it wrong.
Player 7: 0, 1.7 – Didn't take too long, partially correct.
Player 8: 1, 2.5 – Took a while, got it right.
Player 9: 0, 2.7 – Took a while, partially correct.

Assuming that the testers have equivalent educations and mathematical skills, this tells me a few things about the question. There is a clear correlation between time taken to answer and accuracy of answer. Sorting by time to answer reveals another interesting thing:

Player 0: 1, 5.8 – Took a long time, got it right.
Player 4: 1, 3.1 – Took a while, got it right.
Player 3: 1, 2.9 – Took a while, got it right.
Player 9: 0, 2.7 – Took a while, partially correct.
Player 8: 1, 2.5 – Took a while, got it right.
Player 5: 0, 1.8 – Didn't take too long, partially correct.
Player 7: 0, 1.7 – Didn't take too long, partially correct.
Player 2: 0, 1.5 – Didn't take too long, partially correct.
Player 6: -1, 1.1 – Did it pretty fast, got it wrong.
Player 1: -1, .8 – Did it really fast, got it wrong.

Take a look at scores for players 3, 9, 8, and 5. This is where the correct answer times overlap with the partially correct answer times. If we assume that the time it takes to answer the question and get it correct is a valid indicator of the difficulty of the question, then we could assign this question a difficulty rating of 2.6. That's the average of the time for the lowest correct answer and the highest partially correct answer. If you wanted to, you could discard the results for players 9 and 8 as outliers and take the average times for players 3 and 5, yielding a difficulty rating of 2.35 ((2.9+1.8)/2).

Wow, I just ran down a rabbit hole I didn't intend to enter. I guess what I'm saying is that approximating the shortest amount of time it takes to get a question correct is one way of assessing the difficulty of a question. Another way would be to test a bunch of education-equivalent peers, giving a set amount of time (1 minute?) to answer every question. Count the number of times a question was answered correctly. Higher numbers (more correct answers) indicate easier questions and low numbers (fewer correct answers) indicate harder questions.

Example: I have a 10-question test. I give this test to 100 education-equivalent people. Each participant has 1 minute to answer each question. The results (number of correct answers) for each question are:

Question 0: 73
Question 1: 13
Question 2: 55
Question 3: 100
Question 4: 89
Question 5: 83
Question 6: 56
Question 7: 33
Question 8: 26
Question 9: 90

Sorted by number correct, from most correct to least, I get:

Question 3: 100
Question 9: 90
Question 4: 89
Question 5: 83
Question 0: 73
Question 6: 56
Question 2: 55
Question 7: 33
Question 8: 26
Question 1: 13

This tells me that Question 3 is the easiest (and may be too easy, in fact). Questions 9, 4, 5, and 0 are of roughly equal difficulty and are not too hard. Questions 6 and 2 are of roughly equal difficulty and are fairly hard. Questions 7, 8, and 1 are of roughly equal difficulty and are pretty hard, with question 1 possibly being too difficult.

I think I prefer this way of assessing difficulty.

Okay. I think that's enough for now. Of course, you could always just use your own judgment. Imagining how many people out of 100 you'd expect to get the question correct given a minute to answer it may be sufficient for you to assign difficulty ratings to your questions.

Best of luck with your game!

Excellent answer by InvisibleJon!

My 2 cents: to some extent, all that info (or similar) has already been collected and analyzed. A math teacher will probably be able to tell you, without much effort, which topics and questions are easier or harder.

If you can't have a teacher as a consultant, you may be able to use school books or curricula as a reference. We all know calculating 2+2 is easier than 2x2, which is easier than the square root of 36, etc. No school plan teaches you how to solve an advanced analysis problem before you learn to do simple calculations. First grade math is easier than third grade, and so on.

Regarding the rating of just questions, you should ask them to as many people as you possibly can. Something that might be simple to you might be unheard of to someone else, and the opposite is true also.

Two quiz shows that make use of graduated questions are Jeopardy! and Who Wants to be a Millionaire? If you watch a few episodes of each, you'll get an idea for what's easy and what's hard, at least in terms of the overall program.

Why do you want to avoid using correct answers as the measuring stick? That seems to make the most sense. If you're right, you score a hit. If not, you fail.