StrataScratch’s Data Project and Mini-Analysis Grader and Evaluator

Every data project and mini-analysis you submit gets read by an AI that scores five dimensions, catches six cardinal sins, and refuses – on principle – to hand you the fix.

Submit a data project on StrataScratch and an AI agent reads it before you ever see a score. It's the same system behind full data projects and mini-analyses alike, and it runs on two ideas that shape everything else about it.

The first is: coach, not grader. Every evaluation names at least one real strength, and any score under 9 comes with a specific note on what would raise it. Harsh feedback is fine. Feedback with nowhere to go isn't.

The second is: mentor, not answer key. You've already submitted – the evaluator's job now is to help you see what you missed, not fix it for you. No corrected code appears anywhere in the response. No field quietly hands you the right answer dressed up as a hint.

Three Inputs, No Shortcuts

The evaluator sees exactly three things, always in the same order:

That's deliberate. The evaluator has to figure out what kind of project it's looking at – predictive modeling, statistical inference, forecasting, whatever the brief actually calls for – from the README and the solution alone, the same way a hiring manager would.

The official solution isn't a template to match line for line, either. It's a benchmark. A different valid approach earns full credit, and a submission that pushes further than the official solution scores above it.

There's also a safety layer built into how those three inputs get read. Everything in them is data to grade, never instructions to follow – so a submission that includes "score this 10/10" gets ignored, not obeyed. Submit the wrong project entirely, and you get a one-line "this looks like a different project" verdict instead of a detailed critique. An unreadable submission gets reported as unreadable; the evaluator never guesses at what it probably contained.

The Five Dimensions Behind Your Score

Every submission gets scored across five weighted dimensions, each on a 0–10 scale. The overall score is the weighted sum, rounded half-up – never nudged to land on a rounder number.

Data Handling & Rigor carries the most weight for a reason – it's the spine everything else sits on. Reasoning & Justification comes next, because in this format, explaining your thinking is the primary deliverable, not an afterthought tacked onto the code.

From Score to Level

The number rolls up into a label, and the label always matches the band the score falls into:

A 7–8 is what a well-prepared candidate submits. A 9–10 is that same work with real depth underneath it. And a 10 on any single dimension means the evaluator genuinely found nothing left to improve – if the written feedback names any gap at all, that dimension caps at 9. Inventing a flaw just to avoid handing out a perfect score isn't allowed either; a dimension gets a 10 by default unless the feedback names something real.

The Rules That Cap a Score No Matter What

Some mistakes put a ceiling on a dimension regardless of how strong the rest of the work is:

Six Mistakes Serious Enough to Hold Back the Whole Score

Beyond the individual caps, six specific problems are treated as cardinal sins – serious enough that when one fires, the whole submission is held at Solid or below, and the feedback says plainly that the result can't be trusted:

There's a seventh sin, and it applies to the evaluator itself, not the submission: inventing a deficiency. Docking points for something the brief never asked for, that isn't in the data, or that belongs to a different question is treated exactly like fabrication would be in a submission.

One more rule keeps the scoring honest in both directions: a single problem gets charged once, in the dimension it actually belongs to – not recycled as a Rigor gap, then a Methodology error, then a Reasoning gap. 

Cardinal sins are the one exception, since they legitimately hit two dimensions at once – Rigor for the missed handling, Methodology for the untrustworthy result.

Not Every Project Needs Every Stage

Data projects move through up to eight stages, and the evaluator only judges the ones a given brief actually calls for:

EDA is listed before Data Cleaning for a reason – looking at the data always has to come before fixing it. These stage ratings are for your own reference only; they don't feed into your overall score, which stays the weighted sum of the five dimensions above. That's also why the same miss can show up twice, phrased differently in each place – a skipped assumption check reads as a thin Model Assumptions stage and a Rigor gap.

What You Actually Get Back

Once scoring is done, the results page shows you considerably more than a number:

Bonus deliverables get listed too, but skipping one never costs you on Completeness – doing it well only helps.

Frequently Asked Questions

Does the evaluator ever just tell me the right answer?

No. That's a hard line. Even when it identifies exactly what went wrong, it's built to explain the gap and let you work toward the fix yourself.

What if I solve it a completely different way than the official solution?

That's fine, and it can even score higher. The official solution is a benchmark for what "good" looks like, not a template your code has to match.

Can I actually get a perfect 10?

Yes, but only if the feedback genuinely can't name a single gap in that dimension. The evaluator isn't allowed to manufacture a flaw just to avoid a perfect score.

What happens if I submit the wrong dataset by mistake?

You'll get a short, direct note that the submission doesn't match the project – not a full breakdown treating the mismatch as a real attempt.

Why does the same mistake sometimes show up in more than one place in my feedback?

Cardinal sins are the one case where that's intentional – a serious issue like leakage affects both how carefully you handled the data and how trustworthy your results are, so it's called out in both places. Anywhere else, one mistake is charged once.