Amazon Bedrock's LLM-as-a-Judge
Teaching one AI to grade another on Amazon Bedrock.
- Year
- 2026
- Type
- Writing
- From
- Internship
Context
Grading AI answers by hand doesn't scale. Amazon Bedrock can have one model grade another, so I wrote a working walkthrough for people preparing for AWS certifications. Amazon Nova Lite writes the answers and Claude 3.5 Sonnet grades them.
My contribution
- Set up least-privilege access: three custom IAM policies and a service role limited to the evaluation job.
Engineering
- Amazon Bedrock
- Amazon S3
- IAM
Challenges
I couldn't honestly write about an evaluation I hadn't run. So I ran the full evaluation job on Bedrock first, then wrote every step from what actually happened.
Outcome
Published on Tutorials Dojo in January 2026.