Building an AI Rubric Assessment System with Java and OpenAI

#

As part of my work with AI-driven applications, I wanted to explore how large language models can be integrated into traditional backend applications rather than only being used through chat interfaces.

The goal of this project was to build an application capable of assessing internship reports using AI.

Instead of generating random feedback, the system evaluates reports using:

• Learning goals from the curriculum
• Internship report requirements
• Dare, Share, Care values
• A weighted rubric system

The result is a structured AI-powered assessment system built with Java and OpenAI.

The Goal of the Project

#

The assignment focused on integrating an LLM API into a real application architecture.

That meant:

• Building a backend API
• Creating prompts programmatically
• Handling JSON responses
• Structuring AI output
• Managing external files and data
• Handling API errors and invalid input

The idea was not just to “use AI”, but to make AI part of a real software workflow.

Building the Backend

#

The backend was built using:

• Java
• Javalin
• Maven
• OpenAI API

The application exposes a REST endpoint:

POST /api/assess

A markdown file containing a student report is sent to the backend, which then:

1. Reads the report
2. Loads learning goals and grading criteria
3. Builds prompts dynamically
4. Sends everything to OpenAI
5. Returns structured JSON feedback

Using Markdown Files as Context

#

One important design decision was separating the assessment criteria into external markdown files instead of hardcoding everything directly in Java.

The project uses files like:

learning-goals.md
report-requirements.md
dare-share-care.md
student1.md
student2.md
student3.md

This made the system much more flexible.

Instead of changing Java code every time requirements change, the AI can simply read updated markdown files as context.

It also made the architecture feel much more realistic and maintainable.

Designing the Rubric System

#

To make the evaluations more structured, I created a weighted rubric system in Java.

Each criterion contains:

• A name
• A description
• Performance levels
• A weight value

For example:

{
  "name": "Opfyldelse af læringsmål",
  "weight": 20
}

The AI receives both:

• The original requirement documents
• The structured rubric

This helped make the responses significantly more consistent and relevant.

Prompt Engineering

#

One of the most interesting parts of the project was prompt design.

The application separates prompts into:

• A system prompt
• A user prompt

The system prompt defines:

• The AI’s role
• Rules and limitations
• Output requirements
• Tone and behavior

The user prompt contains:

• Learning goals
• Report requirements
• Dare, Share, Care descriptions
• The rubric
• The student report itself

The AI is instructed to return structured JSON instead of free-form text.

This makes the response much easier to use programmatically.

Building a Small Frontend

#

After the backend was working, I added a simple frontend to make the application easier to test and interact with.

The frontend was built as a lightweight HTML page served directly from the Javalin backend.

It includes:

• A textarea for pasting internship reports
• A button for submitting the report
• A fetch request to the backend API
• A result area showing the AI-generated JSON assessment

This transformed the project from a backend-only API into a complete end-to-end AI application.

Instead of testing everything manually with curl commands, I could now interact with the system directly in the browser.

Handling Real-World Problems

#

One thing I learned quickly is that AI integration is not just about prompts.

A large part of the work involved handling practical software problems like:

• Invalid JSON
• Escaping markdown content
• API key management
• HTTP requests
• Error handling
• Parsing AI responses
• Environment variables

At one point, markdown files containing quotation marks completely broke the JSON request body. Fixing that required proper escaping and eventually using jq to safely send markdown content through curl requests.

That was probably one of the biggest takeaways from the project:

The difficult part is often not the AI itself — it is everything around it.

What I Learned

#

This project gave me a much better understanding of how LLMs fit into real software systems.

Before this, AI mostly felt like something you interacted with through a chat interface. After building this project, I started seeing LLMs more like external backend services that need proper architecture around them.

I also learned how important structure is when working with AI systems.

The better:

• prompts
• input data
• output formats
• validation
• system design

…the more stable and useful the results become.

Another important takeaway was how unpredictable AI output can be. Even when asking for JSON, the model can still occasionally return unexpected formatting or incomplete data. That made validation and error handling much more important than I originally expected.

I also learned that even a very small frontend changes how an application feels. Once the browser interface was added, the system immediately felt much more like a real product rather than just a technical experiment.

Conclusion

#

What started as a small school assignment turned into a really interesting exploration of AI integration and backend architecture.

The project combines:

• Java backend development
• REST APIs
• prompt engineering
• markdown processing
• structured JSON output
• frontend integration
• AI integration

into one complete application.

The most interesting part is that the AI is not just generating text randomly — it is evaluating documents based on structured academic requirements and returning usable programmatic output.

Overall, this project made AI feel much less abstract and much more like a real software component that can be integrated into practical applications.