Add iterative refinement example for COBOL to Java refactoring

[neubig]

Description

This PR adds a new example demonstrating an iterative refinement workflow for COBOL to Java refactoring.

Changes

• Added examples/01_standalone_sdk/31_iterative_refinement.py - A comprehensive example that demonstrates:
  1. A refactoring agent that converts COBOL files to Java files
  2. A critique agent that evaluates the quality of each conversion and provides scores
  3. An iterative loop that continues until the average score reaches 90% or higher

How it works

The workflow operates in a loop:

1. Refactoring Phase: An agent reads COBOL files and creates corresponding Java files
2. Critique Phase: A separate agent evaluates each conversion on:
   • Correctness (0-25 pts)
   • Code Quality (0-25 pts)
   • Completeness (0-25 pts)
   • Best Practices (0-25 pts)
3. Iteration: If the average score is below 90%, the refactoring agent is called again with the critique file location so it can address the issues

The example includes sample COBOL files based on the AWS CardDemo application structure.

Source Data Reference

The example recommends using COBOL files from:
https://github.com/aws-samples/aws-mainframe-modernization-carddemo/tree/main/app/cbl

Fixes #1414

Testing

• Pre-commit hooks pass
• CI passes

Documentation

Documentation PR: OpenHands/docs#191

---

Agent Server images for this PR

• GHCR package: https://github.com/OpenHands/agent-sdk/pkgs/container/agent-server

Variants & Base Images

Variant	Architectures	Base Image	Docs / Tags
java	amd64, arm64	eclipse-temurin:17-jdk	Link
python	amd64, arm64	nikolaik/python-nodejs:python3.12-nodejs22	Link
golang	amd64, arm64	golang:1.21-bookworm	Link

Pull (multi-arch manifest)

# Each variant is a multi-arch manifest supporting both amd64 and arm64
docker pull ghcr.io/openhands/agent-server:16da1e7-python

Run

docker run -it --rm \
  -p 8000:8000 \
  --name agent-server-16da1e7-python \
  ghcr.io/openhands/agent-server:16da1e7-python

All tags pushed for this build

ghcr.io/openhands/agent-server:16da1e7-golang-amd64
ghcr.io/openhands/agent-server:16da1e7-golang_tag_1.21-bookworm-amd64
ghcr.io/openhands/agent-server:16da1e7-golang-arm64
ghcr.io/openhands/agent-server:16da1e7-golang_tag_1.21-bookworm-arm64
ghcr.io/openhands/agent-server:16da1e7-java-amd64
ghcr.io/openhands/agent-server:16da1e7-eclipse-temurin_tag_17-jdk-amd64
ghcr.io/openhands/agent-server:16da1e7-java-arm64
ghcr.io/openhands/agent-server:16da1e7-eclipse-temurin_tag_17-jdk-arm64
ghcr.io/openhands/agent-server:16da1e7-python-amd64
ghcr.io/openhands/agent-server:16da1e7-nikolaik_s_python-nodejs_tag_python3.12-nodejs22-amd64
ghcr.io/openhands/agent-server:16da1e7-python-arm64
ghcr.io/openhands/agent-server:16da1e7-nikolaik_s_python-nodejs_tag_python3.12-nodejs22-arm64
ghcr.io/openhands/agent-server:16da1e7-golang
ghcr.io/openhands/agent-server:16da1e7-java
ghcr.io/openhands/agent-server:16da1e7-python

About Multi-Architecture Support

• Each variant tag (e.g., 16da1e7-python) is a multi-arch manifest supporting both amd64 and arm64
• Docker automatically pulls the correct architecture for your platform
• Individual architecture tags (e.g., 16da1e7-python-amd64) are also available if needed

[neubig]

Example Run Results ✅

The iterative refinement example completed successfully!

Summary

• Total iterations: 1
• Final score: 94.0%
• Quality threshold: 90.0%
• Example cost: $0.60

Final Critique Report Snippet

# COBOL to Java Refactoring Critique Report

## Summary
The COBOL to Java refactoring demonstrates a high-quality conversion with strong adherence to Java best practices and OOP principles. All three programs successfully preserve the original business logic while modernizing the codebase.

## File Evaluations

### CBACT01C.cbl
- **Java File**: AccountDisplay.java
- **File Score**: 93/100

### CBCUS01C.cbl
- **Java File**: CustomerInformation.java
- **File Score**: 93/100

### CBTRN01C.cbl
- **Java File**: TransactionProcessing.java
- **File Score**: 95/100

## Overall Score
- **Average Score**: 94/100
- **Recommendation**: PASS

The refactoring successfully converts all COBOL programs to Java with high fidelity to the original business logic while embracing modern Java practices. The code is production-ready with professional-quality documentation, error handling, and structure.

Created Java Files

• AccountDisplay.java
• CustomerInformation.java
• TransactionProcessing.java

The example demonstrates the iterative refinement workflow working as expected - the refactoring agent achieved a high-quality conversion on the first iteration, meeting the 90% threshold.

[openhands-ai]

Looks like there are a few issues preventing this PR from being merged!

• GitHub Actions are failing:
  • [Optional] Docs example

If you'd like me to help, just leave a comment, like

@OpenHands please fix the failing actions on PR #1415 at branch `openhands/iterative-refinement-example`

Feel free to include any additional details that might help me get this PR into a better state.

_{^{You can manage your notification settings}}

[csmith49]

Some minor suggestions, but nothing blocking. LGTM!

[neubig]

@OpenHands do minimal changes to reflect csmith49's comments above.

[openhands-ai]

I'm on it! neubig can track my progress at all-hands.dev

[openhands-ai]

I've addressed csmith49's review comments with minimal changes and pushed to the PR branch. Here's a summary:

Changes Made

Addressing csmith49's comments:

1. Environment variables for configuration (comment 2):

   • Changed QUALITY_THRESHOLD and MAX_ITERATIONS from hardcoded constants to environment variables with defaults
   • Now configurable via QUALITY_THRESHOLD and MAX_ITERATIONS env vars

2. Source traceability in JavaDoc (comment 1):

   • Added requirement Port over BashSession and tests #7 to include @source <program>:<line numbers> in JavaDoc comments
   • This enables post-hoc analysis by linking each Java class/method to its original COBOL source

3. Scoring rubric (comment 3):

   • No changes made - csmith49 explicitly stated "Don't think it's necessary here" as it would blow up the prompt without adding instructive value for the example

Checklist

• Exposed config as environment variables (like LLM configuration)
• Added source traceability requirement for JavaDoc comments
• Pre-commit hooks pass
• Changes are minimal and focused
• Pushed to openhands/iterative-refinement-example branch

The PR has been updated: #1415

_{View full conversation}