W20D2 - Design Your CartPole Solution

Your Mission

Build an AI that learns to balance a pole on a moving cart. You'll design the solution, use AI to help write the code, then test and improve it.

The Goal

Train an agent to score 475+ points by keeping the pole balanced for 100 games in a row.

Pole starts upright
Cart can move left or right
+1 point for each moment balanced
Game ends if pole falls or cart goes off screen

The Rules

You're the architect. AI is your developer. You're the quality checker.

YOU decide what approach to use
AI helps write the code
YOU test if it actually works
Document everything in your journal

The Timeline

You have 3 hours to complete this challenge.

Phase 1: Design (45 min)
Phase 2: Build with AI (60 min)
Phase 3: Test & Improve (45 min)
Phase 4: Document (30 min)

Step 1: What Do You Want?

Before picking an approach, let's figure out what matters most to you.

What's your target score?

The minimum to "solve" CartPole is 195. Perfect is 500.

475

195 (Minimum) 350 (Good) 475 (Great) 500 (Perfect)

What matters more to you?

Keep It Simple

I want something easy to understand, even if it's slower

Learn Fast

I want the agent to figure it out quickly

Stable Training

I don't want random crashes or weird behavior

Best Results

I want the highest possible score, no matter what

How comfortable are you with neural networks?

Step 2: Pick Your Approach

Each approach has trade-offs. Pick the one that matches your goals from Step 1.

DQN (Deep Q-Network)

The "brain that learns what's valuable"

How it works: The agent learns which actions are valuable in each situation, like learning that "moving left when pole tilts right" is good.

Best for: Your first real RL project. Well-documented, lots of examples.

Medium

Difficulty

~300

Episodes

475+

Score

PPO (Proximal Policy Optimization)

The "careful improver"

How it works: The agent directly learns a strategy for acting, improving it bit by bit without making changes that are too big.

Best for: When you want stable, reliable training. Industry favorite.

Harder

Difficulty

~200

Episodes

500

Score

Genetic Algorithm

The "survival of the fittest"

How it works: Create many random agents, keep the best ones, mix their "DNA", repeat. Like evolution!

Best for: Understanding evolution-based learning. No gradients needed!

Easier

Difficulty

~50 gen

Generations

450+

Score

Why did you pick this algorithm?

Step 3: Design the Details

Now let's decide on the specific settings for your solution.

Network Size

Bigger networks can learn more complex patterns, but take longer to train.

Small (32 neurons)

Fast training, simple patterns. Good for CartPole!

Medium (64 neurons)

Balanced choice. Most examples use this.

Large (128 neurons)

More power, but overkill for CartPole.

Enhancements to Include

These options are specific to your chosen algorithm.

Please select an algorithm in Step 2 first to see available enhancements.

Training Settings

Learning Rate (how fast to learn)

Too high = unstable. Too low = slow. Try 0.001 to start.

Training Episodes (how many games to practice)

More = better learning but takes longer. Start with 500.

Step 4: Plan Your Experiments

Scientists test ideas systematically. Plan 2-3 variations to compare.

1

Baseline (your starting config)

What I expect to happen

2

Variation A (change one thing)

What I expect to happen

3

Variation B (try something different)

What I expect to happen

Your AI Prompt

Copy this prompt and paste it into Claude or ChatGPT. The AI will write code based on YOUR design!

Your personalized AI prompt will appear here based on your design choices.

Complete Steps 1-4 to generate your prompt!

Tips for Working with AI

If the code doesn't work, ask AI to explain the error
Ask "Can you add more comments?" if you don't understand something
If training fails, try: "The agent isn't learning. What should I change?"
Save all your prompts in your journal!

How You'll Be Evaluated

Here's how your project will be graded. Make sure you can explain your choices!

Criteria	Weight	What We're Looking For
Does it work?	40%	Agent achieves 475+ average score. Partial credit for lower scores.
Training efficiency	20%	How quickly does it learn? Fewer episodes = better.
Code quality	20%	Is the code readable? Are there comments? Is it organized?
Design justification	20%	Can you explain WHY you made each choice? What alternatives did you consider?

Be Ready to Answer

"Why did you choose this algorithm over others?"
"What was the biggest improvement you made?"
"If you had more time, what would you try next?"
"How did you use AI? What did you have to fix?"

Open Journal →