← Back to selected projects

Independent research ARC-AGI

Why did a perfect grid score zero?

I trained Qwen, a language model, to solve visual puzzles from a few examples. On 138 public puzzles I’d kept out of that training, the original model solved four and my version solved seven. I opened the three extra answers to see what had changed.

Explore the three puzzles
Puzzle with gaps
What both models drew
Black squares mark the gaps in the original puzzle.

Every color was in place.

Both models filled all 46 gaps correctly. The checker expected a list of grids, even for a single answer. The original sent the grid on its own. My trained version put the same grid in the list and got credit.

684594894594684
Original answer
684594894
After training
Stripes mark the two extra rows.

It kept drawing.

The puzzle asked for a missing square patch. The original model drew all nine squares correctly, then added two rows. My trained version stopped after the third row, exactly where the patch ended.

234567334567344567345567345678
Original answer
234567334567344567345567345667
After training
Top-left crop. The outline marks the first difference in row five.

One color had to repeat.

The repeated color moves one square to the right in each row. In the fifth row, the original model moved on to 7. My version repeated the 6. Across this puzzle, the trained model got 35 more squares right.

ARC-AGI puzzles by François Chollet · Apache 2.0
See the rule in action

Which squares can you reach from the edge?

In an earlier experiment, I asked both models to fill the holes inside a shape. Both drew the wrong answer. I then asked my trained model to write a Python program. Its first attempt ran out of room. The next program started at the edge, marked every empty square it could reach, and filled the rest in yellow. I ran that rule on the examples and on a new grid. It worked on both.

An example from the ARC puzzle
Trace the rule

The dots mark the squares you can reach from the edge by moving up, down, left or right without crossing green. The program fills the empty squares left over.

ARC-AGI · François Chollet · Apache 2.0

Would the door have opened anyway?

Next I want to give an AI a game it can restart. If it steps on a switch just before a door opens, will it think to walk past the switch on its next try?

Discuss an AI internship