An experiment with developers learning a new Python library found lower quiz scores among those using AI. How they used the tool also mattered.
AI makes it possible to build a small program before you understand much of the code behind it. That can be useful when you need a quick result. It becomes a problem when building the program was supposed to teach you how it works.
Consider a script that sorts a folder of files. It runs correctly on your first folder, so you move on. A week later it overwrites a file with the same name. Fixing that requires more than another generated version: you need to understand how the program chooses names, what it does when a file already exists, and which other cases the change might affect. If AI handled every earlier decision, you may have little knowledge to draw on.
A study by Judy Hanwen Shen and Alex Tamkin examined this tradeoff while developers learned an unfamiliar Python library. It gives beginners a reason to distinguish between using AI to finish a task and using AI to understand a task. Both are possible goals, but one does not automatically deliver the other.
Judy Hanwen Shen and Alex Tamkin studied developers learning to use Trio, a Python tool for handling work that can wait or run alongside other work. Their 2026 paper reports a randomized experiment comparing work with and without AI help. The AI group did worse on a later quiz about the material. The paper also reports that the way people used AI mattered. Asking for explanations was different from handing over the whole task. That second finding describes patterns of use; it does not prove that one prompt caused better learning.
The version reviewed here is an author preprint, a paper shared before completed journal review. It concerns a specific new coding task and a near-term test. It does not show that all AI use weakens coding skill.
Imagine a tiny program that reads a list of names. It removes blank lines and prints a greeting for each person. AI writes it for you. Your list has three names. All three greetings look right. Then consider a name with a leading space, a file with no names, and a duplicate entry. Decide what should happen in each case before asking the tool to implement it. The program cannot be meaningfully checked until those requirements are clear.
If you only ask the tool to fix each new problem, the work can turn into a chain of patches you cannot explain. One fix may break an earlier rule. You may not notice because your test list never checks that rule again. If every new problem is sent straight back to the same tool, it becomes difficult to judge the changes independently. You need at least a small set of expected behaviours against which to check its work.
Before you add a feature, pick one tiny input. Follow it through the program by hand. Write what comes in, what changes, and what comes out. For the name list, start with a blank line. Does the program remove spaces before it checks whether the line is empty? If it checks too soon, a line made of spaces may slip through. That is a real reason to care about the order of two lines of code.
Follow a normal name as well as a blank line. Identify where it is stored, where the greeting is assembled, and which line prints it. Tracing one example with actual values is often more useful than requesting a broad description of the entire program. Ask AI about the step you cannot follow. A useful request is: Explain what happens to this input at this line. Use the actual values. That is easier to check than a long claim that the program is clean and robust.
Once the path makes sense, try a small change yourself. Make the greeting end with a full stop. Or skip a name that appears twice. Choose one change, not a whole new app. Write what you expect before you run the code. Then compare the result with that note. If it differs, find the first place where the values stop matching your plan. Debugging means finding the point where the program stops behaving as intended. A prediction about a particular input gives you something testable. Replacing the whole program after each error can make that process harder because several things may change at once.
If you get stuck, show the tool your guess and the result. Keep the question narrow. Do not throw away everything and ask for a fresh program each time.
AI-generated tests also need review. A test can pass because it repeats the program’s mistaken assumption, so a green result is not enough by itself. Check the expected output against the requirements you chose, including the awkward cases that motivated the test. For the name program, make a short list of cases in plain words. A normal name gets a greeting. A blank line gets none. A name with spare spaces is cleaned. A repeated name follows the rule you chose. Then read the tests against that list. Check the expected output yourself. These are your rules. The tool should not quietly choose them for you.
A useful learning check is to make a small change without asking the tool to rewrite the program. Explain which lines need to change, predict what will happen, and test that prediction. If you cannot do that yet, ask for an explanation of those lines before adding more features.
For experienced developers working on familiar tasks, handing over routine code may be a reasonable way to save time. For someone learning a library for the first time, the same shortcut can remove needed practice. I would judge AI's value in that situation by whether the developer can explain and repair the result, not by whether the first version happens to run.
