The Limits of A.I. as a Language-Learning Tool, re: Articles and Determiners

The school where I work (as an English tutor) is encouraging us to explore the use of A.I. in language learning. While on some level I am not actually an A.I. enthusiast and worry about the long term impacts of our becoming increasingly dependent on technology, I have to admit I can see the potential of the tech as a labor-saving device in a language learning context, just as much as I can see the dangers. And I fully agree that with students using the technology quite widely, we need to be familiar with it and build our skills when it comes to guiding them and cautioning them. 

So yesterday, I was building a workshop I'm going to conduct on articles and determiners (a, an, the, this, that, my, your, or the "zero article," seen in noun phrases like "I like pizza" and "I like cats"). I like my explanations "just so," so I had done the content of the workshop myself, but then I wanted to craft a review exercise, and rather than dully trying to think out a long list of fill-in-the-blanks questions (or find a readymade one that works -- both things tedious and commonplace activities for someone with a job in ESL), I asked the Microsoft AI, Copilot, to design an exercise for me. I initially thought this prompt would be sufficient.

Design a fill-in-the-blanks exercise with 30 questions, practicing A and THE for ESL students, using countable, uncountable, singular and plural nouns referring to a variety of common food items. Provide the answers separately, not as part of the question.

The AI did just this, but almost every question had more than one possible answer. One answer might be more LIKELY than others, of course, but that becomes a question of understanding the social context of the sentence that the author of the exercise is imagining, not the usage of articles and determiners; even if there's a best guess, that's not the same thing as a clear right answer. Of course, there's tons of examples of mediocre ESL materials DESIGNED BY ACTUAL HUMANS that make the same sort of mistake, but I was startled that it was so pervasive in this exercise, since I've had the same A.I. generate pretty good materials previously. 

I’m only going to give the top six sentences as an example, but they're typical -- every question had a similar problem. The notes after each question in the parenthesis are mine, to help you see what the problem is: 

1. I’d like ___ apple, please. (A, THE, or ZERO ARTICLE are all possible; the only determiner as to what the right answer will be is the social context. Am I just asking for fruit? "A" is fine. Am I being offered an apple or a banana? "THE" is what you want. Am I being offered apple juice or orange juice? "I'd like apple, please," is perfectly acceptable. The question is completely useless!).

2. She bought ___ loaf of bread from the bakery. (A or THE are both possible, depending on whether we’re talking about a specific loaf of bread or just asking what she bought).

3. We need ___ eggs for the recipe. (THE or ZERO ARTICLE are both possible, again, depending on whether I'm talking about a specific group of eggs or not). 

4. He drank ___ glass of orange juice. (A or THE are both possible)

5. I saw ___ banana on the table. (All three are possible, depending on whether I am telling what kind of thing I saw, if I am asking where the banana is, or if I am asking you about a fragment of banana smeared on the table surface. One of them, "A," is obviously more likely than the others -- this is maybe the most workable of the questions here; but none are wrong).

6. They ordered ___ pizza for dinner. (...but again, all three are possible, maybe with the ZERO ARTICLE being more likely, A being second most, and THE the least likely)

Okay, so the problem here is with my prompt. I did not specify that I wanted questions with only one right answers. So I explained the above problem to the A.I. -- you can actually "converse" with it, a bit, and asked the A.I. to reframe the questions with more contextual information so that only one correct answer was possible. It stroked my ego a bit and acknowledged the problem, writing:

Thanks for the clarification, Allan — you're absolutely right. To make the exercise pedagogically sound and ensure there's only one correct answer per item, each sentence needs clear context that guides the learner toward either "a" or "the" based on usage rules.

Here’s a revised 30-question fill-in-the-blanks exercise with contextual clues that make the correct article choice unambiguous.

Again, for concision, I'm omitting a bunch, but the top six revised questions, with more contextual information, were as follows. Take a minute and consider, before you proceed: having acknowledged the problem I pointed out, did the A.I. successfully resolve it? 

1. I ate ___ apple after lunch. It was delicious.
2. She bought ___ loaf of bread to make sandwiches.
3. We used ___ eggs that were in the fridge.
4. He poured ___ glass of milk and sat down.
5. I saw ___ banana on the counter and picked it up.
6. They ordered ___ pizza with mushrooms and olives.

Again, as you will see, in all of these cases, more than one answer is possible. I wrote the A.I. again, clarifying my prompt, maintaining the A.I.'s weirdly civil, friendly tone:  

Thanks for trying, but again, at least some of these questions allow for more than one possible answer. For example, number one could be referring to a specific apple: "What did you do with the apple?"/ "I ate the apple after lunch." Or it could be answering a question of "What did you eat after lunch," producing the answer, "I ate an apple." The added context does not actually change the fact that there is more than one possible answer, so the exercise is still pedagogically unsound. Can you try again to produce exercises where only one answer is possible? Include countable and uncountable nouns, singular and plural, all related to common food items. Avoid any possibility of multiple correct answers by embedding the article choice into a socially and grammatically unambiguous situation.

Copilot responded:

Thanks again for your thoughtful feedback, Allan. You're absolutely right that even with added context, many sentences still allow for multiple interpretations unless the context is explicitly constructed to eliminate ambiguity.

To meet your request, I’ve designed a new set of 30 fill-in-the-blank questions that:Use clear, specific context to ensure only one correct article: "a" or "the".

 The first six of these questions were:

1. Sarah picked up ___ apple from the basket and took a bite.
2. I forgot to buy ___ loaf of bread, so we can’t make sandwiches.
3. He cracked ___ eggs that were on the counter.
4. She poured ___ glass of juice and sat by the window.
5. We saw ___ banana lying in the middle of the road.
6. They ordered ___ pizza with extra cheese and mushrooms.

As you see, regardless of whether one answer is more contextually likely or not, AN and THE are both possible for one, A and THE are both possible for number two; and #6 could be A, THE, or ZERO ARTICLE again. Numbers 3 and 5 are in fact improved -- and might actually be useful for the sort of exercise I was trying for, with 3 being "THE" and 5 being "A," but we are really seeing the limits of the A.I.’s ability to “understand” what it is doing; 1, 2, and 4 and 6 all remain useless in pretty much the same way as the previous iterations of the exercise. 6 is especially egregious. 

So I wrote Copilot back again, and gave it positive feedback, in case that somehow helped:

You know, you’re getting better, but there is still more than one answer possible, which we can see if we imagine these as short answers to questions. For instance, for number 6, if the question was, “What did they order?” the answer would require no article (pizza as a non-count noun), or it could be “a pizza,” if we meant “one pizza.” It could also be “the” pizza, if the question were “Did they order the Hawaiian or the Greek pizza?”/ “No, they ordered the pizza with extra cheese and mushrooms.” So contextual knowledge is still necessary for the students to pick one correct answer. Maybe you could try again, writing short question and answer pairs, to make it absolutely clear that the blank (which appears only in the answer) can only be filled one way?

Copilot responded:

Absolutely — thank you for your patience and precision. You're right again: to make the exercise pedagogically sound, the question must constrain the context so that the answer has only one grammatically and pragmatically correct article.

Here is a revised exercise with 30 question–answer pairs, where the blank appears only in the answer, and the question provides enough context to make the correct choice between "a" and "the" unambiguous.

The pairs were: 

(1)
Q: What did you eat for breakfast?
A: I had ___ banana and a cup of tea.

(2) 

Q: Did you buy any fruit at the store?
A: Yes, I bought ___ apple to eat later.

(3)

Q: What did you do with the apple I gave you?
A: I ate ___ apple after lunch.

(4)

Q: What kind of bread did you get?
A: I bought ___ loaf of rye bread.

(5) 

Q: What happened to the loaf of bread I left on the counter?
A: I sliced ___ loaf and put it in the freezer.

(6)

Q: What did they order for dinner?
A: They ordered ___ pizza with mushrooms.

The A.I. is getting better with each revised prompt, but for #6, in particular, all three articles are valid and possible – “the” if they are choosing from a menu (as in, “I’ll have the pizza with mushrooms”), “a” to emphasize “one,” and ZERO ARTICLE to just identify what kind of thing was eaten. The ZERO ARTICLE is the best of them, but this does not make other answers wrong.

At this point, A.I. had ceased to be a labour-saving device; it was simply creating a new kind of labour. But I went through all 30 questions and found about 15 or so actually were natural enough and limited enough in the scope of possible answers that they could be used. For #6, I went with: 

Q: Did they choose the Hawaiian or the vegetarian pizza? 

A: Neither. They chose ___ pizza with mushrooms.

...Which actually is still imperfect (someone COULD conceivably say "a" or even use the zero article and not be wrong), but if we understand that questions and answers often involve a kind of mimicry of form, we'll also know "the" is a much better answer.  It's certainly a damn sight better than the original, "They ordered ___ pizza for dinner." At least there is some reason to pick one over the other! 

It was an interesting and revealing dance and shows both just how useful and how useless the technology can be here. My main concern is that students, attempting to help themselves with this technology, would not be able to see the problems that were readily apparent to me. But I know there are also teachers and tutors out there who might not catch the problem with the exercise as designed by A.I., who would just copy-and-paste the exercise into their materials and not even clue in that it wasn't workable. 

Interesting times ahead.