Written by Dr. Karsten Schulz, Chief Black Box Opener, Digital Technologies Institute Brisbane
A student asks a generative AI system to write a program, explain a concept or help produce an essay. Within seconds, a polished response appears.
It may be convincing. It may even be correct.
But when the student is asked why the answer works, whether it can be trusted or how it should be improved, the confidence often disappears. The student has operated the tool but may not understand the reasoning behind it.
This raises an important question for schools: are we teaching students to work intelligently with AI, or merely teaching them how to use it?
At the Digital Technologies Institute, our mission is to open technological black boxes and give students opportunities to investigate what is inside. We want young people to see technology not as something mysterious that simply delivers answers, but as something designed by people, built from understandable components and open to investigation.
This is also why we created MyComputerBrain: to help students look beneath the surface of digital technology and explore the ideas and systems that make computers work.
Prompting is not enough
Learning how to communicate effectively with generative AI is an important digital skill. Students need to formulate clear requests, provide context, refine outputs and use AI responsibly.
However, prompting alone is not AI literacy.
A student who can write a good prompt may be a capable user, just as someone who can conduct an effective internet search is a capable user of a search engine. But neither skill necessarily provides an understanding of how the underlying system works, why it sometimes fails or how its output should be evaluated.
Students need to understand that generative AI does not know things in the same way people do. It identifies patterns in large quantities of data and generates statistically plausible outputs. Its responses can therefore be useful, but also incomplete, biased, outdated or wrong.
A fluent answer is not necessarily a trustworthy one.
The new black box
Modern technology is moving steadily towards greater convenience and less transparency. Search engines rank information, navigation apps choose routes, streaming platforms recommend content and smart assistants generate answers, yet the processes, assumptions and design decisions behind these results remain largely hidden.
Behind the response are layers of software, training data, algorithms, computing infrastructure and human design decisions. Yet almost all of these remain invisible.
When students cannot see inside the system, they may assume that it “just knows”. They may treat its output as an authority rather than as the result of a computational process that should be questioned.
Opening the black box means giving students clear enough mental models to ask intelligent questions. For example: What data might the system have learned from? Why can it produce different answers to similar questions? Why might a response sound confident but be incorrect? Where do human assumptions and choices enter the system? Is a Generative AI similar to a human brain?
These questions move AI from the realm of apparent magic into the realm of technology.
Digital literacy makes technology visible
Digital technologies education gives students the conceptual tools needed to investigate black boxes.
Through programming, students learn that computers follow instructions. Through data activities, they discover that information must be represented and interpreted. Through algorithms, they see that outputs depend on processes. Through digital systems, they learn that apparently simple actions rely on many interacting components.
These ideas are foundations of meaningful AI literacy.
Students should understand the difference between a rule written by a programmer and a pattern
learned from data. They should recognise that automated decisions depend on what information
is collected, how it is classified and what outcomes designers value.
These are not merely technical ideas. They affect fairness, privacy, trust, accountability and human agency.
From consumer to investigator
The goal of AI education should not be to turn students into passive consumers of intelligent services. It should help them become informed users, critical investigators and creators of digital solutions.
A useful progression is: Use → Question → Investigate → Build
Students might begin by using AI for an authentic task, such as classifying information or suggesting a solution.
They should then question the result. Is it accurate? What evidence supports it? What assumptions has the system made? Where might it fail?
Next, students investigate what may be happening inside the black box. What information is being used? Where might uncertainty or bias enter the process?
Finally, students should have opportunities to build, modify or test a digital solution themselves.
For example, students could ask an AI system to identify suspicious emails. They could examine the indicators it appears to use, such as urgent language, misleading links or requests for personal information, then create a simple program that scores messages.
Their program would be less sophisticated than an AI model, but its decision-making process would be visible. Students would need to decide which indicators matter and when a message should be classified as suspicious.
By rebuilding a simplified version of the black box, students discover that automated judgements are constructed rather than magical.
Preserving human agency
Schools can help students look beneath the interface and understand the systems shaping their lives.
When students use AI, they should explain, test and improve its output. Assessment can value their reasoning, judgement and verification process alongside the finished product.
Hands-on experiences with programming, data, electronics and digital systems are therefore becoming more important. They give students the confidence to explore how technology works and the knowledge to question the answers it provides.
This principle lies at the heart of MyComputerBrain. It enables students to explore computing from the top down and the bottom up—from artificial intelligence, software and data, all the way to binary representation, digital logic and the way a CPU processes instructions.
When students can investigate technology, they begin to see it differently. What first appeared mysterious becomes understandable. What appeared authoritative becomes open to examination. What appeared fixed becomes something they can shape, improve and help build.
That is the purpose of opening the black box.
Dr. Karsten Schulz
Chief Black Box Opener
Digital Technologies Institute
Brisbane
https://www.digital-technologies.institute/
