· By Nicholas “Nick” O'Sullivan, Townsville
Staff are building apps with AI. Who looks after them next?
A staff member has a repetitive task, an idea and access to an AI coding tool. Soon there is a script or small web app that saves time. That initiative is worth taking seriously. So is the point at which the prototype starts collecting student details, accessing business systems or becoming something colleagues depend on every day.
AI-built apps in schools are among the topics being discussed within MITIE. The same questions apply to a charity’s volunteer tool, a church’s booking form or a small business’s internal dashboard: what data does it touch, who can use it, and who will maintain it after the original builder moves on?
Give staff a clear path from idea to approved use
Our recommendation is a short review before real organisational data goes into a prototype. Ask the builder to demonstrate the problem it solves using synthetic data, explain where it runs and identify the accounts or services it needs. Check whether an existing approved tool already meets the need. Scale the review to the consequences of failure and the sensitivity of the information involved.
Review more than whether the demo works
A useful review covers:
- Data flows: What goes into prompts, logs, databases and external APIs? Use approved tools and keep personal or confidential information out of unapproved services.
- Access controls: Verify users can only see and change the records they are meant to access. Test with separate accounts and roles; a hidden button is not an access control.
- Code and dependencies: Have a capable reviewer inspect the implementation, third-party packages, input handling and error behaviour. Automated scans and AI feedback can assist, but are not approval on their own.
- Secrets: Keep passwords and API keys out of source code, browser-delivered code and prompts. Give service accounts only the permissions they need and replace any exposed credentials.
- Hosting: Use an approved environment with organisational ownership, updates, logging, access management and recoverable backups where data is stored.
- Lifecycle: Name an owner and deputy, document the source and deployment process, and set review and retirement dates.
For reviewers, the OWASP Application Security Verification Standard provides a structured reference for checking web application security controls. Choose the relevant checks for the application and record the results; simply mentioning a standard does not establish that a tool has passed review.
Test the awkward situations
Try an account that should have no access. Try an invalid upload, a duplicated form submission and a failed connection to a dependent service. Check what happens when the owner leaves, a paid API reaches its limit or a restore is required. These tests often reveal operational gaps that a successful demonstration leaves hidden.
Record an explicit decision: approved for a limited pilot, approved with conditions, needs more work or unsuitable for the proposed data. For a pilot, limit access and data, set an end date and agree who can stop it. Keep an existing process available while reliability is being established.
Keep the useful idea and make it maintainable
The aim is to give staff a workable route to building useful things. A short intake process, a named reviewer and an approved hosting option make it easier to bring an experiment into the open before it becomes an undocumented dependency.
School ICT teams can compare approaches through MITIE. Suburban Secure can help organisations define ownership, review requirements and technology risk through governance work or a technology review. Start with the tool people are already using and the information it actually handles.
