How to Prepare Your Data for AI

Data preparation is the unglamorous foundation of every successful AI implementation. In our experience, it consumes 30–50% of total project time — and it's almost always underestimated in initial scoping. Here's what it actually involves and how to approach it.

Step 1: Data Audit

Before touching anything, map what you have:

• What data sources exist? (Systems, spreadsheets, documents, emails, external feeds)
• What's in each source? (Data types, volumes, update frequency)
• Who owns each source? (Data owner, system administrator)
• What are the access permissions and privacy obligations for each source?
• What's the data quality? (Completeness, consistency, accuracy)

Step 2: Data Cleaning

Identify and fix the quality issues that will undermine AI performance:

• Missing values: How are missing records handled? Imputation, removal, flagging?
• Inconsistencies: The same thing described multiple ways (e.g., "NSW", "New South Wales", "nsw")
• Duplicates: Duplicate records across systems or within a system
• Outliers: Genuine anomalies vs data entry errors — both need handling
• Stale data: Old records that are no longer accurate and shouldn't influence AI models

Step 3: Data Structuring

Organise data into formats AI can work with effectively. For structured AI applications, this means consistent schemas, standardised formats and clear relationships between data entities. For unstructured data (documents, emails), this may mean creating metadata and classification systems.

Step 4: Data Governance

Establish the rules for your data before AI touches it:

• What data can be used for AI, and what can't? (Privacy Act obligations)
• How long is data retained?
• Who can access the AI-processed data?
• What audit trail is required?