AI, big data, data warehousing, data mining and knowledge management
Big data is defined by its Vs; warehouses store it for analysis, mining finds patterns and AI learns from it. Knowledge management turns tacit into explicit knowledge.
AI: key terms
| Term | Meaning |
|---|---|
| Artificial intelligence | Machines doing tasks that need human intelligence; term coined by John McCarthy (1956) |
| Turing test | Alan Turing (1950): can a machine's replies pass as a human's? |
| Machine learning | Systems learn patterns from data: supervised (labelled data), unsupervised (unlabelled data), reinforcement (rewards) |
| Deep learning | Machine learning with many-layered neural networks; powers image and speech recognition |
| Natural language processing | Understanding and generating human language, as in chatbots and translation |
| Generative AI | Creates new text, images or code from patterns learnt in data |
| Expert system | Copies an expert's decisions using a knowledge base and an inference engine |
| Narrow vs general AI | Narrow AI does one task; general AI would match humans across tasks and does not yet exist |
The Vs of big data
| V | Meaning | Example |
|---|---|---|
| Volume | Huge size of data | Billions of UPI payments each month |
| Velocity | Speed at which data arrive and must be processed | Live stock-market feeds |
| Variety | Structured, semi-structured and unstructured forms | Text, video, sensor data, social media posts |
| Veracity | Trustworthiness and quality of the data | Fake reviews, faulty sensors |
| Value | Business usefulness extracted from the data | Better credit scoring |
Data warehousing terms
| Term | Meaning |
|---|---|
| Data warehouse | Central store of integrated historical data for analysis; Inmon is called its father |
| ETL | Extract, transform, load: moves data from source systems into the warehouse |
| Data mart | A smaller warehouse for one department, such as marketing |
| OLAP | Online analytical processing: multidimensional analysis by slice, dice, drill-down, roll-up and pivot |
| OLTP | Online transaction processing: the day-to-day operational databases |
| Star schema | A central fact table linked to dimension tables (Ralph Kimball's dimensional approach) |
| Inmon vs Kimball | Inmon is top-down: build the enterprise warehouse first, then data marts. Kimball is bottom-up: build dimensional data marts first and link them |
| Data lake | Raw data of all types stored as-is for later use |
Data mining techniques
| Technique | Finds | Example |
|---|---|---|
| Association (market basket) | Items that occur together | Customers who buy bread also buy butter |
| Classification | Which predefined class a record belongs to | Loan applicant: likely to default or not |
| Clustering | Natural groups with no predefined labels | Customer segmentation |
| Regression (prediction) | Numeric forecasts | Next month's sales |
| Sequence analysis | Patterns over time | A phone purchase followed by a phone cover |
| Anomaly detection | Unusual records | Credit card fraud |
Knowledge discovery in databases (KDD)
Fayyad, Piatetsky-Shapiro and Smyth (1996). Data mining is one step of KDD, not the whole process.
SECI model (Nonaka and Takeuchi, 1995)
- back to Socialisation (tacit → tacit)
Michael Polanyi (1966) observed that we know more than we can say: that unspoken part is tacit knowledge.
Tacit vs explicit knowledge
Knowledge management is the process of creating, capturing, storing, sharing and applying an organisation's knowledge.
NET asks you to expand and order the Vs of big data, to match SECI modes with knowledge conversions, and to define Inmon's warehouse properties or mining techniques such as association and clustering.
Typical MCQ: “Turning a senior engineer's know-how into a written manual is which SECI mode?” Externalisation (tacit to explicit). Trap: combination is explicit to explicit.
Typical MCQ: “Which data mining technique finds products that are bought together?” Association rule mining (market basket analysis). Trap: clustering groups customers, not item sets.
SECI = 'Share, Express, Combine, Internalise'. Inmon's warehouse is SITN: Subject-oriented, Integrated, Time-variant, Non-volatile.