Big Data Course

This module serves as an introductory exploration into the world of big data, providing participants with essential knowledge and insights into its significance and practical applications. Participants will gain an understanding of the core characteristics of big data and various data processing methods. Additionally, they will explore the different types of data, including structured, unstructured, and semi-structured data, along with key concepts such as ETL processes, data lakes, data warehouses, and data marts.

• What is Big Data
• Big Data Characteristics
• Data Processing Methods
• Structured Data
• Unstructured Data
• Semi-Structured Data
• ETL (Extract, Transform, Load)
• ELT (Extract, Load, Transform) & Comparison with ETL
• Data Lake
• Data Warehouse
• Data Mart

In this module, participants will delve into the fundamental tools and frameworks essential for managing and analyzing big data. The primary focus will be on two key technologies: Hadoop and Apache Spark. Through a combination of theoretical insights and hands-on practical exercises, participants will gain a comprehensive understanding of these powerful frameworks. From installation and setup to real-world applications, this module equips learners with the skills needed to harness the potential of big data in their respective fields.

• Hadoop
• HDFS (Hadoop Distributed File System)
• MapReduce
• Apache Spark Introduction
• Comparison with Hadoop
• Spark RDDs
• Spark Architecture
• Components of Apache Spark
• Anaconda Installation & Setup
• PySpark Installation & Setup
• DataBrick & Cloud Computing Introduction
• Databricks Setup for PySpark
• Spark Applications
• Spark Session
• Job
• Stage
• Task
• Spark Context
• Structured APIs
• PySpark RDDs
• Data Frames
• DataSet
• Read & Write CSV and Operations
• Withcolumn, Rename Column ( Column Operations )
• Cast, Contains, Describe
• Rank & Dense Rank
• Functions
• Data Frames Functions
• Window Functions
• Filtering
• Aggregation
• Sorting
• Descriptive statistics
• Handling Missing Values
• Correlation Analysis
• Chi Square Test
• Covariance
• Distinct
• Grouped (Sum, Min, Max)
• Grouped (Mean, Average)
• Count Distinct, Concat, Collect
• like & rlike
• isin & substr
• Pyspark SQL
• Pyspark TempView
• Pyspark GlobalView
• SQL Queries using Temp & Global View
• Pyspark Projects

In this module, participants will embark on a journey into the realm of cloud computing, exploring its transformative potential and the foundational principles underpinning Amazon Web Services (AWS). Through a comprehensive exploration of cloud computing concepts, AWS services, and architectures, participants will develop a robust understanding of how cloud technologies are reshaping industries and revolutionizing the way businesses operate. By delving into the intricacies of AWS, participants will not only gain theoretical knowledge but also practical insights into leveraging cloud resources effectively to drive innovation and achieve business objectives in the digital age.

• Module-01 (Introduction)
  • What is Cloud Computing?
  • Cloud Service Models
  • Cloud Deployment Models
  • Introduction to AWS
  • AWS Architecture
• Module-02 (AWS Compute Services)
  • What is an Instance?
  • What is EC2?
  • Types of Instances
  • AWS Lambda
  • AWS SDKs
  • What is AWS Elastic Beanstalk?
  • PaaS (Platform as a Service)
  • Web Hosting Platforms
• Module-03 (Storage Services)
  • What is Cloud Storage?
  • Cloud Storage Practices
  • Buckets & Objects
  • Versioning & Cross-Region Replication
  • Transfer Acceleration
  • Module-04 (Data Collection)
  • AWS Database Services
  • Introduction to AWS Kinesis

Machine learning is a subset of artificial intelligence that enables computers to learn from data and make predictions or decisions without being explicitly programmed. It involves the use of algorithms, statistical models, and data to build predictive models and uncover insights from large datasets.

• Data Preprocessing
• Supervised Learning
• Unsupervised Learning
• Regression Models
  • Simple Linear Regression
  • Multiple Linear Regression
  • Polynomial Regression
  • Random Forest Regression
  • Topics such as
    • Assessing a Regression Model
    • Bias vs Variance
    • Regularisation
    • Gradient Descent
• Classification Models
  • Decision Tree Classification
  • K-Nearest Neighbor
  • Logistic Regression
  • Na誰ve Bayes
  • Random Forest Classification
  • Support Vector Machines
  • Additional Topics
    • Assessing a Classification Model
    • Adaboost
    • Gradient Boosting
    • XGBoost
    • Grid Search CV
• Clustering Models
  • Hierarchical
  • K-Means Clustering
• Association
  • Apriori
  • Eclat
• Build Dashboards for Data Visualisation
• Solved Sample Code Files for easy practice
• Access to Multiple Datasets
• 7+ Real world data projects

Python is a widely-used programming language in machine learning. It offers powerful libraries, tools, and frameworks for data analysis, modeling, and visualization. Python's simplicity and flexibility make it a popular choice for building machine learning models and deploying them in real-world applications.

• Python Setup and What is Python?
• Data Types and Syntax
• Comparison Operators
• Python Loop
• Python Statements
• Logical Operators
• Methods and Functions
• Error and Exception Handling
• Modules Packages and libraries
• Debugging
• Advanced python Modules (DateTime)
• File Management
• Multiple Activities to Perform
• Multiple Projects to Build

SQL is critical in machine learning, allowing efficient data retrieval and analysis from relational databases. It ensures accurate analysis of large data sets, with solid understanding essential for success.

• DDL and DML in MySQL and setup
• ERD Diagrams and Relational Mapping
• Data normalization
• Basic Queries
• Database Manipulation
• Table Manipulation
• Relational Algebra
• Advanced SQL - Joining, Subquery, Views
• Database Security
• Multiple Activities to Perform

ETL (Extract, Transform, Load) and ELT (Extract, Load, Transform) are data integration processes used to extract data from various sources, transform it into a desired format, and load it into a target data warehouse or data lake for analysis and reporting. Both ETL and ELT play a crucial role in consolidating, cleaning, and preparing data for business intelligence and analytics purposes.

• Introduction to ETL/ELT
  • Extract
  • Transform
  • Load
• Software Tools
  • Talend Studio
  • Apache Hadoop
  • Apache Kafka
• Real-world data projects

The "Docker" module is designed to provide a thorough introduction and foundational understanding of Docker, a key tool in modern software development. This module is ideal for developers, system administrators, and IT professionals who are looking to gain practical knowledge in containerization technology, which is pivotal in achieving efficiency and consistency across various computing environments.

• Intro to Docker
• Docker Basis
• Creating your own Image

This module provides a comprehensive overview of Jenkins, a leading automation server widely used for continuous integration and continuous delivery (CI/CD) processes. Participants will delve into the fundamentals of Jenkins, its operations, and integration with other tools such as GitHub for creating efficient CI/CD pipelines.

• Jenkins Intro
• Jenkins Basics
• Jenkins Operations
• GitHub Integration & CI/CD Test Pipeline