Big Data Hadoop Architect Master Program

·       Introducing Scala

·       Deployment of Scala for Big Data applications and Apache Spark analytics

·       Scala REPL, lazy values, and control structures in Scala

·       Directed Acyclic Graph (DAG)

·       First Spark application using SBT/Eclipse

·       Spark Web UI

·       Spark in the Hadoop ecosystem.

·       The importance of Scala

·       The concept of REPL (Read Evaluate Print Loop)

·       Deep dive into Scala pattern matching

·       Type interface, higher-order function, currying, traits, application space and Scala for data analysis

·       Learning about the Scala Interpreter

·       Static object timer in Scala and testing string equality in Scala

·       Implicit classes in Scala

·       The concept of currying in Scala

·       Various classes in Scala

·       Learning about the Classes concept

·       Understanding the constructor overloading

·       Various abstract classes

·       The hierarchy types in Scala

·       The concept of object equality

·       The val and var methods in Scala

·       Understanding sealed traits, wild, constructor, tuple, variable pattern, and constant pattern

·       Understanding traits in Scala

·       The advantages of traits

·       Linearization of traits

·       The Java equivalent

·       Avoiding of boilerplate code

·       Implementation of traits in Scala and Java

·       Handling of multiple traits extending

·       Introduction to Scala collections

·       Classification of collections

·       The difference between iterator and iterable in Scala

·       Example of list sequence in Scala

·       The two types of collections in Scala

·       Mutable and immutable collections

·       Understanding lists and arrays in Scala

·       The list buffer and array buffer

·       Queue in Scala

·       Double-ended queue Deque, Stacks, Sets, Maps, and Tuples in Scala

·       Introduction to Scala packages and imports

·       The selective imports

·       The Scala test classes

·       Introduction to JUnit test class

·       JUnit interface via JUnit  suite for Scala test

·       Packaging of Scala applications in the directory structure

·       Examples of Spark Split and Spark Scala

·       Introduction to Spark

·       Spark overcomes the drawbacks of working on MapReduce

·       Understanding in-memory MapReduce

·       Interactive operations on MapReduce

·       Spark stack, fine vs coarse-grained update, Spark stack, Spark Hadoop YARN, HDFS Revision, and YARN Revision

·       The overview of Spark and how it is better than Hadoop

·       Deploying Spark without Hadoop

·       Spark history server and Cloudera distribution

·       Spark installation guide

·       Spark configuration

·       Memory management

·       Executor memory vs driver memory

·       Working with Spark Shell

·       The concept of resilient distributed datasets (RDD)

·       Learning to do functional programming in Spark

·       The architecture of Spark

·       Spark RDD

·       Creating RDDs

·       RDD partitioning

·       Operations and transformation in RDD

·       Deep dive into Spark RDDs

·       The RDD general operations

·       Read-only partitioned collection of records

·       Using the concept of RDD for faster and efficient data processing

·       RDD action for the collect, count, collects map, save-as-text-files, and pair RDD functions

·       Understanding the concept of key-value pair in RDDs

·       Learning how Spark makes MapReduce operations faster

·       Various operations of RDD

·       MapReduce interactive operations

·       Fine and coarse-grained update

·       Spark stack

·       Comparing the Spark applications with Spark Shell

·       Creating a Spark application using Scala or Java

·       Deploying a Spark application

·       Scala built application

·       Creation of the mutable list, set and set operations, list, tuple, and concatenating list

·       Creating an application using SBT

·       Deploying an application using Maven

·       The web user interface of Spark application

·       A real-world example of Spark

·       Configuring of Spark

·       Learning about Spark parallel processing

·       Deploying on a cluster

·       Introduction to Spark partitions

·       File-based partitioning of RDDs

·       Understanding of HDFS and data locality

·       Mastering the technique of parallel operations

·       Comparing repartition and coalesce

·       RDD actions

·       The execution flow in Spark

·       Understanding the RDD persistence overview

·       Spark execution flow, and Spark terminology

·       Distribution shared memory vs RDD

·       RDD limitations

·       Spark shell arguments

·       Distributed persistence

·       RDD lineage

·       Key-value pair for sorting implicit conversions like CountByKey, ReduceByKey, SortByKey, and AggregateByKey

·       Introduction to Machine Learning

·       Types of Machine Learning

·       Introduction to MLlib

·       Various ML algorithms supported by MLlib

·       Linear regression, logistic regression, decision tree, random forest, and K-means clustering techniques

·       Why Kafka and what is Kafka?

·       Kafka architecture

·       Kafka workflow

·       Configuring Kafka cluster

·       Operations

·       Kafka monitoring tools

·       Integrating Apache Flume and Apache Kafka

·       Introduction to Spark Streaming

·       Features of Spark Streaming

·       Spark Streaming workflow

·       Initializing StreamingContext, discretized Streams (DStreams), input DStreams and Receivers

·       Transformations on DStreams, output operations on DStreams, windowed operators and why it is useful

·       Important windowed operators and stateful operators

·       Introduction to various variables in Spark like shared variables and broadcast variables

·       Learning about accumulators

·       The common performance issues

·       Troubleshooting the performance problems

·       Learning about Spark SQL

·       The context of SQL in Spark for providing structured data processing

·       JSON support in Spark SQL

·       Working with XML data

·       Parquet files

·       Creating Hive context

·       Writing data frame to Hive

·       Reading JDBC files

·       Understanding the data frames in Spark

·       Creating Data Frames

·       Manual inferring of schema

·       Working with CSV files

·       Reading JDBC tables

·       Data frame to JDBC

·       User-defined functions in Spark SQL

·       Shared variables and accumulators

·       Learning to query and transform data in data frames

·       Data frame provides the benefit of both Spark RDD and Spark SQL

·       Deploying Hive on Spark as the execution engine

·       Learning about the scheduling and partitioning in Spark

·       Hash partition

·       Range partition

·       Scheduling within and around applications

·       Static partitioning, dynamic sharing, and fair scheduling

·       Map partition with index, the Zip, and GroupByKey

·        Spark master high availability, standby masters with ZooKeeper, single-node recovery with the local file system and high order functions

·       Introduction to Splunk and Splunk developer roles and responsibilities

·       Writing Splunk query for search

·       Auto-complete to build a search

·       Time range

·       Refine search

·       Working with events

·       Identifying the contents of search

·       What is a Field

·       How to use Fields in search

·       Deploying Fields Sidebar and Field Extractor for REGEX field extraction

·       Delimiting Field Extraction using FX

·       Writing Splunk query for search, sharing, saving, scheduling and exporting search results

·       How to create alerts

·       Understanding alerts

·       Viewing fired alerts

·       Describe and configure scheduled reports

·       Introduction to Tags in Splunk

·       Deploying Tags for Splunk search

·       Understanding event types and utility

·       Generating and implementing event types in search

·       What is a Macro

·       What are variables and arguments in Macros

·       Creating get, post and search workflow actions

·       Studying the search command

·       The general search practices

·       What is a search pipeline

·       How to specify indexes in search

·       Highlighting the syntax

·       Deploying the various search commands like fields, tables, sort, rename, rex and erex

·       Using top, rare and stats commands

·       Using following commands and their functions: addcoltotals, addtotals, top, rare and stats

·       iplocation, geostats, geom and addtotals commands

·       Explore the available visualizations

·       Create charts and time charts

·       Omit null values and format results

·       Calculating and analyzing results

·       Value conversion

·       Roundoff and format values

·       Using the eval command

·       Conditional statements

·       Filtering calculated search results

·       How to search the transactions

·       Creating report on transactions

·       Grouping events using time and fields

·       Comparing transactions with stats

·       Learning data lookups

·       Examples and lookup tables

·       Defining and configuring automatic lookups

·       Deploying lookups in reports and searches

·       Creating search charts, reports and dashboards

·       Editing reports and dashboards

·       Adding reports to dashboards

·       Working with raw data for data extraction, transformation, parsing and preview

·       Describe pivot

·       Relationship between data model and pivot

·       Select a data model object

·       Create a pivot report

·       Create instant pivot from a search

·       Add a pivot report to dashboard

·       What is a Splunk CIM

·       Using the CIM Add-On to normalize data

·       Introduction to the architecture of Splunk

·       Various server settings

·       How to set up alerts

·       Various types of licenses

·       Important features of Splunk tool

·       The requirements of hardware and conditions needed for installation of Splunk

·       How to install and configure Splunk

·       The creation of index

·       Standalone server’s input configuration

·       The preferences for search

·       Linux environment Splunk installation

·       The administering and architecting of Splunk

·       How to install Splunk in the Linux environment

·       The conditions needed for Splunk

·       Configuring Splunk in the Linux environment

·       Introducing Splunk distributed management console

·       Indexing of clusters

·       How to deploy distributed search in Splunk environment

·       Forwarder management

·       User authentication and access control

·       Introduction to the Splunk app

·       How to develop Splunk apps

·       Splunk app management

·       Splunk app add-ons

·       Using Splunk-base for installation and deletion of apps

·       Different app permissions and implementation

·       How to use the Splunk app

·       Apps on forwarder

·       Details of the index time configuration file

·       The search time configuration file

·       Understanding of Index time and search time configuration filesin Splunk

·       Forwarder installation

·       Input and output configuration

·       Universal Forwarder management

·       Splunk Universal Forwarder highlights

·       Implementing the Splunk tool

·       Deploying it on the server

·       Splunk environment setup

·       Splunk client group deployment

·       Understanding the Splunk Indexes

·       The default Splunk Indexes

·       Segregating the Splunk Indexes

·       Learning Splunk Buckets and Bucket Classification

·       Estimating Index storage

·       Creating new Index

·       Understanding the concept of role inheritance

·       Splunk authentications

·       Native authentications

·       LDAP authentications

·       Splunk installation, configuration

·       Data inputs

·       App management

·       Splunk important concepts

·       Parsing machine-generated data

·       Search indexer and forwarder

·       Introduction to Splunk Configuration Files

·       Universal Forwarder

·       Forwarder Management

·       Data management, troubleshooting and monitoring

·       Converting machine-generated data into operational intelligence

·       Setting up the dashboard, reports and charts

·       Integrating Search Head Clustering and Indexer Clustering

·       Understanding the input methods

·       Deploying scripted, Windows and network

·       Agentless input types and fine-tuning them all

·       Splunk user authentication and job role assignment

·       Learning to manage, monitor and optimize Splunk Indexes

·       Understanding parsing of machine-generated data

·       Manipulation of raw data

·       Previewing and parsing

·       Data field extraction

·       Comparing single-line and multi-line events

·       Distributed search concepts

·       Improving search performance

·       Large-scale deployment and overcoming execution hurdles

·       Working with Splunk Distributed Management Console for monitoring the entire operation

·       Cluster indexing

·       Configuring individual nodes

·       Configuring the cluster behavior, index and search behavior

·       Setting node type to handle different aspects of cluster like master node, peer node and search head

·       RDBMS, types of relational databases,

·       challenges of RDBMS,

·       NoSQL database,

·       its significance,

·       how NoSQL suits Big Data needs,

·       introduction to MongoDB and its advantages,

·       MongoDB installation,

·       JSON features,

·       data types and examples

·       Installing MongoDB,

·       basic MongoDB commands and operations,

·       MongoChef (MongoGUI) installation and MongoDB data types

·       The need for NoSQL,

·       types of NoSQL databases,

·       OLTP,

·       OLAP,

·       limitations of RDBMS,

·       ACID properties,

·       CAP Theorem,

·       Base property,

·       learning about JSON/BSON,

·       database collection and documentation,

·       MongoDB uses,

·       MongoDB write concern—acknowledged,

·       replica acknowledged,

·       unacknowledged,

·       journaled—and Fsync

·       Understanding CRUD and its functionality,

·       CRUD concepts,

·       MongoDB query and syntax and read and write queries and query optimization

·       Concepts of data modelling,

·       difference between MongoDB and RDBMS modelling,

·       model tree structure,

·       operational strategies,

·       monitoring and backup

·       In this module, you will learn MongoDB Administration activities such as health check, backup, recovery, database sharding and profiling, data import/export, performance tuning, etc.

·       Concepts of data aggregation and types and data indexing concepts,

·       properties and variations

·       Understanding database security risks,

·       MongoDB security concept and security approach and MongoDB integration with Java and Robomongo

·       Implementing techniques to work with variety of unstructured data like images, videos, log data and others and understanding GridFS MongoDB file system for storing data

·       Big Data characteristics,

·       understanding Hadoop distributed computing,

·       the Bayesian Law,

·       deploying Storm for real-time analytics,

·       Apache Storm features, comparing Storm with Hadoop,

·       Storm execution and learning about Tuple, Spout and Bolt.

·       Installing Apache Storm and various types of run modes of Storm.

·       Understanding Apache Storm and the data model.

·       Installation of Apache Kafka and its configuration.

·       Understanding advanced Storm topics like Spouts, Bolts, Stream Groupings and Topology and its life cycle and learning about guaranteed message processing

·       Various grouping types in Storm, reliable and unreliable messages,

·       Bolt structure and life cycle,

·       understanding Trident topology for failure handling,

·       process and call log analysis topology for analyzing call logs for calls made from one number to another.

·       Various grouping types in Storm, reliable and unreliable messages,

·       Bolt structure and life cycle,

·       understanding Trident topology for failure handling,

·       process and call log analysis topology for analyzing call logs for calls made from one number to another.

·       Various components, classes and interfaces in Storm like Base Rich Bolt Class, i RichBolt Interface, i RichSpout Interface and Base Rich Spout Class and various methodologies of working with them.

·       Understanding Cassandra, its core concepts, its strengths and deployment.

·       Twitter Boot Stripping,

·       detailed understanding of Boot Stripping,

·       concepts of Storm,

·       Storm development environment.

·       Understanding what is Apache Kafka,

·       the various components and use cases of Kafka,

·       implementing Kafka on a single node.

·       Learning about the Kafka terminology,

·       deploying single node Kafka with independent Zookeeper,

·       adding replication in Kafka,

·       working with Partitioning and Brokers,

·       understanding Kafka consumers,

·       the Kafka Writes terminology,

·       various failure handling scenarios in Kafka.

·       Introduction to multi node cluster setup in Kafka,

·       the various administration commands,

·       leadership balancing and partition rebalancing,

·       graceful shutdown of kafka Brokers and tasks,

·       working with the Partition Reassignment Tool,

·       cluster expending,

·       assigning Custom Partition,

·       removing of a Broker and improving Replication Factor of Partitions.

·       Understanding the need for Kafka Integration,

·       successfully integrating it with Apache Flume,

·       steps in integration of Flume with Kafka as a Source.

·       Detailed understanding of the Kafka and Flume Integration,

·       deploying Kafka as a Sink and as a Channel,

·       introduction to PyKafka API and setting up the PyKafka Environment.

·       Connecting Kafka using PyKafka,

·       writing your own Kafka Producers and Consumers,

·       writing a random JSON Producer,

·       writing a Consumer to read the messages from a topic,

·       writing and working with a File Reader Producer,

·        writing a Consumer to store topics data into a file.

·       Introduction to Cassandra, its strengths and deployment areas

·       Significance of NoSQL,

·       RDBMS Replication,

·       Key Challenges,

·       types of NoSQL,

·       benefits and drawbacks,

·       salient features of NoSQL database

·       CAP Theorem,

·       Consistency.

·       Installation,

·       introduction to Cassandra,

·       key concepts and deployment of non relational database,

·       column-oriented database,

·       Data Model – column,

·       column family,

·       Token calculation,

·       Configuration overview,

·       Node tool,

·       Validators,

·       Comparators,

·       Expiring column,

·       QA

·       How Cassandra modelling varies from Relational database modelling,

·       Cassandra modelling steps,

·       introduction to Time Series modelling,

·       comparing Column family Vs. Super Column family,

·       Counter column family,

·       Partitioners,

·       Partitioners strategies,

·       Replication,

·       Gossip protocols,

·       Read operation,

·       Consistency,

·       Comparison

·       Creation of multi node cluster,

·       node settings,

·       Key and Row cache,

·       System Key space,

·       understanding of Read Operation,

·       Cassandra Commands overview,

·       VNodes,

·       Column family

·       JSON,

·       Hector client,

·       AVRO,

·       Thrift,

·       JAVA code writing method,

·       Hector tag

·       Cassandra management,

·       commands of node tool,

·       MapReduce and Cassandra,

·       Secondary index,

·       Datastax Installation

·       Rules of Cassandra data modelling,

·       increasing data writes,

·       duplication, and reducing data reads,

·       modelling data around queries,

·       creating table for data queries

·       Understanding the Java application creation methodology,

·       learning key drivers,

·       deploying the IDE for Cassandra applications,

·       cluster connection and data query implementation

·       Learning about Node Tool Utility,

·       cluster management using Command Line Interface,

·       Cassandra management and monitoring via DataStax Ops Center.

·       Cassandra client connectivity,

·       connection pool internals,

·       API,

·       important features and concepts of Hector client,

·       Thrift,

·       JAVA code,

·        Summarization.