Unit 2: Deep Neural Networks
Master ASSEB Class XII AI Unit 2: CNNs, RNNs, Attention Mechanism, Transformers, and PyTorch MNIST image classification lab.
Unlock Unit 2: Deep Neural Networks
Access verified code errata manual, 50 external viva questions, period lesson scripts, and practical examination rubrics.
Class XII • Unit 2 Exercise Solutions
Human-crafted academic answers for all exercise questions (Pages 26–29) formulated to secure full marks in the ASSEB Board Examinations. Includes step-by-step calculations, diagrams, and examiner scoring tips.
Deep Neural Networks contain multiple ____ layers between input and output.
CNNs are mainly used for processing ____ data such as images.
Pooling layers in CNNs help reduce ____ complexity.
RNNs are designed to process ____ data where order matters.
The vanishing gradient problem makes it difficult to learn ____ dependencies.
LSTM and GRU are examples of ____ recurrent architectures.
Sequence-to-Sequence models consist of an encoder and a ____.
Transformers rely on the ____ mechanism instead of recurrence.
Multi-head attention allows the model to focus on different ____ in a sequence.
Positional encoding provides information about the ____ of elements in a sequence.
Which of the following best defines a Deep Neural Network?
CNNs are especially effective for:
Which layer type reduces the size of feature maps in CNNs?
RNNs are mainly used for:
Which problem commonly affects vanilla RNNs?
Which architecture helps RNNs remember long-term information?
Transformers differ from RNNs because they:
What is the role of positional encoding?
Which Transformer variant is mainly used for language understanding?
Vision Transformers are mainly applied to:
What is a Deep Neural Network?
A Deep Neural Network (DNN) is an artificial neural network architecture containing multiple hidden layers (typically two or more, up to hundreds of layers) between the input and output layers.
Working Principle: Each layer transforms representation from the preceding layer, enabling the network to learn hierarchical representations of data automatically from raw inputs.
Expressive Power: Deep architectures model highly non-linear, multi-dimensional relationships far more efficiently than shallow networks with fewer total parameters.
Why are CNNs suitable for image processing tasks?
Convolutional Neural Networks (CNNs) are uniquely tailored for images due to three structural properties:
Spatial Grid Exploitation: CNNs process 2D/3D grids directly, preserving spatial adjacency between neighboring pixels.
Local Receptive Fields & Parameter Sharing: Instead of fully connecting every pixel (which causes parameter explosion), small kernels (e.g. 3×3) slide across the image sharing the same weights, drastically reducing model size.
Translation Invariance: Because weights are shared across spatial locations, a filter trained to detect an edge or eye recognizes it regardless of where it appears in the frame.
What is the purpose of pooling in CNNs?
Pooling layers (such as Max Pooling and Average Pooling) serve three core purposes:
Dimensionality Reduction: Compresses the spatial height and width of feature maps, lowering memory usage and computational cost for subsequent layers.
Translation Invariance: Retains summary features (e.g. maximum activation in a 2×2 window) even if features shift slightly in position.
Overfitting Prevention: Reducing spatial dimensions limits excess capacity, encouraging the network to focus on dominant structural patterns.
Spatial Dimension Formula: O = ⌊(W - K + 2P) / S⌋ + 1
How do RNNs handle sequential information?
Recurrent Neural Networks (RNNs) process sequential data by maintaining an internal Hidden State (h_t) that functions as memory:
At each discrete time step t, the network ingests the current input token xt alongside the previous hidden state h{t-1}.
It updates its memory via a recurrent equation:ht = tanh(Whh · h{t-1} + Wxh · x_t + b)
This cyclical feedback loop enables past context to influence future predictions across time.
What causes the vanishing gradient problem?
The Vanishing Gradient Problem is caused by the mathematical chain rule during backpropagation:
When using saturating activation functions (like Sigmoid or Tanh) whose derivatives are strictly less than 1.0 (maximum 0.25 for Sigmoid), gradients are repeatedly multiplied across layers or temporal time steps.
As the computational graph unrolls deeply (∂Loss/∂w = ∏ ∂hi/∂h{i-1}), this repeated multiplication of fractions causes the gradient signal to shrink exponentially toward zero, preventing earlier layers from updating their weights.
How do LSTM or GRU networks solve long-term dependency issues?
LSTMs and GRUs overcome vanishing gradients through Gated Memory Architectures:
LSTM Solution: Features a dedicated Cell State (C_t) channel that runs linearly through time with minimal interactions. Multiplicative gates (Forget Gate, Input Gate, and Output Gate) use Sigmoid activations to add or remove information linearly, creating an uninterrupted gradient pathway across hundreds of time steps.
GRU Solution: Employs a streamlined design with a merged hidden state and two gates (Reset Gate and Update Gate) to preserve long-term historical context with fewer parameters.
What is the role of the encoder in Seq2Seq models?
In Sequence-to-Sequence (Seq2Seq) architectures:
The Encoder is responsible for reading the variable-length input sequence (such as a source sentence in English) token by token.
It compresses the entire semantic meaning and context of the input into a single fixed-length vector called the Context Vector (the final hidden state of the encoder), which is then passed to the decoder to initiate output generation.
What is attention in the Transformer architecture?
Attention is a mechanism that allows a model to dynamically compute direct connections between all tokens in a sequence, irrespective of their distance:
Instead of compressing an entire sequence into a single fixed vector, attention assigns dynamically computed numerical weights to different tokens based on relevance.
In Self-Attention, tokens query all other tokens in the same sequence using Query (Q), Key (K), and Value (V) representations:Attention(Q, K, V) = softmax((Q · K^T) / √d_k) · V
Why is positional encoding necessary in Transformers?
Positional encoding is mandatory because Transformers contain no recurrent loops and no convolution operations:
Self-attention processes all input tokens concurrently in parallel, making the operation mathematically permutation-invariant (the sentences "dog bites man" and "man bites dog" would appear identical without order information).
Positional encodings add unique, deterministic sinusoidal frequency vectors to the token embeddings, providing the network with explicit information regarding word order and relative position.
Give two real-world applications of deep learning models.
Two major real-world applications highlighted in the ASSEB curriculum are:
- 1.Autonomous Driving & Computer Vision: Deep CNNs and Vision Transformers process multi-camera feeds in real time for obstacle detection, pedestrian tracking, lane segmentation, and traffic sign recognition.
- 1.Natural Language Processing & Healthcare Translation: Large language models (such as BERT and GPT) power automated clinical diagnosis from medical records, real-time voice translation across indigenous languages, and conversational chatbots.
Explain the structure and working of Deep Neural Networks.
- 1.Architectural Structure
A Deep Neural Network (DNN) extends basic artificial neural networks by introducing architectural depth. A standard DNN consists of three major tiers:
Input Layer: Ingests the initial raw input feature vector (x) without mathematical transformation.
Multiple Hidden Layers: Stacked sequence of intermediate computational layers (h₁, h₂, ..., h_L). Each layer consists of artificial neurons connected to subsequent layers by weight matrices (W) and bias vectors (b).
Output Layer: Produces final model predictions (ŷ), mapped to probabilities via Softmax/Sigmoid for classification or unbounded linear outputs for regression.
[Input Layer: x] ---> [Hidden Layer 1: h1] ---> [Hidden Layer 2: h2] ---> ... ---> [Output Layer: ŷ] (Raw Features) (Low-Level Edges) (Abstract Motifs) (Final Decision)
- 1.Operational Working Mechanism
Forward Propagation: Data moves forward layer by layer. At each layer l, neurons compute a linear combination followed by a non-linear activation:z^[l] = W^[l] · a^[l-1] + b^[l]a^[l] = f(z^[l])
Loss Evaluation: The network's prediction ŷ is evaluated against true ground truth labels y via an objective loss function (e.g. Mean Squared Error or Cross-Entropy).
Backpropagation: Errors are propagated backward using the calculus Chain Rule to compute the partial derivative of the loss with respect to every weight (∂Loss/∂W^[l]) across all hidden layers.
Parameter Optimization: An optimizer (such as Adam or SGD) updates parameters:W^[l] ← W^[l] - η · (∂Loss / ∂W^[l])
Highlight that the key operational breakthrough of DNNs is representation learning—the ability to learn features automatically from data without manual engineering.
Describe the architecture of CNNs and their applications in computer vision.
- 1.Architecture of a Convolutional Neural Network
A CNN consists of three primary layer types arranged in a hierarchical pipeline:
- 1.Convolutional Layer: The foundational building block. Small learnable filters/kernels (e.g. 3×3) slide across the input image computing dot products. It utilizes two vital principles:
Local Receptive Fields: Neurons connect only to localized sub-regions of pixels.
Parameter Sharing: The same kernel weights are reused across the entire image, giving the model translation equivariance.
- 1.Activation Layer (ReLU): Rectified Linear Unit (max(0, z)) is applied element-wise to introduce non-linearity while preserving positive gradient flow.
- 1.Pooling Layer: Downsamples feature maps (e.g. 2×2 Max Pooling with stride 2). Reduces spatial dimensions by 50%, decreases computational load, and adds translation invariance.Spatial Output Dimension: O = ⌊(W - K + 2P) / S⌋ + 1
- 1.Flattening & Fully Connected (Dense) Layers: The final 2D feature maps are flattened into a 1D vector and passed through standard dense layers with Softmax to produce classification probabilities.
- 1.Prominent Computer Vision Applications
Application Area
Task Description
Representative Models
Image Classification
Assigning an entire image to a single categorical label.
ResNet, VGGNet, AlexNet
Object Detection
Locating and classifying multiple objects with 2D bounding boxes.
YOLO (You Only Look Once), Faster R-CNN
Semantic Segmentation
Assigning every pixel in an image to a semantic class.
U-Net, Mask R-CNN
Medical Diagnostics
Detecting tumors in MRI/CT scans, retinal fundus disease screening.
Deep CNN Classifiers
Discuss the limitations of vanilla RNNs and how LSTM/GRU overcome them.
- 1.Limitations of Vanilla RNNs
Vanishing Gradient Problem: During Backpropagation Through Time (BPTT), multiplying weight matrices across many time steps causes gradients to decay exponentially. Consequently, vanilla RNNs forget contextual information from earlier tokens in long sequences.
Exploding Gradient Problem: Conversely, if weight eigenvalues are greater than 1, gradients can grow exponentially, causing numerical overflow and training divergence.
Sequential Computation Bottleneck: Computing step t strictly requires the hidden state from step t-1, preventing parallel execution on modern GPUs.
- 1.How LSTM Overcomes These Limitations
Long Short-Term Memory (LSTM) networks introduce an independent Cell State (C_t) that acts as an uninterrupted conveyor belt through time, regulated by three multiplicative Sigmoid gates:
Forget Gate (ft): Decides what information to discard from the previous cell state:ft = σ(Wf · [h{t-1}, xt] + bf)
Input Gate (it): Decides which new values to store into the cell state:it = σ(Wi · [h{t-1}, xt] + bi); Çt = tanh(Wc · [h{t-1}, xt] + b_c)
Cell State Update: Information is added linearly: Ct = ft C_{t-1} + i_t Ç_t, preventing exponential gradient decay.
Output Gate (ot): Regulates what portion of the cell state forms the hidden state:ot = σ(Wo · [h{t-1}, xt] + bo); ht = ot * tanh(C_t)
- 1.Gated Recurrent Unit (GRU) Simplification
The GRU merges cell state and hidden state, using only two gates: a Reset Gate (rt) (how to combine new input with previous memory) and an Update Gate (zt) (acts simultaneously as forget and input gate). This achieves comparable long-term retention with fewer parameters and faster training.
Explain the Sequence-to-Sequence learning framework with an example.
- 1.The Sequence-to-Sequence (Seq2Seq) Framework
The Seq2Seq framework (Sutskever et al., 2014) is designed to map an input sequence of arbitrary length to an output sequence of a potentially different length. It consists of two primary recurrent networks:
The Encoder: Ingests the source sequence token by token. Its final hidden state encapsulates the semantic essence of the entire input as a fixed-length Context Vector.
The Decoder: Takes the context vector as its initial hidden state and generates the target sequence autoregressively, one token at a time, until emitting an end-of-sequence token (<EOS>).
[Source: "Good morning"] ---> [ENCODER RNN] ---> [Context Vector: C] ---> [DECODER RNN] ---> [Target: "Suprabhat"] ^ | +-(Feedback)+
- 1.Concrete Real-World Example: Machine Translation
Consider translating the English phrase "How are you?" into Assamese / Hindi ("Aap kaise hain?"):
Encoding Phase: The encoder ingests "How" (updating h₁), then "are" (updating h₂), and finally "you?" (updating h₃). The final state h₃ forms the Context Vector C.
Decoding Phase:
Step 1: The decoder receives start token <SOS> and context vector C → predicts "Aap".
Step 2: Fed with previous prediction "Aap" → predicts "kaise".
Step 3: Fed with "kaise" → predicts "hain".
Step 4: Predicts <EOS>, concluding translation.
Structural Bottleneck: Compressibility is limited for long sentences (50+ words). This limitation led to the introduction of Attention mechanisms.
Describe the Transformer architecture and the role of attention.
- 1.Overview of the Transformer Architecture
Introduced in 2017 by Vaswani et al. in the landmark paper "Attention Is All You Need", the Transformer discards recurrence and convolution entirely. It consists of an Encoder Stack and a Decoder Stack built on attention mechanisms and feedforward networks.
- 1.Scaled Dot-Product Attention
Attention calculates dynamic relationships between tokens using three vectors derived from input embeddings:
Query (Q): Represents the current token asking what to focus on.
Key (K): Represents other tokens acting as reference addresses.
Value (V): The actual information content stored in each token.
Attention(Q, K, V) = softmax((Q · K^T) / √d_k) · V
The scaling factor 1 / √d_k prevents dot products from growing excessively large, avoiding vanishing gradients during softmax.
- 1.Multi-Head Attention
Instead of computing a single attention function, Multi-Head Attention projects Queries, Keys, and Values into h parallel representation subspaces (e.g. 8 heads). This enables the model to simultaneously attend to syntax, semantics, and coreference relations across different positions.
- 1.Positional Encoding & Feedforward Sublayers
Positional Encoding: Injects sequential word order via deterministic sinusoidal waves:PE(pos, 2i) = sin(pos / 10000^(2i/dmodel))PE(pos, 2i+1) = cos(pos / 10000^(2i/dmodel))
Residual Connections & LayerNorm: Each sublayer applies LayerNorm(x + Sublayer(x)), stabilizing deep training up to hundreds of layers.
Compare RNNs and Transformers for sequence modeling tasks.
- 1.Comparative Overview
While Recurrent Neural Networks (RNNs) dominated sequence modeling for decades, the Transformer architecture has replaced them across virtually all natural language processing benchmarks.
- 1.Side-by-Side Architectural Comparison
Dimension
Recurrent Neural Networks (RNN/LSTM)
Transformer Architecture
Basic Mechanism
Sequential step-by-step recurrence with an internal hidden state.
Self-Attention mechanism connecting all tokens directly.
Parallelization
Poor: Sequential nature prevents GPU parallelization during training (step t requires t-1).
Massive: All tokens in a sequence are processed concurrently in parallel on GPUs.
Long-Range Dependencies
Struggles with long contexts due to vanishing gradients across unrolled time steps.
Direct token-to-token path lengths (O(1) operations), easily capturing long-range context.
Positional Awareness
Inherent to the sequential recurrence order.
Requires explicit Positional Encoding vectors added to embeddings.
Computational Complexity
O(n) computational steps across sequence length n.
O(n²) memory and compute complexity with respect to sequence length.
State-of-the-Art Use
Lightweight edge devices and continuous time-series sensor monitoring.
Modern foundational models (BERT, GPT-4, LLaMA, Gemini).
Discuss the advantages of deep learning in real-world applications.
- 1.Major Advantages of Deep Learning
Automated Feature Learning (End-to-End Learning): Traditional machine learning requires human domain experts to hand-engineer features (e.g., SIFT for vision, TF-IDF for text). Deep learning learns optimal feature representations directly from raw pixels, audio waves, or text.
High Capacity with Big Data: While classical algorithms (like SVMs or Logistic Regression) plateau in performance beyond modest data sizes, deep neural networks continually improve as more training data is provided.
Transfer Learning & Foundation Models: Models pre-trained on massive web-scale corpora (e.g. ImageNet, Common Crawl) can be fine-tuned on small localized datasets (such as Assamese dialect translation or regional crop disease detection) with minimal additional training.
Handling Unstructured Data: Deep learning natively ingests multimodal unstructured inputs—including high-resolution satellite imagery, streaming sensor telemetry, and unstructured clinical notes.
Explain how hierarchical feature learning occurs in deep neural networks.
- 1.Concept of Hierarchical Compositionality
Hierarchical feature learning is the process where deep architectures learn to represent the world in a progression of concept levels, with higher-level abstract concepts defined in terms of simpler, lower-level primitives.
- 1.Layer-by-Layer Progression in Computer Vision
Layer 1 (Edges & Gradients): Neurons act as localized edge detectors, identifying transitions in light, dark, orientations, and colors.
Layer 2 (Textures & Motifs): Combines adjacent edges into corners, angles, repetitive textures, and simple geometric contours.
Layer 3 (Object Parts): Combines motifs into recognizable parts (such as wheels, eyes, noses, petals, and handles).
Layer 4 / Output (Full Semantic Objects): Combines object parts into high-level holistic categories (e.g. human face, bicycle, tiger, or house).
Mathematical Efficiency: Stacking layers hierarchically achieves exponential parameter efficiency compared to a shallow network, which would require an astronomically large number of neurons to memorize all combinatorial variations.
Describe the importance of large datasets and computational power in deep learning.
- 1.The Symbiosis of Data and Compute
Deep learning models possess millions or billions of trainable parameters. Their real-world success relies directly on the convergence of two foundational drivers:
- 1.Importance of Large Datasets
Preventing Overfitting: High-capacity neural networks can easily memorize small datasets. Massive datasets (e.g., ImageNet with 14M images, Common Crawl with billions of web pages) force the model to discover generalizable statistical patterns rather than memorizing individual samples.
Capturing Natural Variance: Real-world phenomena feature immense variability (different lighting, camera angles, dialects, accents). Large datasets provide the necessary diversity to make models robust in production.
- 1.Importance of Computational Power (GPUs & TPUs)
Accelerating Parallel Matrix Math: Training a deep network requires quadrillions of floating-point multiplications. Highly parallel architectures (GPUs and TPUs) execute thousands of tensor calculations simultaneously, reducing training times from decades to days.
Iterative Research & Experimentation: Fast computation enables rapid hyperparameter tuning, neural architecture search, and the scaling of model capacity from millions to hundreds of billions of parameters.
Write a detailed note on modern AI systems built using deep neural network architectures.
- 1.Introduction to Modern AI Paradigms
Modern Artificial Intelligence has evolved beyond isolated task-specific classifiers into unified, highly capable Foundation Models built on deep neural network architectures. These systems power major aspects of contemporary technology.
- 1.Key Architecture Pillars of Modern AI
- 1.Generative Large Language Models (LLMs): Built on autoregressive Transformer decoders (e.g., GPT-4, LLaMA, Gemini). They demonstrate advanced reasoning, zero-shot problem solving, code generation, and conversational capabilities.
- 1.Vision-Language & Multimodal Systems: Combine CNN/ViT image encoders with Transformer language models (e.g. CLIP, LLaVA). These models can interpret visual charts, describe medical radiographs, and answer questions about physical scenes.
- 1.Diffusion Models for Media Generation: Deep networks (such as Stable Diffusion, Midjourney, and Sora) that learn to generate high-fidelity photorealistic images and videos by iteratively reversing noise processes.
- 1.Autonomous Robotics & Embodied AI: Integrate deep reinforcement learning with spatial perception models, enabling autonomous vehicles, warehouse robotics, and surgical automation to perceive, plan, and act in dynamic environments.
Curriculum Reflection: As highlighted in the ASSEB curriculum, understanding these deep architectures empowers Class XII students to participate actively in India's digital AI economy.
© ASSEB Class XII Artificial Intelligence (Subject Code: 985) • Master Facilitator Answer Series • Bamunimaidam, Guwahati