Parameters in Neural Networks

A parameter in an artificial neural network is a numerical value that the network learns from training data. Parameters chiefly consist of weights, which determine the strength and direction of connections between units, and biases, which shift a unit’s activation. Together, their learned values determine how the network transforms an input into an output.

Weights and biases

For a simple artificial neuron, the output may be represented as y = f(Σwᵢxᵢ + b), where xᵢ are inputs, wᵢ are weights, b is the bias and f is an activation function.

• A weight determines how strongly a particular input influences a neuron’s computation.
• A positive or negative weight can increase or decrease the weighted sum associated with an input.
• A bias provides an adjustable offset, allowing the activation to shift independently of the input values.
• A neural network’s parameter count is the total number of such adjustable numerical values across its layers.

How parameters are learned

Training seeks parameter values that reduce a chosen loss function, which measures the difference between the network’s predictions and the desired outputs.

• The forward pass uses the current parameters to generate predictions.
• Backpropagation computes how the loss changes with respect to each trainable parameter.
• An optimisation algorithm, such as gradient descent, uses these gradients to update the parameters.
• After training, the learned parameters are used during inference to produce outputs for new inputs.

Parameters and hyperparameters

Parameters are learned from data, whereas hyperparameters are settings selected before or during the training process to control the model or its learning procedure.

• Weights and biases are parameters.
• The learning rate, batch size and number of training epochs are hyperparameters.
• Architectural choices, such as the number of layers or units per layer, determine the arrangement and number of parameters but are not themselves learned weights or biases.

How UPSC asks this