Project_18_1_KNN-1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "19d1158b",
   "metadata": {},
   "source": [
    "## Q1. What is the KNN algorithm?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0fd1a5e9",
   "metadata": {},
   "source": [
    "The K-Nearest Neighbors (KNN) algorithm is a simple machine learning technique used for classification and regression tasks. It makes predictions by finding the k-nearest data points in the training dataset to a new data point and using the majority class (for classification) or the average value (for regression) of those neighbors to predict the target value for the new data point. KNN is easy to understand and implement but can be computationally expensive for large datasets."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eeb7c665",
   "metadata": {},
   "source": [
    "## Q2. How do you choose the value of K in KNN?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ea21179",
   "metadata": {},
   "source": [
    "The value of K in KNN can be choosen by:\n",
    "\n",
    "1. Smaller K values (e.g., 1, 3, 5) make the model sensitive to noise, potentially leading to overfitting. They capture fine-grained patterns but may not generalize well. Larger K values (e.g., 10, 20, or more) smooth the decision boundary, making the model less sensitive to noise, but they can underfit if the data has complex patterns, capturing more global trends.\n",
    "\n",
    "2. Preferably choosing an odd value for K in binary classification to avoid ties when voting for the majority class, ensuring a clear winner. For multiclass classification, consider the number of classes and the potential for ties when deciding whether to use an odd or even K.\n",
    "\n",
    "3. Using cross-validation to evaluate K's performance on a validation set.\n",
    "\n",
    "4. Trying a range of K values and selecting the one that results in the best model performance (e.g., accuracy for classification, mean squared error for regression).\n",
    "\n",
    "5. Being mindful of computational resources when selecting K."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c740058a",
   "metadata": {},
   "source": [
    "## Q3. What is the difference between KNN classifier and KNN regressor?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6988fbb5",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "| Aspect                   | KNN Classifier                              | KNN Regressor                               |\n",
    "|--------------------------|--------------------------------------------|--------------------------------------------|\n",
    "| Task                     | Classification (assigns a class label)     | Regression (predicts a numerical value)     |\n",
    "| Output                  | Class labels (discrete categories)         | Numerical values (continuous prediction)    |\n",
    "| Objective Function       | Accuracy, F1-score, etc.                   | Mean squared error (MSE), R-squared, etc.  |\n",
    "| Prediction Method        | Majority vote among K nearest neighbors    | Average of target values from K neighbors   |\n",
    "| Suitable for             | Categorical target variables               | Continuous target variables                |\n",
    "| Typical K Values         | Odd values for binary classification, even or odd for multiclass    | Typically odd values, choice depends on data       |\n",
    "| Robustness               | Sensitive to class imbalance               | Sensitive to outliers                      |\n",
    "| Scaling of Features     | Important, as distance metrics are used    | Important, as distance metrics are used    |\n",
    "| Evaluation Metrics      | Accuracy, Precision, Recall, F1-score, etc. | MSE, R-squared, MAE, etc.                  |\n",
    "| Use Cases               | Text categorization, image classification, | Predicting house prices, stock prices,     |\n",
    "|                          | spam detection, sentiment analysis, etc.   | temperature prediction, etc.               |\n",
    "\n",
    "In summary, KNN Classifier is used for classification tasks, where it assigns data points to discrete classes or categories, while KNN Regressor is used for regression tasks, where it predicts numerical values. Both rely on the concept of proximity to neighbors but apply different methods for making predictions."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f880cde9",
   "metadata": {},
   "source": [
    "## Q4. How do you measure the performance of KNN?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8852ee77",
   "metadata": {},
   "source": [
    "The performance of K-Nearest Neighbors (KNN) using the following metrics:\n",
    "\n",
    "**For KNN Classification:**\n",
    "- Accuracy for overall correctness.\n",
    "- Precision for the ratio of true positives to positive predictions.\n",
    "- Recall for the ratio of true positives to actual positives.\n",
    "- F1-Score for a balance between precision and recall.\n",
    "- Confusion Matrix for detailed classification results.\n",
    "- ROC Curve and AUC for binary classification.\n",
    "\n",
    "**For KNN Regression:**\n",
    "- Mean Squared Error (MSE) for average squared prediction errors.\n",
    "- Root Mean Squared Error (RMSE) for interpretable error in the same units.\n",
    "- Mean Absolute Error (MAE) for average absolute prediction errors.\n",
    "- R-squared (R2) for explaining the variance in the target variable.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e7f87d7",
   "metadata": {},
   "source": [
    "## Q5. What is the curse of dimensionality in KNN?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cbcf4c8f",
   "metadata": {},
   "source": [
    "The curse of dimensionality in K-Nearest Neighbors (KNN) refers to the challenges and limitations that arise when dealing with high-dimensional data. In high-dimensional spaces, the volume increases exponentially as the number of dimensions grows, resulting in sparse data and making it difficult to find meaningful nearest neighbors. This sparsity leads to increased computational complexity, reduced discriminative power, and a need for larger datasets to maintain data density. As a result, KNN may perform poorly in high-dimensional settings, requiring careful feature selection, dimensionality reduction, or alternative distance metrics to mitigate these issues.\n",
    "\n",
    "In short as the number of dimensions increases (ie,number of features) in a dataset only upto a certain number of features the increase in accuracy is observed once there threshold number of dimensions is crossed the the accuracy of the model then decreases with the increase in number of features,this is called curse of dimensionality."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e13186b4",
   "metadata": {},
   "source": [
    "## Q6. How do you handle missing values in KNN?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2645955b",
   "metadata": {},
   "source": [
    "To handle missing values in K-Nearest Neighbors (KNN) we use these methods :\n",
    "\n",
    "1.  **Remove Instances:** If only a few instances have missing values, consider removing them if data loss is acceptable.\n",
    "\n",
    "2. **Impute with Mean/Median:** Fill missing values with the mean (for numeric data) or median (robust to outliers) of the respective feature.\n",
    "\n",
    "3. **KNN Imputation:** Use KNN to estimate missing values by averaging values from the K nearest neighbors for each missing data point.\n",
    "\n",
    "4. **Predictive Models:** Train predictive models to predict missing values based on other features.\n",
    "\n",
    "5. **Multiple Imputation:** Create multiple imputed datasets with different imputed values and analyze them separately to account for uncertainty.\n",
    "\n",
    "6. **Weighted Distances:** Assign different feature weights when calculating distances in KNN to reduce the impact of missing values in less important features."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d158b2d6",
   "metadata": {},
   "source": [
    "## Q7. Compare and contrast the performance of the KNN classifier and regressor. Which one is better for which type of problem?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0139783b",
   "metadata": {},
   "source": [
    "The K-Nearest Neighbors (KNN) classifier and regressor are two variations of the KNN algorithm, each suited for different types of machine learning tasks.\n",
    "\n",
    "**KNN Classifier:**\n",
    "- The KNN classifier is designed for classification tasks, where the goal is to predict discrete class labels for data points. \n",
    "- It works by assigning a data point to the majority class among its K-nearest neighbors, where K is a user-defined hyperparameter. \n",
    "- KNN classifiers are effective when dealing with categorical or nominal target variables. \n",
    "- They calculate distances (typically Euclidean) between data points in feature space and classify the data point based on the most common class among its neighbors. \n",
    "- KNN classifiers are sensitive to the choice of K and the distance metric used. \n",
    "- They are suitable for tasks like text classification, image classification, disease diagnosis, and fraud detection. However, they can be sensitive to class imbalance and may require class balancing techniques like oversampling or undersampling.\n",
    "\n",
    "**KNN Regressor:**\n",
    "- The KNN regressor is designed for regression tasks, where the objective is to predict continuous numerical values for data points.\n",
    "- It also uses a user-defined K to identify the K-nearest neighbors of a data point. \n",
    "- Instead of class labels, the KNN regressor predicts the target variable's value as the average (or weighted average) of the target values of its neighbors. \n",
    "- KNN regressors are sensitive to the choice of K and the distance metric, just like classifiers.\n",
    "- They are well-suited for tasks like house price prediction, stock price forecasting, and temperature prediction. However, they are sensitive to outliers and may require preprocessing techniques to handle them effectively.\n",
    "\n",
    "**Choosing Between KNN Classifier and Regressor:**\n",
    "The choice between KNN classifier and regressor depends on the nature of the target variable and the problem context. If the target variable represents discrete categories or classes, the KNN classifier is the natural choice. Conversely, if the target variable is continuous and represents numerical values, the KNN regressor should be used. It's essential to consider the characteristics of the data and the specific objectives of the machine learning task when deciding between the two variants."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "35a25747",
   "metadata": {},
   "source": [
    "## Q8. What are the strengths and weaknesses of the KNN algorithm for classification and regression tasks, and how can these be addressed?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "55c9e993",
   "metadata": {},
   "source": [
    "### Strengths\n",
    "- **Simplicity**: Easy to understand and implement.\n",
    "- **Versatility**: Applicable to both classification and regression tasks.\n",
    "- **Adaptability**: Can capture complex decision boundaries.\n",
    "\n",
    "### Weaknesses \n",
    "- **Computational Complexity**: Can be slow for large datasets.\n",
    "- **Sensitivity to Noise and Outliers**: Prone to noisy data and outliers.\n",
    "- **Impact of Irrelevant Features**: Treats all features equally.\n",
    "- **Hyperparameter Sensitivity**: Choice of K and distance metric crucial.\n",
    "\n",
    "### Addressing Weaknesses:\n",
    "- **Dimensionality Reduction**: Use PCA or feature selection to reduce dimensionality.\n",
    "- **Outlier Handling**: Apply outlier detection and removal techniques.\n",
    "- **Feature Engineering**: Carefully select and engineer relevant features.\n",
    "- **Distance Metric Selection**: Experiment with different distance metrics.\n",
    "- **Class Balancing**: Employ class balancing methods for imbalanced data.\n",
    "- **Cross-Validation**: Use cross-validation for hyperparameter tuning.\n",
    "- **Ensemble Methods**: Combine KNN models or use ensemble techniques."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "21900dfe",
   "metadata": {},
   "source": [
    "## Q9. What is the difference between Euclidean distance and Manhattan distance in KNN?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f66700c",
   "metadata": {},
   "source": [
    "| Aspect                 | Euclidean Distance                      | Manhattan Distance                      |\n",
    "|------------------------|----------------------------------------|----------------------------------------|\n",
    "| Formula                | d = √[(x2-x1)^2 - (y2-y1)^2]|          d = [mod(x2-x1) + mod(y2-y1)]\n",
    "| Calculation            | Considers straight-line distance.      | Considers the sum of horizontal and vertical distances (city block distance). |\n",
    "| Sensitivity to Scale   | Sensitive to scale differences.        | Less sensitive to scale differences.   |\n",
    "| Dimensionality         | Works well in lower-dimensional spaces. | Effective in high-dimensional spaces. |\n",
    "| Performance            | May perform better when features have a similar scale. | May perform better when features have different scales or in scenarios with sparse data. |\n",
    "| Geometry Interpretation| Represents the shortest path between two points in a Euclidean space. | Represents the distance traveled when navigating a grid-like city block network. |\n",
    "| Use Cases              | Commonly used for spatial data, image processing, and when the relationship between features is roughly isotropic (equal in all directions). | Suitable for cases where the distance metric should reflect the effort or time to travel in a grid-like network, e.g., route planning or feature engineering in text mining. |\n",
    "\n",
    "Euclidean distance is suitable when features have similar scales and the relationship between them is isotropic, while Manhattan distance is effective in scenarios with differing feature scales or when you want to consider grid-like movements."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "108a3e06",
   "metadata": {},
   "source": [
    "## Q10. What is the role of feature scaling in KNN?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "675355ed",
   "metadata": {},
   "source": [
    "The role of feature scaling in K-Nearest Neighbors (KNN) are:\n",
    "\n",
    "1. **Equal Contribution:** Feature scaling ensures all features contribute equally to distance calculations.\n",
    "\n",
    "2. **Distance Metric Consistency:** Scaling makes distance metrics meaningful and unbiased.\n",
    "\n",
    "3. **Convergence Improvement:** It can help KNN converge faster, especially in high dimensions.\n",
    "\n",
    "4. **Robustness to Outliers:** Scaling can make KNN more robust to extreme values.\n",
    "\n",
    "5. **Effective in High Dimensions:** Especially important in high-dimensional datasets.\n",
    "\n",
    "6. **Common Scaling Methods:** Include Min-Max scaling, Z-score scaling, and robust scaling.\n",
    "\n",
    "Feature scaling is essential to ensure KNN provides accurate and unbiased results across various datasets and dimensions."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}