Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update phishing-model-card.md #1680

Merged
merged 1 commit into from
May 8, 2024
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 4 additions & 44 deletions models/model-cards/phishing-model-card.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,92 +28,73 @@ limitations under the License.
* Devlin J. et al. (2018), BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding https://arxiv.org/abs/1810.04805 <br>

## Model Architecture:

**Architecture Type:**

* Transformers <br>

**Network Architecture:**

* BERT <br>

## Input: (Enter "None" As Needed)

**Input Format:**

* Evaluation script downloads the smsspamcollection.zip and extract tabular information into a dataframe <br>

**Input Parameters:**

* SMS/emails <br>

**Other Properties Related to Output:**

* N/A <br>

## Output: (Enter "None" As Needed)

**Output Format:**

* Binary Results, Fraudulent or Benign <br>

**Output Parameters:**

* N/A <br>

**Other Properties Related to Output:**

* N/A <br>


## Software Integration:

**Runtime(s):**

* Morpheus <br>

**Supported Hardware Platform(s):** <br>

* Ampere/Turing <br>

**Supported Operating System(s):** <br>

* Linux <br>

## Model Version(s):

* v1 <br>

# Training & Evaluation:

## Training Dataset:

**Link:**

* http://archive.ics.uci.edu/ml/machine-learning-databases/00228/smsspamcollection.zip <br>

**Properties (Quantity, Dataset Descriptions, Sensor(s)):**

* Dataset consists of SMSs <br>

## Evaluation Dataset:

**Link:**

* https://github.com/nv-morpheus/Morpheus/blob/branch-24.06/models/datasets/validation-data/phishing-email-validation-data.jsonlines <br>

**Properties (Quantity, Dataset Descriptions, Sensor(s)):**

* Dataset consists of SMSs <br>

## Inference:

**Engine:**

* Triton <br>

**Test Hardware:** <br>

* DGX (V100) <br>

## Ethical Considerations:
Expand All @@ -124,48 +105,38 @@ NVIDIA believes Trustworthy AI is a shared responsibility and we have establishe
## Model Card ++ Bias Subcard

### What is the language balance of the model validation data?

* English

### What is the geographic origin language balance of the model validation data?

* UK

### Individuals from the following adversely impacted (protected classes) groups participate in model design and testing.

* Not Applicable

### Describe measures taken to mitigate against unwanted bias.

* Not Applicable

## Model Card ++ Explainability Subcard

### Name example applications and use cases for this model.
* The model is primarily designed for testing purposes and serves as a small pre-trained model specifically used to evaluate and validate the phishing detection pipeline. Its application is focused on assessing the effectiveness of the pipeline rather than being intended for broader use cases or specific applications beyond testing.

### Fill in the blank for the model technique.

### Intended Users.
* This model is designed for developers seeking to test the phishing detection pipeline with a small pre-trained model.

### Name who is intended to benefit from this model.

* The intended beneficiaries of this model are developers who aim to test the performance and functionality of the phishing pipeline using synthetic datasets. It may not be suitable or provide significant value for real-world phishing messages.

### Describe the model output.
* This model output can be used as a binary result, Phishing/Spam or Benign

### List the steps explaining how this model works.
### Describe how this model works.
* A BERT model gets fine-tuned with the dataset and in the inference it predicts one of the binary classes. Phishing/Spam or Benign.

### Name the adversely impacted groups (protected classes) this has been tested to deliver comparable outcomes regardless of:
* Not Applicable

### List the technical limitations of the model.
* For different email/SMS types and content, different models need to be trained.

### Has this been verified to have met prescribed NVIDIA standards?

* Yes

### What performance metrics were used to affirm the model's performance?
Expand All @@ -182,25 +153,18 @@ NVIDIA believes Trustworthy AI is a shared responsibility and we have establishe
### Link the location of the training dataset's repository.
* http://archive.ics.uci.edu/ml/machine-learning-databases/00228/smsspamcollection.zip

### Is the model used in an application with physical safety impact?
* No

### Describe life-critical impact (if present).
### Describe the life critical impact (if present).
* None

### Was model and dataset assessed for vulnerability for potential form of attack?
* No

### Name applications for the model.

* The primary application for this model is testing the Morpheus phishing detection pipeline

### Name use case restrictions for the model.
* This pretrained model's use case is restricted to testing the Morpheus pipeline and may not be suitable for other applications.

### Name target quality Key Performance Indicators (KPIs) for which this has been tested.
* N/A

### Is the model and dataset compliant with National Classification Management Society (NCMS)?
* No

Expand Down Expand Up @@ -230,16 +194,12 @@ NVIDIA believes Trustworthy AI is a shared responsibility and we have establishe
* Unknown

### Is a mechanism in place to honor data subject right of access or deletion of personal data?

* N/A

### If PII collected for the development of this AI model, was it minimized to only what was required?
* N/A

### Is data in dataset traceable?
* N/A

### Are we able to identify and trace source of dataset?
### Is there data provenance?
* N/A

### Does data labeling (annotation, metadata) comply with privacy laws?
Expand Down
Loading