Degui Zhi Kirk Roberts Licong Cui Xiaqian Jiang Tuan Amith and Cui Tao An ontology representing the model card structure https://creativecommons.org/licenses/by/3.0/ Model Card Ontology 2023-03-07 has version additional information abou the citation associated with the model citation text value of the confidence interval attributed to the Performance Metric concept confidence interval value Data pertaining to the contact information of the model's contact information. This is associated to the Owner Information class of the Model Card Ontology contact text Additional textual information attributed to the licensing information. custom_text Detailed textual information of the model card that is attributed to specific concepts of the Model Card Ontology. This contrasts with the "overview" data property as the "overview" has a summarized version of "documentation" documentation Model Card Report Ontology specific data properties that provide attributes to its concepts. model card text Name of the owner of the model. This is linked to the Owner Information concept of the Model Card Ontology name text Summary textual information for the concepts of the Model Card Ontology overview general numeric value associated with the Performance Metric concepts performance metric value "A description of any sensitive data that may be present in adataset. Be sure to note PII information such as names, addresses, phone numbers, etc. Preferably, such info should be scrubbed from a dataset if possible. Note that even non-identifying information, such as zip code, age, race, and gender, can be used to identify individuals when aggregated. Please describe any such fields here" Google LLC sensitive data This is data property is specific to Perfomance Metric concepts. It attributes information about the data partition used to experiment on the model. slice This is a value pertaining to a threshold value that is attributed to the Performance Metric concept for the Model Card ontology. threshold value The Model Card (MC) is the document designed for transparent reporting of AI model provenance, usage, and ethics-informed evaluation. A report that details characteristics of machine learning model to supplement its release to the public. Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, Timnit Gebru. 2019. Model Cards for Model Reporting. In FAT* ’19: Conference on Fairness, Accountability, and Transparency, January 29–31, 2019, Atlanta, GA, USA. ACM, New York, NY, USA, 10 pages. https://doi.org/10.1145/3287560. 3287596 Model Card Report Section content in this section should describe information in "how close or far off a given set of measurements (observations or readings) are to their true value" https://en.wikipedia.org/wiki/Accuracy_and_precision Accuracy Information Section The content describes information pertaining to the area under the ROC curve. Area Under the Curve Information Section AUC information relating to the citation for the targetted model. This is part of the model details component of the model card report Citation Information Section This section pertains to "Factors" of the model card report by Mitchell et al., 2019. The considerations section includes qualitative information about your model, including some analysis of its risks and limitations. As such, this section usually requires careful consideration, and conversations with many relevant stakeholders, including other model developers, dataset producers, and downstream users likely to interact with your model, or be affected by its outputs. Consideration Information Section Factors Information about the data that was used to develop the model Dataset Information Section Content information that is a type of risk information that is specific to ethical consideration if using the model. Ethical Consideration Section The section is devoted in describing the evaluation dataset. "Evaluation datasets should include datasets that are publicly available for third-party use. These could be existing datasets or new ones provided alongside the model card analyses to enable further benchmarking... the evaluation datasets should not only be representative of the model’s typical use cases but also anticipated test scenarios and challenging cases." Details on the dataset(s) used for the quantitative analyses in the card. - Datasets - Motivation - Preprocessing" (Mitchell et al., 2019) Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, Timnit Gebru. 2019. Model Cards for Model Reporting. In FAT* ’19: Conference on Fairness, Accountability, and Transparency, January 29–31, 2019, Atlanta, GA, USA. ACM, New York, NY, USA, 10 pages. https://doi.org/10.1145/3287560. 3287596 Evaluation Data Information Section The F1 score is the harmonic mean of the precision and recall. This section describes information relating to the F1 score revealed by the performance of the machine learning model. https://en.wikipedia.org/wiki/F-score F1-Score Information Section The section describes information about the false discovery rate of the machine learning model. "The false discovery rate (FDR) is a method of conceptualizing the rate of type I errors in null hypothesis testing when conducting multiple comparisons. FDR-controlling procedures are designed to control the FDR, which is the expected proportion of "discoveries" (rejected null hypotheses) that are false (incorrect rejections of the null)". quivalently, the FDR is the expected ratio of the number of false positive classifications (false discoveries) to the total number of positive classifications (rejections of the null). The total number of rejections of the null include both the number of false positives (FP) and true positives (TP). Simply put, FDR = FP / (FP + TP). FDR-controlling procedures provide less stringent control of Type I errors compared to familywise error rate (FWER) controlling procedures (such as the Bonferroni correction), which control the probability of at least one Type I error. Thus, FDR-controlling procedures have greater power, at the cost of increased numbers of Type I errors." https://en.wikipedia.org/wiki/False_discovery_rate False Discovery Rate Information Section This section describes information relating to false ommision rate of the performance of the machine learning model. FOR is the probability that the true value is positive. False Ommission Rate Section Information FOR This content describes the information pertaining to the model's false postive rate. "A false positive error, or false positive, is a result that indicates a given condition exists when it does not. For example, a pregnancy test which indicates a woman is pregnant when she is not, or the conviction of an innocent person. A false positive error is a type I error where the test is checking a single condition, and wrongly gives an affirmative (positive) decision." https://en.wikipedia.org/wiki/False_positives_and_false_negatives#False_positive_error False Positive Rate Information Section This content summarizes the input and output data format for the model. Format Information Section A section describing the set of graphics accompanying the model card's dataset Graphic Collection Section This secton content serves to describe the inputed data's format for the machine learning model. Input Format Information Section Section for information about the licensing for the machine learning model. License Information Section Content section relating to the technical limits of the machine learning model (e.g. trained on ungeneralized data). Limitation Information Section Content section of the model card that addresses the risk or ethical consideration of the model. Mitigation Strategy Section Content section summarizing the architecture of the model (software used, algortihm, etc.) Model Architecture Information Section a graph content specifically for a model card report Model Card Graphic Basic information about the model that includes licensing information, owner information, the architecture of the model (algorthim employed), references (cited papers), and versioning information. Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, Timnit Gebru. 2019. Model Cards for Model Reporting. In FAT* ’19: Conference on Fairness, Accountability, and Transparency, January 29–31, 2019, Atlanta, GA, USA. ACM, New York, NY, USA, 10 pages. https://doi.org/10.1145/3287560. 3287596 Model Detail Section Section summarizing the content relating to the parameters for the development of the model. Model Parameter Section "for use on black-and-white images only; please consider our research group’s full-color-image classifier for color images.” "not for use on text examples shorter than 100 tokens" "Here, the model card should highlight technology that the model might easily be confused with, or related contexts that users could try to apply the model to. This section may provide an opportunity to recommend a related or similar model that was designed to better meet that particular need, where possible. This section is inspired by warning labels on food and toys, and similar disclaimers presented in electronic datasheets." Content that is a type of use case information but specific to use cases that are out of scope of the intended use case. Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, Timnit Gebru. 2019. Model Cards for Model Reporting. In FAT* ’19: Conference on Fairness, Accountability, and Transparency, January 29–31, 2019, Atlanta, GA, USA. ACM, New York, NY, USA, 10 pages. https://doi.org/10.1145/3287560. 3287596 Out of Scope Use Case Section Information This secton content serves to describe the outputted data's format for the machine learning model. Output Format Information Section Section for the information about the individuals involved in the development of the machine learning model. This includes contact information, the names of the individuals, etc. Owner Information Section This section is for the description of the precision and recall information of the machine learning model's performance. Percision-Recall Curve Information Section Section content here summarizing the information about the metrics (precision, recall, f1 score, etc.) that measured the performance of the model. One can report the values and portion of the data (slice). Performance Metric Information Section personally identifiable information as "any information about an individual maintained by an agency, including (1) any information that can be used to distinguish or trace an individual's identity, such as name, social security number, date and place of birth, mother's maiden name, or biometric records; and (2) any other information that is linked or linkable to an individual, such as medical, educational, financial, and employment information." The section should specificially describe information about the personally identifiable datasets. https://en.wikipedia.org/wiki/Personal_data https://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-122.pdf Personally Identifiable Data Information Section "This section details whether the model was developed with general or specific tasks in mind (e.g., plant recognition worldwide or in the Pacific Northwest). The use cases may be as broadly or narrowly defined as the developers intend. For example, if the model was built simply to label images, then this task should be indicated as the primary intended use case." Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, Timnit Gebru. 2019. Model Cards for Model Reporting. In FAT* ’19: Conference on Fairness, Accountability, and Transparency, January 29–31, 2019, Atlanta, GA, USA. ACM, New York, NY, USA, 10 pages. https://doi.org/10.1145/3287560. 3287596 Primary Intended Use Case Information Section This section describes the primary intended user for the machine learning model. "For example, was the model developed for entertainment purposes, for hobbyists, or enterprise solutions? This helps users gain insight into how robust the model may be to different kinds of inputs." Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, Timnit Gebru. 2019. Model Cards for Model Reporting. In FAT* ’19: Conference on Fairness, Accountability, and Transparency, January 29–31, 2019, Atlanta, GA, USA. ACM, New York, NY, USA, 10 pages. https://doi.org/10.1145/3287560. 3287596 Primary Intended User Information Section content overview of the quantatitve analysis of the model, specifically the performance of the model Quantative Analysis Section need to connect evaluation data information and training data information should their be a specified graphic collection for Quantative Analysis? Quantative Analysis information for any references related to the machine learning model (research papers, website, etc.) Reference Information Section Content information relating to potential uncertainity about utilizing the model Risk Information Section Information relating to sensitive datasets that contain info like personal information that was used to develop the machine learning model Sensitive Data Information Section The section is devoted in descrbing the synthetic evaluation datasets used for a machine learning model. Synthetic evaluation datasets are a type of evaluation dataset. "It is often difficult to find datasets that represent populations outside of the initial domain used in training. In some of these situations, synthetically generated datasets may provide representation for use cases that would otherwise go unevaluated" (Mitchell et al., 2019). This section should described the aforementioned. Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, Timnit Gebru. 2019. Model Cards for Model Reporting. In FAT* ’19: Conference on Fairness, Accountability, and Transparency, January 29–31, 2019, Atlanta, GA, USA. ACM, New York, NY, USA, 10 pages. https://doi.org/10.1145/3287560. 3287596 Synthetic Evaluation Data Information Section Content involving known trade-offs for the use of the model. Trade offs might include performance or technical or ethical. "This section should list additional concerns that were not covered in the previous sections. For example, did the results suggest any further testing? Were there any relevant groups that were not represented in the evaluation dataset? Are there additional recommendations for model use? What are the ideal characteristics of an evaluation dataset for this model?" Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, Timnit Gebru. 2019. Model Cards for Model Reporting. In FAT* ’19: Conference on Fairness, Accountability, and Transparency, January 29–31, 2019, Atlanta, GA, USA. ACM, New York, NY, USA, 10 pages. https://doi.org/10.1145/3287560. 3287596 Trade-off Information Section Caveats Recommendations The section is for information about the training data used for evaluation of a machine learning model. Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, Timnit Gebru. 2019. Model Cards for Model Reporting. In FAT* ’19: Conference on Fairness, Accountability, and Transparency, January 29–31, 2019, Atlanta, GA, USA. ACM, New York, NY, USA, 10 pages. https://doi.org/10.1145/3287560. 3287596 Training Data Information Section The content here involves information about intential uses of the model. "This section should allow readers to quickly grasp what the model should and should not be used for, and why it was created. It can also help frame the statistical analysis presented in the rest of the card, including a short description of the user(s), use-case(s), and context(s) for which the model was originally developed." Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, Timnit Gebru. 2019. Model Cards for Model Reporting. In FAT* ’19: Conference on Fairness, Accountability, and Transparency, January 29–31, 2019, Atlanta, GA, USA. ACM, New York, NY, USA, 10 pages. https://doi.org/10.1145/3287560. 3287596 Use Case Information Section Intended Use Content information pertaining to intended users of the model. This includes individuals that may be impacted as an outcome of the use of the model. target user for the model User Information Section Intended User Information This content section describes any versioning information for the machine learning model. Version Information Section