observable:NTFSFileFacet Enhancement Request #645
Comments
|
@ajnelson-nist I updated this after initially posting it after realizing that there were several more NTFS properties I needed to include. This is for my ongoing File Provenance Explorer app dev effort at Project VIC. I am working to build on the MFTECmd work by Eric Zimmerman. |
|
@vulnmaster , thank you for posting this. I have a few notes for processing:
A few notes for discussion, from the perspective of those who populate these properties in their data:
|
|
Hi @ajnelson-nist I updated the change proposal significantly based on your last round of questions. Yes, some of these properties are extracted from USN Journal records. My proposal includes properties from different file types into this one observable:NTFSFileFacet, but we could also break them up into a series of facets if we had too. I just thought it might be more accessible if we put all 24 properties onto one facet. |
|
I'm a bit concerned with some of the degrees of specificity on data sources. Timestamp resolution does not seem like it would be a file-specific property, but instead would be gathered from some more-global resource. I still can't recall where that is stored, though---somewhere in the file system superblock? In There are subtleties in how some of these properties should be populated. Having the file-scoped properties go into I'm fine working towards implementing parts of this proposal---I do agree that several of the properties are worth encoding---but I would really prefer the proposal's properties be demonstrable or testable. I'm aware of some public NTFS images that contain artifacts. E.g., the NIST "Data leakage case" (data here -- link corrected) has |
|
Citations to documentation, such as near this Microsoft page1 on the Master File Table, would also be significantly helpful for property source verification. Footnotes
|
|
Also, when certain system files, attributes, or flags are noted as sources, can they be cross-checked for the spellings in these appendices from the Microsoft File System Control Codes documentation1? Footnotes
|
|
There are also several versions of journal structures: |
|
@ajnelson-nist I added in a References section up near the top of the change proposal to capture the references you highlighted above as they are significant. I will be digging deeper into this over the coming days/weeks and then submit modifications or additional analysis. Since this change proposal is so large, I think it might be easier to make changes to the proposal itself rather than stringing along changes in the comment section. Can I suggest that you add a references section in the UCO Change Proposal Template? |
UCO Change Proposal: Extended NTFS File System Properties
Change Proposal Created with the Assistance of AI LLM Services: Claude Sonnet 3.5, GPT 4o1, and Google Gemini
Executive Summary
This proposal enhances the
observable:NTFSFileFacetto better represent NTFS metadata in digital forensics investigations. By adding twenty-four comprehensive properties to the existing facet, we enable a more complete representation of NTFS metadata, facilitating better correlation of file system artifacts. These additions maintain backward compatibility while significantly improving the ability to represent and analyze NTFS-specific artifacts.Key Benefits:
Primary Beneficiaries:
References
Requirements
Core Requirements
The UCO ontology must:
Support complete representation of NTFS metadata structures:
Enable accurate forensic analysis capabilities:
Facilitate cross-referencing between:
Support forensic tool integration:
Success Criteria
Completeness
Accuracy
Usability
Performance
Compatibility
Requirements to Success Criteria Mapping
Property Source Mapping
Note: Properties marked as "Derived" or "System-wide" are computed or obtained from system configuration rather than direct NTFS attributes.
Type
Enhancement
Sub-Categories
Description
This change proposal suggests adding additional properties to the existing
observable:NTFSFileFacetto better support NTFS file system metadata obtained through MFT analysis. These properties are essential for digital forensics investigations as they provide crucial information about file system entries, their lifecycle, and relationships within the NTFS structure.Scope
Future Extensibility
This enhancement lays the groundwork for:
Motivation
When analyzing NTFS Master File Table (MFT) entries during digital forensics investigations, several critical pieces of metadata are currently not representable in the CASE/UCO ontology. These properties are essential for:
Real-World Use Cases
Ransomware Investigation
Data Exfiltration Analysis
Anti-Forensics Detection
Stakeholder Input
This proposal incorporates feedback from:
Current Situation
The current
observable:NTFSFileFacetincludes basic NTFS properties but lacks several important fields that are available in MFT entries. This limits the ability to represent complete NTFS metadata in CASE/UCO graphs.Proposed Solution
Add the following properties to
observable:NTFSFileFacet:{ "@type": "uco-observable:NTFSFileFacet", "properties": { "sequenceNumber": { "@type": "xsd:integer", "description": "The sequence number for this MFT entry, used to track reuse of MFT entries", "range": "xsd:nonNegativeInteger" }, "logfileSequenceNumber": { "@type": "xsd:integer", "description": "The sequence number in the NTFS log file (USN Journal) for this entry", "range": "xsd:nonNegativeInteger" }, "updateSequenceNumber": { "@type": "xsd:integer", "description": "The update sequence number for this MFT entry", "range": "xsd:nonNegativeInteger" }, "referenceCount": { "@type": "xsd:integer", "description": "The number of hard links to this file", "range": "xsd:nonNegativeInteger" }, "nameType": { "@type": "xsd:string", "description": "The type of filename entry in NTFS", "range": { "oneOf": [ "POSIX", "Win32", "DOS8.3", "Win32AndDOS" ] } }, "entryNumber": { "@type": "xsd:integer", "description": "MFT entry number", "range": "xsd:nonNegativeInteger" }, "inUse": { "@type": "xsd:boolean", "description": "Whether the MFT entry is in use" }, "parentEntryNumber": { "@type": "xsd:integer", "description": "Parent directory's MFT entry number", "range": "xsd:nonNegativeInteger" }, "parentSequenceNumber": { "@type": "xsd:integer", "description": "Parent directory's sequence number", "range": "xsd:nonNegativeInteger" }, "parentPath": { "@type": "xsd:string", "description": "Full path of the parent directory" }, "recordType": { "@type": "xsd:string", "description": "Type of MFT record", "range": { "oneOf": [ "File", "Directory" ] } }, "securityId": { "@type": "xsd:integer", "description": "Security ID reference", "range": "xsd:nonNegativeInteger" }, "dataType": { "@type": "xsd:string", "description": "Type of data", "range": { "oneOf": [ "Resident", "Non-resident" ] } }, "timestampResolution": { "@type": "xsd:string", "description": "Resolution of timestamps in the record" }, "logicalSize": { "@type": "xsd:integer", "description": "Logical size of the file", "range": "xsd:nonNegativeInteger" }, "physicalSize": { "@type": "xsd:integer", "description": "Physical size on disk", "range": "xsd:nonNegativeInteger" }, "flags": { "@type": "xsd:string", "description": "File flags" }, "siFlags": { "@type": "xsd:string", "description": "Standard Information flags" }, "maxVersions": { "@type": "xsd:integer", "description": "Maximum versions allowed", "range": "xsd:nonNegativeInteger" }, "versionNumber": { "@type": "xsd:integer", "description": "Current version number", "range": "xsd:nonNegativeInteger" }, "classId": { "@type": "xsd:string", "description": "Class ID (if applicable)" }, "ownerId": { "@type": "xsd:integer", "description": "Owner ID", "range": "xsd:nonNegativeInteger" }, "quotaCharged": { "@type": "xsd:integer", "description": "Quota charged", "range": "xsd:nonNegativeInteger" }, "usnJournalId": { "@type": "xsd:string", "description": "USN Journal ID" } } }Property Details
MFT Entry Properties
entryNumber
1234identifies the 1234th entry in the MFTsequenceNumber
inUse
falseindicates a deleted/unallocated entryFile Reference Properties
parentEntryNumber
5678links to parent directory's entryparentSequenceNumber
2indicates second use of parent entryparentPath
Record Attributes
recordType
ntfsDataType
Size Information
logicalSize
physicalSize
Attribute Flags
ntfsFlags
standardInformationFlags
Security Context
securityId
ownerId
Change Tracking
usn
usnJournalId
Additional Properties
maxVersions
5indicates a maximum of five versions allowedversionNumber
3indicates third version of the fileclassId
ownerId
1002references specific user/groupquotaCharged
500bytes charged to quotatimestampResolution
updateSequenceNumber
42indicating 42nd updateusnJournalId
Property Constraints and Validation Rules
maxVersions
versionNumber
classId
ownerId
quotaCharged
timestampResolution
updateSequenceNumber
usnJournalId
Property Interdependencies
graph TD A[Entry Identification] --> B[entryNumber] A --> C[sequenceNumber] A --> D[inUse] E[Parent Reference] --> F[parentEntryNumber] E --> G[parentSequenceNumber] E --> H[parentPath] I[Data Characteristics] --> J[recordType] I --> K[ntfsDataType] I --> L[logicalSize] I --> M[physicalSize] N[Attributes] --> O[ntfsFlags] N --> P[standardInformationFlags] Q[Security] --> R[securityId] Q --> S[ownerId] T[Change Tracking] --> U[usn] T --> V[usnJournalId] B --> F C --> G J --> K L --> M O --> P U --> VError Handling
Validation Errors
Consistency Checks
Recovery Procedures
SPARQL Examples
Query 1: Find File Changes in Time Range with USN Journal Correlation
Purpose: Identifies file modifications within a specific timeframe and correlates them with USN Journal entries for change validation.
Use Case: Investigating system changes during a specific incident window.
Query 2: Find Files with Multiple Hard Links
Purpose: Identifies files that have multiple directory entries, useful for understanding file system structure and potential data hiding.
Use Case: Detecting potential data exfiltration through hard links.
Query 3: Track MFT Entry Reuse
Purpose: Analyzes MFT entry reuse patterns to identify potential anti-forensic activities or system usage patterns.
Use Case: Detecting evidence tampering through MFT entry manipulation.
Query 4: Analyze Compressed Files
Purpose: Identifies compressed files with significant size differences, useful for storage analysis and potential hidden data detection.
Use Case: Investigating storage anomalies and compression ratio outliers.
Query 5: Complex File System Analysis
Purpose: Performs comprehensive analysis of file system hierarchy and relationships.
Use Case: Reconstructing file system state and analyzing structural changes.
See End of Change Proposal for Additional SPARQL Query Examples
Performance Considerations
Indexing Strategy
Query Optimization
Data Organization
Turtle Format Representation
[Previous Turtle format content remains the same...]
Implementation Plan
Timeline
Phase 1: Design Review
Phase 2: Implementation
Phase 3: Testing
Phase 4: Documentation and Release
Risk Management
1. Technical Risks
2. Project Risks
3. Operational Risks
Collaboration Tools
References
OWL-in-Turtle Representation
Note: Property names have been adjusted to avoid namespace conflicts and improve clarity:
dataType→ntfsDataTypeflags��ntfsFlagssiFlags→standardInformationFlagsProperties requiring further discussion (timestampResolution, maxVersions, etc.) have been omitted pending clarification of their source and applicability.
Forensic Relevance of Properties
sequenceNumber
maxVersions and versionNumber
timestampResolution
ntfsFlags and standardInformationFlags
logfileSequenceNumber vs usn
nameType
securityId and ownerId
Enhanced Error Handling
Validation Errors
Missing Data Recovery
Inconsistency Resolution
Additional SPARQL Examples
Digital Evidence Investigation Examples
File Provenance Analysis
The following SPARQL query demonstrates how the extended NTFS properties support comprehensive file provenance analysis in digital forensics investigations:
Forensic Analysis Benefits
Evidence Integrity
Timeline Construction
Chain of Custody
Anti-Forensics Detection
Implementation Notes
Data Protection
Performance Optimization
Validation Rules
Security Property Validation
Version Control Validation
Error Handling and Recovery
Missing Required Properties
Invalid Values
Inconsistency Resolution
Change Tracking Properties Clarification
USN Journal Properties
The proposal consolidates USN-related properties to avoid confusion:
usnJournalId(Recommended): References specific USN Journal entries with full contextusn(Deprecated): Simple reference number, replaced byusnJournalIdfor completenessChange Tracking Comparison
Property Optionality
Required Properties
Optional Properties
Future Extensibility
Integration Points
Planned Extensions
Compatibility Notes
The text was updated successfully, but these errors were encountered: