Preferred method for storing Array(s) and List(s) in ObjectStorage

[sbrptdev2]

I'm in the process of writing the session save-restore functionality into one of the classes handling data that should survive across program cycles. It's my understanding that the ObjectStorage put*() methods operate in a FIFO fashion during save() and the inverse during restore().

The overall functionality is fantastic but I have found one limitation: suppose I need to store a sequence (List) of bytes, which normally for a single byte[] instance you can do away with putBytes(). What's the best approach to serialize or store a List or Array of byte[] objects?

• I could also imagine that storing a Map might be beneficial. In the case of a sequence of byte[] objects I think something generic should already exist. Alas I'm either missing it or there isn't one.

• Suppose the extension is to be proposed for merging at a later junction, are other serialize-to-string methods acceptable? It seems like having ObjectStorage and all the effort behind it should at least offer a path that does not involve a kludge like serialized strings.

Also: is there any documentation for extension developers on how to connect the Saveable to a Manager class? Can extensions save data without creating their own DBAdapter-like class? I have been reading the BookmarkTypeDBAdapter and OldBookmarkManager classes to have an idea on how to implement my Saveable class storage in the context of the plugin.

• Is there a way to save a List (collection) of Saveable objects in the ProgramUserData or ProgramDB, without implementing custom DB types?

[sbrptdev2]

@dragonmacher Harassing you again as you seem to know the internals very well. I'm hitting a road block with this specific side of the project. I'm averse to implementing out-of-band serialization when an integrated system exists in place (ProgramDB), also because polluting the user's filesystem is bad SW engineering etiquette.

[sbrptdev2]

@ryanmkurtz Anyone else?

[ryanmkurtz]

Sorry, I don't really know anything about this area.

[sbrptdev2]

No problem. Could someone (preferably the person(s) who wrote the ProgramDB internals) actually comment on this, or even better, write some initial documentation or pointers? Seems like a large number of discussions go unanswered.

[sbrptdev2]

Reanimating this discussion: per #968 it seems I could rely on getProgramUserData() (@ghidra1 might be able to contribute here).

What are the constraints I should be aware of? What examples of Saveable(s) exist that I can use for reference?

The Saveable object I need to integrate/save to the ProgramDB is defined here:
https://github.com/subreption/ghidra_yara/blob/main/ghidra_yara/src/main/java/ghidra_yara/GhidraYaraRule.java

A more specific question: what is the ideal/preferred way to store arrays/lists of Saveable objects when using getProgramUserData as target? (the original overarching question)

The object definition itself is quite straightforward and I believe Ghidra integrates a JSON library I can easily integrate for serialization, if needed.

[ghidra1]

I am rather confused about what you are attempting to store and it's relationship to a Program (e.g., user-specific or stored within program, address-specific or simple string-key). We have a variety of mechanisms for possible adhoc data:

1. Program.getUsrPropertyManager() provides (PropertyMapManager) address-keyed maps for storing various data forms within a program. This data is versioned with a program.
2. Program Options provides key-based name/value data storage within a program that is visible and editable by a user via the GUI (Tool menu Edit->Options for program...). The key names use a dot notation to form a hierarchical mapping. Care must be taken using this as multi-user merge support is not currently supported.
3. Program.getProgramUserData() provides (ProgramUserData) user data storage specific to a program and stored separately from the program database. This provides an independent but related database stored within the user's project. It is important to note that the data should not be too sensitive to the program's state since it is not versioned and will be present when any version of the associated program is opened by the user.
   The biggest concern with the use of Saveable is the potential revision of its schema. It may be appropriate to incorporate a version number into the stored data to facilitate this and support potential data upgrade/migration. There are not many examples of Saveable use since it is problematic for mutli-user merge and schema versioning. It is certainly best suited for cases where it will never change its schema over time.

[sbrptdev2]

Thank you for stepping in! I'm trying to store GhidraYaraRule objects (YARA rules) generated from the plugin itself, which the user can leave there or export them for use across multiple programs inside and outside of Ghidra.

Program.getUsrPropertyManager() provides (PropertyMapManager) address-keyed maps for storing various data forms within a program. This data is versioned with a program.

Program Options provides key-based name/value data storage within a program that is visible and editable by a user via the GUI (Tool menu Edit->Options for program...). The key names use a dot notation to form a hierarchical mapping. Care must be taken using this as multi-user merge support is not currently supported.

Options are a different issue, in the case of GhidraYara some program-specific settings might make sense, but thus far I have failed to find any settings needed outside of the analyzer, which is already handled properly along the rest of the analyzer settings. For now, it's mostly corner cases or performance problems when analyzing large binaries (100MB static ELF executables for example), where the monolithic scanner seems to fail. I have a regression with 11.2 where the plugin seems to fail silently when using monolithic mode (loads the entire binary in a contiguous buffer, instead of doing block by block scanning through the memory map regions), but need to verify this on my own.

Regarding schema: so, the TL;DR is as follows: each GhidraYaraRule is backed by a TableModel and a List, all I need is to store the rules for the current program (generated and adjusted by the user) in the ProgramDB. The GhidraYaraRule can be serialized to a string (all YARA rules are strings in the end).

	@Override
	public void save(ObjectStorage objStorage) {
		objStorage.putString(ruleName);
		objStorage.putString(author);
		objStorage.putString(description);

(...)
		objStorage.putStrings((String[]) tags.toArray());

		saveMetaMap(objStorage);

		List<String> encodedBytes = byteArrays.stream().map(ByteArrayUtils::encode).collect(Collectors.toList());

		objStorage.putStrings((String[]) encodedBytes.toArray());
	}

I understand that the order for restore/save has to be exactly the same. How can I store a List of these Saveable objects then?
The ObjectPropertyMap class does not seem to be the right way to do this.

The YARA rules are technically tied to the lifetime of the forms that render the table listing them (as managed by the user), so they should be restored/saved to allow the table to be populated correctly when the program is reopened/closed.

[ghidra1]

Program Options are not restricted to analyzer use only and would seem to make the most sense for this if you want them to be stored within the program's database using a storage key. They provide the most flexibility, particularly if associated with a GUI Plugin which can register a complex editor to view and manipulate the Option. See StoredAnalyzerTimes for an example which employs a CustomObject and editor associated with the AutoAnalysisPlugin (see programActivate(Program) plugin method used to register the option and its editor). The CustomObject provides readState and writeState methods using GProperties for storage access.

If you choose to use Saveable with its ObjectStorage, I suggest employing a String serialization format for each rule and using an array of string for storage. My only concern would be the use of either 1) an address-based property map, which seems klunky since you would be using an arbitrary address key and allocating a DB table for the map, or 2) locally-stored user data which is not stored within the program database (which is certanily an option).

Since a Program database is intended to be shared with other users via a Ghidra Server, deciding to expose or not expose the YARA data to other users of the Program is a key deciding factor.