QuestieDB organizes a variety of data types crucial for the functionality of the Questie addon. The database is structured as a Lua table, where each entry is keyed by a unique identifier (ID) such as QuestId, ItemId, etc. These IDs are generated by an external tool and are guaranteed to be unique.
The source data is generated by running the createStatic.lua script in the .generate_database folder. The script will then generate the <DATATYPE>Data.lua-table files into it's correct directory in the Database folder structure.
The corrections system in QuestieDB allows for modifying and extending the generated data. Corrections are loaded into the database and use various dump functions defined in the corresponding "Meta.lua" files.
The main components of the corrections system are:
-
Corrections.lua: This file serves as the central hub for registering and applying corrections. It provides functions for registering static and dynamic corrections for different data types. -
<DATATYPE>Meta.lua(e.g., ItemMeta.lua): These files define the structure and metadata for each data type. They specify the keys, types, and dump functions for the data. The dump functions determine how the data is processed and formatted when applying corrections. -
<EXP>ItemFixes.lua(e.g., classicItemFixes.lua): These files contain the actual corrections for each data type in a specific era (e.g., classic). They define the modifications to be made to the generated data, such as adding or overriding values for specific entries.
The corrections are applied during the database initialization process. The registered corrections are loaded and merged with the generated data, allowing for customization and fixes to be made without modifying the original source data.
For example, in ItemMeta.lua, you'll find the structure definition for item data, including keys like name, npcDrops, objectDrops, etc. The file also defines dump functions for each key, specifying how the data should be processed.
The classicItemFixes.lua file contains the actual corrections for items in the classic era. It defines modifications to specific item entries, such as adding or overriding drop sources, quest relations, and other properties.
Dump functions are used to process and format the data when applying corrections. They are defined in the DumpFunctions.lua file and are utilized in the <DATATYPE>Meta.lua files.
The purpose of dump functions is to convert the data from its raw format into a specific representation that can be used by the Questie addon. They handle tasks such as converting values to strings, formatting arrays, and ensuring data consistency.
The DumpFunctions.lua file provides a set of predefined dump functions that can be used for common data types. These functions include:
dump: A generic function that converts a value to a string representation.dumpAsArray: Converts an array of values into a string representation.dumpAsArraySorted: Converts an array of values into a sorted string representation.combine: Combines multiple values into a single string representation based on specified types. In the<DATATYPE>Meta.luafiles, the dump functions are assigned to specific keys in thedumpFuncstable. This allows for customizing how each key's value is processed and formatted when applying corrections.
For example, in ItemMeta.lua, the dumpFuncs table associates the name key with the dump function, the npcDrops key with the dumpAsArraySorted function, and so on. This ensures that the corresponding values are properly formatted when corrections are applied.
Never done, but my guess would be to stand in the root folder and execute the lua file from there.
An easy way to run it on any OS is to just run the docker-compose.yml file, it is important that you run it in the .generate_database folder for relative paths to work correctly.
The Lua table is a large collection of data entries, each corresponding to a specific game element like quests, items, NPCs, objects, and translations. The structure of an entry is as follows:
{
[uniqueID] = {data1, data2, data3, ..., dataN},
[31] = {'Old Lion Statue',{248,249},{94},{[44]={{84.49,46.83}}},44,},
[32] = {'Sunken Chest',nil,nil,{[44]={{41.52,54.66}}},44,},
[33] = {'Locked Chest',nil,{140},{[40]={{25.97,16.91}}},40,},
-- More entries follow...
}Each data index within an entry holds specific information:
- Index 1: Name of the item or quest.
- Index 2: Table containing information about quest starters (for quests) or similar metadata for other data types.
- ... (additional indexes hold other predefined types of data)
The order of data within each entry is critical. The database treats each index as a fixed position for a specific type of data, as defined in a meta file within the addon. If an index is removed or altered, the subsequent data would shift, leading to incorrect data mapping. For example, if the 'Name' at index 1 is removed, the 'Quest Starters' data intended for index 2 would erroneously occupy index 1. To prevent such issues, nil values are used as placeholders to maintain the correct order of data.
All data is stored as strings within the database files, with the exception of pattern values, which are concatenated using a semicolon (;) as a delimiter. This raw string format allows for a consistent method of data storage, which the addon later interprets and converts into appropriate Lua data types.
QuestieDB serves as a static database injected into the addon, providing comprehensive information about game elements. A Python script reads this Lua table and processes it into a SimpleHTML structure, which is then utilized by the addon to load data on demand.
SimpleHTML is a frame type in World of Warcraft's UI system typically used for displaying HTML content. In QuestieDB, we leverage SimpleHTML in an unconventional way to store and retrieve data on demand. The data is not loaded into memory until explicitly called, which optimizes performance and memory usage.
To access the data, we use hidden SimpleHTML frames—never visible to the user—created with the frame:hide() method. The GetRegions() function is employed to extract all <p> tags from a SimpleHTML template file, effectively reading each line of data. Each <p> tag corresponds to a region, and we encapsulate the return values in a table using {frame:GetRegions()}.
QuestieDB's data files are sequentially ordered, but to manage sparse datasets, we utilize an "Index to Id" mapping. This allows us to skip to the desired data index efficiently. When an ID is loaded, we cache the corresponding GetText() function handles in a table, enabling quick access to data without reloading.
Initially, frames were loaded at startup, but this caused noticeable delays. To enhance user experience, we now load frames on demand, significantly reducing startup time. When a data file is accessed, we cache all IDs within it, which minimizes load times during gameplay as we rarely need to access every data entry in a session.
The World of Warcraft client imposes specific limitations on SimpleHTML content, restricting <p> tags to a maximum of 4000 characters and HTML files to 45000 characters. To adhere to these constraints and maintain efficient load times, we limit the number of <p> tags per file, with a self-imposed cap of 50 IDs per file due to a kind of exponential increase in load times when too many <p> tags are present in a file.
QuestieDB organizes its data into distinct files, each serving a specific purpose and structured to facilitate efficient data management and retrieval. These files are akin to tables in a relational database, with each file type tailored to handle different aspects of the data.
The Id File is an essential index within QuestieDB, cataloging all data entries for a specific data type. Each data type, such as quests or items, has its own Id File, named using the convention <DATANAME>DataIds.html (e.g., QuestDataIds.html for quests).
The Id File is an HTML document with a <body> element that contains multiple <p> tags. Due to character length limitations imposed by the game client, the list of IDs may be split across several <p> tags. However, each <p> tag contains the same comma-separated list of IDs, ending with a trailing comma. During runtime, the Questie addon concatenates the text from all <p> tags to form the complete list of IDs.
<!-- QuestDataIds.html example for quest data -->
<html><body>
<p>1001,1002,1003,1004,</p>
<p>1001,1002,1003,1004,</p>
<!-- Additional <p> tags may follow if the list is long -->
</body></html>The Id File acts as a static index for the addon to reference and verify the presence of data entries. It is generated from the Lua source table through a Python script, ensuring an accurate reflection of the source data. When the addon needs to access the data, it retrieves the text from all <p> tags within the Id File, combines them into a single string, and then extracts the individual IDs.
The "File Names" segment of QuestieDB serves as a crucial reference that maps data types to their corresponding data files stored on disk. This mapping is necessary because the addon cannot directly list files on the disk; instead, it relies on predefined frame names to load the appropriate data.
The file "ItemdataTemplates.html" contains a list of frame names, each corresponding to a specific data file. These frame names are defined in an XML file and follow a naming convention that includes the data type and the ID range covered by the data file. The format is <DATATYPE>Data<START_ID>-<END_ID>, for example, "ItemData25-37".
<!-- This contains all the ranges for the files that are generated -->
<html><body>
<p>ItemData25-37,ItemData38-51</p>
</body></html>During runtime, when the addon needs to access data for a particular ID, it performs a binary search on the list of frame names to find the closest matching file name that includes the requested ID range. This method allows for quick and efficient data retrieval.
If a user requests information about Item 26, the addon uses a binary search to locate the frame name "ItemData25-37". It then uses the CreateFrame function in-game to load the data from the corresponding file on disk.
Data files in QuestieDB are meticulously organized to store and manage the game element data, such as quests, items, NPCs, and more. Each file is named to reflect a range of IDs but contains only the specific IDs present within that range, as the data can be sparse.
Each data file begins with an index list, detailing the exact IDs included in the file. This list is crucial for the addon to identify which IDs are available without loading the entire file into memory. Following the index list, the file is divided into "chunks," with each chunk representing a single data entry.
A chunk is structured as follows:
- The first
<p>tag contains a comma-separated list of data indexes, indicating the types of data stored for the corresponding ID. - Subsequent
<p>tags contain the actual data, one piece per tag, in the order specified by the data index.
Data within the files is stored as strings, which are later converted into appropriate Lua types using various methods:
- Tables are encapsulated in strings and loaded using the
loadstring()function, which interprets the string as a Lua table. - Numbers are converted from strings using the
tonumber()function. - Pattern-values, which store multiple values separated by semicolons, are split using Lua patterns.
- Strings require no conversion as they are already in the correct format.
To maintain a balance between the amount of data per file and performance, we impose a limit of approximately 50 IDs per file. This cap helps prevent the exponential increase in load times that occurs with an excessive number of <p> tags.
Here's an example of what a data file might look like:
File: "25-37.html"
<html><body>
<!-- Index to Id table -->
<p>25,35,36,37</p>
<!-- Data for ID 25 -->
<p>1,7</p>
<p>Worn Shortsword</p>
<p>0;0;2;1;0;2;7</p>
<!-- Data for ID 35 -->
<p>1,7</p>
<p>Bent Staff</p>
<p>0;0;2;1;0;2;10</p>
<!-- Data for ID 36 -->
<p>1,7</p>
<p>Worn Mace</p>
<p>0;0;2;1;0;2;4</p>
<!-- Data for ID 37 -->
<p>1,7</p>
<p>Worn Axe</p>
<p>0;0;2;1;0;2;0</p>
</body></html>In this example, each ID has a corresponding set of data indexes followed by the data itself. The pattern-values are delimited by semicolons and will be split into separate values when processed by the addon.
Meta example:
<html><body>
<!-- Index to Id table -->
<p><-- All Ids that exist in the file --></p>
<!-- first ID -->
<p><-- All Data Index for the specific Id--></p>
<p><-- Data index 1 --></p>
<p><-- Data index 2 --></p>
<p>...</p>
<!-- second ID -->
<p><-- All Data Index for the second specific Id--></p>
<p><-- Data index 1 --></p>
<p><-- Data index 2 --></p>
<p>...</p>
</html></body>QuestieDB stores all game element data as strings within its files, ensuring a consistent format for easy parsing. When the addon retrieves this data, it converts the strings back into their original Lua data types using specific methods:
- Tables: Encapsulated as strings, tables are reconstructed using the
loadstring()function, which interprets the string as executable Lua code to recreate the table structure. - Numbers: Numeric strings are converted to actual numbers with the
tonumber()function. - Pattern-Values: A custom format where multiple values are stored in a single string, separated by semicolons (
;). For example, the string"0;0;2;1;0;2;0"represents a sequence of values. These are split into separate values using Lua patterns during processing. - Strings: As the native format, strings require no conversion.