I want to validate my mission and performance analysis tool. Please guide.

[Ushahid05]

Hi,
I am a student of masters and I am working on the development of mission and performance analysis tool. I need to validate it. Can anyone please guide me about it? I was trying searching the relevant validation data in the Aviary documentation but its so vast and I am unable to find a consolidated document where I could have specific engine decks, aero deck, geometrical properties and mission profiles along with results. Though, I am able to find some engine decks but I am not sure which mission profile make use of which engine and aero data.
Moreover, If anyone can suggest me research paper or any published report using which I could validate my tool, I will be really thankful.

Thankyou in advance.

Best Regards,

[jkirk5]

Hey there, thanks for reaching out!

Tool validation is a tricky subject and different organizations have their own opinions on methods and levels of rigor. In general, I would start by validating individual parts of your code against other codes and real-world sources. If you are doing any vehicle sizing (mass, geometry, etc.) I suggest validating those parts separately. Then validate the mission analysis part of your code by comparing with other tools without any sizing - fixed mass, no resizing of propulsion system, etc. Aviary can be an individual tool to validate your own against, or trying out the legacy tools Aviary is build on (FLOPS and GASP). It shouldn't be your only source of "truth". In my opinion, you should always include real-world data in your validation exercise.

To help out with running our examples, you can start out here.
The information on what propulsion, aero data, etc. goes with the aircraft is in the .csv file that defines the model. All of our examples use the same mission definition file, which Aviary automatically defaults to if you don't specify one. To run one of our examples (the .csv files inside aviary/models/{example_name}), the easiest way is using the command line: aviary run_mission models/{example_name}/{example_filename}.csv

A example command is aviary run_mission models/small_single_aisle/small_single_aisle_GwGm.csv. When writing this up I actually found a bug preventing directly calling the examples by name, so you'll have to use the filepath for now.

To review results, run aviary dashboard small_single_aisle_GwGm after the run is finished

[HamzaAwan1538]

Hi @jkirk5,

Thank you for your detailed comment on this issue. I encountered the same problem while performing validation. When I attempted to run your command, it prompted me to install pyoptsparse first. After successfully cloning the repository, I ran the mission using the provided command, but it seems to require SNOPT, which isn't available as open-source.

Could you kindly suggest an alternative to SNOPT that I can use? Additionally, can you guide me on whether there’s an option to use an open-source optimizer like SLSQP, IPOPT, or any other alternatives? Alternatively, is it possible to simply turn off the optimization process altogether?

Best regards,

[Ushahid05]

Hi Jason, @jkirk5

Thank you for the insights you've shared regarding validation and using Aviary. However, to ensure my validation process is robust and credible in front of my professors and committee, I would like to refer to more formal documentation or source material that outlines the expected results for certain analyses, such as:

1. Aerodeck data: Can you point me to any resources that show the standard or typical aerodynamic data for the aircraft models used in Aviary?

2. Engine deck data : It would be helpful to have references to documented engine performance parameters or data, as used within the Aviary framework.

3. Mission profile: I need to cite a reliable mission profile for the models and analysis conducted with Aviary, so I can compare against other sources.

4. Mission analysis results: Is there a report, publication, or documentation that outlines the results of mission analyses that I can use to cross-check my findings?

Having access to this type of reference material would be incredibly helpful, as it will allow me to support the validation of my methodology in a manner that avoids ambiguity or potential errors in running the tool myself.

Thank you again for your help!

Best regards,

[jkirk5]

Hi @jkirk5,

Thank you for your detailed comment on this issue. I encountered the same problem while performing validation. When I attempted to run your command, it prompted me to install pyoptsparse first. After successfully cloning the repository, I ran the mission using the provided command, but it seems to require SNOPT, which isn't available as open-source.

Could you kindly suggest an alternative to SNOPT that I can use? Additionally, can you guide me on whether there’s an option to use an open-source optimizer like SLSQP, IPOPT, or any other alternatives? Alternatively, is it possible to simply turn off the optimization process altogether?

Best regards,

@HamzaAwan1538, Aviary supports SLSQP and IPOPT as open-source alternatives. run_mission has an --optimizer argument that allows you to select which optimizer you'd like to use.

A lot of things default to SNOPT right now because in the early days of Aviary it was one of the only optimizers that could reliably converge. Nowadays things are more polished and the open-source optimizers are usually sufficient. The 2 degree-of-freedom missions are more complex and SNOPT is helpful, the energy-method missions usually converge with any optimizer. The FLOPS-based energy-method example models don't have .csv files yet so they can't be run from the command line right now. Here's an example of doing it via a Python script.

from aviary.api import run_aviary, default_height_energy_phase_info
from aviary.models.large_single_aisle_1.large_single_aisle_1_FLOPS_data import inputs

run_aviary(inputs, default_height_energy_phase_info, optimizer="IPOPT")

Right now use of an optimizer is required in most cases because Aviary uses a global optimization technique to solve for trajectories. There is a shooting method available that does not use an optimizer, but the implementation is not complete and it has limitations on what kinds of problems it can solve. https://openmdao.github.io/Aviary/user_guide/SGM_capabilities.html

[jkirk5]

@Ushahid05, unfortunately we haven't published a "canon" set of results for any of our Aviary example and reference models. The type of information you want for results can currently only be gotten through running the example models in Aviary and pulling the results you want out of the dashboard.

1. Aero Data Our example models use empirical methods to compute drag data during aircraft sizing - because it's generated at run-time, we don't have that data available in the repository. At some point we plan to have the option to output the table of drag data directly but that isn't available at the moment.
2. Engine Data All the engine data tables used by our example models are directly available under aviary/models/engines
3. Mission Profile After running a case, mission profile graphs are generated by the dashboard. You can save those plots as images, and the trajectory data itself can be pulled from the dashboard's results/Timeseries Mission Output tab.
4. Mission Results We haven't published any results that include the full data for mission profiles - some of our validation exercises are shown here (https://ntrs.nasa.gov/citations/20240007608) but we only show plots in the paper. The models in that paper are the examples provided in the repository, although they may have slightly changed since that paper as we continued to polish those models and fix bugs in the code.

Sorry we don't have the data you'd like available in any publications. Running the example models with IPOPT should be able to generate repeatable results, but I understand that may not be sufficient for your committee.

[jkirk5]

The FLOPS tool also comes with some example models, and those are more "stable" than our Aviary set (I don't think those have been modified in years, possibly decades). If you request a copy from software.nasa.gov that may be more acceptable as a reference model, although those don't have published results anywhere either. At the very least they don't require an optimizer and as such the results are repeatable across any setup, whereas the Aviary results will change based on choice of optimizer. Good luck on your project!