Review of new TOM creation process and documentation

[rachel3834]

The purpose of this issue is to review the creation of a basic TOM following the documentation, to identify any broken features or missing documentation. Items that have been checked off have been tested and found to work.

For reference, this testing was done on a Macbook Pro running OS/X Sonoma, using Python 3.11 in a new virtual environment.

Documentation

• PRIORITY Getting started doc should have a more obvious button at the top of the TOM's readthedocs page and/or a 'Quick start' link in the navbar.
• Github README should list development team and add a section on how to cite/acknowledge the Toolkit
• Readthedocs should have a clear section on how to cite the Toolkit (inc plugins crediting contributors)
• Getting started guide indicates python >=3.7 is required - is this still current? Do we still support 3.6?
• Getting started guide: "./manage.py tom_setup.py" will only work if the user has cd'd into the project's top level directory first. This step should be explained.
• Getting started guide procedure works cleanly. But the user can't do much in the TOM without logging in - this step should be described at the end.
• Documentation could use a new page describing Targets, and how the TOM handles different parameters. For example, the page on TOM Targets gives useful detailed references but no overall overview of what a Target is, and the parameters that are stored (i.e. the attributes on the Target model) and what are not - and why it was designed this way. This leads into a discussion of the role of Tags and Extra Parameters and what the difference is.
• Documentation needs a description of Target Groups and how to use them.
• PRIORITY Documentation needs a high-level page describing the overall principle of how observatories interact with the TOM, and how the authentication and proposal information is configured in the settings. Then it needs to link to pages on how to configure each of the default supported observatories, and the observing modes that they offer. Gemini in particular since the template system (and how to create them) isn't obvious.
• Documentation on LCO's observing modes, and how to use the TOM's forms to configure them. For example, the observing request form offers me "configurations" and "instrument configurations" - what's the difference?
• What is the difference between "observations" and "observation groups" under the TOM's top navbar options for the
  "Observations" menu? Not mentioned in the documentation.
• Add documentation section to describe TOM Users and User Groups. This should include a description of what the AnonymousUser is for.

Functionality

Target Import

• Manual creation of non-sidereal target works
• Target input via SIMBAD catalog search works
• Target input via NED catalog search works
• add help text for TNS and JPL Horizons Target upload #780
• TNS search form needs user guidance on how to configure access to it, and a page dedicated to it in the readthedocs. Typing 'AT2023xys' into the search box raised an error indicating that I was unauthorized to use it. This directs me to a relevant URL in the help docs (which is good) but that page doesn't tell me how to configure TNS in my TOM so that the search can be done.
• Downloading the example file of targets in CSV format for ingest works. Desirable extra: add examples of Solar System targets.
• After a target is ingested, and the TOM is displaying the TargetPage, clicking on the Target pulldown menu in the navbar doesn't work.

TargetListPage

• PRIORITY Update Broker Data Crashes in tom-demo-dev #792

TargetDetailPage

• Observe tab: I have no Gemini observing templates configured yet when I click the "GEM" button I get to an apparently-valid observation request form. This should present a warning with a direction to the user to the documentation on how to configure this.
• Observation request form has no "back to the Target you started from" button.
• PRIORITY Clicking the LCO, SOAR and LT buttons without having a proposal configured crashes the TOM; this should be handled gracefully with a warning.
• When I click on the LCO observation request button, there is a noticable pause in loading the form. Some kind of "Contacting the observatory for your proposal information, please wait" message would help a lot here.

LCO Observation Request Form

• Start and end fields should have time as well as date, and the fields should indicate that these are UTC.
• PRIORITY Form for spectrum request for FLOYDS does not work: I requested a set of configs for arc+flat+spectrum+arc+flat with a 1.2arcsec slit, and I get the following error "non_field_errors: optical element slit_1.0as of type slit is not available". I guess this is because the form has the default set of instrument configs in the JSON dictionary and 1.0arcsec slit is the default (which FLOYDS doesn't have?).
• PRIORITY Spectroscopic sequence tab should be removed (or hidden by default) from the form because it can't work - there is no way to request the required arcs and flats, which can mislead users into wasting observing time on observations they will not be able to use. The user is currently unable to create a template with spectroscopic calibration frames.

Alerts

• Alert broker query results table should offer the user a "select all" checkbox
• Naive query for "All SN1a" in Alerce works
• Add Antares Broker Documentation in light of Python 3.10 incompatibility. #777
• Gaia broker queries on name and coordinates work
• Workflow for Lasair query works, including information on where to get account info.
• PRIORITY Scout query composition form: link to Scout API reference docs is broken. Without this I can't really test the form; it presents very limited options. #772
• PRIORITY TNS query form should warn the user if they do not have account credentials rather than crashing the TOM when the query is run. #774
• PRIORITY I got a warning when I tried to install tom_fink (documented below), followed by an error which means I cannot now runserver. This was corrected only by uninstalling tom_fink, but many users wouldn't know that - so this looks like a serious break to them.
• Alert broker queries search functionality works.

Observations

• Under Observation Templates, the buttons to create observation templates for LT and GEM crash. These should gracefully indicate this functionality is not available for these Facilities.
• Form to create an Observation Template for LCO and SOAR lists all of the user's proposals from the LCO Portal, regardless of whether they apply to the template concerned. For example, my LCO proposals should not be listed when I'm created a template for SOAR because that is invalid. If the user has no proposal for a facility, then ideally the option to create a template for that facility should be greyed out or otherwise discouraged.
• The Observation Template form allows me to create non-sensical templates, for example to request slit configurations for an imager.
• Search functionality works for list of observation templates
• Priority Observation Groups -- Create new Grouping crashes #791
• LCO observations do not show status immediately (need update status)
• add existing observation (Target Detail page) redirects to default tab instead of observations tab
• when adding an observation (target detail page) prompt text should be more precise ("Observation ID should be "Request ID" for OCS facilities.)

Users

• Create/update/delete user works, including change password
• Create/update/delete works for User Groups, including add/delete users
• User profile update works

[rachel3834]

Also:

• Add section to documentation describing how users can implement test code for their TOM.

[jchate6]

"After a target is ingested, and the TOM is displaying the TargetPage, clicking on the Target pulldown menu in the navbar doesn't work."
is actually fixed in the most recent TOM release: tomtoolkit 2.15.13 from issue #761

[jchate6]

Warning installing tom_fink

...
Attempting uninstall: django-bootstrap4
Found existing installation: django-bootstrap4 23.2
Uninstalling django-bootstrap4-23.2:
Successfully uninstalled django-bootstrap4-23.2
Attempting uninstall: tomtoolkit
Found existing installation: tomtoolkit 2.15.12
Uninstalling tomtoolkit-2.15.12:
Successfully uninstalled tomtoolkit-2.15.12
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
crispy-bootstrap4 2023.1 requires django-crispy-forms>=2.0, but you have django-crispy-forms 1.14.0 which is incompatible.
Successfully installed django-bootstrap4-22.3 django-crispy-forms-1.14.0 django-filter-22.1 elasticsearch-7.17.9 elasticsearch-dsl-7.4.1 pillow-9.5.0 tom-fink-0.4.5 tomtoolkit-2.12.1 urllib3-1.26.18

SUBSEQUENT ERROR:
When I subsequently attempt to runserver following this warning, the TOM crashes:
Watching for file changes with StatReloader
Watching for file changes with StatReloader
Performing system checks...

registering new views: args: ('groups', <class 'tom_common.api_views.GroupViewSet'>, 'groups'), kwargs: {}
registering new views: args: ('targets', <class 'tom_targets.api_views.TargetViewSet'>, 'targets'), kwargs: {}
registering new views: args: ('targetextra', <class 'tom_targets.api_views.TargetExtraViewSet'>, 'targetextra'), kwargs: {}
registering new views: args: ('targetname', <class 'tom_targets.api_views.TargetNameViewSet'>, 'targetname'), kwargs: {}
Exception in thread django-main-thread:
Traceback (most recent call last):
File "/Users/rstreet1/software/tom_demo_venv/lib/python3.11/site-packages/django/template/utils.py", line 69, in getitem
return self._engines[alias]
~~~~~~~~~~~~~^^^^^^^
KeyError: 'django'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/threading.py", line 1038, in _bootstrap_inner
self.run()
File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/threading.py", line 975, in run
self._target(*self._args, **self._kwargs)
File "/Users/rstreet1/software/tom_demo_venv/lib/python3.11/site-packages/django/utils/autoreload.py", line 64, in wrapper
fn(*args, **kwargs)
File "/Users/rstreet1/software/tom_demo_venv/lib/python3.11/site-packages/django/core/management/commands/runserver.py", line 133, in inner_run
self.check(display_num_errors=True)
File "/Users/rstreet1/software/tom_demo_venv/lib/python3.11/site-packages/django/core/management/base.py", line 485, in check
all_issues = checks.run_checks(
^^^^^^^^^^^^^^^^^^
File "/Users/rstreet1/software/tom_demo_venv/lib/python3.11/site-packages/django/core/checks/registry.py", line 88, in run_checks
new_errors = check(app_configs=app_configs, databases=databases)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/rstreet1/software/tom_demo_venv/lib/python3.11/site-packages/django/contrib/admin/checks.py", line 78, in check_dependencies
for engine in engines.all():
^^^^^^^^^^^^^
File "/Users/rstreet1/software/tom_demo_venv/lib/python3.11/site-packages/django/template/utils.py", line 94, in all
return [self[alias] for alias in self]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/rstreet1/software/tom_demo_venv/lib/python3.11/site-packages/django/template/utils.py", line 94, in
return [self[alias] for alias in self]
~~~~^^^^^^^
File "/Users/rstreet1/software/tom_demo_venv/lib/python3.11/site-packages/django/template/utils.py", line 85, in getitem
engine = engine_cls(params)
^^^^^^^^^^^^^^^^^^
File "/Users/rstreet1/software/tom_demo_venv/lib/python3.11/site-packages/django/template/backends/django.py", line 24, in init
options["libraries"] = self.get_templatetag_libraries(libraries)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/rstreet1/software/tom_demo_venv/lib/python3.11/site-packages/django/template/backends/django.py", line 42, in get_templatetag_libraries
libraries = get_installed_libraries()
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/rstreet1/software/tom_demo_venv/lib/python3.11/site-packages/django/template/backends/django.py", line 116, in get_installed_libraries
return {
^
File "/Users/rstreet1/software/tom_demo_venv/lib/python3.11/site-packages/django/template/backends/django.py", line 116, in
return {
^
File "/Users/rstreet1/software/tom_demo_venv/lib/python3.11/site-packages/django/template/backends/django.py", line 105, in get_template_tag_modules
for name in get_package_libraries(pkg):
File "/Users/rstreet1/software/tom_demo_venv/lib/python3.11/site-packages/django/template/backends/django.py", line 128, in get_package_libraries
module = import_module(entry[1])
^^^^^^^^^^^^^^^^^^^^^^^
File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/importlib/init.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "", line 1204, in _gcd_import
File "", line 1176, in _find_and_load
File "", line 1147, in _find_and_load_unlocked
File "", line 690, in _load_unlocked
File "", line 940, in exec_module
File "", line 241, in _call_with_frames_removed
File "/Users/rstreet1/software/tom_demo_venv/lib/python3.11/site-packages/tom_dataproducts/templatetags/dataproduct_extras.py", line 19, in
from tom_dataproducts.forms import DataProductUploadForm, DataShareForm
File "/Users/rstreet1/software/tom_demo_venv/lib/python3.11/site-packages/tom_dataproducts/forms.py", line 48, in
class DataProductUploadForm(forms.Form):
File "/Users/rstreet1/software/tom_demo_venv/lib/python3.11/site-packages/tom_dataproducts/forms.py", line 60, in DataProductUploadForm
widget=forms.ClearableFileInput(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/rstreet1/software/tom_demo_venv/lib/python3.11/site-packages/django/forms/widgets.py", line 427, in init
raise ValueError(
ValueError: ClearableFileInput doesn't support uploading multiple files.
Traceback (most recent call last):
File "/Users/rstreet1/software/tom_demo_venv/lib/python3.11/site-packages/django/template/utils.py", line 69, in getitem
return self._engines[alias]
~~~~~~~~~~~~~^^^^^^^
KeyError: 'django'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/rstreet1/software/tom_demo/./manage.py", line 22, in
main()
File "/Users/rstreet1/software/tom_demo/./manage.py", line 18, in main
execute_from_command_line(sys.argv)
File "/Users/rstreet1/software/tom_demo_venv/lib/python3.11/site-packages/django/core/management/init.py", line 442, in execute_from_command_line
utility.execute()
File "/Users/rstreet1/software/tom_demo_venv/lib/python3.11/site-packages/django/core/management/init.py", line 436, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/Users/rstreet1/software/tom_demo_venv/lib/python3.11/site-packages/django/core/management/base.py", line 412, in run_from_argv
self.execute(*args, **cmd_options)
File "/Users/rstreet1/software/tom_demo_venv/lib/python3.11/site-packages/django/core/management/commands/runserver.py", line 74, in execute
super().execute(*args, **options)
File "/Users/rstreet1/software/tom_demo_venv/lib/python3.11/site-packages/django/core/management/base.py", line 458, in execute
output = self.handle(*args, **options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/rstreet1/software/tom_demo_venv/lib/python3.11/site-packages/django/core/management/commands/runserver.py", line 111, in handle
self.run(**options)
File "/Users/rstreet1/software/tom_demo_venv/lib/python3.11/site-packages/django/core/management/commands/runserver.py", line 118, in run
autoreload.run_with_reloader(self.inner_run, **options)
File "/Users/rstreet1/software/tom_demo_venv/lib/python3.11/site-packages/django/utils/autoreload.py", line 671, in run_with_reloader
start_django(reloader, main_func, *args, **kwargs)
File "/Users/rstreet1/software/tom_demo_venv/lib/python3.11/site-packages/django/utils/autoreload.py", line 660, in start_django
reloader.run(django_main_thread)
File "/Users/rstreet1/software/tom_demo_venv/lib/python3.11/site-packages/django/utils/autoreload.py", line 343, in run
autoreload_started.send(sender=self)
File "/Users/rstreet1/software/tom_demo_venv/lib/python3.11/site-packages/django/dispatch/dispatcher.py", line 176, in send
return [
^
File "/Users/rstreet1/software/tom_demo_venv/lib/python3.11/site-packages/django/dispatch/dispatcher.py", line 177, in
(receiver, receiver(signal=self, sender=sender, **named))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/rstreet1/software/tom_demo_venv/lib/python3.11/site-packages/django/template/autoreload.py", line 43, in watch_for_template_changes
for directory in get_template_directories():
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/rstreet1/software/tom_demo_venv/lib/python3.11/site-packages/django/template/autoreload.py", line 16, in get_template_directories
for backend in engines.all():
^^^^^^^^^^^^^
File "/Users/rstreet1/software/tom_demo_venv/lib/python3.11/site-packages/django/template/utils.py", line 94, in all
return [self[alias] for alias in self]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/rstreet1/software/tom_demo_venv/lib/python3.11/site-packages/django/template/utils.py", line 94, in
return [self[alias] for alias in self]
~~~~^^^^^^^
File "/Users/rstreet1/software/tom_demo_venv/lib/python3.11/site-packages/django/template/utils.py", line 85, in getitem
engine = engine_cls(params)
^^^^^^^^^^^^^^^^^^
File "/Users/rstreet1/software/tom_demo_venv/lib/python3.11/site-packages/django/template/backends/django.py", line 24, in init
options["libraries"] = self.get_templatetag_libraries(libraries)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/rstreet1/software/tom_demo_venv/lib/python3.11/site-packages/django/template/backends/django.py", line 42, in get_templatetag_libraries
libraries = get_installed_libraries()
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/rstreet1/software/tom_demo_venv/lib/python3.11/site-packages/django/template/backends/django.py", line 116, in get_installed_libraries
return {
^
File "/Users/rstreet1/software/tom_demo_venv/lib/python3.11/site-packages/django/template/backends/django.py", line 116, in
return {
^
File "/Users/rstreet1/software/tom_demo_venv/lib/python3.11/site-packages/django/template/backends/django.py", line 105, in get_template_tag_modules
for name in get_package_libraries(pkg):
File "/Users/rstreet1/software/tom_demo_venv/lib/python3.11/site-packages/django/template/backends/django.py", line 128, in get_package_libraries
module = import_module(entry[1])
^^^^^^^^^^^^^^^^^^^^^^^
File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/importlib/init.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "", line 1204, in _gcd_import
File "", line 1176, in _find_and_load
File "", line 1147, in _find_and_load_unlocked
File "", line 690, in _load_unlocked
File "", line 940, in exec_module
File "", line 241, in _call_with_frames_removed
File "/Users/rstreet1/software/tom_demo_venv/lib/python3.11/site-packages/tom_dataproducts/templatetags/dataproduct_extras.py", line 19, in
from tom_dataproducts.forms import DataProductUploadForm, DataShareForm
File "/Users/rstreet1/software/tom_demo_venv/lib/python3.11/site-packages/tom_dataproducts/forms.py", line 48, in
class DataProductUploadForm(forms.Form):
File "/Users/rstreet1/software/tom_demo_venv/lib/python3.11/site-packages/tom_dataproducts/forms.py", line 60, in DataProductUploadForm
widget=forms.ClearableFileInput(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/rstreet1/software/tom_demo_venv/lib/python3.11/site-packages/django/forms/widgets.py", line 427, in init
raise ValueError(
ValueError: ClearableFileInput doesn't support uploading multiple files.

[jchate6]

Error installing tom_antares

...
In file included from /private/var/folders/4z/rk7n9jss74g3shm9z6msmyvh0000gp/T/pip-install-clvss3og/confluent-kafka_c4be6e80a1524b3fa91b9ab93fbddaef/src/confluent_kafka/src/Admin.c:17:
/private/var/folders/4z/rk7n9jss74g3shm9z6msmyvh0000gp/T/pip-install-clvss3og/confluent-kafka_c4be6e80a1524b3fa91b9ab93fbddaef/src/confluent_kafka/src/confluent_kafka.h:23:10: fatal error: 'librdkafka/rdkafka.h' file not found
#include <librdkafka/rdkafka.h>
^~~~~~~~~~~~~~~~~~~~~~
1 error generated.
error: command '/usr/bin/clang' failed with exit code 1
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for confluent-kafka
Building wheel for bson (pyproject.toml) ... done
Created wheel for bson: filename=bson-0.5.10-py3-none-any.whl size=11978 sha256=73ee958cf4ecfbf0aea6c7899974ae9f835ebc5035d1db7924f3529bee33a6ea
Stored in directory: /Users/rstreet1/Library/Caches/pip/wheels/cb/f3/45/c859e83339943dfe2f43e1c9aaebdc00db321191a6fe120947
Successfully built antares-client bson
Failed to build confluent-kafka
ERROR: Could not build wheels for confluent-kafka, which is required to install pyproject.toml-based projects