Releases: swirlai/swirl-search
SWIRL AI 4.1, Community Edition
Team SWIRL is thrilled to announce the release of SWIRL AI Search 4.1, Community Edition!!
PLEASE STAR OUR REPO: https://github.com/swirlai/swirl-search
PLEASE VISIT OUR WEBSITE: https://www.swirlaiconnect.com/
New Features
🔹 The DESELECT ALL option on the results page now updates to a SELECT ALL link after it's clicked allowing you to quickly reset the original selections:
🔹 A full Unit Test suite was implemented for the Galaxy UI source code, and unused components and services were removed from the codebase.
Improvements
🔹 A variety of fixes and improvements were made to the new Galaxy UI interface, including the addition of the SWIRL version to the User Profile menu.
🔹 SWIRL AI has been validated on Python 3.12.9.
Changes
🔹 Please note that SWIRL is no longer publishing our Google API keys!!
Community users will need to obtain their own Google API keys to use SWIRL on Google Web, LinkedIn or with the SWIRL documentation site. (These sources are accessed via Google's Programmable Search Engine, which is not free.). Arxiv, European PMC and Google News will continue to return results. Users may search the SWIRL documentation site using the search box at the top of each page.
To obtain your own Google API key, please follow these instructions: https://cloud.google.com/docs/authentication/api-keys
🔹 The arXiv.org SearchProvider configuration now quotes the user's search terms by default, which will return fewer, more precise results from this source.
Known Issues
🔹 Clicking a Microsoft Teams result sometimes results in this error from the Teams app:
We cannot take you to that message because it's in a chat you're not in.
For Teams links to resolve correctly, you must have the Microsoft Teams app open, and be authenticated, prior to clicking the result link.
🔹 Creating searches from a browser with q= can sometimes create two Search objects.
This is because of browser prefetch AKA predictive service. If this bothers you or your users, please turn off the Chrome prediction service](https://www.ghacks.net/2019/04/23/missing-chromes-use-a-prediction-service-setting/) or the Safari prefetch.
Upgrading
Documentation
SWIRL's Documentation site has been updated and reorganized!
SWIRL Overview | Quick Start Guide | User Guide | RAG Configuration | Glossary | Admin Guide | Installation Guide | M365 Guide | SearchProviders Guide | Troubleshooting | Developer Guide | Developer Reference | Tutorial: Extending SWIRL
Support
🔹 Join the SWIRL Community on Slack!
🔹 Email support@swirlaiconnect.com with issues, requests, questions, etc. - we'd love to hear from you!
SWIRL AI 4.0, Community Edition
Team SWIRL is thrilled to announce General Availability of SWIRL AI 4.0 Community edition!
The release includes an updated Galaxy user interface!
Thanks to all SWIRL community members who contributed feedback! Thank you!
PLEASE STAR OUR REPO: https://github.com/swirlai/swirl-search
PLEASE VISIT OUR WEBSITE: https://www.swirlaiconnect.com/
New Features
Updated Galaxy User Interface
Galaxy has been updated to the latest versions of all components, and re-arranged for improved visual clarity!
Improvements
🔹 The Galaxy UI now supports multiple chart displays!
For more about this feature, read: Visualizing Structured Data
🔹 SWIRL now reports the total number of results found from all sources, in addition to the number of results retrieved by SWIRL.
🔹 Galaxy now offers a "Show/Hide All Details" switch.
🔹 SWIRL has been validated on Python 3.12.8.
Changes
🔹 The LinkedIn SearchProvider is now deactivated by default. If you are interested in using LinkedIn with SWIRL, please Contact SWIRL!!
Known Issues
🔹 Clicking a Microsoft Teams result sometimes results in this error from the Teams app:
We cannot take you to that message because it's in a chat you're not in.
For Teams links to resolve correctly, you must have the Microsoft Teams app open, and be authenticated, prior to clicking the result link.
🔹 Creating searches from a browser with q= can sometimes create two Search objects.
This is because of browser prefetch AKA predictive service. If this bothers you or your users, please turn off the Chrome prediction service](https://www.ghacks.net/2019/04/23/missing-chromes-use-a-prediction-service-setting/) or the Safari prefetch.
Upgrading
Documentation
Overview | Quick Start Guide | User Guide | Admin Guide | M365 Guide | Developer Guide | Tutorial: Extending Swirl | Developer Reference | AI Guide | Enterprise Guide
Support
🔹 Join the SWIRL Community on Slack!
🔹 Email support@swirl.today with issues, requests, questions, etc. - we'd love to hear from you!
SWIRL AI Connect 3.9, Community Edition
Team SWIRL is happy to announce General Availability of SWIRL AI Connect 3.9, Community Edition!
The release enables Django database migrations, adds an Unselect All button when using the "Select Items" option for RAG, and provides validation of SWIRL on Python 3.12.6. Many thanks to everyone in the SWIRL community who contributed feedback addressed in this release.
PLEASE STAR OUR REPO: https://github.com/swirlai/swirl-search
New Features
🔹 New Unselect All button when using the "Select Items" option for RAG
🔹 SWIRL database schema changes will now be handled as part of Django migrations. Each schema change will have a corresponding migration file, which will be version-controlled and included with each release.
🔹 SWIRL AI Connect has been validated on Python 3.12.6.
Improvements
🔹 The Galaxy UI more gracefully handles longer result titles and browser window resizing.
🔹 A new "Dev" tag for searching across development-centric SearchProviders has been added
🔹 The Elastic SearchProvider now correctly returns the configured number of results
🔹 The FILE_SYSTEM directive was removed from default OneDrive SearchProvider configuration; and the default Outlook Email result_processors were reordered to improve threaded conversation handling
Known Issues
🔹 Clicking a Microsoft Teams result sometimes results in this error from the Teams app:
We cannot take you to that message because it's in a chat you're not in.
For Teams links to resolve correctly, you must have the Microsoft Teams app open before clicking the result link.
🔹 Creating searches from a browser with q= can sometimes create two Search objects.
This is because of browser prefetch AKA predictive service. Turn off Chrome prediction service. Turn off Safari prefetch.
Please report any issues with this to support.
Upgrading
Documentation
Please visit our updated and expanded Documentation site: https://docs.swirlaiconnect.com/
Support
🔹 Join the SWIRL Community on Slack!
🔹 Email support@swirl.today with issues, requests, questions, etc. - we'd love to hear from you!
SWIRL AI Connect 3.8, Community Edition
Team SWIRL is happy to announce General Availability of SWIRL AI Connect 3.8, Community Edition!
The release includes PII detection and removal based on MS Presidio, improved citation presentation for RAG results, and validation of SWIRL on Python 3.12.5. Many thanks to everyone in the SWIRL community who contributed feedback addressed in this release.
PLEASE STAR OUR REPO: https://github.com/swirlai/swirl-search
New Features
🔹 PII Detection and Removal with MS Presidio
🔹 Updated Citation Format for SWIRL's RAG Results
🔹 SWIRL AI Connect has been validated on Python 3.12.5.
Improvements
🔹 Resolved a problem with the PATCH option at the Django endpoints
Known Issues
🔹 Clicking a Microsoft Teams result sometimes results in this error from the Teams app:
We cannot take you to that message because it's in a chat you're not in.
For Teams links to resolve correctly, you must have the Microsoft Teams app open before clicking the result link.
🔹 Creating searches from a browser with q= can sometimes create two Search objects.
This is because of browser prefetch AKA predictive service. Turn off Chrome prediction service. Turn off Safari prefetch.
Please report any issues with this to support.
Upgrading
Documentation
Please visit our updated and expanded Documentation site: https://docs.swirlaiconnect.com/
Support
🔹 Join the SWIRL Community on Slack!
🔹 Email support@swirl.today with issues, requests, questions, etc. - we'd love to hear from you!
SWIRL AI Connect 3.6.0.1
SWIRL AI Connect 3.6.0.1 Community edition is now available!
This release includes improvements to LinkedIn searching and resolves an issue with the GenAI Connector, along with updates throughout to the new swirlaiconnect.com domain.
PLEASE STAR OUR REPO: https://github.com/swirlai/swirl-search
Updates
🔹 The default LinkedIn SearchProvider has improved matching for person names and job titles.
🔹 An issue was resolved with the GenAI Connector.
🔹 Codebase references to SWIRL domains were updated to the new swirlaiconnect.com domain.
Known Issues
🔹 Clicking a Microsoft Teams result sometimes results in this error from the Teams app:
We cannot take you to that message because it's in a chat you're not in.
For Teams links to resolve correctly, you must have the Microsoft Teams app open before clicking the result link.
🔹 Creating searches from a browser with q= can sometimes create two Search objects.
This is because of browser prefetch AKA predictive service. Turn off Chrome prediction service. Turn off Safari prefetch.
Please report any issues with this to support.
Upgrading
Documentation
Please visit our updated and expanded Documentation site: https://docs.swirlaiconnect.com/
Support
🔹 Join the SWIRL Community on Slack!
🔹 Email support@swirlaiconnect.com with issues, requests, questions, etc. - we'd love to hear from you!
SWIRL AI Connect 3.6.0
Team SWIRL is excited to announce General Availability of SWIRL AI Connect 3.6.0 Community edition!
The release includes a production QDrantDB Connector and SearchProvider, plus support for RAG processing based solely on search results content.
Thank you to Anush008 and others in the SWIRL community who contributed feedback addressed in this release.
PLEASE STAR OUR REPO: https://github.com/swirlai/swirl-search
New Features
🔹 Production support for searching QDrantDB sources, including a new Connector and SearchProvider:
🔹 SWIRL's RAG processing will now fallback to using the search result content when pages are not available to be fetched.
🔹 SWIRL AI Connect has been validated on Python 3.12.4.
Known Issues
🔹 Clicking a Microsoft Teams result sometimes results in this error from the Teams app:
We cannot take you to that message because it's in a chat you're not in.
For Teams links to resolve correctly, you must have the Microsoft Teams app open before clicking the result link.
🔹 Creating searches from a browser with q= can sometimes create two Search objects.
This is because of browser prefetch AKA predictive service. Turn off Chrome prediction service. Turn off Safari prefetch.
Please report any issues with this to support.
Upgrading
Documentation
Overview | Quick Start Guide | User Guide | Admin Guide | M365 Guide | Developer Guide | Tutorial: Extending Swirl | Developer Reference | AI Guide | Enterprise Guide
Support
🔹 Join the SWIRL Community on Slack!
🔹 Email support@swirl.today with issues, requests, questions, etc. - we'd love to hear from you!
SWIRL AI Connect 3.5.0
Team SWIRL is excited to announce General Availability of SWIRL AI Connect 3.5 Community edition!
The release includes a production Pinecone connector, support for running SWIRL in the VS Code debugger, plus GDPR and dataset visualization support in Galaxy.
Thank you to simsong, deg and others in the SWIRL community who contributed feedback addressed in this release.
PLEASE STAR OUR REPO: https://github.com/swirlai/swirl-search
New Features
🔹 The Galaxy UI now renders a chart visualization for results from SearchProviders with the DATASET option specified in the result_mappings field.
🔹 Added GDPR cookie consent to the Galaxy UI.
🔹 Use the new PineconeDB Connector and SearchProvider to search vector database sources.
🔹 Developers can now run SWIRL in the VSCode debugger.
Improvements
🔹 The default SearchProvider configurations that ship with SWIRL have been updated; the multiple Google PSEs were consolidated into a new, simple provider definition, and many freely available sources are now enabled by default.
🔹 The Elastic and OpenSearch Connectors were refactored.
🔹 SWIRL's public API documentation was updated and is now generated by the drf-spectacular package with support for the OpenAPI v.3 schema.
🔹 Scripts and other supporting utilities have been moved to the new DevUtils/ directory in the repository.
🔹 Logging has been refactored; detailed messaging can be selectively enabled using the new SWIRL_LOG_DEBUG entry in SWIRL's .env file. For example:
SWIRL_LOG_DEBUG='swirl.page_fetch.web_page, swirl.processors.rag'🔹 CSRF support now available via the new CSRF_TRUSTED_ORIGINS entry in SWIRL's .env file. For example:
CSRF_TRUSTED_ORIGINS='http://localhost:8000'🔹 SWIRL was validated on Python 3.12.2.
Resolved Issues
🔹 The Galaxy UI now reports the total number of results returned for a search instead of the total number of results configured in the active SearchProviders.
🔹 The URL presented with each Galaxy results is now clickable, just like the result title.
🔹 The .install.sh script was adjusted to keep pip and other essential packages properly updated.
Known Issues
🔹 Clicking a Microsoft Teams result sometimes results in this error from the Teams app:
We cannot take you to that message because it's in a chat you're not in.
For Teams links to resolve correctly, you must have the Microsoft Teams app open before clicking the result link.
🔹 Creating searches from a browser with q= can sometimes create two Search objects.
This is because of browser prefetch AKA predictive service. Turn off Chrome prediction service. Turn off Safari prefetch.
Please report any issues with this to support.
Upgrading
Documentation
Overview | Quick Start Guide | User Guide | Admin Guide | M365 Guide | Developer Guide | Tutorial: Extending Swirl | Developer Reference | AI Guide | Enterprise Guide
Support
🔹 Join the SWIRL Community on Slack!
🔹 Email support@swirl.today with issues, requests, questions, etc. - we'd love to hear from you!
Swirl 3.2.0
Team Swirl is excited to announce General Availability of Swirl 3.2!
This release includes the first version of the AutomaticPayloadMapperResultProcessor, RequireQueryStringInTitleResultProcessor, DropIrrelevantPostResultProcessor and NoModQueryProcessor, along with new SearchProviders for OpenSanctions.org and LittleSis.org. We've also added support for running OpenAI in Azure, storing query templates in JSON, plus updated Python, Django, and OpenAI API versions, and resolved issues found in previous releases. Finally, the Galaxy UI has been updated to ⭐ results above a particular relevancy score.
We're seeking anyone interested in testing new Connectors to Collibra, Thoughtspot and Pinecone DB. Please contact Swirl via email or Slack for access.
PLEASE STAR OUR REPO: https://github.com/swirlai/swirl-search
New Features
🔹 New AutomaticPayloadMapperResultProcessor which profiles response data to find good strings for Swirl's title, body, and date_published fields. It is intended for SearchProviders that would otherwise have few (or no) good result_mappings options. It should be place after the MappingResultProcessor, and the result_mappings field should be blank.
"result_processors": [
"MappingResultProcessor",
"AutomaticPayloadMapperResultProcessor",
"CosineRelevancyResultProcessor"
],
...
"result_mappings": "",
🔹 New NoModQueryProcessor that only removes leading SearchProvider Tags and does not modify the query terms in any way. It is intended for repositories that allow non-search characters (such as brackets).
"query_processors": [
"NoModQueryProcessor"
],
🔹 New RequireQueryStringInTitleResultProcessor that drops results that do not contain the query_string_to_provider in the result title field. It should be added after the MappingResultProcessor and is now included by default in the "LinkedIn - Google PSE" SearchProvider.
"result_processors": [
"MappingResultProcessor",
"RequireQueryStringInTitleResultProcessor",
"DateFinderResultProcessor",
"CosineRelevancyResultProcessor"
],
🔹 New DropIrrelevantPostResultProcessor drops results with swirl_score < settings.MIN_SWIRL_SCORE (set to 500 by default) and results with no swirl_score.
🔹 Swirl now supports using Azure OpenAI models for RAG processing in addition to OpenAI's ChatGPT models.
2024-01-29 16:36:07,335 INFO get model AZUREAI gpt-35-turbo
2024-01-29 16:36:10,542 INFO HTTP Request: POST https://swirltest-openai.openai.azure.com/openai/deployments/gpt-35-turbo/chat/completions?api-version=2023-10-01-preview "HTTP/1.1 200 OK"
🔹 The Galaxy UI now highlghts results with a swirl_score above a configurable threshold with a star in the results list. The swirl_score configuration is available in theminimumSwirlScore entry of static/api/config/default, and the default is 100.
🔹 New SearchProvider.query_template_json field which stores the query_template as JSON. The MongoDB SearchProvider has been updated to use this feature.
"query_template_json": {
"$text": {
"$search": "{query_string}"
}
},
🔹 Support for lowercasing of URLs is now available via a new LC_URL option in SearchProvider result_mappings.
"result_mappings": "title=resource.name,body=resource.description, ... LC_URL,NO_PAYLOAD",
🔹 Added support for list-of-list responses from source APIs, where the first list element is the column names.
New SearchProviders
🔹 Search the LittleSis free database of "who-knows-who at the heights of business and government" with the new LittleSis.org SearchProvider.
🔹 Find sanctions targets and persons of interest with the new OpenSanctions.org SearchProvider.
🔑 Note: Additional SearchProviders contributed by the Swirl Community may be found in the SearchProviders/untested directory.
Improvements
🔹 Lists of SearchProviders can be added to the endpoint with copy/paste, and Swirl will load them all.
🔹 When signing in to Swirl via Microsoft OIDC, users are now automatically authorized to the M365 SearchProviders.
🔹 Swirl was validated on Python 3.12.1.
🔹 Swirl now runs on the new Django 5.0.x series Python packages.
🔹 Swirl now uses the latest 1.6.x OpenAI Python API.
🔹 The QueryProcessors base classes were updated.
🔹 The default AI Summary timeout value can now be overridden with a URL parameter in the Galaxy UI. For example: http://localhost:8000/galaxy/?q=gig%20economics&rag=true&rag_timeout=90000
🔹 The Snowflake SearchProvider query_template was revised.
🔹 The BigQuery SearchProvider no longer dedupes results.
🔹 Improved zero results feedback and logfile messages.
Resolved Issues
🔹 Removed duplicate static/api/config/default configuration file.
🔹 Removed unused timeout settings (SWIRL_Q_WAIT and SWIRL_RERUN_WAIT).
🔹 Fixed Elasticsearch authorization and addressed a certificate error.
🔹 Updated the index_email_elastic.py script and related documentation.
🔹 Stopwords are no longer highlighted in search results or AI Summaries.
🔹 Various small fixes to Galaxy UI behavior.
Known Issues
🔹 Clicking a Microsoft Teams result sometimes results in this error from the Teams app.
We cannot take you to that message because it's in a chat you're not in.
For Teams links to resolve correctly, you must have the Microsoft Teams app open before clicking the result link.
🔹 Creating searches from a browser with q= can sometimes create two Search objects.
This is because of browser prefetch AKA predictive service. Turn off Chrome prediction service. Turn off Safari prefetch.
Please report any issues with this to support.
Upgrading
Documentation
Overview | Quick Start | User Guide | Admin Guide | M365 Guide | Developer Guide | Developer Reference | AI Guide | Azure Marketplace Guide
Support
🔹 Join the Swirl Community on Slack!
🔹 Email support@swirl.today with issues, requests, questions, etc - we'd love to hear from you!
Swirl 3.1.0
Team Swirl is announcing General Availability of Swirl 3.1.0!
Swirl 3.1.0 includes new Connectors to MongoDB Atlas, Snowflake and Oracle, updates to Python and OpenAI API versions, and resolves numerous issues with previous versions.
Community Contributions
🏆 This version of Swirl includes additional contributions from the open source community. Many thanks to GitHub users Harsh Mahajan (HarshMN2345) and Harshil Khamar (Harshil0512) for their accepted, merged PRs; and also to Janvi Thakkar (Janvi-Thakkar) for inspiring us to add the Internet Archive as a source for Swirl!
PLEASE STAR OUR REPO: https://github.com/swirlai/swirl-search
New Features
🔹 New MongoDB Atlas Connector & SearchProvider for the sample_mflix collection, movies sample table
🔹 New Snowflake Connector & SearchProvider for the FreeCompanyResearch dataset
🔹 New Oracle Connector, tested against 23c Free (and presumably supporting earlier versions)
New SearchProviders
🔹 Search your Cards in Atlassian Trello
🔹 Find Tasks in Asana
🔹 Query the Internet Archive Library for almost anything
🔑 Note: Additional SearchProviders contributed by the Swirl Community may be found in the SearchProviders/untested directory.
Improvements
🔹 Swirl RAG is now available through a single API call, e.g. ?qs=metasearch&rag=true
🔹 Swirl now uses the latest 1.3.x OpenAI Python API.
🔹 Swirl 3.1.0 was validated on Python 3.11.6 (note that Python 3.12.0 support is awating one final dependency update).
🔹 The ServiceNow Knowledge SearchProvider was updated for the "Vancouver" release.
🔹 Pip requirements are now frozen on the main branch for releases (addresses Issue #912).
🔹 Default page-fetch configurations were added to the Enterprise Search, Mergers and Acquisition, Strategy Consulting, and Swirl Documentation Google PSE SearchProviders.
🔹 Configurations for a default timeout value (30 seconds) and the text to display when the timeout is exceeded were added to RAG processing.
🔹 Swirl's OpenAPI specification was updated: a new Swirl Postman collection and swagger.json are now available in the repo.
🔹 Matching query terms are now highlighted in the AI Response results.
🔹 Swirl now uses the latest 4.2.x Django packages.
Resolved Issues
🔹 Back button behavior with RAG processing was improved.
🔹 Addressed an issue in the version checker.
🔹 Resolved an issue with some RequestsPost searches returning 500 errors.
🔹 The Galaxy UI was updated to better handle an empty results set returned from some sources.
🔹 The Galaxy UI now displays ChatGPT SearchProvider results in the main results list.
🔹 The "Generate AI Response" option in the Galaxy UI is disabled until the full set of search results have returned.
🔹 Addressd a regression with search re-run functionality in the Galaxy UI.
Known Issues
🔹 Clicking a Microsoft Teams result sometimes results in this error from the Teams app.
We cannot take you to that message because it's in a chat you're not in.
For Teams links to resolve correctly, you must have the Microsoft Teams app open before clicking the result link.
🔹 Creating searches from a browser with q= can sometimes create two Search objects.
This is because of browser prefetch AKA predictive service. Turn off Chrome prediction service. Turn off Safari prefetch.
Please report any issues with this to support.
Upgrading
Documentation
Overview | Quick Start | User Guide | Admin Guide | M365 Guide | Developer Guide | Developer Reference | AI Guide
Support
🔹 Join the Swirl Community on Slack!
🔹 Email support@swirl.today with issues, requests, questions, etc - we'd love to hear from you!
Swirl 3.0.1
Team Swirl is announcing General Availability of Swirl 3.0.1!
This update addresses issues with two of our default GitHub SearchProviders, improves Swirl Unix timestamp handling, and pins the version of OpenAI's Python library in Swirl's requirements.
PLEASE STAR OUR REPO: https://github.com/swirlai/swirl-search
Resolved Issues
🔹 The GitHub "Issues" and "Pull Requests" SearchProvider configurations were updated to realign them with the source's Search API. Please review the query_template value in SearchProviders/github.json for Issues and Pull Requests and update existing configurations to match.
🔹 Swirl's handling of Unix timestamps was enhanced to better handle millisecond values when present in the source's response.
🔹 OpenAI's Python library is now pinned to a known-good version for Swirl. To ensure Swirl is configured with this package version:
- Stop Swirl:
python swirl.py stop - Re-run
./install.shin your<swirl-home>directory - Start Swirl:
python swirl.py start
Known Issues
🔹 Clicking a Microsoft Teams result sometimes results in this error from the Teams app:
We cannot take you to that message because it's in a chat you're not in.
For Teams links to resolve correctly, you must have the Microsoft Teams app open before clicking the result link.
🔹 Creating searches from a browser with q= can sometimes create two Search objects.
This is because of browser prefetch AKA predictive service. Turn off Chrome prediction service. Turn off Safari prefetch.
Please report any issues with this to support.
Upgrading
redis. You must now have redis installed and running before starting Swirl.
Documentation
Overview | Quick Start | User Guide | Admin Guide | M365 Guide | Developer Guide | Developer Reference | AI Guide
Support
🔹 Join the Swirl Community on Slack!
🔹 Email: support@swirl.today with issues, requests, questions, etc - we'd love to hear from you!