Releases 1.4.0 - 1.4.1
Release notes - Hopsworks - Version 1.4.0
Improvement
- [HOPSWORKS-1314] - It should be possible to start Jupyter with a Shared Dataset as base directory
- [HOPSWORKS-1655] - Refactor FeatureGroupService for the new clients
- [HOPSWORKS-1657] - Generate per system user X.509 certificates
- [HOPSWORKS-1752] - Changning Deployment Repos for Enterprise Edition
- [HOPSWORKS-1762] - Add support for the Spark-Tfrecord library
- [HOPSWORKS-1786] - xattr refactor - tags and featurestore search
- [HOPSWORKS-1791] - [ePipe] Add support for large XAttrs
- [HOPSWORKS-1804] - Integrate Prometheus with Consul
- [HOPSWORKS-1815] - manage kubernetes through kagent
- [HOPSWORKS-1816] - Use consul for hive-site and metrics.conf configuration
- [HOPSWORKS-1825] - Connect to MySQLd's using 127.0.0.1 instead of private IP
- [HOPSWORKS-1829] - Register Kibana with consul
- [HOPSWORKS-1834] - Add XAttrs configurations to chef
- [HOPSWORKS-1852] - Jupyter server admin page returns 500
- [HOPSWORKS-1854] - Add datanode replacement upon failure parameters
- [HOPSWORKS-1883] - Add JOB audience to feature group delete endpoint
- [HOPSWORKS-1892] - TrainingDataset/FeatureGroup create automatically increments version if no version provided by user
- [HOPSWORKS-1897] - Add example for submitting Spark/PySpark jobs remotely
- [HOPSWORKS-1899] - Add user and project create functions in hops Python library
- [HOPSWORKS-1921] - Show certificate password in the UI instead of sending it by email.
- [HOPSWORKS-1924] - Add Kafka scope to API keys
- [HOPSWORKS-1925] - Upgrade conda to version 2020.07
- [HOPSWORKS-1935] - Show project ID in the project settings
- [HOPSWORKS-1948] - Distribute HSFS in the base environment
- [HOPSWORKS-1954] - Use hopsworks version as a tag for all docker images
- [HOPSWORKS-1972] - Add new api_key_value to featurestore connect method
- [HOPSWORKS-1974] - [DOC] large xattributes - usage and upgrade details
- [HOPSWORKS-1975] - Add support for passing job arguments in the Airflow workflow builder
- [HOPSWORKS-1976] - New cloud installer script
- [HOPSWORKS-1979] - Pydoop in hops-util-py should be able to handle hopsfs scheme
- [HOPSWORKS-1996] - Allow setting additional trusted registry in container-executor
- [HOPSWORKS-1999] - Fix project cleanup
- [HOPSWORKS-2006] - Fall back to safe default if image name not set for project
- [HOPSWORKS-2008] - Bump Maggy version and update hops-examples for TF2.2
- [HOPSWORKS-2033] - UI update of jobs
New Feature
- [HOPSWORKS-1005] - Index Jupyter notebook server logs in elastic
- [HOPSWORKS-1691] - Use docker to run user code.
- [HOPSWORKS-1702] - Add user registration page in Admin UI and add option to disable default sign up page
- [HOPSWORKS-1839] - Jupyter GitLab support
- [HOPSWORKS-1849] - ACL for datasets
- [HOPSWORKS-1896] - Generate API key for admin Role
- [HOPSWORKS-1902] - Add support for managed docker registries
- [HOPSWORKS-1908] - Add flink remote client in hops-examples
Bug
- [HOPSWORKS-1421] - Hopsworks ignores chef parameter to set default max number of user projects
- [HOPSWORKS-1504] - Workflow builder should sanitize job names
- [HOPSWORKS-1568] - Installing Node.js must work for airgapped deployments
- [HOPSWORKS-1623] - Overwrite config.ini for ndb-chef
- [HOPSWORKS-1633] - Sparkmagic links to Spark UI and driver logs are broken
- [HOPSWORKS-1634] - MaggyFacade throws getSingleResult for findByAppId in parallel experiments
- [HOPSWORKS-1671] - increase systemctl timeout for ndbmtd
- [HOPSWORKS-1672] - Featurestore Descriptions containing ' cannot be handled by hive
- [HOPSWORKS-1740] - HopsKafkaAdminClient initialisation fails if kafka is down/unreachable and then never recovers
- [HOPSWORKS-1768] - Restrict the usage of scan operations
- [HOPSWORKS-1782] - Set C_INCLUDE_PATH and CPLUS_INCLUDE_PATH when installing libraries in Anaconda
- [HOPSWORKS-1793] - Increase certificate keysize
- [HOPSWORKS-1794] - Make Hopsworks logo consistent with the branding
- [HOPSWORKS-1795] - Close connections in AirflowDagFacade
- [HOPSWORKS-1796] - hops-util(-py) should use consul domain to reach Hopsworks
- [HOPSWORKS-1797] - Hopsworks should listen on an internal port with certificate signed by HopsCA
- [HOPSWORKS-1799] - Is kubernetes enabled is not evaluated correctly in front-end after variables service refactoring
- [HOPSWORKS-1800] - cache provenance-epipe scan operations & fixes
- [HOPSWORKS-1805] - Discover docker registry through consul
- [HOPSWORKS-1806] - Clean up docker images from registry
- [HOPSWORKS-1807] - Hadoop 3 canot be installed without hopsworks
- [HOPSWORKS-1808] - Fix Python environment upgrades from 1.3 to 1.4
- [HOPSWORKS-1810] - fix the base image
- [HOPSWORKS-1811] - manage docker registry with kagent.
- [HOPSWORKS-1813] - Put back conda environment import
- [HOPSWORKS-1819] - Query constructor should return the online query equivalent
- [HOPSWORKS-1820] - Use Miniconda in Docker image
- [HOPSWORKS-1823] - [hops-util-py] Add metric_key as an argument to distributed training
- [HOPSWORKS-1824] - Fix chrome driver version
- [HOPSWORKS-1826] - Add set -e to Docker script
- [HOPSWORKS-1836] - Pin nbdime-jupyterlab version
- [HOPSWORKS-1840] - Sklearn serving PermissionError
- [HOPSWORKS-1842] - Library installation/uninstallation should increment Docker image version for the project
- [HOPSWORKS-1850] - Consul health check for Logstash uses private IP
- [HOPSWORKS-1855] - Set PATH and LD_LIBRARY_PATH in Dockerfile
- [HOPSWORKS-1856] - sklearn iris classifier import joblib broken
- [HOPSWORKS-1858] - [chef] consul does not create hops gid needed by docker
- [HOPSWORKS-1859] - make dn and nn ports configurable
- [HOPSWORKS-1862] - Fix cyclic dependency error
- [HOPSWORKS-1864] - Revert HopsFS data dir location to be compatible with HopsFS upgrades
- [HOPSWORKS-1865] - Schema problem when updating from 1.2 to 1.3
- [HOPSWORKS-1869] - Add 1.3 migration in expat-site.xml.rb in chef
- [HOPSWORKS-1871] - Add admin DNs to all elasticsearch nodes
- [HOPSWORKS-1875] - Kubernetes master recipe should run after Consul default
- [HOPSWORKS-1879] - update gpu machine - docker packages
- [HOPSWORKS-1880] - Can not create project with a member
- [HOPSWORKS-1885] - hopsworks ee should use hops ee
- [HOPSWORKS-1886] - Service start fail if consul not ready.
- [HOPSWORKS-1887] - Hive warehouse should not be set as DB policy when HopsFS-S3 is enabled.
- [HOPSWORKS-1890] - Bump Tez minor version to account for Hadoop 3.2 dependencies
- [HOPSWORKS-1895] - Spark History server should listen to 0.0.0.0
- [HOPSWORKS-1900] - epipe mishandling of large xattr when logged xattr size differs from current xattr size
- [HOPSWORKS-1901] - JWTRequired filter not working if ApiKey is set
- [HOPSWORKS-1904] - Livy needs access to users' certificates
- [HOPSWORKS-1905] - reset_session at the beginning of maggy_spec
- [HOPSWORKS-1911] - TensorFlow version in variables should be 2.2
- [HOPSWORKS-1913] - Can't write to S3 bucket from a Docker container
- [HOPSWORKS-1914] - Docker image name regex sanitizer should be a shared attribute in Chef
- [HOPSWORKS-1915] - resolv.conf is regenerated on reboot on CentOS
- [HOPSWORKS-1917] - Fix the notebook names in the ruby tests after tf 2.2 upgrade of hops examples
- [HOPSWORKS-1919] - Consule race condition
- [HOPSWORKS-1920] - Airflow Job Sensor does not pass the API key to connection hook
- [HOPSWORKS-1928] - Add fs when returning HopsFS training datasets to the clients
- [HOPSWORKS-1929] - enforce unique names in training dataset splits
- [HOPSWORKS-1931] - online fs grants not updated correctly when changing user roles
- [HOPSWORKS-1933] - Services hang up startup for 10min if consul failed to start
- [HOPSWORKS-1938] - Jupyter fails to start as it does not find the launcher script
- [HOPSWORKS-1953] - update expat to use hops 3.2.0.0-SNAPSHOT
- [HOPSWORKS-1959] - kube-hops::master should run after consul::default
- [HOPSWORKS-1963] - Fix download_url in setup-chef
- [HOPSWORKS-1970] - navigating to featurestore fg/td/f page does not enable all functionality
- [HOPSWORKS-1977] - use kagent/enabled, not kagent/test
- [HOPSWORKS-1980] - aux is a reserved keyword in windows and cannot be used for directories
- [HOPSWORKS-1983] - datanode not starting if bind-utils not installed on centos
- [HOPSWORKS-1985] - delete from shared dataset with full path fails
- [HOPSWORKS-1989] - Python jobs sometimes finishes immediately and no logs are available
- [HOPSWORKS-1993] - Upgrade Petastorm from 0.7.6 to 0.9.4
- [HOPSWORKS-1995] - Move docker image regex sanitizer from hopsworks to conda chef
- [HOPSWORKS-1998] - Project ID in Jupyter configuration is incorrectly templated with ","
- [HOPSWORKS-2002] - Chef fixes for upgrading 1.3 -> 1.4
- [HOPSWORKS-2004] - get_latest_featuregroup_version not updating cache
- [HOPSWORKS-2012] - Fix typo in the link to Hopsworks Build status in README.md
- [HOPSWORKS-2018] - non admin hops users cannot view spark executors logs
- [HOPSWORKS-2020] - Available GPUs not calculated correctly
- [HOPSWORKS-2021] - TensorBoard too slow loading many experiment runs and times out
- [HOPSWORKS-2024] - Fix consul hostname resolution on ubuntu containers
- [HOPSWORKS-2028] - Starting two jobs with appPath pointing to .ipynb fails if started directly after each other
- [HOPSWORKS-2036] - TensorFlow should not be imported in hops-util-py util module
- [HOPSWORKS-2047] - Non LTS Nvidia Driver version doesn't work on Azure VMs
- [HOPSWORKS-2048] - Jupyter-git missing configuration for git_backend
- [HOPSWORKS-2059] - Base directory when starting Jupyter in python mode not set correctly
Task
- [HOPSWORKS-1414] - Update readthedocs revision history
- [HOPSWORKS-1642] - Upgrade to TensorFlow 2.2.0 and PyTorch 1.5.1
- [HOPSWORKS-1745] - Fix tests for sklearn, tfserving
- [HOPSWORKS-1821] - [hops-util-py] Replace pydoop imports with hdfs module
- [HOPSWORKS-1835] - Make files/directories created by Jobs/notebooks group writable by default
- [HOPSWORKS-1838] - Set NCCL_SOCKET_NTHREADS and NCCL_NSOCKS_PERTHREAD for distributed training
- [HOPSWORKS-1843] - Move download_url from kagent to conda
- [HOPSWORKS-1845] - Collect flask server logs in sklearn serving
- [HOPSWORKS-1857] - Bump Maggy version in Docker base image
- [HOPSWORKS-1873] - Bump hops dependencies in docker-images
- [HOPSWORKS-1877] - Add requests as dependency to hops-util-py
- [HOPSWORKS-1889] - hdfs.add_module should reload the python module each time
- [HOPSWORKS-1891] - Remove commons-beanutils dependency
- [HOPSWORKS-1893] - [hops-util-py] Add project and dataset modules docs
- [HOPSWORKS-1909] - Bump spark and hive versions in hops-examples
- [HOPSWORKS-1912] - Do not download Docker image if it's already there
- [HOPSWORKS-1916] - Add documentation on how to configure the Hopsworks launch operator to run on external Airflow
- [HOPSWORKS-1922] - Bump up service-discovery-clients version
- [HOPSWORKS-1923] - Remove users from privileged certificates group
- [HOPSWORKS-1927] - Move Cloud related code to separate module
- [HOPSWORKS-1934] - Upgrade TensorFlow to 2.3 and PyTorch to 1.6
- [HOPSWORKS-1936] - Use Miniconda instead of Anaconda for system python environments
- [HOPSWORKS-1943] - python and notebook execution tests mnist example takes too long
- [HOPSWORKS-1946] - [API] State provenance refactoring
- [HOPSWORKS-1973] - Remove master-cloud branch from hopsworks-chef
- [HOPSWORKS-1994] - Add example with TensorFlow profiling and custom image returned in wrapper
- [HOPSWORKS-2005] - model name directory in Models dataset should be group writable
- [HOPSWORKS-2007] - Bump hops dependencies to 3.2.0.0-RC1
- [HOPSWORKS-2011] - Revoke host certificates with regular expression
- [HOPSWORKS-2013] - Cleanup docker images from managed registries
- [HOPSWORKS-2017] - Bump Kubernetes minor version to 1.18.8
- [HOPSWORKS-2034] - Bump hops dependencies to 3.2.0.0-RC2
Sub-task
- [HOPSWORKS-1277] - update spark to work with hadoop3
- [HOPSWORKS-1281] - update hopsworks to work with hadoop3 and docker
- [HOPSWORKS-1592] - Add support for feature store s3 storage connectors to encrypted buckets
- [HOPSWORKS-1692] - makes flink work with hadoop3 and docker
- [HOPSWORKS-1693] - make livy depend on hadoop3
- [HOPSWORKS-1694] - Update chef cookbooks to run user job in docker
- [HOPSWORKS-1695] - make hive work with hadoop3
- [HOPSWORKS-1732] - Search iteration 2
- [HOPSWORKS-1776] - [API] provenance - ops & link artifacts - feature groups, training datasets, models, experiments
- [HOPSWORKS-1801] - update karamel-chef cluster definitions
- [HOPSWORKS-1809] - Remove deprecated Python management functionality
- [HOPSWORKS-1910] - [UI] provenance - link of feature groups, training datasets, models, experiments
- [HOPSWORKS-2000] - Sklearn should use base image instead of the hops one
- [HOPSWORKS-2001] - TFServing should use base image instead of the hops one
Release notes - Hopsworks - Version 1.4.1
Bug
- [HOPSWORKS-2057] - Unique constraint violation when installing multiple llibraries and update conda docker gc
- [HOPSWORKS-2065] - Restart systemd units after failure