Release 1.0.0
Release notes - Hopsworks - Version 1.0.0
Bug
- [HOPSWORKS-432] - Clean Kafka ACLs when unsharing a topic or deleting a project.
- [HOPSWORKS-1095] - Localhost (127.0.0.1) cluster definition support
- [HOPSWORKS-1114] - Rework password recovery
- [HOPSWORKS-1146] - Kagent stops reporting if env creation fails with non default repo
- [HOPSWORKS-1152] - Dataset browser not displaying contents of shared datasets with naming conflicts
- [HOPSWORKS-1172] - Upload progress not visible in the beginning of uploading large files
- [HOPSWORKS-1175] - yarnapp/user attribute not propagated correctly to yarn-site.xml
- [HOPSWORKS-1176] - Security risk for default admin password pops up even if the password has been changed
- [HOPSWORKS-1178] - [airflow-chef] Add default timezone for Airflow
- [HOPSWORKS-1180] - Chef upgrade mixes kafka brokers ids
- [HOPSWORKS-1185] - Jupyter not working when Hopsworks runs behind a LB/reverse proxy w/ TLS termination
- [HOPSWORKS-1190] - [featurestore] missing argument when writing petastorm datasets
- [HOPSWORKS-1195] - Bad handling of Job resource missing
- [HOPSWORKS-1197] - Fix broken admin and agent tests
- [HOPSWORKS-1198] - Cleanup hadoop sbin scripts
- [HOPSWORKS-1199] - ErrorCode range mismatch and TestRestCodes not testing all ErrorCodes
- [HOPSWORKS-1203] - [ePipe] Fixes for JSON handling in the XAttr table
- [HOPSWORKS-1207] - Hopsworks broken on Ubuntu 18.04
- [HOPSWORKS-1209] - filebeat_kagent does not start with kagent start-all services script
- [HOPSWORKS-1210] - Jupyter now requires the python kernel.json to be under python{2, 3} dir.
- [HOPSWORKS-1215] - Use OSProcessExecutor for Python library search
- [HOPSWORKS-1216] - tensorflow-chef install recipe CentOS hiccups
- [HOPSWORKS-1218] - Python service not working when searching or installing libraries containing 'http'
- [HOPSWORKS-1225] - add attribute for url to download spark in spark-chef
- [HOPSWORKS-1228] - Remove livy session name from sparkmagic conf file
- [HOPSWORKS-1233] - [ePipe] hivecleaner fail during project removal
- [HOPSWORKS-1234] - Remove sudo command from sudoers scripts
- [HOPSWORKS-1236] - Project cleanup does not work after a failed project creation
- [HOPSWORKS-1246] - [ePipe] Check for the existence of elastic fields before delete
- [HOPSWORKS-1251] - images not rendering in Jupyter notebooks
- [HOPSWORKS-1252] - Airflow HopsworksLaunchOperator waits for wrong Job statuses
- [HOPSWORKS-1253] - [ePipe] Rename operation should read the inodes info from the DB
- [HOPSWORKS-1263] - Project cleanup fails if X.509 to be revoked does not exist
- [HOPSWORKS-1268] - Sanitize python library name/version/reposity/url
- [HOPSWORKS-1271] - Bump HopsHive to use ORC 1.5.5
- [HOPSWORKS-1276] - Anaconda import yml file not working, removal of explicit enabling of python kernel
- [HOPSWORKS-1278] - [ePipe] The tailer should guarantee the order of epochs during recovery and while waiting for events
- [HOPSWORKS-1279] - [ePipe] Check for the existence of the xattr parent field before delete
- [HOPSWORKS-1280] - [hops-util-py] add_module not working in python kernel
- [HOPSWORKS-1282] - Exclude netty from Hudi hdfs dependency
- [HOPSWORKS-1285] - Livy is missing Yarn GENERATING_SECURITY_MATERIAL state
- [HOPSWORKS-1290] - [hops-util-py] numpy_helper and pandas_helper documentation not rendering
- [HOPSWORKS-1297] - Model Serving view not sorted
- [HOPSWORKS-1298] - Cannot remove job associated with featuregroups
- [HOPSWORKS-1300] - [kagent-chef] Change deprecated node.set on install recipe
- [HOPSWORKS-1315] - Hopsworks wrongly sets umask for HDFS client to 000
- [HOPSWORKS-1326] - setup-chef is broken
- [HOPSWORKS-1328] - [airflow-chef] Package libjpeg-devel is no longer available
- [HOPSWORKS-1329] - [hopsworks-cloud-sdk] Fix connecting from Databricks
- [HOPSWORKS-1336] - [kagent] Host private key is not converted to PKCS#1 if it's an upgrade
- [HOPSWORKS-1343] - Airflow Hopsworks connection hook ignores configured connections
- [HOPSWORKS-1350] - [featurestore] default training dataset connector not found
- [HOPSWORKS-1355] - Conda solver is problematic
- [HOPSWORKS-1357] - flyway_schema_history fix for versions < 0.6 doesn't work after 1.0.0
- [HOPSWORKS-1358] - Handle glassfish upgrade from 4.1.2.174 to 4.1.2.181
- [HOPSWORKS-1359] - MySQL server reinitialized during update to 1.0.0
- [HOPSWORKS-1360] - Data validation JS controller does not properly include mainCtrl
- [HOPSWORKS-1365] - When upgrading ignore anaconda base env's lib dir of previous installations
- [HOPSWORKS-1368] - NullPointerException when deleting a project without OnlineFeaturestore
- [HOPSWORKS-1370] - Online featurestore need mysql driver in the spark classpath
Sub-task
- [HOPSWORKS-1132] - [featurestore] improve UI
- [HOPSWORKS-1133] - [featurestore] drop dependency functionality
- [HOPSWORKS-1141] - [online featurestore] python API
- [HOPSWORKS-1142] - [online featurestore] Java/Scala API
- [HOPSWORKS-1155] - [featurestore] join features into training datasets from UI
- [HOPSWORKS-1156] - [featurestore] support feature defintions stored remotely
- [HOPSWORKS-1158] - [featurestore] custom data sources
- [HOPSWORKS-1159] - [featurestore] custom data sinks
- [HOPSWORKS-1206] - [featurestore] multiple jobs linked to feature group / training dataset
- [HOPSWORKS-1258] - Use hops-util from the local installation instead of Hops-FS
- [HOPSWORKS-1364] - Remove telegraf and kapacitor from VersionsDTO
New Feature
- [HOPSWORKS-34] - Hudi for unified Stream/Batch processing
- [HOPSWORKS-640] - Support for Flink, Beam, TFX
- [HOPSWORKS-1020] - Support supplying user provided version for installing python libraries
- [HOPSWORKS-1107] - Monitoring and Notifications
- [HOPSWORKS-1135] - API keys
- [HOPSWORKS-1139] - [Umbrella] Online Feature Store
- [HOPSWORKS-1183] - Secrets storage
- [HOPSWORKS-1186] - Make NDB disk data files configurable in chef
- [HOPSWORKS-1221] - [ePipe] Add metrics endpoint
- [HOPSWORKS-1231] - Support accessing the feature store from Amazon SageMaker
- [HOPSWORKS-1232] - Support accessing the Hopsworks feature store from a Databricks notebook
- [HOPSWORKS-1243] - Git support for Jupyter Notebook server
- [HOPSWORKS-1264] - [featurestore] import functionality
- [HOPSWORKS-1302] - [cloud-sdk] Add create training dataset API and job
- [HOPSWORKS-1304] - Add support for triggering jobs for featuregroup imports - CE
Improvement
- [HOPSWORKS-453] - Spark jobs should use local spark jars instead of hdfs archive
- [HOPSWORKS-781] - [ePipe] Integrate hive cleaner into ePipe
- [HOPSWORKS-782] - [ePipe] Synchronize recovery tailer and table tailers to avoid inconsistencies during recovery
- [HOPSWORKS-1054] - Reimplement the schemaless metadata API to use the extended attributes added in HOPS-1312
- [HOPSWORKS-1063] - Cleanup NDB configuration in ndb-chef
- [HOPSWORKS-1124] - optimize scp-deploy-frontend.sh
- [HOPSWORKS-1144] - Update conda distribution binary
- [HOPSWORKS-1162] - Make hops home readable for everybody
- [HOPSWORKS-1163] - Set the Jupyter token explicitly
- [HOPSWORKS-1164] - Refactor Hopswork to prepare support for non-local Jupyter
- [HOPSWORKS-1179] - [hops-examples] Add Maggy MNIST example notebook to DL tour
- [HOPSWORKS-1189] - Remove kagent/network/interface attribute
- [HOPSWORKS-1196] - Feature Store V2
- [HOPSWORKS-1200] - upgrade apexcharts
- [HOPSWORKS-1204] - [ePipe] Use configuration files instead of command line parameters
- [HOPSWORKS-1208] - Replace Jupyter Notebook with JupyterLab
- [HOPSWORKS-1217] - Configure Livy to not upload internal libraries to stagingDir
- [HOPSWORKS-1219] - [ePipe] Update CMakeLists to use C++14, Boost 1.70, and CMake 3.5.0
- [HOPSWORKS-1226] - [ePipe] Use Boost:Beast http client library instead of Libcurl
- [HOPSWORKS-1227] - forward kafka authorizer logs to a separate file
- [HOPSWORKS-1230] - Support changing the schema of an existing Kafka topic
- [HOPSWORKS-1237] - Improve the Installation / Configuration Documentation
- [HOPSWORKS-1242] - [Featurestore] update hops-petastorm to track petastorm 0.7.6
- [HOPSWORKS-1247] - Download restricted artifacts when installing Hopsworks EE
- [HOPSWORKS-1256] - Add flag to disable creation of users, groups and user-group mapping
- [HOPSWORKS-1269] - [featurestore] make S3 connector compatible with IAM role
- [HOPSWORKS-1284] - Add support for installing python libraries by name and/or version
- [HOPSWORKS-1299] - Forbid S3 ft connector access/secret key if using instance role
- [HOPSWORKS-1307] - [Karamel] Fixes to release Karamel 0.6
- [HOPSWORKS-1312] - Airflow operator to fetch Feature store validation result
- [HOPSWORKS-1321] - [featurestore] Utility function to list feature names in feature group
- [HOPSWORKS-1322] - [hopsworks-cloud-sdk] Configure credential per IAM role not per project and IAM role
- [HOPSWORKS-1337] - [featurestore] Utility function to list names of features in training dataset
- [HOPSWORKS-1366] - Upgrade default maggy version
Task
- [HOPSWORKS-609] - Get rid of Obama from the 404 page
- [HOPSWORKS-901] - confusing number of projects
- [HOPSWORKS-909] - Upgrade Kafka to 2.3.0
- [HOPSWORKS-1021] - Replace http.client with requests library in hops-util-py
- [HOPSWORKS-1081] - Upgrade Spark to 2.4.3
- [HOPSWORKS-1119] - Upgrade ROCm to 2.6 and tensorflow-rocm 1.14
- [HOPSWORKS-1145] - Upgrade to TF 1.14, Cuda10, Cudnn7.6, Pydoop2.0.0
- [HOPSWORKS-1154] - Improve creation speed for Deep Learning and Featurestore tours
- [HOPSWORKS-1166] - [hops-util-py] change development status to production/stable
- [HOPSWORKS-1181] - Remove hopsfacets as a dependency
- [HOPSWORKS-1184] - Add dfs.exclude and dfs.datanode.balance.max.concurrent.moves to hdfs-site.xml
- [HOPSWORKS-1229] - Update conda version to 4.7
- [HOPSWORKS-1235] - Reduce logging level of Airflow Gunicorn
- [HOPSWORKS-1239] - Replace script pinging for Jupyter with Jupyter REST API
- [HOPSWORKS-1240] - Pin TensorFlow Model Server to 1.14.0 in Dockerfile
- [HOPSWORKS-1261] - Use kagent to start and stop all local services instead of static bash scripts
- [HOPSWORKS-1262] - Remove unused and insecure Airflow related code
- [HOPSWORKS-1266] - [ePipe] Fix Licence headers to AGPL v3
- [HOPSWORKS-1273] - Install base admin software on the machines
- [HOPSWORKS-1275] - Upgrade Nvidia driver to 430 to support Ubuntu 18
- [HOPSWORKS-1286] - Hopsworks Airflow Operators should be able to use user provided API key
- [HOPSWORKS-1289] - [karamel-chef] Add swap on dev VMs if they don't have already
- [HOPSWORKS-1293] - [cloud-sdk] adjust hops-py-util code to work with hive instead of spark