change.log

20220430

 - Change: Added single_pass variable which runs the sync process once before exiting out.

20220428

 - Change: Merged PR from  eliempje (https://github.com/boredazfcuk/docker-icloudpd/pull/155) which adds Openhab functionality.

20220411
 - Change: Apparently Synology devices have a Photos app which doesn't update when a download takes place. The newly created files need to be touched for it to trigger a re-index. I've added a new variable which will touch a file to trigger the update, then set it back to what it was to maintain integrity.

20220304

 - Change: Added a check for the IP address lookup performed during initialisation. Previously this could fail if the DNS server was not available (due to container starting on same system as a DNS container). It will now retry the lookup for 2mins before failing.
 - Change: Changed cookie generation name from Generate2FACookie to GenerateCookie as it generates both types of cookie and is run no matter what cookie type you use. Also, changed the log output so it doesn't say "Generating 2FA Cookie" when generating a Web cookie.
 - Change: Urgh. I dunno about you, but sometimes I delete a bunch of stupid memes off my phone and it ends up leaving my folder_structure with empty directories. I've now added a variable delete_empty_directories which will tidy those up.

20220220

 - Change: The timezone variable is not being set globally if it's not specified, so the underlying Python script generates a warning. The TZ variable is now defaulted to UTC in the Dockerfile which should take care of this.
 - Change: Added an extra synchronisation interval of 21600, which synchronises every 6 hours. This generates a warning message detailing the potential issue and fix, and includes a 2 minute wait on container start time to make the warning prominent.
 - Change: Pushover notifications not showing downloaded/deleted file previews
 - Change: Added function to allow deletion of HEIC accompanying files. These would be the converted JPG files (if using the HEIC to JPG conversion feature) and the_HEVC.MOV files which come with live photos.
 - Change: Bug found with sending Telegram notifications if e-mail address has an _ character in it. The _ character wasn't escaped correctly when passing the text to the Telegram API.
 - Change: The Notify function is always called, even if notifications aren't configured. A log entry was still being processed and the logic for determining a success/failure of the sent message was being performed too, resulting in an 'out of range' error, as an empty variable was being processsed.

20220218

 - Change: Cookie expiry notification period was being defaulted in the Initialise function and also for every notification type configured in the ConfigureNotifications function. Removed these as they're just duplicates. Also corrected a bug which would override user configured period, configuring it to 7.
 - Change: Logic for skipping the file check was checking if value it true, then following two if statements had to check if it was set to false. Don't really like the OR tests that needed to be done so adjusted the logic two set the check_exit code and check_files_count to values that will always perform a sync, as they should never change due to CheckFiles function never being run.
 - Change: Script doesn't seem to reliably send the "once per day" notifications during the nofitication expiry period. So have added a log entry which details the next notification time
 - Change: Huge overhaul of notification system. Probably broken everything lol. I've tested Telegra, Prowl, Discord and Pushover. I don't have any of the other methods to test though, so can't guarantee they work, or are formatted correctly. I've standardised the way notifications are sent to the Notify function. This way, only one notification command is needed to send notifications for all types and the formatting of the messages is now handled by the Notify function.

20220211

 - Change: Introduced a bug with my error handling from the last update. Now I create the error and exit_code log files in the Initialisation function of the script so that the correct permisions are applied. Problem was that the files were being created as root, and then the python script didn't have permission to them.
 - Change: Status code checking for the curl command when sending notifications would only accept 200 as a success. Discord reports 204. I've amended the checking code to just accept 2xx.

20220210

 - Change: Handle errors differently. Added suggestions for authentication failure issues.
 - Change: Eliminated full stops from log entries. Some lines had them, others didn't, so standardised on not having them. We don't need them. They take up precious bytes. They giving me flashbacks to COBOL.

20220209

 - Change: Broke out logging into three separate functions. Just because. I dunno. Having tonnes of $(date '+%Y-%m-%d %H:%M:%S') commands in the script just looked a bit weird. Plus if I even need to change it, it's three lines instead of a find/replace of a bazillion.
 - Change: Set ConfigureNotifications function to check for presence of Dingtalk token, instead of ID, to decide whether to fail or not. Same as the other methods (except webhook, which doesn't use tokens).
 - Change: Altered priority for Pushover warning/error notifications. Previously they were set to -2, which gives no alert, only changes the icon badge number. Now it sends a standard priority alert for all notification types, except the one that alerts you that your cookie will run out that day, which is a "quiet hours breaking" notification (same for Prowl).

20220206

 - Change: Amended the code which sends to Pushover notifications. Originally this was two separate commands, depending on whether the pushover_sound variable was sent. This is not needed, as Pushover will use the default sound is a blank value is sent for the "sound" argument

20220204

 - Change: Changed notification failure log messages to be more accurate and display http status code.
 - Change: Prowl notifications logging not quite right.
 - Change: Standard notification log message.
 - Change: Moved some stuff to prevent duplication of code.

20220123

 - Change: Added variable to set custom sounds for Pushover notification type. This can be used to silence notifications.

20211212

 - Change: Fix bug introduced by moving to useradd/groupadd.

20211211

 - Change: Amended domain and route checks to be compatible with icloud_china variable. No point checking the route to icloud.com is OK when the container downloads from icloud.com.cn.
 - Change: Change to IYUU notifications, which should now get these working, and integrated changes to live.
 - Change: Users and Groups now created with useradd/groupadd instead of adduser/addgroup to allow the creation of users/group with an id greater than 256,000.

20211130

 - Change: Final text alert for cookie expiration not received. Made a couple of changes. Will check again in 3 months time.

20211127

 - Change: Added the IYUU notification type. This is a 3rd party service which supports forwarding messages to WeChat.

20211125

 - Change: Added variable icloud_china so that photos are downloaded from icloud.com.cn instead of icloud.com 
 - Change: Removed readme information about obsolete variable interavtive_only. The readme is nearing it's maximum size on Dockerhub, so this helps keep tthe number of characters down.

20211121

 - Change: Amended healthcheck so that it does not set the status to unhealthy if notification days is less than 7. This was causing my auto-heal container to automatically restart it every 5mins. This behaviour now only occurs when the cookie has actually expired.
 - Change: Made script launch parameters case insensitive and also allowed for non-English spelling of Initialise.

20210024

 - Change: Set icloud.com DNS query type to 'a' records only so that it returns the IPv4 IP address and not IPv6. Also, removed the local loopback addresses, just to remove more unwanted data

20211020

 - Change: Discord notifications added by @ibtvt

20211017

 - Change: Added detection for when a file has been downloaded, but also exists in 'Recently Deleted', so gets removed post download. In these cases, it should get rid of the missing file errors.

20210921

 - Change: Added DNS lookup and traceroute to check network connectivity to icloud.com

20210915

 - Change: Removed the code which removes hyphen characters from the Telegram chat ID. The hyphen character is used to denote the bot has been added to a group to send messages, rather than a standard user chat.

20210913

 - Change: Replicated recent whitespace changes to additional functions for completeness
 - Change: Force convert HEICs functions now remove pre-existing JPG files as I don't think the convertion tool is overwriting pre-existing files

20210912

 - Change: Bug introduced by recent HEIC to JPEG change. Fixed.
 - Change: Added set owner and permissions function to run after HEIC to JPEG conversion functions
 - Change: Added --ForceConvertAllmntHEICs
 - Change: Fixed bug in above new command line option as it wasn't handling spaces safely

20210906

 - Change: HEIC to JPEG coversion utility has a bug and rotates JPEG files when it shouldn't. Replaced with ImageMagick which doesn't have this problem.
 - Change: Added --ForceConvertAllHEICs command line option to overwrite JPEG files. This is so over-rotated files can be replaced with crorrectly oriented versions.

20210902

 - Change: Added a log entry to show the UID of the account that is running the script. It seems some users are having permissions issues and this will help with debugging
 - Change: Also changed the symbolic link command to force linking of python_keyring directory if something that isn't a link exists in its place.

20210901

 - Change: Added logging for --ConvertAllHEICs function
 - Change: Amended way script handles expired cookies

20210830

 - Change: Amended script to wait for the cookie file to be created, then continuing once it's appeared. Container will auto restart after 30mins if file does not appear, just in case something has gone wrong.

20210827

 - Change: Better way of calculating elapsed time.
 - Change: Amended the logging so it lists the expiry time of expired 2FA cookies for clarity.
 - Change: Amended the way the script waits for the keyring and failsafe files to be created. Instead of checking every 5 minutes before proceeding, script now checks every 5 seconds. This way, script is a bit more responsive when being initialised. If the script doesn't detect the file it's checking for after 30 minutes, it will exit the container, just in case there is a detection problem.
 - Change: Added logging info explaining that the download check takes a long time on large collections and that nothing is being downloaded during that phase.
 - PR Merge: Changed how the webhook notification is handled for compatibility across more notification platforms.
 - Change: Script now records time taken per synchronisation and subtracts this from the synchronisation_interval. This means a sync with a 12 hour interval will start 12 hours later and not skew by the amount of time it takes the sync to complete.

20210827

 - Change: Locked version of tzlocal to 2.1 as lateat version introduces dependency which is not available in Python 3.8
 - Change: Added variable manipulation on the Telegram chat ID to remove minus sign if it's included in the variable. It needs to be omitted when posting the data to the API.

20210801

 - Change: Added some extra logging info

20210724

 - Change: Added function to sanitise command line parameters passed to script.

20210719

 - Change: Download and Delete notification webhook payload JSON fix
 - Change: Remove ambiguity in DownloadedFilesNotification

20210716

 - Change: Sanitise notification_title variable
 - Change: Rebase to Alpine Linux 3.14.0

20210713

 - Change: Modify 20210709 JSON escaping and fix minor change.log text

20210712

 - Change: Removed apple_password variable. Password must now be added to the system keyring.
 - Change: Default mode now to download, interactive_only variable removed. This process was far more complex than it needed to be for Synology/QNAP devices, so a small change to the default operation makes things easier for everyone.
 - Change: --Generate2FACookie command line option changed to --Initialise as it is also used for saving the password to the system keyring. Not just cookie generation

20210709

 - Change: Changed JSON escaping to prevent further issues.

20210708

 - Change: Download check and actual download functions both exit with same error message. Changed for clarity about which section is failing.

20210705

 - Change: JPEG quality mismatch in documentation and script. Default JPEG quality now referred to correctly in README.MD as 90.

20210617

 - Change: Added script hash to make sure that the sync-icloud.sh file is correct file as date stamp seems to be affected by timezone.
 - Change: Oddities occuring when creating password/cookie on devices which always run interactively

20210614

 - Change: Accurate health check output when approaching cookie expiration date.
 - Change: Cookie expiration date checking improved.
 - Change: Wasn't setting notification period for some of the notification options.
 - Change: Notifications weren't firing in the notification period. Should now be fixed.

20210607

 - Change: Code to remove empty keyring file requires logic to detect f file actually exists, otherwise brand new containers enter a file deletion loop. Logic amended.

20210522

 - Change: Rebase to Alpine Linux 3.13.5
 - Change: Now logs full version ID rather than just major/minor version.
 - Change: Reddit user Vinnipinni made a comment that the documentation was great... so I've revisited it again to make sure it's up to date, corrected some spellings and grammar, and made a few improvements for clarity.

20210522

 - Change: --size parameter was incorrectly set as --photo-size.

20210519

 - Change: Added code to remove keyring file if it doesn't contain credentials

20210515

 - Change: New download link for Boost source as location changed

20210515

 - Change: Amended log output for when attempting to add a user to a pre-existing system group.

20210322

 - Change: Amended log text after cookie generation. Script no longer claims 2FA cookie generated for all cases, but states 2FA/Web based on which was actually generated.
 - Chnage: Add Litecoin address. BTC network fees are expensive AF!

20210304

 - Change: Added notification_title variable.
 - Change: Readme changes to add full stops on lines where they were missing.

20210128

 - Change: Added logging of $TERM variable.

20210127

 - Change: Modified when the script check for the the --Generate2FACookie so that it can be invoked from NAS devices such a Synology and QNAP.

20210119

 - Change: Added the --Generate2FACookie command line option.

20210118

 - Change: Rebuilt on Alpine 3.13

20201221

 - Change: Convert HEIC to JPEG functions now allow specifying the quality with and integer value from 0 to 100.

20201216

 - Change: Sanitise apple_id variable so all letters are lower case. This prevents a case mismatch between the cookie file name that is saved, and the name that the icloudpd Python script creates the cookie file with, due to it respecting case.

20201204

 - Change: Removed Pushbullet notifications. The Pushbullet service has introduced version 2 of its API so notifications no longer work. Pushbullet have also removed their app from the iOS App Store, so if it's no longer supported on iOS, there's no way to realistically support it here.

20201203

 - Change: Upgraded pip
 - Change: Pushover notification bugfix. PR courtesy of @larstomas
 - Change: Webhook notification notification standardisation. PR courtesy of @larstomas again.
 - Change: Altered failure piority for pushover notifications from -2 to 2 so they become emergency level notifications rather than silent. Change recommended by @larstomas
 - Change: Added variable to allow skipping of the new file check. On large photo collections, this can significantly increase the time taken to download.

20201128

 - Change: Added log line to show the exact command line used to invole iCloudPD

20201121

 - Change: Due to changes in iOS14 the 2FA synchronisation interval is now limited to a maximum of once every 12 hours. It seems that 2FA authentication is now restricted to a maximum of 26 logins before re-authentication is required. Synchronisation is now restricted to specific time periods with the shortest being 12 hours.
 - Change: New variable synchronisation_delay added. This is so that the first synchronisation can be dalayed for a number of minutes. This is so that multiple containers can stagger their downloads

201112

 - Change: Bugfix. Convert HEIC to JPEG function would only run if download notifications were enabled.

20201111

 - Change: Additional changes required for Pushover notifications.

20201110

 - Change: Added Pushover notifications
 - Change: Failsafe file mount checking changed slightly regarding logging
 - Change: Added variable for the --until-found option as must have missed this on the 20201024 update.
 - Removed: Checking downloaded files list against files stored on the server is no longer required. Issue has been fixed upstream.

20200411

 - Change: Bugfix on healthcheck not detecting non-zero exit code.

20201101

 - Removed: Multi-thread capability has been removed from upstream so removed capability to specify number of threads
 - Removed: Speed test capability. Was put in to test speed differences between mutli-thread and single-thread download options. No longer possible due to previous change.
 - Change: Bugfix. Convert HEIC to JPEG function - missed a bit.

20201030

 - Change: Bugfix. Convert HEIC to JPEG function failing to set correct timestamp.

20201029

 - Change: Remove pip cache folder /root/.cache during container build.
 - Removed: Recently files only doesn't work as expected. Removed.
 - Change: Added --CorrectJPEGTimestamps command line option to correct timestamps of JPEG files
 - Change: Convert HEIC to JPEG functions don't preserve timestamps. Filestamp of JPEG now taken from HEIC file.
 - Change: Default to downloading new files only by checking what files exist vs what needs downloading. This prevents it attempting to download all files upon every synchronisation.
 - Change: Typo for Synchronisation Interval variable. This change will break users configurations that set the old variable name, so default to 24hr sync period for these users.

20201027

 - Change: Silence warning about depreceated command_line_options varaible if it's actually blank to begin with.
 - Change: Remove Python package Future. This is for Python 2 to Python 3 compatibility, however, upstream application now fully Python 3 compatible.
 - Change: Python keyring and keyrings.alt no longer version restricted. Upstream application is now compatible with newer Python versions than before so these older packages no longer need to be installed.

20201027

 - Change: Bugfix relating to Prowl & Pushbullet notifications.

20201024

 - Change: Added new variables: set_exif_datetime, auto_delete, photo_size, skip_live_photos, live_photo_size & skip_videos. Eventually command_line_options variable will be removed.
 - Change: Added a new "CommandLineBuilder" function to create the command line options list to be passed to the download command.

20201022

 - Change: Added a speed test mode which ignores the requirement for a mounted filesystem. It limits the download to last 500 files.
 - Change: Multi-threaded mode can now be enabled by setting the multi_thread variable. This will set the thread count to the number of available processors, multipled by five. If the variable is not configured, it will default to single threaded node. Please note: Multithreaded mode has known issues withe files that have duplicate file names. Use with caution.

20201012

 - Change: Lowered 60 second delay for warning about password not being stored in keyring to 15 seconds as 60's a bit long.
 - Change: Removed the check for files that have been downloaded already, as new icloudpd version uses photo fingerprinting to allow downloading of different pictures with identical names.
 - Change: Added two variables, download_notifications and delete_notifications. If they are not set, they will default to True. To disable, set to False.
 - Change: Dockerfile now installs pip dependencies using requirements.txt instead of having them specified in the dockerfile.
 - Change: sync-icloud.sh script now shows the version of Python, icloudpd and pyicloud-ipd that are being used.
 - Change: Noted that folder structure can now be set to 'none' to download to flat file structure.
 - Remove: diff file that applies the patch to skip incorrectly named files. Pull request has now been merged.

20200909

 - Change: Undersocres now being used in cookie file name. Amended to reflect change.

20200905

 - Change: Made the webhook path configurable.

20200825

 - Change: Removed the Powershell script to build separate images for ARM architectures as I'm now producing a single multi-arch image without using Windows.

20200823

 - Change: Version is now date and time of last modification of the sync-icloud.sh script.

20200822

 - Change: Added force_gid variable so that user can be added to system groups to maintain consistency with host OS.
 - Change: Rewrote the functions to create the user/group as it would fails if a system reseved group id is chosen.

20200815

 - Change: Amended the interactive terminal check to see if /dev/stdin is a pipe. Odd issue seen on some systems.

20200808

 - Change: Updated READ.ME.

20200807

 - Change: Updated READ.ME.
 - Change: Apply diff patch to fix album naming issue raised here: https://github.com/ndbroadbent/icloud_photos_downloader/issues/150.

20200806

 - Change: Removed manual versioning in Dockerfile in favour of MD5 script hash. Means I don't need to track version numbers, or remember to increment them, but latest version can be confirmed quite easily from the log.
 - Change: Added owner change to /config dir at start of ConfigurePassword function so password is successfully stored first time around.

20200803

 - Change: Converted docker-compose.yml file to be 3.8 compatible. Removed networking configuration.

20200803

 - Change: When using keyring it was claiming plaintext password was in use and vice versa.
 - Change: Added a version number in Dockerfile to display when script first starts up.

20200802

 - Change: Found that the --directory option is required. I wasn't including it when doing the 2FA authentication cookie generation which lead to an error.

20200608

 - Change: Rebased to Alpine 3.12.
 - Change: Display IP Address in log.

20200528

 - Alpha change: Changed Webhook notification messages.
 - Alpha change: Forgot to add webhook_id to the check which validates that the API keys have been set.

20200527

 - Alpha change: Webhook host_ip_address variable changed to webhook_server. Added webhook_port so a non-default port can be added.

20200526

 - Alpha: Webhook notifications implemented for Home Assistant. Based on Telegram notifications, but implemented without testing. Using them will probably break things as I'm unsure as to the format of the webhook payload.

20200519

 - Change: Prefixed some of the files stored in /tmp/icloudpd with icloudpd_, just for consistency. Plus it's easier to remove them all with `rm icloudpd*` instead of multiple arguments to rm.
 - New: Keyring password saving. Previously, the Apple ID user's password could be seen in the Docker logs, the variables (docker inspect) and disturbingly, if any user did `ps -ef | grep password", it would be seen in the command line of any running connection to iCloud. Script now heavily pushes users towards saving credentials into the secure encrypted keyring.
 - Removal: Saving password to text file has been removed (never made it to a release, but is doumented in the change log).
 - Change: su doesn't pass variables to the account it uses. This meant that some variables that I thought were optional (user, config_dir, folder_structure) were actually mandatory. Command now built with $0 $1 $2 etc and variables passed via command line ( -- $bunch $of $options).
 - Removal: Debug logging variable removed. Always logging errors to screen.
 - Change: Telegram notification success/failure messages now confirm message type (startup, download, delete, failure).
 - Change: Cookie checking function would hang system without error if 2FA cookie does not exist.

20200518

 - Change: Container now saves the Apple ID password to a file located in the /config volume. Once this has been done, the apple_password variable not longer needs to be passed to the container and should be removed.

20200517

 - Change: Moved some of the logging commands so they triggered in more appropriate places.
 - Change: File download check error notification incorrectly labelled as a startup notification.
 - Change: Script wasn't properly handling passwords with special characters. Should be OK now.
 - Change: Amended the failsafe error message so that the name and path of the file it is looking for is visible. Makes it easier for people to spot the capitalisation in iCloud.
 - Change: File check exit code grabbing logic was flawed. Was actually grabbing the exit code of su, which was always 0.

20200514

 - Change: Previous change didn't work. This actually fixes the issue that the list of files is no longer growing. However, it's only highlighted a different issue. A number of files are exhibiting weird behaviour. 5 files are being listed as downloaded, but only 3 filenames are unique. These files are then also being listed as having been deleted from the system. They are all stil in my photostream on iCloud.com and not in recently deleted. Coincidentally, they all start with camphoto_ but that may not be indicitive of anything.

20200508

 - Change: My list of downloaded/deleted files seemed to be getting longer without me actively using my phone's camera. Guessing some variable isn't being cleared, so have localised a fair few variables to the functions they run in.

20200430

 - Change: Added $debug_logging variable, which outputs errors to the screen instead of hiding them.

20200413

 - Remove: Removed PR 132. PR 132 was based off PR 110, which was raised in 2018. Neither had been merged. Obviously, after I spent an hour or so chainging my Dockerfile to incorporate the patch, it was merged upstream!

20200412

 - New: Added PR 132: "Multi-threaded downloads and caching" from https://github.com/ndbroadbent/icloud_photos_downloader/pull/132.
 - New: Added quotes to variables to prevent globbing in download check and photo download functions.

20200313

 - New: Added interactive_only variable. Some hosts only run containers interactively (looking at you Synology) and this means the script gets stuck attempting to create a 2FA cookie every time. Setting interactive_only will force the script to bypass the cookie generation function and sync files instead.

20200313

 - New: Log now displays OS version.

20200309

 - Removal: Removed feature to delete JPG file if the HEIC has been removed. Floored logic in that all files exist in a single location, which doesn't match their original location. Also, JPG files received by WhatsApp would be removed as would JPG files saved from Safari. Can't work in practice.
 - New: Script now has a command line parameter that can be run from an interactive session. --ConvertAllHEICs will scan all HEIC files and convert them to JPG, provided the JPG doesn't exist already.
 - Change: Convert HEIC to JPEG not working correctly.

20200217

 - Change: Running the sync client with "--only-print-filenames" actually creates folders too. File check now run as user account instead of root to avoid ownership issues.

20200211

 - Change: Fancy Telegram notifications caused the underscore characters in the file previews to be displayed in italics. Should be correctly escaped now.

20200209

 - Change: Odd issue where owner on config directoy changed to root. Added chown at start of sync to fix this, and also to Owner/Permissions setting function.
 - New: Fancy pants Telegram notifications with emojis.
 - Removal: Telegram URL Encoding function. Now sending markdown text and UTF-8 for special characters.
 - Change: Renamed $syncronisation_interval variable to $synchronisation_interval.

20200123

 - Update: Now based on Alpine Linux 3.11.
 - New: Apple ID displayed on push notifications.
 - Update: ARM images rebuilt to latest version.

20200122 - Changes:

 - New: HEIC to JPEG conversion.
 - New: JPEG removal if original HEIC  removed (experimental, backup your files!).
 - Change: Code changes for readability and simplicity.
 - New: Added example docker-compose.yaml.

20200118

 - Change: Variable name changes for readability and naming accuracy.
 - New: Pre-download check for new files so a download run will only occur if new files exist.
 - New: Telegram notifications.
 - New: Synchronisation summary. Number of new files downloaded. Number of deleted files (if --auto-delete enabled).
 - New: Telegram notifications only: List of downloaded filenames (10 max). List of deleted files (10 max).
 - New: Startup notification.
 - New: Configurable permissions on the download destination directory.
 - Change: Logic re-writes for simplicity and optimisation.
 - Change: Additional logging.
 - Change: Code clean-ups.
 - Change: Healthcheck update.