Icloud not fully supported

[tobixen]

Updated summary

Updated 2021-03-08.

The library can be used for basic access into events on calendars in iCloud, as well as to save changes.

As far as I've understood, Apple has never officially said that they do support CalDAV in iCloud. iCloud does use some variant of the Apple CalendarServer, which is supposed to be very standards-compliant, but development on the CalendarServer has unfortunately been discontinued ... and whatever version Apple is using, there are lots of regression issues, pushing out new features in iCloud obviously have higher priority than to keep the calendar compliant with the standards. The things that doesn't work now will probably never work, and I haven't planned to do more work on improving iCloud compatibility as for now.

I will close this issue, as no more work is planned to be done on icloud support.

Some things to be aware of:

• Ref @markus0m one may need to use app-specific passwords, see https://support.apple.com/en-us/HT204397
• It doesn't seem like iCloud supports journal entries (but who cares ... does anyone use those anyway?)
• iCloud doesn't support freebusy-requests.
• While iCloud do have some "task" system, it does not seem like it's possible to access those through the caldav interface (If you have iCloud and a tasklist with tasks - try using .objects() rather than .todos() and see if you get any tasks?)
• All test code dealing with recurring events have been disabled, it seems like they don't support recurring events through their CalDAV interface. It's probably possible to do more research on this one ... but it's not on my list.
• Icloud uses calendar URLs like https://p12-caldav.icloud.com/17112341234/calendars/12345@example.com/ - the numbers (even in the host name) differs from account to account. There is no need to wonder too much about this, specify the caldav url to be https://caldav.icloud.com/, and the library can figure out the rest - code examples at Icloud not fully supported #3 (comment)
• As per the comment above ... I'm not much happy with the way the specific URL gets stored, I think it's a quite dirty hack (objects.py, Principal.calendar_home_set, the url in the associated DAVClient objects gets changed), an unexpected side effect that is bound to cause unexpected behaviour further down the road.
• The test code is (by default) creating and deleting the same calendar repeatedly. I did observe some times that events stored on the calendar would not disappear if the calendar was deleted and then recreated. I consider that to be a bug, but a bug that never should be encountered in ordinary use cases. (could be that the bug is in the test code - did not investigate much).
• It seems that one unique calendar object with one unique uid cannot be on two calendars at the same time. I'm not sure what the RFC says about this - I think it's a common use case scenario to have one and the same event on multiple calendars, and then it ought to have the same uid also.
• A test doing a propfind failed and has been skipped. Not sure if it's something wrong with the iCloud or if it's something wrong with the test.
• The method object_by_uid (as well as event_by_uid) fails. I vaguely remember someone telling me there was an issue with it on Google as well. Should be investigated, perhaps the logic should be rewritten or some workaround should be found for iCloud (and Google?).
• I had to do some minor workaround in the code for fetching event lists - the calendar URL is sometimes passed in the response together with the events, causing the library to handle it as an extra event. I haven't bothered checking the RFC, but wrote up some lines to check for this and throw it away.

Except for those things ... all tests passes by now.

This ticket is closed as for now, but feel free to update it if you have more information. If it's possible to work around any of the problems above without introducing too much complexity in the library, this issue may be reopened.

[tobixen]

The quick TLDR; people have reported various issues when using this caldav library towards the icloud. I don't seem to have a working test-account towards the icloud at the moment, hence I'm not working actively on research here. People using icloud is encouraged to do research.

(actually, I vaguely remember someone provided me with a test account once upon a while - it may be that I've simply lost the credentials for it).

[maxandersen]

you can setup a free account on icloud.com and just need to login to it once from iphone or mac to get calendar access. I can also do that if you don't have access - to where should I send the credentials?

[tobixen]

@maxandersen your reply here went completely under my radar, sorry for the slow response. Please send to t-caldav@tobixen.no as I don't have an Apple device myself.

@kyloe has successfully managed to create events on his iCloud account using the caldav library, but he had to write up his own discovery methods to find the appropriate URLs. I hope we can get his work integrated into the caldav library soon, so caldav access towards iCloud will just work. I probably won't have any time looking into this myself until mid-September, best case.

[tobixen]

commit 975a4d1 gives an example on how to get the correct URLs on the iCloud platform (though, only tested for one account).

[kyloe]

Hi Tobias Yes, you're right - requests is simpler - have simplified the code now to use the requests library My guess is that I should - Update locally - Commit - Push to my fork - Make another pull request Just checking before I make a fool of my self rgds Ian B

…

On 9 August 2017 at 10:14, Tobias Brox ***@***.***> wrote: commit 975a4d1 <975a4d1> gives an example on how to get the correct URLs on the iCloud platform (though, only tested for one account). — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#3 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AGiw9ezQmWIwVd3CkbqBPwnXgkSWg-n2ks5sWXiFgaJpZM4LrxN0> .

[tobixen]

My guess is that I should - Update locally

Yes, in general you should pull from the official repo. You will probably be a minor doc change if you haven't pulled it in already.

- Commit - Push to my fork - Make another pull request

Seems correct. :-) With the request lib in place, it will be easier to read your code, and hence also easier to understand how the caldav library should be modified to support this out of the box.

…

-- Tobias Brox Senior System Consultant Redpill Linpro AS - Changing the game Mobil: +47 917 000 50 Kontor: +47 215 441 68

[maxandersen]

@maxandersen your reply here went completely under my radar, sorry for the slow response. Please send to t-caldav@tobixen.no as I don't have an Apple device myself.

i set one up and send info to tobixen.no - but actually it looks like you can now create icloud accounts without an apple device.

[tobixen]

Finally getting back to this one. Running tests now towards the account provided by @maxandersen

[tobixen]

caldav.lib.error.AuthorizationError: AuthorizationError at 'https://caldav.icloud.com/', reason 'Unauthorized'. I can't log in there with the credentials given, was that the wrong URL maybe?

I do get into https://icloud.com/ interactively, but I see no calendaring options there.

[maxandersen]

I saw your login in my notifications. It should all be there - I just created a normal account.

[markus0m]

Are you using app-specific passwords for caldav?
https://support.apple.com/en-us/HT204397

Apart from that, you'll have to build the URL as described here:
https://www.techrepublic.com/article/how-to-find-your-icloud-calendar-url/

Hope this helps you test, desperately waiting for that todo list support ;)

[jendas1]

Same situation here, getting 500 Internal Server Error when listing todos.

client.principal().calendars()[0].todos()

Any workaround or a solution would be highly appreciated.

[iVlozz]

From the two comments above it sounds like its working with calendars and its just not working with todos. Im only trying to get events from my iCloud calendar but everything I tried so far wouldn't work. Even the icloud example code isn't working for me. I have absolutely no clue what Im doing wrong, could somebody maybe help me there?

[forgoty]

Hello.
Big thanks for the work that had been done. I really appreciate it.

My question is:
Do the caldav library support expanding icloud calendar recurrent events?
In calendar I have only one recurring daily event. When I am trying to:

 events = calendar.date_search(start=start, end=end, expand=True)

There is only one master event without recurrent ones.

@tobixen, can you help me just to clarify what I am doing wrong?

Many thanks!

[tobixen]

I have observed the same on other calendar servers also, so I should investigate. Will raise a separate issue for it.

[tobixen]

@forgoty - I've done a quite thorough investigation on the expand-issue. I'm assuming that the iCloud handles expand correctly, but that the caldav interface is simply not intuitive enough (and should be documented as well!)

So, in a calendar with one recurring event, and doing a date search covering multiple recurrences of the same event, one will get a list with only one Event object. However, this object does contain multiple vEvent entries.

events = calendar.date_search(start=start, end=end, expand=True)
print(events[0].data.count('DTSTART'))

(this would print i.e. 2 if there are two occurrences in the date range)

This is confusing, especially as the vobject class does not seem to handle it very well.

events = calendar.date_search(start=start, end=end, expand=True)
print(events[0].instance.vevent.dtstart.value)

... this will print only one starting time.

I'm planning to switch to icalendar in the 1.0-release, and find some better/more intuitive solution to this later.

[forgoty]

@tobixen
Thanks for clarify this behavior. Now I am tremendously waiting for 1.0-release

For all of us who is using icalendar and wants to get all recurrent events you can try this:

from icalendar import Event

events = [Event.from_ical(event.data) for event in calendar.date_search(start=start, end=end, expand=True)]
return [event for event in events if len(event.subcomponents) < 2 else event.subcomponents]

# [event1, event2, [recurrent_event1, recurrent_event2], event3]

[tobixen]

it's also possible to call .components on a vobject-instance, that's what I did in calendar-cli eventually.

There has been two pull requests so far for replacing vobject with icalendar, I've rejected them because I think it's important to preserve backward-compatibility. However, at some point I renamed .instance to .vobject_instance with the alias .instance for preserving backward-compatibility. My plan was to introduce .icalendar_instance. It shouldn't be many extra lines of code, perhaps I should sneak it into the upcoming 0.7.

[tobixen]

Looking into some old test/conf_private.py I found an account bramblecucumber@icloud.com with a given password. I can log into the web console.

Entering caldav.icloud.com as the caldav URL, I get a 401.

So, if I read https://www.techrepublic.com/article/how-to-find-your-icloud-calendar-url/ correctly, the caldav URL is actually secret. I follow the instructions there and https://p34-caldav.icloud.com/8253573109/calendars/Work should be a valid calendar URL for the bamblecucumber user - but no dice, still get a 401.

This is for a propfind to find the principal URL, but I suppose it's a generic problem with either the password or the authorization or the permissions. Hm.

I'm going to look into that example file now and see if I have any luck with it.

[tobixen]

Same thing with the example file ...

Failed to retrieve Principal: 401

I'll try to reset the password, maybe that will help.

[tobixen]

My goodness, to administer the account I need to answer some security questions. I'll try to create a new account, hopefully I'll get access to the calendar without associating it with an apple unit.

[tobixen]

"Your account cannot be created at this time."

Perhaps too much messing around with attempts to log in from the same IP/browser, duplicated accounts with same name and birth date, perhaps the email address I used was already associated with another test account.

I give up as for today.

[jdo-2002]

I was able to create a calendar entry if i use a app specific password and changed in the icloud sample file the header: 'Depth': '0', instead of 'Depth': '1',

[tobixen]

I'm running the tests towards iCloud now.

Some good news. I'm able to specify the caldav_url simply as https://caldav.icloud.com/ - and the library figures out the rest.

conn = DAVClient(username='myuser@example.com', password='hunter2', url='https://caldav.icloud.com')
principal = conn.principal()

principal.url looks like URL(https://caldav.icloud.com/17112341234/principal/)

old_calendars = principal.calendars() 
new_calendar = principal.make_calendar(name="Yep", cal_id='12345@example.com')

new_calendar.url looks like URL(https://p12-caldav.icloud.com/17112341234/calendars/12345@example.com/)

I'm able to run some tests successfully - but then it stops for me, a newly created calendar already contains an event! Possibly some race condition/synchronization issue at the icloud side (my tests does delete and recreate the same calendar many times - I could probably rewrite tests to make unique IDs for each calendar)

[tobixen]

I've been running through all the tests now. I had to do one fix or perhaps workaround in the code, and I also had to skip quite many tests (all tests involving VTODO, VJOURNAL, VFREEBUSY, events with RRULE and some more) - haven't done any research on weather the problem is in the tests, in the library or on iCloud. (Does icloud support all those?).

The summary at the top of this issue has been updated with the findings.

[tobixen]

I'm planning to include some iCloud-specific notes in the documentation, as part of #120

[tobixen]

As far as I've understood, Apple never officially supported CalDAV in iCloud. iCloud does use some variant of the Apple CalendarServer, which is supposed to be very standards-compliant, but development on this one has unfortunately been discontinued.

I've done a lot of testing recently towards iCloud, and ... what is working now works, and what doesn't work will probably never work.

There is one thing I feel uneasy about ... in Principal.calendar_home_set, the URL in the associated DAVClient object is changed to reflect the correct URL in their load balancing scheme. This feels like an unacceptable side effect that is bound to create unexpected behaviour later down the road ... but at the other hand, it seems to work out pretty well.

I will close this issue, as no more work is planned to be done on icloud support.

[tobixen]

Looking through the thread;

@markus0m @jendas1 ... if you have a task list in iCloud that contains tasks ... can you please try to use calendar.objects() and see if that helps on accessing the tasks?

[tobixen]

@forgoty ... unfortunately, none of the test code that deals with recurring events has been passing when running towards iCloud. I haven't investigated, perhaps it's possible to find work-arounds for it, but it's not on my list currently. Pull requests would probably be accepted.

[tobixen]

Also, @forgoty ... the master branch (and the latest release, if I remember correct) does support the icalendar library (though, icalendar is not yet in the list of dependencies). Just use my_event.icalendar_instance to access the data as an icalendar object. Data in the instance can be modified and will be saved back to the server when doing my_event.save().

[SolidHal]

Just figured I'd note this here, setting the filter in objects.py todos() to simply

            filters1 = cdav.CompFilter("VTODO")

let me retrieve the list of todos from an icloud account. This is annoying since we have to then do the filtering on the client side, but its better than the 500 error that occurs otherwise.