Where does the reporting happen?

[michaelherger]

I've had a report of a user who happens to be an artist, too. He claims that when he's listening to his own music using my implementation of librespot, the tracks would not be reported. This is definitely not what I want to happen, therefore I'm a bit concerned.

And I must admit I've been wondering why my kids' listening habits hadn't polluted my profile yet... (but I was less concerned about that :-D).

Is there code in librespot which actively does report when a track has been listened to? I was assuming that telling librespot to play a track would do all the magic for me.

I'm using a slightly modified fork of librespot (https://github.com/michaelherger/librespot), and I'm not sure whether one of my modifications is causing this mis-behavior or not.

[michaelherger]

Anyone?

[devgianlu]

How can we test whether the playback history is being recorded or not? I am willing to fix this issue for librespot-java, but I need a way to test it properly.

[michaelherger]

That's a good point and something I'll try to do: you can get the recently played tracks using the web API (the call is still beta... hopefully that won't confuse even more...).

https://developer.spotify.com/documentation/web-api/reference/player/get-recently-played/

[devgianlu]

@michaelherger I've checked and it seems that tracks played from librespot aren't reported by that endpoint. I will investigate this further.

[michaelherger]

Thanks for the confirmation!

[devgianlu]

Unluckily I don't see any specific call to report that a track is playing. I am afraid that it is done via the event-service endpoint which hasn't been reversed engineered at all. The only (mercury) packet that has the track uri in it is the following:

0078  # Changes every time, not incremental

5549496e746572616374696f6e # "UIInteraction"

0935  # Padding + "5"

0909  # Double padding
706c61796c697374  # "playlist"

09  # Padding
73706f746966793a6170703a706c61796c697374  # "spotify:app:playlist"

0909  # Double padding
32  # "2", could be track index in playlist

09  # Padding
73706f746966793a747261636b3a3367696379666945564d474f4e677a796770576a4e54  # Track uri

0909  # Double padding
6c6973742d6974656d2d73656c6563746564  # "list-item-selected"

09  # Padding
31353637303834303738323536  # Timestamp in millis

There's no reporting in the HTTP API either. Android doesn't seem to have any of this either.

I've also noticed that the Recently played API updates quite slowly, so maybe librespot-java works with reporting? Don't know.

[ashthespy]

@devgianlu There is also an activity-manager endpoint that I recall seeing, but didn't investigate further.

[devgianlu]

@devgianlu There is also an activity-manager endpoint that I recall seeing, but didn't investigate further.

I did not see any of that. Maybe it has been deprecated when they upgraded the protocol.

[ashthespy]

This was from an older version indeed. I need to find my notes.
Also might be of interest:

hm://played-state/v1/items
hm://played-state/v1/updates
hm://played-state/v1/unsubscribe
hm://played-state/v1/items?fromTimestamp=%Lu

[devgianlu]

I think that's for episodes only.

[ashthespy]

You are probably right again, its been a while I looked at this!

[caba8685]

Any updates on this?

[devgianlu]

We are stuck on trying to find a way to check if the changes on the client work.

[sashahilton00]

It’s probably handled via the events service. I’d be surprised if Spotify does the play reporting client side, would make it too easy to abuse. Then again, I can’t see how they track it server side unless they’re doing some sort of calculation based on the number of chunks sent to a client

[ashthespy]

I was thinking about this some time back, and I guess they probably have metrics based on client type, and since we report librespot they could just not be propagating those metrics?

[devgianlu]

More work has been done and I finally got it working. There is still some work to do as they don't pick up tracks that end because of pressing some button (aka works only with trackdone).

There are also many fields that I don't know the purpose of. Here's a breakdown of the most interesting packet:

17 # Incremental ID
6cfc77983f62f1d2150a0eb7350ad303f8074ea5 # Device ID
012c7792bd5fe36573369d32966002c0 # Playback ID
00000000000000000000000000000000 # Parent playback ID (?)
library-collection appload # Where/how the track started
library-collection endplay # Where/how the track ended
648668 # Something related to the file size (may be equal to it)
3767776 # File size (twice)
29385 29385 # When the track ended (twice?)
191743 # Track duration
45 0 0 0 1 # No clue
75877 # When the track started
269 -1 context 146 0 0 0 1 0 # No clue
29385 29385 # When the track ended (twice, again?)
0 # No clue
160000 # Bitrate
spotify:user:11145089019:collection # Context URI
vorbis # Encoding
5988f2d40bcb442b92e324f7c486657f # Track ID
b5b84892efb54ddda3a24deb216f6b46 # Parent track ID (?) (usually absent) 
0 1586783725809 0 context # Some fixed values + timestamp
spotify:app:collection-songs 1.1.26 # Stuff from play_origin
com.spotify none none local na none # Fixed values

I've enhanced the dissect script to better reverse engineer the event service: https://github.com/devgianlu/spotify-analyze

[devgianlu]

Just a quick question on the topic. Do you think it is better to send an event with some standard values that I didn't figure out or just not send it because it is not needed?

[humanperzeus]

its better to send them if they are generated to behave like a original client.
is something happened here?

[devgianlu]

is something happened here?

What do you mean?

[humanperzeus]

does librespot now sends correctly the playing track to the endpoint? because it seems like that still hearing a song, does not appearing to the endpoint of the artist

[ashthespy]

No, only librespot-java does it..

[humanperzeus]

im using librespot-java and there is still something not working regarding that endpoint. listening to a song does not appear on the artist site as someone is listening. What exactly does work now, and what not? Does it only work if a song without pressing any button like above mentioned (only via trackdone) is listened, that the endpoint get the information about trackdone?

[schox-nor]

Hi - as I'd really appreciate if streaming through librespot gets credited to the artists - is it possible to get the partial solution from librespot-java implemented in the main librespot branch - possibly also find a solution to the limitations of the current fix.

I don't like artists getting "cheated" out of the royalties for streaming.

[devgianlu]

In librespot-java I have tried to replicate the same events being sent by the official client, but it doesn't seem to work anyway. Recently they also removed a lot of events ruling out a lot of possible problems.

[ashthespy]

@Johannesd3
Stream count != play count :-)
Again, I'm no artist so it would be great to get some insight from an artist.. From what I gather they do seem to distinguish between streams and plays..

Edit: Hmm, from Spotify's Docs downloaded plays are included that we do not report.

[schox-nor]

@schox-nor Could you elaborate about this? Why do you believe streaming through librespot doesn't get credited? From what I recall, the event-services is mainly for outward facing API and/or metrics tracking. Stream count is handled directly in Spotify's backend, it would be rather naive of them to expose an API that can be spammed to inflate stream counts..

I have no way of knowing or testing - I started to suspect something was odd when I noticed that my "Spotify year in review" was way off "Last.fm year summary" - then I found this thread.

I've talked to an artist I know - and he had a retired band called Monumentum witch receive less than 10 streams a day for this track (Whore):

https://open.spotify.com/track/5scAew6okD9CUIwumeswc6?si=KqYqb1NxQH2ey594c3YHgg

If more people test some streams each through Librespot - I will report back if the stats is updated with a significant change in a couple of days. (Please don't use official apps 😅)

• OK?

[michaelherger]

@schox-nor Could you elaborate about this? Why do you believe streaming through librespot doesn't get credited? From what I recall, the event-services is mainly for outward facing API and/or metrics tracking. Stream count is handled directly in Spotify's backend, it would be rather naive of them to expose an API that can be spammed to inflate stream counts..

I think many of the questions here already have been answered towards the top of this discussion. The simplest indication is that what you play wouldn't show up in your play history. My Weekly Discovery or similar playlist has nothing to do with what I actually listen to...

I've worked with close to a dozen streaming services over the years. And it's not unusual to have the reporting implemented on the client. Most services would give the client a URL to play, but leave the control to the implementation. Whether the stream was played or not, only the client application knows. And, as has been mentioned here today, even for Spotify there must be something like that, as the offline mode would not allow Spotify to see what has been played while offline. Keep in mind that this is not "being naive", but part of a contract between the services and the developer of a client: either the reporting is implemented correctly, or you (as a developer) get blocked. librespot is different in this regard, as there's no contract, but things have been reverse engineered. And it's my understanding that the development of librespot always tried to play fair.

[devgianlu]

Using librespot-java I don't seem to have this problem. My personal playlists make sense and my yearly summary had quite a lot of artists listened to, too many to be coming from my phone alone.

I haven't checked in a while, but songs appear in the recently played API endpoint too. The only thing that was broken at the time was skipping a track.

[Johannesd3]

Btw, what happens if a downloaded track is played offline in the official client? Does that appear on the artist site?

[devgianlu]

It will probably be reported when the client goes back online. The same happens when you close the client, the last played song events are sent on the next startup.

[humanperzeus]

it does report when you go online
or it does report that value even in offline mode secretly.
Real having no internet connection, and then deleting Spotify after playing a song 1000 times means, it get never reported

[chaseadam]

Checking to see if "reporting" has made any progress. Would be good to make sure artists get the (small) sum they are due and there are other feature requests from at least one downstream client which may be related (i.e. last.fm reporting).

[roderickvd]

I totally agree and porting it from librespot-java is on my list in #917.

[agneevX]

It's honestly baffling that Spotify's API is this lacking.

[roderickvd]

I’m not sure if it’s lacking, it’s just that we haven’t fully figured it out yet.

[humanperzeus]

can just tell that
"Report played tracks back to Spotify"
does still not work on librespot-java

[roderickvd]

No, I haven't gotten around to porting it yet. Also it'll be what's in librespot-java, so whatever works or doesn't work there will then be the same in Rust librespot.

[Calamigos]

It would be great if this was implemented. Spotify would probably try to curb librespot if it was reporting back playcounts - so I can understand why it hasn't been added yet.

As it stands, using this project to
listen to Spotify facilitates paying a giant corporation ~$150/year in premium subscription and ghost-stream music. The artist are getting $0, when they would normally at least get ~$.006/play. Spotify won't take any issue with this - more money for them.

[michaelherger]

I believe reporting should be implemented if possible. Lack of reporting can be reason enough for Spotify to kill the project. There really is no reason not to do it - except for the lack of documentation, time and resources of course 😁.

I like the fact how the librespot project has continuously (tried to) shut down or stop illegal use of the code (eg. music file downloaders). It has been working hard to be a good citizen in the Spotify landscape. Implementing the reporting would only be in line with this stance.

[Calamigos]

I can supply access to artist account that has no previous traffic if that would help with testing.

[kingosticks]

I believe reporting should be implemented if possible. Lack of reporting can be reason enough for Spotify to kill the project. There really is no reason not to do it - except for the lack of documentation, time and resources of course 😁.

If you kill an entirely reverse engineered project for not reverse engineering a particular part of your private system then I think you're just looking for a reason to kill the project, because that's a ridiculous stance.

I'm not sure this reason not to implement it actually matters but it is something to think carefully about and implement carefully. Fortunately it's moot until someone works out how to do it!

Edit: here's some context https://ppcprotect.com/blog/ad-fraud/spotify-streaming-farms/

[sashahilton00]

Also regardless of the legal complexities, reverse engineering reporting and reproducing the functionality 1-1 is all we should be aiming to do - preventing potential fraud is not really within the scope of librespot, and anyone using librespot provides Spotify with all the identifying information that they need to be able to counter fake plays if they want to.

It's not our problem to deal with.

[roderickvd]

Great discussion everyone, thanks.

Given the entire reverse engineering effort was based of libspotify, I still do think there might some rudimentary server side logging for playback counts from Spotify's side.

No-one except Spotify knows for certain of course. Technically I wonder if it'd be different depending on the handshake. We don't authenticate as PRODUCT_LIBRESPOT (and haven't since I got on board since 0.1.6 I think) but as PRODUCT_PARTNER in dev and as PRODUCT_CLIENT in new-api. More and more we're leaning towards acting like a client (and looking at that traffic).

Since clients can play tracks offline, I guess that they would buffer playback history locally and report all that on connection. Again this is a total guess (I haven't looked at the traffic yet) but if so, that would be another nudge towards client-based reporting instead of server-based reporting.

EDIT: https://gitter.im/librespot-org/spotify-connect-resources?at=5fd0d65f2a954a51a9be8428 So there seems to be some metrics being collected?

Is that the same thing? I can attest that when you have librespot running and authenticated, but inactive and playing on another device, you still see spirc frames pushed but targeted at the other connection ID.

Also regardless of the legal complexities, reverse engineering reporting and reproducing the functionality 1-1 is all we should be aiming to do - preventing potential fraud is not really within the scope of librespot, and anyone using librespot provides Spotify with all the identifying information that they need to be able to counter fake plays if they want to.

It's not our problem to deal with.

Amen to that.

[glk1001]

Hi, I'm curious about the status of the reporting back to Spotify of a played track. Is this still an issue or has it been fixed somewhere else? Thanks

[roderickvd]

Still not implemented.

[gurv1r]

Is this likely to ever be implemented? Especially with the rise of apps like stats.fm, as the lack of reporting breaks the app. The mentioned app/site is however a good way to test the reporting functionality.

[sashahilton00]

It will require someone to put the legwork in to reverse and implement it. The thus far unspoken challenge is that without it artists are missing out on a few fractions of a cent, and with it implemented someone may potentially try to cheat.

[roderickvd]

Librespot-Java has got parts figured out. I just did not get around to porting it.

[CaptainBoggle]

Looks like librespot-java isn't working in this regard now though

[CaptainBoggle]

although, my own tests seem to show that the issue isn't reproducible.

[CaptainBoggle]

Perhaps an interesting method of achieving this feature would be analysing how the client stores then reports offline listens once connection is reestablished. It would probably be easier to get this right, but would make live listen information (think discord now playing etc).