Including FSSSignalDiscovered event to list of allowed events to send over EDDN #152

norohind · 2021-10-29T22:58:46Z

It will allow EDDN listeners to:

Track FCs movements
Track combat zones
Track Resources extraction points (not quite useful since body info isn't included)
Track NonHuman signals
Track high grade emissions signals
Track installations
Match FCs IDs and names

In order to decide which fields have to be removed I analyzed all keys in all my own journals record for FSSSignalDiscovered event, here is the list with some comments:

SignalName
TimeRemaining - on USS, I don't think it can be useful for anybody but I also don't think it is private information
timestamp
SpawningFaction_Localised - candidate to removing
SignalName_Localised - candidate to removing
IsStation
USSType
USSType_Localised - candidate to removing
SpawningState
SpawningFaction
event
SpawningState_Localised - candidate to removing
ThreatLevel
SystemAddress

There is also an event type that can be considered as personal information, it is missions related signals

{ "timestamp":"2021-10-27T20:33:30Z", "event":"FSSSignalDiscovered", "SystemAddress":3107509342922, "SignalName":"$USS; ", "USSType":"$USS_Type_MissionTarget;", "USSType_Localised":"Mission target", "SpawningState":"$FactionState_None;", "SpawningState_Localised":"None", "SpawningFaction":"$faction_none;", "SpawningFaction_Localised":"None", "ThreatLevel":4 }

Also please note that every jump for single player can generate more than 100 FSSSignalDiscovered events.

Additional values:
horizons
odyssey
StarSystem
StarPos
With appropriate values as in journal-v1.0.json schema.

The text was updated successfully, but these errors were encountered:

Tkael · 2021-11-01T23:45:09Z

Localized names are already typically scrubbed from the EDDN messages, I see nothing controversial there.

I see limited value in reporting USS with TimeRemaining, since the time wherein each signal exists is quite limited and the most useful of these USS signals (HGE) can already be readily predicted from faction states and allegiances.

I would also agree with removing USS like the $USS_Type_MissionTarget; USS type event identified by the OP as personal information.

Combat zones (except perhaps for nonhuman combat zones) follow naturally from faction states as well - there is limited value to be gained from reporting individual combat zones to EDDN.

Without body information, is resource extraction data still useful?

Athanasius · 2021-11-02T11:44:42Z

There is an issue with some FSSSignalDiscovered events being written to the Journal before a jump has actually been made/completed, certainly before the matching FSDJump or CarrierJump event (maybe even before Location on login?).

Now, we could just leave this event as only having SystemAddress from the actual journal event. The point of those augmentations is for listeners to get 'full' data on a system they've not seen before without them needing to cross-check against data they hold. But this is not 100% necessary if it's going to be problematic.

If we insist on SystemName and StarPos then senders would need to cross-check that SystemAddress with a 'location' event before adding those augmentations. If it doesn't match then they can either:

Just not send this event (but this loses data).
Not include the augmentations in this mesage when sent.

Athanasius · 2021-11-02T11:47:07Z

Localized names are already typically scrubbed from the EDDN messages, I see nothing controversial there.

Agreed.

I see limited value in reporting USS with TimeRemaining, since the time wherein each signal exists is quite limited and the most useful of these USS signals (HGE) can already be readily predicted from faction states and allegiances.

Yes, no point to including this.

I would also agree with removing USS like the $USS_Type_MissionTarget; USS type event identified by the OP as personal information.

Yup, personal, and not really useful to anyone else.

Combat zones (except perhaps for nonhuman combat zones) follow naturally from faction states as well - there is limited value to be gained from reporting individual combat zones to EDDN.

Knowing how many CZ there are, and if the data is there, of what type/size, can be useful, even if you don't yet know exactly where they are. I'd leave this up to listeners to decide on the usefulness.

Without body information, is resource extraction data still useful?

Again, it might be useful, so I see no reason to not include it.

Someone has mentioned that perhaps some sort of batching of these events into a single message might be useful. Timing that post-jump might be tricky, but should be considered.

clonedArtie · 2021-11-02T12:00:33Z

Yes, the FSSSignalDiscovered can happen even before Location event at the startup, so to avoid complications (and possible incorrect values) with stuff like star system coords and name I suggest to have these properties as SHOULD rather than MUST.

The batching is a MUST, to not compromise the server (I can clearly imagine that for highly populated system congested with fleet carriers, with just a few senders at the same moment the server will need to handle hundreds of HTTP requests at once instead of just a couple of requests. I guess it will be better even for the users as their machines won't be making 100+ HTTP requests at the same time.

It can follow an easy mechanics in journal parsing:

If the event is FSSSignalDiscovered, store it to the temporal list and proceed to next event.
If the next event is also FSSSignalDiscovered, repeat 1.
If the next event is any other or there is no other event for more than 10 seconds, send the temporal list in a single message to EDDN.
???
Profit! ;)

chennin · 2021-11-02T12:25:22Z

CZs/RES

The difficulty is useful information -- I've seen requests for this information. Even without body information it's better than nothing

Athanasius · 2021-11-04T08:44:42Z

Something just mentioned on Discord is that currently Canonn gain the names (rather than just call signs) of Fleet Carriers from FSSSignalDiscovered events. Currently EDDB etc only get data with the call sign, giving no way to map them to the names.

The format is a little awkward though, as you get both the name and the call sign in the same field:

{ "timestamp":"2021-10-18T08:42:23Z", "event":"FSSSignalDiscovered", "SystemAddress":3240309557611, "SignalName":"WARD'S OLOGIES X3F-N5Z", "IsStation":true }

clonedArtie · 2021-11-04T11:16:48Z

It's easy to parse though. I certainly recommend for the specs that senders MUST NOT alter the SignalName any way.

norohind · 2021-11-04T14:42:41Z

I also think messages should be transmitted as is with minimum changes. Receivers might want to implement some special logic to parse it.

chennin · 2021-11-04T16:04:49Z

Confirming those 14 fields are all I have in my journals

Possible formats:

USS - the following may or may not appear:

SpawningState_Localised
SpawningFaction_Localised
SignalName_Localised
TimeRemaining

{
  "timestamp": "2021-01-15T02:54:18Z",
  "event": "FSSSignalDiscovered",
  "SystemAddress": 1900279744859,
  "SignalName": "$USS_HighGradeEmissions;",
  "SignalName_Localised": "Unidentified signal source",
  "USSType": "$USS_Type_ValuableSalvage;",
  "USSType_Localised": "Encoded emissions",
  "SpawningState": "",
  "SpawningFaction": "Free Marlinists of Carinae",
  "ThreatLevel": 0,
  "TimeRemaining": 718.508789
}
{
  "timestamp": "2021-09-23T14:53:10Z",
  "event": "FSSSignalDiscovered",
  "SystemAddress": 1733119939282,
  "SignalName": "$USS_DegradedEmissions;",
  "SignalName_Localised": "Unidentified signal source",
  "USSType": "$USS_Type_Salvage;",
  "USSType_Localised": "Degraded emissions",
  "SpawningState": "$FactionState_None;",
  "SpawningState_Localised": "None",
  "SpawningFaction": "$faction_none;",
  "SpawningFaction_Localised": "None",
  "ThreatLevel": 0,
  "TimeRemaining": 975.480469
}
{
  "timestamp": "2021-01-15T02:12:30Z",
  "event": "FSSSignalDiscovered",
  "SystemAddress": 1900279744859,
  "SignalName": "$USS; ",
  "USSType": "$USS_Type_MissionTarget;",
  "USSType_Localised": "Mission target",
  "SpawningState": "$FactionState_None;",
  "SpawningState_Localised": "None",
  "SpawningFaction": "$faction_none;",
  "SpawningFaction_Localised": "None",
  "ThreatLevel": 4
}
{
  "timestamp": "2020-07-31T12:46:05Z",
  "event": "FSSSignalDiscovered",
  "SystemAddress": 938173532515,
  "SignalName": "$USS_SalvageHaulageWreckage;",
  "SignalName_Localised": "Unidentified signal source",
  "USSType": "$USS_Type_Aftermath;",
  "USSType_Localised": "Combat aftermath",
  "SpawningState": "$FactionState_Outbreak_desc;",
  "SpawningState_Localised": "An outbreak reflects the spread of a virulent disease among members of the faction.",
  "SpawningFaction": "Noblemen of Andancan",
  "ThreatLevel": 0,
  "TimeRemaining": 655.460205
}

Station/FC

{
  "timestamp": "2021-10-30T17:43:06Z",
  "event": "FSSSignalDiscovered",
  "SystemAddress": 84456968626,
  "SignalName": "BUYING PAINITE 230K  H8N-TXQ",
  "IsStation": true
}

{
  "timestamp": "2021-10-30T12:40:33Z",
  "event": "FSSSignalDiscovered",
  "SystemAddress": 5068732310921,
  "SignalName": "Al-Jazari Orbital",
  "IsStation": true
}

CZ/RES/etc

{
  "timestamp": "2021-10-29T11:56:53Z",
  "event": "FSSSignalDiscovered",
  "SystemAddress": 1733119906514,
  "SignalName": "$Warzone_PointRace_High:#index=1;",
  "SignalName_Localised": "Conflict Zone [High Intensity]"
}

Full list of $ signal names I've encountered. I've noted the indexes I've seen, but there could be larger - includes USS (above)

$Aftermath_Large:#index=1; # Goes from 1-2
$AttackAftermath;
$FIXED_EVENT_CAPSHIP;
$FIXED_EVENT_CHECKPOINT;
$FIXED_EVENT_CONVOY;
$FIXED_EVENT_DEBRIS;
$FIXED_EVENT_DISTRIBUTIONCENTRE;
$FIXED_EVENT_HIGHTHREATSCENARIO_T5;
$FIXED_EVENT_HIGHTHREATSCENARIO_T6;
$FIXED_EVENT_HIGHTHREATSCENARIO_T7;
$Fixed_Event_Life_Cloud;
$Fixed_Event_Life_Ring;
$Gro_controlScenarioTitle;
$ListeningPost:#index=1;
$MULTIPLAYER_SCENARIO14_TITLE;
$MULTIPLAYER_SCENARIO42_TITLE;
$MULTIPLAYER_SCENARIO77_TITLE;
$MULTIPLAYER_SCENARIO78_TITLE;
$MULTIPLAYER_SCENARIO79_TITLE;
$MULTIPLAYER_SCENARIO80_TITLE;
$MULTIPLAYER_SCENARIO81_TITLE;
$NumberStation:#index=1;
$USS; # This one has a trailing space!
$USS_CeremonialComms;
$USS_ConvoyDispersalPattern;
$USS_DegradedEmissions;
$USS_DistressCall;
$USS_HighGradeEmissions;
$USS_NonHumanSignalSource;
$USS_SalvageHaulageWreckage;
$USS_Type_Aftermath;
$USS_Type_Convoy;
$USS_Type_DistressSignal;
$USS_Type_NonHuman; $USS_ThreatLevel:#threatLevel=4; # Goes from 4-9. Yes this is one entry
$USS_Type_WeaponsFire;
$USS_WeaponsFire;
$Warzone_PointRace_High:#index=1; # Goes from 1-3
$Warzone_PointRace_Low:#index=1; # Goes from 1-10
$Warzone_PointRace_Med:#index=1; # Goes from 1-4

Installations/Beacons/Outposts(!)

{
  "timestamp": "2021-10-26T21:13:26Z",
  "event": "FSSSignalDiscovered",
  "SystemAddress": 121569805492,
  "SignalName": "Enlightened Mathematic Analysis"
}
{
  "timestamp": "2021-10-29T12:02:34Z",
  "event": "FSSSignalDiscovered",
  "SystemAddress": 5068732310921,
  "SignalName": "SVA-046 Dionysus-class Cropper"
}
{
  "timestamp": "2021-10-26T20:25:20Z",
  "event": "FSSSignalDiscovered",
  "SystemAddress": 24859942069665,
  "SignalName": "What is Interpol?"
}
{
  "timestamp": "2021-10-26T21:35:04Z",
  "event": "FSSSignalDiscovered",
  "SystemAddress": 4305444669811,
  "SignalName": "Sekelj Port"
}

For my reference as I don't see a list on Google, EN localized names for the non-obvious ones:

$FIXED_EVENT_HIGHTHREATSCENARIO_T5; $FIXED_EVENT_HIGHTHREATSCENARIO_T6; $FIXED_EVENT_HIGHTHREATSCENARIO_T7;: Pirate Activity Detected
$MULTIPLAYER_SCENARIO14_TITLE;: "Resource Extraction Site"
$MULTIPLAYER_SCENARIO42_TITLE;: "Nav Beacon"
$MULTIPLAYER_SCENARIO77_TITLE;: "Resource Extraction Site [Low]"
$MULTIPLAYER_SCENARIO78_TITLE;: "Resource Extraction Site [High]"
$MULTIPLAYER_SCENARIO79_TITLE;: "Resource Extraction Site [Hazardous]"
$MULTIPLAYER_SCENARIO80_TITLE;: "Compromised Navigation Beacon"
$MULTIPLAYER_SCENARIO81_TITLE;: "Salvageable Wreckage"

norohind · 2021-11-04T16:52:08Z

Out of curiousity, have you got any with IsStation = False?

chennin · 2021-11-04T16:53:36Z

Nope, none

kenjij · 2021-12-30T07:14:02Z

One valuable info that can be obtained from this schema would be Nav Beacon status.

{ "timestamp":"2021-11-12T05:16:07Z", "event":"FSSSignalDiscovered", "SystemAddress":147949848947, "SignalName":"$MULTIPLAYER_SCENARIO42_TITLE;", "SignalName_Localised":"Nav Beacon" }

{ "timestamp":"2021-11-12T05:15:10Z", "event":"FSSSignalDiscovered", "SystemAddress":3996240546163, "SignalName":"$MULTIPLAYER_SCENARIO80_TITLE;", "SignalName_Localised":"Compromised Navigation Beacon" }

Athanasius · 2022-01-09T20:10:51Z

For anyone following this and wondering if it's been forgotten, it hasn't.

However I do have some concerns about the possible flood of data adding this would cause, so there are some caveats.

We need a sender implementation - if anyone wants to beat me to a PR against ED Market Connector, please do. The purpose of this is to actually test it in action to ensure that we identify any issues with things like batching the events for arrival in a new system into just a single message. During this testing phase it should only attempt sending to beta.eddn.edcd.io (which isn't always up and working).
That will match up with a beta implementation of the schema running on beta.eddn.edcd.io.
We then need to at least get an idea of the message and data rate when e.g. a commander flies around the populated (more signals from things like FCs) bubble for an hour. We'd then combine this with some guesstimate of how many Cmdrs might do that over the course of a week to get an idea of any extra load it would cause on both the EDDN Gateway and listeners.
Some listeners may well not want this data at all. Ideally they would use ZeroMQ features to filter it out. However the version of zeromq that is supported by the legacy version we're currently using for all of EDDN doesn't perform server-side subscription filtering, i.e. the data would get sent to listeners anyway, even if they then subscription-filter it on their end. Thus:
1. We would need to complete the migration to Python3 and verify that server-side subscription filtering works with the version that uses.
2. Verify that we are definitely setting ZeroMQ topic correctly per message so that this actually functions.

So, apart from anything else this is now dependent on first completing the migration to Python3. If no-one has the ability and time to assist me in verifying that #155 doesn't break any functionality or introduce regressions, then this is further dependent on implementing as many automated unit and functional tests as possible. That's a whole project in itself.

chennin · 2022-06-12T15:49:13Z

I parsed all my journals. Of hours that have at least one FSSSignalDiscovered:

Top 10:

Hour FSSSignalDiscoveredCount
2022-03-26T18 3963
2022-03-25T18 3019
2021-09-20T12 2780
2022-03-25T17 2376
2022-06-11T23 2360
2022-06-11T21 1834
2022-03-25T19 1729
2021-09-20T14 1672
2022-03-25T00 1613
2022-01-08T14 1534

Median: 64
Average: 132

If anyone else wants to measure, get your journals on a Linux-like system, then here's one way to do it:

for i in $(ls Journal*); do grep FSSSignalDiscovered $i | jq -r '.timestamp'  | awk -F: '{ count[$1]++ } END { for (i in count) print i, count[i] }' ; done | tee output
#top: 
cat output | sort -k2 -nr | head
#median: 
cat output | sort -k2 -nr | awk '{print $2}' | awk '{a[NR]=$0}END{print(NR%2==1)?a[int(NR/2)+1]:(a[NR/2]+a[NR/2+1])/2}'
#average: 
cat output | sort -k2 -nr | awk '{print $2}' | awk '{x+=$0}END{print x/NR}'

Athanasius · 2022-08-04T14:02:10Z

The new schema has been live for many weeks now without anyone bringing up an issue, so closing this.

norohind mentioned this issue Nov 7, 2021

FSSSignalDiscovered event support #154

Merged

Athanasius added enhancement help wanted journal event schemas labels Jan 9, 2022

This was referenced Jun 12, 2022

Implement FSSSignalDiscovered sender chennin/EDMarketConnector#2

Closed

Implement FSSSignalDiscovered in EDDN plugin EDCD/EDMarketConnector#1586

Closed

Athanasius closed this as completed Aug 4, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Including FSSSignalDiscovered event to list of allowed events to send over EDDN #152

Including FSSSignalDiscovered event to list of allowed events to send over EDDN #152

norohind commented Oct 29, 2021 •

edited

Loading

Tkael commented Nov 1, 2021 •

edited

Loading

Athanasius commented Nov 2, 2021

Athanasius commented Nov 2, 2021

clonedArtie commented Nov 2, 2021 •

edited

Loading

chennin commented Nov 2, 2021

Athanasius commented Nov 4, 2021 •

edited

Loading

clonedArtie commented Nov 4, 2021

norohind commented Nov 4, 2021

chennin commented Nov 4, 2021 •

edited

Loading

norohind commented Nov 4, 2021

chennin commented Nov 4, 2021

kenjij commented Dec 30, 2021

Athanasius commented Jan 9, 2022

chennin commented Jun 12, 2022 •

edited

Loading

Athanasius commented Aug 4, 2022

Including FSSSignalDiscovered event to list of allowed events to send over EDDN #152

Including FSSSignalDiscovered event to list of allowed events to send over EDDN #152

Comments

norohind commented Oct 29, 2021 • edited Loading

Tkael commented Nov 1, 2021 • edited Loading

Athanasius commented Nov 2, 2021

Athanasius commented Nov 2, 2021

clonedArtie commented Nov 2, 2021 • edited Loading

chennin commented Nov 2, 2021

Athanasius commented Nov 4, 2021 • edited Loading

clonedArtie commented Nov 4, 2021

norohind commented Nov 4, 2021

chennin commented Nov 4, 2021 • edited Loading

norohind commented Nov 4, 2021

chennin commented Nov 4, 2021

kenjij commented Dec 30, 2021

Athanasius commented Jan 9, 2022

chennin commented Jun 12, 2022 • edited Loading

Athanasius commented Aug 4, 2022

norohind commented Oct 29, 2021 •

edited

Loading

Tkael commented Nov 1, 2021 •

edited

Loading

clonedArtie commented Nov 2, 2021 •

edited

Loading

Athanasius commented Nov 4, 2021 •

edited

Loading

chennin commented Nov 4, 2021 •

edited

Loading

chennin commented Jun 12, 2022 •

edited

Loading