Script keeps looping

[TumGuven]

After I run the script (with node.js) the script keeps repeating itself. it looks like it's in a never ending loop:

Also the files are being stored in seperate csv files instead one single lineage.csv file:

[mountaindude]

Hi @TumGuven

Butler Spyglass is going through intensive development right now. I suspect you used v1.0.0?

I quickly realised that there were things in that version that did not work well when applied to larger Sense environments. Version 1.1.0 fixed som issues, and with 1.2.0 around the corner (I expect to release it tonight) I hope to have a tool that works also in very large Sense environments.

You are right about many vs one lineage file.
Initially I stored all lineage data in a single file. But I then realised (the hard way..) that some apps might generate 10s of megabytes of lineage data... For example apps with very large inline tables would cause this to happen.

A more scalable solution was to
a) store lineage data for each app in its own file, similar to how the app's load script is stored.
b) Truncate the lineage data so only the first 1000 characters is stored. From what I have seen, the interesting lineage stuff (data source names etc) always appear first in the lineage data, it should thus be safe to cut off anything beyond 1000 characters.

Finally, regarding the looping - you are once again correct.
That is actually a feature. I wanted a setup that dumped lineage and scripts with a regular interval, for example daily or weekly. The frequency of the extraction runs is controlled by the extractFrequency parameter in the config file.

In version 1.2 there is however a new config option that will enable/disable these automatic extraction runs, making it possible to use Spyglass for one-off extraction runs (that could be scheduled by cron or some other scheduler if needed).

[mountaindude]

Additional comment re v1.2:

The readme file will be a bit more comprehensive in 1.2, which is a good thing.
There are however also more config options in the config file, so you will need to go through those options and make sure they work in your context.

Finally, you should ALWAYS get the Butler Spyglass tool from the releases page. Anything taken straight from main branch (or some other branch) might be in-development.
I know, main branch should be left alone and I usually work like that... but it's good practice to always use the packed releases - so I recommend that in this case too.

[TumGuven]

Great! Thank you for your input. I indeed took the files from the main branch. I will wait for the new version to come out.

[mountaindude]

@TumGuven
Version 1.2.1 is out.
AFAICT it will do everything you asked for, and a bit more... :)

1.2.0 had a bug in it, so go for 1.2.1.
It's been put through a fair bit of real-life testing over the past couple of weeks and I am thus pretty confident that it is working.

The one thing missing is handling of failed extracts. I.e. there are no retries if the tool for some reason gets an error back from Sense, or for some other reason fails extracting data for an app.
Fixing that is on the todo list as #1 .

Closing this ticket for now.