Storage isn't purged for non-crawlee Apify SDKv3 projects

[mvolfik]

I have a project using "apify": "^3.1.0" in package.json, but I don't specify any Crawlee packages directly. However, in my code, I use Actor.openKeyValueStore() & friends. However, now the project is not detected as CRAWLEE, nor as PRE_CRAWLEE_APIFY_SDK, so no storage purge happens with run -p

This is the section of the 0.19.1..0.19.2 diff that broke it, this is from PR #475

[B4nan]

Hmm, but with sdk v3 you should get the purge by default anyway, or not?

[mvolfik]

that's what i thought
but looking into the code, just a line before this diff above, the CLI sets CRAWLEE_PURGE_ON_START = '0', and as you cans see, it reverts that to the default 1 only if it detects crawlee, which it doesn't for my project.

[B4nan]

I see, @vladfrangu do you remember why it's like that? Feel's wrong, I'd say we should remove that line.

It looks like now when you use the Apify CLI, it won't purge stuff by default even for crawlee projects and will require explicit --purge flag? It shouldn't behave like that, it should be the exact opposite, purge by default for crawlee, and require --no-purge to opt-out (which disables the auto purge via env var).

[B4nan]

Maybe better to keep those changes for the upcoming v1 release as they are indeed breaking and could surprise users if it currently works in the opposite way as it should.

edit: on the other hand, we broke it just with the latest version ~month ago...

[vladfrangu]

it should be the exact opposite, purge by default for crawlee, and require --no-purge to opt-out (which disables the auto purge via env var).

This is edging undefined behavior terrain very fast. I also don't see why pre-crawlee should be purge opt-in but post-crawlee should be purging opt-out.

I'd say lets clean up all of this logic in v1 (including for older sdks) -> always purge, opt out only. What I tried with that PR (but missed this case, which is easy to fix actually) is to make the crawlee behavior also opt in, like for old sdk.

[B4nan]

I also don't see why pre-crawlee should be purge opt-in but post-crawlee should be purging opt-out.

Because that's what we changed on purpose with v3. In v2 you had to use the CLI to purge, with v3 we purge by default, and we wanted to keep that behavior as changing it is breaking.

always purge, opt out only

Yeah, that might be a good approach for v1. Thoughts @mvolfik?

[mvolfik]

always purge, opt out only

Yeah, that might be a good approach for v1. Thoughts @mvolfik?

If the --no-purge flag is easily found, and you don't have to dig for some CRAWLEE_NO_PURGE env var (this is a part of the reason I was opposed to this change with v3 when we spoke about this last time), let's go with it. A disadvantage is that the default behavior is destructive (deletes data), and I'm so used to running apify run -p that I don't consider it a hassle anymore, but for common development workflow probably default=purge makes more sense.

[mvolfik]

Nonetheless, unless the 1.0 is next week, can you please fix the original issue of this thread for 0.19? thanks

[B4nan]

and I'm so used to running apify run -p

Yeah, and that's the root of the problem, people in Apify are used to this while newcomers are confused by it.

@vladfrangu let's fix this in v0.19 too please

If the --no-purge flag is easily found

How often do you actually want this behavior? I would say you want this only when testing resurrection of an actor, otherwise never? Which was one of the reasons we did the behaviour switch in the first place.

[B4nan]

@mvolfik 0.19.3 is out, let us know how it works

[B4nan]

Closing as the fix is now also in the master branch.