Add a way to persist dataset even if exception is thrown

[amritghimire]

Refactoring upon the changes at #494 from the comments, this introduces the way to allow a way to persist dataset even if exception is thrown. With this change, the cleanup is now done at the active context. So, we can use the session context as checkpoints now. Example implementation is as:

datachain/tests/scripts/feature_class_exception.py

Lines 1 to 34 in 029e677

	import logging
	from datachain import Session
	from datachain.lib.dc import DataChain
	logging.basicConfig(level=logging.INFO)
	# Dataset with variable session. Will persist.
	session = Session("asVariable")
	dv = DataChain.from_values(key=["a", "b", "c"], session=session)
	dv.save("passed_as_argument")
	# A datachain created in global context.
	# This will be reverted back due to the global exception.
	DataChain.from_values(key=["a", "b", "c"]).save("global_test_datachain_v1")
	with Session("local"):
	# A datachain created in local context.
	# This will persist since error occur in global context.
	DataChain.from_values(key=["a", "b", "c"]).save("local_test_datachain")
	try:
	with Session("local_failure"):
	# A datachain created in local context.
	# This will not persist since error occur in local context.
	DataChain.from_values(key=["a", "b", "c"]).save("local_test_datachain_v2")
	raise ValueError("Local failure class")
	except ValueError:
	pass
	# We return to global context. So, this will also be reverted.
	DataChain.from_values(key=["a", "b", "c"]).save("global_error_class_v2")
	raise Exception("This is a test exception")

In this file, local_test_datachain should only persist since it is the only dataset where the exception doesn't occur in current context.

Closes #500

[cloudflare-workers-and-pages]

Deploying datachain-documentation with    Cloudflare Pages

Latest commit:	501ad01
Status:	✅  Deploy successful!
Preview URL:	https://87fed5ff.datachain-documentation.pages.dev
Branch Preview URL:	https://amrit-persist-error.datachain-documentation.pages.dev

View logs

[codecov]

Codecov Report

Attention: Patch coverage is 95.83333% with 1 line in your changes missing coverage. Please review.

Project coverage is 87.11%. Comparing base (e0c654e) to head (501ad01).
Report is 3 commits behind head on main.

Files with missing lines	Patch %	Lines
src/datachain/query/session.py	95.45%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #504   +/-   ##
=======================================
  Coverage   87.10%   87.11%           
=======================================
  Files          92       92           
  Lines        9852     9832   -20     
  Branches     1350     1349    -1     
=======================================
- Hits         8582     8565   -17     
  Misses        917      917           
+ Partials      353      350    -3     

Flag	Coverage Δ
datachain	87.08% <95.83%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[shcheklein]

session can be also passed to DataChain explicitly (check from_storage, for example) ... it's probably expected that we pass it directly into all the datacnain factories even when we use contextmanager ... not sure get_current_context is the right approach ... at least it might be changing the assumption ... can we check docs / existing tests please?

[amritghimire]

Updated the logic along with example in the file where the variable session is used. PTAL.

[dreadatour]

Looks good to me 👍

[shcheklein]

Approving to move forward, but this design Session -> Catalog -> Sessions looks a bit weird to me.

[amritghimire]

Approving to move forward, but this design Session -> Catalog -> Sessions looks a bit weird to me.

Thanks. Modified the chain logic.