Feature/write with blob

[MariusWirtz]

The successor of #881

Changes:

• use private application
• don't create an actual CSV file on the disk
• name application and process with hash based on tm1 session + local thread
• break process creation into a separate function

[MariusWirtz]

Things to consider before the merge

• Gather stats on the performance of. raw TI vs use_ti vs use_blob. use_blob and raw TI should presumably be en par
• Provided that use_blob is faster than use_ti we should change the write_async function to make use of use_blob as well
• challenge the uniqueness of process names and application names in multithreading environment

[rclapp]

Awesome! Thanks!

[MariusWirtz]

Here are some stats (I'm running TM1 and Python on a notebook with 4 CPUs). Huge progress IMO

Function	Runtime
write with use_ti	0:03:37
write_async with use_ti	0:00:59
write with use_blob	0:00:36
Turbo Integrator	0:00:33
write_async with use_blob	0:00:11

To reproduce:

from datetime import datetime

from TM1py.Objects.Cube import Cube
from TM1py.Objects.Dimension import Dimension
from TM1py.Objects.Hierarchy import Hierarchy
from TM1py.Services.TM1Service import TM1Service

sdata_params = {
    "address": "",
    "port": 8010,
    "ssl": True,
    "user": "Admin",
    "password": "apple"
}

CUBE = "TM1py Cube"


def setup(tm1: TM1Service):
    hierarchy = Hierarchy("TM1py Dimension 1", "TM1py Dimension 1")
    hierarchy.remove_all_elements()
    for e in range(1_000_000):
        element = str(e).zfill(7)
        hierarchy.add_element(element, "Numeric")
    dimension = Dimension(name="TM1py Dimension 1", hierarchies=[hierarchy])
    tm1.dimensions.update_or_create(dimension)

    hierarchy = Hierarchy("TM1py Dimension 2", "TM1py Dimension 2")
    hierarchy.add_element("Measure", "Numeric")
    dimension = Dimension(name="TM1py Dimension 2", hierarchies=[hierarchy])
    tm1.dimensions.update_or_create(dimension)

    cube = Cube(name="TM1py Cube", dimensions=["TM1py Dimension 1", "TM1py Dimension 2"])
    tm1.cubes.update_or_create(cube)
    print("Created Cube")


with TM1Service(**sdata_params) as tm1:
    setup(tm1)

    cells = dict()
    element2 = "Measure"
    for e in range(1_000_000):
        element1 = str(e).zfill(7)
        value = e
        cells[element1, element2] = value

    for _ in range(2):
        tm1.processes.execute_ti_code([f"CubeClearData('{CUBE}');"])
        before = datetime.now()
        tm1.cells.write(CUBE, cells, use_ti=True)
        elapsed_time = datetime.now() - before
        print(f"Write use_ti: {elapsed_time}")

        tm1.processes.execute_ti_code([f"CubeClearData('{CUBE}');"])
        before = datetime.now()
        tm1.cells.write_async(CUBE, cells, max_workers=4, slice_size=250_000)
        elapsed_time = datetime.now() - before
        print(f"Write_async: {elapsed_time}")

        tm1.processes.execute_ti_code([f"CubeClearData('{CUBE}');"])
        before = datetime.now()
        tm1.cells.write(CUBE, cells, use_blob=True)
        elapsed_time = datetime.now() - before
        print(f"Write use_blob: {elapsed_time}")

[rclapp]

With such a big improvement can you push this to PyPi sometime soon?

[ajain86]

I am intrigued by seeing such a big difference in use_ti vs use_blob. In my experience, the creation of the previous unbound TI with the CellPutN statements, did not add a significant processing time. Any insight on what might be creating this improvement.

[MariusWirtz]

With such a big improvement can you push this to PyPi sometime soon?

Yes. We should do this short term.

[MariusWirtz]

I am intrigued by seeing such a big difference in use_ti vs use_blob. In my experience, the creation of the previous unbound TI with the CellPutN statements, did not add a significant processing time. Any insight on what might be creating this improvement.

Here is my hypothesis:
The TM1 server does some kind of "scanning" when a process is created. That operation is expensive in terms of performance and the cost scales linear with the number of lines in the process.
#774 (comment)

In the old approach, we would create a process with 1 million lines to write 1 million cells. Now we always create a process with 15 lines or so, even when we write 1 million cells.

[MariusWirtz]

Any more feedback is welcome on this one!
Since we release it to PyPI soon we should get the namings and arguments and design decisions right short term.

Once it's out on PyPI we can't really change it much anymore without breaking compatibility

[rclapp]

That would just use the contents endpoint, right? Similar to how arc let's you upload a source file.

Sent from my mobile device On Mar 29, 2023 6:18 PM, Marius Wirtz ***@***.***> wrote: @MariusWirtz commented on this pull request.

________________________________ In TM1py/Services/CellService.py<#882 (comment)>:

+ csv_content = StringIO()

+ csv_writer = csv.writer( + csv_content, + delimiter=",", + quoting=csv.QUOTE_ALL) + csv_writer.writerows( + list(elements) + [value.replace('\r', '').replace('\n', '') if isinstance(value, str) else value] + for elements, value + in cellset_as_dict.items()) + + # Define the path and file name + filename = f'{unique_hash}.csv' + + # Upload CSV to TM1 server using ApplicationService: + # Create a folder TM1py + tm1py_folder = FolderApplication(path='', name=folder_name) I think you have a point there @gbryant-dev<https://github.com/gbryant-dev>. We should change it to use blobs and ignore the concepts of TM1 applications. — Reply to this email directly, view it on GitHub<#882 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AEK7GZSPYJP4ERSYN56SCU3W6SYKPANCNFSM6AAAAAAWJUQTKI>. You are receiving this because you commented.Message ID: ***@***.***>

[macsir]

How exciting with such a big performance improvement! Can't wait for testing it in the cloud environment. 👍

[gbryant-dev]

That would just use the contents endpoint, right? Similar to how arc let's you upload a source file.

That's correct, a document can be uploaded to /Contents('Blobs') instead of /Contents('Applications').

[MariusWirtz]

That would just use the contents endpoint, right? Similar to how arc let's you upload a source file.

That's correct, a document can be uploaded to /Contents('Blobs') instead of /Contents('Applications').

I'm on it now :)

I will introduce a new service class FileService to create, read, update, delete blobs as we can do in Arc.

[gbryant-dev]

Looks good! Naming isn't my one of my strengths so feel free to ignore the suggestion! 🙂

[MariusWirtz]

Looks good! Naming isn't my one of my strengths so feel free to ignore the suggestion! 🙂

Good catch. Thanks!

I think we might be able to use the same approach to speed up the execute_mdx_csv and execute_mdx_dataframe functions.
I'm not expecting anything near 10x performance but if we get even 10% or 20% performance improvement that would add a lot of value IMO.

Does anyone want to take a shot at #884?
with #884 and #848 done we can release v 1.11 next week.