Argument lifetime issue when calling COM/ATL methods with array of strings (and probably others) as of at least Python 3.8.3 (which worked with Python 3.7.x)

[whannes]

Consider the following, very simple COM/ATL project made with MSVC 2017 (15.9.24, Windows 10 Pro 2004): comProjectWithArray-v2.zip. It does not contain any functionality and is used to reproduce the empty array as described below only.

Registration is disabled in the project settings here, so to test it you have to register
ATLTestServer.exe /RegServer
regsvr32 ATLTestServerPS.dll

After successful registration, it can be tested with the following Python-Snippet:
from comtypes.client import CreateObject
obj = CreateObject("ATLTestServer.ATLTestObject")
obj.concatenateStrings(["New", " ", "York"])

When using comtypes 1.1.7 (6fb4403) with Python 3.7.7 x64 the call from the Python snipped gets properly routed to the C++ method:

However, when using the same comtypes version with Python 3.8.3 x64, the array in C++ contains only nullptr:

Is this a known issue?

[whannes]

Update: It seems it is not related to the cache files generated. Although they are not exactly the same on my machine with regard to the order of the definitions, using the ones created with Python 3.7 in Python 3.8 and vice versa does not changes the behavior: When calling the server with Python 3.8, the array of variants is empty while with Python 3.7 everything is fine.

I also updated the test project (implemented concatenate). Now this issue seems to be only partially reproducible, this is really strange:
Consider this very simple script without a breakpoint before calling the method:

As mentioned before, the vector of variants is actually not arriving at the com server and thus the returned string is empty.

However, if a run the same snipped with a breakpoint before calling this method and then continue, it correctly forwards the array:

So there is definitely a lifetime issue around the caller arguments. When defining them externally, the arguments are always passed correctly to the COM server also with Python 3.8:

[karpierz]

Hi All
Any news?. IMHO it is very important bug/thing which in practice makes comtypes impossible to use in Py >=3.8
PS: I'v observed the same behaviour/issue using comtypes with Excell for Py >=3.8.
Thanks,
Adam

[vasily-v-ryabov]

@karpierz I see the importance of the issue. I will try to find time to work on it in June. Since pywinauto is not affected, it's P2 for me. Thanks for understanding.

[karpierz]

Vasilij
I tried to investigate this issue in sources but.. without any succes/seems for me too difficult (I am not familiar with comtypes source for now;).
Of course I can help you, but with your direction. Do not hesitate to use me :)
PS: Yes, it is important issue. It is blocker for use Python >=3.8 in our production env.
Adam

[vasily-v-ryabov]

I'm trying to make this example working, but variable a is always None for me in clean Python 3.7.3 (started from cmd.exe) even with explicit tagVARIANT creation. But I built the C++ code using VS 2015. And I'm using comtypes-1.1.9. No ideas yet what is wrong on my side. Maybe running under debugger initializes something important.

[irfanilgin]

I have a similar problem. In my case the last string argument provided to python function is replacing other string arguments coming before it in the underlying C/C++ code. Somehow comtypes is doing this when running with >python 3.8. I have the following solution that can highlight the underlying issue. I provided string arguments inside a VARIANT object, which helped me.

IMHO this issue is very important and if @vasily-v-ryabov can provide some guidance I am happy to help.

Irfan

[vasily-v-ryabov]

@whannes what do you expect in returned value when the attached implementation is empty?

STDMETHODIMP CATLTestObject::concatenateStrings(VARIANT arrayOfStrings, BSTR* concatenatedStrings)
{
	// TODO: Add your implementation code here

	return S_OK;
}

I mean what do you expect while running your example without debugger? I suspect you have more detailed implementation to reproduce the problem but you forgot to attach the updated code. Or I missed something.

All I need is a reproducing code (without debugger). Can someone prepare it?

[vasily-v-ryabov]

@karpierz @irfanilgin if you have your own reproducing example, please attach it ASAP. I have free time on Monday probably. And today which seems already missing for this bug.

I can quickly compile C++ code with VS2015 (hope it's OK). I'd like first to run it without VS debugger and without Python debugger (python.exe -d). Python 3.8+ and Python 3.7 should show me the difference. I hope both Pythons are 64-bit (it sounds consistent at least).

[karpierz]

from comtypes.client import CreateObject
xl = CreateObject("Excel.Application")
from comtypes.gen.Excel import xlRangeValueDefault
xl.Workbooks.Add()
xl.Range["A1", "C1"].Value[xlRangeValueDefault] = (10, "20", 31.4)
print(xl.Range["A1", "C1"].Value[xlRangeValueDefault])
xl.Visible = True
input("Press a key")
xl.Quit()

Please compare

D:>py -3.7 z_COM.py
((10.0, 20.0, 31.4),)
Press a key

D:>py -3.8 z_COM.py
10.0
Press a key

[whannes]

@whannes what do you expect in returned value when the attached implementation is empty?

STDMETHODIMP CATLTestObject::concatenateStrings(VARIANT arrayOfStrings, BSTR* concatenatedStrings)
{
	// TODO: Add your implementation code here

	return S_OK;
}

I mean what do you expect while running your example without debugger? I suspect you have more detailed implementation to reproduce the problem but you forgot to attach the updated code. Or I missed something.

All I need is a reproducing code (without debugger). Can someone prepare it?

Hi all, I can‘t do it within the next week as I have no machine with me right now. If any of you can provide some implementation code earlier, please go ahead.

[karpierz]

Hi All
Any news/update ?
Thanks,
Adam

[karpierz]

Hi All
Any news/update ?
Thanks,
Adam

[junkmd]

@karpierz
I am restoring some tests in comtypes.

I am trying to restore test_excel.py(currently skipped because it depends on specific environment, see #298 (comment)), and I was trying to use unittest.skipIf to allow both testing in non-Excel-installed-env by AppVeyor and testing in Excel-installed-env by the developer.

I noticed that it fails in Python 3.8 or later as you reported.

However, CreateObject(... dynamic=True)(test_latebind), the test succeeded even with 3.8 or later.

I wonder why it occurred.

I have been always use cell ranges as ("A1:C1")(with-colon style) since Python 3.7 before this bug occurred, so I did not face this problem.

But I think it's large problem that different behaviors in different versions.

If this can be fix by modifying the comtypes code, we would like to do so, otherwise we think we need to clarify the writing style supported by different versions for those who use comtypes.

[bennyrowland]

Just run into this problem myself trying to test a new comtypes based server. I have put together a fairly compact minimum example, the .idl file looks like this:

import "oaidl.idl";
import "ocidl.idl";

[
    uuid(D690C6D9-3C64-486D-9A5A-D7B70987514B),
    dual,
    oleautomation
]
interface ITest : IDispatch {
    HRESULT ReceiveTuple([in] VARIANT tup, [out, retval] INT *retval);
}

[
    uuid(2563935A-9C03-40D5-9068-FB58C9828F1D)
]
library COMTest
{
    importlib("stdole2.tlb");

    [uuid(F979C04E-6346-44E6-A529-2E13A645EF20)]
        coclass Application {
        [default] interface ITest;
    };
};

which I compile with the MIDL compiler, then the Python code looks like this

import comtypes.client
import comtypes.server.register

testcom = comtypes.client.GetModule("test.tlb")

class Test(testcom.Application):
    def ReceiveTuple(self, tup):
        print(f"Received {tup}")
        return 0

if __name__ == "__main__":
    comtypes.server.register.register(Test)
    server = comtypes.client.wrap(Test().QueryInterface(comtypes.IUnknown))
    server.ReceiveTuple("test")
    server.ReceiveTuple(["a", "b"])

This needs to be run from an admin console to allow the registration part to happen (at least the first time it is run), and the result I get is:

Received VARIANT(vt=0x8, 'test')
Received VARIANT(vt=0x200c, (None, None))

Process finished with exit code -1073740940 (0xC0000374)

So calling with a single value (string or numeric) is fine, but with a list or tuple we not only get an array of None variants (of the same size as the passed list) but this is followed by a crash due to heap corruption, which may be because memory is being released twice or something.

My current setup is with comtypes 1.1.11 and Python 3.9.7.

I am not that familiar with the comtypes internals, but if @vasily-v-ryabov or another maintainer can give me some pointers about where to look I am happy to dive in to try and solve this issue.

[junkmd]

@bennyrowland
Your test.tlb example may help me solve #303 at the same time.

I am also trying to figure out how to solve this problem, so if I come up with any ideas, I would like to share them.

[bennyrowland]

@junkmd, it would be nice to solve this. For the moment I have been able to solve it in my case by changing the parameters to VARIANT *, in that case the array is passed correctly. Interestingly I have also tried accessing the Test server from VB.NET, this also fails with test.tlb above, but works with VARIANT * argument, I think that VB.NET is only capable of passing arrays by reference, and seems to release the values after it passes them, but for compatibility with both clients, VARIANT * may be the way to go.

[junkmd]

@bennyrowland
I don't know much about c-pointers, so I may be guessing, but I thought a double or more pointer might solve the problem.

In terms of the code generated by comtypes, define POINTER(POINTER(...(VARIANT)...))).

but I would like to hear from other contributors that is a reasonable modification.

[junkmd]

I noticed.

comtypes/comtypes/test/test_excel.py

Lines 49 to 63 in db0931d

	# Test with empty-tuple argument
	xl.Range["A1", "C1"].Value[()] = (10,"20",31.4) # XXX: in Python >= 3.8.x, cannot set values to A1:C1
	xl.Range["A2:C2"].Value[()] = ('x','y','z')
	# Test with empty slice argument
	xl.Range["A3:C3"].Value[:] = ('3','2','1')
	# not implemented:
	# xl.Range["A4:C4"].Value = ("3", "2" ,"1")
	# call property to retrieve value
	expected_values = ((10.0, 20.0, 31.4),
	("x", "y", "z"),
	(3.0, 2.0, 1.0))
	# XXX: in Python >= 3.8.x, fails below
	self.assertEqual(xl.Range["A1:C3"].Value(),
	expected_values)

Add the following assertion to this test.

self.assertEqual(xl.Range("A1", "C1").Address(), "$A$1:$C$1")

This line passes for dynamic=True and does not pass for dynamic=False.

FAIL: test (comtypes.test.test_excel.Test_EarlyBind.test)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\komoj\comtypes\comtypes\test\test_excel.py", line 52, in test
    self.assertEqual(xl.Range("A1", "C1").Address(), "$A$1:$C$1")
AssertionError: '$C$1' != '$A$1:$C$1'
- $C$1
+ $A$1:$C$1

[junkmd]

diffs between 3.7.7 and 3.8.3

python/cpython@v3.7.7...v3.8.3#diff-402bc51df93eaaca2e2f2551ec0206d8af190722ae4703c178554b565d7c14b4

[junkmd]

I am investigating the issue on VARIANT until there is progress on the PR on type annotation.

[junkmd]

There seems to be some difference in the calling methods of automation.IDispatch and client.lazybind.Dispatch.

After #378 is merge, I'm thinking of adding a type hint to client.lazybind to analyze its behavior while I have a handful of work related to #327.

[junkmd]

It was reported in python/cpython#82929 that the callbacks were behaving strangely starting with Python 3.8.

I was going to look for a workaround on comtypes side.
However, this issue may be a regression of cpython.

Would I call out to the cpython core developers?

[bennyrowland]

@junkmd it seems quite plausible to me that the issue is indeed not with com types but with ctypes or something else further up the chain. Indeed it is quite hard to imagine how com types itself could cause the problem without any C code. However, the cpython issue doesn't seem to have seen much progress in a while so who knows if/when it will get resolved.

[junkmd]

@michaelDCurran

I saw you in python/cpython#82929.
Any feedback on this issue?

[michaelDCurran]

I don't believe it is directly related to the libffi bug in python/cpython#82929, but I can't rule out that it is a separate libffi bug.
We have not experienced this comtypes bug in our particular project to date, though it is worth noting that our project is 32 bit not 64 bit.

[junkmd]

@michaelDCurran

Thank you for your response.

I have not encountered this bug either, but I am concerned about the regression in how Excel cells are specified as reported by @karpierz.

If there is no movement on python/cpython#82929 in the next week I will open another issue on cpython to resolve this.

[michaelDCurran]

For reference python/cpython#82929 has already been fixed in Python main. See python/cpython#97512 and in the Python 3.11 branch, see python/cpython#97513. However this missed out on getting into Python 3.11.0. I'm not sure why python/cpython#82929 has not been closed as fixed yet. I can't do it as I was not the original reporter. So it is worth testing comtypes with Python main to see if this particular bug still exists.

[karpierz]

@michaelDCurran

For reference python/cpython#82929 has already been fixed in Python main.

Unfortunatelly the bug still exist in the newest (6.12.2022) versions of Python (3.10.9, 3.11.1).

For such code:

xl = CreateObject("Excel.Application")
from comtypes.gen.Excel import xlRangeValueDefault
xl.Workbooks.Add()
xl.Range["A1", "C1"].Value[xlRangeValueDefault] = (10, "20", 31.4)
print(xl.Range["A1", "C1"].Value[xlRangeValueDefault])
xl.Visible = True
input("Press a key")
xl.Quit()

we have results:

D:\Py>py -3.7 z_COM.py
((10.0, 20.0, 31.4),)
Press a key

D:\Py>py -3.10 z_COM.py
10.0
Press a key

D:\Py>py -3.11 z_COM.py
10.0
Press a key

PS: This is an important/real barrier to upgrading Python to versions newer than 3.7.

[junkmd]

@karpierz

I posted bug report to cpython, python/cpython#99952.

It would be helpful if you could post any information you have there as well.

[junkmd]

I think python/cpython#100169 by @ynkdir might help this.

I am watching for python/cpython#100169 as it may be possible to resolve this issue on the cpython side.

[junkmd]

Good news.

python/cpython#100169 has been merged.

The changes will be backported to Py3.11 and Py3.10.
python/cpython#101339
python/cpython#101340

I will see that Excel in my environment would work when they are released.

I think that the involved members should also check to see that it works well for your projects.

[junkmd]

Python 3.10.10 and 3.11.2 is now available.
https://www.python.org/downloads/release/python-31010/
https://www.python.org/downloads/release/python-3112/

And there is a mention about python/cpython#99952 in the CHANGELOG.

gh-99952: Fix a reference undercounting issue in ctypes.Structure with from_param() results larger than a C pointer.

https://docs.python.org/release/3.10.10/whatsnew/changelog.html#library
https://docs.python.org/release/3.11.2/whatsnew/changelog.html#library

I installed 3.11.2 and removed the skip decorator in test_excel, then it works in my env!

...\comtypes> python -m unittest comtypes.test.test_excel -v
test (comtypes.test.test_excel.Test_EarlyBind.test) ... ok
test (comtypes.test.test_excel.Test_LateBind.test) ... ok
----------------------------------------------------------------------
Ran 2 tests in 13.654s
OK

I am thinking if it is possible to test this behavior with a non-environment dependent COM library.

Any opinions would be appreciated.

[junkmd]

I am thinking of closing this issue as resolved.

However, this error is very confusing when getting into it (like #494), so I am thinking of a way to make it known to comtypesers.

For example, adding a sentence to README.md recommending that Python>=3.10.10 and Python>=3.11.2 should be used rather than Python3.8 and Python3.9.

• This does not mean dropping 3.8 or 3.9 from support now. Because only a few parts methods and arguments cause errors, and projects that do not use them should have no problems.

Any opinions would be appreciated.

[junkmd]

I have described in the README.md that there are bugs in Python 3.8 and Python 3.9, and they have been fixed in Python>=3.10.10 and Python>=3.11.2.

I think this is sufficient for disseminating the information.