Added QueryWorkingSetEx to PSAPI

[Crain-32]

Added QueryWorkingSetEx to PSAPI

[Crain-32]

We'll call my failed Builds and commits an fair trade for Author Amendments (And I say this because I feel bad about spamming your email), and not because my IDE wouldn't lint my code due to poor project configuration on my part.

[dbwiddis]

Thanks for continuing to work this. A few more comments.

[Crain-32]

@dbwiddis I believe that wraps up the changes.
When should I expect to see this released?

[dbwiddis]

@Crain-32 Thanks, I'm on vacation but did a quick glance and it looks good. Could you also add an entry to the CHANGES.MD file similar to others where a new method was added?

We don't have a regular release cycle, typically it's done when there have been substantive changes, so I wouldn't expect it for a couple of months. You can add this mapping to your own program temporarily while awaiting the next version, though.

[dbwiddis]

In addition to the change log, a few more comments.

[dbwiddis]

pv should be a Pointer as the user is passing an array.

Also to simplify mappings for users, use int for cb rather than DWORD.

[dbwiddis]

This should be an array of ULONG_PTR.

[dbwiddis]

This isn't right, the bitmask is on the higher bits per lines 417-419, but you are shifting to the lowest bits before masking. You need to either:

• shift first and then mask the lowest "length" bits, or
• mask the specific bits and then shift. Using the >>> operator will save you the need to do special treatment

[Crain-32]

Okay, I think this next iteration will be correct.

private int getBitFieldValue(final int maskLength, final int rightShiftAmount) {
    long bitMask = 0;
                       // Having this Conditional be <= was going give the mask an extra bit.
    for (int l = 0; l < maskLength; l++) {
        bitMask |= 1 << l;
    }
    return (int) ((innerValue >>> rightShiftAmount) & bitMask);
}

Now if maskLength = 4 and rightShiftAmount = 3

innerValue = 101100111000 // Assume Leading Ones so the Unsigned Shift Prevents Sign Flipping
// Construct Mask
1111
// Shift
101100111000 >> 3 = 101100111
//Apply Mask
101100111 &
000001111 
------------
000000111

[dbwiddis]

Yes, I agree if you are shifting first before applying the bitmask, then you are using a 0-indexed array and < is appropriate. Ultimately you probably should have tested this (and I will when you commit) with some debug output to convince yourself it works.

[dbwiddis]

What's "It"? Let's make the messages stand alone. And this should be an assertNotEqual().

[dbwiddis]

Looking good, getting closer. Haven't seen the commit for the changes that you thumbsed-up. In your next commit, we tend to like the change logs to reference the PR rather than the issue, and to include the full package path (with the prefixes abbreviated c.s.j....

[matthiasblaesing]

I have one comment: I'm surprised over the getters for the attributes of PSAPI_WORKING_SET_EX_BLOCK. I would expect them following java bean convention, i.e. isValid(), getShareCount(), getWin32Protection() and so on. Any reason not to?

[Crain-32]

Sorry for not pushing my changes yet. I didn't get it as aligned as I wanted to, so I didn't push yet. Hoping to do that tonight.

@matthiasblaesing It doesn't follow Java Bean Conventions, but from my perspective these would ideally be public properties of the class, and they aren't because honestly I didn't feel like putting in the extra work in the read() overwrite, and because I felt like it would remove the clarity of them being bitfield values. (Albeit once Compiled that clarity is lost anyways :/) I've also been doing a lot of Java 17 as of late, and got used to the records from JEP 395, which I've taken to use a lot more Data-Specific classes.

[dbwiddis]

LGTM, minor nit with = rather than - in the changelog.

I'll be away from internet for a few days so I'll let @matthiasblaesing decide about get*/set* or the record-style getters and whether they should be lowercased. I'm happy either way.

[matthiasblaesing]

@matthiasblaesing It doesn't follow Java Bean Conventions, but from my perspective these would ideally be public properties of the class, and they aren't because honestly I didn't feel like putting in the extra work in the read() overwrite, and because I felt like it would remove the clarity of them being bitfield values. (Albeit once Compiled that clarity is lost anyways :/) I've also been doing a lot of Java 17 as of late, and got used to the records from JEP 395, which I've taken to use a lot more Data-Specific classes.

JNA was created way before 17, compiles on 8 and is theoretically compatible with 6. Even is records came into consideration, they don't match, most of the JNA binds are mutable and it already a liability, that the public fields are used as carrier for the native values. That prevents moving from read-eagerly (when transitioning between native and java) to read-when-needed. Not sure if the original authors of JNA agree, but looking back, I would not map over public field and keep this implementation detail hidden.

What is more many other bindings exists in the jna-platform, that already follow java bean convention at least loosely, so please switch these.

[Crain-32]

With respect, if you wish to lint the Function names using a convention not mentioned in the Contribution Doc, feel free to create a new PR that swaps them to your preferred convention.

The PR's implementation works fine, and as the only person who has ever mentioned needing QueryWorkingSetEx in the Github Issues, I don't feel a need to change them.

[matthiasblaesing]

@Crain-32 it took you most probably longer to write a reply than simply adjusting the names according to normal java convention. For me it is a waste of time to discuss it and will fix the code afterwards.

[Crain-32]

As no other classes in Psapi had private methods, I moved them all out to PsapiUtil, and made sure to include the typing. This aligned more with how the rest of the surrounding code was.

[matthiasblaesing]

At the risk again stepping on toes. I don't see the value of moving the functions out, the original location looked good. They are tightly bound to the `object and only map the documented semantics of the structure. See for example these:

jna/contrib/platform/src/com/sun/jna/platform/win32/Ddeml.java

Lines 435 to 442 in 999c60a

	public String getStr() {
	int offset = fieldOffset("str");
	if(W32APITypeMapper.DEFAULT == W32APITypeMapper.UNICODE) {
	return getPointer().getWideString(offset);
	} else {
	return getPointer().getString(offset);
	}
	}

jna/contrib/platform/src/com/sun/jna/platform/win32/Sspi.java

Lines 626 to 633 in 999c60a

	/**
	* Get buffer bytes.
	* @return
	* Raw buffer bytes.
	*/
	public byte[] getBytes() {
	return pvBuffer == null ? null : pvBuffer.getByteArray(0, cbBuffer);
	}

jna/contrib/platform/src/com/sun/jna/platform/win32/OaIdl.java

Lines 246 to 280 in 999c60a

	public Date getAsJavaDate() {
	long days = (((long) this.date) * MICRO_SECONDS_PER_DAY) + DATE_OFFSET;
	double timePart = 24 * Math.abs(this.date - ((long) this.date));
	int hours = (int) timePart;
	timePart = 60 * (timePart - ((int) timePart));
	int minutes = (int) timePart;
	timePart = 60 * (timePart - ((int) timePart));
	int seconds = (int) timePart;
	timePart = 1000 * (timePart - ((int) timePart));
	int milliseconds = (int) timePart;
	Date baseDate = new Date(days);
	baseDate.setHours(hours);
	baseDate.setMinutes(minutes);
	baseDate.setSeconds(seconds);
	baseDate.setTime(baseDate.getTime() + milliseconds);
	return baseDate;
	}
	public void setFromJavaDate(Date javaDate) {
	double msSinceOrigin = javaDate.getTime() - DATE_OFFSET;
	double daysAsFract = msSinceOrigin / MICRO_SECONDS_PER_DAY;
	Date dayDate = new Date(javaDate.getTime());
	dayDate.setHours(0);
	dayDate.setMinutes(0);
	dayDate.setSeconds(0);
	dayDate.setTime(dayDate.getTime() / 1000 * 1000); // Clear milliseconds
	double integralPart = Math.floor(daysAsFract);
	double fractionalPart = Math.signum(daysAsFract) * ((javaDate.getTime() - dayDate.getTime()) / (24d * 60 * 60 * 1000));
	this.date = integralPart + fractionalPart;
	}
	}

jna/contrib/platform/src/com/sun/jna/platform/win32/WinNT.java

Lines 373 to 380 in 999c60a

	public byte[] getBytes() {
	int len = Advapi32.INSTANCE.GetLengthSid(this);
	return getPointer().getByteArray(0, len);
	}
	public String getSidString() {
	return Advapi32Util.convertSidToStringSid(this);
	}

jna/contrib/platform/src/com/sun/jna/platform/win32/WinCrypt.java

Lines 354 to 370 in 999c60a

	public String[] getRgpszUsageIdentier() {
	if (cUsageIdentifier == 0) {
	return new String[0];
	} else {
	return rgpszUsageIdentifier.getStringArray(0, cUsageIdentifier);
	}
	}
	public void setRgpszUsageIdentier(String[] array) {
	if (array == null || array.length == 0) {
	cUsageIdentifier = 0;
	rgpszUsageIdentifier = null;
	} else {
	cUsageIdentifier = array.length;
	rgpszUsageIdentifier = new StringArray(array);
	}
	}

[Crain-32]

As the build issues are resolved, I'll write my final comments I have on this.

I don't see the value of moving the functions out, the original location looked good

I thought the original location was good as well. However as per the Contribution Guideline changes today, I followed "the surrounding code" and removed all class functions. I personally don't see the value of Jean Bean Naming on Data Classes.

Even is records came into consideration, they don't match

If you mean the function names didn't match what record would make, they did.

public record foo(Integer bar) {
   public Integer retrieveBar() {
      return this.bar(); // this is valid, and the expected function call to get the value.
   }
}

it took you most probably longer to write a reply than simply adjusting the names according to normal java convention.

Oh it did! Being asked to change perfectly fine function names to an unstated convention on a PR with a working Build is rather annoying. When people get annoyed they tend to do what you say instead of what you want. You said to follow conventions and the surrounding code, and I did that to the fullest I could.

(From the Contribution PR Today)

we can't expect people to follow simple requests if not spelled explicitly.

As I mentioned in the second comment of the PR my IDE wasn't indexing anything for me. Your "simple request" is not right click and refactor the function for me, it's instead jumping between different files and doing Commits/Pushes to use your Pipeline to catch symbol errors for me.

If my behavior comes off as unprofessional, please understand that my desire was to provide this function in JNA for when others need it, which from my perspective I had already achieved. The team has done a fantastic Job of making JNA expandable from outside the project, which I will keep in mind the next time I need something.

With a working PR, I have no intentions of continuing to Iterate. If there are changes you'd like to see before merging, feel free to create a PR on my Fork.

[matthiasblaesing]

@dbwiddis please wait. The definition is not correct - I'm currently trying to create a real unittest and see if my modified bindings work.

[dbwiddis]

Hey @Crain-32, first off, thank you for your contribution so far. Even if you chose not to complete the PR, the work you did is helpful and your contributions are appreciated. As the number of active maintainers is small and we do this all on our volunteer time (and I'm popping in briefly from vacation during some down time) it's always helpful when the community can do more work.

Oh it did! Being asked to change perfectly fine function names to an unstated convention on a PR with a working Build is rather annoying.

JNA is a community-maintained project, and I myself have been annoyed at the lack of overall convention requirements (beyond some checkstyle tests). This is where the convention becomes "try to match how it's been done in the past as much as possible." Unstated, but at least somewhat consistent. Multiple other places in JNA when there are convenience methods for members of a class they use getters. Even in the bitfield example you cited in your research, there were (a lot more) getters.

There are some benefits to the Java Bean notation. For example, Jackson's ObjectMapper can automatically assume field names with them, for one, which is very helpful when serializing/deserializing in a REST model. (Yes it can be worked around, but the "easy" assumption is generally better to follow.)

Records are fun, but some of the newer and shinier stuff is going to be in JDK 21 with a whole new foreign function and memory interface that will probably be preferred to JNA for new projects, while JNA continues to support the old JDK 6+ world.

Anyway, I pushed a commit going back to how I think was discussed. Leaving this PR for further review/discussion.

[matthiasblaesing]

@Crain-32 @dbwiddis: Based on an older commit from this branch I build a new PR, which from my perspective fixes issues found here. This started as a "just change the naming of the getters" change. But then I had a look at the data structure definitions again and found, that PSAPI_WORKING_SET_EX_BLOCK was not correct. It was bound as a structure consisting of multiple ULONG_PTRs. But this is not the case, it is just one ULONG_PTR with the bitfield structure used as an accessor.

The change that drops PSAPI_WORKING_SET_EX_BLOCK and changes the names of the getters can be found here:

dbf8751

Considering how to test this, I realized, that we need a way to change a single attribute from the memory properties. The easiest seemed to be the virtual memory lock. The corresponding function were not yet bound, which was fixed by:

04ae48f

In the test cases a big problem was, that the structures were not read after the native call, so they basicly were no-ops. In

45647d5

both the positive and the negative case are checked and by locking the memory location and doing a second read it ensured, that the read is not some random value.

[dbwiddis]

It was bound as a structure consisting of multiple ULONG_PTRs. But this is not the case, it is just one ULONG_PTR with the bitfield structure used as an accessor.

But the documentation showed a union that clearly indicated ULONG_PTR covering a 64-bit field. WIth the variable size structure it's effectively the same thing for a single one, but with the windows docs mapping would consist of two. Although digging further it does indicate that second 32-bit field isn't there on 32-bit systems.

More strange is that Flags is listed differently than the nameless union. So is the _INFORMATION structure 2 * POINTER_SIZE or 3 * POINTER_SIZE?

[matthiasblaesing]

It was bound as a structure consisting of multiple ULONG_PTRs. But this is not the case, it is just one ULONG_PTR with the bitfield structure used as an accessor.

But the documentation showed a union that clearly indicated ULONG_PTR covering a 64-bit field. WIth the variable size structure it's effectively the same thing for a single one, but with the windows docs mapping would consist of two. Although digging further it does indicate that second 32-bit field isn't there on 32-bit systems.

More strange is that Flags is listed differently than the nameless union. So is the _INFORMATION structure 2 * POINTER_SIZE or 3 * POINTER_SIZE?

This is the structure from the windows sdk headers:

typedef union _PSAPI_WORKING_SET_EX_BLOCK {
    ULONG_PTR Flags;
    union {
        struct {
            ULONG_PTR Valid : 1;
            ULONG_PTR ShareCount : 3;
            ULONG_PTR Win32Protection : 11;
            ULONG_PTR Shared : 1;
            ULONG_PTR Node : 6;
            ULONG_PTR Locked : 1;
            ULONG_PTR LargePage : 1;
            ULONG_PTR Reserved : 7;
            ULONG_PTR Bad : 1;

#if defined(_WIN64)
            ULONG_PTR ReservedUlong : 32;
#endif
        };
        struct {
            ULONG_PTR Valid : 1;            // Valid = 0 in this format.
            ULONG_PTR Reserved0 : 14;
            ULONG_PTR Shared : 1;
            ULONG_PTR Reserved1 : 15;
            ULONG_PTR Bad : 1;

#if defined(_WIN64)
            ULONG_PTR ReservedUlong : 32;
#endif
        } Invalid;
    };
} PSAPI_WORKING_SET_EX_BLOCK, *PPSAPI_WORKING_SET_EX_BLOCK;

What I see is this:

1. _PSAPI_WORKING_SET_EX_BLOCK is a union over two members, one named Flags and one is an anonymous union.
2. the inner anonymous union consists of two structure definitions, where the first structure is anonymous and the second one is named Invalid
3. both structures are bitfield definitions
4. the bitfields are based on ULONG_PTR as carrier
5. both structures are cover 32bits on 32bit platform and 64bit on 64bit platforms (the final ReservedUlong in both structures is only present on 64bit)
6. ULONG_PTR is 32bit wide on 32bit platforms and 64bit wide on 64bit platforms
7. From 5+6: The structures exactly match the size of their carrier type (regardless of platform bitness). I don't see padding in this case.
8. From 7: The anonymous union over the two structures has a size of 32bit on 32bit platforms and 64bit on 64bit platforms
9. From 2 + 6 + 8: The two definitions for the _PSAPI_WORKING_SET_EX_BLOCK both have the same size: 32bit on 32bit platforms and 64bit on 64 bit platforms.
10. From 9: In _PSAPI_WORKING_SET_EX_INFORMATION are two pointer sized blocks, where the first encodes the memory location (VirtualAddress) and the second the flags (Flags).

Does that make sense?

[dbwiddis]

I completely missed that _BLOCK was a union, and the headers I found didn’t include the #if defined(_WIN64) part. It’s clear now.

[dbwiddis]

Thanks for the contribution. Closing this PR as the change commits will be included in #1459.