Document Stream Format

[rationalcoder]

I'm having an issue with a serialized block size having a zero in it, despite there actually being a block there. I'm having a really hard time debugging it without fully understanding the contents of a stream.

[cas4ey]

Here is description of the latest file format

[rationalcoder]

@cas4ey Nice! That is better than I was thinking it would be. Some of the ends of the titles are being cut off, btw, but that isn't a big issue.

I'm probably going to write some code that can write theses files as text for debugging purposes. Also, what program did you use to make the diagrams?

[cas4ey]

I have used draw.io and it has cut some ends of the labels when converted diagram to PNG image

[rationalcoder]

Is the format specifically little-endian? If not, I think it should be. That might avoid a little bit of confusion.

On a different note, I just got tripped up by the documentation. The file version is documented as major, minor, and a patch number, and they look like separate fields. In my simple little test program, I ended up creating a struct for the header like this:

struct FileVersion
{
    uint8_t major = 0;
    uint8_t minor = 0;
    uint16_t patch = 0;
};

When really it's packed into a uint32_t.
I think It would be nice if you put a (<type>) in the blocks so you can tell what the type is e.g. char[], uint64_t, int64_t, etc.

[cas4ey]

Is the format specifically little-endian? If not, I think it should be.

Good point, thanks.

The file version is documented as major, minor, and a patch number, and they look like separate fields.

We keep version in a single uint32_t. But it could be represented as a struct you created (or a union) if it is really needed.

As for the rest: OK, I will modify the diagram, but until that moment you can find the types here and here.

[rationalcoder]

In the next version of the .prof format, could you make it so that Total blocks number and Descriptors number are next to each other, followed by the memory usage? That would resolve a padding issue and make it to where you can just fread/memcpy right into a file header struct. The way it is now, you can't do that easily.

[cas4ey]

It is better to use packed structs when writing/reading from files and currently all data in .prof file are aligned to 1 byte boundary so I don't really understand what padding are you talking about?
Could you please clarify what you mean?

[rationalcoder]

@cas4ey Why is better to use packed structs? That is what I was talking about. If you make a structure exactly like the documentation, you get padding between the uint32_ts and the uint64_t, making it to where you can't copy directly into it without using a non-portable packing attribute. If is like that for a reason, it's okay. I was just pointing out that re-arranging to have both the sizes next to each other gets rid of that issue and makes it to where further element access in such a header struct is not an unaligned access. I realize that you don't fread data into structs in your code, that is just what I am doing in my debugging code (a full .prof parser).

Do you do many unaligned accesses in the code for this kind of thing. My ARM platform is ARMv5 and that might be part of the issue (which I am still having, btw. It would seem that you misread my last email :) ).

[cas4ey]

Doesn't struct aligned to 1 byte boundary more portable than default aligned struct (while we are profiling on one CPU type and viewing results on another one)?

Do you do many unaligned accesses in the code for this kind of thing

There are a lot of accesses from gui, but not from the core.

and that might be part of the issue (which I am still having, btw. It would seem that you misread my last email :) )

Yes it seems that I have misunderstood your last email. My bad :-)

[rationalcoder]

@cas4ey It wouldn't make a difference in portability of the format, as it would still technically be aligned to a 1 byte boundary. It's just that it would make reading things like the header faster and easier. Plus, reading into a packed struct could actually make your code less portable on platforms that don't allow unaligned access.

No matter the platform(as far as I know), a FileHeader struct like this:

struct FileVersion
{
    uint16_t patch = 0;
    uint8_t minor = 0;
    uint8_t major = 0;
};

struct FileHeader
{
    uint32_t signature = 0;
    FileVersion version;
    uint64_t profiledProcessId = 0;
    int64_t cpuFrequencyRatio = 0;
    uint64_t beginTime = 0;
    uint64_t endTime = 0;
    uint32_t numBlocks = 0;
    uint32_t numDescriptors = 0;
    uint64_t blocksMemoryUsage = 0;
    uint64_t descriptorsMemoryUsage = 0;
};

Won't have any padding in it, so I can just fread into it, where as a struct like this(The current format):

struct FileHeader
{
    uint32_t signature = 0;
    FileVersion version;
    uint64_t profiledProcessId = 0;
    int64_t cpuFrequencyRatio = 0;
    uint64_t beginTime = 0;
    uint64_t endTime = 0;
    uint32_t numBlocks = 0;
    // char pad1[4];
    uint64_t blocksMemoryUsage = 0;
    uint32_t numDescriptors = 0;
    // char pad2[4];
    uint64_t descriptorsMemoryUsage = 0;
};

Forces you to read some fields individually. It's not really that big of a deal in the long run, it's just annoying for what I am trying to do. However, if you read into a packed struct and use it, you could run into problems on platforms that forbid unaligned access. In the best case, it will still slow things down.

In other news, I am getting close to the issue on my platform. Block names are sometimes being corrupted and blocks that are there are getting a zero size in them. I'm writing a little third-party .prof parser for this kind of debugging. Once I narrow it down, I will let you know.

[rokups]

@cas4ey for what its worth i think @rationalcoder is right. It costs nothing to move struct members around. If it makes working with these structs easier then its a free win.

[cas4ey]

I don't mind against rearranging a couple of elements. I just always thought that padding is system dependent (mainly on CPU) and the main goal of packing data to 1 byte boundary is portability of this data between different systems. That is why I'm posting a lot or comments here :-)

[rationalcoder]

@rokups I would just like to make sure it's clear that the structs I have shown don't exist in easy_profiler, only in my third-party program that I am using to debug some issues. What I am asking is for the next version of the .prof format to move the Total descriptors number to be right after "Total blocks number" (and potentially anything else that is similar). That way, you can make a struct like I did with the members in order and just fread into them. That kind of change simplifies reading and writing code and could offer a small speed boost.

@cas4ey My last point was that, no matter the alignment of uint64_t on different platforms, having the members in order isn't going to produce any padding, so you don't need to pack it. Also, I have just realized that unaligned access in C and C++ don't cause any behavioral problems. However, on my ARMv5 arch. and others, unaligned access has to be done with extra instructions. Even on later versions with hardware support for unaligned access, it is still faster to do naturally aligned access. Rearanging fields in the .prof format doesn't affect the 1 byte alignment of the format, parts of it would just happen to be able to be read into a naturally aligned struct, making some code simpler and further access faster.

Honestly, it's not that big of a deal, though, as long as you aren't using using packed structs in your project. The end result is saving a few fread calls.

[cas4ey]

I have updated a picture (underlying types has been added).

However, on my ARMv5 arch. and others, unaligned access has to be done with extra instructions. Even on later versions with hardware support for unaligned access, it is still faster to do naturally aligned access.

Well, as I know this is true also for desktop CPUs, not only for ARMs :-)

Honestly, it's not that big of a deal, though, as long as you aren't using using packed structs in your project. The end result is saving a few fread calls.

Then it would be better to postpone these changes :-) I'll rearrange file-header members, but other structures may cause serious troubles.

[cas4ey]

File header has been rearranged

struct EasyFileHeader
{
    uint32_t signature = 0;
    uint32_t version = 0;
    processid_t pid = 0; // uint64_t
    int64_t cpu_frequency = 0;
    profiler::timestamp_t begin_time = 0; // uint64_t
    profiler::timestamp_t end_time = 0; // uint64_t
    uint64_t memory_size = 0;
    uint64_t descriptors_memory_size = 0;
    uint32_t total_blocks_number = 0;
    uint32_t total_descriptors_number = 0;
};

[cas4ey]

Latest file format