XDK Heap Profiler collector and XDK CPUProfiler

improvement: adding total time to CPU profiler

This commit  is squashed from several commits and correspond for two big
features:
1. XDK heap profiler collector
2. CpuProfiler: Annotation of source lines by total time in CPU Profiler
   this feature is in the process of upstream to chromium:
      https://codereview.chromium.org/1477623003/

-- feature 1:
This heap profiler allows to perform deepper analysis of the JavaScript objects
and is more concentrated on the allocation point of the object.

Here are main deifference between CDT and XDK heap profilers:
1. CDT Heap Profiler care of the alive objects, XDK collects all info even
for released ones.
This allows to show the allocation chart of objects which are used by the app in
each moment.
Neither CDT Timeline nor CDT Heap Profiler don't allow doing this. CDT Timeline
shows the whole pool size which is much more than currently refered objects, CDT
Heap Profiler shows only bars of allocated and alive objects by the end of
collection. The drawback of this approach - we have to call garbage collection
regularly. On the other hand CDT Heap Profiler calls Garbage Collection the same
way.
2. To reduce the amount of information XDK profiler groups the objects to
buckets with unique key including the allocation time, release time, type,
allocation call site.
This allow to select any period on the timeline and analyze which objects are
allocated and not released by the end of the collection. This analysis type is
useful in case of big memory spikes. Just need to select on the chart where this
spike starts till the maximum of the allocation memory.
3. The allocation callsite allows to annotate the sources by the self and total
values of allocated memory. And it's useful to know that certain line calling
library function implicitly allocates and retains megabates of data.
4. Like in CDT, XDK profiler can collect the object's retention information, but
this info is stored for allocation callsite/type. It is done because we don't
show unique objects and we have to show all retentions for group. It's not
such efficient like for unique objects but also has a value.

-- feature 2:
CpuProfiler: Annotation of source lines by total time in CPU Profiler
upstream to chromium: https://codereview.chromium.org/1477623003/

This patch adds source lines annotations by total time values for the CPU profile.
This functionality is required for XDK CPU profiling feature and can be used for
CDT CPU profiling feature after frontend modification.

For source lines annotation by total time was added new StackEntry structure that
substitutes CodeEntry class inside CPU profiling collection. StackEntry structure
includes a pointer to CodeEntry object and source line where certain sample
happened or the callee function was called. This allows recording source line
number for each node in the call tree. Previously, the source line was recorded
only for the sample place, so we were not able to recover source lines of caller,
that prevented calculating total time for source lines.

Thus the parsing stack functions use StackEntry's instances instead of CodeEntry.
To store them in the hash was changed the hash function. It starts using not only
CodeEntry, but also source line. Currently it looks like: CodeEntryHash(old function
for calculating hash) XOR (source_line from StackEntry structure). It is needed for
separation of equal CodeEntries with different source lines. Efficiency of the new
hash function was checked in a separate test and obtained even better distribution
inside HashMap than with the old function.

These changes don't affect current CPU profiling results under CDT. The call tree
stay the same because frontend merges the Nodes in the tree by CodeEntry hash only.
And the source view is not annotated by the lines because it's an open question
how to do this properly for CDT views. CDT frontend should be extended to support
both self-time annotations of the source lines and total times annotation.

Regarding transferring results to the frontend: these changes are transmitted to the
frontend by adding new parameter src_line in ProfileNode token in the JSON based
package.

Were added wrappers for several functions which accept StackEntry by the
functions, which accept old CodeEntry values to support existing tests.
The new test for this functionality was added to 'test-profile-generator'

include/v8-profiler.h

@@ -80,6 +80,11 @@ class V8_EXPORT CpuProfileNode {
    */
   int GetColumnNumber() const;
 
+  /**
+   * Returns source line connected with current ProfileNode
+  */
+  int GetSrcLine() const;
+
   /**
    * Returns the number of the function's source lines that collect the samples.
    */
@@ -173,6 +178,18 @@ class V8_EXPORT CpuProfile {
   void Delete();
 };
 
+/**
+ * HeapEventXDK contains the latest chunk of heap info
+ */
+class V8_EXPORT HeapEventXDK {
+ public:
+  const char* getSymbols();
+  const char* getFrames();
+  const char* getTypes();
+  const char* getChunks();
+  const char* getRetentions();
+  unsigned int getDuration();
+};
 
 /**
  * Interface for controlling CPU profiling. Instance of the
@@ -341,6 +358,18 @@ class V8_EXPORT OutputStream {  // NOLINT
   virtual WriteResult WriteHeapStatsChunk(HeapStatsUpdate* data, int count) {
     return kAbort;
   }
+
+  /**
+   * Writes XDK object
+   */
+  virtual WriteResult WriteHeapXDKChunk(const char* symbols, size_t symbolsSize,
+                                        const char* frames, size_t framesSize,
+                                        const char* types, size_t typesSize,
+                                        const char* chunks, size_t chunksSize,
+                                        const char* retentions,
+                                        size_t retentionSize) {
+    return kAbort;
+  }
 };
 
 
@@ -681,6 +710,14 @@ class V8_EXPORT HeapProfiler {
    */
   void SetRetainedObjectInfo(UniqueId id, RetainedObjectInfo* info);
 
+  void StartTrackingHeapObjectsXDK(int stackDepth, bool retentions,
+                                   bool strict_collection = false);
+  /**
+   * @author amalyshe
+   */
+  void GetHeapXDKStats(OutputStream* stream);
+  HeapEventXDK* StopTrackingHeapObjectsXDK();
+
  private:
   HeapProfiler();
   ~HeapProfiler();

src/api.cc

@@ -49,12 +49,12 @@
 #include "src/profiler/heap-snapshot-generator-inl.h"
 #include "src/profiler/profile-generator-inl.h"
 #include "src/profiler/sampler.h"
-#include "src/property.h"
 #include "src/property-descriptor.h"
 #include "src/property-details.h"
+#include "src/property.h"
 #include "src/prototype.h"
-#include "src/runtime/runtime.h"
 #include "src/runtime-profiler.h"
+#include "src/runtime/runtime.h"
 #include "src/simulator.h"
 #include "src/snapshot/natives.h"
 #include "src/snapshot/snapshot.h"
@@ -65,7 +65,7 @@
 #include "src/v8threads.h"
 #include "src/version.h"
 #include "src/vm-state-inl.h"
-
+#include "src/xdk-allocation.h"
 
 namespace v8 {
 
@@ -8179,6 +8179,10 @@ int CpuProfileNode::GetColumnNumber() const {
       entry()->column_number();
 }
 
+int CpuProfileNode::GetSrcLine() const {
+  const i::ProfileNode* node = reinterpret_cast<const i::ProfileNode*>(this);
+  return node->src_line();
+}
 
 unsigned int CpuProfileNode::GetHitLineCount() const {
   const i::ProfileNode* node = reinterpret_cast<const i::ProfileNode*>(this);
@@ -8420,6 +8424,41 @@ void HeapSnapshot::Delete() {
   }
 }
 
+const char* HeapEventXDK::getSymbols() {
+  const i::HeapEventXDK* eventXDK =
+      reinterpret_cast<const i::HeapEventXDK*>(this);
+  return eventXDK->symbols();
+}
+
+const char* HeapEventXDK::getFrames() {
+  const i::HeapEventXDK* eventXDK =
+      reinterpret_cast<const i::HeapEventXDK*>(this);
+  return eventXDK->frames();
+}
+
+const char* HeapEventXDK::getTypes() {
+  const i::HeapEventXDK* eventXDK =
+      reinterpret_cast<const i::HeapEventXDK*>(this);
+  return eventXDK->types();
+}
+
+const char* HeapEventXDK::getChunks() {
+  const i::HeapEventXDK* eventXDK =
+      reinterpret_cast<const i::HeapEventXDK*>(this);
+  return eventXDK->chunks();
+}
+
+const char* HeapEventXDK::getRetentions() {
+  const i::HeapEventXDK* eventXDK =
+      reinterpret_cast<const i::HeapEventXDK*>(this);
+  return eventXDK->retentions();
+}
+
+unsigned int HeapEventXDK::getDuration() {
+  const i::HeapEventXDK* eventXDK =
+      reinterpret_cast<const i::HeapEventXDK*>(this);
+  return eventXDK->duration();
+}
 
 const HeapGraphNode* HeapSnapshot::GetRoot() const {
   return reinterpret_cast<const HeapGraphNode*>(ToInternal(this)->root());
@@ -8538,6 +8577,20 @@ AllocationProfile* HeapProfiler::GetAllocationProfile() {
   return reinterpret_cast<i::HeapProfiler*>(this)->GetAllocationProfile();
 }
 
+void HeapProfiler::GetHeapXDKStats(OutputStream* stream) {
+  reinterpret_cast<i::HeapProfiler*>(this)->PushHeapObjectsXDKStats(stream);
+}
+
+void HeapProfiler::StartTrackingHeapObjectsXDK(int stackDepth, bool retentions,
+                                               bool strict_collection) {
+  reinterpret_cast<i::HeapProfiler*>(this)->StartHeapObjectsTrackingXDK(
+      stackDepth, retentions, strict_collection);
+}
+
+HeapEventXDK* HeapProfiler::StopTrackingHeapObjectsXDK() {
+  return reinterpret_cast<HeapEventXDK*>(
+      reinterpret_cast<i::HeapProfiler*>(this)->StopHeapObjectsTrackingXDK());
+}
 
 void HeapProfiler::DeleteAllHeapSnapshots() {
   reinterpret_cast<i::HeapProfiler*>(this)->DeleteAllSnapshots();

src/profiler/heap-profiler.cc

@@ -9,6 +9,7 @@
 #include "src/profiler/allocation-tracker.h"
 #include "src/profiler/heap-snapshot-generator-inl.h"
 #include "src/profiler/sampling-heap-profiler.h"
+#include "src/xdk-allocation.h"
 
 namespace v8 {
 namespace internal {
@@ -67,7 +68,7 @@ HeapSnapshot* HeapProfiler::TakeSnapshot(
     v8::HeapProfiler::ObjectNameResolver* resolver) {
   HeapSnapshot* result = new HeapSnapshot(this);
   {
-    HeapSnapshotGenerator generator(result, control, resolver, heap());
+    HeapSnapshotGenerator generator(this, result, control, resolver, heap());
     if (!generator.GenerateSnapshot()) {
       delete result;
       result = NULL;
@@ -137,6 +138,42 @@ void HeapProfiler::StopHeapObjectsTracking() {
   }
 }
 
+void HeapProfiler::StartHeapObjectsTrackingXDK(int stackDepth, bool retentions,
+                                               bool strict_collection) {
+  ids_->UpdateHeapObjectsMap();
+  is_tracking_object_moves_ = true;
+  DCHECK(!is_tracking_allocations());
+  allocation_tracker_xdk_.Reset(
+      new XDKAllocationTracker(this, ids_.get(), names_.get(), stackDepth,
+                               retentions, strict_collection));
+  heap()->DisableInlineAllocation();
+  // init pre collected objects
+  allocation_tracker_xdk_->CollectFreedObjects(false, true);
+}
+
+void HeapProfiler::PushHeapObjectsXDKStats(OutputStream* stream) {
+  // get the garbage here
+  if (!allocation_tracker_xdk_.is_empty()) {
+    allocation_tracker_xdk_->CollectFreedObjects();
+    OutputStream::WriteResult result =
+        allocation_tracker_xdk_->SendChunk(stream);
+    // TODO(amalyshe): it's interesting why CDT can return kAbort. Need to
+    // investigate if we need add better error generation in the
+    // allocation_tracker_xdk_->SendChunk
+    if (result == OutputStream::kAbort) return;
+    stream->EndOfStream();
+  }
+}
+
+v8::internal::HeapEventXDK* HeapProfiler::StopHeapObjectsTrackingXDK() {
+  HeapEventXDK* event = NULL;
+  if (!allocation_tracker_xdk_.is_empty()) {
+    event = allocation_tracker_xdk_->stopTracking();
+    allocation_tracker_xdk_.Reset(NULL);
+    heap()->EnableInlineAllocation();
+  }
+  return event;
+}
 
 size_t HeapProfiler::GetMemorySizeUsedByProfiler() {
   size_t size = sizeof(*this);
@@ -169,9 +206,13 @@ SnapshotObjectId HeapProfiler::GetSnapshotObjectId(Handle<Object> obj) {
 
 void HeapProfiler::ObjectMoveEvent(Address from, Address to, int size) {
   base::LockGuard<base::Mutex> guard(&profiler_mutex_);
-  bool known_object = ids_->MoveObject(from, to, size);
-  if (!known_object && !allocation_tracker_.is_empty()) {
-    allocation_tracker_->address_to_trace()->MoveObject(from, to, size);
+  if (allocation_tracker_xdk_.is_empty()) {
+    bool known_object = ids_->MoveObject(from, to, size);
+    if (!known_object && !allocation_tracker_.is_empty()) {
+      allocation_tracker_->address_to_trace()->MoveObject(from, to, size);
+    }
+  } else {
+    allocation_tracker_xdk_->OnMove(from, to, size);
   }
 }
 
@@ -181,6 +222,9 @@ void HeapProfiler::AllocationEvent(Address addr, int size) {
   if (!allocation_tracker_.is_empty()) {
     allocation_tracker_->AllocationEvent(addr, size);
   }
+  if (!allocation_tracker_xdk_.is_empty()) {
+    allocation_tracker_xdk_->OnAlloc(addr, size);
+  }
 }
 
 

src/profiler/heap-profiler.h

@@ -17,6 +17,8 @@ class AllocationTracker;
 class HeapObjectsMap;
 class HeapSnapshot;
 class SamplingHeapProfiler;
+class HeapEventXDK;
+class XDKAllocationTracker;
 class StringsStorage;
 
 class HeapProfiler {
@@ -45,6 +47,11 @@ class HeapProfiler {
 
   SnapshotObjectId PushHeapObjectsStats(OutputStream* stream,
                                         int64_t* timestamp_us);
+  void PushHeapObjectsXDKStats(OutputStream* stream);
+  void StartHeapObjectsTrackingXDK(int stackDepth, bool retentions,
+                                   bool strict_collection = false);
+  v8::internal::HeapEventXDK* StopHeapObjectsTrackingXDK();
+
   int GetSnapshotsCount();
   HeapSnapshot* GetSnapshot(int index);
   SnapshotObjectId GetSnapshotObjectId(Handle<Object> obj);
@@ -66,7 +73,8 @@ class HeapProfiler {
 
   bool is_tracking_object_moves() const { return is_tracking_object_moves_; }
   bool is_tracking_allocations() const {
-    return !allocation_tracker_.is_empty();
+    return (!allocation_tracker_.is_empty() ||
+            !allocation_tracker_xdk_.is_empty());
   }
 
   Handle<HeapObject> FindHeapObjectById(SnapshotObjectId id);
@@ -83,6 +91,7 @@ class HeapProfiler {
   base::SmartPointer<StringsStorage> names_;
   List<v8::HeapProfiler::WrapperInfoCallback> wrapper_callbacks_;
   base::SmartPointer<AllocationTracker> allocation_tracker_;
+  base::SmartPointer<XDKAllocationTracker> allocation_tracker_xdk_;
   bool is_tracking_object_moves_;
   base::Mutex profiler_mutex_;
   base::SmartPointer<SamplingHeapProfiler> sampling_heap_profiler_;

src/profiler/heap-snapshot-generator-inl.h

@@ -27,7 +27,7 @@ HeapSnapshot* HeapGraphEdge::snapshot() const {
 
 
 int HeapEntry::index() const {
-  return static_cast<int>(this - &snapshot_->entries().first());
+  return static_cast<int>(this - &entries_->first());
 }