Drop dmlc-core commits that break MXNet build

[hcho3]

Description

Partially reverts #12129 by dropping commits dmlc/dmlc-core#409 and dmlc/dmlc-core#399.

These commits by themselves are fine, but they tigger double free memory error inside CSVIter. In #12139, the CSVIter test crashed on CentOS. The memory error has been inside CSVIter but remained undetected until dmlc/dmlc-core#409 and dmlc/dmlc-core#399 were added. See address_sanitizer_log.txt for a detailed log of double-free error.

Note that the dmlc-core submodule will now track the mxnet-stable branch of dmlc/dmlc-core.

Checklist

Essentials

Please feel free to remove inapplicable items for your PR.

• Changes are complete (i.e. I finished coding on this PR)
• All changes have test coverage:
• Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
• Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
• Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
• Code is well-documented:
• For user-facing API changes, API doc string has been updated.
• For new C++ functions in header files, their functionalities and arguments are documented.
• For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
• Check the API doc at http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
• To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Changes

• Revert dmlc-core to last working point
• Selectively add dmlc-core commits so that the double-free memory error in CSVIter is not triggered
• Enabled the disabled test test_CSVIter (disabled in Broken test test_io.test_CSVIter #12139)

Comments

• This is a temporary measure. We'll need to come back to CSVIter and address memory error. Address Sanitizer reveals double free issues in CSVIter.

[anirudh2290]

This should be fine as a temporary solution but we should move back to dmlc-core master branch instead of mxnet-stable once the issues are fixed.

[hcho3]

@anirudh2290 I found the bug. It was actually the threaded parser in dmlc-core, not MXNet's CSVIter: dmlc/dmlc-core#505. Investigating now.

[hcho3]

Output from Thread Sanitizer: log.txt

[anirudh2290]

@stu1130

[hcho3]

Minimal reproduction of data race, with combination of MXNet CSVIter and dmlc-core:

/* mxnet_integ.cc */
#include <mxnet/c_api.h>
#include <mxnet/io.h>
#include <dmlc/registry.h>
#include <type_traits>
#include <cstring>

int main(void) {
  const char* keys[] = {
    "data_csv", "data_shape", "label_csv", "batch_size", "dtype"
  };
  const char* vals[] = {
    "./incubator-mxnet/data.t", "(8, 8)", "./incubator-mxnet/label.t", "100", "float32"
  };
  static_assert(std::extent<decltype(keys)>::value == std::extent<decltype(vals)>::value,
                "keys[] and vals[] must be of same length");

  DataIterHandle csv_iter_handle;

  unsigned int size;
  DataIterCreator* handle;
  CHECK_EQ(MXListDataIters(&size, &handle), 0);
  LOG(INFO) << "size = " << size;

  for (unsigned int i = 0; i < size; ++i) {
    const char *name, *desc;
    unsigned int num_args;
    const char **arg_names, **arg_types, **arg_descs;
    CHECK(handle[i]);
    CHECK_EQ(MXDataIterGetIterInfo(handle[i], &name, &desc, &num_args, &arg_names, &arg_types, &arg_descs), 0);
    if (std::strcmp(name, "CSVIter") == 0) {
      LOG(INFO) << "Creating CSVIter";
      CHECK_EQ(MXDataIterCreateIter(handle[i], std::extent<decltype(keys)>::value, keys, vals, &csv_iter_handle), 0);
    }
  }
  CHECK_EQ(MXDataIterBeforeFirst(csv_iter_handle), 0);
  int next_res;
  CHECK_EQ(MXDataIterNext(csv_iter_handle, &next_res), 0);
  while (next_res) {
    LOG(INFO) << "Got a batch from the iterator";
    void* hdl;
    CHECK_EQ(MXDataIterGetData(csv_iter_handle, &hdl), 0);
    CHECK_EQ(MXDataIterGetLabel(csv_iter_handle, &hdl), 0);
    int pad;
    CHECK_EQ(MXDataIterGetPadNum(csv_iter_handle, &pad), 0);
    uint64_t index_size;
    uint64_t* index_data;
    CHECK_EQ(MXDataIterGetIndex(csv_iter_handle, &index_data, &index_size), 0);

    CHECK_EQ(MXDataIterNext(csv_iter_handle, &next_res), 0);
  }

  CHECK_EQ(MXDataIterFree(csv_iter_handle), 0);

  return 0;
}

Compile this example with

g++ -g -O0 -fsanitize=thread -o mxnet_integ mxnet_integ.cc -Wl,--whole-archive \
./incubator-mxnet/lib/libmxnet.a ./incubator-mxnet/3rdparty/tvm/nnvm/lib/libnnvm.a \
-I./incubator-mxnet/include/ ./incubator-mxnet/3rdparty/dmlc-core/libdmlc.a \
./incubator-mxnet/deps/lib/libprotobuf.a ./incubator-mxnet/deps/lib/libzmq.a \
-Wl,--no-whole-archive -DMSHADOW_USE_MKL=0 -DMSHADOW_USE_CUDA=0 \
-DMXNET_USE_OPENCV=0 -lgomp -lpthread -lrt -lz -lopenblas

Make sure that libmxnet.a is also compiled with flags -g -O0 -fsanitize=thread.