Hi, how could I reproduce results for code documentation as described in the paper

[rishab-32]

Sorry to bother you again. I want to know how could I run code2seq for code documentation. I am trying to test code2seq on my dataset.

[urialon]

Hi @rishab-32 ,
Is this question different than your question in #41 ?

[rishab-32]

Hi @urialon,

Yes, it is the same I have been able to get the results for the code captioning task, thanks for your help.

However, now I am trying to get the result for code documentation. From your previous comments. What I understand is

1. Convert Hu et al. dataset from one-large-JSON into a directory of many files (for each of training/validation/test) - Done using the following script.
   https://gist.github.com/urialon/6ce2ffab7b675d9437b730246dc07827

2. Run preprocess.sh on the generated directories. This creates three c2s each for the train, test and valid -Done

3. Finally, you need to "insert" the code sequences into the produced files, using this script:
   https://gist.github.com/urialon/7c4d129f6cb45c2cdba669a15f132fb4

I am confused about step 3, do I have to run the script in step 3 on the train, test, and valid directories(containing .java files and align.txt) or the c2s files generated after preprocess.sh?

And what is the next step I should follow after this?

Thanks for your help,

Best,
Rishab Sharma.

[urialon]

Hi @rishab-32 ,

I didn't upload this pipeline officially because that dataset was problematic (if you take a look, you'll see that the NL labels contain many: " " ), and the paper's results were far from accurate.

If I remember correctly, step3 should be used here: https://github.com/tech-srl/code2seq/blob/master/preprocess.sh#L53

• after calling the JavaExtractor, but before performing the following preprocessing and creating the c2s files.

Afterward, run the following lines: https://github.com/tech-srl/code2seq/blob/master/preprocess.sh#L58
but on the outputs of step 3 rather than the output of the JavaExtractor.

Best,
Uri

[rishab-32]

Hi @urialon, thanks a lot for being so helpful and cooperative. Yes, you are right. Even I have seen different results being reported for the same dataset.

One last question, so the c2s files are arranged in the following form.

method_name context.

Right? where methodname acts as the target sequence while training. If I am able to replace the method_name(nonduplicates) with the natural language summary, would this be the right approach to train the model and report results, as I am using code2seq as one of the baseline to compare with my approach for the code documentation task?

[urialon]

Sorry, I meant that many NL labels in the dataset contain: "<!-- begin-user-doc --> <!-- end-user-doc -->" as their NL documentation!

Regarding your question:
Exactly! The files that are outputted from the JavaExtractor are arranged in this form.
The c2s files are also arranged in this form, except that they are truncated if they contain too many contexts, and padded with spaces if they have too few contexts.
This truncation+padding happens in this step: https://github.com/tech-srl/code2seq/blob/master/preprocess.sh#L63

Once you manage to train a model on this dataset, I recommend using these hyperparameters that are better suited for these long documentations (the default hyperparams are optimized for short method names):

config.WORDS_MIN_COUNT = 20
config.TARGET_WORDS_MIN_COUNT = 2
config.EMBEDDINGS_SIZE = 128 * 4
config.RNN_SIZE = 128 * 4
config.DECODER_SIZE = 512
config.NUM_DECODER_LAYERS = 2

Or use the same sizes/layers as your model.

Additionally, set the max documentation length config.MAX_TARGET_PARTS similarly as your model and the other baselines. Otherwise, the model will be limited to predict only 6 words in the output.

Best,
Uri

[rishab-32]

Thanks a lot, @urialon for clearing things up and sure I would use the above mentioned recommended parameters.

Best,
Rishab Sharma.

[rishab-32]

Hi @urialon, while I was running the train.sh, I encountered the following error. could you suggest any possible change I need to make?

tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
(0) Invalid argument: Expect 1001 fields but have 30 in record
[[{{node IteratorGetNext}}]]
[[IteratorGetNext/_25]]
(1) Invalid argument: Expect 1001 fields but have 30 in record
[[{{node IteratorGetNext}}]]
0 successful operations.
0 derived errors ignored.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "code2seq.py", line 39, in
model.train()
File "/content/code2seq/model.py", line 96, in train
_, batch_loss = self.sess.run([optimizer, train_loss])
File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/client/session.py", line 956, in run
run_metadata_ptr)
File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/client/session.py", line 1180, in _run
feed_dict_tensor, options, run_metadata)
File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/client/session.py", line 1359, in _do_run
run_metadata)
File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/client/session.py", line 1384, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
(0) Invalid argument: Expect 1001 fields but have 30 in record
[[node IteratorGetNext (defined at /tensorflow-1.15.2/python3.6/tensorflow_core/python/framework/ops.py:1748) ]]
[[IteratorGetNext/_25]]
(1) Invalid argument: Expect 1001 fields but have 30 in record
[[node IteratorGetNext (defined at /tensorflow-1.15.2/python3.6/tensorflow_core/python/framework/ops.py:1748) ]]
0 successful operations.
0 derived errors ignored.

Original stack trace for 'IteratorGetNext':
File "code2seq.py", line 39, in
model.train()
File "/content/code2seq/model.py", line 77, in train
config=self.config)
File "/content/code2seq/reader.py", line 43, in init
self.output_tensors = self.compute_output()
File "/content/code2seq/reader.py", line 192, in compute_output
return self.iterator.get_next()
File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/data/ops/iterator_ops.py", line 426, in get_next
name=name)
File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/ops/gen_dataset_ops.py", line 2518, in iterator_get_next
output_shapes=output_shapes, name=name)
File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/framework/op_def_library.py", line 794, in _apply_op_helper
op_def=op_def)
File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/framework/ops.py", line 3357, in create_op
attrs, op_def, compute_device)
File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/framework/ops.py", line 3426, in _create_op_internal
op_def=op_def)
File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/framework/ops.py", line 1748, in init
self._traceback = tf_stack.extract_stack()

[urialon]

Expect 1001 fields but have 30 in record

I think that didn't run the preprocess.py on the output of the scripts.

preprocess.py is supposed to pad targets that have less than 1000 contexts.

[rishab-32]

Thanks, Now everything is working fine.