C API Parser and Example Lifetime

Current

void start_parser(vw& all);
void end_parser(vw& all);

bool is_ring_example(vw& all, example* ae);

// The next commands deal with creating examples.  Caution: VW does not all allow creation of many examples at once by 
// default.  You can adjust the exact number by tweaking ring_size.
/* The simplest of two ways to create an example.  An example_line is the literal line in a VW-format datafile.
 */
example* read_example(vw& all, char* example_line);
example* read_example(vw& all, std::string example_line);

// The more complex way to create an example. 
// after you create and fill feature_spaces, get an example with everything filled in. 
example* import_example(vw& all, const std::string& label, primitive_feature_space* features, size_t len); 

// callers must free memory using release_example 
// this interface must be used with care as finish_example is a no-op for these examples. 
// thus any delay introduced when freeing examples must be at least as long as the one 
// introduced by all.l->finish_example implementations. 
// e.g. multiline examples as used by cb_adf must not be released before the finishing newline example. 
example* alloc_examples(size_t, size_t); 
void dealloc_example(void (*delete_label)(void*), example& ec, void (*delete_prediction)(void*) = nullptr); 

void parse_example_label(vw& all, example& ec, std::string label); 
void setup_examples(vw& all, v_array<example*>& examples); 
void setup_example(vw& all, example* ae); 
example* new_unused_example(vw& all); 
example* get_example(parser* pf); 

// after export_example, must call releaseFeatureSpace to free native memory primitive_feature_space* export_example(vw& all, example* e, size_t& len); 
void releaseFeatureSpace(primitive_feature_space* features, size_t len); 

Deprecate

example* alloc_examples(size_t, size_t) // Deprecate this, use new_unused_example instead 

void dealloc_example(void (*delete_label)(void*), example ec, void (*delete_prediction)(void*) = nullptr) // Deprecate this. Use finish_example (which internally calls is_ring_example().. maybe too slow?) 

void setup_examples(vw& all, v_array<example*>& examples); // Deprecate, Used in CLI. Don’t expose v_array 

void finish_example(vw& all, multi_ex& ec); // Deprecate, don't expose c++ object 

Proposed

ErrorCode start_parser(vw*) // Do we want to specify which parser to start here? Currently its done via options
ErrorCode end_parser(vw*) 

bool is_ring_example(const vw*, const example*)  // Returns false if a user-defined example factory was used to create an example

ErrorCode read_example(vw*, const char*, example_factory_t, void* example_factory_context, example**, size_t*)  // might return a multiline example in the case of the json parser

ErrorCode import_example(vw*, const char*,  primitive_feature_space*, size_t, example_factory_t, void* example_factory_context, example**, size_t*)  // primitive_feature_space can be modified or destroyed after this call without consequence 

ErrorCode parse_example_label(vw*, example*, const char* label)

ErrorCode setup_example(vw*, example*);  // Maybe deprecate this. Seems to be doing stuff that should be done in read/import_example

example* new_unused_example(vw*, example_factory_t, void* example_factory_context)  // block if pool is used and there are no free examples, factory behavior if factory is used 

example* get_parsed_example(vw*)  // What happens if there are no parsed examples? Block or nullptr 

void finish_example(vw*, example*, size_t); // finish one or more examples? How do we handle multi_ex? 

void clear_example(vw*, example*);

primitive_feature_space* create_primitive_feature_space(size_t)

primitive_feature_space* export_example(example*, size_t*)

void destroy_primitive_feature_space(primitive_feature_space*)

Open questions

How do we handle multiline examples?

• Should we allow read_example to pass in multi-line text and we return an example* and size?
• How do we handle json parser vs text parser discrepancy (single example vs vector of examples)?