Replace LAMA with Dask: grouping methods to migrate #295
Comments
sadielbartholomew
added
high priority
performance
|
Hi Sadie - this is great, just what we need. Thanks. (Is there a line 8 missing from the Python code?) |
|
Thanks @davidhassell, after our conversation I thought it was needed to allow us to organise the work. I have started to add in some basic groupings that may be helpful when thinking about methods with similar requirements and ones to tackle as a group, etc. I will add the properties in as a group too, though I don't think there are that many sadly (since as you mentioned they count for three in the table). Please edit the table as and when you wish to, there is plenty of info and groupings that could be added but I was going to chip away at it as useful rather than doing it all at once. I will ensure to keep it up to date in terms of what methods are in a PR that is open or merged.
Ah yes, good spot, I must have missed it when copying lines from the interactive session. I will add that back in now. |
|
I should add, I am aware you aren't a fan of emojis but the green and red colouring here makes it much easier to process at a glance whether a method is done or not than Booleans or Y/N... |
|
colon smiley colon |
|
Hi @davidhassell, here's the code I have quickly written to grab the total count of completed methods: # TODO: copy table as string here, heeding warning below.
# Make sure table is copied such that the first line of table starts as the
# first character, with no newline, and the string ends at end of final line.
# Basically as the line below indicates. (Code relies on that format.)
table = """<insert the current table here!>"""
def get_count_of_completed_methods(table_string):
"""Get count (and total) of (un)daskified methods from table of PR 295."""
lines = table_string.split("\n")
total_number_methods = len(lines) - 2 # subtract two heading lines
count = 0
validate = 0
for line in lines[2:]: # skip 2 header lines
if ":heavy_check_mark:" in line:
count += 1
elif ":heavy_multiplication_x:" in line:
validate += 1
else:
print(f"WARNING: POSSIBLY DODGY LINE? CHECK:'{line}'")
# Check that nothing dodgy has happened, that all lines/methods got covered
missing_methods = total_number_methods - count - validate
print("NUMBER OF METHODS UNACCOUNTED FOR IS", missing_methods)
return count, total_number_methods # include total for extra info
results = get_count_of_completed_methods(table)
print(
"COUNT OF DASKIFIED METHODS IS {} FROM TOTAL {}.".format(*results)
)
print(f"THAT'S A COMPLETION PERCENTAGE OF {results[0]/results[1]*100:.1f}")Right now (see comment timestamp later if needed as a reference) is: $ python daskification-table.py
NUMBER OF METHODS UNACCOUNTED FOR IS 0
COUNT OF DASKIFIED METHODS IS 228 FROM TOTAL 315.
THAT'S A COMPLETION PERCENTAGE OF 72.4As you predicted, once I put my |
|
@davidhassell I've just updated the table to account for #409 and the scores on the doors are now: $ python daskification-table.py
NUMBER OF METHODS UNACCOUNTED FOR IS 0
COUNT OF DASKIFIED METHODS IS 306 FROM TOTAL 315.
THAT'S A COMPLETION PERCENTAGE OF 97.1So we're very close now, as the green to red ratio indicates just from scrolling down the table! Just getting up PRs for |
|
Data test suite now passes! |
Table for #182
(See comment update datetime for author and datestamp of last table update)
Note:
306 methods in totalnow 315 methods in total in the table.Datamethod nameHDF_chunksUnits_HDF_chunks_Units_YMDhms__abs____add____and____array____bool____contains____data____deepcopy____div____doc_template____docstring_package_depth____docstring_substitutions____eq____float____floordiv____ge____getitem____gt____hash____iadd____iand____idiv____ifloordiv____ilshift____imod____imul____init____int____invert____ior____ipow____irshift____isub____iter____itruediv____ixor____le____len____lshift____lt____mod____module____mul____ne____neg____new____or____pos____pow____query_set____query_wi____query_wo____radd____rand____rdiv____reduce____reduce_ex____rfloordiv____rlshift____rmod____rmul____ror____round____rpow____rrshift____rshift____rsub____rtruediv____rxor____setitem____sub____truediv____xor___all_axes_all_axis_names_asdatetime_asreftime_atol_auxiliary_mask_auxiliary_mask_add_component_auxiliary_mask_from_1d_indices_auxiliary_mask_return_auxiliary_mask_subspace_auxiliary_mask_tidy_axes_binary_operation_change_axis_names_chunk_add_partitions_collapsedask >= 2022.03.0_collapse_create_weights_collapse_finalise_collapse_mask_collapse_optimize_weights_collapse_subspace_combined_units_create_auxiliary_mask_component_custom_cyclic_default_del_Array_del_component_dtype_equals_equals_preprocess_flag_partitions_for_processing_flip_get_Array_get_component_has_component_initialise_netcdf_is_abstract_Array_subclass_isdatetime_item_move_flip_to_partitions_ndim_new_axis_identifier_package_parse_axes_parse_indices_pmaxes_pmndim_pmshape_pmsize_rtol_set_Array_set_CompressedArray_set_component_set_partition_matrix_set_subspace_shape_share_lock_files_share_partitions_size_unary_operationadd_partitionsalldask >= 2022.6.0.allcloseanydask >= 2022.6.0.apply_maskingarccosarccosharcsinarcsinharctanarctanhargmaxarraycftime >= 1.6.0asdatabinary_maskceilchange_calendarchunksclipclosecompressedcompressed_arraycomputeconcatenateconcatenate_dataconvolution_filtercopycoscoshcountdask >= 2022.03.0.count_maskeddask >= 2022.03.0.creation_commandscumsumcyclicdatadatetime_arraycftime 1.6.0datetime_as_stringdatumdaydel_calendardel_fill_valuedel_unitsdiffdigitizedtarraydtypedumpdumpddumpsemptyequalsexpfilesfill_valuefilledfirst_elementfits_in_memoryfits_in_one_chunk_in_memoryflatflattenflipfloorfullfuncget_calendarget_compressed_axesget_compressed_dimensionget_compression_typeget_countget_dataget_filenamesget_fill_valueget_indexget_listget_unitshaloconvolution_filter.hardmaskhas_calendarhas_fill_valuehas_unitshourin_memoryinsert_dimensioninspectintegraliscloseis_maskedispartitionedisscalarlast_elementloaddloadslogmaskmask_fpemask_invalidmasked_allmasked_invalidmaxmaximummaximum_absolute_valuemeanmean_absolute_valuemean_of_upper_decilemedianmid_rangeminminimumminimum_absolute_valueminutemonthnbytesnc_clear_hdf5_chunksizesnc_hdf5_chunksizesnc_set_hdf5_chunksizesndimndindexonesouterproductoverride_calendaroverride_unitspartition_boundariespartition_configurationpartitionspercentilepersistrangerechunkreconstruct_sectioned_datareshaperintrollroot_mean_squareroundsample_sizesave_to_disksdsecondsecond_elementsectionset_calendarset_fill_valueset_unitsseterrshapesinsinhsizesourcesquaresqrtsqueezestandard_deviationstdstatssumsum_of_squaressum_of_weightssum_of_weights2swapaxestantanhto_dask_arrayto_diskto_memorytolisttransposetruncuncompressuniquedask >= 2022.6.0.varvariancevarraywhereyearzerosCode to re-generate table
(In case a different form proves useful. Uses the library python-tabulate for ease.)
List of groupings of methods referenced in above table
See 'Group with (if at all)' column in table, which is designed to denote methods which are lunked in some useful way (say, being similar so can be migrated in a set of related PRs). Any groups referenced there should be added here.
Example: name group A for all of the trig. methods, and list 'A' in the 'Group with' column for all such methods, as well as noting A with all methods in it here.
arctan2is special (as takes two args) and not yet implemented, but now can be.__add__,__sub__,__mul__,__div__,__floordiv__,__mod__, etc.__abs__,__pos__,__neg__,__invert__, etc.__eq__,__ne__,__gt__,__lt__,__ge__,__le__, etc.__and__,__or__,__xor__, etc._rtol,_atol__lshift__,__ilshift__,__rlshift__,__rshift__,__irshift__,__rrshift__cfdmmax,mean,_collapseto_memoryThe text was updated successfully, but these errors were encountered: