Remove apex deps

[saforem2]

• Add DEBUG option to toggle set -euxo pipefail in train_llama_alcf.sh

• Add option to switch between launching with deepspeed and mpiexec (default)

• Remove file: generate_config.sh and move logic for creating deepspeed_config.json into ALCF/helpers.sh directly

• Remove hard dependency on NVIDIA/apex

  • Explicitly, changes to:

    • megatron/model/__init__.py
    • megatron/optimizer/clip_grads.py
    • megatron/optimizer/distrib_optimizer.py
    • megatron/utils.py

  • This is done by replacing explicit import apex (or similar) calls with

    # from deepspeed.accelerator import get_accelerator
    # if get_accelerator().device_name() == 'cuda':
    try:
        from apex.multi_tensor_apply import multi_tensor_applier
        import amp_C
        HAS_APEX = True
    except Exception:
        HAS_APEX = False

    and using the default fallback methods when HAS_APEX is False

[saforem2]

• add ALCF/test_sunspot.sh to run simple test on Sunspot

[saforem2]

• add ALCF/test_sirius.sh