Fairseq安装问题

[xy1137030414]

1.虚拟机：PyTorch 2.1.0 Python 3.10(ubuntu22.04) Cuda 12.1

2.新建了python虚拟环境，并指定python版本为3.11.10.

3.之后按照mcts文档下载相应版本的包

4.安装fairseq：git clone→ cd fairseq→ 按照文档下载特定版本的fairseq→pip install 特定的版本

5.显示安装成功，但有报错和警告：
DEPRECATION: omegaconf 2.0.6 has a non-standard dependency specifier PyYAML>=5.1.*. pip 23.3 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of omegaconf or contact the author to suggest that they release a version with a conforming dependency specifiers. Discussion can be found at https://github.com/pypa/pip/issues/12063 Installing collected packages: fairseq Attempting uninstall: fairseq Found existing installation: fairseq 1.0.0a0+06c65c8 Uninstalling fairseq-1.0.0a0+06c65c8: Successfully uninstalled fairseq-1.0.0a0+06c65c8 Successfully installed fairseq-1.0.0a0+06c65c8 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv WARNING: There was an error checking the latest version of pip.

6.利用python -c "import fairseq; print(fairseq.version)"查询是否安装成功，报错如下：

(forst) root@autodl-container-98e944be4b-90b41232:/fairseq# python -c "import fairseq; print(fairseq.version)"
common - <dataclasses._MISSING_TYPE object at 0x7fb1f649c290>
Traceback (most recent call last):
File "", line 1, in
File "/root/fairseq/fairseq/init.py", line 31, in
hydra_init()
File "/root/fairseq/fairseq/dataclass/initialize.py", line 25, in hydra_init
cs.store(name=k, node=v)
File "/root/miniconda3/envs/forst/lib/python3.11/site-packages/hydra/core/config_store.py", line 85, in store
cfg = OmegaConf.structured(node)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/forst/lib/python3.11/site-packages/omegaconf/omegaconf.py", line 140, in structured
return OmegaConf.create(obj, parent)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/forst/lib/python3.11/site-packages/omegaconf/omegaconf.py", line 177, in create
return OmegaConf._create_impl(obj=obj, parent=parent)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/forst/lib/python3.11/site-packages/omegaconf/omegaconf.py", line 245, in _create_impl
format_and_raise(node=None, key=None, value=None, msg=str(e), cause=e)
File "/root/miniconda3/envs/forst/lib/python3.11/site-packages/omegaconf/_utils.py", line 694, in format_and_raise
_raise(ex, cause)
File "/root/miniconda3/envs/forst/lib/python3.11/site-packages/omegaconf/_utils.py", line 610, in _raise
raise ex # set end OC_CAUSE=1 for full backtrace
^^^^^^^^
omegaconf.errors.ValidationError: Object of unsupported type: '_MISSING_TYPE'
full_key:
reference_type=None
object_type=None

7.显示错误如上，我对下载好的特定版本的fairseq的config配置文件进行了修改：

https://github.com/DataClass
class FairseqConfig(FairseqDataclass):
common: CommonConfig = field(default_factory=CommonConfig)
common_eval: CommonEvalConfig = field(default_factory=CommonEvalConfig)
distributed_training: DistributedTrainingConfig = field(default_factory=DistributedTrainingConfig)
dataset: DatasetConfig = field(default_factory=DatasetConfig)
optimization: OptimizationConfig = field(default_factory=OptimizationConfig)
checkpoint: CheckpointConfig = field(default_factory=CheckpointConfig)
bmuf: FairseqBMUFConfig = field(default_factory=FairseqBMUFConfig)
generation: GenerationConfig = field(default_factory=GenerationConfig)
eval_lm: EvalLMConfig = field(default_factory=EvalLMConfig)
interactive: InteractiveConfig = field(default_factory=InteractiveConfig)
model: Any = field(default_factory=lambda: "default_model") # 提供默认值或工厂函数
task: Optional[Any] = None # 标记为可选字段
criterion: Optional[Any] = None
optimizer: Optional[Any] = None
lr_scheduler: Optional[Any] = None
scoring: Optional[Any] = None
bpe: Optional[Any] = None
tokenizer: Optional[Any] = None
ema: EMAConfig = field(default_factory=EMAConfig)

8.然后还是不行。

事实上在pip命令提示安装成功后，我就利用bash /root/mcts/scripts/simplification/data_process.sh命令尝试运行bpe.py和预处理文件，bpe文件可以正常运行输出，然后process文件就一开始缺少必要的依赖项，我按照提示下载了依赖项：omegaconf==2.0.6 和 hydra-core==1.0.7。之后，我又重新安装了fairseq，重新sh命令，这时候开始显示各种fairseq的依赖问题（命令行输出我已经找不到了，忘记记录了），我在chatgpt的指引下修改了config文件。但是一直还是不行，我刚才贴出的报错代码是在我修改了一番后的最新报错。

9.我的pip list：

(forst) root@autodl-container-98e944be4b-90b41232:/fairseq# pip list
DEPRECATION: Loading egg at /root/miniconda3/envs/forst/lib/python3.11/site-packages/apex-0.1-py3.11.egg is deprecated. pip 23.3 will enforce this behaviour change. A possible replacement is to use pip for package installation..
Package Version Editable project location

antlr4-python3-runtime 4.8
apex 0.1
apex 0.1
autocommand 2.2.2
backports.tarfile 1.2.0
bitarray 2.9.2
certifi 2024.8.30
cffi 1.17.1
charset-normalizer 3.3.2
colorama 0.4.6
Cython 3.0.11
fairseq 1.0.0a0+06c65c8 /root/fairseq
filelock 3.16.1
fsspec 2024.9.0
huggingface-hub 0.25.1
hydra-core 1.0.7
idna 3.10
importlib_metadata 8.0.0
importlib_resources 6.4.0
inflect 7.3.1
jaraco.collections 5.1.0
jaraco.context 5.3.0
jaraco.functools 4.0.1
jaraco.text 3.12.1
Jinja2 3.1.4
lxml 5.3.0
MarkupSafe 2.1.5
more-itertools 10.3.0
mpmath 1.3.0
networkx 3.3
numpy 2.1.1
nvidia-cublas-cu12 12.1.3.1
nvidia-cuda-cupti-cu12 12.1.105
nvidia-cuda-nvrtc-cu12 12.1.105
nvidia-cuda-runtime-cu12 12.1.105
nvidia-cudnn-cu12 9.1.0.70
nvidia-cufft-cu12 11.0.2.54
nvidia-curand-cu12 10.3.2.106
nvidia-cusolver-cu12 11.4.5.107
nvidia-cusparse-cu12 12.1.0.106
nvidia-nccl-cu12 2.20.5
nvidia-nvjitlink-cu12 12.6.77
nvidia-nvtx-cu12 12.1.105
omegaconf 2.0.6
packaging 24.1
pip 23.2.1
platformdirs 4.2.2
portalocker 2.10.1
pycparser 2.22
PyYAML 6.0.2
regex 2024.9.11
requests 2.32.3
sacrebleu 2.4.3
safetensors 0.4.5
setuptools 75.1.0
sympy 1.13.3
tabulate 0.9.0
tokenizers 0.20.0
tomli 2.0.1
torch 2.4.1
torchaudio 2.4.1
tqdm 4.66.5
transformers 4.45.1
triton 3.0.0
typeguard 4.3.0
typing_extensions 4.12.2
urllib3 2.2.3
wheel 0.44.0
zipp 3.19.2

10.我在怀疑是不是pip、python的版本太高导致，但是我觉得我实在是处理不了了，我可能写得不专业，因为我完全是小白，如有问题，请联系我。谢谢！