[WIP] Refactor to Introduce Backend Abstraction #2011
Conversation
…parallelization_strategy
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
| Mark tie information right before we run passes because dynamo tracing will alter the parameter name while our | ||
| passes don't. |
There was a problem hiding this comment.
| Mark tie information right before we run passes because dynamo tracing will alter the parameter name while our | |
| passes don't. | |
| Mark information about tied parameters right before running passes because dynamo tracing alters the names of the parameters while our passes do not. |
There was a problem hiding this comment.
Are you sure naming this pre_process is the most adapted considered it just marks for tied weights?
There was a problem hiding this comment.
I think generally it means something that needs done before we run passes, and weights tying info marking happens to be one of them
| ) -> nn.Module: | ||
| raise NotImplementedError | ||
|
|
||
| def pre_process(self, graph_module: GraphModule, ctx: "ParallelExecutionCtx", config: "Config") -> GraphModule: |
There was a problem hiding this comment.
a Config is a data class which records static configurations during the whole process
| raise ValueError( | ||
| "`sequence_parallel` can not be activated when `tp_mode` is not set to `REDUCE_SCATTER` in nanotron backend" | ||
| ) |
There was a problem hiding this comment.
Is it best to fail like that or change the setting in Nanotron ourselves?
|
@zhenglongjiepheonix what's the status? |
|
the nanotron backend is not tested, for the default backend, everything works fine, it contains the newest update and has addressed comments |
|
Ok so ready for final review? |
Basically It's for reference, if someone is working on the support for nanotron indeed, then the correctness of nanotronbackend needs to be verified and additional tests are needed, but in my opinion this PR marks the boundary of optimum and nanotron, the rest of work should be implemented inside nanotron and optimum just expose |
What does this PR do?
The
NanotronBackendis still WIP and untested, but it would be nice to get some feedbacks first.