Bug Report: infinite loop in tmState.ChangeTabletType

[mvh-stripe]

Overview of the Issue

		// Update the tablet record first.
		_, err := topotools.ChangeType(ctx, ts.tm.TopoServer, ts.tm.tabletAlias, tabletType, PrimaryTermStartTime)
		if err != nil {
			log.Errorf("error in changing type in tablet record in topo for tablet %s :- %v", topoproto.TabletAliasString(ts.tm.tabletAlias), err)
			log.Errorf("will keep trying to read from the topo server")
			// In case of a topo error, we aren't sure if the data has been written or not.
			// We must read the data again and verify whether the previous write succeeded or not.
			// The only way to guarantee safety is to keep retrying read until we succeed
			for {
				ti, errInReading := ts.tm.TopoServer.GetTablet(ctx, ts.tm.tabletAlias)
				if errInReading != nil {
					time.Sleep(100 * time.Millisecond)
					continue
				}
                                 ....
                         }

8b3f61f
^ A recent change introduced the possibility of an infinite loop when the context gets canceled. In the for loop, we never check if the context has an error, and so it keeps trying to call topoServer.GetTablet every 100ms and never exit. This is bad because this goroutine also holds the mutex for the entire duration. So retries of the ChangeTabletType rpc also get stuck on the mutex.

Reproduction Steps

N/A

Binary Version

N/A

Operating System and Environment details

N/A

Log Fragments

No response

[deepthi]

/assign @vitessio/cluster-management

[GuptaManan100]

Fixed by #9842