Loop forever or for n steps, or for given time.

[MerlinSmiles]

During measurements I often encounter situations where I want to run a loop over and over again, i.e. measuring IV-traces during cooldown.

Is there an easy solution for that?

What I can do right now is something like this:

step_param = ManualParameter('step', initial_value=0)
swp = SweepFixedValues(step_param,start=0,stop=100,step=1)
data = qc.Loop(swp).loop(k1v.sweep(-0.006, 0.006, num=51), 0.01).each(curr, vdrp).run()

It would be nice if we had loops that don't do anything than repeating the inner loop, or loops that would repeat the inner loop for a given amout of time.

qc.Loop_forever().each(BreakIf(sunshine==True), qc.Loop(...)
qc.Loop_forever(steps=100).loop(...)
qc.Loop_forever(time=3600).loop(...)

@alexcjohnson or anyone else, any ideas on how to do this?

[AdriaanRol]

@MerlinSmiles we use a NoneSweep for that, it corresponds basically to the step param you introduced. I think using a manual parameter in that fashion is simple enough and does not warrant introducing extra syntax.

I do however like the BreakIf statement. This could be combined with a Loop to e.g. 1e8 with a break if timemark>end_time. I don't think it warrants introducing the Loop_forever().

I have to admit I am a bit biased though, I much prefer to be allowed to use standard python (which allows all sorts of constructions you can come up with) within the loop than to introduce an ad hoc syntax for every special case we come up with. The reason I don't like it is that I think it is at use time very hard to know what all the specific commands are that are available, not to mention teaching this to your new master student. Therefore I am of the opinion simpler is better.

[MerlinSmiles]

@AdriaanRol generally I agree with your statement of 'simpler is better' but I'm more in favor of usability.
Especially for the new master student it might not be fun to reinvent a manual parameter just to do something which basically is a for or while loop, without knowing any stuff behind the scenes.
Digging through source code to understand how to make a parameter, and that the students needs a manualParameter for that?

[AdriaanRol]

Fair point, would it be easier if a ManualParameter that does nothing was predefined? The reason that this seems so natural to me is that we used to have sweep functions (which were basically wrappers around parameters) and we had the NoneSweep as one of the most basic ones.

To me this really seems the simplest generalisation of the concept of a loop.
You want to loop something but not change any parameter other than the index that marks the data. Therefore you would loop over some parameter that does nothing.

I think the question boils down to how many new concepts does someone need to learn.
If I would be explaining someone how QCodes works I would tell them about the concept of instruments, parameters and how to do a 1D and a 2D loop.

The question then is what is harder to learn, the ins and outs of the Loop syntax (+special cases) or how to use the concepts you have to do a repeated loop. As the None parameter involves 1 line of code this seems simple to me, especially as at this point there is no good overview of the different ways the Loop can be used. However in the ideal world we will end up with a good manual for the Loop and this may be the preferred way.

So to summarise my opinion on this: Yes, this is a basic functionality that should be there. And it is a matter of taste what is simpler to learn for a new user and this depends a lot on how we structure our concepts and explanations.

[MerlinSmiles]

You raise good points there, and you're right, the parameter belongs to the basic concept of qcodes so it has to be introduced anyways. I'm not in favor of a loop_forever anymore 😃

There is another point I didnt mention before, but that is more a bug / unhappy feature of the dataset, When the loop is created, an array to hold the data is created with it. for live-plotting that array is synced all the time.
When the step-loop is 1e9 and the inner loop has 1001 datapoints I get to a point where live-plotting is unusable.

[alexcjohnson]

If this is a parameter, then you'd use it like Loop(none_sweep[0:100:1]).each(...)?
That would certainly work, but it's not obvious what it does just by looking at it. I realize that it would be an extra word to learn, but it seems so much clearer to write Loop(repeat(100)).each(...). repeat would of course just create or use a none_sweep parameter, but without the awkward syntax, and it could support infinite loops like repeat().

This brings up that we really need to support expanding the data arrays during a Loop. Shouldn't be difficult, and I imagine we can just build it into the DataArray that if you try to set a point with an invalid index it automatically expands. Not sure how happy numpy will be to keep expanding again and again with every point, perhaps we will be better off expanding in chunks, but that's an implementation detail.

When the step-loop is 1e9 and the inner loop has 1001 datapoints I get to a point where live-plotting is unusable.

Right - to get started quickly I just had data_set.sync() push the WHOLE set every time. This is obviously not the right way to go long-term, but depending on how generic we want sync to be (what if data arrives out of order? what if multiple clients are listening to the DataServer at once?) it may be a bit tricky to get a minimal sync to work correctly.

[MerlinSmiles]

I have bad experience with expanding (almost) single points, expanding in chunks is much nicer, I've been doing newsize = oldsize * 2 a lot.

As for the sync, wouldnt it be enough to call data.sync(lastindex=xx) and in return the plot or whatever get what is new since then and the newest latest index. the plot can then store that internaly...

[alexcjohnson]

I have bad experience with expanding (almost) single points, expanding in chunks is much nicer, I've been doing newsize = oldsize * 2 a lot.

Sounds reasonable

As for the sync, wouldnt it be enough to call data.sync(lastindex=xx) and in return the plot or whatever get what is new since then and the newest latest index. the plot can then store that internaly...

Yes, as long as we don't allow data to arrive out of order. The plot wouldn't need to store it, the DataSet would.

[MerlinSmiles]

Why would the dataset have to save it? if I connect several plots, the dataset would have to keep track of all connected plots, no?
Then it sounds easier if an individual plot asks for 'give me whatever since index'

[alexcjohnson]

I suppose there are two sync steps here - first the DataSet has to sync with its clone in the DataServer, then the plot has to sync its own data with the DataSet. I suppose for pyqtgraph, since we're remotely connecting to a separate plotting process, BOTH of these steps need to send the data through some sort of pipe so could be slow... but I don't know how to make the latter step feed data incrementally (do you?) so I'm focused on the first step. If we can do that, then yes, the plot can ask a similar question.

[MerlinSmiles]

Ah, I didnt think about that... will think about it.