qcodes overhead

[peendebak]

In qcodes there seems to be quite some overhead, both in the qc.Loop and in the different objects living on servers. An example script is included below. The output of this script on my system:

## server
DataSet: DataMode.LOCAL, location='2016-05-24/12-18-47'
   chan0: chan0
   amplitude: amplitude
started at 2016-05-24 12:18:50
qc.Loop: dt 2.94 (length 2000)
simple get: dt 1.31
## local
DataSet: DataMode.LOCAL, location='2016-05-24/12-18-51'
   chan0: chan0
   amplitude: amplitude
started at 2016-05-24 12:18:51
qc.Loop: dt 0.19 (length 2000)
simple get: dt 0.02

For a small dataset of 2000 points the qc.Loop takes about 3 seconds! I set all delays to zero, and the mock model does no calculations at all. The same "measurement" with a simple loop and local objects takes hardly any time.

Can we reduce this overhead or am I missing some options than I should set?

#%% Packages
import numpy as np
import qcodes as qc
import time
from toymodel import MockGates, MockMeter, MockModel

#%% Make toy model

server_name=None
server_name='test%d' % (1000+np.random.randint(1000) )

class MockModelLocal:
    ''' Same as MockModel, but without a server    '''
    def __init__(self, name='Model-{:.7s}'):
        pass

    def ask(self, *query):
        return self.do_query(*query)            

    def write(self, *query):
        self.do_query(*query)            

    def do_query(self, *query):
        query_args = query[1:]
        query = query[0].split(':')

        instrument = query[0]

        param = query[1]
        if param[-1] == '?' and len(query) == 2:
            getter = getattr(self, instrument + '_get')
            return getter(param[:-1])
        elif len(query) <= 3:
            value = query[2] if len(query) == 3 else None
            setter = getattr(self, instrument + '_set')
            setter(param, value)
        else:
            raise ValueError


class AModel(MockModel):
    def __init__(self):
        self._gates = [0.0, 0.0, 0.0]
        super().__init__()

    def gates_set(self, parameter, value):
        pass

    def gates_get(self, parameter):
        return 0

    def meter_get(self, parameter):
        return np.random.rand()

class AModelLocal(MockModelLocal):
    def __init__(self):
        self._gates = [0.0, 0.0, 0.0]
        super().__init__()

    def gates_set(self, parameter, value):
        pass

    def gates_get(self, parameter):
        return 0

    def meter_get(self, parameter):
        return np.random.rand()


#%% Test speed

model = AModel()
gates = MockGates('gates', model=model, server_name=server_name)
meter = MockMeter('meter', model=model, server_name=server_name)
station = qc.Station(gates, meter, server_name=server_name)
station.set_measurement(meter.amplitude)


steps=gates.chan0[0:50:.025]
t0=time.time()
data = qc.Loop(steps, 0.0).run(data_manager=False, background=False)
dt=time.time()-t0
print('qc.Loop: dt %.2f (length %d)'  % (dt, len(steps)) )

t0=time.time()
for x in range(2000):
    val=meter.amplitude.get()
dt=time.time()-t0
print('simple get: dt %.2f'  % dt)


#%% Test speed (local)

server_name=None
model = AModelLocal()
gates = MockGates('gates', model=model, server_name=server_name)
meter = MockMeter('meter', model=model, server_name=server_name)
station = qc.Station(gates, meter, server_name=server_name)
station.set_measurement(meter.amplitude)


print('## local')
steps=gates.chan0[0:50:.025]
t0=time.time()
data = qc.Loop(steps, 0.0).run(data_manager=False, background=False)
dt=time.time()-t0
print('qc.Loop: dt %.2f (length %d)'  % (dt, len(steps)) )

t0=time.time()
for x in range(2000):
    val=meter.amplitude.get()
dt=time.time()-t0
print('simple get: dt %.2f'  % dt)

[AdriaanRol]

I am experiencing similar issues, I have never bothered to quantify this but this is definitely something we want to tune a bit.

[giulioungaretti]

Out of curiosity what happens if you double and triple the number of points ?
(I would do it myself but I am currently computer-less, and yeah qcodes does not run on my ipad just yet :P )

[peendebak]

Running time seems linear in the number of points (see data below):

for nn in [1, 10,100,1000,3000, 6000]:
    steps=gates.chan0[0:100:(100/nn)]
    numsteps=len(steps)
    t0=time.time()
    data = qc.Loop(steps, 0.0).run(data_manager=False, background=False, quiet=True)
    dt=time.time()-t0
    print('qc.Loop: number of steps %d: dt %.2f (length %d)'  % (numsteps, dt, len(steps)) )

Results in

qc.Loop: number of steps 1: dt 0.01 (length 1)
qc.Loop: number of steps 10: dt 0.01 (length 10)
qc.Loop: number of steps 100: dt 0.10 (length 100)
qc.Loop: number of steps 1000: dt 0.84 (length 1000)
qc.Loop: number of steps 3000: dt 2.40 (length 3000)
qc.Loop: number of steps 6000: dt 4.47 (length 6000)

[peendebak]

I think we should at least add a performance test to catch regressions. I am not familiar with unit tests relating to performance. The calculation time in seconds does not tell everything because the test can be executed on different platforms. For another project I compared the speed of a certain function to the speed calculating a fixed function (e.g. f(x)=2+sin(x), for x=1:100000), but I guess there are better ways to do this.

@giulioungaretti Can you set up something for this?

[jenshnielsen]

The loop will eventually be retired. Performance benchmarks are a good idea but will not be implemented fro the loop