Skip to content
This repository has been archived by the owner on Aug 29, 2023. It is now read-only.

Cant open esacci.CLOUD.mon.L3C variable subset #279

Closed
forman opened this issue Jul 4, 2017 · 4 comments
Closed

Cant open esacci.CLOUD.mon.L3C variable subset #279

forman opened this issue Jul 4, 2017 · 4 comments
Assignees
@mzuehlke
Labels
bug ds wontfix
Milestone
IPM6

Comments

@forman
Copy link
Member

forman commented Jul 4, 2017

Expected behavior

Opening a local data source in GUI should display its first variable.

Actual behavior

Cate Desktop, version 0.9.0-dev.2

unsupported operand type(s) for +: 'int' and 'tuple'

An error (code 10) occurred while executing a backend process:

Traceback (most recent call last):
  File "C:\Users\Norman\IdeaProjects\ccitools\cate-core\cate\util\web\jsonrpchandler.py", line 163, in send_service_method_result
    result = future.result()
  File "C:\Users\Norman\Miniconda3\envs\cate\lib\concurrent\futures\_base.py", line 398, in result
    return self.__get_result()
  File "C:\Users\Norman\Miniconda3\envs\cate\lib\concurrent\futures\_base.py", line 357, in __get_result
    raise self._exception
  File "C:\Users\Norman\Miniconda3\envs\cate\lib\concurrent\futures\thread.py", line 55, in run
    result = self.fn(*self.args, **self.kwargs)
  File "C:\Users\Norman\IdeaProjects\ccitools\cate-core\cate\util\web\jsonrpchandler.py", line 216, in call_service_method
    result = method(*method_params, monitor=monitor)
  File "C:\Users\Norman\IdeaProjects\ccitools\cate-core\cate\webapi\websocket.py", line 294, in set_workspace_resource
    return workspace.to_json_dict()
  File "C:\Users\Norman\IdeaProjects\ccitools\cate-core\cate\core\workspace.py", line 205, in to_json_dict
    ('resources', self._resources_to_json_list())
  File "C:\Users\Norman\IdeaProjects\ccitools\cate-core\cate\core\workspace.py", line 217, in _resources_to_json_list
    resource_descriptor = self._get_resource_descriptor(res_id, res_update_count, res_name, resource)
  File "C:\Users\Norman\IdeaProjects\ccitools\cate-core\cate\core\workspace.py", line 238, in _get_resource_descriptor
    variable_descriptors.append(self._get_xarray_variable_descriptor(variable))
  File "C:\Users\Norman\IdeaProjects\ccitools\cate-core\cate\core\workspace.py", line 306, in _get_xarray_variable_descriptor
    image_config = self._get_variable_image_config(variable)
  File "C:\Users\Norman\IdeaProjects\ccitools\cate-core\cate\core\workspace.py", line 358, in _get_variable_image_config
    max_size, tile_size, num_level_zero_tiles, num_levels = ImagePyramid.compute_layout(array=variable)
  File "C:\Users\Norman\IdeaProjects\ccitools\cate-core\cate\util\im\image.py", line 777, in compute_layout
    tile_size = (compute_tile_size(max_size[0], chunk_size=chunk_width, int_div=int_div),
  File "C:\Users\Norman\IdeaProjects\ccitools\cate-core\cate\util\im\utils.py", line 126, in compute_tile_size
    num_chunks = cardinal_div_round(ts, chunk_size)
  File "C:\Users\Norman\IdeaProjects\ccitools\cate-core\cate\util\im\utils.py", line 66, in cardinal_div_round
    return int(num + denom - 1) // int(denom)
TypeError: unsupported operand type(s) for +: 'int' and 'tuple'

Steps to reproduce the problem

  1. create local data source local.2010.CLOUD.mon.L3C.CLD_PRODUCTS.AVHRR.multi-platform.AVHRR-AM.2-0.r1
    • Time: 2010-01-01, 2010-21-31
    • Variable: cfc
  2. open data source
  3. get error shown above

Specifications

Cate 0.9.0.dev2
@forman forman added the bug label Jul 4, 2017
@forman forman added this to the IPM6 milestone Jul 4, 2017
@forman
Copy link
Member Author

forman commented Jul 4, 2017

Strange, with esacci.CLOUD.mon.L3C.CLD_PRODUCTS.multi-sensor.multi-platform.ATSR2-AATSR.2-0.r1 as remote source it works fine

@forman forman assigned forman and mzuehlke and unassigned forman Jul 5, 2017
@mzuehlke
Copy link
Collaborator

mzuehlke commented Jul 7, 2017 •
edited
Loading

The problem lies in the not lat and lon coordinate variables that are not aligned throughout the dataset.

201005-ESACCI-L3C_CLOUD-CLD_PRODUCTS-AVHRR_METOPA-fv2.0.nc
lat = [-89.75       -89.25       -88.75       -88.25       -87.75       -87.25
 -86.75       -86.25       -85.75       -85.25       -84.75       -84.25
 -83.75       -83.25       -82.75       -82.25       -81.75       -81.25
 -80.75       -80.25       -79.75       -79.25       -78.75       -78.25
 -77.75       -77.25       -76.75       -76.25       -75.75       -75.25
 -74.75       -74.25       -73.75       -73.25       -72.75       -72.25
 -71.75       -71.25       -70.75       -70.25       -69.75       -69.25
 -68.75       -68.25       -67.75       -67.25       -66.75       -66.25
 -65.75       -65.25       -64.75       -64.25       -63.75       -63.25
 -62.75       -62.25       -61.75       -61.25       -60.75       -60.25
 -59.75       -59.25       -58.75       -58.25000381 -57.75000381
 -57.25000381 -56.75000381 -56.25000381 -55.75000381 -55.25000381
 -54.75000381 -54.25000381 -53.75000381 -53.25000381 -52.75000381
 -52.25000381 -51.75000381 -51.25000381 -50.75000381 -50.25000381
 -49.75000381 -49.25000381 -48.75000381 -48.25000381 -47.75000381
 -47.25000381 -46.75000381 -46.25000381 -45.75000381 -45.25000381
 -44.75000381 -44.25000381 -43.75000381 -43.25000381 -42.75000381
 -42.25000381 -41.75000381 -41.25000381 -40.75000381 -40.25000381
 -39.75000381 -39.25000381 -38.75000381 -38.25000381 -37.75000381
 -37.25000381 -36.75000381 -36.25000381 -35.75000381 -35.25000381
 -34.75000381 -34.25000381 -33.75000381 -33.25000381 -32.75000381
 -32.25000381 -31.75000381 -31.25000381 -30.75000381 -30.25000381
 -29.75000381 -29.25000381 -28.75000381 -28.25000381 -27.75000381
 -27.25000381 -26.75000381 -26.25000381 -25.75000381 -25.25000381
 -24.75000381 -24.25000381 -23.75000381 -23.25000381 -22.75000381
 -22.25000381 -21.75000381 -21.25000381 -20.75000381 -20.25000381
 -19.75000381 -19.25000381 -18.75000381 -18.25000381 -17.75000381
 -17.25000381 -16.75000381 -16.25000381 -15.75000477 -15.25000477
 -14.75000477 -14.25000477 -13.75000477 -13.25000477 -12.75000477
 -12.25000477 -11.75000477 -11.25000477 -10.75000477 -10.25000477
  -9.75000477  -9.25000477  -8.75000477  -8.25000477  -7.75000477
  -7.25000525  -6.75000525  -6.25000525  -5.75000525  -5.25000525
  -4.75000525  -4.25000525  -3.75000525  -3.25000525  -2.75000525
  -2.25000525  -1.75000536  -1.25000536  -0.75000542  -0.25000545
   0.2499945    0.74999446   1.2499944    1.7499944    2.24999428
   2.74999428   3.24999428   3.74999428   4.24999428   4.74999428
   5.24999428   5.74999428   6.24999428   6.74999428   7.24999428
   7.74999428   8.24999428   8.74999428   9.24999428   9.74999428
  10.24999428  10.74999428  11.24999428  11.74999428  12.24999332
  12.74999332  13.24999332  13.74999332  14.24999332  14.74999332
  15.24999332  15.74999332  16.24999428  16.74999428  17.24999428
  17.74999428  18.24999428  18.74999428  19.24999428  19.74999237
  20.24999237  20.74999237  21.24999237  21.74999237  22.24999237
  22.74999237  23.24999237  23.74999237  24.24999237  24.74999237
  25.24999237  25.74999237  26.24999237  26.74999237  27.24999237
  27.74999237  28.24999237  28.74999237  29.24999237  29.74999237
  30.24999237  30.74999237  31.24999237  31.74999237  32.24999237
  32.74999237  33.24999237  33.74999237  34.24999237  34.74999237
  35.24999237  35.74999237  36.24999237  36.74999237  37.24999237
  37.74999237  38.24999237  38.74999237  39.24999237  39.74999237
  40.24999237  40.74999237  41.24999237  41.74999237  42.24999237
  42.74999237  43.24999237  43.74999237  44.24999237  44.74999237
  45.24999237  45.74999237  46.24999237  46.74999237  47.24999237
  47.74999237  48.24999237  48.74999237  49.24999237  49.74999237
  50.24999237  50.74999237  51.24999237  51.74999237  52.24999237
  52.74999237  53.24999237  53.74999237  54.24999237  54.74999237
  55.24999237  55.74999237  56.24999237  56.74999237  57.24999237
  57.74999237  58.24999237  58.74999237  59.24999237  59.74999237
  60.24999237  60.74999237  61.24999237  61.74999237  62.24999237
  62.74999237  63.24999237  63.74999237  64.24999237  64.74999237
  65.24999237  65.74999237  66.24999237  66.74999237  67.24999237
  67.74999237  68.24999237  68.74999237  69.24999237  69.74999237
  70.24999237  70.74999237  71.24999237  71.74999237  72.24999237
  72.74999237  73.24999237  73.74999237  74.24999237  74.74999237
  75.24999237  75.74999237  76.24999237  76.74999237  77.24999237
  77.74999237  78.24999237  78.74999237  79.24999237  79.74999237
  80.24999237  80.74999237  81.24999237  81.74999237  82.24999237
  82.74999237  83.24999237  83.74999237  84.24999237  84.74999237
  85.24999237  85.74999237  86.24999237  86.74999237  87.24999237
  87.74999237  88.24999237  88.74999237  89.24999237  89.74999237]

201006-ESACCI-L3C_CLOUD-CLD_PRODUCTS-AVHRR_METOPA-fv2.0.nc
lat = [-89.75 -89.25 -88.75 -88.25 -87.75 -87.25 -86.75 -86.25 -85.75 -85.25
 -84.75 -84.25 -83.75 -83.25 -82.75 -82.25 -81.75 -81.25 -80.75 -80.25
 -79.75 -79.25 -78.75 -78.25 -77.75 -77.25 -76.75 -76.25 -75.75 -75.25
 -74.75 -74.25 -73.75 -73.25 -72.75 -72.25 -71.75 -71.25 -70.75 -70.25
 -69.75 -69.25 -68.75 -68.25 -67.75 -67.25 -66.75 -66.25 -65.75 -65.25
 -64.75 -64.25 -63.75 -63.25 -62.75 -62.25 -61.75 -61.25 -60.75 -60.25
 -59.75 -59.25 -58.75 -58.25 -57.75 -57.25 -56.75 -56.25 -55.75 -55.25
 -54.75 -54.25 -53.75 -53.25 -52.75 -52.25 -51.75 -51.25 -50.75 -50.25
 -49.75 -49.25 -48.75 -48.25 -47.75 -47.25 -46.75 -46.25 -45.75 -45.25
 -44.75 -44.25 -43.75 -43.25 -42.75 -42.25 -41.75 -41.25 -40.75 -40.25
 -39.75 -39.25 -38.75 -38.25 -37.75 -37.25 -36.75 -36.25 -35.75 -35.25
 -34.75 -34.25 -33.75 -33.25 -32.75 -32.25 -31.75 -31.25 -30.75 -30.25
 -29.75 -29.25 -28.75 -28.25 -27.75 -27.25 -26.75 -26.25 -25.75 -25.25
 -24.75 -24.25 -23.75 -23.25 -22.75 -22.25 -21.75 -21.25 -20.75 -20.25
 -19.75 -19.25 -18.75 -18.25 -17.75 -17.25 -16.75 -16.25 -15.75 -15.25
 -14.75 -14.25 -13.75 -13.25 -12.75 -12.25 -11.75 -11.25 -10.75 -10.25
  -9.75  -9.25  -8.75  -8.25  -7.75  -7.25  -6.75  -6.25  -5.75  -5.25
  -4.75  -4.25  -3.75  -3.25  -2.75  -2.25  -1.75  -1.25  -0.75  -0.25
   0.25   0.75   1.25   1.75   2.25   2.75   3.25   3.75   4.25   4.75
   5.25   5.75   6.25   6.75   7.25   7.75   8.25   8.75   9.25   9.75
  10.25  10.75  11.25  11.75  12.25  12.75  13.25  13.75  14.25  14.75
  15.25  15.75  16.25  16.75  17.25  17.75  18.25  18.75  19.25  19.75
  20.25  20.75  21.25  21.75  22.25  22.75  23.25  23.75  24.25  24.75
  25.25  25.75  26.25  26.75  27.25  27.75  28.25  28.75  29.25  29.75
  30.25  30.75  31.25  31.75  32.25  32.75  33.25  33.75  34.25  34.75
  35.25  35.75  36.25  36.75  37.25  37.75  38.25  38.75  39.25  39.75
  40.25  40.75  41.25  41.75  42.25  42.75  43.25  43.75  44.25  44.75
  45.25  45.75  46.25  46.75  47.25  47.75  48.25  48.75  49.25  49.75
  50.25  50.75  51.25  51.75  52.25  52.75  53.25  53.75  54.25  54.75
  55.25  55.75  56.25  56.75  57.25  57.75  58.25  58.75  59.25  59.75
  60.25  60.75  61.25  61.75  62.25  62.75  63.25  63.75  64.25  64.75
  65.25  65.75  66.25  66.75  67.25  67.75  68.25  68.75  69.25  69.75
  70.25  70.75  71.25  71.75  72.25  72.75  73.25  73.75  74.25  74.75
  75.25  75.75  76.25  76.75  77.25  77.75  78.25  78.75  79.25  79.75
  80.25  80.75  81.25  81.75  82.25  82.75  83.25  83.75  84.25  84.75
  85.25  85.75  86.25  86.75  87.25  87.75  88.25  88.75  89.25  89.75]

When opening these dataset using xarray.open_mfdatset the alignment of the individual datasets results in a combination of the latitude values from both datasets.

[-89.75       -89.25       -88.75       -88.25       -87.75       -87.25
 -86.75       -86.25       -85.75       -85.25       -84.75       -84.25
 -83.75       -83.25       -82.75       -82.25       -81.75       -81.25
 -80.75       -80.25       -79.75       -79.25       -78.75       -78.25
 -77.75       -77.25       -76.75       -76.25       -75.75       -75.25
 -74.75       -74.25       -73.75       -73.25       -72.75       -72.25
 -71.75       -71.25       -70.75       -70.25       -69.75       -69.25
 -68.75       -68.25       -67.75       -67.25       -66.75       -66.25
 -65.75       -65.25       -64.75       -64.25       -63.75       -63.25
 -62.75       -62.25       -61.75       -61.25       -60.75       -60.25
 -59.75       -59.25       -58.75       -58.25000381 -58.25       -57.75000381
 -57.75       -57.25000381 -57.25       -56.75000381 -56.75       -56.25000381
 -56.25       -55.75000381 -55.75       -55.25000381 -55.25       -54.75000381
 -54.75       -54.25000381 -54.25       -53.75000381 -53.75       -53.25000381
 -53.25       -52.75000381 -52.75       -52.25000381 -52.25       -51.75000381
 -51.75       -51.25000381 -51.25       -50.75000381 -50.75       -50.25000381
 -50.25       -49.75000381 -49.75       -49.25000381 -49.25       -48.75000381
 -48.75       -48.25000381 -48.25       -47.75000381 -47.75       -47.25000381
 -47.25       -46.75000381 -46.75       -46.25000381 -46.25       -45.75000381
 -45.75       -45.25000381 -45.25       -44.75000381 -44.75       -44.25000381
 -44.25       -43.75000381 -43.75       -43.25000381 -43.25       -42.75000381
 -42.75       -42.25000381 -42.25       -41.75000381 -41.75       -41.25000381
 -41.25       -40.75000381 -40.75       -40.25000381 -40.25       -39.75000381
 -39.75       -39.25000381 -39.25       -38.75000381 -38.75       -38.25000381
 -38.25       -37.75000381 -37.75       -37.25000381 -37.25       -36.75000381
 -36.75       -36.25000381 -36.25       -35.75000381 -35.75       -35.25000381
 -35.25       -34.75000381 -34.75       -34.25000381 -34.25       -33.75000381
 -33.75       -33.25000381 -33.25       -32.75000381 -32.75       -32.25000381
 -32.25       -31.75000381 -31.75       -31.25000381 -31.25       -30.75000381
 -30.75       -30.25000381 -30.25       -29.75000381 -29.75       -29.25000381
 -29.25       -28.75000381 -28.75       -28.25000381 -28.25       -27.75000381
 -27.75       -27.25000381 -27.25       -26.75000381 -26.75       -26.25000381
 -26.25       -25.75000381 -25.75       -25.25000381 -25.25       -24.75000381
 -24.75       -24.25000381 -24.25       -23.75000381 -23.75       -23.25000381
 -23.25       -22.75000381 -22.75       -22.25000381 -22.25       -21.75000381
 -21.75       -21.25000381 -21.25       -20.75000381 -20.75       -20.25000381
 -20.25       -19.75000381 -19.75       -19.25000381 -19.25       -18.75000381
 -18.75       -18.25000381 -18.25       -17.75000381 -17.75       -17.25000381
 -17.25       -16.75000381 -16.75       -16.25000381 -16.25       -15.75000477
 -15.75       -15.25000477 -15.25       -14.75000477 -14.75       -14.25000477
 -14.25       -13.75000477 -13.75       -13.25000477 -13.25       -12.75000477
 -12.75       -12.25000477 -12.25       -11.75000477 -11.75       -11.25000477
 -11.25       -10.75000477 -10.75       -10.25000477 -10.25        -9.75000477
  -9.75        -9.25000477  -9.25        -8.75000477  -8.75        -8.25000477
  -8.25        -7.75000477  -7.75        -7.25000525  -7.25        -6.75000525
  -6.75        -6.25000525  -6.25        -5.75000525  -5.75        -5.25000525
  -5.25        -4.75000525  -4.75        -4.25000525  -4.25        -3.75000525
  -3.75        -3.25000525  -3.25        -2.75000525  -2.75        -2.25000525
  -2.25        -1.75000536  -1.75        -1.25000536  -1.25        -0.75000542
  -0.75        -0.25000545  -0.25         0.2499945    0.25         0.74999446
   0.75         1.2499944    1.25         1.7499944    1.75         2.24999428
   2.25         2.74999428   2.75         3.24999428   3.25         3.74999428
   3.75         4.24999428   4.25         4.74999428   4.75         5.24999428
   5.25         5.74999428   5.75         6.24999428   6.25         6.74999428
   6.75         7.24999428   7.25         7.74999428   7.75         8.24999428
   8.25         8.74999428   8.75         9.24999428   9.25         9.74999428
   9.75        10.24999428  10.25        10.74999428  10.75        11.24999428
  11.25        11.74999428  11.75        12.24999332  12.25        12.74999332
  12.75        13.24999332  13.25        13.74999332  13.75        14.24999332
  14.25        14.74999332  14.75        15.24999332  15.25        15.74999332
  15.75        16.24999428  16.25        16.74999428  16.75        17.24999428
  17.25        17.74999428  17.75        18.24999428  18.25        18.74999428
  18.75        19.24999428  19.25        19.74999237  19.75        20.24999237
  20.25        20.74999237  20.75        21.24999237  21.25        21.74999237
  21.75        22.24999237  22.25        22.74999237  22.75        23.24999237
  23.25        23.74999237  23.75        24.24999237  24.25        24.74999237
  24.75        25.24999237  25.25        25.74999237  25.75        26.24999237
  26.25        26.74999237  26.75        27.24999237  27.25        27.74999237
  27.75        28.24999237  28.25        28.74999237  28.75        29.24999237
  29.25        29.74999237  29.75        30.24999237  30.25        30.74999237
  30.75        31.24999237  31.25        31.74999237  31.75        32.24999237
  32.25        32.74999237  32.75        33.24999237  33.25        33.74999237
  33.75        34.24999237  34.25        34.74999237  34.75        35.24999237
  35.25        35.74999237  35.75        36.24999237  36.25        36.74999237
  36.75        37.24999237  37.25        37.74999237  37.75        38.24999237
  38.25        38.74999237  38.75        39.24999237  39.25        39.74999237
  39.75        40.24999237  40.25        40.74999237  40.75        41.24999237
  41.25        41.74999237  41.75        42.24999237  42.25        42.74999237
  42.75        43.24999237  43.25        43.74999237  43.75        44.24999237
  44.25        44.74999237  44.75        45.24999237  45.25        45.74999237
  45.75        46.24999237  46.25        46.74999237  46.75        47.24999237
  47.25        47.74999237  47.75        48.24999237  48.25        48.74999237
  48.75        49.24999237  49.25        49.74999237  49.75        50.24999237
  50.25        50.74999237  50.75        51.24999237  51.25        51.74999237
  51.75        52.24999237  52.25        52.74999237  52.75        53.24999237
  53.25        53.74999237  53.75        54.24999237  54.25        54.74999237
  54.75        55.24999237  55.25        55.74999237  55.75        56.24999237
  56.25        56.74999237  56.75        57.24999237  57.25        57.74999237
  57.75        58.24999237  58.25        58.74999237  58.75        59.24999237
  59.25        59.74999237  59.75        60.24999237  60.25        60.74999237
  60.75        61.24999237  61.25        61.74999237  61.75        62.24999237
  62.25        62.74999237  62.75        63.24999237  63.25        63.74999237
  63.75        64.24999237  64.25        64.74999237  64.75        65.24999237
  65.25        65.74999237  65.75        66.24999237  66.25        66.74999237
  66.75        67.24999237  67.25        67.74999237  67.75        68.24999237
  68.25        68.74999237  68.75        69.24999237  69.25        69.74999237
  69.75        70.24999237  70.25        70.74999237  70.75        71.24999237
  71.25        71.74999237  71.75        72.24999237  72.25        72.74999237
  72.75        73.24999237  73.25        73.74999237  73.75        74.24999237
  74.25        74.74999237  74.75        75.24999237  75.25        75.74999237
  75.75        76.24999237  76.25        76.74999237  76.75        77.24999237
  77.25        77.74999237  77.75        78.24999237  78.25        78.74999237
  78.75        79.24999237  79.25        79.74999237  79.75        80.24999237
  80.25        80.74999237  80.75        81.24999237  81.25        81.74999237
  81.75        82.24999237  82.25        82.74999237  82.75        83.24999237
  83.25        83.74999237  83.75        84.24999237  84.25        84.74999237
  84.75        85.24999237  85.25        85.74999237  85.75        86.24999237
  86.25        86.74999237  86.75        87.24999237  87.25        87.74999237
  87.75        88.24999237  88.25        88.74999237  88.75        89.24999237
  89.25        89.74999237  89.75      ]

Checking the values using ncdump confirmed them.

@mzuehlke
Copy link
Collaborator

mzuehlke commented Jul 7, 2017

After some checking it is not simply possible to use an epsilon when doing the equal check of two different coordinate indices.

@mzuehlke mzuehlke mentioned this issue Jul 7, 2017
Closed
@mzuehlke
Copy link
Collaborator

mzuehlke commented Jul 7, 2017

This is an issue of the data set: https://github.com/CCI-Tools/cate-core/wiki/Problems-with-ODP-datasets

@mzuehlke mzuehlke closed this as completed Jul 7, 2017
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@mzuehlke mzuehlke

Labels
bug ds wontfix
Projects
None yet
Milestone
IPM6
Development

No branches or pull requests
2 participants
@mzuehlke @forman