Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improper Memoization with Blink Table attributes #5209

Closed
abaranec opened this issue Mar 1, 2024 · 0 comments · Fixed by #5216
Closed

Improper Memoization with Blink Table attributes #5209

abaranec opened this issue Mar 1, 2024 · 0 comments · Fixed by #5216
Assignees
Labels
bug Something isn't working core Core development tasks query engine

Comments

@abaranec
Copy link
Contributor

abaranec commented Mar 1, 2024

Description
When BlinkTable attribute is set memoization causes surprising results with certain operations

Steps to reproduce
Run the following query (You may need book builder, ask andy for it)

Symbols = ["SPY", "PFF", "FB", "DIS", "USO", "XLF", "XLU", "REM", "REZ", "GME", "HDSN"]
Exchanges = ["NYSE", "INET", "EDGX", "BATS", "EDGA" , "MEMX", "BATY", "XOTC", "OTCU"]
Traders = ["John", "Mark", "Paul", "Bill", "Steve", "Brian", "Kevin", "Miranda", "Shirley", "Diana", "Grettle", "Clarissa"]

priceMap = ["PFF":37.50, "SPY":395.00, "USO":40.00, "XLF":31.50, "XLU":62.50, "REM":33.33, "REZ":71.33, "HDSN":1.85, "FB":270.00, "DIS":190.00, "GME":17.50]
pricerFunc = { final String USym, final double E ->;
    def _aPrice = priceMap[USym]
    if (_aPrice == null) {
        priceMap[USym] = abs((int)E)/100.0
    } else {
        priceMap[USym] = ((int)(_aPrice*100) + (int)(E/2000))/100.0
    }
    return (double)priceMap[USym]
}

distributerFunc = { final int cnt ->;
    def n = new Random().nextInt((int)((cnt * (cnt + 1)) / 2)) + 1
    for (def i = 0; i < cnt; i++) {
        n -= (cnt-i)
        if (n <= 0) {
            return (int)i;
        }
    }
}

TickingTable = timeTable("PT0.01S")
        .update("Date=`2023-03-14`",
                "A=new Random().nextInt(10)",
                "B=new Random().nextInt(2)",
                "C=new Random().nextInt(50)",
                "D=(new Random().nextDouble()-.5) * 20000.0",
                "USym=Symbols[(int)distributerFunc.call(Symbols.size())].toString()",
                "Size=Math.max((new Random().nextInt(11))*100, (new Random().nextInt(149))+1)",
                "Price=(double)pricerFunc.call(USym, D)",
                "Side= (int)(ii % 2)",
                "Op=0",
                "Exchange=Exchanges[C%Exchanges.size()].toString()",
                "Trader=Traders[C % Traders.size()].toString()")

import io.deephaven.book.PriceBook
import io.deephaven.engine.table.impl.BlinkTableTools
import io.deephaven.engine.table.impl.sources.ring.RingTableTools
book = PriceBook.build(TickingTable, 4, false, "Timestamp", "Size", "Side", "Op", "Price", "USym")
// latestby = book.removeBlink().partitionBy("Key").transform(t -> RingTableTools.of(t, 1)).merge();
latestby1 = book.removeBlink().lastBy("Key");
latestby2 = book.lastBy("Key");


println System.

Expected results

latestby2 should be a proper lastBy and rows should never go away

Actual results

rows go away

Versions

  • Deephaven: 0.30.4
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working core Core development tasks query engine
Projects
None yet
3 participants