Kybra Modules Init and Stdlib

[lastmjs]

• build the intermediate canister, consider how we can reuse it in Azle
• remove ic-cdk-optimizer (I might be able to remove this and just use ic-wasm)
• remove ic-wasm (actually I'm hoping to keep this and that it works with wasm32-wasi)
• create python -m kybra deploy command
• upgrade to dfx 0.14.0 and test the no artificial delay
• Remove everything that is no longer needed
• Update all documentation
  • Explain the caveats of subnets and stuff for deployment to production
• Look into using temporary files and directories, this could clean up a lot of the junk that we have laying around
• Make sure to deal with the location of the optimized wasm binaries in .dfx/local, I wonder if it changes based on the network being deployed to or something...maybe it's just the network configuration name
• Add the metadata back in using the link section method
• Azle might not even need optimization in many cases...seems gzipping might be enough? Perhaps we should just instruct the user to use optimize in dfx.json only if they need to
• Use post-install for Kybra dynamic module uploading
• Keep in mind that with Kybra, the binary hash will basically be useless, as much of the actual code/functionality will change and not be in the binary. Perhaps we could send the Python source code up in the binary to help with this?
  • We could probably hash all source code and include the hash in the binary though
• It would be nice to figure out how to use temp files more, but I couldn't get it to work for some reason...maybe flushing?
• For Azle consider detecting if the binary is too big, and only then downloading and installing ic-cdk-optimizer and using it with gzip
• We must differentiate between init and post upgrade so that we can call the appropriate function after uploading everything...I am just not sure how we can detect that
• We might need to explain in the Kybra caveats section that object unpacking doesn't work
• We need to explain in the docs that object unpacking shouldn't be used: Dictionary unpacking not working? RustPython/RustPython#4932
  • ethereum_json_rpc and outgoing_http_requests might have these problems
• Get these instructions out: https://forum.dfinity.org/t/kybra-python-std-lib/19632/12
• Explain in stable structures section that memory id 255 is taken
• we need params on init and post upgrade
• We need permissions on init and post upgrade
• This means the sender and receiver type has diverged, and can cause data loss. this is happening in Kybra, I think it was fixed in Azle?
• Test new hooks API for Boa: Fix Date for dynamic timezones boa-dev/boa#2877
• Make an MRE for this: Dictionary unpacking not working? RustPython/RustPython#4932
• Make Python Wasm binary instructions for DFINITY
• In the documentation explain that memory id 254 is taken
• I removed guard functions entirely from init and post_upgrade, is that okay?
• Should we add a --locked option to our cargo commands?
• probably call the timer in init and post_upgrade still
• full stdlib
• remove the custom_modules
• Make sure we are complying with the PSF license...are we a derivative work of Python? Not sure, we might just have to summarize the changes we've made
• document in license that I created a custom _signal.py
• get wasi randomness to work better, for example uuid will always return the same uuids
• get rid of custom_modules entirely, depend on ic-py directly
• Python compilation needs to end if there's an error, that isn't the case right now
• document that we only have 8 bytes of randomness instead of 32 temporarily
• Consider if we should get Azle to compile to Wasi now
• Figure out how to get the final optimizations that we need, I think we might need to use the intermediate canister method
• We should be able to build the Python binary only one time
• Make it so that only the python source directory has to recompile, but the stdlib only compiles once
• change initial installation messages
• make sure to do inline name, cdk name, keywords, context etc correctly with the kybra modules init
• consider setting all of the init ref cells to none once we are done with them, they'll just sit there taking up memory for no reason
• verbose should override the kybra_modules_init log file
• Make sure using init params also in post_upgrade is safe
• inspect_message should reject nicely on any other other method call but the kbyrA_module_init methods, it should reject nicely on those other methods
• Consider an inspect_message that blocks everything unless the kybra_modules_init is finished
• make sure that guard functions run in the same update call as their guarded functions
• we need to figure out how to keep our stdlib files up to date
• The ic polyfill seems to be introducing some overhead
  • It it just the printing? Why is it calling these things so much?
• consider storing the stdlib in stable memory so that it only has to be uploaded one time
• maybe we should rename kybra_modules_init to kybra_post_install
• make sure the production work-flow will work as well, we might need to figure something out to understand --network ic and which canister to chunk upload to...
• remove all .gz paths in dfx.json as we don't need them for the deployer canister, and soon dfx will automatically gzip and won't expect the .gz extension
• Kybra build process
  • Deploy the deployer canister only once
  • Build the stdlib only once
  • Chunk upload the Python binary, compile the Python binary in the VM
  • Chunk upload the stdlib bytecode
  • Call install code
  • Use post_upgrade just as it was
  • Use init just as it was if possible
  • Use get settings from management canister to check for controller only
  • update the ic-wasi-polyfill to the latest
  • update to the latest polyfills and candid...right now we have a pretty bad dependency resolution issue, and I don't know how to fix it. Perhaps manually editing the Cargo.lock will do it
• Add in good error handling if it exists (if the infrastructure exists)
• Seems like the kybra module in python_source is taking a very long time to freeze...I wonder if it will be relatively simple to fix this by only including the actual python code necessary
• gzip the final binary being uploaded, that will save a few chunks
• don't forget DFX_NETWORK for when we are deploying live, we will have to change the commands in post_install
• what happens if init fails in the deployer? if the user tries again, we need the init to always be init and the post_upgrade to always be post_upgrade
• add kybra_rebuild to all test package.json's
• init and post_upgrade arguments
• document that the basic dfx deploy flow is what works, if you want to do anything else you'll have to manually figure it out
• why does post_upgrade not have #[candid::candid_method(init)]
• remove all unnecessary code, for example inspect_message and guard function stuff
• we should add an error check in the compiler to ensure that the init and post_upgrade params are the same. If they are both specified, then they need to be the same.
• Memory ids 0, 1, 2, 233, 234 are all taken
• watch out for this error: thread 'main' panicked at 'Error: "\u{1b}[31mError: \u{1b}(B\u{1b}[mFailed update call.\n\u{1b}[33mCaused by: \u{1b}(B\u{1b}[mFailed update call.\n The Replica returned an error: code 5, message: \"Canister rrkah-fqaaa-aaaaa-aaaaq-cai violated contract: RequestInPrep: current_size 12 exceeded the allowed limit local-subnet 10485760 remote-subnet 2097152\"\n"', src/main.rs:172:9 note: run with RUST_BACKTRACE=1environment variable to display a backtrace
• The installer of a Kybra canister will now always be the canister itself, you will not know the actual identity that started the installation process...is that a problem?
  • If it is a problem we can create an installer command, and in the deployer canister we can store the installer in stable memory and allow it be retrieved through ic.installer() or something like that
  • randomness is broken, I cannot initialize the seed again it seems...add some logs in the cdk_framework to try and track this down
  • what if we called the random seed callback before the init callback? Then we could make sure the randomness was live before init was called
  • fix installer guard to use is_controller
• figure out how to copy RustPython Lib over locally
• breaking change...memory ids 0, 1, 2
• document all contributor stuff
• really make sure to document installer issue
• comment in the whoami example about installer differences
• don't use notify, expect the error if it let's us get rid of the extra-post-install wait
• declarify Python wasm binary builder function
• Don't use fully-qualified names in deployer canister
• don't use unwrap
• Declarify and Resultify everything
• update the date example and try to use datetime and get rid of the rustpython custom datetime stuff
• add tests for stdlib, maybe add test for azle/kybra imports
• see if we can remove time and randomness hacks in Kybra
• fix cdk_framework with conditional compile for wasm32 and wasi
• Make sure kybra_post_install doesn't swallow any errors
• output chunks and the number of the chunks
• I think the post_install process should actually always output I'm guessing, at least for the stdlib part. It does take a while
• rename initialized_map
• switch kybra/bin/version to kybra/version/bin so as to make room for other directories in the future
• let's make constants for the memory ids that we are using
• filter out non .py files in the Python bundler
• consider if kybra_generate should also only be compiled once, just like Azle. This should save a good 5-6 seconds
• Rust directories should be in their own directories
• kybra post_install should be compiled one time probably
• look into if the build.rs script is running or not for real, we need to know if it runs every time or if it doesn't, because that could affect stuff.
• consider removing the canister as controller if it wasn't already there
• test all Kybra post install error handlers
• the Lib directory showed up in kybra_post_install for some reason
• This means the sender and receiver type has diverged, and can cause data loss.
• ic-wasi-polyfill #403
• Try updating the ic wasi polyfill and see if the tests all pass after that
• figure out could not convert value to vec, vec etc errrors, this has happened in the tests in the ledger_canister now and it has happened locally to me with outgoing_http_requests...it seems intermittent which is scary
• Lots of scary intermittent errors in the tests
  • make sure post_upgrade stdlib initialization is completely every time
  • make sure we aren't swallowing any errors during vm initialization
  • perhaps our unsafe code is breaking down, asynchronous stuff not working, race conditions, heap being overwritten, etc
  • Why 18 chunks for python stdlib sometimes?
• Fix ledger canister wasm url: https://forum.dfinity.org/t/access-denied-on-ledger-canister-wasm/20920
• make sure ic-wasi-polyfill can log
• make sure that ic_cdk println! works still
• See if canister_methods_tuple should be returned to its former state.
• Horrible intermittent errors #409
• Code and tests #406
• Declarify code #404
• Resultify #405
• check heartbeat guard test?
• Remove python stdlib from the git history
• get rid of the .gitgnore for lib if you can remove it from the repository and the rust analyzer thing is fixed too
• Not calling ic.acceptMessage seems to be throwing different errors now, check the inspect_message tests as well
• Look into the guard functions example tests unallowedMethod, it is now broken. I wonder if it is broken in Azle on df 0.14.2-beta.2 as well
• Increased Wasm binary limit #414
• Is there some way to use automatic gzipping and the optimization built into dfx so that we can get rid of ic-wasm entirely?
• how should we handle errors in init and post_upgrade now? We don't get the errors returned to us. Check that init and post_upgrade can log and have panics
  • ic0 -1 callback (still doesn't give you errors)
  • instead of using post_upgrade, use a custom function to initialize
  • Rust index hack: Rust Wasm function table hack #419
  • what if we don't await
• Test how preupgrade works
• Do a full unwrap_or_trap and add logging...maybe we just keep around a fixed amount of errors so there is no memory leak...make sure post_upgrade has this everywhere, then simply make a function to check these errors after deploy
• Use the is_controller check everywhere
• The logging in unwrap_or_trap should be very useful going forward in production
• Review code one more time
• Test init, post_upgrade, and pre_upgrade one more time
• Write documentation: 0.5.0 #384
• Tell people that to upgrade to Kybra 0.5.0 they should delete ~/.config/kybra if it exists
• Should we wait for this to be resolved? https://forum.dfinity.org/t/not-calling-accept-message-in-inspect-message-not-rejecting-immediately-in-dfx-0-14-2-beta-2/21105
• Think about randomness, let's also do the settimout randomness thing just in case??
  • detect if randomness is empty in post_upgrade and fail if so
• I'm running into a double Lib thing locally again, where somehow the Lib directory is copied into the Lib directory