Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

integrate libcon4m #269

Open
ee7 opened this issue Apr 11, 2024 · 0 comments
Open

integrate libcon4m #269

ee7 opened this issue Apr 11, 2024 · 0 comments
Assignees
@ee7

Comments

@ee7
Copy link
Contributor

ee7 commented Apr 11, 2024 •
edited
Loading

con4m was rewritten, most importantly to:

  • Decrease the chalk startup time
  • Supported multi-threaded attribute access

Current flow

Currently, building chalk execs the con4m executable to generate c4autoconf.nim for Nim-native data structures and getter procs:

chalk/chalk.nimble

Lines 54 to 60 in 4157b58

if [ ! ${OUTFILE} -nt ${SPEC} ] ; then
echo Config schema changed. Regenerating c4autoconf.nim.
con4m gen ${SPEC} --language=nim --output-file=${OUTFILE}
else
echo No change to chalk.c42spec
fi
""")

Then running chalk:

  1. Runs con4m code to load and validate the chalk config (which doesn't change)
  2. Uses that environment to load and validate the built-in configs
  3. Uses that environment to load and validate any user-supplied config

This also involves flushing the con4m attribute state to the Nim-native data structures a few times, so it stays synced.

Future flow

libcon4m allows chalk to save and restore runtime state by embedding an object file into the chalk binary, such that chalk can perform only the necessary work that would change per-invocation (like user-supplied config). This should reduce complexity overall in configuration loading - there's no more stacking.

That is, with libcon4m, building chalk should:

  1. Run the chalk configuration, generating an object file
  2. Embed that object file into chalk

Then running chalk should:

  1. Restore the state from the object file
  2. Partially process the command-line to find external configs
  3. Process those external configs
  4. Perform the requested operations

Eventually we may want to allow pre-compiling user-configs for later runs, but that introduces significant complexity for much less important gains.

There are also breaking changes in the language.

Refs: #26
Refs: #214
@ee7 ee7 self-assigned this Apr 11, 2024
@ee7 ee7 mentioned this issue Jun 13, 2024
Merged
4 tasks
@miki725 miki725 changed the title integrate con4m v2 integrate libcon4m Jun 20, 2024
ee7 added a commit that referenced this issue Jul 24, 2024
@ee7
refactor: reduce AttrScope usage, toward fully-dotted keys (#337)
6ce7679 
Continue to prepare for the new con4m, by refactoring so that we get/set
everything using the top-level chalk scope with fully-dotted keys. That
helps prepare for the new con4m lookup interface, which in con4m v2 was:

    proc lookup*[T](ctx: RuntimeState, key: string): Option[T] =

where `key` is fully-dotted. And similarly in libcon4m:

    extern c4m_value_t *
    c4m_vm_attr_get(c4m_vmthread_t *tstate,
                    c4m_str_t      *key,
                    bool           *found);

Previously we were passing subscopes around, but `AttrScope` doesn't
exist in the new con4m (where there's essentially one big dictionary of
fully-dotted keys).

We will refactor this further later.

With this commit, the only remaining `AttrScope`s are:

    $ git grep --ignore-case --break --heading AttrScope
    src/run_management.nim
    20:proc getChalkScope*(): AttrScope =
    27:    if v.isA(AttrScope):
    30:proc sectionExists*(scope: AttrScope, s: string): bool =
    41:proc con4mAttrSet*(attrs: AttrScope, fqn: string, value: Box, attrType: Con4mType) =
    
    src/sinks.nim
    177:proc getAuthConfigByName*(name: string, attr: AttrScope = AttrScope(nil)): Option[AuthConfig] =

Refs: #269
@ee7 ee7 mentioned this issue Aug 5, 2024
Merged
4 tasks
ee7 added a commit that referenced this issue Aug 19, 2024
@ee7
refactor: remove most getChalkScope (#395)
7e35a57 
Previous preparation for the new con4m culminated in nearly every get
and set occurring from the top level [1]. Refactor so that helper procs
always get and set from the top level, so that we generally no longer
write `getChalkScope()` when getting or setting.

Before this commit, there was a lot of `getChalkScope()` noise:

    $ git grep --ignore-case 'getChalkScope' -- '*.nim' | wc -l
    266

With this commit, there's much less:

    $ git grep --ignore-case 'getChalkScope' -- '*.nim' | wc -l
    14
    $ git grep --break --heading --ignore-case 'getChalkScope' -- '*.nim'
    src/chalk_common.nim
    484:    if getChalkScope() != nil and attrGet[bool]("chalk_debug"):
    
    src/confload.nim
    109:  if tb != "" and getChalkScope() == nil or attrGet[bool]("chalk_debug"):
    
    src/run_management.nim
    20:proc getChalkScope*(): AttrScope =
    24:  getChalkScope().getObjectOpt(s).isSome()
    27:  get[T](getChalkScope(), fqn)
    30:  getOpt[T](getChalkScope(), fqn)
    33:  getObject(getChalkScope(), fqn)
    55:  doAssert attrSet(getChalkScope(), fqn, value, attrType).code == errOk
    58:  discard attrLookup(getChalkScope(), fqn.split('.'), ix = 0, op = vlSecDef)
    
    src/selfextract.nim
    153:  if getChalkScope() != nil and attrGet[bool]("chalk_debug"):
    
    src/sinks.nim
    47:    if (llStr in toLogLevelMap) and getChalkScope() != nil and
    153:  if getChalkScope() != nil and attrGet[bool]("chalk_debug"):
    185:    attrRoot = if attr != nil: attr else: getChalkScope()
    273:      discard setOverride(getChalkScope(), section & ".pinned_cert_file", some(pack(path)))

This commit squashes the below refactors:

- Replace top-level get with attrGet.

- Replace top-level getOpt with attrGetOpt.

- Replace top-level getObject with attrGetObject.

- run_management: make getChalkSubsections use attrGetObject.

- conffile: replace a top-level attrLookup with attrGetOpt.

- Make sectionExists work from top level. And for now, inline the use
  of sectionExists in sinks.nim that wasn't necessarily from the
  top level.

- Make con4mAttrSet work from top level.

- config, run_management: add a con4mSectionCreate helper proc.

Refs: #269

[1] 6ce7679
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ee7 ee7

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@ee7