Reduce Output in Atmos Tests

[Cerebrovinny]

what

• Introduced verbosity levels. Allows customizable detail.
• Updated GitHub Actions: Modified test execution. Added verbosity control.
• Improved Logging: Validated logs-level flag. Ensures correct log levels.
• Updated README: Documented test verbosity. Guides on usage.

why

• Reduces redundant output.
• CI/CD: Provides clearer feedback

references

[mergify]

Important

Cloud Posse Engineering Team Review Required

This pull request modifies files that require Cloud Posse's review. Please be patient, and a core maintainer will review your changes.

To expedite this process, reach out to us on Slack in the #pr-reviews channel.

[mergify]

Important

Cloud Posse Engineering Team Review Required

This pull request modifies files that require Cloud Posse's review. Please be patient, and a core maintainer will review your changes.

To expedite this process, reach out to us on Slack in the #pr-reviews channel.

[aknysh]

@Cerebrovinny please resolve the conflicts
Does this PR have unresolved conversations or comments that you think need to be addressed?

[mergify]

💥 This pull request now has conflicts. Could you fix it @Cerebrovinny? 🙏

[osterman]

@Cerebrovinny I’ve been reflecting on this, and I think my ideal implementation would suppress all logs unless there’s a failure. In the case of a failure, it would then emit the full log. Is that how it works?

[Cerebrovinny]

@Cerebrovinny I’ve been reflecting on this, and I think my ideal implementation would suppress all logs unless there’s a failure. In the case of a failure, it would then emit the full log. Is that how it works?

The quiet mode shows minimal output and only failures
But it's not buffering logs either - it's just filtering what gets printed

see here
https://github.com/cloudposse/atmos/actions/runs/12897213551/job/35962047152?pr=923#step:11:1

[osterman]

@Cerebrovinny I’ve been reflecting on this, and I think my ideal implementation would suppress all logs unless there’s a failure. In the case of a failure, it would then emit the full log. Is that how it works?

The quiet mode shows minimal output and only failures But it's not buffering logs either - it's just filtering what gets printed

see here https://github.com/cloudposse/atmos/actions/runs/12897213551/job/35962047152?pr=923#step:11:1

Can you show me what a failure looks like? Is it the same, but just says failed? If so, that won't fly.

Would it be possible to buffer the logs? I don't think we can use quiet mode in CI, b/c it will make supporting open source contributions too cumbersome if the maintainers have to check out the branch, just to see what the failure/error was.

[osterman]

Is testing a supported section in the README.md.tmpl?

[mergify]

💥 This pull request now has conflicts. Could you fix it @Cerebrovinny? 🙏

[osterman]

@Cerebrovinny I would prefer a different approach that doesn’t involve wrapping existing functions and introducing a new lexicon for testing.

Ideal requirements:

• non need to wrap individual functions with feature flags to silence output
• just work for anything that produces output.
• buffer output and only emit when there’s any error are errors
• Feature flags (env) or argument, that disables buffering, but it should default to enabled

Pseudo code (gpt):

package mypackage

import (
	"bytes"
	"fmt"
	"os"
	"testing"
)

func captureOutput(fn func()) (string, error) {
	// Save the original os.Stdout and os.Stderr
	origStdout := os.Stdout
	origStderr := os.Stderr

	// Create pipes to capture output
	rOut, wOut, err := os.Pipe()
	if err != nil {
		return "", err
	}
	rErr, wErr, err := os.Pipe()
	if err != nil {
		return "", err
	}

	// Redirect os.Stdout and os.Stderr to the pipes
	os.Stdout = wOut
	os.Stderr = wErr

	// Create a channel to capture the output
	outputChan := make(chan string)
	go func() {
		var buf bytes.Buffer
		buf.ReadFrom(rOut)
		buf.ReadFrom(rErr)
		outputChan <- buf.String()
	}()

	// Run the provided function
	fn()

	// Restore the original os.Stdout and os.Stderr
	wOut.Close()
	wErr.Close()
	os.Stdout = origStdout
	os.Stderr = origStderr

	// Get the captured output
	output := <-outputChan
	return output, nil
}

func TestExample(t *testing.T) {
	// Wrap the test logic in the output capture
	output, err := captureOutput(func() {
		// Simulate commands or deeply nested code that outputs
		fmt.Println("This is stdout")
		fmt.Fprintln(os.Stderr, "This is stderr")
		fmt.Println("Another stdout message")
	})
	if err != nil {
		t.Fatalf("Failed to capture output: %v", err)
	}

	// Example assertion
	if false { // Simulate a test failure condition
		t.Errorf("Test failed. Output:\n%s", output)
	}
}

[osterman]

I think there's a misunderstanding how the README.yaml works. We run this though gomplate and apply this template: https://github.com/cloudposse/.github/blob/main/README.md.gotmpl

There's no section we look for called content or testing.

Also, after updating the README.yaml, run make readme to rebuild it.

Note, we're going to do this with atmos soon, see:

• Support atmos docs generate feature natively #934

And the confusion here, shows we should have a JSON schema.

[osterman]

We also need to capture stderr