There's this fascinating problem that you hit when you try to build out high-velocity code-generating AI. It revolves around complexity metrics, but it's outside the scope of the complexity tracking that I've designed so far. See, while I'm accounting for the module and file complexity of an app, as the AI implements it... I haven't been accounting for the overall cognitive score for the app, which I do have the numbers for. And that's interesting.
When complexity melts continuity
Too much unchecked complexity within the entire app itself will also lead to brain melt, even in situations where you're responsible, use dependency injection, and build out your modules properly.
But, see, there's another problem in this ball of things to be aware of. The complexity score threshold for cognitive melt, which involves hallucinated refactoring, isn't especially high, and happens in apps that humans would consider relatively small code bases of about a dozen fairly complex application files.
On the surface, it would seem this is a showstopper.
Go too far, by holding continuity across too many sessions, and you nuke your app, because you're limited by the limitations of current LLM technology, right?
Wrong!
Flow tests as architectural anchors
My solution is a new spin on flow testing. Flow testing is a type of unit test for integration testing that focuses on data flow across complex systems. It's about mapping the pathways from input to output of the entire app, and figuring out what it takes to go from point A to point B in terms of core functionality.
Typical unit tests only track the input and output of specific functions, modules, and classes. Write a bad test, get bad results. But, this method reinforces the design of your entire app from start to finish at the moment of inception. Locking you into good design principles, and a transparent, cognitively straightforward process.
Map the flow, then enforce it
Here's my approach to it: at the very start and at every feature milestone, I mock out the application's entire data flow. I outline the key parts of the core API and map out how data moves from one side of the decision tree to the other.
With flow tests, you can test the entire app, checking how data flows across the whole system no matter how complex it gets. They show you what's working and what's broken. There's no such thing as partial coverage here—partial coverage means the app is broken.
I treat these flow tests as immutable truths that can't be changed during a coding cycle. And by tying them into Git pre-commit hooks, you simply can't commit if a flow test fails.
I think it's elegant.
Isn't NLP interesting?