Notebook

Despite the fact that most of my actual research interest in AI is focused on LLMs, as an end user I use image models at least as often. I have an insatiable need for images for my TTRPG worldbuilding hobby, and tools like Midjourney and Ideogram have been more than happy to take my money and enable me.

It was through that frequent use of image models that I first started thinking there might be some relationship between generative AI and certain cognitive processes. I started to get a feel for the types of mistakes different models made, and I started to notice that several common types of mistakes lined up with visual distortions I recognized from my own dreams.  In particular there were parallels between the way clocks and text tend to distort in my dreams and the distortions that often appear in images generated by Midjourney and similar diffusion-based systems.

I also thought a bit about just how differently I approach creating images when I’m conscious vs. when I’m dreaming. In my waking mind, I start with structure. If I am attempting to draw a clock, first I will draw a circle, then place numbers, etc. I’m far more focused on named shapes and functionality. My dreams, by contrast, seem to start with a sort of abstract concept of ‘clock’ and then meander, often only hinting at visual details. Text not directly looked at often melts in my dreams in a way that reminds me quite a bit of the garbled letters in diffusion-generated text. I noticed this sort of melting/fading effect long before diffusion models were a thing, and seeing them echoed visually was kind of eerie. I started to really wonder just how much that association would hold up if I started to look into it.

The more I thought about it, the more I realized that this kind of associative generation was not limited to dreams and images. I do a version of it while speaking too quickly or when tired. I can reach for a word and find the wrong one, misremember a fact, or start talking before my mind has fully caught up to what I am trying to say. Most of the time my waking mind catches and corrects these errors quickly enough that they do not become very noticeable. But the underlying tendency is still there as this sort of almost automatic speaking mechanism which is part of me.

Now, to be frank, as cool as I find image generation, I am not brave enough to wade too deep academically into those waters just yet. I am not completely new to it. I have populated algorithms in what could charitably be called “writing” a simple diffusion model, and I have trained that model on pre-provided data to pretend, poorly, to generate handwritten numbers. I have read the latent diffusion paper, thought it was cool, and would even say I somewhat understand the architecture. But that is just about the sum total of my understanding of image generation.

So for me, looking into this question through LLMs made more sense. I feel more academically comfortable with language models, and this was already after my hallucinating transformer model and a fair amount of other reading on LLM accuracy and reliability. I felt I already knew enough of the foundational research to give myself a good head start. Based on that, and my general experience, my guess was that I would find some similar patterns between the visual errors matching my dream’s errors and the language model matching my errors of casual speech.

To further support my internal argument that I could move this insight from images to text I reasoned that structurally there are also some broad similarities between modern diffusion models and LLMs.  Both work by translating human-facing inputs into learned internal representations: image models work with latent representations of images, while LLMs work with token embeddings that represent words or word-pieces as vectors. The architectures are very different, but both are operating in learned representational spaces rather than directly manipulating meaning in the way a person consciously would. Both are also trained, in broad terms, to learn patterns from existing “good” outputs and generate new outputs that are plausible relative to those learned expectations. I do not want to flatten the differences too much, but the overlap was enough to make the comparison feel meaningful.

So this was nothing but an intuition, but the alignment of the errors between the less conscious parts of my mind (dreams, careless speech) and the generative model’s output seemed interesting enough to be worth looking into, and this is when I began to engage a bit more of an academic lens.

One of the hypotheses I generally find persuasive in cognitive science is mental modularity, at least in its more restrained original form by Jerry Fodor. This is the idea that parts of the mind may operate as somewhat specialized systems for tasks like vision and language. I am not especially interested here in the later, more elaborate evolutionary versions of that argument, where the mind gets carved into a large number of highly specific adaptive modules. But the broader idea that the mind is not one smooth, unified reasoning machine but instead has specialized sub-components each of which has their own representations and specialized mechanisms which then transmit information to other systems in the mind has always seemed basically right to me.

So my first thought was that there could be some alignment between the LLM model and one of these specialized systems. The challenge with that hypothesis is that it very quickly became the kind of idea I enjoy thinking about but could do absolutely nothing with. It is interesting to say that LLMs might resemble some specialized cognitive process, but much harder to say which process, in what way, and how I would prove that the resemblance was more than an evocative metaphor. So while I still think there may be something there, it was not a very usable academic question. It was cute, but it did not want to be operationalized.

That pushed me toward a more useful question. If LLMs are, in some broad sense, semi-automatic associative word predictors, and if I also seem to have something like a semi-automatic associative word predictor running in my own head, then why do I usually make more sense than early language models?  (… taking as an assumption that I do, in fact, make more sense… we are not going to test this hypothesis too harshly…).

So, taking my own sense-making as a given,  what’s the mechanism that is helping me audit my thoughts and keeps me from being a general pile of hallucinations?

That question led me to dual-process theory, which argues that human cognition often involves a relationship between one part of the mind which generated a sort of intuitive, associative, and somewhat automatic process which interacted with a slower, more deliberate, and more controlled process.

That framework looked much more promising because it shifted the comparison away from “which mental module is this like?” and toward “what happens when fast generation is not enough?” In other words, the interesting question was not just whether LLMs resemble some part of human cognition, but whether they resemble the kind of cognition that produces plausible outputs before those outputs have been fully checked, filtered, or corrected. It also potentially offered a mechanism for fixing that defect, which now could be very academically interesting.

And that, in short, was the genesis for the term project which I’ll be covering in the next few posts.  

Image: This image was generated by taking the photos of Irises from my garden (see last post) and having Ideogram generate a text output from it. Effectively, this helps find the closest place in the model to describing the image and will let me try to make images as similar as possible while using the model directly. I had originally planned to take this and then have Ideogram add in a clock to make my point. However, it looks like Ideogram has some sort of very specific ‘get clocks right’ method which plants this sort of ugly clock-face on things so I couldn’t get it to work.

Instead, I ended up taking the description, modifying it slightly, feeding it into Midjourney to get something very similar, and using that. The exact prompt I ended up using was: “A garden bed of blooming bearded irises in various shades of purple and pink. The flowers are arranged in a diagonal line from the bottom left to the top right of the image. The irises have tall, upright green stems with broad leaves at their base. The flowers range from deep violet to light lavender, with some having white accents. In front of the irises is a golden clock on a pole like a lamp-post facing forward and easy to read. Behind the irises is a well-maintained green shrub border. The garden bed is edged with dark mulch and has a paved pathway visible on the left side of the frame. The grass is a vibrant green and is freshly cut. The lighting in the image is bright and natural, creating clear definition of the flower petals and plant textures. The composition shows multiple flower spikes at different heights, creating a layered effect in the garden display. –ar 3:2″

Cognitive Science is an interesting discipline. The name sounds like it should mean “a more mind-focused version of psychology,” but the field is much broader. Cognitive science is the interdisciplinary study of mind which draws together psychology, neuroscience, philosophy, computer science, and several other fields. In particular, it’s the field that asks questions like: How do minds represent the world? How do they reason, learn, remember, perceive, and make decisions? 

As an interdisciplinary field that deals with a somewhat touchy set of concepts, cognitive science can also be internally contentious. There is some debate about what gets to count as a “mind,” what counts as “cognition,” and what should instead be treated as an analogy or mind-like object of study. Sometimes this is mostly semantic: people may avoid calling a machine, organization, or system a “mind” because they do not want the term to become misleading outside an academic context. 

Sometimes however the disagreement represents a more material difference of opinion or area of focus. Some traditions treat cognition as an abstract process of representation, computation, and behavior independent of the ‘medium’ (person, machine, system) in which it occurs. Others tie cognition much more closely to the brain and would argue other forms of ‘mind’ aren’t really valid except as models. Still others would argue that ‘mind’ actually can’t include the brain alone and that mind cannot really be separated from the environment, tools, or social setting, because those things can store information and affect how decisions are made. Interestingly, this last school of thought has become one of the more visible and influential movements in recent cognitive science, even if it remains far from a universal consensus. 

I mention most of this to contextualize my own little space in this tumultuous debate. This is the post where I want to clarify some of my assumptions to hopefully mitigate confusion further down the line.

To begin, there are really a few specific questions within CogSci that I’m interested in:

1. How do AI minds (primarily LLMs) work in CogSci terms? 
2. How are AI minds similar to, and different from, human minds or specific human cognitive processes?
3. How can we create better AI minds, especially in terms of accuracy, reliability, and explainability?  
4. What can AI minds, when used as models or analogies, teach us about human cognition?
5. How is the increasing use of AI impacting human cognition? 

Now I use the word ‘mind’ here in the broad academic sense: as a system that performs reasoning, representation, and decision-making under uncertainty. By this definition machines (and organizations, etc.) can in fact be ‘minds’. I don’t use the word ‘mind’ to imply ‘conscious’ in any form or fashion. On my more philosophical days I do think about consciousness, but it’s not something I am going to attempt to seriously academically address. 

I’ve studied a few different areas of cognitive science at different times in my life, but the one which I’ve recently been spending the most time with (and thus the one which will likely make up the next several posts) is dual-process theory.  I’ll go into this in more depth later, but to summarize for now: In its simplest form, dual-process theory distinguishes between fast, automatic, seemingly ‘effortless’ day-to-day cognition and slower, more deliberate, effortful reasoning. The terminology varies a bit depending on the writer and the time (I’ve taken to using ‘Type 1’ and ‘Type 2’ as is the typical modern parlance) but those two processes or some variation of them are the core idea behind the model. 

I find dual-process theory useful for thinking about large neural networks, and LLMs in particular. My working view is that the model itself often behaves something like Type 1 cognition: fast, fluent, associative but also prone to characteristic errors and plausible-but-wrong answers. If that analogy is useful, then many problems in LLM accuracy and reliability may need to be addressed in a similar way to problems in human cognition: by adding interruption (‘executive function’), verification, external memory, tool use and calls to more structured reasoning. 

That is the set of topics I mean to explore in this writing. I am not especially interested in examining ideas about future AI consciousness, or in settling whether AI is or could ever “really” be a mind in some philosophical sense. My interest is in the more grounded cognitive questions: how AI systems attempt to reason, why they fail in the ways they do, and what kinds of structures might make them more reliable. Those questions do point in the direction of bigger debates about machine cognition and perhaps even one day “AGI”, but my starting point is much more immediate and humble: trying to understand the pitfalls and plausible solutions to the models we already have.  

I also, when I’m feeling particularly brave, muse about questions related to human minds. Here, though, I should admit the limits of my expertise. I am far more qualified to talk about machine learning and computers than I am to talk about neurons and brains. Human cognition fascinates me, and I may occasionally be reckless enough to write about it publicly, but those posts will be more exploratory than authoritative and I’ll try to caveat them appropriately. Still, this is my personal space, so I can’t promise I’ll spare you from the occasional speculative ramble. 

Image: Photo of my front garden Iris bed, taken May 2024 on my phone because I thought they were beautiful in full bloom. This photo will be a reference point I’m going to use later for similar(ish) AI image generation.

So I’m primarily a long-term investor with some interest in the mechanisms of the markets and economics in general. While I do have some ambitions of seeking financial independence a bit earlier than social security would support, my interests in the market tend to be driven more by curiosity than practicality. I have an MBA, but to be honest what I learned there was very introductory and superficial. I find markets interesting.

Lately I’ve been reading a book called “Stocks for the Long Run” by Jeremy J. Siegel. The book is essentially a long historical argument for why equities have rewarded patient investors despite all the short-term volatility, crashes, and noise of the market. I’ve also found it’s been a good book to deepen my understanding of equity markets in general. Historical returns may not predict future results – but it’s also the only data we have.

One insight I found particularly helpful in re-wiring some of my intuition about stocks was in chapter 5, where Siegel addresses how most stocks actually have under-market returns, and why it’s so difficult to make market returns in a non-diversified portfolio.

The average investor believes that if he or she picks one or two of those stocks at random, on average, they will realize the average rate of return, which has been almost 7 percent per year after inflation.

This was me. I always believed that the reason stock picking was bad came down mainly to two things: First, you were taking on a lot more risk. The market is volatile, but individual stocks can be even more volatile, and the market isn’t going to pay any additional risk premium for hyper-focusing in on just one stock when you could buy the market as a whole. Second, for most people it tends to be a form of emotional investing. Big-name stocks tend to be traded based on feelings, feelings often shared by lots of other people and thus most people tend to be emotionally drawn to some of the most over-priced stocks. In the short run riding a higher price to earnings ratio may help you enjoy the ride, and if the market is doing well generally then making even a below-market return may feel like a success even when it’s not. Individual stock picking is an area where it’s very easy to be ‘dumb money’.

Well, while there may be some merit to all the above, it turns out there’s also a mechanical reason why picking individual stocks is statistically less likely to generate an above market return which Siegel lays out in an example.

Let’s say, hypothetically, that the market’s expected return is 0% for the period.  Let’s say a ‘good’ outcome for the year is +10% and a bad outcome is -10%. Let’s ignore inflation in our very simplified example and say we’re going to run for 2 years.

If we invest $100 in our stock there are only 4 possible outcomes:

• Good Year -> Good Year = $110 -> $121
• Bad Year – > Good Year = $90 -> $99
• Good Year -> Bad Year = $110 -> $99
• Bad Year -> Bad Year = $90 -> $81

So essentially, the arithmetic average ending value is $100, but the median outcome is only $99. Three of the four paths leave you below the starting point, even though the average is held up by the one strong outcome. 

Fail to include those exceptional winners in your stock picks, and you’re going to end up picking below-market. 

Now, obviously this is an oversimplification – the baseline is market, not 0, ‘good’ and ‘bad’ are sliding scales, and on average actually slightly less than half of stocks actually outperform the market on any given year (so it’s worse than this looks). But the notion that over time the market’s value becomes highly concentrated and that in order to win you don’t just have to pick the best half of the market but rather successfully find the few winners, remains. 

I realize I’m discovering old news here. In fact I found another blog (no affiliation – found them on google) which extrapolates this out using real data and shows that over a 20 year period 90% of companies in practice will end up underperforming their market benchmark. Even over only a 5 year period, depending on sector, you’re looking at 84% on all domestic funds which underperform (less in some sectors – because some sectors did better overall). 

That puts you in a pretty tough bind:

• If you pick a single stock and hold, your chances over the long term of that stock consistently outperforming the market get pretty bad. 
• If you try to jump around between individual stocks to find the next winner, then the temptation to end up following the market (buying high and selling low) will be extreme.

It’s not a great bind to be in for a longer-term investor. 

And this is how I realized individual stock picking was even worse than I had assumed. I already understood the risks of added volatility and the temptations of emotion, but hadn’t really thought through the mechanical issues around concentration and how large a class of ‘bad’ options that creates. If you don’t hold that relatively small number of exceptional winners, you may never actually receive the return you think “the market” provides.

I’m still interested in looking at things like sector tilts, but individual stock picking now feels less like an interesting calculated risk and more like a roulette table I don’t need to sit down at.

Image: GenAI Image using Midjourney 8.1 with the prompt “A stock market chart showing stocks performing poorly and crashing. In front of them a roulette wheel/table in the foreground, Light colored background. –p –ar 3:2”.

Like many people, I became fascinated with machine learning primarily because of the astonishing things large generative models started to accomplish around 2023. Having spent almost two decades in software development with much of that time spent trying to get people to break problems down into clear logical processes with precise definitions, the ‘fuzzy’ behavior of these new generative models felt confusing, kinda spooky, and completely disconnected from ground truths I was used to in computing. I could work with the models and even got comfortable with prompt engineering. But I still felt more like I was playing with some strange magic box rather than a computer, and I found myself lacking any real intuition for how it all actually worked.

That bothered me more than was probably reasonable, and was my main reason for returning to graduate school. I wanted to achieve a real understanding of the machinery inside that strange box.

Deep Learning, which explicitly covered larger neural networks, was a class I had been looking forward to since I had first put in my grad school application. The class was exactly what I was looking for – a moderately rigorous class (in the mathematical sense) which gave me a genuine understanding of various model architectures, why they behave as they do, and helped me connect how models are built to why they work certain ways. It gave me hands-on experience building and evaluating model architectures with PyTorch alongside reading papers and trying to address the math. 

There were a lot of really good moments, but the one that really stuck with me happened in assignment 3, in which we implemented five different model types, including a small transformer model (think: a very small version of the model type used in LLMs) to do german to english translations. 

To explain the results I saw (taken from my final report):

Qualitatively my best Seq2Seq model is still frankly pretty terrible – it repeats words, seems to have very little conception of rarer word combinations (for ‘orange hat’ it just repeated the word – ‘hat hat’). It does seem to be able to structure sentences somewhat (especially when compared to the encoder-only model) but the word repetition generally breaks up the feel of the sentence being sensible. My Seq2Seq model, even with attention, still seems to be unaware of what it has said previously (‘mother’, ‘mother’, ‘and’, ‘her’, ‘mother’, ‘son’,). While LSTM does theoretically carry context forward the compressed context seems to provide a meaningful limitation. 

My transformer model on the other hand creates sensible sentences and plausible scenarios, and even some of its substitutions feel a lot more ‘natural’ (i.e. ‘yellow dog’ for ‘terrier’ – ‘performing a board’ for ‘breaking a stick’). The transformer’s ability to ingest context and its bias for fluency in even this tiny model is almost uncanny. These generated answers could be regarded as very small hallucinations – but they’re hallucinations informed by context. The errors in even this tiny model are coherent. 

Behind that observation was a startling revelation – my model, my very tiny model which I had mostly coded and entirely trained by hand, running in a completely deterministic fashion, was generating the phenomena of hallucinations – introducing content (‘yellow’) which wasn’t in the original input. 

I’ll admit it: I ended up spending a bit of time googling if terrier had some specific association with ‘yellow’ in German. Then I realized I was being silly and started to try to figure out how this had happened. That’s when the learning really started.

Terrier was a relatively rare word in my training data. There were only 7 instances of it in the set, compared to over 2000 for “dog”. So when the model was trying to predict ‘likeliness’ with the full context of the sentence, even though there was no case in which ‘terrier’ ever directly translates to ‘dog’ in my data set, it went with the most likely and common word given the context. This is how powerful self-attention can be. Then because ‘terrier’ and ‘dog’ map relatively closely as vector word representations, it was a fairly easy switch for the model to make. Thus why my model picked ‘dog’ over the more literal ‘terrier’ despite being wrong.

More puzzling to me was the dog specifically being yellow. So as I looked through my training data I realized that the pattern and specifically “dog” was unreasonably common. It was far from universal, but of the ~2000 dog references from my training data well over half had some sort of color pattern associated with them. The literal string “yellow dog” appeared only 25 times, but other variants were much more common – “brown dog” appeared 318 times, “white dog” 291 times, “black dog” 272. The word ‘terrier’ was only associated with color once (it was usually associated with a place-name, like “Boston Terrier”) – but ‘dog’ was most often proceeded by a color. So while I couldn’t quite figure out why it would come to the conclusion that the dog was yellow specifically, I could very clearly see how when trying to find the most common structure for the language the model tried to put some sort of color there. 

What surprised me most was that this hallucination did not require randomness. There was no temperature setting nudging the model toward being inventive. The model was simply following a path made most likely in its architecture. In transformers, hallucinations emerge from the same machinery that makes the model useful. The model is trained to produce contextually plausible next tokens, with attention helping to bind those continuations into coherent context (and structure). So when the model doesn’t have a clear piece of information to place into that structure (generally because the available examples are rare or ambiguous) that pull towards coherence can cause the model to favor fluent inventions rather than obvious failures. 

This is also why no amount of creativity-removal will ever work to completely eliminate hallucinations – because to the model this invention was the most structurally correct answer it had available, just like remembering to use “a” or “an” appropriately in a sentence is structurally correct and required even if those articles didn’t exist in the language being translated from. 

This insight would eventually lead me into a whole world of study on accuracy and reliability in LLMs. But for now, it was enough to make the strange magic box feel a little less like magic, while also revealing just how fundamental the problem of LLM reliability really is. 

Image: Screen shot from my notebook taken at a kind of random point while training my transformer model.

Welcome to my blog.

I started this site because I wanted a place to collect thoughts, and questions that didn’t really fit anywhere else.

A lot of the things I find myself thinking about sit somewhere on the borders between professional experience, academic interest, and personal curiosity. I tend to be drawn to questions about technology, the mind, and systems. That curiosity covers several different interrelated topics. Some of those thoughts are directly connected to my professional work. Others come from school, reading, conversations, or simply noticing patterns in the world and wanting somewhere to put them.

For a while, I thought about using LinkedIn for more of this. But LinkedIn has a particular tone I don’t feel fits these thoughts. It can be useful, but it often feels very corporate, performative, and lends itself to short-form content engineered to garner views and reactions. These observations aren’t really announcements or selling my personal brand as a thought leader. Many of these thoughts aren’t really polished enough to be articles, but they are still worth keeping.

I also think we have moved away from long-form blogging in a way that feels like a loss to me. There is something useful about having a personal place on the internet where ideas can accumulate over time. Perhaps this is nostalgia for an earlier internet, but not every thought needs to be optimized to generate visibility. Sometimes it is valuable just to write something down, explain it clearly, and have a stable place to refer back to it later.

So that is the purpose of this site. It is partly a notebook, partly a professional blog, and partly a place for open-ended inquiry. I expect some posts will be fairly practical, especially the more professional posts around data, technology, and business systems. Others may be more academic or reflective, especially around AI, cognitive science, and research. Some may also just be attempts to understand or explore something interesting to me a bit more clearly.

I am not trying to make this blog into a publication in the formal sense. I am trying to create a space where I can think, record observations for myself, and occasionally share ideas that may be useful or interesting to others. My hope is that over time this becomes a useful archive of what I have been reading, learning, building, and thinking about. If it ever truly achieves my wildest ambitions for it, then it’ll be a place where these thoughts can lead to interesting conversations.

Image Description: This photo was taken from my office window on an especially beautiful spring day. For about one week each year, the decorative cherry tree outside my window is in full bloom. This was one of those days.