My Private Personal Agent

Access to a private, good coding model[1] has re-invigorated my creative spirit, kicking off a multi-week-long quest to customize my own personal agent.[2] Something about knowing that everything I share with this agent is mine, ours even, really lends itself to this. In a more practical sense, it’s freeing to be able to dump all company docs, other peoples information, and any api keys without worrying about them leaving my control. I want to explain the agent configuration I have set up around this model, and why, and provide an open source implementation of it all. Tangent on privacy: In the limit I believe private intelligence is necessary. My government-assigned AI representative needs to know everything about me to bargain on my behalf. If I want a user aligned model, I need it to know me intimately. Right now, on my computer I have a section of private memories and preferences. While none of this is yet super valuable, it will increasingly[3] be. Companies would love to have it. Governments would love to have it. My preferences, perspective, taste, may well become my moat. I might as well start protecting it now.  For the body of my agent I chose the Pi coding harness. It’s called a coding harness, and referred to as a coding agent, but this is what a general ‘agent’ should be. Like all coding agents it has file access & bash, which is all you need to basically do what a human on a computer can.[4] 80% of the reason I chose pi is their tagline: There are many agent harnesses but this one is yours Pi makes it easy to add extensions and is very minimal, which is great for our purposes here, and is open source, an absolute requirement.  Table of Contents I'll talk about the following things: Maintaining privacy - GitHub, sharing data between models, and more. Tinfoil Extensions Search Images Voice (external) Miscellaneous extensions & fixes  Prompt caching fix & system reminders VSCode integrations Thinking signature fix You can follow along with this example repository, that shows all sections.[5] Maintaining Privacy As I mentioned in the introduction, I try to entirely use GLM 5.2, as it’s a good coding model. (One small drawback is that it isn’t multi-modal, which I deal with in Tinfoil Extensions > Images)  Even so, I sometimes need another model. My model of choice up until now has been Claude, and I wanted to be able to share a specific selection of context / system prompts with Claude so I'm not constantly repeating myself. To do this, we can use @import syntax so that different agents get different prompts, composed of fragments. Claude does this natively but we have to extend Pi to respect it. For instance, Pi gets both the encrypted, private preferences, and the shared preferences, while Claude would only get the latter. Pi's prompt looks like: ## Preferences @~/agents/fragments/encrypted/preferences.md ## Weaker Preferences @~/agents/fragments/shared-preferences.md Next, I’ll want this to be stored somewhere. Git / GitHub is my preferred option, and allows me to see differences, so I can revert and monitor agent changes. No brain surgery without a reset button. That along with the constraint of sharing instructions led to a central agents folder at my root, with everything inside it: ~/agents/ ├── pi ├── claude/ └── fragments/ └── encrypted/ I want to prevent GitHub from taking my data that I so carefully prevented AI labs from getting. Git-crypt comes in handy here! It preserves the git tree, which means it shows diffs locally and works like normal git, but stores encrypted ciphertext on GitHub[6]. I use this for the private prompts and memory. There’s one more trick – AI can just do things. To prevent this I made a little information-guard extension. This prevents agents from committing or pushing in the ~/agents/ folder. It also prevents specific agents (Claude) from reading private files like memories, using apple sandboxing.[7] Sandboxing is a big thing, and I tried not to get lost in the weeds here, so it's very minimal. From what I’ve seen, sandboxes (reasonably) tend to be more worried about preventing unauthorized actions, than about what data is read by whom. In the future I’d like to extend on this and data provenance in general. There’s a lot of options for how to do this, though it’s annoying that apple sandboxing is actively hostile to its users.  I’m a visual person. It’s nice to check that something is correct at a glance. I have a little script to show what data is visible to which agent, and what is encrypted. I also have a script that produces a stitched together prompt markdown file.  The slop visual created by visualize-provenance.mjs Tinfoil Extensions Websearch Search is absolutely essential. I’m not going to read docs. Pi comes without this, so we get to add a simple extension that allows search and fetch. I use Tinfoil’s websearch endpoint because it’s accurate, it gives you the full page, it checks for prompt injections, and of course it’s weakl