Record how you work, then automate the repetitive parts
Hi Hacker News, I'm Louis. Task: Hi Hacker News, I'm Louis. I built Screenpipe (https://screenpipe.com), an app that records your screen and audio locally (only!), and gives AI agents a searchable memory of what you've seen, said, and heard. This makes it easier to automate your repetitive tasks, turn them into SOPs (Standard Operating Procedure) and so on.I made a HN-style demo video at https://www.tella.tv/video/build-your-ai-second-brain-with-s... and there’s a marketing video at https://www.youtube.com/watch?v=c1jV6E9pyug.I’ve been obsessed with this for a long time. I’ve been maintaining a “second brain” since 2020, in which I would store journals, handwritten notes, music I listen to, projects I'm working on, conversations I have with people, personal CRM etc. I experimented a lot of RAG in the early days with ParlAI, hundreds of fine-tuned GPT2 models, and GPT3 (https://forum.obsidian.md/t/fine-tuning-openai-api-gpt3-on-y...). Later I built Ava, the first Obsidian AI plugin, which grew to a few thousands of users quickly. It then became Embedbase, an API to make it easier to build AI apps powered by RAG.What I learned from all this is how important it is for the models to have context about what you’re doing on your computer, in order to get them to do what you want.In the early days there was fine tuning but it was too much pain, then there was tool calling so that AI can access software you use but still kinda not autonomous enough. needing micro management. Then MCP came, but it felt too static, and non technical users struggled to build and use MCP. Then we got skills. Most recently we’ve seen Karpathy’s LLM-maintained wiki, Garry's GBrain, etc., where an agent incrementally maintains a persistent collection of Markdown pages. New sources update entity pages, strengthen or contradict existing claims, and improve a synthesis that compounds over time. I like this pattern, but it still begins with someone selecting and importing the sources. There is still no way AI can know what you and your company are doing every day, across apps, not just inside of apps.Of course, not everyone wants this. But I do! I want AI to know what I'm doing and never lose memory ever again, and I want it to use the same software that humans do, without painful context switches.I started building Screenpipe for myself in 2024 - a CLI to record your screen and plug this context into AI. An HN user posted it in 2024 (https://news.ycombinator.com/item?id=41695840) and that discussion influenced the product. The most useful criticism concerned recording consent, local security, CPU usage, signal-to-noise, and whether agents could act on top of the data.The naive implementation started from continuously recording video and running OCR over every frame. But that creates duplicate data, consumes substantial resources (it basically turns your computer into a space heater!), and discards structure the operating system already knows. Screenpipe now instead listens for events such as app switches, clicks, typing pauses, scrolling, and idle fallbacks. When something meaningful changes, it pairs a screenshot with the operating system’s accessibility tree at the same timestamp. OCR is used when structured accessibility data is unavailable. We also capture audio continuously, identify speakers and transcribe locally through Parakeet/Whisper or using cloud models.Everything is indexed in a local SQLite database, mp4 files, and sometimes md files. An AI friendly API on port 3030 is open for agents, with authentication and a MCP and skills.Once Screenpipe has been up and running for a while, you can use it through our built-in chat, Claude, ChatGPT, Hermes, Openclaw, or any agent, to do things like:- adding context to your current chat, e.g. "gather all context about task X", then requiring less prompts to achieve your goal- retrieve information, e.g.; tools: custom-built or unnamed agent. Human role: The original post does not say how a person stayed involved.; limitations: Prerequisites include a compatible OS (macOS, Windows, Linux) and sufficient local storage for continuous recording. Constraints involve strict privacy compliance (e.g., GDPR, employee monitoring laws) and potential security risks if agents are over-permissioned. Operating costs are low for local processing, but commercial use requires a license. Human approval is critical for any action taken by agents based on recorded context..
Tools
custom-built or unnamed agent
Published on Agent Work Atlas:
Overview
Hi Hacker News, I'm Louis.
Workflow
Hi Hacker News, I'm Louis. I built Screenpipe (https://screenpipe.com), an app that records your screen and audio locally (only!), and gives AI agents a searchable memory of what you've seen, said, and heard. This makes it easier to automate your repetitive tasks, turn them into SOPs (Standard Operating Procedure) and so on.I made a HN-style demo video at https://www.tella.tv/video/build-your-ai-second-brain-with-s... and there’s a marketing video at https://www.youtube.com/watch?v=c1jV6E9pyug.I’ve been obsessed with this for a long time. I’ve been maintaining a “second brain” since 2020, in which I would store journals, handwritten notes, music I listen to, projects I'm working on, conversations I have with people, personal CRM etc. I experimented a lot of RAG in the early days with ParlAI, hundreds of fine-tuned GPT2 models, and GPT3 (https://forum.obsidian.md/t/fine-tuning-openai-api-gpt3-on-y...). Later I built Ava, the first Obsidian AI plugin, which grew to a few thousands of users quickly. It then became Embedbase, an API to make it easier to build AI apps powered by RAG.What I learned from all this is how important it is for the models to have context about what you’re doing on your computer, in order to get them to do what you want.In the early days there was fine tuning but it was too much pain, then there was tool calling so that AI can access software you use but still kinda not autonomous enough. needing micro management. Then MCP came, but it felt too static, and non technical users struggled to build and use MCP. Then we got skills. Most recently we’ve seen Karpathy’s LLM-maintained wiki, Garry's GBrain, etc., where an agent incrementally maintains a persistent collection of Markdown pages. New sources update entity pages, strengthen or contradict existing claims, and improve a synthesis that compounds over time. I like this pattern, but it still begins with someone selecting and importing the sources. There is still no way AI can know what you and your company are doing every day, across apps, not just inside of apps.Of course, not everyone wants this. But I do! I want AI to know what I'm doing and never lose memory ever again, and I want it to use the same software that humans do, without painful context switches.I started building Screenpipe for myself in 2024 - a CLI to record your screen and plug this context into AI. An HN user posted it in 2024 (https://news.ycombinator.com/item?id=41695840) and that discussion influenced the product. The most useful criticism concerned recording consent, local security, CPU usage, signal-to-noise, and whether agents could act on top of the data.The naive implementation started from continuously recording video and running OCR over every frame. But that creates duplicate data, consumes substantial resources (it basically turns your computer into a space heater!), and discards structure the operating system already knows. Screenpipe now instead listens for events such as app switches, clicks, typing pauses, scrolling, and idle fallbacks. When something meaningful changes, it pairs a screenshot with the operating system’s accessibility tree at the same timestamp. OCR is used when structured accessibility data is unavailable. We also capture audio continuously, identify speakers and transcribe locally through Parakeet/Whisper or using cloud models.Everything is indexed in a local SQLite database, mp4 files, and sometimes md files. An AI friendly API on port 3030 is open for agents, with authentication and a MCP and skills.Once Screenpipe has been up and running for a while, you can use it through our built-in chat, Claude, ChatGPT, Hermes, Openclaw, or any agent, to do things like:- adding context to your current chat, e.g. "gather all context about task X", then requiring less prompts to achieve your goal- retrieve information, e.g.
Reported result
The source describes the product's capabilities and architecture but does not provide specific business outcomes, efficiency metrics, or failure rates for a particular company. It mentions that early naive implementations caused high CPU usage, which was later optimized. Actual impact on business processes remains unknown and requires a measurable acceptance check, such as tracking time saved on repetitive tasks after a pilot period.
Human role
The original post does not say how a person stayed involved.
Limitations
Prerequisites include a compatible OS (macOS, Windows, Linux) and sufficient local storage for continuous recording. Constraints involve strict privacy compliance (e.g., GDPR, employee monitoring laws) and potential security risks if agents are over-permissioned. Operating costs are low for local processing, but commercial use requires a license. Human approval is critical for any action taken by agents based on recorded context.
Try it
I want to pilot Screenpipe to automate my daily reporting. First, install Screenpipe locally using `npx screenpipe record` and configure it to only capture my work-related apps (e.g., Slack, Jira) while excluding personal browsers. Set up a local AI agent connected to the Screenpipe API on port 3030. Ask the agent to retrieve all tasks I worked on between 9 AM and 5 PM today and draft a summary email. Review the draft for accuracy and privacy compliance before sending. If the summary is correct, ask the agent to create a reusable SOP for this daily report. Do not allow the agent to send emails or modify files without my explicit confirmation. Measure success by comparing the time taken to generate the report manually versus using the agent.
Source reports are not independently verified. Publication date is the catalog publication date, not the original article date.