Automate a weekly arXiv paper triage
This is a post about how I use Codex and a few surrounding tools to remove repeated friction from AI safety research. Task: This is a post about how I use Codex and a few surrounding tools to remove repeated friction from AI safety research. It is not a guide to autonomous research, and it is not an argument for outsourcing your own judgement. My idea of leveraging LLMs is: let agents search, organise, test, check, and remember, while I keep responsibility for the research question, verify the evidence and what I publish. I am writing this because the setup has started to change how I work, not in a magical way though. It mostly shortens loops that used to cost a lot of attention: literature search, experiment setup, debugging, source tracking, and remembering project-specific quirks. Two constraints matter before any setup details: Don’t overoptimize on the get go. If you try to automate everything, maintaining and understanding your system becomes the work. Don’t overoptimize on the get go. If you try to automate everything, maintaining and understanding your system becomes the work. Do not outsource judgement. The agent can make you faster, including faster at being wrong. Do not outsource judgement. The agent can make you faster, including faster at being wrong. Therefore you should also not let an LLM decide if the mentioned workflows and setup are worth adapting. The purpose of this article is to motivate you to explore the possibilities of current agentic systems and how they can assist you in your work. Why I Started Automating Parts Of My Research Why automate your work with agents? Sometimes it can be difficult to relinquish responsibility to an agentic system. You might not trust the system’s output, prefer to stay in control, or think that the system is inferior and unable to complete the task. When I started my master’s thesis last year, I mostly used AI for guidance and clarification and still did most things manually. That made sense at first: I did not fully trust agentic systems, I wanted to stay in control, and I often assumed the model would do a worse job than I would or they weren’t as capable. However, it didn’t take long before I realized that this approach was not feasible. I had to catch up on a lot of knowledge in a fast-moving field. Therefore, saving time and knowing what to prioritize, which papers to read, and how to quickly iterate experiments became crucial. The sooner I realize I’m on the wrong path, the sooner I can correct my course and focus my most valuable resource, time, on a more promising approach. The goal became simple: spend less attention on repeated setup and more attention on the parts that actually require my judgement. My Baseline: I Mostly Work In Codex Most of this setup is built around the Codex app. I can recommend Codex for this kind of workflow, but I have not done a fair comparison against Claude Code, OpenCode, OpenHands, Hermes, or other agent frameworks. Why it works well for me is it’s combination: local files, coding, browser/computer use, skills, subagents, thread management, and enough app-level UX that I don’t have to leave Codex to get work done. The principles should transfer to other agent harnesses. The implementation details in this post are mostly Codex-specific. Problem 1: Too Many Papers The first workflow I automated was literature triage. Rather than having an agent decide which papers are relevant to my work, I wanted it to prefilter papers based on the criteria I set. The basic loop: Run a weekly Codex automation that searches arXiv for papers matching keywords I care about (e.g. AI alignment). Run a weekly Codex automation that searches arXiv for papers matching keywords I care about (e.g. AI alignment). Save the results in a CSV file to avoid the same paper appearing every week. Save the results in a CSV file to avoid the same paper appearing every week. Triage only the delta from the previous run. Triage only the delta from the previous run. Produce an initial prioritisation list: keep, skim, drop, duplicate, outside scope or weak method. Produce an initial prioritisation list: keep, skim, drop, duplicate, outside scope or weak method. Add promising papers to my Zotero inbox, where they are automatically embedded in a vector database. Add promising papers to my Zotero inbox, where they are automatically embedded in a vector database.; tools: Codex. Human role: The original post does not say how a person stayed involved.; limitations: Prerequisites include access to Codex, arXiv, and Zotero. The author warns against over-optimizing initially, as maintaining the system can become the work itself. Crucially, the agent must not outsource judgment; it can make users faster at being wrong. The workflow relies on the human understanding the test and inspecting outputs. Source reliability is secondary, based on a personal blog post without controlled benchmarks..
Tools
Published on Agent Work Atlas:
Overview
This is a post about how I use Codex and a few surrounding tools to remove repeated friction from AI safety research.
Workflow
This is a post about how I use Codex and a few surrounding tools to remove repeated friction from AI safety research. It is not a guide to autonomous research, and it is not an argument for outsourcing your own judgement. My idea of leveraging LLMs is: let agents search, organise, test, check, and remember, while I keep responsibility for the research question, verify the evidence and what I publish. I am writing this because the setup has started to change how I work, not in a magical way though. It mostly shortens loops that used to cost a lot of attention: literature search, experiment setup, debugging, source tracking, and remembering project-specific quirks. Two constraints matter before any setup details: Don’t overoptimize on the get go. If you try to automate everything, maintaining and understanding your system becomes the work. Don’t overoptimize on the get go. If you try to automate everything, maintaining and understanding your system becomes the work. Do not outsource judgement. The agent can make you faster, including faster at being wrong. Do not outsource judgement. The agent can make you faster, including faster at being wrong. Therefore you should also not let an LLM decide if the mentioned workflows and setup are worth adapting. The purpose of this article is to motivate you to explore the possibilities of current agentic systems and how they can assist you in your work. Why I Started Automating Parts Of My Research Why automate your work with agents? Sometimes it can be difficult to relinquish responsibility to an agentic system. You might not trust the system’s output, prefer to stay in control, or think that the system is inferior and unable to complete the task. When I started my master’s thesis last year, I mostly used AI for guidance and clarification and still did most things manually. That made sense at first: I did not fully trust agentic systems, I wanted to stay in control, and I often assumed the model would do a worse job than I would or they weren’t as capable. However, it didn’t take long before I realized that this approach was not feasible. I had to catch up on a lot of knowledge in a fast-moving field. Therefore, saving time and knowing what to prioritize, which papers to read, and how to quickly iterate experiments became crucial. The sooner I realize I’m on the wrong path, the sooner I can correct my course and focus my most valuable resource, time, on a more promising approach. The goal became simple: spend less attention on repeated setup and more attention on the parts that actually require my judgement. My Baseline: I Mostly Work In Codex Most of this setup is built around the Codex app. I can recommend Codex for this kind of workflow, but I have not done a fair comparison against Claude Code, OpenCode, OpenHands, Hermes, or other agent frameworks. Why it works well for me is it’s combination: local files, coding, browser/computer use, skills, subagents, thread management, and enough app-level UX that I don’t have to leave Codex to get work done. The principles should transfer to other agent harnesses. The implementation details in this post are mostly Codex-specific. Problem 1: Too Many Papers The first workflow I automated was literature triage. Rather than having an agent decide which papers are relevant to my work, I wanted it to prefilter papers based on the criteria I set. The basic loop: Run a weekly Codex automation that searches arXiv for papers matching keywords I care about (e.g. AI alignment). Run a weekly Codex automation that searches arXiv for papers matching keywords I care about (e.g. AI alignment). Save the results in a CSV file to avoid the same paper appearing every week. Save the results in a CSV file to avoid the same paper appearing every week. Triage only the delta from the previous run. Triage only the delta from the previous run. Produce an initial prioritisation list: keep, skim, drop, duplicate, outside scope or weak method. Produce an initial prioritisation list: keep, skim, drop, duplicate, outside scope or weak method. Add promising papers to my Zotero inbox, where they are automatically embedded in a vector database. Add promising papers to my Zotero inbox, where they are automatically embedded in a vector database.
Reported result
The author reports that this workflow shortens loops costing attention, such as literature search and source tracking. It reduces the cost of asking 'is this even worth pursuing?' but does not eliminate the need for human validation. The author notes that the setup changes how they work by allowing them to focus on judgment rather than repeated setup, though they explicitly state they have not measured specific efficiency gains or success rates.
Human role
The original post does not say how a person stayed involved.
Limitations
Prerequisites include access to Codex, arXiv, and Zotero. The author warns against over-optimizing initially, as maintaining the system can become the work itself. Crucially, the agent must not outsource judgment; it can make users faster at being wrong. The workflow relies on the human understanding the test and inspecting outputs. Source reliability is secondary, based on a personal blog post without controlled benchmarks.
Try it
Act as a research assistant using Codex. I need to set up a weekly arXiv paper triage workflow for my research on [replaceable topic, e.g., AI alignment]. 1. Create a script or automation that searches arXiv for papers matching [replaceable keywords] from the last 7 days. 2. Compare results against a local CSV file [replaceable path to history.csv] to identify new papers only. 3. For each new paper, generate a prioritization label: 'keep', 'skim', 'drop', 'duplicate', 'outside scope', or 'weak method' based on the abstract and title. 4. Output a summary table with these labels and links. 5. Do not automatically add papers to Zotero yet. First, show me the summary table for one sample week of data. 6. Wait for my confirmation before setting up the actual weekly automation or modifying my Zotero library. Ensure the human remains in the loop for final decisions. If the search yields no new papers, report that clearly. If the CSV is missing, create it with headers: title, arxiv_id, date, status.
Source reports are not independently verified. Publication date is the catalog publication date, not the original article date.