Build data pipelines and analysis for a research paper
How I vide code papers with it. Task: How I vide code papers with it Numerous blog posts and YouTube videos discuss how good Claude Code is as an SWE agent. However, I would like to discuss my experience using it for data analysis. And my conclusion is that it’s revolutionary and has completely changed my way of working. I’ll share my setups and the lessons I learned. But if you haven’t used Claude Code before, the best thing you can do is stop reading this and give it a try. I’m a researcher. In my day-to-day work, I need to write (mostly) Python code to collect and process data, gain insights, and then turn them into visually appealing figures and tables. Finally, I write papers about the findings. A significant portion of my coding tasks involves building data pipelines and exploring patterns in the data (i.e., the dirty work), which can be tedious and time-consuming. With the development of various AI agents, I started to wonder what a data analysis agent would look like and even considered building one myself. Then I played with Claude Code (which I’ll refer to as CC from here on), and I realized that it was exactly what I was looking for. In the past few weeks, I have been using CC to perform the data analysis for a paper I’m working on. It was a fascinating journey. Ultimately, I believe it generated over 98% of the code. And I think I will never go back to my old working routine again. I’m using a combination of the following for coding tasks: Claude Code: This is my main coding agent. Claude Code: This is my main coding agent. Lazygit: Git is your best friend for vibe coding (more on this later), and lazygit is a great terminal UI for Git. Lazygit: Git is your best friend for vibe coding (more on this later), and lazygit is a great terminal UI for Git. Cursor: I primarily use it for reviewing the code generated by CC and making minor modifications as needed. The tab completion feature is still super handy. Cursor: I primarily use it for reviewing the code generated by CC and making minor modifications as needed. The tab completion feature is still super handy. Good old Jupyter Notebook: Although Cursor supports notebooks now, the experience is just not good. So I would still open the notebook in my browser. Good old Jupyter Notebook: Although Cursor supports notebooks now, the experience is just not good. So I would still open the notebook in my browser. I have two monitors so that I would have the Cursor in one and a browser window (with Jupyter Notebooks) in the other. Moreover, I would use my iPad as the third screen with CC in it. The screen size of the iPad is actually perfect for CC. Although one can run CC directly in Cursor, I prefer running it in the terminal. What about the alternatives? There are many other coding agents, and I have tried the following: OpenAI Codex CLI: I only tried it briefly; somehow, it sucks so bad. OpenAI Codex CLI: I only tried it briefly; somehow, it sucks so bad. Gemini CLI: Similar to Codex, not even close to CC. But I think it has a lot of potential. Guess we will see. Gemini CLI: Similar to Codex, not even close to CC. But I think it has a lot of potential. Guess we will see. Cursor agent: It’s decent, but not at the CC level. Additionally, with the new pricing strategy for the Pro plan, the included API calls are pretty limited. Cursor agent: It’s decent, but not at the CC level. Additionally, with the new pricing strategy for the Pro plan, the included API calls are pretty limited. I haven’t tried others, but based on others' feedback, I’m convinced that CC is currently the best in terms of coding ability. CC is also great due to the incredible value it provides, given its price. I'm on the Pro plan, which only costs $17 a month if you pay yearly. It's rate-limited but enough for me (at least for now). And you can easily burn over $100 worth of API calls each month using the Pro subscription (you can use the ccusage tool to monitor your usage). When I reach the rate limit but need to complete a task, I switch to the bring-your-own-key mode. Opus 4 is not available in the Pro plan, but I find Sonnet 4 more than competent. Configuring Claude Code IDE integration This should work for all VSCode-based editors. See instructions here. It allows CC to be aware of which file you are focusing on in the editor. You can select specific lines of code to highlight them. The integration works even when you are running CC in a separate terminal window. Hooks; tools: Claude Code. Human role: The original post does not say how a person stayed involved.; limitations: The workflow relies on Claude Code Pro subscription ($17/month yearly) and macOS for specific notification hooks. CC has rate limits, requiring a switch to bring-your-own-key mode when exceeded. The author found CC less effective for precise figure creation, preferring manual editing. Source reliability is secondary, based on a single user's blog post..
Tools
Published on Agent Work Atlas:
Overview
How I vide code papers with it
Workflow
How I vide code papers with it Numerous blog posts and YouTube videos discuss how good Claude Code is as an SWE agent. However, I would like to discuss my experience using it for data analysis. And my conclusion is that it’s revolutionary and has completely changed my way of working. I’ll share my setups and the lessons I learned. But if you haven’t used Claude Code before, the best thing you can do is stop reading this and give it a try. I’m a researcher. In my day-to-day work, I need to write (mostly) Python code to collect and process data, gain insights, and then turn them into visually appealing figures and tables. Finally, I write papers about the findings. A significant portion of my coding tasks involves building data pipelines and exploring patterns in the data (i.e., the dirty work), which can be tedious and time-consuming. With the development of various AI agents, I started to wonder what a data analysis agent would look like and even considered building one myself. Then I played with Claude Code (which I’ll refer to as CC from here on), and I realized that it was exactly what I was looking for. In the past few weeks, I have been using CC to perform the data analysis for a paper I’m working on. It was a fascinating journey. Ultimately, I believe it generated over 98% of the code. And I think I will never go back to my old working routine again. I’m using a combination of the following for coding tasks: Claude Code: This is my main coding agent. Claude Code: This is my main coding agent. Lazygit: Git is your best friend for vibe coding (more on this later), and lazygit is a great terminal UI for Git. Lazygit: Git is your best friend for vibe coding (more on this later), and lazygit is a great terminal UI for Git. Cursor: I primarily use it for reviewing the code generated by CC and making minor modifications as needed. The tab completion feature is still super handy. Cursor: I primarily use it for reviewing the code generated by CC and making minor modifications as needed. The tab completion feature is still super handy. Good old Jupyter Notebook: Although Cursor supports notebooks now, the experience is just not good. So I would still open the notebook in my browser. Good old Jupyter Notebook: Although Cursor supports notebooks now, the experience is just not good. So I would still open the notebook in my browser. I have two monitors so that I would have the Cursor in one and a browser window (with Jupyter Notebooks) in the other. Moreover, I would use my iPad as the third screen with CC in it. The screen size of the iPad is actually perfect for CC. Although one can run CC directly in Cursor, I prefer running it in the terminal. What about the alternatives? There are many other coding agents, and I have tried the following: OpenAI Codex CLI: I only tried it briefly; somehow, it sucks so bad. OpenAI Codex CLI: I only tried it briefly; somehow, it sucks so bad. Gemini CLI: Similar to Codex, not even close to CC. But I think it has a lot of potential. Guess we will see. Gemini CLI: Similar to Codex, not even close to CC. But I think it has a lot of potential. Guess we will see. Cursor agent: It’s decent, but not at the CC level. Additionally, with the new pricing strategy for the Pro plan, the included API calls are pretty limited. Cursor agent: It’s decent, but not at the CC level. Additionally, with the new pricing strategy for the Pro plan, the included API calls are pretty limited. I haven’t tried others, but based on others' feedback, I’m convinced that CC is currently the best in terms of coding ability. CC is also great due to the incredible value it provides, given its price. I'm on the Pro plan, which only costs $17 a month if you pay yearly. It's rate-limited but enough for me (at least for now). And you can easily burn over $100 worth of API calls each month using the Pro subscription (you can use the ccusage tool to monitor your usage). When I reach the rate limit but need to complete a task, I switch to the bring-your-own-key mode. Opus 4 is not available in the Pro plan, but I find Sonnet 4 more than competent. Configuring Claude Code IDE integration This should work for all VSCode-based editors. See instructions here. It allows CC to be aware of which file you are focusing on in the editor. You can select specific lines of code to highlight them. The integration works even when you are running CC in a separate terminal window. Hooks
Reported result
The author reports that CC generated over 98% of the code for the paper's analysis. However, the source does not provide specific metrics on time saved or error rates. The author notes that CC makes mistakes and requires human review to ensure correctness. No independent verification of the 98% claim is provided.
Human role
The original post does not say how a person stayed involved.
Limitations
The workflow relies on Claude Code Pro subscription ($17/month yearly) and macOS for specific notification hooks. CC has rate limits, requiring a switch to bring-your-own-key mode when exceeded. The author found CC less effective for precise figure creation, preferring manual editing. Source reliability is secondary, based on a single user's blog post.
Try it
Act as a data analysis assistant using Claude Code. I am a researcher working on [replaceable: paper topic]. I have raw data in [replaceable: file paths/format]. 1. Create a plan in a Markdown file for data cleaning, analysis for [replaceable: research question], and output generation. 2. Generate Python scripts for each step one by one. 3. After each script, pause for my review. I will check the code in Cursor and run it in Jupyter. 4. Use Git to commit changes after each successful step. Do not proceed to the next step until I confirm the current output is correct. If you encounter errors, explain them and propose fixes. Start by drafting the plan for [replaceable: specific analysis task].
Source reports are not independently verified. Publication date is the catalog publication date, not the original article date.