Data Pipelines and Automation Services

I learned to code because manual data work stopped scaling. Reports took too long, feeds had to be rebuilt by hand, and no tool did exactly what the business needed.

That origin shows in how I build. I start with the work being repeated, then build the smallest system that removes it.

Problems this solves

• Someone copies data between websites, spreadsheets, and systems every week.
• Inventory or product data has to reach several platforms in different formats.
• Reports are assembled by hand before every meeting.
• A public data source has what you need but no export button.
• An outside API limits how fast you can send requests.

What I build

Web scrapers and data collection

I build scrapers with Python, Selenium, and Playwright, including multi-worker designs. At Aquifer Investments I developed a multi-threaded data collection scraper that gathered public records to inform business decisions.

ETL pipelines and data feeds

I wrote a Go ETL pipeline and API that syncs vehicle inventory to Google and Facebook ad platforms. It splits feeds by brand and unique identifier, and it is still used for inventory marketing today.

Large file processing

I used Python to filter and process a 3.5 GB Texas sold-vehicle dataset into reporting data the business could use. It was my first big data problem, and it is what pushed me into software development.

Reporting automation

I build Google Sheets and Google Apps Script automation that feeds Google Data Studio reports. One example is a KPI tracker that normalized caller data into metrics, so a company could decide which call data lists to keep marketing.

Ad account automation

I wrote Google Ads scripts that enforce budgets and clean up search terms. At World Car, those scripts supported ad accounts with $10K - $40K monthly budgets across 13 stores.

No-code workflow automation

When a managed tool is the better fit, I set up Zapier and n8n workflows. Not every problem needs custom code, and I will tell you when yours does not.

Designing around constraints

Good pipelines expect things to go wrong. Three patterns come up on most projects:

• Rate limits. My phone scrubber takes an uploaded CSV, fans it into a rate-limited queue for the Trestle API, and exports a cleaned CSV. The queue keeps the job inside the provider’s limits. Read the phone scrubber write-up.
• Worker failure. My Playwright crawler runs in Docker replicas with a shared proxy pool and recovery for stalled work. See the distributed crawler project.
• Visibility. Health checks and monitoring show when a job stalls, so you hear about it before your customers do.

How a project runs

1. Map the manual process. We write down every step a person takes today.
2. Pick the data source and target. We confirm what is allowed, what is stable, and where the output should land.
3. Build and test. I build with tests around messy input, since real data is always messy.
4. Run and watch. I add logging and alerts so failures are visible.
5. Document. You receive notes on how it runs and how to restart it.

Frequently asked questions

Is web scraping legal?

It depends on the site, the data, and how you use it. I check terms of service and robots rules with you before building, and I do not scrape private or login-protected data without permission. This is not legal advice, so involve your attorney for higher-risk cases.

Which language will you use?

Python for most scraping and data jobs, Go when performance matters, and Google Apps Script for work that lives in Sheets. I choose by the job, not by habit.

Can you automate reports in Google Sheets and Data Studio?

Yes. I have built many Sheets with Apps Script for reporting, and Google Data Studio dashboards on top of them.

What happens when a data source changes?

Sources do change, so I build with monitoring and clear error messages. Ongoing maintenance can be part of a retainer.

How do I know what to automate first?

Start with the task that is repeated most often and follows clear rules. I can help you rank the list.

Share the process you want gone

Tell me what someone does by hand today and how often. I will reply with how I would automate it.

Request a quote or see how engagements work.