Solution to error: failed to push some refs on GitHub Actions¶
One of my favorite things to do with GitHub (specifically GitHub Actions) is writing scrapers that automatically update CSV files.
Sometimes, though, you'll run into the error error: failed to push some refs. It might look something like this:
Run git config user.name "Automated"
[main 0cc3a28] Latest data: Tue Jan 3 17:27:58 UTC 2023
3 files changed, 1300 insertions(+)
To https://github.com/jsoma/sample-repo
! [rejected] main -> main (fetch first)
error: failed to push some refs to 'https://github.com/jsoma/sample-repo
hint: Updates were rejected because the remote contains work that you do
hint: not have locally. This is usually caused by another repository pushing
hint: to the same ref. You may want to first integrate the remote changes
hint: (e.g., 'git pull ...') before pushing again.
hint: See the 'Note about fast-forwards' in 'git push --help' for details.
Error: Process completed with exit code 1.
Technically the Updates were rejected because the remote contains work that you do not have locally part is the useful/descriptive part of the error message, but oddly no one seems to ever read that far! Let's just stick with failed to push some refs.
What the failed to push some refs error means¶
When you're running an automatic scraper with GitHub Actions, there are four steps that have to happen:
- Copy the repository with
git clone - Scrape/update/change the files with
python scraper.py(or whatever the file is) - Commit the changes with
git commit - Push the data to the repository with
git push
You don't think about it too much, though: these all happen in your workflow's yml file, and you probably only changed the scraping script.
The failed to push some refs error comes up in the last step, when your Action tries to run git push. The error typically means there are updates in the "live" repository that aren't in your "copied" repository.
This almost always happens when you tell your scraper to run multiple times, with each run quickly after another. Maybe you keep clicking "run" a few times, or it ran automatically when you pushed and then you forced it to run again, or anything else like that.
The first job clones the repo, scrapes, commits and pushes, no problem.
The second job clones the repo at around the same time as the first job, but finishes later. This means the data from the first job is already in the repo when the second job tries to git push. GitHub tells your second job "hey hey, you aren't working from the most recent updates!", your job panics, and it just gives up and throws an error.
How to fix the failed to push some refs error¶
You have two approaches to solve this problem.
First, you can edit the yml workflow file to run git pull before it runs git push. This will allow git to combine the first job and the second job's updates without any trouble.
- name: Commit and push any changes
run: |-
git config user.name "Automated"
git config user.email "actions@users.noreply.github.com"
git add -A
timestamp=$(date -u)
git commit -m "Latest data: ${timestamp}" || exit 0
git pull
git push
Second, you can... just ignore it! All of the data the second job captured was probably already captured by the first, so do you really need to care about when the conflict happens?