After a small hiatus due to exams and job interviews, I wanted to come back with a new project that I’ve been working on for the past few days. Due to the fact that sometime organizations use another platform than the one you use to host your repositories, I wanted to create a unified contributions viewer that would show contributions from both Github and Gitlab.
The idea is to have a github-like contributions graph that would show contributions from both platforms so that you can have a better overview of your contributions. This can also benefit you for job interviews as not only you can show that you contributed to both platforms but you can also demonstrate that your contributions are consistent despite them being low since you are working professionally on another platform.
In the near future, I will probably add Bitbucket to the list as well as other platforms that could be interesting.
The code is open-source and available on Github. I decided to use NestJS as the backend as I wanted to quickly create a REST API that would fetch the contributions from the different platforms and return them in a unified format.
We will also see how exactly you can use this project to update automatically your contributions graph on your Github profile using Github Actions.
Getting contributions from Github and Gitlab
Thankfully, both platforms provide us easy ways to get the contributions of a user.
Github
Github allows us to use it’s graphql API to get the contributions of a user. The explorer also allows us for interactive query testing which is really useful to get and determine the fields we need.
By providing a login and the following query, we can get the contributions of a user:
{
user(login: "mpellouin") {
contributionsCollection {
contributionCalendar {
weeks {
contributionDays {
contributionCount
date
}
}
}
}
}
}
Let’s take a look at what Github’s API returns:
{
"data": {
"user": {
"contributionsCollection": {
"contributionCalendar": {
"weeks": [
{
"contributionDays": [
{
"contributionCount": 0,
"date": "2023-05-14T00:00:00.000+00:00"
},
{
"contributionCount": 6,
"date": "2023-05-15T00:00:00.000+00:00"
},
{
"contributionCount": 8,
"date": "2023-05-16T00:00:00.000+00:00"
},
{
"contributionCount": 3,
"date": "2023-05-17T00:00:00.000+00:00"
},
{
"contributionCount": 0,
"date": "2023-05-18T00:00:00.000+00:00"
},
{
"contributionCount": 2,
"date": "2023-05-19T00:00:00.000+00:00"
},
{
"contributionCount": 0,
"date": "2023-05-20T00:00:00.000+00:00"
}
]
},
{
"contributionDays": [
...
]
},
...
]
}
}
}
}
}
The fact that we are using a login and not the viewer is important as we want to get the contributions of a specific user and not the viewer’s contributions. However, if you want to get the viewer’s contributions, you can use the viewer query instead of the user query. This will not be important as we are using github’s personal access token to get the contributions.
Gitlab
Gitlab provides a route on it’s API to get the contributions directly from the calendar. This is really useful as we don’t need to do any additional processing to get the contributions.
By providing the username and the following route, we can get the contributions of a user:
curl https://gitlab.com/users/mpellouin/calendar.json
{"2023-05-17":5,"2023-05-18":8,"2023-05-20":18,"2023-05-21":8,"2023-05-25":9,"2023-05-26":20,"2023-05-27":1,"2023-05-29":110,"2023-06-01":18,"2023-06-04":18,"2023-06-07":39,"2023-06-08":53,"2023-06-09":25,"2023-06-12":7,"2023-06-13":5,"2023-06-18":6,"2023-06-22":6,"2023-06-30":1,"2023-07-05":2,"2023-07-19":2,"2023-08-30":2,"2023-09-03":3,"2023-09-04":3,"2023-09-25":16,"2023-09-26":1,"2023-09-29":1,"2023-09-30":12,"2023-10-01":41,"2023-10-02":58,"2023-10-03":43,"2023-10-04":1,"2023-10-11":8,"2023-10-16":12,"2023-11-07":10,"2023-11-08":3,"2023-11-09":7,"2023-11-13":1,"2023-11-19":12,"2023-11-20":1,"2023-11-27":9,"2023-11-28":15,"2023-11-29":11,"2023-11-30":15,"2023-12-05":1,"2023-12-06":2,"2023-12-07":1,"2023-12-08":1,"2024-01-01":5,"2024-01-02":25,"2024-01-03":37,"2024-02-04":2,"2024-02-05":1,"2024-02-06":1,"2024-02-25":16,"2024-02-26":9,"2024-03-01":7,"2024-03-04":6,"2024-03-05":2,"2024-03-07":14,"2024-03-15":1,"2024-03-18":2,"2024-03-21":1,"2024-03-27":2,"2024-04-02":5,"2024-04-04":1,"2024-04-05":29,"2024-04-09":13,"2024-04-11":3,"2024-04-12":6,"2024-04-16":8,"2024-04-17":8,"2024-04-18":1,"2024-04-19":1,"2024-04-20":2,"2024-04-28":12,"2024-05-01":20,"2024-05-02":21,"2024-05-03":3,"2024-05-05":13,"2024-05-06":41,"2024-05-07":7}
Once again this toute is public and doesn’t require any authentication as it is the route fetched by the Gitlab’s contributions graph.
Creating the unified contributions viewer
After fetching the contributions from both platforms, I decided to create two different endpoints on the backend to get or visualize the contributions.
The first endpoint is /contributions?github_id=<github_id>&gitlab_id=<gitlab_id> which will return the contributions of the user from both platforms as a JSON object.
The second endpoint is /contributions/heatmap?github_id=<github_id>&gitlab_id=<gitlab_id> which returns the contributions as a heatmap in a github fashioned way.
However, since I had never built heatmaps outside of python and the matplotlib and seaborn libraries, I had to do some research on how to create a heatmap in javascript. And then I decided to go with Plotly as it is a really powerful library that allows us to create a lot of different types of graphs.
Creating the heatmap
To create the heatmap we first need to merge the contributions from both platforms into a single object. After normalizing the contributions, we can do this by using the following function of the ContributionsService:
computeTotalContributions(contributionsA: NormalizedContributions, contributionsB: NormalizedContributions): NormalizedContributions {
const data = { ...contributionsA };
for (const date in contributionsB) {
if (data[date]) {
data[date] += contributionsB[date];
} else {
data[date] = contributionsB[date];
}
}
return data;
}
This allows to add the contributions of the second object to the first object while making sure to not override the contributions of the first object.
Now we need to understand how to create a heatmap with Plotly. By looking at the examples and the documentation, we quickly understand that if we want to create a heatmap like the one on Github, we need to create a 2D array where each row is a day and each column is a week. This is because Plotly will accept the data just like it is displayed.
Therefore, since in our intended design (shown below) is expected to have each row as a day and each column as a week, we need to rearrange the data to fit this format.
This is done by the following function of the ContributionsService:
computeHeatmapData(data: NormalizedContributions): (number | null)[][] {
const contributionsPerDay = [[], [], [], [], [], [], []];
Object.keys(data).forEach((date) => {
const testDate = parse(date, 'yyyy-MM-dd', new Date());
const day = testDate.getDay();
if (data[date] === 0) contributionsPerDay[day].push(null);
else
contributionsPerDay[day].push(data[date]);
});
// Normalization of data so that the heatmap is correctly displayed
const firstContributionDay = parse(Object.keys(data)[0], 'yyyy-MM-dd', new Date()).getDay();
const lastContributionDay = parse(Object.keys(data)[Object.keys(data).length - 1], 'yyyy-MM-dd', new Date()).getDay();
for (let i = 0; i < firstContributionDay; i++) {
contributionsPerDay[i].unshift(null);
}
for (let i = lastContributionDay + 1; i < 7; i++) {
contributionsPerDay[i].push(null);
}
return contributionsPerDay.reverse();
}
Let me further break down the function:
- We create an array of 7 arrays that will represent the days of the week.
- We iterate over the contributions and add them to the correct day of the week while making sure to add
nullif there are no contributions. This is key since if we do not addnull, the heatmap will not be displayed correctly or some squares will be displayed as if we had contributions. - We then need to “normalize” the data so that the heatmap is correctly displayed. Since the contributions retrieved from the platforms date up to 365 days, if the first date retrieved is a Tuesday, then the displayed data would be Sunday, Monday from week 2 followed by the other days from week 1, and this would continue up to the current week. That is why we need to add some
nullvalues to the days that are not in the first week and the last week.
From there, we have the data needed to create the heatmap with the createHeatmap method:
async createHeatmap(data: NormalizedContributions, githubId: string, gitlabId: string): Promise<string> {
let returnURL = '';
const contributionsPerDay = this.computeHeatmapData(data);
const totalContributions = Object.values(data).reduce((acc, val) => acc + val, 0);
const heatmapData = [
{
z: contributionsPerDay,
y: ['Sat', 'Fri', 'Thu', 'Wed', 'Tue', 'Mon', 'Sun'],
x: this.generateDateLegend(),
type: 'heatmap',
colorscale: 'Greens',
xgap: 2,
ygap: 2,
}
]
const layout = {
title: {
text: `Github user ${githubId} and Gitlab user ${gitlabId} total contributions: ${totalContributions}`,
font: {
color: '#EEEEEE'
}
},
yaxis: {
tickvals: [0, 2, 4, 6],
color: '#EEEEEE'
},
xaxis: {
dtick: 2,
color: '#EEEEEE'
},
width: 1000,
height: 300,
xgap: 1,
paper_bgcolor: '#31363F',
plot_bgcolor: '#222831',
font: {
color: '#EEEEEE'
}
}
const graphOptions = {
fileopt: 'overwrite',
filename: githubId + '_' + gitlabId,
layout
};
await new Promise((resolve, reject) => {
this.plotly.plot(heatmapData, graphOptions, function (err, msg) {
if (err) {
reject(err);
} else {
resolve(msg.url + '.png');
}
});
}).then((url: string) => {
returnURL = url;
}).catch((error) => {
console.error(error);
throw new InternalServerErrorException("Error while creating heatmap");
});
return returnURL;
}
This method will create the heatmap and return the URL of the heatmap that can then be used to display the heatmap on some frontend, or to automatically update the contributions graph on Github.
Automatically updating the contributions graph on Github with Github Actions
The final idea I had was to automatically update some README.md file on a repository with the heatmap of the contributions. Let’s see how we can do just that by setting up a cron job with Github Actions.
What is a cron job?
A cron job is used to schedule tasks to run periodically at fixed times or intervals. It is commonly used to automate system maintenance or administration. However, we can also use it to automate the update of the contributions graph on Github.
A cron expression is a string that consists of at least five fields separated by white spaces. It looks like this:
* * * * * *
where each one of the fields represents a different time unit. The fields are as follows:
- Minutes
- Hours
- Day of the month
- Month
- Day of the week
Therefore the cron expression 0 0 * * * would run the task every day at midnight while the cron expression 0 0 * * 0 would run the task every Sunday at midnight or while the cron expression */5 * * * 0 would run the task every 5 minutes on Sunday.
Our Github Actions workflow
Setting up the workflow
To set up the workflow, we need to create a .github/workflows directory at the root of the repository and create a update-contributions.yml file in it.
The workflow will be divided in 4 main steps:
- Checkout the repository
- Install and run the backend
- Create the heatmap
- Download the heatmap to the repository
name: Update README Image
on:
schedule:
- cron: "0 0 * * *"
jobs:
update_readme_image:
runs-on: ubuntu-latest
environment: CI
steps:
- name: Checkout repository
uses: actions/checkout@v2
- name: Start API server
run: |
yarn --frozen-lockfile
yarn start &
sleep 10
env:
PLOTLY_USER: ${{ secrets.PLOTLY_USER }}
PLOTLY_API_KEY: ${{ secrets.PLOTLY_API_KEY }}
GITHUB_TOKEN: ${{ secrets.TOKEN }}
- name: Api check
run: |
curl http://localhost:3000
- name: Generate image link and download
run: |
GITHUB_ID="mpellouin"
GITLAB_ID="mpellouin"
API_URL="http://localhost:3000"
IMAGE_URL="$(curl -s -X GET "$API_URL/contributions/heatmap?github_id=$GITHUB_ID&gitlab_id=$GITLAB_ID")"
echo $IMAGE_URL
wget $IMAGE_URL -O new_image.jpg
- name: Replace old image in repository
continue-on-error: true
run: |
IMG_PATH="assets/README_image.jpg"
GITHUB_USER="mpellouin"
cp new_image.jpg $IMG_PATH
git config --global user.name $GITHUB_USER
git config --global user.email '$GITHUB_USER@users.noreply.github.com'
git add $IMG_PATH
git commit -m "Automatic update of README image"
git push
As you can see we asks Github to run the workflow every day at midnight. The workflow will then checkout the repository, install the backend, create the heatmap, and download the heatmap to the repository. However, those steps will not work out unless we setup secrets and unless you modify the GITHUB_ID, GITLAB_ID and GITHUB_USER variables.
As to why last step continue-on-error: true is used, it is because the workflow will fail if there is no change to commit. It is realistic that on some days, you will not have produced any contributions AND the day 365 days ago was contributionless as well. Therefore, the heatmap state will change from inserting a null value to “normalize” the null value is it is a day more than 365days ago or a future day. This will result in the heatmap being the same as the previous day and therefore no change to commit.
Even though there is no change to commit, the workflow did what it was supposed to do and this problem arises solely because no contributions were made. Therefore, we can safely ignore this error.
Let’s set-up the secrets now so that our workflow is complete:
Setting up the secrets
To set up the secrets, you need to go to the repository settings and then to the secrets tab. There create an environment called CI as we specified in the workflow and add the following secrets:
PLOTLY_USER: Your plotly usernamePLOTLY_API_KEY: Your plotly API key to generate both of them, head to the plotly chart studio settings.TOKEN: A github-personal access token. Click here to know more
Once all of these are set-up, TADA! You have a cron job that will update the contributions graph on your Github profile every day at midnight.
Conclusion
In this post, we saw how to create a unified contributions viewer from Github and Gitlab. We also saw how to automatically update the contributions graph on your Github profile using Github Actions and cron jobs. I hope this post was useful and that you will be able to use this project or what you learned in your own projects. If you have any questions or suggestions, feel free to leave a comment below. I will be happy to answer them.
In the meantime, be sure to check out and star the repository on github!, and I will see you in the next post!