While GitHub Copilot offers significant advantages, it is essential to understand its limitations and accurately measure its impact on development processes. This unit explores the constraints of AI-assisted coding and presents methods for quantifying GitHub Copilot’s influence on productivity.

Identifying GitHub Copilot’s limitations

GitHub Copilot is a powerful tool, but it has certain limitations that developers should be aware of:

Code quality and accuracy

• Potential for errors: GitHub Copilot may sometimes suggest code that contains bugs or does not fully meet requirements.
• Security issues: Generated code may not always follow best security practices, requiring careful review.
• Misinterpretation of context: It may misunderstand the overall context, leading to inappropriate suggestions.

Language and framework specificity

• Variable performance: GitHub Copilot’s effectiveness can vary depending on the programming languages and frameworks used.
• Niche technologies: For less common or newer technologies, suggestions may be less accurate or relevant.

Dependence on training data

• Bias in suggestions: Suggestions reflect patterns found in training data, which may include biases or outdated practices.
• Copyright concerns: Generated code raises ongoing debates about intellectual property implications.

Solving complex problems

• Limitations in high-level design: GitHub Copilot performs well on low-level tasks but may not grasp complex architectural decisions.
• Creativity constraints: While helpful, it cannot replace human creativity in solving novel problems.

Measuring productivity gains

Understanding the productivity gains offered by GitHub Copilot is essential to maximize its benefits. The GitHub Copilot usage metrics REST API and the GitHub Copilot developer survey provide powerful ways to measure and analyze its influence on your workflow.

Using REST API endpoints for usage metrics

GitHub provides a REST API to access GitHub Copilot usage metrics for enterprise members, teams, and organizations. These metrics offer insights into daily usage of GitHub Copilot, including completions, chat interactions, and user engagement by editor and language.

Getting a summary of GitHub Copilot usage for enterprise members
Endpoint: GET /enterprises/{enterprise}/GitHub Copilot/usage
This endpoint provides a daily breakdown of aggregated usage metrics for completions and GitHub Copilot Chat interactions across all users in an enterprise. It includes details on suggestions, acceptances, and active users, with breakdowns by editor and language.

curl -L \
  -H "Accept: application/vnd.github+json" \
  -H "Authorization: Bearer <YOUR-TOKEN>" \
  https://api.github.com/enterprises/ENTERPRISE/GitHub Copilot/usage

curl -L \
  -H "Accept: application/vnd.github+json" \
  -H "Authorization: Bearer <YOUR-TOKEN>" \
  https://api.github.com/enterprises/ENTERPRISE/GitHub Copilot/usage

Response:
Status code: 200 OK
Body: JSON table containing daily metrics, including suggestions, acceptances, active users, as well as a breakdown by editor and language.

Get a summary of GitHub Copilot usage for an enterprise team

Endpoint: GET /enterprises/{enterprise}/team/{team_slug}/GitHub Copilot/usage
This endpoint provides a daily breakdown of aggregated usage metrics for GitHub Copilot completions and interactions with GitHub Copilot Chat within a specific enterprise team.

Example request:

curl -L \
  -H "Accept: application/vnd.github+json" \
  -H "Authorization: Bearer <YOUR-TOKEN>" \
  https://api.github.com/enterprises/ENTERPRISE/team/TEAM_SLUG/GitHub Copilot/usage

curl -L \
  -H "Accept: application/vnd.github+json" \
  -H "Authorization: Bearer <YOUR-TOKEN>" \
  https://api.github.com/enterprises/ENTERPRISE/team/TEAM_SLUG/GitHub Copilot/usage

Response:
Status code: 200 OK
Body: JSON table containing the team’s daily metrics, including suggestions, acceptances, active users, as well as a breakdown by editor and language.

Get a summary of GitHub Copilot usage for organization members

Endpoint: GET /orgs/{org}/GitHub Copilot/usage
This endpoint provides a daily breakdown of aggregated usage metrics for GitHub Copilot completions and interactions with GitHub Copilot Chat across an entire organization.

Example request:

curl -L \
  -H "Accept: application/vnd.github+json" \
  -H "Authorization: Bearer <YOUR-TOKEN>" \
  https://api.github.com/orgs/ORG/GitHub Copilot/usage

curl -L \
  -H "Accept: application/vnd.github+json" \
  -H "Authorization: Bearer <YOUR-TOKEN>" \
  https://api.github.com/orgs/ORG/GitHub Copilot/usage

Response:
Status code: 200 OK
Body: JSON table containing the organization’s daily metrics, including suggestions, acceptances, active users, as well as a breakdown by editor and language.

Implementing a measurement framework

To systematically assess the impact of GitHub Copilot, you can adopt the following framework, using the GitHub Copilot usage metrics API at each stage:

1. Evaluation

During the initial adoption phase of GitHub Copilot, focus on early indicators such as developer satisfaction and task completion rates. Use the API to collect metrics like:

• Average number of daily active users
• Overall acceptance rate
• Number of accepted lines of code

2. Adoption

As GitHub Copilot becomes more integrated into your team’s workflow, continue monitoring productivity and engagement indicators. The API can provide insights into user engagement and help identify training needs.

3. Optimization

Once GitHub Copilot is fully adopted, use the REST API to refine its impact on broader organizational goals, such as reducing time-to-market or improving code quality.

4. Sustained Efficiency

Continuously evaluate GitHub Copilot’s effectiveness as your organization evolves. The API enables regular tracking and adjustments to ensure long-term productivity gains.

Using the GitHub Copilot Developer Survey

The GitHub Copilot Developer Survey is a valuable tool designed to gather feedback from your teams. It helps you understand how GitHub Copilot is being used, its benefits, and the challenges encountered. The survey comes in two formats: short and long, each suited to different stages of evaluation and adoption.

1. Frequency and Timing

To avoid fatigue while collecting useful data:

• Short survey: Can be conducted every two weeks to gather frequent feedback, especially alongside other channels (online or in-person discussions).
• Long survey: Should be conducted no more than once every four weeks, especially at the end of evaluation and adoption phases, to gather in-depth feedback.

2. Structuring the Survey

Questions should be tailored to your organization’s specific needs to ensure the relevance of the data collected.

• Short survey: Aims to gather immediate feedback on satisfaction, challenges, and time saved or lost.
  Example questions:
  • “How would you feel if you could no longer use GitHub Copilot?”
  • “When I use GitHub Copilot, I enjoy coding more / I write better quality code / I complete tasks faster.”
  • “What challenges have you encountered with GitHub Copilot since the last survey?”
• Long survey: Provides a deeper analysis of GitHub Copilot’s impact, usage, and effects on team dynamics.
  Example questions:
  • “I use GitHub Copilot to code in a familiar language / explore a new language / write repetitive code.”
  • “Thanks to GitHub Copilot, my team conducts better code reviews / merges code to production faster.”
  • “What challenges have you encountered with GitHub Copilot since the last survey?”

3. Analyzing Results

Once surveys are completed, analyze the results in a structured way:

• Privacy compliance: Ensure responses are anonymized and cannot be traced back to individuals.
• Data tracking: Integrate responses into your Business Intelligence (BI) tools or spreadsheets for easier analysis. Track trends over time to make informed decisions.

4. Continuous Improvement

Use insights from the surveys to adjust GitHub Copilot deployment in your organization. Focus on resolving identified issues, amplifying reported benefits, and optimizing tool usage to maximize productivity.

The GitHub Copilot Developer Survey is a key element in understanding and improving the tool’s usage within your teams.

By combining the REST API and the developer survey, you move beyond subjective impressions to obtain concrete data on GitHub Copilot’s impact. This data-driven approach enables informed decision-making and continuous improvement of its role in the development workflow.