What happens when employees at some of the world’s largest organizations like Microsoft and RedHat use personal GitHub repos for their side projects? They can unknowingly expose corporate secrets and credentials opening the doors for a security incident. Unfortunately, this isn’t just a hypothetical situation.
In a recent study, we explained how we analyzed Kubernetes configurations and found in them many exposed secrets. To emphasize the gravity of these exposed secrets we will talk about the major corporations that were at risk from them. Surprisingly, in most cases, these secrets were not found in an official organization repository, but in a personal employee’s public GitHub repo, demonstrating a case of cloud native Shadow IT.
A significant discovery we reported to Microsoft was a privileged Azure Container Registry Token that granted unauthorized access to several Internal Azure projects, including Azure IoT Edge, Akri, Apollo, and others. This token had the potential to retrieve or overwrite private images, posing risks to both internal operations of Microsoft and Azure users.
In addition, we identify similar security incidents for other companies like RedHat, Tigera and more, where personal public repositories of current or former employees contained sensitive company secrets. This blog delves into these cases, highlighting the perils of personal repositories becoming corporate nightmares.
What is Shadow-IT
“Shadow IT” refers to the use of IT systems, devices, software, applications, and services without explicit IT department approval within an organization, and worse without any proper security controls that monitor these IT resources. Traditionally, this includes scenarios such as employees using personal data storage like OneDrive, Dropbox, Google Drive for work documents or installing unauthorized software on company computers. However, in the context of cloud native development, this term should be extended to employees’ personal source code management platforms (for instance GitHub).
The risk of Shadow IT on GitHub is significant because threat actors can easily crawl exposed sensitive data on public GitHub repositories using GitHub’s regex search feature. While corporate official repositories are usually scanned with various approaches and tools for sensitive data, employees’ personal repositories are not. Thus, each employee’s personal repository has the potential to extend the attack surface to the organization.
Past research recap and research insights
In past research, we searched for Kubernetes configuration files (dockerconfigjson and dockercfg) in public GitHub repositories, while we focused on ones that contained encoded (base64) secrets. Many of the encoded secrets we found contained usernames and passwords to container registries, which are used by the k8s cluster to pull container images.
During our research, we were able to connect these registries to companies in all sizes. We then noticed that many of the secrets were actually found in personal repositories rather than official company repositories, which might explain why they had not been discovered and rotated earlier.
After a thorough analysis, it turned out that 66% of the valid tokens of companies that we found were under personal employee GitHub repositories or with external contractors of the companies. It is important to note that while this research and findings focus on Kubernetes in the context of Shadow IT, they could apply to any kind of sensitive data.
For example, in another research we conducted on exposed artifact management repositories, we found that 83% of the exposed repositories that we reported to security teams were categorized by them as Shadow IT. The security teams categorized these assets as either personal, test, staging or other negligible types of repositories, that weren’t monitored by the IT security teams.
In many cases, the security teams were under the impression that these repositories don’t pose any threat to the organization, while in reality they contained secrets to other environments, sensitive information, proprietary or sensitive software components or all the above. In one particular case, the organization tested the artifact repository software, and left an admin token to their cloud service provider. This illustrates the potential blast radius of Shadow IT in cloud native environments.
New Shadow IT Findings
To demonstrate the risks of Shadow IT, we will detail a few examples of secrets we discovered that, in some cases, could lead to serious supply chain attacks against companies.
Microsoft Azure
During our research, we found credentials within a git commit by a Microsoft employee, which granted us access to an internal Azure Container Registry used by Azure. This registry contains images critical to various Azure projects, such as Azure IoT Edge, Akri, and Apollo. The exposed token provided privileged access, allowing us to download private images and upload/overwrite images. The potential implications of this are significant – an attacker could leverage this access to overwrite existing images and run malicious code within the internal Azure environment, potentially impacting both Azure and its customers across various organizations.
Figure 1 – Login succeeded to the internal Azure Registry
Figure 2 – Listing of existing container images – as you can see, we were able to push an image, meaning we have high privileges
We reported this issue to Microsoft, which then promptly invalidated the token, deleted the employee’s commit, and assigned this security incident an important severity.
RedHat
During our investigation, we discovered several instances where Red Hat employees unintentionally exposed tokens for internal Red Hat container registries containing highly sensitive data linked to vital corporate functions. These tokens grant both pull and push privileges, posing substantial risks to the company. If exploited by a threat actor, this could result in the leakage of sensitive or proprietary information. Moreover, the capability to push changes is a critical security issue that could potentially facilitate a supply chain attack.
Figure 3 – Access to private registries of Red Hat.
We reported these issues to Red Hat, which then promptly invalidated the token. Red Hat also reviewed their internal credentials. During our research, we found many tokens from Red Hat and Quay registries and reported them to Red Hat, who took the initiative to inform the relevant owners of all exposed credentials we had provided.
Tigera
During our research, we discovered credentials for the internal container registry (quay.io/tigera) exposed in a Git commit of other company. This registry contains images from various Tigera projects, such as Calico, and more. We reported this issue to Tigera, who then promptly invalidated the token on the same day. Following a thorough investigation, it was confirmed that, since it was a scoped token, there was no risk posed to Tigera or their software.
Figure 4 – Login succeeded to the Tigera Quay Registry
Conclusions and mitigations
In this blog, we have explored how personal GitHub repositories can pose significant security risks. Our research reveals that personal repositories often expose sensitive corporate data, leading to severe security breaches at major companies. These findings underscore the urgent need for organizations to enforce robust security measures against this type of threat.
We would also like to suggest some mitigations:
- Regularly scan the internet for exposed environments or secrets, for instance you can scan GitHub for unique keys or markers your organization uses. If your artifact repository is hosted on ‘repo.mydomain.com/artifacts’, you can look for any exposed sensitive code that contains this URL.
- Encourage your employees, contractors, or code contributors to regularly scan the personal accounts they use that may contain any corporate secrets or misconfigurations that could lead to security breach.
- Implement Least Privilege with Scoped Keys: Ensure that all keys provided to access resources are scoped—i.e., they are limited to only the privileges necessary for specific tasks.
- Limiting the lifespan of secrets by setting expiration dates.
- Anomaly Detection: Monitor and scan for anomalies in connections, work patterns, and the push/pull of images. Unusual activity could indicate a security breach or misuse of resources, which should be promptly investigated.
- Secure Kubernetes Secrets: Avoid uploading Kubernetes secret manifests in plaintext or encoded forms to public repositories. Utilize tools such as Sealed Secrets or Mozilla SOPS, which allow you to encrypt secrets before they are uploaded, ensuring that they remain secure.
- Utilize Aqua’s open source tools or commercial offering to scan your repositories for secrets; you can also add your employees’ personal public repositories, to better secure their repositories and avoid exposed corporate secrets. Aqua Trivy can be utilized to find encoded (
base64) Kubernetes configuration secrets, Kubernetes solutions can be utilized to detect exposed APIs and many more features in our commercial offering can help you mitigate these risks.
