Whether it’s building, testing, or deploying code, automating manual processes is key to improving developer experience and achieving a successful DevOps strategy.

On GitHub, you can use GitHub Actions to not only implement your CI/CD pipeline, but also automate other processes both on and off GitHub. When you are adopting GitHub Actions on GitHub Enterprise Cloud, you can choose between GitHub-hosted runners and self-hosted runners to run your workloads, and each has its pros and cons.

In this post, we’ll compare GitHub-hosted runners with self-hosted runners across five areas to help you determine which type best fits your GitHub Actions adoption strategy.

What are GitHub-hosted runners and self-hosted runners?

GitHub-hosted runners and self-hosted runners are based on the same open-source software and both support macOS, Windows, and Linux. But they have many differences.

GitHub-hosted runners are fully managed on GitHub’s infrastructure using pre-configured Windows, Linux, and macOS virtual machines. In addition to offering standard runners for typical workloads, hosted runners offer larger runners with more resources (memory, CPU, and storage), custom images, static IP ranges, and Azure Virtual Network integration for enhanced security control.

Self-hosted runners operate on your own infrastructure, whether on-premises or in the cloud. You manage all aspects—configuration, security, and scaling. They also allow you to operate runners in places you couldn’t otherwise—for example, on GitHub Enterprise Server or on custom hardware. They can also be the only way to implement certain compliance requirements, especially when working with highly secured systems.

Both options offer distinct advantages depending on your specific needs and resources. Let’s explore when GitHub-hosted runners may be the right choice for your projects, and when it may be better to use self-hosted runners.

Fully managed or self-managed?

A key distinction between these two options is where they’re hosted, as we’ve pointed out. But that choice comes with several implications.

GitHub-hosted runners provide managed infrastructure with pools of on-demand virtual machines (VMs) that are automatically secured and updated. The environments are ephemeral, with the disks reimaged after each job, preventing files from previous jobs from affecting subsequent runs. The VMs are optimized for GitHub Actions, with pre-installed software and tools, including the latest versions of GitHub CLI, Docker, and common development platforms to ensure fast start times and avoid rate limits.

With GitHub-hosted runners, you can jump right in and start building workflows. There’s nothing to configure or secure before you start, making them ideal when you want to get started quickly. And we all prefer to spend more time on code than infrastructure, right?

Self-hosted runners offer you complete flexibility in defining your solution, but also means you are responsible for managing the infrastructure, images, caches, and security, and monitoring availability and usage against GitHub’s rate limits. This requires expertise in GitHub Actions architecture, VM and container image building, and network and infrastructure management. If your core business offering is scalable infrastructure solutions or Kubernetes, self-hosted runners may make sense.

Let’s take a closer look.

Scalability

To remain productive, it’s important to have highly-available resources available on demand, especially for CI/CD workloads, where waiting for a job to run may mean you’re blocked from working on other tasks. In fact, a single wasted hour each week can cost a company over $4,000 a year per developer!

But scaling highly available, on-demand resources is hard. Even with a well-designed cloud infrastructure, it takes time to provision new virtual machines. You need systems in multiple regions to maintain up time, with 20-25% spare capacity to scale quickly and handle unexpected system failures.

GitHub-hosted runners take advantage of Microsoft’s deep data center and cloud expertise and have dedicated teams to meet our service level agreement (SLA) of 99.9% availability. And that’s without any expertise on your part. In fact, many teams consider self-hosted runners in hopes of beating this availability, but it turns out that’s not even technically possible, as all runnings depend on the same services and control plane. That said, there are conditions where self-hosted runners may work for you.

Self-hosted runners may meet your needs if you need a fixed number of servers, are primarily focused on deployment to non-cloud resources, and don’t need to scale on demand. Just remember that the instances are not natively ephemeral, so you’ll need to have a strategy to keep the instances free from artifacts created by earlier runs. Self-hosted runners also lack automatic scaling capabilities; they require a scaling solution to be able to support large teams or create new instances dynamically.

GitHub’s Actions Runner Controller (ARC) offers a solution, but it has limitations as it requires Kubernetes expertise and only supports Linux runners. Kubernetes relies on containers instead of VMs, which can require you to troubleshoot resource contention and scaling issues. ARC can also offer high availability by having multiple clusters. As we noted before, if your primary business is hosting and managing Kubernetes clusters, then ARC may be the right approach.

ARC does not support macOS or Windows workloads, and both environments present a number of limitations. For example, on macOS, you are required to use Apple hardware, you are limited to two VMs per machine, and containerizing the Apple runtime is not supported. For Windows, virtual machines are supported, but you need a custom orchestrator for scaling the instances. While you can create Windows containers and manage them with Kubernetes, the containers have slow startup times and may not support some of the necessary development and testing tools.

In short, we recommend GitHub-hosted runners for both macOS and Windows workloads.

Security

Security is critical for CI/CD processes, since they may require access to internal or production resources, and builds often use third-party libraries, runtimes, and tools, which can create a large attack surface if not properly secured.

GitHub-hosted runners provide built-in security through a defense-in-depth, zero-trust approach. VMs provide network isolation, preventing exposure to other runners and corporate resources. In fact, access to corporate or cloud resources requires elevating privileges (we recommend OIDC). Their ephemeral nature eliminates code persistence and prevents application execution after job completion, reducing unauthorized access risks.

Storage disks for hosted runners are encrypted at rest, ensuring the code is protected on the disk. All communications are encrypted to GitHub, and deployments to Microsoft Azure are routed through the Azure backbone, minimizing transits through the public internet. We provide regular security updates to both operating systems and runner software. The minimized attack surface and reduced risk of security breaches are key factors in the Department of Defense DevSecOps Reference Design’s recommendation to prefer GitHub-hosted runners for workloads up to Impact Level 5.

Self-hosted runners shift security responsibility entirely to you, requiring management of network, infrastructure, images, containers, and caches—that’s a lot of work. You also need to keep everything up to date, as runners connected to GitHub Enterprise Cloud will not be able to connect if they are more than 30 days behind the current release.

Not to mention, if you operate runners within your network environment with access to corporate resources and production environments, you’ll want to implement a zero-trust, defense-in-depth strategy with time-limited resource access, which demands a high level of network security expertise.

Finally, you’ll need to implement and keep updated both a tool cache and an Actions archive cache. Otherwise, you’re likely to encounter our rate limits as you scale up.

Troubleshooting

Keeping you productive means that problems with workflows or jobs—lack of resources, network issues, outages—need to be solved quickly. As a result, it’s important to have a support strategy.

GitHub-hosted runners come with 24/7 support across all time zones, with premium plans offering dedicated reliability engineers and rapid 30-minute response times for critical issues. This eliminates the need for infrastructure troubleshooting on your part. GitHub handles all runner environment issues, from performance problems to queue times, letting you focus on development while we roll up our sleeves, figure out the problems, and get them fixed.

Self-hosted runners, however, shift first-level support responsibility to you, which means someone will have to troubleshoot performance, network, or queueing issues when they happen, leaving less time for the fun coding stuff. 🙁

Not only that, but GitHub can only assist with the Actions service itself; we cannot assist with your infrastructure, Kubernetes clusters, or custom orchestration solutions. So if they figure out the issue is with your system, you’ll be on your own to solve it. Without sufficient planning, you can spend a lot of time waiting for a solution that lets you get back to writing and deploying code. That can be a big price to pay for self-hosted runners.

Cost management

Finally, there’s the issue of cost. If you are offering Kubernetes or infrastructure management solutions, self-hosted runners may have some advantages. If not, then GitHub-hosted runners are likely the answer here too.

GitHub-hosted runners operate on a pay-as-you-go model with no upfront costs or commitments. Teams optimize expenses through workflow improvements and appropriate runner selection. In addition, there are built-in cost savings. For example, GitHub doesn’t charge network egress fees—a significant advantage when working with large container images on cloud platforms. GitHub also has a partnership with Docker that allows unlimited image pulls from Docker Hub by GitHub-hosted runners, which often eliminates the need to create a pass-through registry or purchase business licenses for your CI/CD processes. Maintaining, supporting, and securing the environment is handled by GitHub, avoiding additional staff and service expenses. Finally, Enterprise accounts benefit from 50,000 free monthly minutes for standard runners.

Self-hosted runners, as in other areas, means organizations assume responsibility for all infrastructure, network, storage, security, and support costs. This gives you a lot of flexibility in defining the environment, right-sizing your resources, and customizing the networking. While per-minute virtual machine expenses might initially seem lower, the total ownership cost can (and often does) exceed GitHub-hosted solutions when accounting for these additional support costs.

Which runner is best for you?

Choosing the right runner depends on your specific needs. Self-hosted runners are most suitable when using GitHub Enterprise Server (which lacks hosted runners), if your core business involves managing infrastructure or Kubernetes, or when you have compliance requirements not met by GitHub Enterprise Cloud with data residency. Scaling and ephemerality challenges make self-hosting less ideal for Windows and macOS workloads. If self-hosting is necessary, consider a hybrid approach and use self-hosted runners just for the specific workloads where they are needed.

For most developers and the vast majority of scenarios, unless you have very unique requirements or are willing to deeply invest in infrastructure to keep your CI/CD system humming, GitHub-hosted runners are likely your best option. They’re especially beneficial for those new to GitHub Actions and they let you spend your time focused on business value, new ideas, and writing code—instead of managing runners.

The post When to choose GitHub-Hosted runners or self-hosted runners with GitHub Actions appeared first on The GitHub Blog.

In today’s rapidly evolving field of machine learning (ML), the efficiency and reliability of deploying models into production environments have become as crucial as the models themselves. Machine Learning Operations, or MLOps, bridges the gap between developing machine learning models and deploying them at scale, ensuring that models are not only built effectively but also maintained and monitored to deliver continuous value.

One of the key enablers of an efficient MLOps pipeline is automation, which minimizes manual intervention and reduces the likelihood of errors. GitHub Actions on Arm64 runners, now generally available, in conjunction with PyTorch, offers a powerful solution to automate and streamline machine learning workflows. In this blog, we’ll explore how integrating Actions with Arm64 runners can enhance your MLOps pipeline, improve performance, and reduce costs.

The significance of MLOps in machine learning

ML projects often involve multiple complex stages, including data collection, preprocessing, model training, validation, deployment, and ongoing monitoring. Managing these stages manually can be time-consuming and error-prone. MLOps applies the principles of DevOps to machine learning, introducing practices like Continuous Integration (CI) and Continuous Deployment (CD) to automate and streamline the ML lifecycle.

CI and deployment in MLOps

CI/CD pipelines are at the heart of MLOps, enabling the seamless integration of new data and code changes, and automating the deployment of models into production. With a robust CI/CD pipeline defined using Actions workflows, models can be retrained and redeployed automatically whenever new data becomes available or when the codebase is updated. This automation ensures that models remain uptodate and continue to perform optimally in changing environments.

Enhancing performance with Arm64 runners

Arm64 runners are GitHub-hosted runners that utilize Arm architecture, providing a cost-effective and energy-efficient environment for running workflows. They are particularly advantageous for ML tasks due to the following reasons:

• Optimized performance: Arm processors have been increasingly optimized for ML workloads, offering competitive performance in training and inference tasks.
• Cost efficiency: Arm64 runners are priced 37% lower than GitHub’s x64 based runners, allowing you to get more workflow runs within the same budget.
• Scalability: easily scalable within Actions, allowing you to handle growing computational demands.

Arm ❤️ PyTorch

In recent years, Arm has invested significantly in optimizing machine learning libraries and frameworks for Arm architecture. For instance:

• Python performance improvements: collaborations to enhance the performance of foundational libraries like NumPy and SciPy on Arm processors.
• PyTorch optimization: contributions to the PyTorch ecosystem, improving the efficiency of model training and inference on Arm CPUs.
• Parallelization enhancements: enhancements in parallel computing capabilities, enabling better utilization of multi-core Arm processors for ML tasks.

These optimizations mean that running ML workflows on Arm64 runners can now achieve performance levels comparable to traditional x86 systems, with cost and energy efficiency.

Automating MLOps workflows with Actions

Actions is an automation platform that allows you to create custom workflows directly in your GitHub repository. By defining workflows in YAML files, you can specify triggers, jobs, and the environment in which these jobs run. For ML projects, Actions can automate tasks such as:

• Data preprocessing: automate the steps required to clean and prepare data for training.
• Model training and validation: run training scripts automatically when new data is pushed or when changes are made to the model code.
• Deployment: automatically package and deploy models to production environments upon successful training and validation.
• Monitoring and alerts: setup workflows to monitor model performance and send alerts if certain thresholds are breached.

Actions offer several key benefits for MLOps. It integrates seamlessly with your GitHub repository, leveraging existing version control and collaboration features to streamline workflows. It also supports parallel execution of jobs, enabling scalable workflows that can handle complex machine learning tasks. With a high degree of customization, Actions allows you to tailor workflows to the specific needs of your project, ensuring flexibility across various stages of the ML lifecycle. Furthermore, the platform provides access to a vast library of pre-built actions and a strong community, helping to accelerate development and implementation.

Building an efficient MLOps pipeline

An efficient MLOps pipeline leveraging Actions and Arm64 runners involves several key stages:

1. Project setup and repository management:
   • Organize your codebase with a clear directory structure.
   • Utilize GitHub for version control and collaboration.
   • Define environments and dependencies explicitly to ensure reproducibility.
2. Automated data handling:
   • Use Actions to automate data ingestion, preprocessing, and augmentation.
   • Ensure that data workflows are consistent and reproducible across different runs.
3. Automated model training and validation:
   • Define workflows that automatically trigger model training upon data or code changes.
   • Use Arm64 runners to optimize training performance and reduce costs.
   • Incorporate validation steps to ensure model performance meets predefined criteria.
4. CD:
   • Automate the packaging and deployment of models into production environments.
   • Use containerization for consistent deployment across different environments.
   • Leverage cloud services or on-premises infrastructure as needed.
5. Monitoring and maintenance:
   • Setup automated monitoring to track model performance in real time.
   • Implement alerts and triggers for performance degradation or anomalies.
   • Plan for automated retraining or rollback mechanisms in response to model drift.

Optimizing workflows with advanced configurations

To further enhance your MLOps pipeline, consider the following advanced configurations:

• Large runners and environments: define Arm64 runners with specific hardware configurations suited to your workload.
• Parallel and distributed computing: utilize Actions’ ability to run jobs in parallel, reducing overall execution time.
• Caching and artifacts: use caching mechanisms to reuse data and models across workflow runs, improving efficiency.
• Security and compliance: ensure that workflows adhere to security best practices, managing secrets and access controls appropriately.

Real-world impact and case studies

Organizations adopting Actions with Arm64 runners have reported significant improvements:

• Reduced training times: leveraging Arm optimizations in ML frameworks leads to faster model training cycles.
• Cost savings: lower power consumption and efficient resource utilization result in reduced operational costs.
• Scalability: ability to handle larger datasets and more complex models without proportional increases in cost or complexity.
• CD: accelerated deployment cycles, enabling quicker iteration and time-to-market for ML solutions.

Embracing CI

MLOps is not a one-time setup but an ongoing practice of continuous improvement and iteration. To maintain and enhance your pipeline:

• Regular monitoring: continuously monitor model performance and system metrics to proactively address issues.
• Feedback loops: Incorporate feedback from production environments to refine models and workflows.
• Stay updated: keep abreast of advancements in tools like Actions and developments in Arm architecture optimizations.
• Collaborate and share: engage with the community to share insights and learn from others’ experiences.

Conclusion

Integrating Actions with Arm64 runners presents a compelling solution for organizations looking to streamline their MLOps pipelines. By automating workflows and leveraging optimized hardware architectures, you can achieve greater efficiency, scalability, and cost-effectiveness in your ML operations.

Whether you’re a data scientist, ML engineer, or DevOps professional, embracing these tools can significantly enhance your ability to deliver robust ML solutions. The synergy between Actions’ automation capabilities and Arm64runners’ performance optimizations offers a powerful platform for modern ML workflows.

Additional resources

• Getting started with GitHub Actions: GitHub Actions Documentation
• Arm architecture and ML: Arm Developer Resources for AI and ML
• Arm community blogs: Arm for AI and ML

The post Streamlining your MLOps pipeline with GitHub Actions and Arm64 runners appeared first on The GitHub Blog.

Automotive software development moves to the cloud

We are at an inflection point for automotive embedded development to move to the cloud. In an era where software has not just eaten the world but is continuously redefining it through AI, the cloud emerges not just as a platform but as the foundational fabric for software engineering. With AI’s increasing demand for computational power driving unprecedented changes in silicon, both at the edge and in the cloud, the need for agile, scalable, and continuously optimized development environments has never been more critical. As the home of the world’s developers, GitHub is the platform to build the next generation of automotive and embedded development environments in the cloud.

Traditional embedded development challenges

Improving the developer experience is at the heart of what GitHub does. We’re dedicated to making coding as smooth as possible by reducing unnecessary complexity. The traditional process for developers working with embedded systems has plenty of friction to remove. Historically, software development has been very hardware-dependent with developers maintaining some combination of test hardware connected to their development machines or an in-house testing farm. There weren’t many alternatives because so much was proprietary.

In recent years, a series of technical advancements have significantly influenced the foundational architectures within the field. Despite these changes, many traditional methods and operational processes remain in use. Key developments include the adoption of more powerful multipurpose processors, the establishment of open standards for the lower-level software stack such as SOAFEE.io for cloud native architecture at the edge, and the increased reliance on open-source resources, facilitating reuse across different domains. These innovations have provided developers with the opportunity to fundamentally rethink their approaches to development, enabling more efficient and flexible strategies.

As the rate of these technical trends and foundational change increases, teams are finding it increasingly difficult to deliver application commitments without significant cost of maintaining these in-house development and test environments.

Virtualization for embedded and automotive development

While virtualization has become a cornerstone of enterprise development, its integration into embedded systems has proceeded at a more cautious pace. The complexities inherent in embedded systems—spanning a vast array of processors, operating systems, and specialized software—pose unique challenges not encountered in the more homogeneous environments of data centers and IT networks. Embedded systems require a nuanced approach to virtualization that goes beyond simply accommodating mainstream operating systems like Windows and Linux on standard Intel architectures.

In a significant development that reflects the evolving landscape of embedded systems, in March 2024, Arm unveiled its new Automotive Enhanced (AE) processors. These cutting-edge processors are designed to boost AI capabilities within the automotive sector, ensuring ISA (Instruction Set Architecture) compatibility. This advancement is poised to revolutionize the way applications are developed and deployed, enabling developers to create software in the cloud and seamlessly transition it to the edge, such as in vehicles, without the need for extensive reconfiguration or modification. This leap forward promises to accelerate the time-to-market for new applications, bridging the gap between cloud development environments and the nuanced world of embedded systems .

This transition exemplifies how advancements in processor technology and virtualization are converging to address the unique challenges of embedded development, paving the way for more integrated and efficient systems across industries. Developers will be able to write, build, and test code in the cloud and then run their applications in virtualized environments with digital twins that mirror their processor targets, even if those targets haven’t even been delivered in the silicon.

Cloud-based continuous integration platform

Continuous integration (CI), a cornerstone of agile methodologies for over two decades, automates the build, test, and deployment processes. This automation accelerates feedback loops, enabling timely verification that the software meets the intended requirements. It also minimizes integration risks and enhances the early detection of defects and security vulnerabilities. While surveys indicate that many embedded development teams have adopted CI as a practice, managing the development environments across multiple hardware configurations and deployment targets is costly and complex.

Implementing CI/CD in a cloud environment leverages the well-established advantages of cloud computing for embedded engineering teams, significantly enhancing their ability to deliver high-quality products within tight market timelines.

• Enhanced Scalability. Cloud-based CI allows teams to dynamically allocate resources and optimize compute spend. Teams can execute workloads in parallel in order to support multiple hardware and software configurations simultaneously. Developers can also participate across geographic regions or even across organizational boundaries within the supply chain.
• Reduced Complexity. Standardizing on cloud-based CI reduces environment setup and tear down times and promotes consistency. Workflows can easily be shared across teams.
• Improved Quality. When compute resources are too constrained or managing the CI environment is brittle, teams may optimize locally onto too narrow a piece of the development. Reducing this friction and thereby increasing the end to end feedback loops can improve quality.

To deliver cloud-based embedded developer environments for the design and build time that feed into the runtime virtualized and simulated targets, GitHub needed to update our infrastructure. In October 2023, GitHub announced native Arm64 support for our hosted CI/CD workflow engine, GitHub Actions. Supporting this platform is important because Arm’s family of processor designs are central to many uses in the embedded and automotive world.

This promises to free embedded developers from being tied to the desktop. By moving jobs to the cloud, development teams will be able to focus more on coding time and less on infrastructure management. We also recently announced the public beta of GPU hosted runners that will enable teams building machine learning models to do complete application testing, including the ML components within GitHub Actions.

Conclusion

The convergence of cloud technologies, advanced virtualization, and cutting-edge processor innovations represents a transformative shift in automotive software development. To further advance and support these transformations across the industry, GitHub has recently joined SOAFEE.io, as well as maintaining our membership in the Connected Vehicle Systems Alliance (COVESA) and supporting Microsoft’s commitment to the Eclipse Software Defined Vehicle project.

GitHub Enterprise Cloud, along with Arm’s latest AE processors, heralds a new era where development and testing transcend traditional boundaries, leveraging the cloud’s vast resources for more efficient, scalable, and flexible software creation. This paradigm shift towards cloud-based development and virtualized testing environments not only addresses the complexities and limitations of embedded system design but also dramatically reduces the overhead associated with physical hardware dependencies. By enabling developers to seamlessly transition applications from the cloud to the edge without extensive rework, the automotive industry stands on the brink of a significant acceleration in innovation and time-to-market for new technologies.

GitHub’s introduction of native Arm64 support and the public beta of GPU hosted runners on its CI/CD platform, GitHub Actions, further underscores this transition. These advancements ensure that the embedded and automotive development communities can fully harness the cloud’s potential, facilitating a shift from local, hardware-constrained development processes to a more agile, cloud-centric approach. As a result, developers can focus more on innovation and less on the intricacies of hardware management, propelling the automotive sector into a future where software development is more integrated, dynamic, and responsive to the rapidly evolving demands of technology and consumers. This transition not only signifies a leap forward in how automotive software is developed but also reflects a broader trend towards the cloud as the backbone of modern software engineering across industries.

The post GitHub Actions, Arm64, and the future of automotive software development appeared first on The GitHub Blog.

CI/CD best practices are easy to talk about and hard to implement. For example, teams want to avoid surprises before deploying code but often stop short of introducing centrally managed policies to help protect their repositories. Let’s face it, there are plenty of challenges when defining, setting up, and enforcing policy at scale—especially scaling to hundreds of repositories and thousands of developers.

Reusable workflows introduced the idea of centrally managed workflows for an organization’s repositories. They helped the shareability of workflows but configuring each repository individually didn’t scale and more importantly, didn’t enforce the success of workflow runs.

We’ve made it better

GitHub has simplified the process of centrally managing workflows. Available today, GitHub Enterprise Cloud customers can require that certain workflows need to run successfully before code can be merged into an organization’s repositories. Earlier this month, we shared that GitHub Actions required workflows would be moving to GitHub repository rulesets. We’re excited to announce that setting up and managing these workflows is now formally part of GitHub repository rulesets. Not only do repository rulesets improve the experience of setting up and managing these workflows but it also allows several other benefits like branch targeting, bypass rules, and dry running rules in an evaluation mode.

Enforceability is key

Enforcing required workflow runs provides a new level of control to the code being added to your repositories. By requiring workflows via repository rulesets, all pull requests on selected repositories will be blocked until specified workflows run successfully. This easily scales from some to all repositories in the organization regardless of the number of contributors. The level of specificity for the workflow is also configurable—choose a workflow by specific branch, tag, current commit or specific SHA to lock in the exact version of the workflow that’s required. These controls will help save engineering time and effort by enforcing workflow runs on code contributions before they’re merged and deployed.

Repository rulesets make it easy

Requiring workflows from repository rulesets keeps set-up and maintenance for all your rules in a centralized UI. This also means that all additional tools provided by rulesets can be applied to your workflow rule. For example, you can create a bypass list that allows specific users to “break the glass” so they can merge even if the workflow doesn’t succeed. You can also set up and evaluate the potential impact of required workflow policies before enforcing them on repositories. This will allow you to fine tune and test the settings without disrupting the flow of contributors.

Auditing

Repository Rule Insights are a powerful tool used to evaluate required workflows. When introducing a workflow rule, it’s necessary to make sure things are working properly before enforcing it throughout your repositories. By placing the rule into “Evaluate” mode, the audit reports the results of the rule without blocking downstream repository contributions. Staying in “Evaluate” mode provides the time necessary to evaluate the results of the rule before enforcing it. Once the rule is working as intended, switching from “Evaluate” to “Activate” enforces the rule and prevents any code associated with the workflow from being merged into the repository.

Return confidence to the dreaded “Friday deploy”

Workflows controlled at the organization level ensure repository contributions adhere to a defined set of automated processes. Defining and enforcing these workflows is now a part of GitHub repository rules, which greatly improves the experience to centrally manage repositories. Now, code contributions are guaranteed to adhere to the CI/CD policies defined by required workflows. Check out our docs to learn more about how to set up and require workflows at the organization level. And don’t forget to join the community discussion about requiring workflows. If you’re reading this on a Friday, don’t worry, set up your required workflows and build confidence in today’s deployments.

The post Enforcing code reliability by requiring workflows with GitHub repository rules appeared first on The GitHub Blog.

Microsoft Power Platform is a powerful suite of tools that allows users to build custom solutions for their organization with low-code tools. GitHub, on the other hand, is a powerful web-based platform that allows developers to manage, secure, and deliver their code. In this blog post, we will explore how to automate a Power Platform deployment using the CI/CD capability of GitHub Actions.

What is GitHub Actions

GitHub Actions is a powerful automation tool that allows developers to automate tasks, such as building, testing and deploying code. It is actually more than just a continuous integration and continuous deployment tool. It provides a wide range of pre-built actions, which are reusable units of code that can be combined to create workflows. GitHub Actions also supports custom actions, which can be built and shared across teams and the community.

Why automation

Why automation? Simply, human error. We all make mistakes. We want to increase our efficiency and productivity, all while reducing our human errors. GitHub Actions are an easy to use automation tool that work directly from a GitHub repository, enabling your deployments to occur closer to your code. Also, there are a lot of platform tools from GitHub that you can integrate into your code base, for example, GitHub Advanced Security. It is all about bringing your tools closer to the developer and enabling you to do more with your code.

Automating your Power Platform deployment with GitHub Actions

Automating your Power Platform deployment with GitHub Actions has several benefits:

1. Consistency: automation ensures that each deployment is identical and prescriptive, eliminating the risk of human error.
2. Efficiency: reduces the time and effort required to deploy solutions, allowing you and your teams to deliver more value at an increased rate.
3. Version Control: track your changes with GitHub as your version control provider, allowing you to have full visibility of code changes and the ability to rollback to a previous version if needed, preventing unnecessary downtime.
4. Collaboration: leveraging GitHub to allow your teams to work on the same code simultaneously, increasing communication across your teams.

Watch the video below to follow along with a hands-on tutorial:

In the above video we cover off everything you need to know about deploying your Power Platform deployment with GitHub Actions; follow the links below to jump into each topic:

• Integrate Power Platform development into GitHub Codespaces
• Leverage GitHub Environments to segment your workflow
• Access the Microsoft Power Platform actions lab from a GitHub repository
• Export your Power Platform solution and check it in
• Release your Power Platform solution from a QA environment through to production
• Review your CI/CD deployment in Power Platform

Automating your Power Platform deployments with GitHub Actions can help you save time, reduce errors, and increase productivity. Good luck automating and remember, you can automate so much more than your CI/CD workflows with GitHub Actions!

Useful Resources

• Further documentation for GitHub Actions and Power Platform
• GitHub Marketplace for Power Platform Actions
• Getting started with GitHub Actions
• Learn more about Microsoft Power Platform

The post How to automate a Microsoft Power Platform deployment using GitHub Actions appeared first on The GitHub Blog.

Doing individual migrations is relatively easy in isolation. However, for companies that have a large and established CI/CD footprint, having tooling available to assist with migrations is key to their ability to adopt Actions at scale. In the time that we’ve been developing and using GitHub Actions Importer in its private preview form, we’ve already encountered numerous customers that have thousands of pipelines—even in excess of 15K—in their legacy environments that need to be migrated. GitHub Actions Importer is designed to help when manual migration is not feasible, such as when customers have a large number of teams that depend on hundreds or thousands of workflows.

The phases of a GitHub Actions Importer migration

Using GitHub Actions Importer to facilitate a migration to Actions can be conceptualized in distinct phases.

Planning

A migration typically starts with an analysis of your existing CI/CD usage and approach, which includes addressing such questions as:

• How many pipelines are currently defined?
• What is the typical makeup of these pipelines? Are they defined using out-of-the-box tasks, or is the product being extended to provide custom behavior?
• Which pipelines are the highest priority to be migrated? Should any pipelines not be migrated?
• What type and how many runners are being used? What mix of GitHub-hosted and self-hosted runners should you select when migrating?

Migrating

Once you begin your migration efforts, you can expect that a small subset of migrated workflows will require manual work to complete. Although the actual conversion rate for a customer will depend on the makeup of each individual pipeline, in practice we’ve found that, on average, GitHub Actions Importer can successfully convert in excess of 90% of the tasks and constructs used in a workflow.

To finalize your migration, you should plan to iterate on the new GitHub Actions workflow to validate that it functions equivalently to the existing pipeline and that constructs that could not be migrated automatically are implemented. This could include—but is not limited to—configuring secrets, self-hosted runners, environments, and branch protection rules. In addition, if you were using any custom tasks or less commonly used public tasks, you will need to port or re-implement that logic manually.

How to get started with GitHub Actions Importer

The user interface for GitHub Actions Importer is an extension to the official GitHub CLI, which delegates to a Docker container. When you get access to GitHub Actions Importer, have installed it and the GitHub CLI, and are ready to try it in your own environment, you’ll typically want to take the following steps.

1. Plan the timeline and complexity of the migration

GitHub Actions Importer provides an audit command that is designed to help analyze the complexity of a potential migration, which can be used to formulate a migration plan. This command will fetch all of the pipelines defined in a specified scope of the existing CI/CD environment, attempt a conversion of these pipelines to their equivalent workflow, and write a summary report with statistics gathered from the audit.

Running an audit on the command line will look something like this:

$ gh actions-importer audit jenkins --output-dir .

The Audit summary report provides details aggregated at the pipeline level, aggregated at the build step level, and identifies manual migration tasks that will need to be completed.

Pipelines

This section contains high-level aggregated statistics on GitHub Actions Importer’s ability to migrate the targeted pipeline automatically.

The Pipelines section provides the following key metrics:

• Successful: identifies the number of pipelines that had all of their constructs and individual items converted automatically to their GitHub Actions equivalent.
• Partially successful: identifies the number of pipelines that had all of their constructs converted, however, there were some individual items (for example,build tasks or build triggers) that could not be converted automatically.
• Unsupported: identifies the number of pipeline definitions that use constructs that are unsupported by GitHub Actions Importer. Please refer to the platform-specific documentation to learn which pipeline types are supported.
• Failed: identifies the number of pipelines where GitHub Actions Importer had a fatal error during an attempt at conversion. This can occur for one of three reasons:
  • The pipeline was misconfigured and not valid in its existing CI/CD platform.
  • GitHub Actions Importer encountered an internal error when converting it.
  • There was an unsuccessful network response, often due to invalid credentials, that caused the pipeline to be inaccessible.

The Job types section summarizes which pipeline types are being used and whether they are supported or unsupported. Please refer to the platform-specific documentation to learn which pipeline types are supported.

Build steps

This section presents an aggregated summary of the individual build steps that are used across all of the target pipelines and how many could be converted automatically.

The Build steps section provides the following details:

• A known build step is one that GitHub Actions Importer can automatically convert to an equivalent action.
• An unknown build step is one that cannot be automatically converted to an equivalent action.
• An unsupported build step is either:
  • Fundamentally not supported by GitHub Actions.
  • Configured in a way that is incompatible with GitHub Actions.
• Actions provides a comprehensive list of actions that would be used in all of the converted workflows. This is important in these scenarios:
  • When you are migrating to GHES and need to sync these actions to your GHES instance.
  • When you are defining an organization-level allowlist.
  • When you need to provide a comprehensive list of actions for your security and/or compliance teams to review and approve.

In addition, the audit summary report includes an equivalent summary of build triggers, environment variables, and other uncategorized items.

Manual tasks

This section presents an overview of the manual tasks that can be identified.

The Manual tasks section can provide the following details:

• Secrets includes a summarized list of the secrets used in the converted pipelines. Because secrets cannot be migrated automatically, they will need to be created manually in order for these workflows to function properly.
• Self-hosted runners summarizes and identifies the labeled runners referenced in the source pipelines. You will need to decide whether a GitHub-hosted or self-hosted runner is appropriate to run these workflows within GitHub Actions.

File manifest

The final section of the audit summary report provides a manifest of all the files generated during the audit. These files can include:

• The original pipeline as it was defined in the source CI/CD platform.
• A log of the network responses used to convert a pipeline.
• The converted workflow file(s).
• A log of any error messages to troubleshoot a failed pipeline conversion.

2. Understand what is not migrated automatically

GitHub Actions Importer cannot migrate everything. It’s important to be mindful of its limitations, which include:

• Secrets and encrypted values are not automatically converted to repository secrets. References to secrets will be converted into context expressions and populating these values will be a manual task.
• Self-hosted build agents are not automatically converted to self-hosted runners. Determining whether to use GitHub-hosted runners or create equivalent self-hosted runners will be a manual task. References to these self-hosted runners will be converted to the same set of labels in a needs statement in the resulting workflow.
• Historical packages are not migrated to GitHub Packages. Any steps that publish or consume pipeline artifacts and caches will be converted using the equivalent actions.
• Permissions for CI/CD pipelines are not migrated automatically and will need to be manually configured.
• Build steps or build triggers that are less commonly used may not be automatically converted by GitHub Actions Importer. This can be a factor for migrations involving Azure DevOps, Jenkins, and CircleCI, all of which can be extended through marketplace customizations.

3. Forecast your Actions usage

The forecast command is designed to help you understand the compute capacity you’re currently using within your CI/CD environment. This command fetches jobs that have been completed over a specified time period and uses that data to calculate usage metrics.

Running a forecast on the command line will look something like this:

$ gh actions-importer audit jenkins –start-date 7/1/22 --output-dir .

The Forecast report includes these metrics:

• Job count: the total number of completed jobs.
• Pipeline count: the number of unique pipelines used.
• Execution time: the amount of time a runner spent on a job. This metric can be used to help set expectations for the cost of GitHub-hosted runners.
• Queue time: the amount of time a job spent waiting for a runner to be available.
• Concurrent jobs: the number of jobs running at any given time. This metric can be used to understand the number of concurrent runners necessary at current levels.

Additionally, these metrics are presented for each runner queue defined in the source CI/CD system. This is useful if you will need a mix of hosted and self-hosted runners and/or if you use a mix of platforms.

4. Customize a workflow’s conversion

You can use the dry-run command to convert an existing pipeline to its equivalent GitHub Actions workflow. The console output of the command will list the path to the file or files that GitHub Actions Importer generated. Before migrating, you should perform a dry run of a pipeline and validate the contents are suitable.

If the conversion of a pipeline was only “partially successful” (that is, it included tasks that could not be converted automatically), the task that was not converted will be included in a commented section. For example, if you were to run the following CLI command:

$ gh actions-importer dry-run jenkins --source-url $SOURCE_URL --output-dir 

The converted workflow will have contents similar to the following.

name: ethanis/universe
on:
  workflow_dispatch:
jobs:
  Build:
    runs-on:
      - self-hosted
      - main
    steps:
    - name: checkout
      uses: actions/checkout@v3
#     # This item has no matching transformer
#     - buildJavascriptApp:
#       - key: deploy
#         value:
#           isLiteral: true
#           value: false

In this situation, you will need to decide how to implement this functionality. If this task is used within multiple pipelines, the recommended approach is to implement a custom transformer that can handle this scenario in every pipeline. If this is a one-off scenario, you can edit the converted workflow file manually.

Let’s assume in this example that the buildJavascriptApp build step is used in multiple workflows and could be easily replicated with a single shell command. This is what that custom transformer would look like:

transform “buildJavascriptApp” do |item|
  {
    name: “Build Javascript App”,
    run: “npm install && npm run build”
  }
end

If you were to add these contents to a file named transformers.rb, they can be provided to GitHub Actions Importer on the command line. For example:

$ gh actions-importer dry-run jenkins --source-url $SOURCE_URL --output-dir . --custom-transformers transformers.rb

5. Perform the production workflow migration

You can use the migrate command to convert an existing pipeline to its equivalent action and open a pull request with the converted workflows and associated files.

$ gh actions-importer migrate jenkins --source-url $SOURCE_URL –target-url $TARGET_URL --output-dir .

Any necessary manual tasks will be included in the description of the pull request. Once these manual tasks and the code reviews are complete, the pull request can be merged and the workflow will have been successfully migrated to GitHub Actions.

Optional: configure self-serve migrations with IssueOps

The GitHub Actions Importer IssueOps template repository provides the foundational functionality required to run GitHub Actions Importer commands through GitHub Actions and Issues. This approach allows you to migrate pipelines to GitHub Actions without needing to have GitHub Actions Importer and Docker installed on a local machine. Organizations that want to enable self-service migrations to GitHub Actions could find this approach useful.

Sign up for GitHub Actions Importer

Request access by completing the waitlist form here.

The post Introducing GitHub Actions Importer appeared first on The GitHub Blog.

As software development has evolved, so too have the expectations for application security processes. It is no longer acceptable to bolt on a vulnerability scan just prior to release; instead it’s seen as an integral part of the end-to-end SDLC. To do this well, security tools need to be deeply embedded and automated, so developers are empowered rather than encumbered. GitHub code scanning, which is powered by the CodeQL engine, surfaces security alerts right at the heart of developers’ workflows so you and your teams can ship more secure code.

This blog post explores how to centrally manage a GitHub code scanning integration that runs the CodeQL analysis engine using third party CI/CD tooling. We then explain how you can leverage CodeQL’s indirect build tracing to perform a scan on an application portfolio in a lightweight, repeatable and consistent manner.

Code scanning at scale

At the individual or team level, open source maintainers and application developers continue to strive for the highest standards of code security by incorporating scans in their CI processes. Large enterprises are often tasked with the responsibility to centrally manage their implementation as it means they can provide a more reliable service and steward adoption. At scale, rolling out the same SAST workflow across thousands of applications can be a challenge.

Getting the rollout right means systematic, thoughtful planning and communication with the application development teams that will consume the service. While this post isn’t meant to cover the human aspect of rolling out that SAST workflow, it is a major component in any successful DevSecOps program. If you’re looking for tips on how to do that well, we’ve written about how customers have gotten it right for communities of thousands of developers.

A great AppSec experience will earn the trust of developers, reduce mean time to remediation, and lead to more secure code. Consistently high‐quality code scanning results, like those CodeQL produces, are therefore essential. CodeQL achieves that consistency by analyzing the full context of an application’s data and call flows during the build. With the architecture below, that build happens in the development teams’ CI/CD agent with the CodeQL engine simply “peering into” the build process. This simplified integration is achieved when those teams wrap their build commands with a library function and continue their pipeline steps while a centralized service performs the scan.

Centralized administration

Increasingly, GitHub customers prefer to use GitHub Actions for CI/CD. GitHub code scanning natively integrates with GitHub Actions to make adopting security scanning tools, like CodeQL, frictionless. However, we recognize that some enterprises prefer to use other CI/CD tools, such as Jenkins or ADO. Even with third-party CI, from a “level of effort” perspective, integrating a new centralized scan job into the developers pipeline should be simple.

Most CI tools provide the ability to create modular, reusable and centrally managed templates (shared libraries on Jenkins or Pipeline Templates on ADO) to load external logic into an existing workflow. By leveraging this integration pattern, an application security team can externalize the core CodeQL functionality (create, analyze, upload) whilst enabling individual developer teams to include custom logic to support more granular configurations. Those configurations might look like: passing in application-specific build commands, running additional queries, or providing additional sources and sinks. This allows for the re-use of the required CodeQL stages, meaning every application team does not have to duplicate the same stages for every pipeline, and also enables application specific customization.

Here is a pseudocode example of an application team’s pipeline that loads a centrally managed security library called “central-security-library.” All of the logic to run CodeQL has been abstracted into the library “central-security-library,” and all the application team’s pipeline needs to do is pass in the build command for that individual application. This pattern is highly reusable and promotes a quick time to value:

### Application Teams Pipeline

@Library(["central-security-library"]) 
# This introduces the reusable step “codescan”, which is used below. 

stage('Build & Scan'){

  steps{

    codescan("$REF_PARAMS"),

    {

### Teach “appsec-workflow” how to build the application via $BUILD_ARGS.

      sh("mvn clean package -f ./main/pom.xml") 

    }

  }

}

Build once, for scan and deploy

Centralizing the code scanning integration with a reusable pipeline component opens the door to another key optimization. The developer’s pipeline can pass build arguments into the reusable component so it can perform the build on the pipeline’s behalf, generating the necessary AppSec insights along the way. That build artifact is then available for its usual testing and deployment or anything in between. To achieve this build, scan and deploy pattern introduce CodeQL CLI indirect build tracing to your reusable library.

Indirect build tracing enables CodeQL to detect all build steps automatically between the init and analyze steps. The reusable workflow requires the application team’s pipeline to pass in build commands for the build to be traced. The centralized security library should sandwich those build commands between CodeQL init and analyze for CodeQL to peer into the build process and assemble a representative CodeQL DB.

Below is an example (ADO pipeline pseudocode) of how you might initialize CodeQL with indirect build tracing, build the application via the “build command” arguments of the caller workflow, and perform an analysis using indirect build tracing. More examples available here.

### Centralized Security Library Workflow

# Initialize the CodeQL database using `codeql database init --begin tracing`.

- task: CmdLine@1

  displayName: Initialize CodeQL database

  inputs:

      # Assumes the source code is checked out to the current working directory.

      # Creates a database at `/codeql-dbs/example-repo`.

      # Ensure you "source" the relevant environment variables after this step!

      script: "codeql database init --language $LANG --source-root . --begin-tracing /codeql-dbs/example-repo"

# Set CodeQL environment variables

- … [docs](https://codeql.github.com/docs/codeql-cli/creating-codeql-databases/#example-of-creating-a-codeql-database-using-indirect-build-tracing)

# Build the App with Args

- task: CmdLine@1

  displayName: Build app with build command Args from caller

  inputs:

      script: $BUILD_ARGS

# Use `codeql database finalize` to complete database creation after the build is done.

- task: CmdLine@2

   displayName: Finalize CodeQL database

   inputs:

      script: 'codeql database finalize /codeql-dbs/example-repo'

# Analyze the database

- task: CmdLine@2

   displayName: Analyze CodeQL database

   inputs:

      script: 'codeql database analyze /codeql-dbs/example-repo java-code-scanning.qls --sarif-category=java --format=sarif-latest --output=/temp/example-repo-java.sarif'

# Upload the results.

- task: CmdLine@2

   displayName: Upload CodeQL results

   inputs:

      script: 'echo "$TOKEN" | codeql github upload-results

    --repository=$BUILD_REPOSITORY_NAME \ --ref=$BUILD_SOURCEBRANCH \  --commit=$BUILD_SOURCEVERSION

    --sarif=/temp/example-repo-java.sarif --github-auth-stdin'

The output of the above indirect build tracing is two-fold: the source code has been extracted into a CodeQL database and is ready for the analysis, and your standard build artifact has been built ready for deployment. All whilst only building your application once. After the CodeQL analysis has been completed, the contents of the SARIF file, containing the analysis results, is uploaded to the application team’s GitHub repository immediately for review. The next stage in the application team’s workflow is invoked without change.

After implementing this workflow, AppDev teams will have meaningful insight into the security of the features they are delivering, and AppSec teams can now meet their objective of providing a scalable integration that provides a high quality experience with little disruption.

Learn more about GitHub Code Scanning

GitHub is a cloud-native software development leader, empowering more than 83 million developers to collaborate using open source and inner source. GitHub is committed to helping build safer and more secure software without compromising on the developer experience.

To learn more about code scanning, visit our code scanning Docs page. For any questions or to see a demo, please contact your GitHub account team.

The post Best practices on rolling out code scanning at enterprise scale appeared first on The GitHub Blog.

Despite this, there are levels of maturity when building a CI/CD pipeline that range from basic implementations to more advanced automation workflows. But wherever you are on your CI/CD journey, there are a few things you can do to level up your CI/CD pipeline.

With that, here are six strategic things I often see missing from CI/CD pipelines that can help any developer or team advance and improve their workflows.

1. Add performance, device compatibility, and accessibility testing

Performance, device compatibility, and accessibility testing are often a manual exercise—and something that some teams are only partially doing. Manually testing for these things can slow down your delivery cycle, so many teams either eat the costs or just don’t do it.

But if these things are important to you—and they should be—there are tools that can be included in your CI/CD pipeline to automate the testing for and discovery of any issues.

Performance and device compatibility testing

One tool, for example, is Playwright which can do end-to-end testing, automated testing, and everything in between. You can also use it to do UI testing so you can catch issues in your product.

Visual regression testing

There’s another class of tools that can help you automate visual regression testing to make sure you haven’t changed the UI when you weren’t intending to do so. That means you haven’t introduced any unexpected UI changes. This can be super useful for device compatibility testing too. If something looks bad on one device, you can quickly correct it.

Accessibility testing

This is another incredibly impactful class of automated tests to add to your CI/CD pipeline. Why? Because every one of your customers should be valuable to you—and if even just a fraction of your customers have trouble using your product, that matters.

There are a ton of accessibility testing tools that can tell you things like if you have appropriate content for screen readers or if the colors on your website make sense to someone with color blindness. A great example is Pa11y, an open source tool you can use to run automated accessibility tests via the command line or Node.js.

2. Incorporate more automated security testing

Security should always be part of your software delivery pipeline, and it’s incredibly vital in today’s environments. Even still, I’ve seen a number of teams and companies who aren’t incorporating automated security tests in their CI/CD pipelines and instead treat security as something that happens after the DevOps process takes place.

Here’s the good news: There are a lot of tools that can help you do this without too much effort—including GitHub-native tools like Dependabot, code scanning, secret scanning, and if you’re a GitHub Enterprise user, you can bundle all the security functionality GitHub offers and more with GitHub Advanced Security. But even with a free GitHub account, you still can use Dependabot on any public or private repository, and code scanning and secret scanning are available on all public repositories, too.

Dependabot, for example, can help you mitigate any potential issues in your dependencies by scanning them for outdated packages and automatically creating pull requests for teams to fix them. It can also be configured to automatically update any project dependencies, too.

This is super impactful. Developers and teams often don’t update their dependencies because of the time it takes—or, sometimes they even just forget to update their dependencies. Dependencies are a legitimate source of vulnerabilities that are all too often overlooked.

Additionally, code scanning and secret scanning are offered on the GitHub platform and can be built into your CI/CD pipeline to improve your security profile. Where code scanning offers SAST capabilities that show if your code itself contains any known vulnerabilities, secret scanning makes sure you’re not leaking any credentials to your repositories. It can also be used to prevent any pushes to your repository if there are any exposed credentials.

The biggest thing is that teams should treat security as something you do throughout the SDLC—and, not just before and after something goes to production. You should, of course, always be checking for security issues. But the earlier you can catch issues, the better (hello DevSecOps). So including security testing within your CI/CD pipeline is an essential practice.

3. Build a phased testing strategy

Phased testing is a great strategy for making sure you’re able to deliver secure software fast and at scale. But it’s also something that takes time to build. And consequently, a lot of teams just aren’t doing it.

Often, developers will put all or most of their automated testing at the build phase in their CI/CD pipelines. That means the build can take a long time to execute. And while there’s nothing necessarily wrong with this, you may find that it takes longer to get feedback on your code.

With phased testing, you can catch the big things early and get faster feedback on your codebase. The goal is to have a quick build that rapidly tests the fundamentals with simpler tests such as unit tests. After this, you may then perhaps deploy your build to a test environment to execute additional tests such as some accessibility testing, user testing, and other things that may take longer to execute. This means you’re working your way through a number of possible issues starting with the most critical elements first.

As you get closer to production in a phased testing model, you’ll want to test more and more things. This will likely include key items such as regression testing to make sure previous bugs aren’t reappearing in your codebase. At this stage, things are less likely to go wrong. But you’ll want to effectively catch the big things early and then narrow your testing down to ensure you’re shipping a very high-quality application.

Oh, and of course, there’s also testing in production, which is its own thing. But you can incorporate post-deployment tests into your production environment. You may have a hypothesis you want to test about if something works in production and execute tests to find out. At GitHub, we do this a lot by releasing new features behind feature flags and then enabling that flag for a subset of our user base to collect feedback.

4. Invest in blue-green deployments for easier rollouts

When it comes to releasing a new version of an application, what’s one word you think of? For me, the big word is “stress” (although “excitement” and “relief” are a close second and third). Blue-green deployments are one way to improve how you roll out a new version of an application in your CI/CD pipeline, but it can also be a bit more complex, too.

In the simplest terms, a blue-green deployment involves having two or more versions of your application in production and slowly moving your users from an older version to a newer one. This means that when you need to update or deploy a new version of an application, it goes to an “unused” production environment, and you can slowly move your users across safely.

The benefit of this is you can quickly roll back any changes by redirecting users to another prod environment. It also leads to drastically reduced downtime while you’re deploying a new application version. You can get everything set up in the environment and then just point people to a new one.

Blue-green deployments are perfect when you have two environments that are interchangeable. In reality with larger systems, you may have a suite of web servers or a number of serverless applications running. In practice, this means you might be using a load balancer that can distribute traffic across multiple locations. The canonical example of a load balancer is nginx—but every cloud has its own offerings (like Azure Front Door or Elastic Load Balancing on AWS).

This kind of strategy is common among organizations using Kubernetes. You may have a number of pods that are running and when you do a deployment, Kubernetes will deploy updates to new instances and redirects traffic. The management of which ones are up and running operates under the same principles as blue-green deployments—but you’re also navigating a far more complex architecture.

5. Adopt infrastructure-as-code for greater flexibility

Infrastructure provisioning is the practice of building IT infrastructure as you need it—and some teams will adopt infrastructure-as-code (IaC) in their CI/CD pipelines to provision resources automatically at specific points in the pipeline.

I strongly recommend doing this. The goal of IaC is that when you’re deploying your application, you’re also deploying your infrastructure. That means you always know what your infrastructure looks like in production, and your testing environment is also replicable to what’s in production.

There are two benefits to building IaC into your CI/CD pipeline:

1. It helps you make sure that your application and the infrastructure it runs on are routinely being tested in tandem. The old school way of doing things was to say that this is a production machine and it looks like this—and this is our testing machine and we want it to be as close to production as possible. But almost always, you’ll find that production environments change over time—and it makes it harder to know what your production environment is.

2. It helps you mitigate any real-time issues with your infrastructure. That means if your production server goes down, it’s not a disaster—you can just re-deploy it (and even automate your redeployment at that).

Last but not least: building IaC into your CI/CD pipeline means you can more effectively do things like blue-green deployments. You can deploy a new version of an application—code and infrastructure included—and reroute your DNS to go to that version. If it doesn’t work, that’s fine—you can quickly roll back to your previous version.

6. Create checkpoints for automated rollbacks

Ideally, you want to avoid ever having to roll back a software release. But let’s be honest. We all make mistakes and sometimes code that worked in your development or test environment doesn’t work perfectly in production.

When you need to roll back a release to a previous application version, automation makes it much easier to do so quickly. I think of a rollback as a general term for mitigating production problems by reverting to a previous version, whether that’s redeploying or restoring from backup. If you have a great CI/CD pipeline, you can ideally fix a problem and roll out an update immediately—so you can avoid having to go to a previous app version.

Looking for more ways to improve your CI/CD pipeline?

Try exploring the GitHub Marketplace for CI/CD and automation workflow templates. At the time I’m writing this, there are more than 14,000 pre-built, community-developed CI/CD and automation actions in the GitHub Marketplace. And, of course, you can always build your own custom workflows with GitHub Actions.

Additional resources

• [Blog] How to build a CI/CD pipeline on GitHub in four steps
• [Blog] 5 DevOps tips to speed up your developer workflow
• [Blog] A beginner’s guide to CI/CD and automation on GitHub

The post 6 strategic ways to level up your CI/CD pipeline appeared first on The GitHub Blog.

Continuous Integration / Continuous Delivery (CI/CD) has long been—and continues to be—the domain of DevOps experts. But with the introduction of native CI/CD to GitHub in 2019 via GitHub Actions, it’s easier than ever to bring CI/CD directly into your workflow right from your repository.

That’s a great thing. As developers, we’re trained to use peer reviews to make sure our code works. But I’m here to tell you we need to disrupt the peer review. If you’re using Git, GitHub, and GitHub Actions to build a CI/CD pipeline, you should have confidence in your code.

I’m going to walk you through exactly how to build your own CI/CD pipeline, right from your repository on GitHub.

Find out how GitHub compares to other DevOps and CI/CD platforms >

Key advantages of using GitHub Actions for CI/CD pipelines

But first, let’s talk through some of the benefits to using GitHub Actions—because let’s be honest, there are a lot of other tools out there. Let me unpack the four big benefits that I’ve come across:

• CI/CD pipeline set-up is simple: GitHub Actions is made by and for developers, so you don’t need dedicated resources to set up and maintain your pipeline. There’s no need to manually configure and set up CI/CD. You don’t have to set up webhooks, you don’t have to buy hardware, reserve some instances out there, keep them up to date, do security patches, or spool down idle machines. You just drop one file in your repo, and it works.

• Respond to any webhook on GitHub: Since GitHub Actions is fully integrated with GitHub, you can set any webhook as an event trigger for an automation or CI/CD pipeline. This includes things like pull requests, issues, and comments, but it also includes webhooks from any app you have integrated into your GitHub repository. Let’s say you’re going to use any one of the many tools that are out there to run part of your development pipeline. With GitHub Actions, you can trigger CI/CD workflows and pipelines of webhooks from these apps (even something simple, like a chat app message, if you’ve integrated your chat app into your GitHub repository, of course).

• Community-powered, reusable workflows: You can share your workflows publicly with the wider GitHub community or access pre-built CI/CD workflows in the GitHub Marketplace (there are more than 11,000 available actions!). Did I mention every action is reusable just by referencing its name? Yes, that too.

• Support for any platform, any language, and any cloud: GitHub Actions is platform agnostic, language agnostic, and cloud agnostic. That means you can use it with whatever technology you choose.

How to build a CI/CD pipeline with GitHub Actions

Before we dive in, here are a few quick notes:

• Be clear about what a CI/CD pipeline is and should do. This is a simple note, but important. A CI pipeline runs when code changes and should make sure all of your changes work with the rest of the code when it’s integrated. It should also compile your code, run tests, and check that it’s functional. A CD pipeline goes one step further and deploys the built code into production.

• GitHub Actions takes a “choose-your-own adventure” type of approach to CI/CD. You’ll see it the first time you open GitHub Actions in a repository. There are plenty of guided options with pre-built CI workflows you can leverage, per your technology requirements. But you can also build your own CI workflow from scratch if you want to.

• Our example features a website built on Astro and deployed via GitHub Pages. For both the CI and CD portion of this guide, we’ll be using a website I built and developed called www.opensauced.pizza. This website is intended to make it easier for first-time open source contributors to find open source projects to work on and prioritizes projects with clear onboarding flows.

• If you’re a visual learner, we have you covered. While I’ll walk you through building a simple CI/CD pipeline with my GitHub project www.opensauced.pizza, I also have a video you can watch where I put these steps into practice and build a quick CI flow for another project of mine: hot.opensauced.pizza. This video shows you exactly where to go to start building a streamlined CI pipeline. Give it a watch if you’re a visual learner.

Okay, let’s do this.

Step 1: Create or choose a repository, and pick a project

This might sound pretty basic, but the first step to building a CI pipeline with GitHub Actions is creating or choosing a repository on GitHub. You can either use an existing project code base, fork a project you like on GitHub, or start from scratch.

For simplicity’s sake, I’m going to use the Open Sauced repository in my Open Sauced project. Feel free to take a closer look by forking this repository and contributing with a fork.

You can see that the Open Sauced repository is relatively simple. The website itself is made with OneGraph, hosted on Netlify , and built with HTML, CSS, and JavaScript. I also leverage Storybook for UI and design work. We’re also using React and npm for package management, installation, and testing—but more on that later.

Step 2: Open GitHub Actions in your repository to start building your CI/CD workflow

To begin building your CI/CD pipeline, open the GitHub Actions tab in your repository’s top navigation bar.

You should see a list of CI/CD and workflow automation templates that match the technology your project uses (side note: We just improved this entire user flow. Check out the details later, if you’re interested).

For this project, we’ll leverage a few different CI/CD workflows to test, build, stage, and deploy our code. These include:

• A development workflow: This workflow runs through a few different jobs whenever a pull request is opened, edited, synchronized, or reopened. These jobs include setting up Node, installing npm packages and dependencies, running npm test, and cycling through a number of lint jobs too (setup node, install npm@latest, install dependencies, lint code… you get the idea).

• A CodeQL Analysis workflow: This workflow runs a series of CodeQL security tests on our code after we merge it to the main branch to ensure there are no known vulnerabilities. Check out the YAML file for yourself. It’s super simple, but effective and something I’d highly recommend.

• A release and build workflow: This workflow runs tests and enforces lint after releasing code changes to Docker and building the application. It also deploys the final code to our production environment, cuts a release using a similar structure to the automated release notes, bundles the site into a container and publishes to ghcr. From there, it bumps the version number and tag in the repository. This is one of the more complex workflows we’re running, and I’m oversimplifying it somewhat, but you can see which jobs and steps happen in the workflow for yourself.

• A storybook deployment workflow: This workflow deploys any UI component changes to our production website through our frontend UI tech Storybook.

And those are our workflows. The point here is that if you’re working on a solo project or something small, building a CI pipeline doesn’t have to be an intimidating process. You can start with a few simple things (like what’s highlighted above) to make your workflow a little easier.

A CI/CD pipeline can and should be more complex than this if you’re building enterprise software, maintaining a big open source project, or working with a big team on any array of things. But if you’re just getting started, don’t worry about making your CI/CD pipeline meet every single need of a big team or ambitious project—just make it work for you.

Step 3: Make changes to your code to trigger your CI/CD pipeline

For the sake of this exercise, we’re going to make a small change to the website headline (“The path to your next open source contribution”) by adding in the “and more pizza” at the end. The final result will look like this: “The path to your next open source contribution and more pizza.”

Let’s get to work.

Open the src folder, and go to the components subfolder. From there, you’ll want to open the hero.js file. In that file, go to this code:

<div>
<h1>The path to your next</h1>
<h2>Open Source</h2>
<h1>contribution.</h1>
</div>

Make the change to the copy by adding in “and more pizza.” It should look like this:

<h1>The path to your next</h1>
<h2>Open Source</h2>
<h1>contribution and more pizza.</h1>

Step 4: Take a look at the workflow visualizer and live logs to get a full look into how your pipeline is running

Once you push the above change, you get to dive into the fun part: seeing your pipeline work in real time with a workflow visualizer and with live logs.

Okay, maybe this isn’t the most fun part, but knowing how to use both of these tools… you’ll thank me later.

Let’s go over the workflow visualizer first. Accessible via the Actions main page, a workflow visualizer can be pulled up by selecting whatever workflow you want to see. You’ll find something like this:

Here, you can see which job in a given workflow happens when—and if they’re working or not with a little green check mark, a yellow sign to show if something’s working, and a red sign to show if a job failed.

This is your YAML workflow, but in visual form, and it makes it easier to see what’s happening when and if it’s working.

Pro tip: Since workflow visualization graphics are color-coded to quickly show successful actions, ongoing actions, and actions that failed at a particular step, try using it after you set up a new workflow and trigger it for the first time.

Now, here’s what you should know about live logs: First off, they’re your best friend (seriously). Live logs are incredibly useful for figuring out exactly what is working, what’s broken, and why the things you think should be working aren’t working at all. But more to the point, you can go through these logs and look through timestamps, the raw logs, or even download the log itself for local reference.

Live logs can be accessed directly through the Actions menu in your repository and by clicking into any job or workflow. If you did everything right when you built out your CI/CD pipeline, you probably won’t need to look at your live logs.

But if something goes wrong, these live logs can be very helpful to reference. Whether you’re looking at timestamps or which part of the process failed, you can determine how to fix the problem.

Pro tip: Timestamps can be incredibly useful if you’re debugging a time-sensitive error. By default, a live log will color code which jobs failed and when in a given workflow, and that makes it easier to immediately address the problem.

Take this with you

Whether you’re working on an open source project, a side project, or a work project, there are some big benefits to adopting CI/CD—like producing more consistent and workable releases. But the biggest benefit of all is trusting that your code works after merging it to your main branch, testing it, and deploying it.

With GitHub Actions, building a CI/CD pipeline is a straightforward process and one that lets you focus more extensively on your code instead of all the things that come after it. Do you have any questions? Shoot me a line on Twitter at @bdougieYO, or find me on GitHub.

Additional resources

• 10 GitHub Actions resources to bookmark from the basics to CI/CD
• 5 automations every developer should be running
• 7 advanced workflow automation features with GitHub Actions

The post How to build a CI/CD pipeline with GitHub Actions in four simple steps appeared first on The GitHub Blog.

Remember, release candidates are a way for you to try the latest features at the earliest time, and they help us gather feedback early to ensure the release works in your environment. They should be tested on non-production environments.

Download the release candidate now or read more about the release candidate process.

Clean lines and and an edge that shines

In the last release of GitHub Enterprise Server, we introduced the much-awaited dark and dimmed themes, and we continue to work on providing options to help you treat your eyes with kindness while keeping them on the task at hand–introducing high contrast dark theme!

For more information on changing your theme, see “Managing your theme settings.”

One, then gone…

Sometimes, we need each job to be run on a new, clean environment. Managing the cycling, registration, and de-registration of runners and automatically scaling runners is now easier with support for ephemeral (single job) runners and a new workflow_job webhook.

Empowering Security Teams with a new role and new permissions

Security is a cross-cutting concern, and professionals responsible for keeping organizations safe need the right access applied consistently and easily across organizations and repositories. The new security manager role addresses these needs allowing the specified teams’ members to manage security alerts and settings across your organization, as well as read permission for all repositories in the organization.

• Read access on all repositories in the organization.
• Write access on all security alerts in the organization.
• Access to the organization-level security tab.
• Write access on security settings at the organization level.
• Write access on security settings at the repository level.

For more information, see “Managing security managers in your organization.”

The security conscious will also welcome the addition of the option to set an expiration date for personal access tokens, new and existing. User renewals will be requested by email and can easily be regenerated with the same properties as the original. When using a personal access token with the GitHub API, a new GitHub-Authentication-Token-Expiration header is included in the response, which indicates the token’s expiration date. For more information, see “Creating a personal access token” or check out the release notes for other security related changes

Try it out today

To learn more about GitHub Enterprise Server 3.3, read the release notes and download it now.

Release candidates should be installed on non-production environments. GitHub Support is here to help with any problems, and hear your feedback.

Not using GitHub Enterprise Server already? Start a free trial to innovate faster with the platform developers know and love.

The post GitHub Enterprise Server 3.3 enhances CI/CD and adds a new security manager role appeared first on The GitHub Blog.