Test automation with Playwright for VS Code extensions
Choosing a better test automation framework for automating tests for VS Code plugins like Data Editor for Mainframe or COBOL Control Flow or others is not so straightforward. The challenge is, how to run those automated tests for some specific test-scenarios, e.g. to find a <iframe> tag and test it for some inputs.
In this blog I will describe why the Playwright framework is a better choice to write automated tests for VS Code extensions compared to Cypress. As a case study, I will give some working examples on how to write and run Playwright tests for VS Code extensions and mention some tools which will be helpful to debug, trace and generate the tests.
Modern Coding & Debugging Experiences Using VS Code for Mainframe Software
As a part of modern tooling and interfaces support for mainframe modernization, Broadcom offers a set of VS Code extensions, called Code4z, for mainframe developers working with z/OS.
In my earlier blogs, I have described the power of the LSP[1] (provides language-smartness features in modern IDEs) and DAP[2] (brings debugging experience) architecture and how these new specifications can be leveraged to deliver a user friendly, consistent experience to developers working on mainframe code (e.g. COBOL program with CICS and DB2 statements) regardless of IDE (VSCode, GitHub Codespaces[3], Gitpod OpenVSCode Server[4], etc.)
To ensure all these extensions are working fine (verification), we need a mechanism to regularly execute automated tests of these extensions. Our preference: Playwright[5] — a simple test automation tool.
Automation Testing Tools — Cypress.io
Automation testing tools are applications (e.g. Selenium, Cypress.io, Puppeteer or Playwright and etc.) designed to verify functional or non-functional requirements via automated test scripts. Being successful in any test automation process, depends on identifying the right tool for the project. Tools differ very much from each other in their implementation, philosophy, applicability and API. Also, to better understand the different solutions and trying them on your product is time-consuming.
Let’s briefly mention Cypress. It is a free and open source automation tool, offering a complete end-to-end testing experience. Cypress is very convenient to create, run and debug tests. Using Cypress you can test anything that runs in a browser.
We have been using Cypress to automate the testing of all code4z extensions, and the tests successfully run on the Theia editor — a browser based IDE. But as we now have new VS Code extensions like Data Editor for Mainframe or COBOL Control Flow, then some new challenges started. We could not write test-cases for both extensions which could be testing elements in different frames (<iframe> in html). The reason is, Cypress has some limitations in cases when you have test-cases inter-connected with points below:
- Limited support for iFrames.
- Cypress only supports JavaScript for creating test cases.
- One cannot use Cypress to drive two browsers at the same time.
- It doesn’t provide support for multi-tabs.
- Currently it doesn’t provide support for browsers like Safari and IE.
To resolve these limitations, different work arounds are possible but this is not a long-term or a complete and stable solution. A better alternative is to use a different testing framework like: Puppeteer by Google or Playwright by Microsoft which handles the above limitations and offers better built-in solutions.
Why is Playwright a better choice?
The answer is simple. We can write any-tests with Playwright, including the ones which could not be done using Cypress ;)
Playwright is a new open-source cross-browser, cross platform [Win, MacOS, and Linux] end-to-end test-automation framework. Written by some of the same people who authored Puppeteer framework.
It offers:
❖ Powerful automation capabilities
❖ Cross-language. Use the Playwright API in TypeScript, JavaScript, Python, .NET, Java.
❖ Execute tests in parallel.
❖ Enjoy context isolation out of the box.
❖ Capture videos, screenshots and other artifacts on failure.
❖ Integrate your POMs as extensible fixtures.
❖ Powerful Tooling — Codegen, Playwright Inspectors, Trace Viewer
With limitations like:
- No legacy MS Edge or IE11 support
- Desktop browsers are used to emulate mobile devices (test with real devices yourself ;)
- Quite new tool, so less community support and docs
Getting Started with Playwright
Let’s create a test-project called “dataeditor”, open your console and type: >npm init playwright dataeditor
It will question you to choose some answers, and based on that it automatically starts installing the playwright tool-set with all supported browsers and will end-up creating a node.js project like below
It has default configuration file called “playwright.config.ts” where you can tune your test requirements like: timeouts, browsers involved in the test, headless mode, storage state, testing options like retries, worker count, emulation of mobile devices or time zones, automatic screenshots, record video and tracing options and many others.
Here is the step-by-step in-detail guide to start writing Playwright tests — Getting Started, or look at the playground to play with quick testing.
Writing Playwright Automated Test for given Test-Case
Let’s assume we have given a below test-case for Data Editor for Mainframe VS Code extension
Step-1: Setup your development environment to run the Playwright tests (see below section dedicated to this point)
//open Zowe explorer and search for data set
Step-2: Zowe should be installed (prerequisite)
Step-3: Add a Zowe profile using ‘<server-name>.broadcom.net:443’ with your mainframe-account
Step-4: Search for data-set ‘AD1QA.FMMVS90.GRAKE02.VSAM.TEST’
Step-5: Press Enter
//add FMP server into configuration
Step-6: Right click VS Code gear-settings-button and click on Settings
Step-7: In ‘Search Setting’ input field, search for ‘Data Editor for Mainframe’ and press enter
Step-8: Specify the URL of your Testing Tools Server instance. Use the format http(s)://host:port
Expected result — your testing-tool server address is saved
//browse data-set with data-editor
Step-8: Right click on your dataset ‘AD1QA.FMMVS90.GRAKE02.VSAM.TEST’
Expected result — you see the ‘Browse With Data Editor for Mainframe’ in the list
Step-9: Click on ‘Browse With Data Editor for Mainframe’
Expected result — As you see, dataset is opened in HEX mode and on the right-corner you see the frames (iframes in html, which causes an issue in Cypress) for Options.
//click options and use layout to change to single mode
Step-10: Find the ‘Options’ frame (this is the <iframe> element in-deed, try VS Code developer-tool to inspect the source-code ;) )
Step-11: Put the value ‘AD1QA.FMMVS90.LAYOUT(DEMO)’ in input-layout field
Step-12: Press enter
Expected result — Single mode button is enabled
Step-13: Click ‘Single’ button to see how application changes the format of dataset
Expected result — Mode is changed from Hex to Single
Writing automated tests for a given test-case is quite straightforward once you follow the instructions described there: just find the elements and apply the actions (click, fill, etc.) according to use-case, and provide respective expectations (assertions). You can also use playwright codegen tool (mentioned in last section) which helps you to generate code for your tests.
Respective automated-test to above test-case looks like below (to make it easy, values are hard-coded here)
How to Prepare the Environment to Run the Automated Tests
To be able to run Playwright or Cypress tests, your VS Code extensions should be running on browser-like applications including: Theia, GitHub Codespaces, or Gitpod OpenVSCode Server.
In this blog, I used Gitpod OpenVSCode Server for my working-example. As a motivational exercise ;) I will leave it to you to use GitHub Codespaces for running your tests. For more about VS Code experience on GitHub Codespaces, read this nice article by our distinguished engineer Venkat Balabhadrapatruni.
Gitpod introduced an open-source project called “OpenVSCode Server” which runs the latest VS Code on a remote machine and allows access through a modern web browser.
To set up your environment you should use either Docker or Linux or use WSL 2 on Windows. I installed Ubuntu on my Windows OS using WSL 2 as in here
Also you need a latest release of ‘openvscode-server’, which you can download it from here, and untar
> tar -xzf openvscode-server-v1.65.2-linux-x64.tar.gz
> cd /mnt/c/openvscode-server-v1.65.2-linux-x64/bin/
For possible entry point arguments (see more) to configure (edit) your ‘openvscode-server’ file, for example I will add my own token here, which will keep server generating same URL for me, otherwise, for each restart of openvscode-server, it will re-generate new random-token, and this will not be so practical, user-friendly for running the tests especially in CI/CD pipeline.
Save the edited file and run run the server
Running Automated Tests for VS Code Extensions in Gitpod OpenVSCode-Server
Server is running, just open the above URL ‘http://localhost:3000/?tkn=mytoken’ on the browser and install the ‘Data Editor for Mainframe’ extension, and create a Zowe-profile.
Well done, your environment is ready to run automated-tests. Let’s open the ‘test-change-layout-value.spec.ts’ file in the IDE and and run the below playwright command in console
> npx playwright test tests/test-change-layout-value.spec.ts –headed
Playwright triggers to open a new instance of ‘OpenVSCode Server’ and opens it in the chrome-browser (modify playwright.config.ts to open in a different browser) and runs the all steps of test as described in test-case.
It opens the browser, automatically fills the required data (user-id, dataset name) for zowe and presses the enter button.
Then it adds FMP server into configuration and saves it.
Browses dataset using data-editor-extension, and changes the format of layout via providing new value in ‘Layout’ and presses the ‘Single’ button. As expected, char mode is changed from hex to single.
All steps are done accordingly and test runs successfully. Test report is available in html-format (can be configured as other formats like json, dot, github etc.) in the shown URL. Tests can be run even faster if you use the ‘headless’ browser mode.
Playwright Powerful Tools
You can also use playwright codegen tool which helps you to generate tests. Very easy to find the html-elements e.g. “<iframe>”, or nested “<div>”, and respective actions like ‘double click’, ‘save’ etc. Then you just copy the generated code and be able to use it in your tests. This makes your life easier.
⮚ npx playwright codegen http://localhost:3000/?tkn=mytoken
Besides generating the test-code, codegen-tool can be used to emulate-devices or color-schemes, geolocation, timezone/local and for many others like for timeouts, saving cookies etc.
> npx playwright codegen — device=”iPhone 11" sahet.net
> npx playwright codegen — save-storage=auth
Use playwright and debugging tools, devtool, and playwright inspector to debug your tests:
and show-tracer can be enabled as like below:
Summary
In this article, I described the why and how to use Playwright framework to write automated tests for VS Code extensions with running examples in Gitpod. If you develop VS Code extensions then this article should be a motivation to you to write automation tests for your application. f you would like to contribute to open source projects, then I welcome you to COBOL Language Support or any other extensions which are the part of the all-in-one VS Code extension pack — Code4z
For further questions and feedback you may contact us on Slack: che4z.slack.com
References
[1] https://microsoft.github.io/language-server-protocol/specifications/specification-current/
[2] https://microsoft.github.io/debug-adapter-protocol/specification
[3] https://medium.com/modern-mainframe/github-codespaces-for-mainframe-development-e0f2145990d0
